DBStream: A holistic approach to large-scale network traffic monitoring and analysis

被引:18
作者
Baer, Arian [1 ]
Casas, Pedro [2 ]
D'Alconzo, Alessandro [2 ]
Fiadino, Pierdomenico [3 ]
Golab, Lukasz [4 ]
Mellia, Marco [5 ]
Schikuta, Erich [6 ]
机构
[1] FTW Forschungszentrum Telekommunikat Wien, Donau City St 1, A-1220 Vienna, Austria
[2] Austrian Inst Technol GmbH, AIT, Vienna, Austria
[3] EURECAT Technol Ctr Catalonia, Ave Diagonal 177,Planta 9, Barcelona 08018, Spain
[4] Univ Waterloo, 200 Univ Ave West, Waterloo, ON, Canada
[5] Politecn Torino, Corso Duca Abruzzi 24, I-10129 Turin, Italy
[6] Univ Vienna, Waehringerstr 29, A-1090 Vienna, Austria
关键词
Network monitoring; Data stream warehouse; Machine-to-machine traffic; On-line traffic classification; Machine learning; Cellular networks; DEGRADATION; MAPREDUCE; YOUTUBE;
D O I
10.1016/j.comnet.2016.04.020
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
In the last decade, many systems for the extraction of operational statistics from computer network interconnects have been designed and implemented. Those systems generate huge amounts of data of various formats and in various granularities, from packet level to statistics about whole flows. In addition, the complexity of Internet services has increased drastically with the introduction of cloud infrastructures, Content Delivery Networks (CDNs) and mobile Internet usage, and complexity will continue to increase in the future with the rise of Machine-to-Machine communication and ubiquitous wearable devices. Therefore, current and future network monitoring frameworks cannot rely only on information gathered at a single network interconnect, but must consolidate information from various vantage points distributed across the network. In this paper, we present DBStream, a holistic approach to large-scale network monitoring and analysis applications. After a precise system introduction, we show how its Continuous Execution Language (CEL) can be used to automate several data processing and analysis tasks typical for monitoring operational ISP networks. We discuss the performance of DBStream as compared to MapReduce processing engines and show how intelligent job scheduling can increase its performance even further. Furthermore, we show the versatility of DBStream by explaining how it has been integrated to import and process data from two passive network monitoring systems, namely METAWIN and Tstat. Finally, multiple examples of network monitoring applications are given, ranging from simple statistical analysis to more complex traffic classification tasks applying machine learning techniques using the Weka toolkit. (C) 2016 Elsevier B.V. All rights reserved.
引用
收藏
页码:5 / 19
页数:15
相关论文
共 50 条
[31]   Neural network acceleration of large-scale structure theory calculations [J].
DeRose, Joseph ;
Chen, Shi-Fan ;
White, Martin ;
Kokron, Nickolas .
JOURNAL OF COSMOLOGY AND ASTROPARTICLE PHYSICS, 2022, (04)
[32]   High Performance Attack Estimation in Large-Scale Network Flows [J].
Freas, Christopher B. ;
Harrison, Robert W. ;
Long, Yuan .
2018 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA), 2018, :5014-5020
[33]   Large-Scale Distributed Kalman Filtering via an Optimization Approach [J].
Hudoba de Badyn, Mathias ;
Mesbahi, Mehran .
IFAC PAPERSONLINE, 2017, 50 (01) :10742-10747
[34]   A Large-Scale Clinical Validation of an Integrated Monitoring System in the Emergency Department [J].
Clifton, David A. ;
Wong, David ;
Clifton, Lei ;
Wilson, Sarah ;
Way, Rob ;
Pullinger, Richard ;
Tarassenko, Lionel .
IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, 2013, 17 (04) :835-842
[35]   A Ranking Approach on Large-Scale Graph With Multidimensional Heterogeneous Information [J].
Wei, Wei ;
Gao, Bin ;
Liu, Tie-Yan ;
Wang, Taifeng ;
Li, Guohui ;
Li, Hang .
IEEE TRANSACTIONS ON CYBERNETICS, 2016, 46 (04) :930-944
[36]   An autoencoder compression approach for accelerating large-scale inverse problems [J].
Wittmer, Jonathan ;
Badger, Jacob ;
Sundar, Hari ;
Bui-Thanh, Tan .
INVERSE PROBLEMS, 2023, 39 (11)
[37]   A semiparametric graphical modelling approach for large-scale equity selection [J].
Liu, Han ;
Mulvey, John ;
Zhao, Tianqi .
QUANTITATIVE FINANCE, 2016, 16 (07) :1053-1067
[38]   A novel approach for large-scale wind energy potential assessment [J].
Dai, Tao ;
Scown, Corinne D. .
RENEWABLE & SUSTAINABLE ENERGY REVIEWS, 2025, 211
[39]   Marina: Realizing ML-Driven Real-Time Network Traffic Monitoring at Terabit Scale [J].
Seufert, Michael ;
Dietz, Katharina ;
Wehner, Nikolas ;
Geissler, Stefan ;
Schueler, Joshua ;
Wolz, Manuel ;
Hotho, Andreas ;
Casas, Pedro ;
Hossfeld, Tobias ;
Feldmann, Anja .
IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, 2024, 21 (03) :2773-2790
[40]   Centrality Approach for Community Detection in Large Scale Network [J].
Behera, Ranjan Kumar ;
Naik, Debadatta ;
Sahoo, Bibhudatta ;
Rath, Santanu Ku. .
COMPUTE 2016, 2016, :115-124