The Service Analysis and Network Diagnosis Data Pipeline

被引:1
作者
Weitzel, Derek [1 ]
McKee, Shawn [2 ]
Bockelman, Brian Paul [3 ]
Thiltges, John [1 ]
Babik, Marian [4 ]
Vukotic, Ilija [5 ]
机构
[1] Univ Nebraska, Lincoln, NE 68588 USA
[2] Univ Michigan, Ann Arbor, MI 48109 USA
[3] Morgridge Inst Res, Madison, WI USA
[4] European Org Nucl Res, Geneva, Switzerland
[5] Univ Chicago, Chicago, IL 60637 USA
来源
PROCEEDINGS OF 8TH WORKSHOP ON INNOVATING THE NETWORK FOR DATA-INTENSIVE SCIENCE (INDIS 2021) | 2021年
关键词
network visualization; data pipeline;
D O I
10.1109/INDIS54524.2021.00006
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Modern network performance monitoring toolkits, such as perfSONAR, take a remarkable number of measurements about the local network environment. To gain a complete picture of network performance, however, one needs to aggregate data across a large number of endpoints. The Service Analysis and Network Diagnosis (SAND) data pipeline collects data from diverse sources and ingests these measurements into a message bus. The message bus allows the project to send the data to multiple consumers, including a tape archive, an Elasticsearch database, and a peer infrastructure at CERN. In this paper, we explain the architecture and evolution of the SAND data pipeline, the scale of the resulting dataset, and how it supports a wide variety of network analysis applications.
引用
收藏
页码:1 / 11
页数:11
相关论文
共 45 条
[1]  
[Anonymous], 2021, SITE RESOURCE TOPOLO
[2]  
Babik M., 2017, CERNITNOTE2017001
[3]  
Babik M., 2021, NETWORK THROUGHPUT S
[4]  
Babik M., 2021, PERFSONAR ETF MONITO
[5]  
Babik M., 2021, Experiments Test Framework (ETF), DOI 10.5281/zenodo.4674413
[6]   Lessons Learned from using the RIPE Atlas Platform for Measurement Research [J].
Bajpai, Vaibhav ;
Eravuchira, Steffie Jacob ;
Schoenwaelder, Juergen .
ACM SIGCOMM COMPUTER COMMUNICATION REVIEW, 2015, 45 (03) :35-42
[7]  
Bird I., 2014, Update of the computing models of the wlcg and the lhc experiments
[8]   WLCG and IPv6-the HEPiX IPv6 working group [J].
Campana, S. ;
Chadwick, K. ;
Chen, G. ;
Chudoba, J. ;
Clarke, P. ;
Elias, M. ;
Elwell, A. ;
Fayer, S. ;
Finnern, T. ;
Goossens, L. ;
Grigoras, C. ;
Hoeft, B. ;
Kelsey, D. P. ;
Kouba, T. ;
Munoz, F. Lopez ;
Martelli, E. ;
Mitchell, M. ;
Nairz, A. ;
Ohrenberg, K. ;
Pfeiffer, A. ;
Prelz, F. ;
Qi, F. ;
Rand, D. ;
Reale, M. ;
Rozsa, S. ;
Sciaba, A. ;
Voicu, R. ;
Walker, C. J. ;
Wildish, T. .
20TH INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP2013), PARTS 1-6, 2014, 513
[9]  
Claise B., 2004, INTERNET REQUESTS CO
[10]  
Dorigo A., 2005, WSEAS Transactions on Computers, V4, P348