High performance RDMA-based MPI implementation over InfiniBand

被引:149
作者
Liu, JX [1 ]
Wu, JS [1 ]
Panda, DK [1 ]
机构
[1] Ohio State Univ, Columbus, OH 43210 USA
关键词
parallel programming; MPI; InfiniBand; RDMA; clusters; parallel computing;
D O I
10.1023/B:IJPP.0000029272.69895.c1
中图分类号
TP301 [理论、方法];
学科分类号
081202 ;
摘要
Although InfiniBand Architecture is relatively new in the high performance computing area, it offers many features which help us to improve the performance of communication subsystems. One of these features is Remote Direct Memory Access (RDMA) operations. In this paper, we propose a new design of MPI over InfiniBand which brings the benefit of RDMA to not only large messages, but also small and control messages. We also achieve better scalability by exploiting application communication pattern and combining send/receive operations with RDMA operations. Our RDMA-based MPI implementation achieves a latency of 6.8 musec for small messages and a peak bandwidth of 871 million bytes/sec. Performance evaluation shows that for small messages, our RDMA-based design can reduce the latency by 24%, increase the bandwidth by over 104%, and reduce the host overhead by up to 22% compared with the original design. For large data transfers, we improve performance by reducing the time for transferring control messages. We have also shown that our new design is beneficial to MPI collective communication and NAS Parallel Benchmarks.
引用
收藏
页码:167 / 198
页数:32
相关论文
共 26 条
[1]  
Alfaro F. J., 2002, INT PAR DISTR PROC S
[2]   MPI-LAPl: An efficient implementation of MPI for IBM RS/6000 SP systems [J].
Banikazemi, M ;
Govindaraju, RK ;
Blackmore, R ;
Panda, DK .
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 2001, 12 (10) :1081-1093
[3]   VIRTUAL-MEMORY-MAPPED NETWORK INTERFACES [J].
BLUMRICH, MA ;
DUBNICKI, C ;
FELTEN, EW ;
LI, K ;
MESARINA, MR .
IEEE MICRO, 1995, 15 (01) :21-28
[4]  
BRIGHTWELL R, 1996, 1996 MPI DEV C JUL
[5]  
Carrera EV, 2002, EIGHTH INTERNATIONAL SYMPOSIUM ON HIGH-PERFORMANCE COMPUTER ARCHITECTURE, PROCEEDINGS, P275
[6]  
CULLER D, 1993, P 4 ACM SIGPLAN S PR, P1
[7]  
DIMITROV R, 1998, EFFICIENT MPI IMPLEM, V6
[8]   The virtual interface architecture [J].
Dunning, D ;
Regnier, G ;
McAlpine, G ;
Cameron, D ;
Shubert, B ;
Berry, F ;
Merritt, AM ;
Gronke, E ;
Dodd, C .
IEEE MICRO, 1998, 18 (02) :66-76
[9]   A high-performance, portable implementation of the MPI message passing interface standard [J].
Gropp, W ;
Lusk, E ;
Doss, N ;
Skjellum, A .
PARALLEL COMPUTING, 1996, 22 (06) :789-828
[10]  
Gupta R., 2003, INT PAR DISTR PROC S