Towards higher efficiency in a distributed memory storage system using data compression

被引:0
作者
Yu, Xiaoyang [1 ,2 ]
Lu, Songfeng [1 ]
Wang, Tongyang [1 ]
Zhang, Xinfang [3 ]
Wan, Shaohua [4 ]
机构
[1] Huazhong Univ Sci & Technol, Hubei Engn Res Ctr Big Data Secur, Sch Cyber Sci & Engn, Wuhan 430074, Peoples R China
[2] Huazhong Univ Sci & Technol, Sch Comp Sci & Technol, Wuhan 430074, Peoples R China
[3] Huazhong Univ Sci & Technol, Sch Mech Sci & Engn, Wuhan 430074, Peoples R China
[4] Univ Elect Sci & Technol China, Shenzhen Inst Adv Study, Shenzhen 518110, Peoples R China
关键词
data storage systems; system performance; data compression; distributed memory storage; ALGORITHM;
D O I
10.1504/IJBIC.2022.128090
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
As the amount of data grows, achieving an appropriate trade-off among computation, storage and network transportation will be beneficial for a distributed memory storage system, leading to higher overall efficiency. To this end, we explore a method to achieve this trade-off by introducing data compression technology in a transparent manner. Instead of focusing on specific compressed data structures, we target block level compression for a general-purpose storage system to incorporate a wide range of existing data analysis frameworks and usage scenarios, especially with big data. A prototype is implemented and evaluated based on the memory-centric distributed storage system Alluxio to provide transparent compression and decompression during write/read operations. The extensive experiments for data with different types of compression ratio are conducted and the experimental results prove that our approach can achieve huge write/read throughput.
引用
收藏
页码:232 / 240
页数:10
相关论文
共 28 条
[1]  
[Anonymous], 2014, P ACM S CLOUD COMP, DOI [10.1145/2670979.2670985, DOI 10.1145/2670979.2670985]
[2]  
[Anonymous], 2010, ACM SIGOPS Operating Systems Review, DOI DOI 10.1145/1713254.1713276
[3]   An ensemble bat algorithm for large-scale optimization [J].
Cai, Xingjuan ;
Zhang, Jiangjiang ;
Liang, Hao ;
Wang, Lei ;
Wu, Qidi .
INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS, 2019, 10 (11) :3099-3113
[4]   Multi-Objective Three-Dimensional DV-Hop Localization Algorithm With NSGA-II [J].
Cai, Xingjuan ;
Wang, Penghong ;
Du, Lei ;
Cui, Zhihua ;
Zhang, Wensheng ;
Chen, Jinjun .
IEEE SENSORS JOURNAL, 2019, 19 (21) :10003-10015
[5]   An Intelligent Platooning Algorithm for Sustainable Transportation Systems in Smart Cities [J].
Chen, Chen ;
Zhang, Yuru ;
Khosravi, Mohammad R. ;
Pei, Qingqi ;
Wan, Shaohua .
IEEE SENSORS JOURNAL, 2021, 21 (14) :15437-15447
[6]   Data Prefetching and Eviction Mechanisms of In-Memory Storage Systems Based on Scheduling for Big Data Processing [J].
Chen, Chien-Hung ;
Hsia, Ting-Yuan ;
Huang, Yennun ;
Kuo, Sy-Yen .
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 2019, 30 (08) :1738-1752
[7]   A long video caption generation algorithm for big video data retrieval [J].
Ding, Songtao ;
Qu, Shiru ;
Xi, Yuling ;
Wan, Shaohua .
FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2019, 93 :583-595
[8]  
Ding XW, 2015, 2015 Third International Conference on Digital Information, Networking, and Wireless Communications (DINWC), P160, DOI 10.1109/DINWC.2015.7054235
[9]   Adaptive Fusion and Category-Level Dictionary Learning Model for Multiview Human Action Recognition [J].
Gao, Zan ;
Xuan, Hai-Zhen ;
Zhang, Hua ;
Wan, Shaohua ;
Choo, Kim-Kwang Raymond .
IEEE INTERNET OF THINGS JOURNAL, 2019, 6 (06) :9280-9293
[10]  
Gu R., 2016, UCBEECS2016133