Data Prefetching for Scientific Workflow Based on Hadoop

被引：0
|
作者：
Chen, Gaozhao [1 ]
Wu, Shaochun [1 ]
Gu, Rongrong [1 ]
Xu, Yongquan [1 ]
Xu, Lingyu [1 ]
Ge, Yunwen [1 ]
Song, Cuicui [1 ]
机构：
[1] Shanghai Univ, Sch Comp Engn & Sci, Shanghai 200072, Peoples R China
来源：
COMPUTER AND INFORMATION SCIENCE 2012 | 2012年 / 429卷
关键词：
Hadoop; data-intensive; scientific workflow; prefetching;
D O I：
暂无
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
Data-intensive scientific workflow based on Hadoop needs huge data transfer and storage. Aiming at this problem, on the environment of an executing computer cluster which has limited computing resources, this paper adopts the way of data prefetching to hide the overhead caused by data search and transfer and reduce the delays of data access. Prefetching algorithm for data-intensive scientific workflow based on the consideration of available computing resources is proposed. Experimental results indicate that the algorithm consumes less response time and raises the efficiency.
引用
收藏
页码：81 / 92
页数：12
相关论文
共 50 条
  • [21] Scientific Workflow Approach (Kepler) for Carbon flux data processing
    Liu, Min
    He, Honglin
    Sun, Xiaomin
    Yu, Guirui
    ICICTA: 2009 SECOND INTERNATIONAL CONFERENCE ON INTELLIGENT COMPUTATION TECHNOLOGY AND AUTOMATION, VOL I, PROCEEDINGS, 2009, : 694 - 697
  • [22] HFetch: Hierarchical Data Prefetching for Scientific Workflows in Multi-Tiered Storage Environments
    Devarajan, Hariharan
    Kougkas, Anthony
    Sun, Xian-He
    2020 IEEE 34TH INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM IPDPS 2020, 2020, : 62 - 72
  • [23] An Efficient Data Extracting Method Based on Hadoop
    Cao, Lianchao
    Li, Zhanqiang
    Qi, Kaiyuan
    Xin, Guomao
    Zhang, Dong
    CLOUD COMPUTING (CLOUDCOMP 2014), 2015, 142 : 87 - 97
  • [24] Key based Deep Data Locality on Hadoop
    Lee, Sungchul
    Jo, Ju-Yeon
    Kim, Yoohwan
    2018 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA), 2018, : 3889 - 3898
  • [25] A Dataflow-Based Scientific Workflow Composition Framework
    Fei, Xubo
    Lu, Shiyong
    IEEE TRANSACTIONS ON SERVICES COMPUTING, 2012, 5 (01) : 45 - 58
  • [26] A system based on Hadoop for radar data analysis
    Chi Yang
    Xiaomin Yang
    Feng Yang
    Journal of Ambient Intelligence and Humanized Computing, 2019, 10 : 3899 - 3913
  • [27] A system based on Hadoop for radar data analysis
    Yang, Chi
    Yang, Xiaomin
    Yang, Feng
    JOURNAL OF AMBIENT INTELLIGENCE AND HUMANIZED COMPUTING, 2019, 10 (10) : 3899 - 3913
  • [28] Management of Unstructured Geological Data Based on Hadoop
    Wei, Dongqi
    Zhu, Yueqin
    IEEE 12TH INT CONF UBIQUITOUS INTELLIGENCE & COMP/IEEE 12TH INT CONF ADV & TRUSTED COMP/IEEE 15TH INT CONF SCALABLE COMP & COMMUN/IEEE INT CONF CLOUD & BIG DATA COMP/IEEE INT CONF INTERNET PEOPLE AND ASSOCIATED SYMPOSIA/WORKSHOPS, 2015, : 432 - 435
  • [29] Study of Hadoop Data Migration Based on Oozie
    Wu, Kehe
    An, Yanwen
    Wu, Tingting
    Zeng, Wenjing
    PROCEEDINGS OF THE 2015 JOINT INTERNATIONAL MECHANICAL, ELECTRONIC AND INFORMATION TECHNOLOGY CONFERENCE (JIMET 2015), 2015, 10 : 81 - 85
  • [30] Characterizing the Impact of Prefetching on Scientific Application Performance
    McCurdy, Collin
    Marin, Gabriel
    Vetter, Jeffrey S.
    HIGH PERFORMANCE COMPUTING SYSTEMS: PERFORMANCE MODELING, BENCHMARKING AND SIMULATION, 2014, 8551 : 115 - 135
12345