Improving the performance of GIS polygon overlay computation with MapReduce for spatial big data processing

被引:48
作者
Wang, Yong [1 ]
Liu, Zhenling [1 ]
Liao, Hongyan [1 ]
Li, Chengjun [1 ]
机构
[1] China Univ Geosci, Sch Comp Sci, Wuhan 430074, Peoples R China
来源
CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS | 2015年 / 18卷 / 02期
关键词
GIS; Polygon overlay processing; MapReduce; Grid index; GPC;
D O I
10.1007/s10586-015-0428-x
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
As one of the important operations in Geographic Information System (GIS) spatial analysis, polygon overlay processing is a time-consuming task in many big data cases. In this paper, a specially designed MapReduce algorithm with grid index is proposed to decrease the running time. Our proposed algorithm can reduce the times of calling intersection computation by the aid of grid index. The experiment is carried out on the cloud framework based on Hadoop built by ourselves. Experimental results show that our algorithm with spatial grid index consumes less time than its peer without spatial index. Moreover, the proposed algorithm has an upward speed-up ratio when more nodes of Hadoop framework are used. Nevertheless, with the increase of nodes, the upward trend of speed-up ratio slows down.
引用
收藏
页码:507 / 516
页数:10
相关论文
共 50 条
[41]   Improving Hadoop MapReduce Performance with Data Compression: A Study using Wordcount Job [J].
Rattanaopas, Kritwara ;
Kaewkeeree, Sureerat .
2017 14TH INTERNATIONAL CONFERENCE ON ELECTRICAL ENGINEERING/ELECTRONICS, COMPUTER, TELECOMMUNICATIONS AND INFORMATION TECHNOLOGY (ECTI-CON), 2017, :564-567
[42]   GIS, Big Data, and a Tweet Corpus Operationalized via Natural Language Processing [J].
Corso, Anthony J. ;
Alsudais, Kareem .
AMCIS 2015 PROCEEDINGS, 2015,
[43]   XHAMI - extended HDFS and MapReduce interface for Big Data image processing applications in cloud computing environments [J].
Kune, Raghavendra ;
Konugurthi, Pramod Kumar ;
Agarwal, Arun ;
Chillarige, Raghavendra Rao ;
Buyya, Rajkumar .
SOFTWARE-PRACTICE & EXPERIENCE, 2017, 47 (03) :455-472
[44]   A Demonstration of GeoSpark: A Cluster Computing Framework for Processing Big Spatial Data [J].
Yu, Jia ;
Wu, Jinxuan ;
Sarwat, Mohamed .
2016 32ND IEEE INTERNATIONAL CONFERENCE ON DATA ENGINEERING (ICDE), 2016, :1410-1413
[45]   Geographical information system parallelization for spatial big data processing: a review [J].
Zhao, Lingjun ;
Chen, Lajiao ;
Ranjan, Rajiv ;
Choo, Kim-Kwang Raymond ;
He, Jijun .
CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS, 2016, 19 (01) :139-152
[46]   A Performance Analysis of MapReduce Task with Large Number of Files Dataset in Big Data Using Hadoop [J].
Pal, Amrit ;
Agrawal, Pinki ;
Jain, Kunal ;
Agrawal, Sanjay .
2014 FOURTH INTERNATIONAL CONFERENCE ON COMMUNICATION SYSTEMS AND NETWORK TECHNOLOGIES (CSNT), 2014, :587-591
[47]   IMPROVING WORKING LIFE SKILLS OF UNIVERSITY STUDENTS: CASE GIS DATA PROCESSING [J].
Vanhamaki, Susanna ;
Hyytiainen, Juha .
ICERI2015: 8TH INTERNATIONAL CONFERENCE OF EDUCATION, RESEARCH AND INNOVATION, 2015, :2925-2931
[48]   An Overview on the Convergence of High Performance Computing and Big Data Processing [J].
Mei, Songzhu ;
Guan, Hongtao ;
Wang, Qinglin .
2018 IEEE 24TH INTERNATIONAL CONFERENCE ON PARALLEL AND DISTRIBUTED SYSTEMS (ICPADS 2018), 2018, :1046-1051
[49]   MapReduce for Graphs Processing: New Big Data Algorithm for 2-Edge Connected Components and Future Ideas [J].
Dahiphale, Devendra .
IEEE ACCESS, 2023, 11 :54986-55001
[50]   Improving the Effectiveness of Burst Buffers for Big Data Processing in HPC Systems with Eley [J].
Yildiz, Orcun ;
Zhou, Amelie Chi ;
Ibrahim, Shadi .
FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2018, 86 :308-318