Research and design of data processing based on ETL framework

被引：0

作者：

Guo, Xiao-Li ^{[1
]}

Chen, Bo ^{[2
]}

机构：

[1] China Univ Min & Technol Beijing, Sch Mech Elect & Informat Engn, Beijing, Peoples R China

[2] Beijing Univ Technol, Sch Software Engn, Beijing, Peoples R China

来源：

MODERN TECHNOLOGIES IN MATERIALS, MECHANICS AND INTELLIGENT SYSTEMS | 2014年 / 1049卷

关键词：

Data processing; data extraction; data conversion; data loading; ETL framework;

D O I：

10.4028/www.scientific.net/AMR.1049-1050.1966

中图分类号：

T [工业技术];

学科分类号：

08 ;

摘要：

ETL is a key link in the construction of data warehouse. On the base of analyzing the mainstream ETL tool Datastage, the data extraction, transformation and loading, proposes a ETL framework based on data processing, and the realization method and steps are discussed in detail. The framework uses HIVE as a data processing station, improve the operating efficiency of the file; data task according to the E, T and L three parts and hierarchical partitioning, conversion of data users to better grasp the process; development data using the configuration file of the task, the development personnel free out from the heavy code, will to shift the focus of the work to the data logical task, which has greatly improved the efficiency of development personnel data processing.

引用

页码：1966 / +

页数：2

共 50 条

[1] UnifiedViews: An ETL Framework for Sustainable RDF Data Processing
Knap, Tomas
Kukhar, Maria
Machac, Bohuslav
Skoda, Petr
Tomes, Jiri
Vojt, Jan
SEMANTIC WEB: ESWC 2014 SATELLITE EVENTS, 2014, 8798 : 379 - 383
[2] ETL process design and quality control research based on radar spatial data
Chen, Xuejun
PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON INTELLIGENT SYSTEMS AND KNOWLEDGE ENGINEERING (ISKE 2007), 2007,
[3] A Quality-based ETL Design Evaluation Framework
El Akkaoui, Zineb
Vaisman, Alejandro
Zimanyi, Esteban
PROCEEDINGS OF THE 21ST INTERNATIONAL CONFERENCE ON ENTERPRISE INFORMATION SYSTEMS (ICEIS), VOL 1, 2019, : 249 - 257
[4] Research on Data Integration Based on ETL and ODS
Yang, Bin
Li, Huihui
2011 INTERNATIONAL CONFERENCE ON FUTURE COMPUTERS IN EDUCATION (ICFCE 2011), VOL III, 2011, : 498 - 500
[5] A framework for the design of ETL scenarios
Vassiliadis, P
Simitsis, A
Georgantas, P
Terrovitis, M
ADVANCED INFORMATION SYSTEMS ENGINEERING, PROCEEDINGS, 2003, 2681 : 520 - 535
[6] RESEARCH ON THE DESIGN OF WEB DATA WAREHOUSE BASED ON ETL META DATA MODEL AND PARTICLE SWARM OPTIMISATION
Jun-Zhou, Li
Nan, Yu
JOURNAL OF THE BALKAN TRIBOLOGICAL ASSOCIATION, 2016, 22 (02): : 1184 - 1192
[7] Research on Data Integration of Credit Cooperative Based on ETL
Yang, Bin
Wang, Lei
2010 INTERNATIONAL CONFERENCE ON MANAGEMENT SCIENCE AND ENGINEERING (MSE 2010), VOL 3, 2010, : 290 - 293
[8] A BPMN-Based Design and Maintenance Framework for ETL Processes
El Akkaoui, Zineb
Zimanyi, Esteban
Mazon, Jose-Norberto
Trujillo, Juan
INTERNATIONAL JOURNAL OF DATA WAREHOUSING AND MINING, 2013, 9 (03) : 46 - 72
[9] NMSTREAM: A SCALABLE EVENT-DRIVEN ETL FRAMEWORK FOR PROCESSING HETEROGENEOUS STREAMING DATA
Xiao, Fei
Li, Chengming
Wu, Zheng
Wu, Yinghao
ISPRS TC IV MID-TERM SYMPOSIUM 3D SPATIAL INFORMATION SCIENCE - THE ENGINE OF CHANGE, 2018, 4-4 : 243 - 246
[10] A Design of ETL for the Construction of Traffic Network Based on Big Data
Liu, Qinan
3RD INTERNATIONAL CONFERENCE ON APPLIED ENGINEERING, 2016, 51 : 451 - 456

← 1 2 3 4 5 →