Research and design of data processing based on ETL framework

被引：0

作者：

Guo, Xiao-Li ^{[1
]}

Chen, Bo ^{[2
]}

机构：

[1] China Univ Min & Technol Beijing, Sch Mech Elect & Informat Engn, Beijing, Peoples R China

[2] Beijing Univ Technol, Sch Software Engn, Beijing, Peoples R China

来源：

MODERN TECHNOLOGIES IN MATERIALS, MECHANICS AND INTELLIGENT SYSTEMS | 2014年 / 1049卷

关键词：

Data processing; data extraction; data conversion; data loading; ETL framework;

D O I：

10.4028/www.scientific.net/AMR.1049-1050.1966

中图分类号：

T [工业技术];

学科分类号：

08 ;

摘要：

ETL is a key link in the construction of data warehouse. On the base of analyzing the mainstream ETL tool Datastage, the data extraction, transformation and loading, proposes a ETL framework based on data processing, and the realization method and steps are discussed in detail. The framework uses HIVE as a data processing station, improve the operating efficiency of the file; data task according to the E, T and L three parts and hierarchical partitioning, conversion of data users to better grasp the process; development data using the configuration file of the task, the development personnel free out from the heavy code, will to shift the focus of the work to the data logical task, which has greatly improved the efficiency of development personnel data processing.

引用

页码：1966 / +

页数：2