Real-Time Data ETL Framework for Big Real-Time Data Analysis

被引:0
作者
Li, Xiaofang [1 ]
Mao, Yingchi [2 ]
机构
[1] Changzhou Inst Technol, Coll Comp & Informat Engn, Changzhou, Peoples R China
[2] Hohai Univ, Coll Comp & Informat Engn, Nanjing, Jiangsu, Peoples R China
来源
2015 IEEE INTERNATIONAL CONFERENCE ON INFORMATION AND AUTOMATION | 2015年
关键词
real-time data warehouse; ETL framework; dynamic mirror; query contention; data skew;
D O I
暂无
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
In the big data era, data become more important for BI and SCADA system operation. The load cycle of traditional data warehouse is fix and longer, which cannot timely response the rapid data change. Real-time data warehouse technology, as an extension of traditional data warehouse, can capture the rapid data change and process the real-time data analysis to meet the requirements of SCADA system. The real-time data access without the processing delay is a challenging task to the real-time data warehouse. In this paper, the real-time data ETL framework is presented to separately process the historical data and real-time data. Then, combining an external dynamic storage area, a dynamic mirror replication technology was proposed to avoid the contention between OLAP queries and OLTP updates. Finally, the experiments is set up based on the TPC-H benchmark to evaluate the performance of the proposed real-time data ETL framework. The experimental results demonstrates the proposed solution to real-time data ETL can effectively mitigate the query contention and data skew.
引用
收藏
页码:1289 / 1294
页数:6
相关论文
共 12 条
  • [1] ANKORION I, 2005, DM REV MAGAZINE JAN
  • [2] Heman S., 2010, P 2010 ACM SIGMOD IN, P543
  • [3] Italiano I. C., 2006, IEEE COMPUTER MAGAZI, V8, P167
  • [4] Two-version based concurrency control and recovery in real-time client/server databases
    Kuo, TW
    Kao, YT
    Kuo, CF
    [J]. IEEE TRANSACTIONS ON COMPUTERS, 2003, 52 (04) : 506 - 524
  • [5] Lahman S., 2011, LAHMAN BASEBALL DATA
  • [6] Langseth J., REAL TIME DATA WAREH
  • [7] Lin Z, 2012, 7 INT C COMP INF TEC
  • [8] Rifaie M, 2008, PROCEEDINGS OF THE 2008 IEEE INTERNATIONAL CONFERENCE ON INFORMATION REUSE AND INTEGRATION, P58
  • [9] Stonebraker Mike., 2005, VLDB'05
  • [10] Vassiliadis P, 2009, ANN INFORM SYST, V3, P19