Data Integration Patterns for Data Warehouse Automation

被引：5
|
作者：
Tomingas, Kalle [1 ]
Kliimask, Margus [2 ]
Tammet, Tanel [1 ]
机构：
[1] Tallinn Univ Technol, EE-19086 Tallinn, Estonia
[2] Eliko Competence Ctr, EE-12618 Tallinn, Estonia
来源：
NEW TRENDS IN DATABASE AND INFORMATION SYSTEMS II | 2015年 / 312卷
关键词：
data warehouse; etl; data mappings; template based sql generation; abstract syntax patterns; metadata management;
D O I：
10.1007/978-3-319-10518-5_4
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
The paper presents a mapping-based and metadata-driven modular data transformation framework designed to solve extract-transform-load (ETL) automation, impact analysis, data quality and integration problems in data warehouse environments. We introduce a declarative mapping formalization technique, an abstract expression pattern concept and a related template engine technology for flexible ETL code generation and execution. The feasibility and efficiency of the approach is demonstrated on the pattern detection and data lineage analysis case studies using large real life SQL corpuses.
引用
收藏
页码：41 / 55
页数：15
相关论文
共 50 条
  • [21] AUTOMATION OF MACROMOLECULAR DATA COLLECTION - INTEGRATION OF DATA COLLECTION AND DATA PROCESSING
    Powell, H.
    Leslie, A. G. W.
    Winter, G.
    Nave, C.
    Duke, E. M. H.
    Kinder, S. H.
    Love, D.
    McSweeney, S.
    Svensson, O.
    Spruce, D.
    Delageniere, S.
    ACTA CRYSTALLOGRAPHICA A-FOUNDATION AND ADVANCES, 2002, 58 : C301 - C301
  • [22] VINEdb: a data warehouse for integration and interactive exploration of life science data
    Hariharaputran, Sridhar
    Toepel, Thoralf
    Brockschmidt, Bjoern
    Hofestaedt, Ralf
    JOURNAL OF INTEGRATIVE BIOINFORMATICS, 2007, 4 (03):
  • [23] Data Warehouse Architect: Migrating legacy data into a data warehouse
    DBMS, 7 (14):
  • [24] YEARBOOK DATA INTEGRATION BASED ON COMMON WAREHOUSE MODEL
    Miao, Gaimei
    Gou, Juanqiong
    ICEIS 2011: PROCEEDINGS OF THE 13TH INTERNATIONAL CONFERENCE ON ENTERPRISE INFORMATION SYSTEMS, VOL 4, 2011, : 569 - 573
  • [25] InterMine: a flexible data warehouse system for the integration and analysis of heterogeneous biological data
    Smith, Richard N.
    Aleksic, Jelena
    Butano, Daniela
    Carr, Adrian
    Contrino, Sergio
    Hu, Fengyuan
    Lyne, Mike
    Lyne, Rachel
    Kalderimis, Alex
    Rutherford, Kim
    Stepan, Radek
    Sullivan, Julie
    Wakeling, Matthew
    Watkins, Xavier
    Micklem, Gos
    BIOINFORMATICS, 2012, 28 (23) : 3163 - 3165
  • [26] Integration Materials Data between Heterogeneous Databases Based on Data Warehouse Technologies
    Yu, Gang
    Chen, Jingzhong
    2009 THIRD INTERNATIONAL SYMPOSIUM ON INTELLIGENT INFORMATION TECHNOLOGY APPLICATION, VOL 2, PROCEEDINGS, 2009, : 233 - 236
  • [27] Data warehouse integration using best fit matching
    Holmes, D
    Maxwell, D
    IKE'03: PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON INFORMATION AND KNOWLEDGE ENGINEERING, VOLS 1 AND 2, 2003, : 177 - 181
  • [28] Investigating the Integration of Supercomputers and Data-Warehouse Appliances
    Oldfield, Ron A.
    Davidson, George
    Ulmer, Craig
    Wilson, Andrew
    EURO-PAR 2013: PARALLEL PROCESSING WORKSHOPS, 2014, 8374 : 855 - 864
  • [29] Selection and classification of external information for the integration in a data warehouse
    Behme, W
    Mucksch, H
    WIRTSCHAFTSINFORMATIK, 1999, 41 (05): : 443 - +
  • [30] Simulation Data Warehouse for Integration and Analysis of Disaster Information
    Zhao, Jing
    Sugiura, Kento
    Wang, Yuanyuan
    Ishikawa, Yoshiharu
    JOURNAL OF DISASTER RESEARCH, 2016, 11 (02) : 255 - 264
12345