Graphical Flow-based Spark Programming

被引:0
|
作者
Tanmaya Mahapatra
Christian Prehofer
机构
[1] Technische Universität München,Lehrstuhl für Software und Systems Engineering, Fakultät für Informatik
来源
Journal of Big Data | / 7卷
关键词
Spark pipelines; IoT mashup tools; Graphical tools; Stream analytics; Flow-based programming;
D O I
暂无
中图分类号
学科分类号
摘要
Increased sensing data in the context of the Internet of Things (IoT) necessitates data analytics. It is challenging to write applications for Big Data systems due to complex, highly parallel software frameworks and systems. The inherent complexity in programming Big Data applications is also due to the presence of a wide range of target frameworks, with different data abstractions and APIs. The paper aims to reduce this complexity and its ensued learning curve by enabling domain experts, that are not necessarily skilled Big Data programmers, to develop data analytics applications via domain-specific graphical tools. The approach follows the flow-based programming paradigm used in IoT mashup tools. The paper contributes to these aspects by (i) providing a thorough analysis and classification of the widely used Spark framework and selecting suitable data abstractions and APIs for use in a graphical flow-based programming paradigm and (ii) devising a novel, generic approach for programming Spark from graphical flows that comprises early-stage validation and code generation of Spark applications. Use cases for Spark have been prototyped and evaluated to demonstrate code-abstraction, automatic data abstraction interconversion and automatic generation of target Spark programs, which are the keys to lower the complexity and its ensued learning curve involved in the development of Big Data applications.
引用
收藏
相关论文
共 50 条
  • [1] Graphical Flow-based Spark Programming
    Mahapatra, Tanmaya
    Prehofer, Christian
    JOURNAL OF BIG DATA, 2020, 7 (01)
  • [2] aFlux: Graphical flow-based data analytics
    Mahapatra, Tanmaya
    Prehofer, Christian
    SOFTWARE IMPACTS, 2019, 2
  • [3] On a Flow-Based Paradigm in Modeling and Programming
    Al-Fedaghi, Sabah
    INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 2015, 6 (06) : 209 - 217
  • [4] Flow-Based Programming for Machine Learning
    Mahapatra, Tanmaya
    Banoo, Syeeda Nilofer
    FUTURE INTERNET, 2022, 14 (02)
  • [5] A Flow-Based Programming Environment for Geometrical Construction
    Nakamura, Kento
    Ahara, Kazushi
    MATHEMATICAL SOFTWARE - ICMS 2020, 2020, 12097 : 426 - 431
  • [6] StatWire: Visual Flow-based Statistical Programming
    Subramanian, Krishna
    Maas, Johannes
    Ellers, Michael
    Wacharamanotham, Chat
    Voelker, Simon
    Borchers, Jan
    CHI 2018: EXTENDED ABSTRACTS OF THE 2018 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS, 2018,
  • [7] Securing the Flow: Security and Privacy Tools for Flow-based Programming
    Ioannidis, Thodoris
    Bolgouras, Vaios
    Politis, Ilias
    Xenakis, Christos
    18TH INTERNATIONAL CONFERENCE ON AVAILABILITY, RELIABILITY & SECURITY, ARES 2023, 2023,
  • [8] Flow-based Attribution in Graphical Models: A Recursive Shapley Approach
    Singal, Raghav
    Michailidis, George
    Ng, Hoiyi
    INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 139, 2021, 139
  • [9] Patterns for visual programming With a focus on flow-based programming inspired systems
    Schmidt, Marc
    PROCEEDINGS OF THE EUROPEAN CONFERENCE ON PATTERN LANGUAGES OF PROGRAMS 2021, EUROPLOP 2021, 2021,
  • [10] Flow-based programming for IoT leveraging fog computing
    Szydlo, Tomasz
    Brzoza-Woch, Robert
    Sendorek, Joanna
    Windak, Mateusz
    Gniady, Chris
    2017 IEEE 26TH INTERNATIONAL CONFERENCE ON ENABLING TECHNOLOGIES - INFRASTRUCTURE FOR COLLABORATIVE ENTERPRISES (WETICE), 2017, : 74 - 79