A cloud-based architecture for explainable Big Data analytics using self-structuring Artificial Intelligence

Big Data is steadily expanding beyond the boundaries of its foundational constructs of three primary Vs, Volume, Velocity and Variety, and two secondary Vs, Veracity and Value. The advent of 5G networks, Edge computing and IoT technologies has transformed Big Data into this modern context. With these new manifestations of Big Data, the focus is not only on the data itself but on the context that it applies to its immediate environment as well as the human and societal perception of this context. It is increasingly challenging for conventional AI algorithms to process and transform this data, analyse and visualise a broad spectrum of insights, and then formulate the explainability of such insights in terms of bias, transparency, safety, ethics, and causality. Self-structuring Artificial Intelligence (SSAI) addresses the limitations of conventional AI by adapting to the inherent structure of the data, incrementally learning and abstracting from this structure. SSAI has not been investigated in a cloud-based setting for generating explainable insights from these new types of Big Data. In this paper we propose a cloud-based architecture for explainable Big Data analytics using SSAI in highly-connected 5G and Edge computing environments. The proposed architecture is empirically evaluated on a commercial scale Big Data use case of Smart Grid for Smart Cities. The results of these experiments confirm the functionality and effectiveness of the proposed architecture. © The Author(s) 2024.