Managing Heterogeneous Data on a Big Data Platform: A Multi-Criteria Decision Making Model for Data-Intensive Science

This paper presents an approach to solving data variety problem of big data through an offline and online decision-making system. We present a graph-based approach to imitate real-world problem domain with a set of criteria and problem solvers. We introduce a Multi-criteria decision-making model to select a set of problem solvers that meets the set of criteria most. Suppose a system is processing Twitter data that comes as a stream of JSON records from multiple data sources. The decision system determines which of the available methods to use for a list of requirements (criteria). When multiple criteria (must meet requirements) coexist in a problem domain, their order of importance against the criteria, the mutual influence on each other and level of indispensability forms a graphic structure. In the proposed model, we consider each vertex of the graph as a criterion or benefit of an agent against the criterion. The mutual influence of multiple agents is denoted by the connecting edges of the graph. We also proposed a fuzzy graph framework to model real-world unpredictability. The model produces benchmarking results for each of the problem solvers in terms of absolute values to support decision making. The model is implemented through TopBread, Resource Description Framework (RDF), and RDF Data Query Language (RDQL). The key advantage of the proposed model over the existing ones is that the framework can operate in a dual mode both as a standalone offline tool and as an online decision-making gateway, it can also be used in high-velocity ingestion scenarios.