Analysis of Massive Industrial Data using MapReduce Framework for Parallel Processing

With the emergence of the 'Big Data' paradigm, more and more industrial data are now available for practitioners and professionals. This data is being generated faster due to the advancement of the new information technologies. For reliability and maintenance engineers, 'Big Data' is an interesting source of information. If analyzed correctly, it can produce useful knowledge-base to help making decisions in an industrial organization. The availability of 'Big Data' is now leading to a new area of researches that are dedicated to the analysis of such data. This paper shows how to analyze massive amount of data generated from an industrial system(s). Those massive data may range from terabytes to petabytes in size; analyzing such sizes cannot be performed on a single commodity computer due to the possibility of memory leakage as the data may not fit into the computer's resources, specifically CPUs. Even if it fits, it will take an unacceptable amount of time. For this purpose, processing industrial large size of data requires the involvement of high performance analytical systems running on distributed environments. Different algorithms can be considered to have such analysis done. Cloud Computing models provide the necessary scalable and flexible infrastructure(s) to adapt the standard analytics algorithms in a distributed manner. We introduce a new distributed training technique that combines the newly widely used framework for big dataflow, namely MapReduce, with the traditional structure of machine learning techniques such as matrix multiplication and linear regression. Parallel processing of the aforementioned types is based on different algorithms to be adapted to MapReduce and its framework. Our considered platform is deployed on top of Google Cloud Platform (App Engine and Compute Engine), also taking into consideration Cloud Amazon EMR services to see how we can benefit from the provisioned resources in each one of them, and make the analysis and the extraction of useful information from the massive industrial data goes faster, i.e. in its computational time.