Large scale MTConnect data collection

Data collection is the first stage of data lifecycle to drive manufacturing activities such as monitoring, prediction and optimization. Data collection at shop floor is more complex since it is affected by both the physical and cyber worlds. Two common issues are: difficult to collect data of various machines which support different interfaces and communication protocols, and hard to analyze inconsistent data. MTConnect provides a standard solution to collect consistent data from various machines and devices. The current MTConnect applications are based on a single computer which is not suitable to collect large-scale MTConnect data. Cloud computing technology can handle this issue easily such as Apache NiFi. In this paper, a NiFi based solution is proposed to address issues of large-scale MTConnect data collection, processing, and long-term storage. The proposed approach also provides a fault tolerant infrastructure for data provenance to ensure historical data integrity for the requirement of enterprise compliance.