Data Column Prediction: Experiment in Automated Column Tagging Using Machine Learning

The lack of tools to rapidly identify and align data from different sources is a critical, needed capability for the Department of Defense especially when it comes to automated ingestion. In the current open source Karma Mapping Tool, the Steiner tree optimization algorithm suggests semantic types during data alignment. We hypothesize that Machine Learning (ML) may perform better than the Steiner approach on a subset of column types, or "labels", where 1.) the data is extremely similar in structure and content and 2.) inferring column type correctly is highly dependent on the interrelated components of the dataset. In this session we discuss the experimental design, our initial results, and a path toward future work in broader applications beginning with intelligence analysis in the maritime domain. The initial results from this experiment show there is promise in using ML to do column prediction in analysis environments where there are many similar or overlapping data.