Research of large scale manifold learning based on MapReduce

With the rapid development of the information technology, it is challenging for the traditional machine learning and data mining algorithms to deal with large scale explosive growth data. Manifold learning is a dimensionality reduction algorithm which can overcome some shortages of traditional linear dimensionality reduction methods. However, it is not useful for large scale data because of high complexity. In order to deal with the dimensionality reduction of large scale data, a distributed manifold learning algorithm is proposed based on MapReduce. Local sensitive hash functions are used to map the similarity points to the same bucket, then the geodesic distance between points in the same bucket can be computed by Euclidean norm according to the local homeomorphisms of Euclidean spaces of manifold and the geodesic distance among points between buckets can be computed by the modified geodesic distance formula which takes use of central points and edge points. Experiments on large scale of manmade dataset and real dataset show that this distributed manifold learning algorithm can approximate the geodesic distance between points effectively and it is useful for large scale dimensionality reduction.