Signature based Malware Detection for Unstructured Data in Hadoop

Hadoop is a very efficient distributed processing framework. It's based on map-reduce approach where the application is divided into small fragments of work, each of which may be executed on any node in the cluster. Hadoop is very efficient tool in storing and processing unstructured, semi-structured and structured data. Unstructured data usually refers to the data stored in files not in traditional row and column way. Examples of unstructured data is e-mail messages, videos, audio files, photos, web-pages, and many other kinds of business documents. Our work primarily focuses on detecting malware for unstructured data stored in Hadoop distributed file system environment. Here we use calm AV's updated free virus signature database. We also propose a fast string search algorithm based on map-reduce approach.