DroidEncoder: Malware detection using auto-encoder based feature extractor and machine learning algorithms

Android Malware detection became a hot topic over the last several years. Although considerable studies have been conducted utilizing machine learning-based methods, little attention has been dedicated to the feature extraction importance which considers an essential factor when using machine learning methods. Thus, in this study, we proposed a new feature extraction method based on the auto-encoder structure. Particularly, we propose DroidEncoder, a novel autoencoder-based model to classify Android malware applications. On the grounds of this, an image-based Android app dataset composed of 3000 malicious apps and 3000 benign apps is constructed. Then, three different auto-encoders, namely ANN-based auto-encoder, CNN-based auto-encoder, and VGG19-based auto-encoder have been proposed to extract features from the visualized Malware dataset. Three different experiments were conducted for extracting features in order to train multiple machine learning algorithms, such as decision tree, extra tree, k-nearest neighbors, LightGBM, XGBoost, Random forest, linear regression, and support vector machine. Furthermore, cross-validation alongside multiple metrics was used for evaluating the performance of the proposed models. According to the obtained results, the proposed method approved its affectivity with superior performance in terms of all metrics.