A Machine Learning Classification Approach to Detect TLS-based Malware using Entropy-based Flow Set Features

Transport Layer Security (TLS) based malware is one of the most hazardous malware types, as it relies on encryption to conceal connections. Due to the complexity of TLS traffic decryption, several anomaly-based detection studies have been conducted to detect TLS-based malware using different features and machine learning (ML) algorithms. However, most of these studies utilized flow features with no feature transformation or relied on inefficient flow feature transformations like frequency-based periodicity analysis and outlier percentage. This paper introduces TLSMalDetect, a TLS-based malware detection approach that integrates periodicity-independent entropy-based flow set (EFS) features generated by a flow feature transformation technique to solve flow feature utilization issues in related research. The effectiveness of EFS features was evaluated in two ways: (1) by comparing them to the corresponding outlier percentage and flow features using four feature importance methods, and (2) by analyzing classification performance with and without EFS features. Moreover, new Transmission Control Protocol features not explored in the literature were incorporated into TLSMalDetect, and their contribution was assessed. This study's results proved that EFS features of the number of packets sent and received were superior to the related outlier percentage and flow features and could remarkably increase the performance up to similar to 42 percent in the case of Support Vector Machine accuracy. Furthermore, using the basic features, TLSMalDetect achieved the highest accuracy of 93.69 percent by Naive Bayes (NB) among the ML algorithms applied. From a comparison view, TLSMalDetect's Random Forest precision of 98.99 percent and NB recall of 92.91 percent exceeded the best relevant findings of previous studies. These comparative results demonstrated TLSMalDetect's ability to detect more malware flows out of total malicious flows than existing works. It could also generate more actual alerts from overall alerts than earlier research.