Objective: Accurate diagnosis and early treatment are crucial for survival in patients with brain metastases. This study aims to expand the capability of radiomics-based classification algorithms with novel features and compare results with deep learning -based algorithms to differentiate the subtypes of lung cancer from MRI of metastatic lesions in the brain.Methods: This study includes 75 small cell lung carci-noma, 72 squamous cell carcinoma, and 75 adenocarci-noma segments. For the radiomics- based algorithm, novel features from the original Laplacian of Gaussian filtered and two-dimensional wavelet transformed images were extracted, and a new three -stage feature selection algo-rithm was proposed for feature selection. Two classifi-cation methods were applied to images to identify the subtypes of lung cancer. Additionally, EfficientNet and ResNet with transfer learning were used as classifiers to compare the results of the proposed algorithm.Results: The sensitivity and specificity values of the radiomics- based classifier are 94.44 and 95.33%, and for the second classifier are 87.67% and 92.62%, respectively. Besides, a one-vs -all approach comparison was made utilizing two deep learning -based classifiers; The sensi-tivity and specificity values of 94.29 and 94.08% were obtained from ResNet- 50. Moreover, mentioned metrics for EfficientNet- b0 are 92.86 and 93.42%. Furthermore, the accuracies of two radiomics- based and two deep learning -based models were 84.68%, 78.37%, 92.34%, and 90.99%, respectively for one -vs -one approach.Conclusion: The results suggest that the proposed radiomics- based algorithm is a helpful diagnostic assis-tant to improve decision-making for treating patients with brain metastases in small datasets. Advances in knowledge: Firstly, the proposed method of this study extracts novel features from transforma-tions of the original images, such as wavelet and Lapla-cian of Gaussian filter for the first time in literature. Secondly, this is the first study that investigates the clas-sification performance of the shallow and deep learning approaches to identify subtypes of lung cancer.