Deep transfer learning techniques with hybrid optimization in early prediction and diagnosis of different types of oral cancer

Oral cancer is a frequent and challenging cancer that has a high fatality rate. It is the fifth most common cancer in India, with 130,000 deaths each year. There are various diagnostic techniques for oral cancer, but they are limited to accurately detecting cancer cells. As a result, deep transfer learning algorithms such as ResNet50, MobileNetV2, VGG19, VGG16, and DenseNet were applied to non-malignant and oral cancerous photographs acquired from histopathologic and real-time datasets in this study. The images were first preprocessed with techniques like Gaussian blur to eliminate noise signals. Then, features were extracted with morphological operations to produce extreme points on the images, subsequently cropped. The retrieved pictures were fed into deep transfer learning models along with optimization techniques as ADAM, SGD, and RMSprop optimizers to obtain optimal results. Accuracy, loss, RMSE, precision, and AUC were used to analyze the findings. During the validation phase, the hybrid optimization techniques using ADAM, RMSprop, and SGD on DenseNet produced effective results, with accuracy of 92.41%, loss of 0.70, and RMSE of 0.29 for the real-time dataset, accuracy of 95.41%, loss of 0.10, and RMSE of 0.03 for oral cancer images, and accuracy of 92.41%, loss of 0.30, and RMSE of 0.09 for non-cancerous images taken from the histopathologic dataset.