Deep learning and computer vision methods are increasingly integrated to help diagnose breast cancer. This paper proposes the classification of Mammogram Images utilizing a Convolutional Neural Network with Local Window Attention Vision Transformer and Honey Badger Optimization (HBO) algorithm (CNN-LWAViT-HBO). Here, the input images are taken through two datasets, they are Chinese Mammography dataset (CMD) and real-time dataset (LHD) from Lakeshore Hospital, Kochi, India. An Adaptive Variational Bayesian Filter (AVBF) is applied for noise elimination during data preprocessing to enhance the image quality. After that, the images are fed to the classification process using Convolutional Neural Network with LWAViT. The Honey Badger Optimization (HBO) algorithm is used to optimize the weight parameters. Finally, Smooth Grad-Cam ++ is utilized to visualize regions of an output image as benign and malignant. The performance metrics, like Sensitivity, Accuracy, and Specificity are considered to confirm the effectiveness of the proposed method. The Chinese mammography dataset and real-time dataset demonstrate the exceptional accuracy of 99.1% and 99.67%, respectively, outperforms the existing methods, like deep learning in Mammography Images Segmentation with Categorization (DL-MISC), Framework for Breast Cancer Classification utilizing Multi-DCNNs (BCC-MDCNN), Medical Image Enhancement Algorithm for Breast Cancer Detection on Mammography Images using Machine Learning (BCC-ML).