Handwritten character recognition through automated techniques is one of the recent innovations in the industry, as it helps in interpreting historical documents, digital scripts, and large records. Deep learning techniques are effective in recognizing complex image patterns like handwritten Telugu scripts, and however, inherent variability in writing styles, unique characteristics, limited data pose a challenging recognition environment. Defining a robust segmentation and classification tool with intelligent deep-learning techniques is one of the possible solutions for handling the variability and challenges within handwritten character recognition. So, this paper presented an effective Telugu handwritten character segmentation and classification model for handling the challenges in recognition of variable length sequences. Initially, the handwritten images are acquired from online data sources and are inputted into the Adaptive Yolov5 (A-YoloV5) model for the segmentation process. Here, the Enhanced Chef-Based Optimization Algorithm (ECOA) is developed for improving the performance of YoloV5 platform and reduces complexity in training. Then, the Adaptive Yolov5 (A-YoloV5) segments the telugu characters from the input handwritten images. After extracting ROI, the extracted images are sent into the newly developed Convolutional Neural Networks with Residual Attention-based Long Short-Term Memory layer (CNN-RA-LSTM) model for classification of the handwritten images. This network combines the CNN and LSTM networks with added residual layers to effectively extract the sequential features and then categorize the text. The efficacy of the CNN-RA-LSTM and A-Yolov5 model is compared with recent Telugu handwritten character recognition and has resulted in 95.41 % text recognition accuracy.