Hyperspectral image (HSI) classification is a significant and demanding research area in the field of remote sensing and earth observation. The effective extraction and utilization of texture features from HSI is a challenging problem. In addition, class imbalance is the common problem in the remote sensing datasets, further complicating the HSI task of achieving optimal performance. To address these two problems, in this paper, a deep learning network W-net with gray-level co-occurrence matrix (GLCM) for HSI classification and a hybrid loss function are proposed. The network can utilize GLCM to extract each texture feature for each band of the HSI. The extracted feature maps and the RGB image are down-sampled simultaneously in the encoder part, and then upsampled to obtain the final classification map in the decoder part. Meanwhile, the hybrid loss function combines focal loss (FL) with softmax equalization loss (SEQL) to adjust the balance between classes and suppress the gradient of negative samples in rare classes. Experimental results show that the proposed network demonstrates the effective integration of texture features from hyperspectral data using GLCM in the training process, while also offering a solution to the problem of class imbalance, resulting in a promising HSI classification performance.