Block pavements require efficient distress detection and segmentation methods for quality control and pavement management systems. This research proposes TransUNet, a hybrid model for block and distress segmentation, combining convolutional neural networks (CNNs) and Vision transformers. The model adopts transfer learning to achieve accurate block segmentation, pre-training the model on a large wall image dataset and then fine-tuning it on a smaller set of block pavement images. This approach yields promising results with 77% intersection over union (IoU), 96.8% precision, 99.5% recall, and 98.1% F1-score, surpassing conventional CNN-based models, UNet and UNet + + . The use of transfer learning not only enhances accuracy but also significantly reduces training time and computational resources, as it eliminates the need for a large dataset. For the block distress segmentation model, the proposed hybrid TransUNet model obtained a mIoU of 71.3% outperforming CNN-based models. The CNN models often struggle to handle the diverse distress types commonly found in block pavements, resulting in sub-optimal distress segmentation outcomes. By automating block and distress segmentation, the proposed models contribute to efficient maintenance planning and the development of sustainable infrastructure.