Breast cancer causes a serious menace to women's health and lives, underscoring the urgency of accurate tumor detection. Detecting both cancerous and non -cancerous breast tumors has become increasingly crucial, with ultrasound imaging emerging as a widely adopted modality for this purpose. However, identifying breast lesions in ultrasound images is a challenging task due to various tumor morphologies, geometry, similar color intensity distributions, and fuzzy boundaries, particularly irregularly shaped malignant tumors. This work proposes an encoder-decoder based U-shaped convolutional neural network (CNN) variant with an attention aggregation -based pyramid feature clustering module (AAPFC) to detect breast lesion regions. The network consists of the U -Net variant as a base network and AAPFC to fuse features extracted at the various levels of the base U -Net using a suitable feature fusion technique. Furthermore, the deformable convolution with adaptive self -attention mechanism is introduced to decode the pyramid features parallel to capture the various geometric features at multi -stages. Two public breast lesion ultrasound datasets consisting 263 malignant, 547 benign and 133 normal images are considered to evaluate the performance of the proposed model and stateof-the-art deep CNN -based segmentation models. The proposed model provides 96% accuracy, 68% Mean-IoU, 97% specificity, 82% sensitivity and 0.747 kappa score respectively. The conducted qualitative and quantitative performance analysis experiments show that the proposed model performs better in breast lesion segmentation on ultrasound images.