For screening breast tumors, different imaging modalities like ultrasound, mammography, computed tomography (CT), magnetic resonance imaging (MRI) have been utilized. Mammography and CT use ionizing radiations and hence are not preferred for pregnant women. Even though MRI has high sensitivity for differentiating between breast tumor types, it is costlier and not available everywhere. Therefore, ultrasound is used more prominently for screening of breast tissue due to its ease of use, portability, low cost and safety. Ultrasound images are marred by speckle noise, hence an accurate diagnosis of abnormalities is challenging even for experienced radiologists. Therefore, increasing amount of interest has been observed among researchers to address these limitations and enhance the diagnostic potential of ultrasound images. Accordingly, in the present work, an exhaustive review of machine learning and deep learning based computer aided diagnostic (CAD) system designs has been conducted and brain storming diagrams have been used to indicate the characterization approaches for each stage i.e. (i) datasets, (ii) pre-processing methods, (iii) data augmentation methods, (iv) segmentation methods, (v) feature extraction methods, (vi) feature selection methods, (vii) classification methods and (viii) evaluation metrics. The paper also presents (a) clinically significant sonographic features for differentiating between breast tumor types, (b) achievements made in the design of CAD systems for breast tumor classification and (c) future challenges in designing such systems. The directions for future research to further enhance the diagnostic potential of ultrasound imaging modality for differential diagnosis between different breast abnormalities have also been highlighted.