In non-cooperative communication sys-tems, wireless interference classification (WIC) is one of the most essential technologies. Recently, deep learning (DL) based WIC methods have been pro-posed. However, conventional DL-based WIC meth-ods have high computational complexity and unsatis-factory accuracy, especially when the interference-to -noise ratio (INR) is low. To this end, we propose three effective approaches. Firstly, we introduce multi -branch convolutional neural networks (CNNs) for in-terference recognition. The multi-branch CNN is con-structed by repeating a layer that aggregates several transformations with the same topology, and it notably improves the recognition ability for WIC. Our design avoids the carefully crafted selection of each transfor-mation. Unfortunately, multi-branch CNNs are com-putationally expensive and memory-inefficient. To this end, we further propose Low complexity multi -branch networks (LCMN), which are mathematically equivalent to multi-branch CNNs but maintain low computing costs and efficient inference. Thirdly, we present novel loss function, which encourages net-works to have consistent prediction probabilities for samples with high visual similarities, resulting in in - creasing recognition accuracy of LCMN. Experimen-tal results demonstrate the proposed methods consis-tently boost the classification performance of WIC without substantially increasing computational over-head compared to traditional DL-based methods.