Due to the low detection accuracy of most anchor-free detectors and the slow detection speed of anchor-based detectors. Therefore, to balance the detection accuracy and speed of traffic scene objects, a new anchor-free detector called FABNet is proposed in this paper. The method is mainly composed of feature pyramid fusion module (FPFM), cascade attention module (CAM), and boundary feature extraction module (BFEM). Firstly, we design a feature pyramid fusion module to generate richer semantic information. The proposal of the feature pyramid fusion module not only improves the detection accuracy of objects, but also solves the problem of detection of objects of different sizes. Secondly, the cascade attention module achieves the local representation of features by exploiting hierarchical attention, spatial attention and channel attention. The proposal of cascade attention module improves the representation ability of object detection head. Finally, to obtain more foreground information under the influence of complex background, we design a boundary feature extraction module to extract the boundary features of the object effectively. We perform sufficient experiments on three public datasets, i.e., BDD100K, PASCAL VOC, and KITTI. The results show that our method achieves state-of-the-art levels in both accuracy and speed.