With the development of intelligent transportation system, the detection method of traffic signs plays an important role in unmanned driving. However, due to the real-time and reliability characteristics of the automatic driving system, each traffic sign needs to be processed in a specific time interval to ensure the precision of the test results. Automatic driving is developing rapidly and has made great progress. Various traffic sign detection algorithms are proposed. Especially, convolutional neural network algorithm is concerned because of its fast execution and high recognition rate. But in the real world of complex traffic conditions, those algorithms still have problems such as poor real-time detection, low precision, false detection and high missed detection rate. To overcome those problems, this paper proposed an improved algorithm named as YOLO-RFB based on YOLO V4 network. Based on YOLO V4 network, the main feature extraction network is pruned, and convolution layer is replaced by RFB structure in two output feature layers. In the detection results of GTSDB data sets, the mAP of improved algorithm achieves 85.59%, 4.76% points higher than the original algorithm, and the FPS reaches 48.72, which is slightly lower than that of the original YOLO V4 algorithm 50.21.