In this paper, using the multi-source data, combined with deep neural network, an automatic assessment model for highway risk and accident number prediction model are established. First, based on the multi-source heterogeneous data collected by different monitoring facilities and sensors on the highway, the features are fused. Secondly, the shuffled frog leaping algorithm is used to optimize the features to reduce the dimension. Thirdly, we improve the traditional deep neural network by connecting the output of each layer of neural network to the last fully connected layer to obtain a fused vector. Finally, the effectiveness of the proposed algorithm is verified in the experimental results.