Machined surface topography is a crucial factor in determining the machine tool machining accuracy. To investigate machined surface topography and further enhance machining accuracy, this paper proposes methods for predicting ball end milling surface topography and analyzing the reliability of machine tool milling accuracy. Initially, the static deflection and dynamic deflection of the cutter are analyzed by the flexible deformation model and the nonlinear dynamics model. The milling surface topography model is developed with the effects of the cutter teeth motion trajectory, cutter deflection, and cutter runout. Furthermore, considering the uncertainty of machine tool parameters, the method for analyzing the reliability of machine tool milling accuracy is presented. Simultaneously, the surface roughness Sa is selected as the characterization parameter for machining accuracy. Subsequently, the established model is validated by ball end milling experiments. Finally, the results of partial machine tool parameters on the ball end milling surface are investigated, and the reliability and global reliability sensitivity of ball end milling accuracy are analyzed. This study assists researchers in predicting machining accuracy before processing, and optimizing the process parameters to develop products with good surface finish.