Carbon fiber-reinforced polymer (CFRP) has the characteristics of high brittleness and high hardness, which easily causes delamination damage and burr damage, resulting in low hole-making efficiency, and the application of cutting fluid in the process will lead to the decrease of mechanical properties of the material. In order to improve the surface quality and processing efficiency of CFRP, an ultrasonic-assisted dry helical milling technology is proposed. Taking tool rotation speed, feed rate, pitch, and ultrasonic amplitude as optimization variables and taking minimum delamination damage, burr damage, and maximum material removal rate as objective functions, multi-objective optimization models are established through experiments and genetic algorithm, and Pareto optimal solution sets are obtained. The results show that the influence weights are in the order of pitch, tool rotation speed, ultrasonic amplitude, and feed rate for the analysis of variance of delamination damage, accounting for 38.7%, 24.9%, 20.9%, and 15.5%, respectively. The maximum weight of pitch is 29.7%, and the minimum weight of ultrasonic amplitude is 18.1%, in the analysis of variance of burr damage. Finally, multi-objective optimization models are verified by experiments, and it is concluded that the established optimization models can provide multiple parameter optimization schemes for different engineering applications with high accuracy.