The cooperation of AC-DC control and protection functions has important influence on the security and stability of the power system with the characteristics of long-distance, heavy capacity, ultra-high voltage and combined operation of AC/DC. HVDC fundamental and second harmonic protections are the basic cooperation function between the AC and DC system faults. The 50 Hz and 100 Hz protection setting principles of the AC/DC hybrid transmission system are studied in depth by EMTDC and RTDS simulation. The setting method based on the longest clearing time of AC faults and the over-load capability of the HVDC main equipments is proposed, and the quantitative method based EMTDC simulation is presented. The proposed 100 Hz protection setting principle has been applied to TSQ-Guangdong, Guizhou-Guangdong I and Guizhou-Guangdong II HVDC links in China Southern Power Grid, and the key problems of the AC-DC protection cooperation of China Southern Power Grid have been solved.