The structure of M-N-4, in which one metal atom is anchored by 4 N atoms, is the main active structure of N-doped carbon-based single-atom catalysts (SACs). However, we still lack efficient strategies for improving their activity. In this DFT study, M-N-4 is doped with phosphorus or sulfur, which can be realized in many experiments by the addition of a P/S source. Previous studies simply attributed the improved activity of the metal site to the electronic effect of non-metal doping using the d-band center theory, which is inaccurate for understanding the ORR performance in the P/S-doped M-N-4. We found that the strong interactions between P/S and O cause the doping site to be occupied by adsorbed O atoms, whose effects on the ORR process have been ignored for a long time. The local collaborative structure further stabilizes OOH*/OH* and weakens on metal sites by hydrogen bond and repulsive interactions, for which the scaling relationship of ORR could be broken. In our P/S-doped models, the onset potentials of ORR in the 4e(-) mechanism on M-1-N-4 (M-1 = Co, Ni, Cu, Rh, Ir, and Pt) and those in the 2e(-) mechanism on M-2-N-4 (M-2 = Ni, Cu, Pd, Ag, Pt, and Au) has increased.