An efficient decision algorithm of the rate-distortion (RD) optimized macroblock mode for scalable video coding (SVC) extension based on H.264 in packet loss environment is presented. A parameter used for temporal level optimization is added into the RD model. Source-coding distortion and error-propagated distortion of each frame are calculated on a encoding block basis after the frame is encoded, which will be stored in a distortion mapping table to calculate the end-to-end distortion of the next frame. The simulation results show that the proposed algorithm has better performance than the original method in terms of error resilience.