In mechanical systems, accurately calculating the frequency response function (FRF) is crucial for predicting the dynamic behavior of the system at various external excitation frequencies. This paper proposes a novel approach called the complex extended transfer matrix method (CETMM) for computing the FRF of mechanical systems. By introducing the concept of the complex extended state vector, this method streamlines the computation process of FRF. The governing equations for standard mechanical components, such as rigid bodies, beams, and elastic hinges, are derived. From these, the corresponding complex extended transfer equations and matrices are constructed. These are then assembled into the overall complex extended transfer equation and matrix for the entire mechanical system, ultimately enabling system's FRF computation. Using spindle-holder-tool and robotic milling systems from existing literature as case studies, the computational accuracy and efficiency of CETMM are validated, while its robustness and versatility in computing the FRF of mechanical systems are demonstrated.