Maintaining precise interaction force in uncertain environments characterized by unknown and varying stiffness or location is significantly challenging for robotic manipulators. Existing approaches widely employ a two-level control structure in which the higher level generates the command motion of the lower level according to the force tracking error. However, the low-level motion tracking error is generally ignored completely. Recognizing this limitation, this paper first formulates the low-level motion tracking error as an unknown input disturbance, based on which a dynamic interaction model capturing both structured and unstructured uncertainties is developed. With the developed interaction model, an observer-based adaptive robust force controller is proposed to achieve accurate and robust force modulation for a robotic manipulator. Alongside the theoretical stability analysis, comparative experiments with the classical admittance control (AC), the adaptive variable impedance control (AVIC), and the adaptive force tracking admittance control based on disturbance observer (AFTAC) are conducted on a robotic manipulator across four scenarios. The experimental results demonstrate the significant advantages of the proposed approach over existing methods in terms of accuracy and robustness in interaction force control. For instance, the proposed method reduces the root mean square error (RMSE) by 91.3%, 87.2%, and 75.5% in comparison to AC, AVIC, and AFTAC, respectively, in the experimental scenario where the manipulator is directed to follow a time-varying force while experiencing significant low-level motion tracking errors.