Non-contact measurement of structural vibrations from video camera has been an emerging method for vibration test and structural health monitoring. However, its accuracy is very sensitive to camera's ego motion, which may be induced by environmental winds, ground motion or unmanned-aerial-vehicle perturbation. In case of highly perturbed ego motion, the observed structural vibration from video is contaminated and even dominated by camera's rigid body motion. Thus, to mitigate the effects of camera's ego motion on vision-based measurement, a camera motion compensation method is proposed within the Lucas-Kanade framework. The realization of the proposed approach is mainly two-fold. At first, the camera's rigid body motion, possibly being large, is estimated with the help of the projection transformation of a reference template. Next, the structural vibration is recovered in conjunction with the estimated camera's ego motion. Numerical simulations and real-world experiment are conducted to verify the efficiency and accuracy of the proposed method for vibration measurement under highly perturbed camera ego motion.