Vanadium dioxide (VO2), a promising multifunctional smart material, has shown strong promise in microactuation, memory, and optical applications. During thermally induced insulator-to-metal phase transition of VO2, the changes of its electrical, mechanical, and optical properties demonstrate pronounced, complex hysteresis with respect to the temperature, which presents a challenge in the utilization of this material. In this paper, an extended generalized Prandtl-Ishlinskii model is proposed to model the hysteresis in VO2, where a nonlinear memoryless function is introduced to improve its modeling capability. A novel inverse compensation algorithm for this hysteresis model is developed based on fixed-point iteration with which the convergence conditions of the algorithm are derived. The proposed approach is shown to be effective for modeling and compensating the asymmetric and non-monotonic hysteresis with saturation between the curvature output and the temperature input of a VO2-coated microactuator, as well as the asymmetric hysteresis with partial saturation between the resistance output and the temperature input of a VO2 film.