To actively suppress vibrations in mechanical structures, this research presents a novel fuzzy sliding mode controller that can be designed without involving mathematical plant modeling and any plant's parameters. The proposed strategy introduces a virtual controlled object (VCO)-based model-free scheme into a sliding mode control (SMC) system while addressing chattering mitigation by adaptive fuzzy inference. A framework of the model-free design is first formulated by interposing the VCO between a real controlled structure and an actuator. Based on a simple controlled frequency band-based design policy, the VCO, which is represented by a single-degree-of-freedom (SDOF) system, is specified to enable model-free active controller design. The VCO-based design process is then combined with SMC, and a Lyapunov function approach is employed to analyze the convergence. Furthermore, the chattering issue, which hinders successful implementation of SMC, is explicitly addressed by employing a fuzzy inference system. The fuzzy rules adjust the control effort online to ensure rapidly convergence to the sliding surface while avoiding drastic switching of the control input, in accordance with variations of the switching function. Finally, experimental verifications are performed by applying the proposed vibration controller to a cantilever plate with characteristic perturbations. The verification results demonstrate that the proposed fuzzy SMC provides sufficient vibration reduction and realizes high robustness to changes in the controlled object.