Efforts have been made to estimate clothing insulation in real time, an element of thermal comfort for occupants. Nevertheless, an effective method to estimate clothing insulation in real time is lacking. In addition, there has been little debate on how to apply clothing insulation to building control in practice. The purpose of this study is to propose a method for estimating clothing insulation using deep learning-based vision recognition, which has recently attracted attention and implement building control based on clothing insulation. The study also evaluates the significance of the method in effective building control. The results demonstrated that the proposed framework, CloNet, showed an accuracy of 94% for the validation image dataset and 86% for the actual built environment. In addition, we proved that the proposed vision-based estimation method is very fast and practical for estimating clothing insulation. The control experiment showed that the CloNet-based predicted mean vote (PMV) control changed the set temperature in response to changes in the subject's clothing. Compared to the traditional PMV control, the CloNet-based PMV control improved the thermal preference and thermal comfort vote. These results prove that clothing insulation estimation can be useful for building control.