Insulators are a key equipment in power systems. Regular detection of defects in the insulator surface and replacement of defective insulators in time are a must for the operation of the safety system. Whereas manual inspection remains a common practice, the recent maturity of unmanned aerial vehicle(UAV) and artificial intelligence(AI) techniques leads the electrical industry to envision an automated, real-time insulator defect detector. However, the existing detection models mainly operate in very limited weather condition, faltering in generalization and practicality in the wild. To aid in the status quo, this paper proposes a new framework that enables accurate detection of insulator defects in adverse weather conditions, where atmospheric particulates can substantially degrade the quality of aerial images on insulator surfaces. Our proposed framework is embarrassingly simple, yet effective. Specifically, it integrates progressive recurrent network(PReNet) and DehazeFormer to derain and dehaze the noisy aerial images, respectively, and tailors you only look once version 7(YOLOv7) with a new structured intersection over union(SIoU) loss function and similarity-based attention module(SimAM) to expedite convergence with better deep feature extraction. Two new benchmark datasets, Chinese power line insulator dataset(CPLID)_Rainy and CPLID_Hazy, are developed for empirical evaluation, and the comparative study substantiates the viability and effectiveness of our proposed framework. We share our code and dataset at https://github.com/CHLNK/Insulator-defect-detection.