Distracted driving causes a large number of fatalities every year and is now becoming an important issue in the traffic safety study. In this paper, we present SafeDrive, a driving safety system that leverages wearable wrist sensing techniques to detect and analyze driver distracted behaviors. Existing wrist-worn sensing approaches, however, do not address challenges under real driving environments, such as less distinguishable gesture patterns due to in-vehicle physical constraints, various gesture hallmarks produced by different drivers and significant noise introduced by various driving conditions. In response, SafeDrive adopts a semi-supervised machine learning model for in-vehicle distracting activity detection. To improve the detection accuracy, we provide online updated classifiers by collecting real-time gesture data, while at the same time utilize smartphone sensing to generate soft hints filtering out anomalies and non-distracted hand movements. In the evaluation, we conduct extensive real-road experiments involving 20 participants (10 males and 10 females) and 5 vehicles (a sedan, a minivan and three SUVs). Our approach can achieve an average classification accuracy of over 90% with a error rate of a few percent, which demonstrate that SafeDrive is robust to real driving environments, and has great potential to help drivers shape safe driving habits. © 2018 ACM.