Standardized Evaluation of Camera-based Driver State Monitoring Systems

Driver state sensing technologies, such as vehicular systems, start to be widely considered by automotive manufacturers. To reduce the cost and minimize the intrusiveness towards driving, the majority of these systems rely on the in-cabin camera(s) and other optical sensors. With their great capabilities in detecting and intervening of driver distraction and inattention, these technologies may become key components in future vehicle safety and control systems. However, to the best of our knowledge, currently, there is no common standard available to objectively compare the performance of these technologies. Thus, it is imperative to develop one standardized process for evaluation purposes. In this paper, we propose one systematic and standardized evaluation process after successfully addressing three difficulties: 1) defining and selecting the important influential individual and environmental factors, 2) countering the effects of individual differences and randomness in driver behaviors, and 3) building a reliable in-vehicle driver head motion tracking tool to collect ground-truth motion data. We have collected data on a large scale on a commercial driver state-sensing platform. For each subject, 30 to 40 minutes of head motion data was collected and included variables, such as lighting conditions, head/face features, and camera locations. The collected data was analyzed based on a proposed performance measure. The results show that the developed process can efficiently evaluate an individual camera-based driver state sensing product, which builds a common base for comparing the performance of different systems.