A gaze-driven manufacturing assembly assistant system with integrated step recognition, repetition analysis, and real-time feedback

Modern manufacturing faces significant challenges, including efficiency bottlenecks and high error rates in manual assembly operations. To address these challenges, we implement artificial intelligence (AI) and propose a gaze-driven assembly assistant system that leverages artificial intelligence for human-centered smart manufacturing. Our system processes video inputs of assembly activities using a Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) network for assembly step recognition, a Transformer network for repetitive action counting, and a gaze tracker for eye gaze estimation. The application of AI integrates the outputs of these tasks to deliver real-time visual assistance through a software interface that displays relevant tools, parts, and procedural instructions based on recognized steps and gaze data. Experimental results demonstrate the system's high performance, achieving 98.36% accuracy in assembly step recognition, a mean absolute error (MAE) of 4.37%, and an off-by-one accuracy (OBOA) of 95.88% inaction counting. Compared to existing solutions, our gaze-driven assistant offers superior precision and efficiency, providing a scalable and adaptable framework suitable for complex and large-scale manufacturing environments.