Predictive intention recognition using deep learning for collaborative assembly

Achieving seamless human-robot collaboration in tasks with uncertainty requires anticipating human intentions. In order to overcome the limitations of classical techniques for inferring human intentions, which often struggle due to delayed articulation and lack of knowledge about spatio-temporal dependencies, this study proposes an approach centered around real-time prediction of human intention using Long-Short Term Memories (LSTMs). To achieve this goal, the human hands and assembly product components were detected in each frame, using a novel dataset, the former is then used as the primary input to our LSTM model. Finally, we demonstrate and validate the effectiveness of this framework in a real industrial assembly scenario, where a robotic agent utilizes the predicted intention to efficiently assist the human in successfully completing the assembly of a product. The results indicate, with high confidence, a response time about three times faster using our approach, on average, compared to waiting for the object detection module to recognize that a component was removed from a work bin.