Towards Assembly Steps Recognition in Augmented Reality

Augmented Reality is a media which purpose is to attach digital information to real world scenes in order to enhance the user experience. It has been used in the field of maintenance in order to show the user the operations he has to perform. Our goal is to go one step further so that our system is able to detect when the user has performed a step of the task. It requires some understanding of what is occurring and where objects are located in order to display correct instructions for the task. This paper is focusing on using an intermediate computation result of the usual augmented reality process, which is the pose computation: we propose to use the transformation matrix not only for objects pose estimation, but also to characterise their motion during an assembly task. With this matrix, we can induce spatial relationship between assembly parts and determine which motion occurs. Then we analyse translation and rotation parameters contained in the transformation matrix during the action. We demonstrate that these data correctly characterise the movement between object's fragments. Therefore, by analysing such a matrix, not only we can achieve the required registration step of the augmented reality process, but we can also understand the actions performed by the user.