Recognizing pointing behavior using image processing for human-robot interaction

This paper presents a gestual interaction system between human and robot using pointing movement. The virtual room, which was based on actual room, can be shown an attended object with blinking. A user pointed out an object, which he/she wants to move. The robot moves to an indicated point, or moves an indicated object to point in actual room. Moreover, to interact at actual rooms, the system that obtained RGB information of a pointed out object was developed. The system recognized a pointing gesture and obtained a feature of pointed out object by a image processing. Two sets of tracking modules comprised of a camera and a PC was prepared in a room. Three-dimensional coordinates of user's head and hand position were calculated using the data, which was obtained by tracking modules. The system recognized indicated point and infers the object, which the user attended. To show what the system attended, some information about the object was needed. Thus, in this research, we constructed RGB data obtaining module. The module obtained coordinates of indicated point and calculate the coordinates in picture. The system was developed using Robot Technology Middleware (RTM). RT Middleware was developed by AIST (Agency of Industrial Science and Technology, Japan) for easily integrating robot systems by modularized software components. By constructing modules based on RT Middleware, we developed modules, which are functional elements to interact with human as components of easily system integration.