A Haptic Interaction Method Using Visual Information and Physically Based Modeling

Haptic feedback can be used to sense a physical environment at a remote site in order to overcome spatial or scale barriers in telemanipulation. The aim of this paper is to develop a haptic interaction method for a deformable object manipulation system by means of image processing and physically based modeling techniques. The interaction forces between the instrument driven by a haptic device and a deformable object are inferred in real time based on the visual information from a slave environment without force sensor. A physically based model of the deformable object is constructed by integrating the geometric information from vision, a priori knowledge of the object mechanical properties, and a pre-defined coordinate system of the slave environment. The forces are then derived from the model, while a boundary condition is updated based on the images (a tool-tip position tracking). In order to demonstrate the applicability and effectiveness of the proposed algorithm, macro- and microscale experimental systems were built and equipped with a telemanipulation system and a commercial haptic display. The proposed method was verified using silicone (macroscale) and zebrafish embryos (microscale).