An AR system with intuitive user interface for manipulation and visualization of 3D medical data

We report on a stereoscopic video-see-through augmented reality system which we developed for medical applications. Our system allows interactive in-situ visualization of 3D medical imaging data. For high-quality rendering of the augmented scene we utilize the capabilities of the latest graphics card generations. Fast high-precision MPR generation ("multiplanar reconstruction") and volume rendering is realized with OpenGL 3D textures. We provide a tracked hand-held tool to interact with the medical imaging data in its actual location. This tool is represented as a virtual tool in the space of the medical data. The user can assign different functionality to it: select arbitrary MPR cross-sections, guide a local volume rendered cube through the medical data, change the transfer function, etc. Tracking works in conjunction with retroreflective markers, which frame the workspace for head tracking respectively are attached to instruments for tool tracking. We use a single head-mounted tracking camera, which is rigidly fixed to the stereo pair of cameras that provide the live video view of the real scene. The user's spatial perception is based on stereo depth cues as well as on the kinetic depth cues that he receives with the viewpoint variations and the interactive data visualization. The AR system has a compelling real-time performance with 30 stereo-frames/second and exhibits no time lag between the video images and the augmenting graphics. Thus, the physician can interactively explore the medical imaging information in-situ.