Mixed Reality in Visceral Surgery Development of a Suitable Workflow and Evaluation of Intraoperative Use-cases

Objective: The paper evaluates the application of a mixed reality (MR) headmounted display (HMD) for the visualization of anatomical structures in complex visceral-surgical interventions. A workflow was developed and technical feasibility was evaluated. Summary of Background Data: Medical images are still not seamlessly integrated into surgical interventions and, thus, remain separated from the surgical procedure. Surgeons need to cognitively relate 2-dimensional sectional images to the 3-dimensional (3D) during the actual intervention. MR applications simulate 3D images and reduce the offset between working space and visualization allowing for improved spatial-visual approximation of patient and image. Methods: The surgeon's field of vision was superimposed with a 3D-model of the patient's relevant liver structures displayed on a MR-HMD. This set-up was evaluated during open hepatic surgery. Results: A suitable workflow for segmenting image masks and texture mapping of tumors, hepatic artery, portal vein, and the hepatic veins was developed. The 3D model was positioned above the surgical site. Anatomical reassurance was possible simply by looking up. Positioning in the room was stable without drift and minimal jittering. Users reported satisfactory comfort wearing the device without significant impairment of movement. Conclusion: MR technology has a high potential to improve the surgeon's action and perception in open visceral surgery by displaying 3D anatomical models close to the surgical site. Superimposing anatomical structures directly onto the organs within the surgical site remains challenging, as the abdominal organs undergo major deformations due to manipulation, respiratory motion, and the interaction with the surgical instruments during the intervention. A further application scenario would be intraoperative ultrasound examination displaying the image directly next to the transducer. Displays and sensor-technologies as well as biomechanical modeling and object-recognition algorithms will facilitate the application of MR-HMD in surgery in the near future.