Leveraging Expert Knowledge for Real-Time Online Adaptation of Intraoperative Liver Registration

In laparoscopic liver surgery, augmented reality can assist surgeons in locating structures of interest, which are invisible in the laparoscopic view. This requires an accurate deformable registration of preoperative patient data with the intraoperative liver model based on established correspondences. Finding these automatically is extremely difficult, due to the two distinct modalities and different noise sources, which lead to very different geometries. This can cause modern neural-network-based registration algorithms to produce imperfect alignments. We aim to alleviate this issue by incorporating additional expert knowledge as input to these networks. We propose Cue-Net for non-rigid registration, and modify it in such a way that surgeons can steer and correct its behavior. This is achieved by incorporating an interaction step in which users mark matching cues, allowing them to pass on their anatomical knowledge to the otherwise automatic system. We evaluated the performance of Cue-Net with user inputs on different datasets from the global and local points of view. The distribution of the matching cues on the operating surface is observed to be the most dominant factor for obtaining an improvement regarding registration accuracy, compared to the morphology of matching cues. Additionally, we performed a user study to present the usability of the system. Code is available at: https://gitlab.com/nct_tso_public/cue-net.