HEARTBEAT4D: An Open-source Toolbox for Turning 4D Cardiac CT into VR/AR

Four-dimensional data sets are increasingly common in MRI and CT. While clinical visualization often focuses on individual temporal phases capturing the tissue(s) of interest, it may be possible to gain additional insight through exploring animated 3D reconstructions of physiological motion made possible by augmented or virtual reality representations of 4D patient imaging. Cardiac CT acquisitions can provide sufficient spatial resolution and temporal data to support advanced visualization, however, there are no open-source tools readily available to facilitate the transformation from raw medical images to dynamic and interactive augmented or virtual reality representations. To address this gap, we developed a workflow using free and open-source tools to process 4D cardiac CT imaging starting from raw DICOM data and ending with dynamic AR representations viewable on a phone, tablet, or computer. In addition to assembling the workflow using existing platforms (3D Slicer and Unity), we also contribute two new features: 1. custom software which can propagate a segmentation created for one cardiac phase to all others and export to surface files in a fully automated fashion, and 2. a user interface and linked code for the animation and interactive review of the surfaces in augmented reality. Validation of the surface-based areas demonstrated excellent correlation with radiologists' image-based areas (R > 0.99). While our tools were developed specifically for 4D cardiac CT, the open framework will allow it to serve as a blueprint for similar applications applied to 4D imaging of other tissues and using other modalities. We anticipate this and related workflows will be useful both clinically and for educational purposes.