An Ophthalmology Clinical Decision Support System Based on Clinical Annotations, Ontologies and Images

With the exponential growth of digital data available in ophthalmology the implementation of effective Computer-Aided Diagnosis Systems can make clinical diagnosis more accurate and accessible and therefore improve the standard of care. We explore a multimodal electronic health record dataset (n = 2348 cases and n = 2348 controls) and propose a new classifier model based on Random Forrest Classifiers for recommendation of an ophthalmic procedure (intravitreal injection of bevacizumab). The dataset comprises structured demographic data, unstructured textual annotations, and clinical images (optical coherence tomography, slit lamp photography and scanning laser ophthalmoscopy). Textual annotations were processed and encoded using a standardized medical ontology nomenclature (SNOMED-CT) to enable higher level modeling and additional meta-features were engineered in the ontology domain. Raster image data were used to train several convolutional neural networks (CNN) which were used as feature generators. Each subset of features was used for clinical event modeling. To the best of our knowledge this is the first work to be published proposing the integration of clinical textual, ontological and image data information for clinical event prediction in ophthalmology. An unoptimized implementation of the model proposed achieves a prediction accuracy of 87.38% when all data modalities are combined. When using demographic, ontological and image features in isolation performances are 72.98%, 84.37% and 78.77% respectively.