Prediction of certainty in artificial intelligence-enabled electrocardiography

Background: The 12-lead ECG provides an excellent substrate for artificial intelligence (AI) enabled prediction of various cardiovascular diseases. However, a measure of prediction certainty is lacking. Objectives: To assess a novel approach for estimating certainty of AI-ECG predictions. Methods: Two convolutional neural networks (CNN) were developed to predict patient age and sex. Model 1 applied a 5 s sliding time-window, allowing multiple CNN predictions. The consistency of the output values, expressed as interquartile range (IQR), was used to estimate prediction certainty. Model 2 was trained on the full 10s ECG signal, resulting in a single CNN point prediction value. Performance was evaluated on an internal test set and externally validated on the PTB-XL dataset. Results: Both CNNs were trained on 269,979 standard 12-lead ECGs (82,477 patients). Model 1 showed higher accuracy for both age and sex prediction (mean absolute error, MAE 6.9 +/- 6.3 years vs. 7.7 +/- 6.3 years and AUC 0.946 vs. 0.916, respectively, P < 0.001 for both). The IQR of multiple CNN output values allowed to differentiate between high and low accuracy of ECG based predictions (P < 0.001 for both). Among 10% of patients with narrowest IQR, sex prediction accuracy increased from 65.4% to 99.2%, and MAE of age prediction decreased from 9.7 to 4.1 years compared to the 10% with widest IQR. Accuracy and estimation of prediction certainty of model 1 remained true in the external validation dataset. Conclusions: Sliding window-based approach improves ECG based prediction of age and sex and may aid in addressing the challenge of prediction certainty estimation.