Predicting seizure recurrence after an initial seizure-like episode from routine clinical notes using large language models: a retrospective cohort study

Background The evaluation and management of first-time seizure-like events in children can be difficult because these episodes are not always directly observed and might be epileptic seizures or other conditions (seizure mimics). We aimed to evaluate whether machine learning models using real-world data could predict seizure recurrence after an initial seizure-like event. Methods This retrospective cohort study compared models trained and evaluated on two separate datasets between Jan 1, 2010, and Jan 1, 2020: electronic medical records (EMRs) at Boston Children's Hospital and de-identified, patient-level, administrative claims data from the IBM MarketScan research database. The study population comprised patients with an initial diagnosis of either epilepsy or convulsions before the age of 21 years, based on International Classification of Diseases, Clinical Modification (ICD-CM) codes. We compared machine learning-based predictive modelling using structured data (logistic regression and XGBoost) with emerging techniques in natural language processing by use of large language models. Findings The primary cohort comprised 14 021 patients at Boston Children's Hospital matching inclusion criteria with an initial seizure-like event and the comparison cohort comprised 15 062 patients within the IBM MarketScan research database. Seizure recurrence based on a composite expert-derived definition occurred in 57% of patients at Boston Children's Hospital and 63% of patients within IBM MarketScan. Large language models with additional domain-specific and location-specific pre-training on patients excluded from the study (F1-score 0 center dot 826 [95% CI 0 center dot 817-0 center dot 835], AUC 0 center dot 897 [95% CI 0 center dot 875-0 center dot 913]) performed best. All large language models, including the base model without additional pre-training (F1-score 0 center dot 739 [95% CI 0 center dot 738-0 center dot 741], AUROC 0 center dot 846 [95% CI 0 center dot 826-0 center dot 861]) outperformed models trained with structured data. With structured data only, XGBoost outperformed logistic regression and XGBoost models trained with the Boston Children's Hospital EMR (logistic regression: F1-score 0 center dot 650 [95% CI 0 center dot 643-0 center dot 657], AUC 0 center dot 694 [95% CI 0 center dot 685-0 center dot 705], XGBoost: F1-score 0 center dot 679 [0 center dot 676-0 center dot 683], AUC 0 center dot 725 [0 center dot 717-0 center dot 734]) performed similarly to models trained on the IBM MarketScan database (logistic regression: F1-score 0 center dot 596 [0 center dot 590-0 center dot 601], AUC 0 center dot 670 [0 center dot 664-0 center dot 675], XGBoost: F1-score 0 center dot 678 [0 center dot 668-0 center dot 687], AUC 0 center dot 710 [0 center dot 703-0 center dot 714]). Interpretation Physician's clinical notes about an initial seizure-like event include substantial signals for prediction of seizure recurrence, and additional domain-specific and location-specific pre-training can significantly improve the performance of clinical large language models, even for specialised cohorts. Funding UCB, National Institute of Neurological Disorders and Stroke (US National Institutes of Health). Copyright (c) 2023 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license.