Comparison of Machine Learning Algorithms Identifying Children at Increased Risk of Out-of-Home Placement: Development and Practical Considerations

ObjectiveTo develop a machine learning (ML) algorithm capable of identifying children at risk of out-of-home placement among a Medicaid-insured population. Study Setting and DesignThe study population includes children enrolled in a Medicaid accountable care organization between 2018 and 2022 in two nonurban Ohio counties served by the Centers for Medicare and Medicaid Services-funded Integrated Care for Kids Model. Using a retrospective cohort, we developed and compared a set of ML algorithms to identify children at risk of out-of-home placement within one year. ML algorithms tested include least absolute shrinkage and selection operator (LASSO)-regularized logistic regression and eXtreme gradient-boosted trees (XGBoost). We compared both modeling approaches with and without race as a candidate predictor. Performance metrics included the area under the receiver operating characteristic curve (AUROC) and the corrected partial AUROC at specificities >= 90% (pAUROC90). Algorithmic bias was tested by comparing pAUROC90 across each model between Black and White children. Study Setting and Design The study population includes children enrolled in a Medicaid accountable care organization between 2018 and 2022 in two nonurban Ohio counties served by the Centers for Medicare and Medicaid Services-funded Integrated Care for Kids Model. Using a retrospective cohort, we developed and compared a set of ML algorithms to identify children at risk of out-of-home placement within one year. ML algorithms tested include least absolute shrinkage and selection operator (LASSO)-regularized logistic regression and eXtreme gradient-boosted trees (XGBoost). We compared both modeling approaches with and without race as a candidate predictor. Performance metrics included the area under the receiver operating characteristic curve (AUROC) and the corrected partial AUROC at specificities >= 90% (pAUROC(90)). Algorithmic bias was tested by comparing pAUROC(90) across each model between Black and White children. Data Sources and Analytic Sample The modeling dataset was comprised of Medicaid claims and patient demographics data from Partners For Kids, a pediatric accountable care organization. Principal Findings Overall, XGBoost models outperformed LASSO models. When race was included in the model, XGBoost had an AUROC of 0.78 (95% confidence interval [CI]: 0.77-0.79) while the LASSO model had an AUROC of 0.75 (95% CI: 0.74-0.77). When race was excluded from the model, XGBoost had an AUROC of 0.76 (95% CI: 0.74-0.77) while LASSO had an AUROC of 0.73 (95% CI: 0.72-0.74). Conclusions The more complex XGBoost outperformed the simpler LASSO in predicting out-of-home placement and had less evidence of racial bias. This study highlights the complexities of developing predictive models in systems with known racial disparities and illustrates what can be accomplished when ML developers and policy leaders collaborate to maximize data to meet the needs of children and families.