A practical method to quantify knowledge-based DVH prediction accuracy and uncertainty with reference cohorts

The adoption of knowledge-based dose-volume histogram (DVH) prediction models for assessing organ-at-risk (OAR) sparing in radiotherapy necessitates quantification of prediction accuracy and uncertainty. Moreover, DVH prediction error bands should be readily interpretable as confidence intervals in which to find a percentage of clinically acceptable DVHs. In the event such DVH error bands are not available, we present an independent error quantification methodology using a local reference cohort of high-quality treatment plans, and apply it to two DVH prediction models, ORBIT-RT and RapidPlan, trained on the same set of 90 volumetric modulated arc therapy (VMAT) plans. Organ-at-risk DVH predictions from each model were then generated for a separate set of 45 prostate VMAT plans. Dose-volume histogram predictions were then compared to their analogous clinical DVHs to define prediction errors Vclin,i-Vpred,i (ith plan), from which prediction bias mu, prediction error variation sigma, and root-mean-square error RMSEpred equivalent to 1N n-ary sumation i Vclin,i-Vpred,i2 approximately equal to sigma 2+mu 2 could be calculated for the cohort. The empirical RMSEpred was then contrasted to the model-provided DVH error estimates. For all prostate OARs, above 50% Rx dose, ORBIT-RT mu and sigma were comparable to or less than those of RapidPlan. Above 80% Rx dose, mu sigma < 3-4% for both models. As a result, above 50% Rx dose, ORBIT-RT RMSEpred was below that of RapidPlan, indicating slightly improved accuracy in this cohort. Because mu approximate to 0, RMSEpred is readily interpretable as a canonical standard deviation sigma, whose error band is expected to correctly predict 68% of normally distributed clinical DVHs. By contrast, RapidPlan's provided error band, although described in literature as a standard deviation range, was slightly less predictive than RMSEpred (55-70% success), while the provided ORBIT-RT error band was confirmed to resemble an interquartile range (40-65% success) as described. Clinicians can apply this methodology using their own institutions' reference cohorts to (a) independently assess a knowledge-based model's predictive accuracy of local treatment plans, and (b) interpret from any error band whether further OAR dose sparing is likely attainable.