Overlooked pitfalls in multi-class machine learning classification in radiation oncology and how to avoid them

In radiation oncology, Machine Learning classification publications are typically related to two outcome classes, e.g. the presence or absence of distant metastasis. However, multi-class classification problems also have great clinical relevance, e.g., predicting the grade of a treatment complication following lung irradiation. This work comprised two studies aimed at making work in this domain less prone to statistical blindsides. In multi-class classification, AUC is not defined, whereas correlation coefficients are. It may seem like solely quoting the correlation coefficient value (in lieu of the AUC value) is a suitable choice. In the first study, we illustrated using Monte Carlo (MC) models why this choice is misleading. We also considered the special case where the multiple classes are not ordinal, but nominal, and explained why Pearson or Spearman correlation coefficients are not only providing incomplete information but are actually meaningless. The second study concerned surrogate biomarkers for a clinical endpoint, which have purported benefits including potential for early assessment, being inexpensive, and being non-invasive. Using a MC experiment, we showed how conclusions derived from surrogate markers can be misleading. The simulated endpoint was radiation toxicity (scale of 0-5). The surrogate marker was the true toxicity grade plus a noise term. Five patient cohorts were simulated, including one control. Two of the cohorts were designed to have a statistically significant difference in toxicity. Under 1000 repeated experiments using the biomarker, these two cohorts were often found to be statistically indistinguishable, with the fraction of such occurrences rising with the level of noise.