The Effect of Assessment Criteria on Inter-rater Variability in the Evaluation of Skin Reactions following Breast Cancer Radiation Therapy

Purpose: Although the Common Terminology Criteria for Adverse Events (CTCAE) grading scales are utilized to assess adverse reactions to radiation therapy, few studies have investigated their reliability. In our previously reported image-based retrospective study, variability between caregivers (radiation oncologists, nurses) in assessing skin reactions in breast cancer radiation therapy using this scale was found to fall within a "moderate" range of concordance (Fleiss kappa score 0.43) per the Landis-Koch criteria. In this work, the potential impact of specific toxicity assessment terms documented by raters in our previous study on overall scale reliability was evaluated. Methods: In an institutional-review board approved retrospective study, clinical notes documented by 8 caregivers were interpreted to assess how discordances among grades of skin lesions following radiation therapy potentially related to the use of freehand terms to describe the adverse events. 25 terms commonly used in the commentary were identified and categorized into those that were and were not outlined in the CTCAE scale. The percentage incidence and free marginal kappa scores for each term was calculated. Results: The free marginal kappa scores for the terms stated in the CTCAE scale ranged from 0.333 to 0.565, suggesting a fairto- moderate level of concordance between grades given by caregivers who used such terms. Certain terms not included within the CTCAE scale such as "hyperpigmentation" exhibited a higher rate of incidence (80%) and concordance (free marginal kappa score of .512) than those included explicitly in the scale. Conclusion: The low kappa scores associated with terms in the CTCAE scale suggest variability in caregivers' interpretation of assessment criteria. Revision of the wording of the scales may be needed to make definitions unambiguous and ensure a reliable grading scheme. The high frequency and kappa scores of terms including "hyperpigmentation" suggests revisions may also require inclusion of new clinical toxicity assessment criteria.