Comparing Methodologies in a Series of Speech Outcome Studies: Challenges and Lessons Learned

Background: Development of the speech audit tool Cleft Audit Protocol for Speech Augmented (CAPS-A) facilitated intercenter comparison of speech outcomes following cleft palate repair. The CAPS-A protocol recommends consensus listening by 3 speech and language therapists, 2 of whom must be CAPS-A trained. Allowing 15 minutes per sample, 15 to 20 samples can be assessed each day. Centers typically have resources to audit 15 to 75 samples per year but not to report speech outcomes of larger data sets for research. This 3-phased outcome study examines how the implementation of the CAPS-A protocol might be modified without compromising reliability. Methodology: In phase 1, 2 external listeners independently rated 42 speech samples; in phase 2, 2 external listeners consensus listened 25% of 140 samples before 1 listener independently rated the remainder; phase 3 compared 124 Great Ormond Street Speech Assessment (GOS.SP.ASS'98) records from live assessments with CAPS-A-rated video samples. Results: Hypernasality, nasal airflow, and passive cleft speech characteristics were rated to identify signs of velopharyngeal dysfunction across all phases. Phase 1 demonstrated suboptimal correlation, intraclass correlation coefficient (ICC) ranging between 0.39 and 0.72. However, the modified CAPS-A consensus listening process in phase 2 achieved a mean ICC of 0.91. Phase 3 revealed only moderate agreement between GOS.SP.ASS'98 and CAPS-A. Conclusion: A large data set of speech samples was successfully managed by establishing good interrater reliability on 25% of the data, which calibrated the listeners and validated a decision for only 1 of 2 listeners to rate the remaining speech samples. The recommended implementation of the CAPS-A protocol can therefore be modified for more efficient speech outcome reporting.