Inter-rater reliability in quality assurance (QA) of pediatric chest X-rays

Purpose: The goal of the study is to determine the inter-rater agreement on multiple factors that were utilized to evaluate the quality of pediatric chest X-ray exams from different levels of healthcare provision in an African setting. Methods: The image quality of pediatric chest X-rays from 3 South African medical centers of varying level of healthcare service were retrospectively assessed by 3 raters for 12 quality factors including: (1) absent body parts; (2) under inspiration; (3) patient rotation; (4) scapula in the way; (5) patient kyphosis/lordosis; (6) artefact/foreign body; (7) central vessel visualization; (8) peripheral vessels visualization; (9) poor collimation; and (10) trachea and bronchi visualization; (11) postcardiac vessel visualization; and (12) absent or wrong image orientation. Analysis was performed using the Brennan--Prediger coefficient of agreement for inter-rater reliability and Cochran's Q statistic and McNemar's test for inter-rater bias. Results: 1077 X-rays were reviewed. The least difference between observers in the frequency of the errors was noticed for factors (1) absent body parts and (12) absent or wrong image orientation with almost perfect agreement between raters. kappa score for these two factors among all raters and between each pair of raters was more than 0.95 with no significant inter-rater bias. Conversely, there was poor agreement for the remaining factors with the least agreed on being factor (3) patient rotation with a kappa score of 0.23. This was followed by factors (2) under inspiration (kappa score of 0.32) and factors (4) scapula in the way (kappa score of 0.35) respectively. There was significant inter-rater bias for all these three factors. Conclusion: Many of the factors used to assess the quality of a chest X-ray in children demonstrate poor reliability despite mitigation against variations in training, standard quality definitions and level of healthcare service provision. New definitions, objective measures and recording tools for assessing pediatric chest radiographic quality are required.