Development of a quality assessment scale for retrospective clinical studies in pediatric surgery

Background/Purpose: More than 97% of the clinical research in pediatric surgery consists of retrospective data. Although these studies have significant limitations, they fundamentally shape clinical practice within the field. In this report, the authors describe the development and potential applications of a standardized quality assessment scale designed for retrospective studies in pediatric surgery. Methods: The authors developed a comprehensive quality assessment instrument incorporating 30 items within 3 subscales. These subscales were designed to assess 3 integral components of study design: clinical relevance, reporting methodology, and the strength of conclusions. Global quality ratings (poor, fair, or good) were derived by combining scores from each subscale. To examine inter-rater reliability, 10 retrospective studies from the pediatric surgery literature were assessed with the instrument by 6 independent reviewers. Results: Inter-rater reliability was excellent as assessed by the level of agreement for all items within the instrument (84.6% concordance, n = 1,573 total item-choices) and for individual subscales (range, 73.3% to 85.8%, n = 60 to 1,258). The extent of agreement among reviewers was 82% for classifying reports into qualitative categories by global quality scores (n = 60). The ranking of papers by individual reviewers was highly predictive of overall ranking by mean quality scores (n = 60 rank pairs, r = 0.83; P < .01). Conclusions: The authors have developed a standardized and reliable quality assessment scale for the analysis of retrospective data in pediatric surgery. Potential applications include: (1) providing the practicing surgeon with a knowledge base to critically evaluate published retrospective data; (2) providing a standardized methodology for the systematic review of existing retrospective data; and (3) developing standardized reporting guidelines for use in peer-reviewed journals. Copyright 2003, Elsevier Science (USA). All rights reserved.