Quality of guideline development assessed by the Evaluation Committee of the Japan Society of Clinical Oncology

The Japan Society of Clinical Oncology started implementing clinical practice guidelines for cancer in 2001. It created a Guideline Committee and has published cancer-related information in collaboration with individual subspecialty cancer societies. The society then established an Evaluation Committee to assess the quality of guidelines. The quality of development and general characteristics of guidelines were reviewed using the AGREE instrument. The six standardized domain scores and 23-item crude scores were described, and items with a low median score or a wide inter-quartile range were explored. Kappa statistics for inter-rater reproducibility were also described. Domains in which the median score was > 50 points in 18 guidelines developed between March 2005 and May 2009 included "scope and purpose," "rigor of development," and "clarity and presentation." Domains with a median score < 50 points were "stakeholder involvement," "applicability," and "editorial independence." Scores in all domains except "stakeholder involvement" were higher during the second half of the period than during the first half of the period, although P values were 0.10-0.93. Crude scores remained low for items 5, 7, 19, 20, 22, and 23, and the inter-quartile ranges of items 2, 6, 10, and 22 were wide. Kappa statistics ranged from -0.02 to 0.64, and they were especially low for items 3, 5, 7, 18, and 23. Guideline quality has tended to improve during the 10 years since the society started this activity. However, issues remain to be improved through continuous revisions.