Reliability among central readers in the evaluation of endoscopic disease activity in pouchitis

Background and Aims: Pouchitis is a common adverse event after proctocolectomy with ileal pouch anal anastomosis for ulcerative colitis. Evaluation of pouchitis disease activity and response to treatment requires use of validated indices. We assessed the reliability of items evaluating endoscopic pouchitis disease activity. Methods: Twelve panelists used a modified RAND appropriateness methodology to rate the appropriateness of items evaluating endoscopic pouchitis disease activity derived from a systematic review and also identified additional potential endoscopic items based on expert opinion. Four central readers then evaluated 50 pouchoscopy videos in triplicate, in random order. Intra-and inter-rater reliability for each item was assessed by calculating and comparing intraclass correlation coefficients (ICCs). A Delphi process identified common sources of disagreement among the readers. Results: Ten existing endoscopic items were identified from the systematic review and an additional 7 exploratory items from the panelists. ICCs for inter-rater reliability were highest for the existing item of pouch ulceration (.72; 95% confidence interval [CI], .60-.82) and for the exploratory item of ulcerated surface in the pouch body (.67; 95% CI, .53-.75). Inter-rater reliability for all other existing and exploratory items was "moderate" (ICC<.60). The item "ulcerated surface in the pouch body" demonstrated the best correlation with a global evaluation of lesion severity (r = .80; 95% CI, .73-.85). Conclusion: Substantial reliability was observed only for the endoscopic items of ulceration and ulcerated surface in the pouch body. Future studies should assess responsiveness to treatment in the next stage toward development of an endoscopic pouchitis disease activity index.