Expert opinions leave space for uncertainty when defining rehabilitation interventions: analysis of difficult decisions regarding categorization of rehabilitation reviews in the Cochrane library

BACKGROUND: In 2017, Cochrane Rehabilitation created an online relational database to crowd-source the identification and categorization of Cochrane publications for relevance to rehabilitation. One of the challenges of this work has been the lack of an operational definition to determine what is or is not a rehabilitation intervention. As such, categorization decisions have been largely based on expert opinion, with two health professionals screening each review, and with disagreements in categorization decisions being adjudicated by the Cochrane Rehabilitation Review Committee. AIM: To analyze the rationale for resolving conflicts in the identification of rehabilitation reviews from all Cochrane reviews to contribute to future work on the scope and definition of rehabilitation interventions. METHODS: We extracted data on decisions made about all Cochrane titles (both protocols and reviews) published between 1 January, 1996, and 31 August, 2019, and identified all titles where there had been disagreement between any people categorizing the reviews. We used thematic analysis methods to classify the reasons for including or excluding reviews from a collection of reviews on rehabilitation interventions. We compared across groups to identify areas of conflict and errors in the initial categorization. RESULTS: Of the 9756 Cochrane titles screened, we identified 894 (9.2%) where some disagreement existed about whether a review was about rehabilitation interventions or not. Of these, 333 (37.2%) had met our original pragmatic criteria for being a "rehabilitation" review, while 561 (52.8%) had not. Seven hundred and nineteen of these reviews (80.4%) could be grouped by inductively created, reportable criteria to justify the initial categorization decisions. Fifty-seven reviews (6.4%) were on topics that were too idiosyncratic to easily group with others for the purposes of categorization. Conflicts in the rationale for categorization decisions were identified in 90 reviews (10.1%) and errors in the initial categorization for 28 reviews (3.1%). CONCLUSIONS: The challenges and conflicts identified in this study clearly indicate the need for better operational definition of rehabilitation interventions. This study provides a foundation for future work to check the utility of any new definition of rehabilitation interventions and to improve the trustworthiness of categorization decisions regarding the Cochrane Rehabilitation database.