The ROBINS-I and the NOS had similar reliability but differed in applicability: A random sampling observational studies of systematic reviews/meta-analysis

Objective: There is a lack of evidence on the usage of the quality assessment tool-the Risk Of Bias In Nonrandomized Studies-of Interventions (ROBINS-I). This article aimed to measure the reliability, criterion validity, and feasibility of the ROBINS-I and the Newcastle-Ottawa Scale (NOS). Methods: A sample of systematic reviews or meta-analyses of observational studies were selected from Medline (2013-2017) and assessed by two reviewers using ROBINS-I and the NOS. We reported on reliability in terms of the first-order agreement coefficient (AC1) statistic. Correlation coefficient statistic was used to explore the criterion validity of the ROBINS-I. We compared the feasibility of the ROBINS-I and NOS by recording the time to complete an assessment and the instances where assessing was difficult. Results: Five systematic reviews containing 41 cohort studies were finally included. Interobserver agreement on the individual domain of the ROBINS-I as well as the NOS was substantial with a mean AC1 statistic of 0.67 (95% CI: 0.50-0.83) and 0.73 (95% CI: 0.65-0.81), respectively. The criterion validity of the ROBNS-I was moderate (K = 0.52) against NOS. The time in assessing a single study by ROBINS-I varied from 7 hours initially to 3 hours compared with 30 minutes for the NOS. Both reviewers rated "bias due to departure from the intended interventions" the most time-consuming domain in the ROBINS-I, items in the NOS were equal. Conclusions: The ROBINS-I and the NOS seem to provide the same reliability but vary in applicability. The over-complicated feature of ROBINS-I may limit its usage and a simplified version is needed.