A Study of an Automatic Stopping Strategy for Technologically Assisted Medical Reviews

Systematic medical reviews are a method to collect the findings from multiple studies in a reliable way. Given budget and time constraints, limiting the recall of a search may undermine the quality of a review to such a degree that the validity of its findings is questionable. In this paper, we investigate a variable threshold approach to tackle the problem of a total recall task in medical reviews proposed by a Cross-Language Evaluation Forum (CLEF) eHealth lab in 2017. Compared to the official results submitted to the CLEF eHealth task, our approach performed consistently better over all the range of thresholds considered achieving a recall greater than 0.95 with 25,000 documents less than the best performing systems. The runs and the source code to generate the analyses of this paper are available at the following GitHub repository (https://github.com/gmdn/ECIR2018).