Identification of randomized controlled trials from the emergency medicine literature: Comparison of hand searching versus MEDLINE searching

Study objective: As part of an ongoing project to identify all the randomized controlled trials (RCTs) in the emergency medicine literature, in association with the Cochrane Collaboration, 2 discrete studies were undertaken; the first, to compare motives for active participation in hand searching of the literature by emergency medicine professionals, and the second, to compare hand searching with MEDLINE searching of a number of emergency medicine journals. Methods: All listed members of the British Association for Emergency Medicine (BAEM) and the Society for Academic Emergency Medicine (SAEM) received a standard letter outlining the objectives of the project, with 1 of 3 headings assigned on an alternate basis. Recruited volunteers hand searched journals prioritized from the emergency medicine literature. Each issue of each journal was hand searched for RCTs. In addition, a comprehensive MEDLINE search was conducted for each journal. The yields of RCTs from the 2 searching methods were compared for all journals and for each journal individually. Results: No clear motivation for participation in this work could be ascertained because of the low response rates from BAEM and SAEM (10.1% and 1.8%, respectively). Only 18 (29.0%) of the 62 journals identified were indexed by MEDLINE. In the 14 journals indexed by MEDLINE for which hand searching was completed, a total of 710 RCTs were identified by a combination of the 2 approaches; of these, 592 (83.4%) were identified by hand searching alone and 483 (68.0%) by MEDLINE searching alone. Both methods identified 365 (51.4%) RCTs; hand searching revealed an additional 227 (32.0%) that were not identified by MEDLINE searching, and MEDLINE searching found 118(16.6%)that were not identified by hand searching. The difference between the proportions identified by hand searching and by MEDLINE searching (15.4%; 95% confidence interval [CI], 12.7% to 17.9%) was statistically significant (McNemar's chi(2) test, 1 df, 33.8; P<.0001). This difference was not significant for 8 of the journals. Conclusion: The response rates from mailing to members of the relevant professional organizations letters requesting participation in this work were very low and suggested that such an approach was not cost-effective. However, no formal costing exercise was undertaken. Searching results showed that, in the 14 emergency medicine journals indexed by MEDLINE for which hand searching was completed, hand searching led to identification of additional RCTs (primary articles) not found through MEDLINE searching. However, hand searching, although statistically significantly better than MEDLINE searching, failed to identify some of the RCTs found by MEDLINE searching, suggesting that hand searching is not a "gold standard" method and that the dual approach, promoted by the Cochrane Collaboration, may be the optimal approach for journals indexed by MEDLINE.