Rating and Ranking of Medical Journals: A Randomised Controlled Evaluation of Impact Factor and Number of Listed Journals

Background: The impact factor is a purely bibliometric parameter built on a number of publications and their citations that occur within clearly defined periods. Appropriate interpretation of the impact factor is important as it is also used worldwide for the evaluation of research performance. Research question: It is assumed that the number of medical journals reflects the extent of diseases and patient populations involved and that the number is correlated with the level of the impact factor. Method: 174 category lists (Subject Categories) are included in the area Health Sciences of the ISI Web of Knowledge of Thomson Reuters, 71 of which belong to the field of medicine and 50 of which have a clinical and/or application-oriented focus. These alphabetically arranged 50 category lists were consecutively numbered, randomized by odd and even numbers, respectively, into 2 equal-sized groups and then grouped according to organ specialities, sub-specialities and cross-disciplinary fields. By tossing up a coin it was decided which group should be evaluated first. Only then the category lists were downloaded and the number of journals, as well as the impact factors of journals ranking number 1 and 2, as well as the impact factors of journals at the end of the first third and at the end of the first half of each category list were compared. Results: The number of journals per category list varies considerably between 5 and 252. The lists of organ specialties and cross-disciplinary fields include more than three times as many journals as those of the sub-specialities; the highest numbers of journals are listed for the cross-disciplinary fields. The level of impact factor of journals that rank number 1 in the lists varies considerably and ranges from 3,058 to 94,333; a similar variability exists for the journals at rank 2. On the other hand, the impact factor of journals at the end of the first third of the lists varies from 1,214 and 3,953, and for those journals at the end of the first half of a respective category list it varies from 0,609 and 2,872. The slope of the straight correlation line between the level of impact factors of journals at rank 1 and 2 with the number of listed journals varies from 0,0756 and 0,2651 (correlation coefficients between 0,49 and 0,96). For the journals ranking further down in the lists the straight correlation lines run almost horizontally or with inverse slope. Conclusions: This current analysis adds to the knowledge for an appropriate interpretation of the impact factor. Generally, greater importance should be given to the ranking of a journal within a corresponding category list.