Evaluating Journal Impact Factor: a systematic survey of the pros and cons, and overview of alternative measures

 

Eugene Mech1,2, Muhammad Muneeb Ahmed2,3, Edward Tamale2, Matthew Holek2, Guowei Li2,4, Lehana Thabane2,5,6,7,8,9

 

1 Department of Biochemistry and Biomedical Sciences, McMaster University, Hamilton, ON, Canada.

2 Department of Health Research Methods, Evidence, and Impact (HEI), McMaster University, Hamilton, ON, Canada.

3 Michael G. DeGroote School of Medicine, McMaster University, Hamilton, ON, Canada.

4 Center for Clinical Epidemiology and Methodology (CCEM), Guangdong Second Provincial General Hospital, China.

5 Biostatistics Unit, Research Institute at St Joseph’s Healthcare, Hamilton, ON, Canada.

6 Department of Anesthesia, McMaster University, Hamilton, ON, Canada.

7 Department of Pediatrics, McMaster University, Hamilton, ON, Canada.

8 School of Nursing, McMaster University, Hamilton, ON, Canada.

9 School of Rehabilitation Science, McMaster University, Hamilton, ON, Canada

 

Abstract

Background: Journal Impact Factor (JIF) has several intrinsic flaws, which highlight its inability to adequately measure citation distributions or indicate journal quality. Despite these flaws, JIF is still widely used within the academic community, resulting in the propagation of potentially misleading information. A critical review of the usefulness of JIF is needed including an overview of the literature to identify viable alternative metrics. The objectives of this study are: (1) to assess the usefulness of JIF by compiling and comparing its advantages and disadvantages; (2) to record the differential uses of JIF within research environments; and (3) to summarize and compare viable alternative measures to JIF.

 

Methods: Three separate literature search strategies using MEDLINE and Web of Science were completed to address the three study objectives. Each search was completed in accordance with PRISMA guidelines. Results were compiled in tabular format and analyzed based on reporting frequency.

 

Results: For objective (1), 84 studies were included in qualitative analysis. It was found that the recorded advantages of JIF were outweighed by disadvantages (18 disadvantages vs. 9 advantages). For objective (2), 653 records were included in a qualitative analysis. JIF was found to be most commonly used in journal ranking (n = 653, 100%) and calculation of scientific research productivity (n = 367, 56.2%). For objective (3), 65 works were included in qualitative analysis. These articles revealed 45 alternatives, which includes 18 alternatives that improve on highly reported disadvantages of JIF.

 

Conclusion: JIF has many disadvantages and is applied beyond its original intent, leading to inaccurate information. Several metrics have been identified to improve on certain disadvantages of JIF. Integrated Impact Indicator (I3) shows great promise as an alternative to JIF. However, further scientometric analysis is needed to assess its properties.

 

Keywords: Journal Impact Factor Bibliometrics Alternative metrics Citations and impact

 

Correspondence: thabanl@mcmaster.ca

 

Received: 27 October 2019; Accepted: 31 July 2020; Published online: 31 August 2020.