View More View Less
  • 1 Social Psychology and Research on Higher Education, ETH Zurich, Muehlegasse 21, 8001 Zurich, Switzerland
  • 2 Evaluation Office, University of Zurich, Muehlegasse 21, 8001 Zurich, Switzerland
Restricted access

Abstract

The journal impact factor (JIF) proposed by Garfield in the year 1955 is one of the most commonly used and prominent citation-based indicators of the performance and significance of a scientific journal. The JIF is simple, reasonable, clearly defined, and comparable over time and, what is more, can be easily calculated from data provided by Thomson Reuters, but at the expense of serious technical and methodological flaws. The paper discusses one of the core problems: The JIF is affected by bias factors (e.g., document type) that have nothing to do with the prestige or quality of a journal. For solving this problem, we suggest using the generalized propensity score methodology based on the Rubin Causal Model. Citation data for papers of all journals in the ISI subject category “Microscopy” (Journal Citation Report) are used to illustrate the proposal.

  • Allison, PD 1999 Logistic regression using SAS: Theory and application SAS Institute Inc. Cary, NC.

  • Bornmann, L, Mutz, R, Neuhaus, C, Daniel, H-D 2008 Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics 8:93102 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Braun, T, Glänzel, W, Schubert, A 1989 Some data on the distribution of journal publication types in the Science Citation Index Database. Scientometrics 15:325330 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cochran, WG 1968 The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 24 2 295313 .

  • Fan, X, Felsöváli, Á, Sivo, SA, Keenan, SC 2001 SAS for monte carlo studies: A guide for quantitative researchers SAS Institute Inc. Cary, NC.

    • Search Google Scholar
    • Export Citation
  • Feng, P., Zhou, X.-H., Zou, Q.-M., Fan, M.-Y., & Li, X.-S. (2011). Generalized propensity score for estimating the average treatment effect of multiple treatments. Statistics in Medicine. doi: (published online February 24, 2011).

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Garfield, E 1955 Citation indexes to science: A new dimension in documentation through association of ideas. Science 122:108111 .

  • Garfield, E 1999 Journal impact factor: A brief review. Journal of the Canadian Medical Association 161 8 979980.

  • Garfield, E 2006 The history and meaning of the Journal Impact Factor. Journal of the American Medical Association 295 1 9093 .

  • Glänzel, W, Moed, H 2002 Journal impact measures in bibliometric research. Scientometrics 53 2 171193 .

  • Guo, S, Fraser, MW 2010 Propensity score analysis—statistical methods and applications Sage London, UK.

  • Hirano, K, Imbens, G 2004 The propensity score with continuous treatments A Gelman X-L Meng eds. Applied Bayesian modeling and causal inference from incomplete-data perspective Wiley London 7384.

    • Search Google Scholar
    • Export Citation
  • Holland, PW 1986 Statistics and causal inference. Journal of the American Statistical Association 81:945970.

  • Imai, K DA van Dyk 2004 Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association 99 467 854866 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Imbens, G 2000 The role of propensity score in estimating dose-response functions. Biometrika 87 3 706710 .

  • Kluve, J, Schneider, H, Uhlendorff, A, Zhao, Z 2012 Evaluating continuous training programmes by using the generalized propensity score. Journal of the Royal Statistical Society A 175 Part 2 131.

    • Search Google Scholar
    • Export Citation
  • Leydesdorff, L, Bornmann, L 2011 How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology 62 2 217229 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lu, B, Greevey, R, Xu, X, Beck, C 2011 Optimal nonbipartite matching and its statistical applications. American Statistician 65 1 2130 .

  • Moed, HF TN van Leeuwen 1995 Improving the accuracy of the Institute for Scientific Information's Journal Impact Factor. Journal of the American Society of Information Science 46:461467 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moed, HF TN Van Leeuwen Reeduk, J 1999 Towards appropriate indicators of journal impact. Scientometrics 46 3 575589 .

  • Mutz, R., & Daniel, H.-D. (2012, in press). Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor. Journal of Infometrics.

    • Search Google Scholar
    • Export Citation
  • Neuhaus, C, Marx, W, Daniel, H-D 2009 The publication and citation impact profile of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: A case study on the limitations of the Journal Impact Factor. Journal of the American Society for Information Science and Technology 60 1 176183 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rosenbaum, PR 2010 Design of observational studies Springer New York .

  • Rosenbaum, PR, Rubin, DB 1984 Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association 79 387 516524.

    • Search Google Scholar
    • Export Citation
  • Rubin, DB 1974 Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66:688701 .

  • Rubin, DB 1977 Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics 2 1 126 .

  • Rubin, DB 2004 Teaching statistical inference for causal effects in experiments and observational studies. Journal of Educational and Behavioral Statistics 29 3 343367 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rubin, DB 2005 Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association 100 469 322331 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rubin, DB 2006 Matched sampling for causal effects Cambridge University Press Cambridge, UK .

  • Rubin, DB 2007 The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine 26:2036 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rubin, DB, Thomas, N 1996 Matching using estimated propensity scores: Relating theory in practice. Biometrics 52:249264 .

  • SAS Institute Inc 2009 SAS/STAT 9.2 user's guide SAS Institute Inc. Cary, NC.

  • Spreeuwenberg, MD, Bartak, A, Croon, MA, Hagenaars, JA, Bussbach, JJV, Andrea, H, Twisk, J, Stijnen, T 2010 The multiple propensity score as control for bias in the comparison of more than two treatment arms. An introduction from a case study in mental health. Medical Care 48 2 166174 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Todorov, R, Glänzel, W 1988 Journal citation measures: A concise review. Journal of Information Science 14:4756 .

  • Vanclay, J. K. (2012). Impact factor: Outdated artifact or stepping-stone to journal certification? Scientometrics (accepted paper).

  • Wang, J, Donnan, PT, Steinke, D, MacDonald, TM 2001 The multiple propensity score for analysis of dose-response relationships in drug safety studies. Pharmacoepidemiology and Drug Safety 10:105111 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zanutto, E, Lu, B, Hornik, R 2005 Using propensity score subclassification for multiple treatment doses to evaluate a national antidrug media campaign. Journal of Educational and Behavioral Statistics 30 1 5973 .

    • Crossref
    • Search Google Scholar
    • Export Citation