View More View Less
  • 1 Department of e-Resources, University Library, Stockholm University, 106 91, Stockholm, Sweden
  • 2 Department of Sociology, Inforsk, Umeå University, 901 87, Umeå, Sweden; cristian.coliander@soc.umu.se; olle.persson@soc.umu.se
Restricted access

Abstract

We compared three different bibliometric evaluation approaches: two citation-based approaches and one based on manual classification of publishing channels into quality levels. Publication data for two universities was used, and we worked with two levels of analysis: article and department. For the article level, we investigated the predictive power of field normalized citation rates and field normalized journal impact with respect to journal level. The results for the article level show that evaluation of journals based on citation impact correlate rather well with manual classification of journals into quality levels. However, the prediction from field normalized citation rates to journal level was only marginally better than random guessing. At the department level, we studied three different indicators in the context of research fund allocation within universities and the extent to which the three indicators produce different distributions of research funds. It turned out that the three distributions of relative indicator values were very similar, which in turn yields that the corresponding distributions of hypothetical research funds would be very similar.

  • Auranen, O, Nieminen, M. University research funding and publication performance-An international comparison. Research Policy 2010 39 6 822834 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Braun, T, Glänzel, W. United Germany–the new scientific superpower. Scientometrics 1990 19 5–6 513521 .

  • Butler, L 2004 What happens when funding is linked to publication counts? HF Moed W Glänzel U Schmoch eds. Handbook of quantitative science and technology research Kluwer Dordrecht 389405.

    • Search Google Scholar
    • Export Citation
  • Debackere, K, Glänzel, W. Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key. Scientometrics 2004 59 2 253276 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fawcett, T. An introduction to ROC analysis. Pattern Recognition Letters 2006 27 8 861874 .

  • Geuna, A. The changing rationale for European university research funding: Are there negative unintended consequences?. Journal of Economic Issues 2001 35 3 607632.

    • Search Google Scholar
    • Export Citation
  • Geuna, A, Martin, BR. University research evaluation and funding: An international comparison. Minerva 2003 41 4 277304 .

  • Hosmer, DW, Lemeshow, S 2000 Applied logistic regression 2 Wiley New York .

  • Lundberg, J. Lifting the crown-citation z-score. Journal of Informetrics 2007 1 2 145154 .

  • Moed, HF RE De Bruin TN van Leeuwen 1995 New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications. Scientometrics 33 3 381422 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Opthof, T, Leydesdorff, L. Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics 2010 4 3 423430 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sandström, U, Sandström, E. The field factor: towards a metric for academic institutions. Research Evaluation 2009 18 3 243250 .

  • Schneider, JW. An outline of the bibliometric indicator used for performance-based funding of research institutions in Norway. European Political Science 2009 8 3 364378 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Seglen, PO. Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science 1994 45 1 111 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sivertsen, G. A performance indicator based on complete data for the scientific publication output at research institutions. ISSI Newsletter 2010 6 1 2228.

    • Search Google Scholar
    • Export Citation
  • AFJ van Raan 1996 Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics 36 3 397420 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weingart, P. Impact of bibliometrics upon the science system: Inadvertent consequences?. Scientometrics 2005 62 1 117131 .