View More View Less
  • 1 National Research Council of Italy, IASI-CNR, Rome, Italy
  • 2 Laboratory for Studies of Research and Technology Transfer, School of Engineering, Department of Management, University of Rome “Tor Vergata”, Rome, Italy giovanni.abramo@uniroma2.itdi.costa@dii.uniroma2.it
Restricted access

Abstract

Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national research assessment exercises, as far as the hard sciences are concerned. In this work we compare rankings lists of universities captured by the first Italian evaluation exercise, through peer review, with the results of bibliometric simulations. The comparison shows the great differences between peer review and bibliometric rankings for excellence and productivity.

  • Abramo, G, D’Angelo, CA 2011 Evaluating research: From informed peer review to bibliometrics. Scientometrics 87 3 499514 .

  • Abramo, G, D’Angelo, CA, Caprasecca, A 2009 Allocative efficiency in public research funding: Can bibliometrics help?. Research Policy 38 1 206215 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Abramo, G, D’Angelo, CA, Di Costa, F 2008 Assessment of sectoral aggregation distortion in research productivity measurements. Research Evaluation 17 2 111121 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Abramo, G, D’Angelo, CA, Viel, F 2010 Peer review research assessment: A sensitivity analysis of performance rankings to the share of research product evacuate. Scientometrics 85 3 705720 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Aksnes, DW, Taxt, RE 2004 Peers reviews and bibliometric indicators: A comparative study at Norvegian University. Research Evaluation 13 1 3341 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • D’Angelo, CA, Giuffrida, C, Abramo, G 2011 A heuristic approach to author name disambiguation in large-scale bibliometric databases. Journal of the American Society for Information Science and Technology 62 2 257269 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • ERA. (2010). The Excellence in Research for Australia Initiative http://www.arc.gov.au/era/.

  • Franceschet, M, Costantini, A 2011 The first Italian research assessment exercise: A bibliometric perspective. Journal of Informetrics 5 2 275291 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Horrobin, DF 1990 The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association 263 10 14381441 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lundberg, J 2007 Lifting the crown-citation z-score. Journal of Informetrics 1 2 145154 .

  • MacRoberts, MH, MacRoberts, BR 1996 Problems of citation analysis. Scientometrics 36 3 435444 .

  • Moed, HF 2002 The impact-factors debate: The ISI's uses and limits. Nature 415:731732.

  • Moed, HF 2005 Citation analysis in research evaluation Springer Dordrecht.

  • Moxam, H, Anderson, J 1992 Peer review. A view from the inside. Science and Technology Policy 5 1 715.

  • Oppenheim, C 1997 The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology. Journal of Documentation 53:477487 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Oppenheim, C, Norris, M 2003 Citation counts and the research assessment exercise V: Archaeology and the 2001 RAE. Journal of Documentation 56 6 709730.

    • Search Google Scholar
    • Export Citation
  • Pendlebury, DA 2009 The use and misuse of journal metrics and other citation indicators. Scientometrics 57 1 111.

  • REF. (2009). Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework. http://www.hefce.ac.uk/pubs/hefce/2009/09_39/.

    • Search Google Scholar
    • Export Citation
  • Rinia, EJ, van Leeuwen, Th N HG van Vuren AFJ van Raan 1998 Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands. Research Policy 27 1 95107 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • AFJ van Raan 2005 Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics 62 1 133143 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • VTR. (2006). Valutazione Triennale (2001-2003) della Ricerca italiana (italian Triennal Assessment Excercise). Risultati delle valutazioni dei Panel di Area. http://vtr2006.cineca.it/php5/vtr_rel_civr_rapp_panel_menu.php?info=-&&sel_lingua=EN&decritta=1&versione=2&aggiorna=0&info=. Accessed July 5, 2010).

    • Search Google Scholar
    • Export Citation