View More View Less
  • 1 CSIC Institute of Public Goods and Policies (IPP-CCHS), Consejo Superior de Investigaciones Científicas (CSIC), C/Albasanz 26-28, E-28037, Madrid, Spain
Restricted access

Abstract

In 1989 the Spanish Government established an individual retrospective research evaluation system (RES) for public researchers. Policy makers have associated the establishment of this evaluation system with the significant increase in the volume of scientific publications attributed to Spain over the last decades. In a similar vein to the analyses of other country cases, some scholars have also claimed that the growth of Spain's international scientific publications is a result of the establishment of the new evaluation system. In this paper, we provide a methodological revision of the validity threats in previous research, including some interrupted time-series analyses and control groups to investigate the effects of this policy instrument on the number of papers produced by Spanish authors. In the years following the establishment of the evaluation system, the results indicate a considerable increase in the number of papers attributed to Spanish authors among those eligible for evaluation (the “treated” group), but also in the control groups. After testing various alternative explanations, we conclude that the growth in Spanish publications cannot be attributed indisputably to the effect of the establishment of the RES, but rather to the increase of expenditure and number of researchers in the Spanish R&D system along with some maturation effects. We take this case as an example of the need to improve and refine methodologies and to be more cautious when attributing effects to research evaluation mechanisms at the national level.

  • Andersen, LB, Pallesen, T 2008 “Not Just for the Money?” How financial incentives affect the number of publications at Danish Research Institutions. International Public Management Journal 11 1 2847 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Braun, T, Schubert, A 1988 World flash on basic research: Scientometric versus socio-economic indicators. Scatter plots for 51 countries. 1978–1980. Scientometrics 13 1–2 19.

    • Search Google Scholar
    • Export Citation
  • Braun, T, Glänzel, W, Schubert, A 1985 Scientometrics indicators. A 32-country comparative evaluation of publishing performance and citation impact World Scientific Singapore, Philadelphia.

    • Search Google Scholar
    • Export Citation
  • Butler, L 2003 Explaining Australia's increased share of ISI publications. The effects of a funding formula based on publication counts. Research Policy 32 1 143155 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Butler, L 2003 Modifying publication practices in response to funding formulas. Research Evaluation 12 1 3946 .

  • Campbell, DFJ 2003 The evaluation of university research in the United Kingdom and the Netherlands, Germany and Austria Ph Shapira S Kuhlman eds. Learning from science and technology policy evaluation Edward Elgar Cheltenham (UK) 98131.

    • Search Google Scholar
    • Export Citation
  • Campbell, DT 1996 Regression artifacts in time-series and longitudinal data. Evaluation and Program Planning 19 4 377389 .

  • Campbell, DT, Ross, HL 1968 The Connecticut crackdown on speeding. Time-series data in quasi-experimental analysis. Law and Society Review 3 1 3254 .

  • Cook, TD, Campbell, DT 1979 Quasi-experimentation: Design and analysis issues for field settings Houghton Mifflin Co Boston.

  • Crespi, GA, Geuna, A 2008 An empirical study of scientific production: A cross country analysis (1981–2002). Research Policy 37 4 565579 .

  • Cruz-Castro, L, Sanz-Menéndez, L 2007 Research evaluation in transition: individual versus organisational assessment in Spain R Whitley J Glaser eds. The changing governance of the sciences. The advent of the research evaluation systems Springer Dordrecht (NL) 205224 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cruz-Castro, L., Sanz-Menéndez, L. & Martínez, C. (2010). Research centers in transition: patterns of convergence and diversity. The Journal of Technology Transfer, doi: (forthcoming).

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Georghiou, L., Howells, J., Rigby, J. Glynn, S., Butler, J., Cameron, H., et al. (2000). Impact of the research assessment exercise and the future of quality assurance in the light of changes in the research landscape. A report produced by PREST, University of Manchester for HEFCE Higher Education Funding Council for England. April 2000. http://www.mbs.ac.uk/research/innovation/publications-archive/reports.aspx. Accessed 10 May 2010.

    • Search Google Scholar
    • Export Citation
  • Geuna, A, Martin, B 2003 University research evaluation and funding: An international comparison. Minerva 41 4 277304 .

  • Gläser, J 2007 The social orders of research evaluation systems R Whitley J Glaser eds. The changing governance of the sciences. The advent of the research evaluation systems Springer Dordrecht (NL) 245266 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gómez, I, Fernández, MT, Zulueta, MA, Camí, J 1995 Analysis of biomedical research in Spain. Research Policy 24 3 459471 .

  • Gómez, I, Sancho, R, Bordons, M, Fernández, MT 2006 La I+D en España a través de publicaciones y patentes J Sebastián E Muñoz eds. Radiografía de la investigación pública en España Biblioteca Nueva Madrid 275302.

    • Search Google Scholar
    • Export Citation
  • Hicks, D 2009 Evolving regimes of multi-university research evaluation. Higher Education 57 4 393404 .

  • Ingwersen, P, Jacobs, D 2004 South African research in selected scientific areas: Status 1981–2000. Scientometrics 59 3 405423 .

  • Jiménez-Contreras, E, Ferreiro-Aláez, L 1996 Publishing abroad: Fair trade or short sell for non-english-speaking authors: A Spanish Study. Scientometrics 36 1 8195 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jiménez-Contreras, E, Moya-Anegón, F, López-Cózar, E 2003 The evolution of research activity in Spain. The impact of National Commission for the Evaluation of Research Activity (CNEAI). Research Policy 32 1 123142 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Laudel, G 2006 The art of getting funding: How scientists adapt to their funding conditions. Science and Public Policy 33 7 489504 .

  • Liefner, I 2003 Funding, resources allocation, and performance in higher education systems. Higher Education 46 4 469489 .

  • Méndez, A, Gómez, I 1986 The Spanish scientific productivity through eight international databases. Scientometrics 10 3–4 207219 .

  • Moed, H 2008 UK research assessment exercises: Informed judgements on research quality or quantity?. Scientometric 74 1 153161 .

  • Moed, HF, Debruin, RE TN Van Leeuwen 1995 New bibliometric tools for the assessment of national research performance: Data base description, overview of indicators and first applications. Scientometrics 33 3 381422 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moed, HF, Leeuwen, TN, Visser, MS 1999 Trends in publication output and impact of universities in the Netherlands. Research Evaluation 8 1 6067 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mohr, LB 2000 Regression artifacts and other customs of dubious desert. Evaluation and Program Planning 23 4 397409 .

  • Moya-Anegón, F, Herrero-Solana, V 1999 Science in America Latina: A comparison of bibliometric and scientific-technical indicators. Scientometrics 46 2 299320 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moya-Anegón, F et al. 2007 Indicadores bibliométricos de la actividad científica Española (1990–2004) FECYT Madrid.

  • Önder, C, Sevkli, M, Altinok, T, Tavukçuoglu, C 2008 Institutional change and scientific research: A preliminary bibliometric analysis of institutional influences on Turkey's recent social science publications. Scientometrics 76 3 543560 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Persson, O, Glänzel, W, Danell, R 2004 Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics 60 3 421432 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rodriguez-Navarro, A 2009 Sound research, unimportant discoveries: Research, universities, and formal evaluation of research in Spain. Journal of the American Society for Information Science and Technology 60 9 18451858 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sanz-Menéndez, L 1995 Policy choices, institutional constraints and policy learning: The Spanish science and technology policy in the eighties. International Journal of Technology Management 10 4/5/6 255274.

    • Search Google Scholar
    • Export Citation
  • Sanz-Menéndez, L 1995 Research actors and the state: Research evaluation and evaluation of science and technology policies in Spain. Research Evaluation 5 1 7988.

    • Search Google Scholar
    • Export Citation
  • Sanz-Menéndez, L 1997 Estado, ciencia y tecnología en España (1939–1997) Alianza Editorial Madrid.

  • Sanz Menéndez, L, Pfretzchner, J 1992 Política científica y gestión de la investigación: El CSIC (1986–1990) en el sistema español de ciencia y tecnología. Arbor 557 Mayo 951.

    • Search Google Scholar
    • Export Citation
  • Scimago, G 2006 Producción ISI y tramos de investigación: Cómo combinarlos en un nuevo indicador. El profesional de la información 15 3 227228 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Scimago, G 2007 Producción ISI y tramos de investigación: Cómo combinarlos en un nuevo indicador (II). El profesional de la información 16 4 510511.

    • Search Google Scholar
    • Export Citation
  • Shadish, WR, Cook, TD, Campbell, DT 2002 Experimental and quasi-experimental designs for generalized causal inference Houghton Mifflin Co Boston, New York.

    • Search Google Scholar
    • Export Citation
  • Shelton, RD 2008 Relations between national research investment and publication output: Application to an American paradox. Scientometrics 74 2 191205 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • SRI International (1988). Research activity in Spain, Portugal and Greece. A 1988 Bibliometric model assessment. (Report prepared by Ailes, Coward and Fresne), Science and Technology Policy Programme, SRI International, Arlington (Vi), mimeo.

    • Search Google Scholar
    • Export Citation
  • Talib, AA 2001 The continuing behavioural modification of academics since the 1992 research assessment exercise. Higher Education Review 33 3 3046.

    • Search Google Scholar
    • Export Citation
  • Taylor, J 2001 The impact of performance indicators on the work of university academics: Evidence from Australian universities. Higher Education Quarterly 55 1 4261 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • AFJ Van Raan 1997 Science as an international enterprise. Science and Public Policy 24 5 290300.

  • Westerheijden, D 1997 A solid base for decisions: Use of the VSNU research evaluation in Dutch Universities. Higher Education 33 4 397413 .

  • Whitley, R 2003 Competition and pluralism in public sciences: The impact of institutional frameworks on the organization of academic science. Research Policy 32 6 10151039 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Whitley, R 2007 The changing governance of the public sciences: The consequences of research evaluation systems for knowledge production in different countries and scientific fields R Whitley J Glaser eds. The changing governance of the sciences. The advent of the research evaluation systems Springer Dordrecht (NL) 327 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • R Whitley J Glaser eds. 2007 The changing governance of the sciences. The advent of the research evaluation systems Springer Dordrecht (NL).

    • Search Google Scholar
    • Export Citation
  • Zitt, M, Perrot, F, Barré, R 1998 The transition from “national” to “transnational” model and related measures of countries’ performance. Journal of the American Society of Information Science 49 1 3042 .

    • Crossref
    • Search Google Scholar
    • Export Citation

Manuscript submission: http://www.editorialmanager.com/scim/

  • Impact Factor (2019): 2.867
  • Scimago Journal Rank (2019): 1.210
  • SJR Hirsch-Index (2019): 106
  • SJR Quartile Score (2019): Q1 Computer Science Apllications
  • SJR Quartile Score (2019): Q1 Library and Information Sciences
  • SJR Quartile Score (2019): Q1 Social Sciences (miscellaneous)
  • Impact Factor (2018): 2.770
  • Scimago Journal Rank (2018): 1.113
  • SJR Hirsch-Index (2018): 95
  • SJR Quartile Score (2018): Q1 Library and Information Sciences
  • SJR Quartile Score (2018): Q1 Social Sciences (miscellaneous)

For subscription options, please visit the website of Springer

Scientometrics
Language English
Size B5
Year of
Foundation
1978
Volumes
per Year
4
Issues
per Year
12
Founder Akadémiai Kiadó
Founder's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Publisher Akadémiai Kiadó
Springer Nature Switzerland AG
Publisher's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
CH-6330 Cham, Switzerland Gewerbestrasse 11.
Responsible
Publisher
Chief Executive Officer, Akadémiai Kiadó
ISSN 0138-9130 (Print)
ISSN 1588-2861 (Online)