View More View Less
  • 1 ISSTI-Institute for the Study of Science, Technology and Innovation, The University of Edinburgh, Old Surgeons’ Hall, Edinburgh EH1 1LZ, Scotland, UK
Restricted access

Abstract

Concerns that the growing competition for funding and citations might distort science are frequently discussed, but have not been verified directly. Of the hypothesized problems, perhaps the most worrying is a worsening of positive-outcome bias. A system that disfavours negative results not only distorts the scientific literature directly, but might also discourage high-risk projects and pressure scientists to fabricate and falsify their data. This study analysed over 4,600 papers published in all disciplines between 1990 and 2007, measuring the frequency of papers that, having declared to have “tested” a hypothesis, reported a positive support for it. The overall frequency of positive supports has grown by over 22% between 1990 and 2007, with significant differences between disciplines and countries. The increase was stronger in the social and some biomedical disciplines. The United States had published, over the years, significantly fewer positive results than Asian countries (and particularly Japan) but more than European countries (and in particular the United Kingdom). Methodological artefacts cannot explain away these patterns, which support the hypotheses that research is becoming less pioneering and/or that the objectivity with which results are produced and published is decreasing.

  • Atkin, PA 2002 A paradigm shift in the medical literature. British Medical Journal 325 7378 14501451 .

  • Bian, ZX, Wu, TX 2010 Legislation for trial registration and data transparency. Trials 11:64 .

  • Bonitz, M, Scharnhorst, A 2001 Competition in science and the Matthew core journals. Scientometrics 51 1 3754 .

  • Browman, HI 1999 The uncertain position, status and impact of negative results in marine ecology: Philosphical and practical considerations. Marine Ecology Progress Series 191:301309 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Csada, RD, James, PC, Espie, RHM 1996 The “file drawer problem” of non-significant results: Does it apply to biological research?. Oikos 76 3 591593 .

  • L de Meis Velloso, A, Lannes, D, Carmo, MS C de Meis 2003 The growing competition in Brazilian science: Rites of passage, stress and burnout. Brazilian Journal of Medical and Biological Research 36 9 11351141 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • M De Rond Miller, AN 2005 Publish or perish—Bane or boon of academic life?. Journal of Management Inquiry 14 4 321329 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Delong, JB, Lang, K 1992 Are all economic hypotheses false. Journal of Political Economy 100 6 12571272 .

  • Doucouliagos, H, Laroche, P, Stanley, TD 2005 Publication bias in union-productivity research?. Relations Industrielles-Industrial Relations 60 2 320347.

    • Search Google Scholar
    • Export Citation
  • Dwan, K, Altman, DG, Arnaiz, JA, Bloom, J, Chan, A-W, Cronin, E et al. 2008 Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3 8 e3081 [Research Support, Non-U.S. Gov't; Review].

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Evanschitzky, H, Baumgarth, C, Hubbard, R, Armstrong, JS 2007 Replication research's disturbing trend. Journal of Business Research 60 4 411415 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fanelli, D 2010 Do pressures to publish increase scientists’ bias? An empirical support from US States Data. Plos One 5 4 e10271 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fanelli, D 2010 “Positive” results increase down the hierarchy of the sciences. Plos One 5 3 e10068 .

  • Feigenbaum, S, Levy, DM 1996 Research bias: Some preliminary findings. Knowledge and Policy: The International Journal of Knowledge Transfer and Utilization 9 2 & 3 135142.

    • Search Google Scholar
    • Export Citation
  • Formann, AK 2008 Estimating the proportion of studies missing for meta-analysis due to publication bias. Contemporary Clinical Trials 29 5 732739 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fronczak, P, Fronczak, A, Holyst, JA 2007 Analysis of scientific productivity using maximum entropy principle and fluctuation-dissipation theorem. Physical Review E 75 2 026103 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gad-el-Hak, M 2004 Publish or perish—An ailing enterprise?. Physics Today 57 3 6162 .

  • Gerber, AS, Malhotra, N 2008 Publication bias in empirical sociological research—Do arbitrary significance levels distort published results?. Sociological Methods & Research 37 1 330 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Howard, GS, Hill, TL, Maxwell, SE, Baptista, TM, Farias, MH, Coelho, C et al. 2009 What's wrong with research literatures? And how to make them right. Review of General Psychology 13 2 146166 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hubbard, R, Vetter, DE 1996 An empirical comparison of published replication research in accounting, economics, finance, management, and marketing. Journal of Business Research 35 2 153164 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ioannidis, JPA 2005 Why most published research findings are false. Plos Medicine 2 8 696701 .

  • Ioannidis, JPA 2006 Evolution and translation of research findings: From to where?. Plos Clinical Trials 1:e36 .

  • Ioannidis, JPA 2008 Perfect study, poor evidence: Interpretation of biases preceding study design. Seminars in Hematology 45 3 160166 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ioannidis, JPA 2008 Why most discovered true associations are inflated. Epidemiology 19 5 640648 .

  • Ioannidis, JPA, Ntzani, EE, Trikalinos, TA, Contopoulos-Ioannidis, DG 2001 Replication validity of genetic association studies. Nature Genetics 29 3 306309 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ioannidis, JPA, Trikalinos, TA 2005 Early extreme contradictory estimates may appear in published research: The proteus phenomenon in molecular genetics research and randomized trials. Journal of Clinical Epidemiology 58 6 543549 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jeng, M 2006 A selected history of expectation bias in physics. American Journal of Physics 74 7 578583 .

  • Jennions, MD, Moller, AP 2002 Publication bias in ecology and evolution: An empirical assessment using the ‘trim and fill’ method. Biological Reviews 77 2 211222 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jennions, MD, Moller, AP 2003 A survey of the statistical power of research in behavioral ecology and animal behavior. Behavioral Ecology 14 3 438445 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jones, KS, Derby, PL, Schmidlin, EA 2010 An investigation of the prevalence of replication research in human factors. Human Factors 52 5 586595 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kelly, CD 2006 Replicating empirical research in behavioral ecology: How and why it should be done but rarely ever is. Quarterly Review of Biology 81 3 221236 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • King, DA 2004 The scientific impact of nations. Nature 430 6997 311316 .

  • Knight, J 2003 Negative results: Null and void. Nature 422 6932 554555 .

  • Kundoor, V, Ahmed, MKK 2010 Uncovering negative results: Introducing an open access journal “Journal of Pharmaceutical Negative Results”. Pharmacognosy Magazine 6 24 345347 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kyzas, PA, Denaxa-Kyza, D, Ioannidis, JPA 2007 Almost all articles on cancer prognostic markers report statistically significant results. European Journal of Cancer 43 17 25592579 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Larsen, PO M von Ins 2010 The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84 3 575603 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lawrence, PA 2003 The politics of publication—Authors, reviewers and editors must act to protect the quality of research. Nature 422 6929 259261 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lortie, CJ 1999 Over-interpretation: Avoiding the stigma of non-significant results. Oikos 87 1 183184 .

  • Maddock, JE, Rossi, JS 2001 Statistical power of articles published in three health psychology-related journals. Health Psychology 20 1 7678 .

  • Marsh, DM, Hanlon, TJ 2007 Seeing what we want to see: Confirmation bias in animal behavior research. Ethology 113 11 10891098 .

  • Meho, LI 2007 The rise and rise of citation analysis. Physics World 20 1 3236.

  • Nicolini, C, Nozza, F 2008 Objective assessment of scientific performances world-wide. Scientometrics 76 3 527541 .

  • Osuna, C, Crux-Castro, L, Sanz-Menedez, L 2011 Overturning some assumptions about the effects of evaluation systems on publication performance. Scientometrics 86:575592 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Palmer, AR 2000 Quasireplication and the contract of error: Lessons from sex ratios, heritabilities and fluctuating asymmetry. Annual Review of Ecology and Systematics 31:441480 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pautasso, M 2010 Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics 85 1 193202 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Qiu, J 2010 Publish or perish in China. Nature 463 7278 142143 .

  • Schmidt, S 2009 Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology 13 2 90100 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Shelton, R. D., Foland, P., & Gorelskyy, R. (2007). Do new SCI journals have a different national bias? Proceedings of ISSI 2007: 11th international conference of the international society for scientometrics and informetrics, Vols I and II (pp. 708717).

    • Search Google Scholar
    • Export Citation
  • Shelton, RD, Foland, P, Gorelskyy, R 2009 Do new SCI journals have a different national bias?. Scientometrics 79 2 351363 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Silvertown, J, McConway, KJ 1997 Does “publication bias” lead to biased science?. Oikos 79 1 167168 .

  • Simera, I, Moher, D, Hirst, A, Hoey, J, Schulz, KF, Altman, DG 2010 Transparent and accurate reporting increases reliability, utility, and impact of your research: Reporting guidelines and the EQUATOR Network. Bmc Medicine 8:24 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Song, F, Parekh, S, Hooper, L, Loke, YK, Ryder, J, Sutton, AJ et al. 2010 Dissemination and publication of research findings: An updated review of related biases. Health Technology Assessment 14 8 1193 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Statzner, B, Resh, VH 2010 Negative changes in the scientific publication process in ecology: Potential causes and consequences. Freshwater Biology 55 12 26392653 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Steen, RG 2011 Retractions in the scientific literature: Do authors deliberately commit research fraud?. Journal of Medical Ethics 37 2 113117 .

  • Sterling, TD, Rosenbaum, WL, Weinkam, JJ 1995 Publication decisions revisited—The effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician 49 1 108112 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tsang, EWK, Kwan, KM 1999 Replication and theory development in organizational science: A critical realist perspective. Academy of Management Review 24 4 759780.

    • Search Google Scholar
    • Export Citation
  • Warner, J 2000 A critical review of the application of citation studies to the Research Assessment Exercises. Journal of Information Science 26 6 453459 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Young, NS, Ioannidis, JPA, Al-Ubaydi, O 2008 Why current publication practices may distort science. Plos Medicine 5 10 14181422 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yousefi-Nooraie, R, Shakiba, B, Mortaz-Hejri, S 2006 Country development and manuscript selection bias: A review of published studies. BMC Medical Research Methodology 6:37 .

    • Crossref
    • Search Google Scholar
    • Export Citation

Manuscript submission: http://www.editorialmanager.com/scim/

  • Impact Factor (2019): 2.867
  • Scimago Journal Rank (2019): 1.210
  • SJR Hirsch-Index (2019): 106
  • SJR Quartile Score (2019): Q1 Computer Science Apllications
  • SJR Quartile Score (2019): Q1 Library and Information Sciences
  • SJR Quartile Score (2019): Q1 Social Sciences (miscellaneous)
  • Impact Factor (2018): 2.770
  • Scimago Journal Rank (2018): 1.113
  • SJR Hirsch-Index (2018): 95
  • SJR Quartile Score (2018): Q1 Library and Information Sciences
  • SJR Quartile Score (2018): Q1 Social Sciences (miscellaneous)

For subscription options, please visit the website of Springer

Scientometrics
Language English
Size B5
Year of
Foundation
1978
Volumes
per Year
4
Issues
per Year
12
Founder Akadémiai Kiadó
Founder's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Publisher Akadémiai Kiadó
Springer Nature Switzerland AG
Publisher's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
CH-6330 Cham, Switzerland Gewerbestrasse 11.
Responsible
Publisher
Chief Executive Officer, Akadémiai Kiadó
ISSN 0138-9130 (Print)
ISSN 1588-2861 (Online)