View More View Less
  • 1 Max Planck Society, Administrative Headquarters, Hofgartenstr. 8, 80539 Munich, Germany
Restricted access

Abstract

Purpose—this paper aims to look at the Hawthorne effect in editorial peer review. Design/methodology/approach—discusses the quality evaluation of refereed scholarly journals. Findings—a key finding of this research was that in the peer review process of one and the same manuscript, reviewers or editors, respectively, arrive at different judgments. This phenomenon is named as “Hawthorne effect” because the different judgements are dependent on the specific conditions under which the peer review process at the individual journals takes place. Originality/value—provides a discussion on the quality evaluation of scholarly journals.

  • Abelson, PH 1980 Scientific communication. Science 209 4452 6062 .

  • Bakanic, V, McPhail, C, Simon, RJ 1987 The manuscript review and decision-making process. American Sociological Review 52 5 631642 .

  • Benda, WGG, Engels, TCE 2011 The predictive validity of peer review: a selective review of the judgmental forecasting qualities of peers, and implications for innovation in science. International Journal of Forecasting 27 1 166182 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L 2008 Scientific peer review. An analysis of the peer review process from the perspective of sociology of science theories. Human Architecture—Journal of the Sociology of Self-Knowledge 6 2 2338.

    • Search Google Scholar
    • Export Citation
  • Bornmann, L 2011 Scientific peer review. Annual Review of Information Science and Technology 45:199245.

  • Bornmann, L, Daniel, H-D 2008 The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition 47 38 71737178 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2008 Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology 59 11 18411852 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bunting, C. (2005, 25 February). Early careers spent grinding teeth, not cutting them. Times Higher Education Supplement, 18.

  • Callaham, M, McCulloch, C 2011 Longitudinal trends in the performance of scientific peer reviewers. [Proceedings Paper]. Annals of Emergency Medicine 57 2 141148 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chew, FS 1991 Fate of manuscripts rejected for publication in the AJR. American Journal of Roentgenology 156 3 627632.

  • Chiesa, M, Hobbs, S 2008 Making sense of social research: how useful is the Hawthorne Effect? [Article]. European Journal of Social Psychology 38 1 6774 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cole, S 1992 Making science, between nature and society Harvard University Press Cambridge.

  • Cronin, B, McKenzie, G 1992 The trajectory of rejection. Journal of Documentation 48 3 310317 .

  • Daniel, H-D 1993 Guardians of science, fairness and reliability of peer review Wiley-VCH Weinheim.

  • DR de Vries Marschall, EA, Stein, RA 2009 Exploring the peer review process: What is it, does it work, and can it be improved?. Fisheries 34 6 270279 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eisenhart, M 2002 The paradox of peer review: Admitting too much or allowing too little?. Research in Science Education 32 2 241255 .

  • Fogg, L, Fiske, DW 1993 Foretelling the judgments of reviewers and editors. American Psychologist 48 3 293294 .

  • French, JRP 1953 Experiments in field settings L Festinger D Katz eds. Research methods in the behavioral sciences Holt, Rinehart and Winston New York 98135.

    • Search Google Scholar
    • Export Citation
  • Gans, JS, Shepherd, GB 1994 How are the mighty fallen—rejected classic articles by leading economists. Journal of Economic Perspectives 8 1 165179 .

  • Gordon, MD 1984 How authors select journals—a test of the reward maximization model of submission behavior. Social Studies of Science 14 1 2743 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gorman, GE 2007 The Oppenheim effect in scholarly journal publishing. Online Information Review 31 4 417419 .

  • Hojat, M, Gonnella, JS, Caelleigh, AS 2003 Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process. Advances in Health Sciences Education 8 1 7596 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jayasinghe, UW, Marsh, HW, Bond, N 2001 Peer review in the funding of research in higher education: the Australian experience. Educational Evaluation and Policy Analysis 23 4 343346 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Khosla, A, McDonald, RJ, Bornmann, L, Kallmes, DF 2011 Getting to yes: the fate of neuroradiology manuscripts rejected by Radiology over a 2-year period. Radiology 260 1 35 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lipworth, WL, Kerridge, IH, Carter, SM, Little, M 2011 Journal peer review in context: a qualitative study of the social and subjective dimensions of manuscript review in biomedical publishing. [Review]. Social Science and Medicine 72 7 10561063 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lock, S 1985 A difficult balance: editorial peer review in medicine ISI Press Philadelphia.

  • Marsh, HW, Jayasinghe, UW, Bond, NW 2008 Improving the peer-review process for grant applications—reliability, validity, bias, and generalizability. American Psychologist 63 3 160168 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Martin, B 2000 Research grants: problems and options. Australian Universities’ Review 43 2 1722.

  • McCook, A 2006 Is peer review broken?. The Scientist 20 2 26.

  • McDonald, RJ, Cloft, HJ, Kallmes, DF 2007 Fate of submitted manuscripts rejected from the American Journal of Neuroradiology: outcomes and commentary. American Journal of Neuroradiology 28 8 14301434 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Opthof, T, Furstner, F M van Geer Coronel, R 2000 Regrets or no regrets? No regrets! The fate of rejected manuscripts. Cardiovascular Research 45 1 255258 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Owen, R 1982 Reader bias. Journal of the American Medical Association 247 18 25332534 .

  • Petty, RE, Fleming, MA 1999 The review process at PSPB: correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin 25 2 188203 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pruthi, S, Jain, A, Wahid, A, Mehra, K, Nabi, SA 1997 Scientific community and peer review system—a case study of a central government funding scheme in India. Journal of Scientific & Industrial Research 56 7 398407.

    • Search Google Scholar
    • Export Citation
  • Pulverer, B 2010 Transparency showcases strength of peer review. [10.1038/468029a]. Nature 468 7320 2931 .

  • Ray, J, Berkwits, M, Davidoff, F 2000 The fate of manuscripts rejected by a general medical journal. American Journal of Medicine 109 2 131135 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ross, PF 1980 The sciences’ self-management: manuscript refereeing, peer review, and goals in science The Ross Company Lincoln.

  • Sharp, DW 1990 What can and should be done to reduce publication bias—the perspective of an editor. Journal of the American Medical Association 263 10 13901391 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Shatz, D 2004 Peer review: a critical inquiry Rowman & Littlefield Lanham.

  • Sternberg, RJ, Hojjat, M, Brigockas, MG, Grigorenko, EL 1997 Getting in: criteria for acceptance of manuscripts in Psychological Bulletin, 1993–1996. Psychological Bulletin 121 2 321323 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • S van Rooyen Godlee, F, Evans, S, Smith, R, Black, N 1998 Effect of blinding and unmasking on the quality of peer review—a randomized trial. Journal of the American Medical Association 280 3 234237 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weller, AC 1996 Editorial peer review: a comparison of authors publishing in two groups of US medical journals. Bulletin of the Medical Library Association 84 3 359366.

    • Search Google Scholar
    • Export Citation
  • Weller, AC 2002 Editorial peer review: its strengths and weaknesses Information Today, Inc Medford.

  • Whitman, N, Eyre, S 1985 The pattern of publishing previously rejected articles in selected journals. Family Medicine 17 1 2628.

  • Wood, FQ, Wessely, S 2003 Peer review of grant applications: a systematic review F Godlee T Jefferson eds. Peer review in health sciences 2 BMJ Books London 1444.

    • Search Google Scholar
    • Export Citation
  • Ziman, J 1968 Public knowledge: an essay concerning the social dimension of science Cambridge University Press Cambridge.

  • Ziman, J 2000 Real science, what it is, and what it means Cambridge University Press Cambridge .

  • Zuckerman, H, Merton, RK 1971 Patterns of evaluation in science: institutionalisation, structure and functions of the referee system. Minerva 9 1 66100 .

    • Crossref
    • Search Google Scholar
    • Export Citation

Manuscript submission: http://www.editorialmanager.com/scim/

  • Impact Factor (2019): 2.867
  • Scimago Journal Rank (2019): 1.210
  • SJR Hirsch-Index (2019): 106
  • SJR Quartile Score (2019): Q1 Computer Science Apllications
  • SJR Quartile Score (2019): Q1 Library and Information Sciences
  • SJR Quartile Score (2019): Q1 Social Sciences (miscellaneous)
  • Impact Factor (2018): 2.770
  • Scimago Journal Rank (2018): 1.113
  • SJR Hirsch-Index (2018): 95
  • SJR Quartile Score (2018): Q1 Library and Information Sciences
  • SJR Quartile Score (2018): Q1 Social Sciences (miscellaneous)

For subscription options, please visit the website of Springer

Scientometrics
Language English
Size B5
Year of
Foundation
1978
Volumes
per Year
4
Issues
per Year
12
Founder Akadémiai Kiadó
Founder's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Publisher Akadémiai Kiadó
Springer Nature Switzerland AG
Publisher's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
CH-6330 Cham, Switzerland Gewerbestrasse 11.
Responsible
Publisher
Chief Executive Officer, Akadémiai Kiadó
ISSN 0138-9130 (Print)
ISSN 1588-2861 (Online)