View More View Less
  • 1 Max Planck Society, Administrative Headquarters, Hofgartenstr. 8, 80539, Munich, Germany
  • 2 Institute of Atmospheric and Climate Science, ETH Zurich, Universitätstrasse 16, 8092, Zurich, Switzerland
  • 3 Professorship for Social Psychology and Research on Higher Education, ETH Zurich, Mühlegasse 21, 8001, Zurich, Switzerland
  • 4 Evaluation Office, University of Zurich, Mühlegasse 21, 8001, Zurich, Switzerland
  • 5 Swiss Federal Laboratories for Materials Science and Technology, Empa, Überlandstrasse 129, 8600, Dübendorf, Switzerland
Restricted access

Abstract

Whereas in traditional peer review a few selected researchers (peers) are included in the manuscript review process, public peer review includes both invited reviewers (who write ‘reviewer comments’) and interested members of the scientific community who write comments (‘short comments’). Available to us for this investigation are 390 reviewer comments and short comments assessing 119 manuscripts submitted to the journal Atmospheric Chemistry and Physics (ACP). We conducted a content analysis of these comments to determine differences in the main thematic areas considered by the scientists in their assessment comments. The results of the analysis show that in contrast to interested members of the scientific community, reviewers focus mainly on (1) the formal qualities of a manuscript, such as writing style, (2) the conclusions drawn in a manuscript, and (3) the future “gain” that could result from publication of a manuscript. All in all, it appears that ‘reviewer comments’ better than ‘short comments’ by interested members of the scientific community support the two main functions of peer review: selection and improvement of what is published.

  • American Association for the Advancement of Science. (1998). Report in AAAS/UNESCO/ICSU Workshop on Developing Practices and Standards for Electronic Publishing in Science. Paris, October 12–14, 1998. Washington, DC: The American Association for the Advancement of Science.

    • Search Google Scholar
    • Export Citation
  • Anon. (2006). Peer review on trial. Nature, 441 (7094), 668.

  • Bakanic, V, McPhail, C, Simon, RJ. Mixed messages: Referees’ comments on the manuscripts they review. Sociological Quarterly 1989 30 4 639654 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bedeian, AG. Peer review and the social construction of knowledge in the management discipline. Academy of Management Learning and Education 2004 3 2 198216 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L. Scientific peer review. Annual Review of Information Science and Technology 2011 45:199245.

  • Bornmann, L, Daniel, H-D. The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition 2008 47 38 71737178 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Marx, W, Schier, H, Thor, A, Daniel, H-D. From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics 2010. Research Evaluation 19 2 105118 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Mutz, R, Daniel, H-D. Latent Markov modeling applied to grant peer review. Journal of Informetrics 2008 2 3 217228 .

  • Bornmann, L, Mutz, R, Daniel, H-D. The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models. Scientometrics 2009 81 2 407411 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Nast, I, Daniel, H-D. Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics 2008 77 3 415432 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Weymuth, C, Daniel, H-D. A content analysis of referees’ comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ?. Scientometrics 2010 83 2 493506 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cohen, J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960 20 1 3746 .

  • Daniel, H-D 1993 Guardians of science. Fairness and reliability of peer review Wiley–VCH Weinheim.

  • Fleiss, J 1981 Statistical methods for rates and proportions Wiley–VCH New York, NY.

  • Gosden, H. ‘Why not give us the full story?’: Functions of referees’ comments in peer reviews of scientific research papers. Journal of English for Academic Purposes 2003 2 2 87101 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Harnad, S. (1978). Inaugural editorial. Behavioral and Brain Sciences, 1 (1).

  • Harnad, S. (2000). The invisible hand of peer review. Exploit Interactive (5).

  • Hosmer, DW, Lemeshow, S 2000 Applied logistic regression 2 Wiley Chichester .

  • House of Commons; Science and Technology Committee on alternative models for the federal funding of science. (2011). Peer review in scientific publications. Eighth Report of Session 2010-12. Volume I: Report, together with formal minutes, oral and written evidence. London: House of Commons.

    • Search Google Scholar
    • Export Citation
  • Janeway, CAJ. JMCI: The last issue. Journal of Molecular and Cellular Immunology 1990 4:293.

  • Krampen, G, Becker, R, Wahner, U, Montada, L. On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications. Scientometrics 2007 71 2 191202 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kumar, P, Rafiq, I, Imam, B. Negotiation on the assessment of research articles with academic reviewers: Application of peer-review approach of teaching. Higher Education 2011 62 3 315332 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kupfersmid, J. Improving what is published: Amodel in search of an editor. American Psychologist 1988 43 8 635642 .

  • Landis, JR, Koch, GG. Measurement of observer agreement for categorical data. Biometrics 1977 33 1 159174 .

  • McCormack, N. Peer review and legal publishing: What law librarians need to know about open, single-blind, and double-blind reviewing. Law Library Journal 2009 101 1 5970.

    • Search Google Scholar
    • Export Citation
  • Mizzaro, S. Quality control in scholarly publishing: Anew proposal. Journal of the American Society for Information Science and Technology 2003 54 11 9891005 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Morrison, J. The case for open peer review. Medical Education 2006 40 9 830831 .

  • Pöschl, U. Interactive journal concept for improved scientific publishing and quality assurance. Learned Publishing 2004 17 2 105113 .

  • Pöschl, U. Interactive open access publishing and peer review: The effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Liber Quarterly 2010 19 3/4 293314.

    • Search Google Scholar
    • Export Citation
  • Rabe-Hesketh, S, Everitt, B 2004 A handbook of statistical analyses using Stata Chapman & Hall/CRC Boca Raton, FL.

  • Schultz, DM. Rejection rates for journals publishing atmospheric science. Bulletin of the American Meteorological Society 2010 91 2 231243 .

  • Shashok, K. (2008). Content and communication: How can peer review provide helpful feedback about the writing? BMC Medical Research Methods, 8(3).

    • Search Google Scholar
    • Export Citation
  • Shum, S. B., & Sumner, T. (2001). JIME: An interactive journal for interactive media. First Monday, 6 (2–5).

  • Smith, R. Opening up BMJ peer review. British Medical Journal 1999 318 7175 45 .

  • StataCorp. 2011 Stata statistical software: Release 12 Stata Corporation College Station, TX.

  • Sumner, T., & Shum, S. B. (1996). Open peer review & argumentation: Loosening the paper chains on journals. ARIADNE, 5.

  • S van Rooyen Godlee, F, Evans, S, Black, N, Smith, R. Effect of open peer review on quality of reviews and on reviewers’ recommendations: Arandomised trial. British Medical Journal 1999 318 7175 2327 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • A von Eye Mun, EY 2005 Analyzing rater agreement. Manifest variable methods Lawrence Erlbaum Associates Mahwah, NJ.

  • Walsh, E, Rooney, M, Appleby, L, Wilkinson, G. Open peer review: Arandomised controlled trial. British Journal of Psychiatry 2000 176:4751 .

  • Wing, JM, Chi, EH. Reviewing peer review. Communications of the ACM 2011 54 7 1011 .

  • Xin, H. (2006). Online sleuths challenge cell paper. Science, 314 (5806), 1669. doi: .

  • Impact Factor (2019): 2.867
  • Scimago Journal Rank (2019): 1.210
  • SJR Hirsch-Index (2019): 106
  • SJR Quartile Score (2019): Q1 Computer Science Apllications
  • SJR Quartile Score (2019): Q1 Library and Information Sciences
  • SJR Quartile Score (2019): Q1 Social Sciences (miscellaneous)
  • Impact Factor (2018): 2.770
  • Scimago Journal Rank (2018): 1.113
  • SJR Hirsch-Index (2018): 95
  • SJR Quartile Score (2018): Q1 Library and Information Sciences
  • SJR Quartile Score (2018): Q1 Social Sciences (miscellaneous)

Manuscript submission: http://www.editorialmanager.com/scim/