View More View Less
  • 1 Max Planck Society, Administrative Headquarters, Hofgartenstr. 8, 80539 Munich, Germany
  • | 2 Center for Psychotherapy Research, University Hospital Heidelberg, Bergheimer Str. 54, 69115 Heidelberg, Germany markus.wolf@med.uni-heidelberg.de
  • | 3 Evaluation Office, University of Zurich, Zurich, Switzerland
  • | 4 ETH Zurich, Mühlegasse 21, MUG, 8001 Zurich, Switzerland
Restricted access

Abstract

Whereas in traditional, closed peer review (CPR) a few, selected scientists (peers) are included in the process of manuscript review, public peer review (PPR) includes, in addition to invited reviewers, a wider circle of scientists who are interested in a manuscript and wish to write a comment on it. In this study, using the data of two comprehensive evaluation studies on the CPR process at Angewandte Chemie—International Edition and the PPR process at Atmospheric Chemistry and Physics, we examined the language characteristics in comments that were written by invited reviewers in CPR and by invited reviewers and interested members of the scientific community in PPR. We used Linguistic Inquiry and Word Count (LIWC), a text analysis software program that counts words in meaningful categories (e.g., positive or negative emotions) using a standardized dictionary. We examined 599 comments from the reviews of 229 manuscripts. The results show that the comments in PPR are much longer than the comments in CPR. This is an indication that PPR reviewing has more of an improvement function and CPR reviewing has more of a selection function. The results also show that CPR is not, as might be expected, more susceptible to the expression of negative emotions than PPR is. On the contrary, positive emotion words are used statistically significantly more frequently in CPR than in PPR.

  • Anon. (2006). Peer review on trial. Nature, 441 (7094), 668.

  • Bakanic, V, McPhail, C, Simon, RJ 1989 Mixed messages: Referees’ comments on the manuscripts they review. Sociological Quarterly 30 4 639654 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bingham, CM, Higgins, G, Coleman, R MB Van Der Weyden 1998 The Medical Journal of Australia Internet peer-review study. Lancet 352 9126 441445 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2008 The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition 47 38 71737178 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2008 Selecting manuscripts for a high impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology 59 11 18411852 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2009 Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition. Journal of Informetrics 3 4 348352 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2009 Reviewer and editor biases in journal peer review: An investigation of manuscript refereeing at Angewandte Chemie International Edition. Research Evaluation 18 4 262272 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2010 How long is the peer review process for journal manuscripts? A case study on Angewandte Chemie International Edition. Chimia 64 1–2 7277 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2010 Reliability of reviewers’ ratings at an interactive open access journal using public peer review: A case study on Atmospheric Chemistry and Physics. Learned Publishing 23 2 124131 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L., & Daniel, H.-D. (2010c). The manuscript reviewing process—empirical research on review requests, review sequences and decision rules in peer review. Library & Information Science Research, 32 (1), 512.

    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2010 The validity of staff editors’ initial evaluations of manuscripts: A case study of Angewandte Chemie International Edition. Scientometrics 85 3 681687 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Marx, W, Schier, H, Rahm, E, Thor, A, Daniel, HD 2009 Convergent validity of bibliometric Google Scholar data in the field of chemistry. Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics 3 1 2735 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Marx, W, Schier, H, Thor, A, Daniel, H-D 2010 From black box to white box at open access journals: Predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics. Research Evaluation 19 2 105118 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Nast, I, Daniel, H-D 2008 Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics 77 3 415432 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Neuhaus, C, Daniel, H-D 2011 The effect of a two-stage publication process on the Journal Impact Factor: A case study on the interactive open access journal Atmospheric Chemistry and Physics. Scientometrics 86:9397 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Schier, H, Marx, W, Daniel, H-D 2011 Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes. Journal of the American Society for Information Science and Technology 62 1 6171 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Weymuth, C, Daniel, H-D 2010 A content analysis of referees’ comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ?. Scientometrics 83 2 493506 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chung, CK, Pennebaker, JW 2007 The psychological function of function words K Fiedler eds. Social communication Psychology Press New York 343359.

    • Search Google Scholar
    • Export Citation
  • Fast, LA, Funder, DC 2008 Personality as manifest in word use: Correlations with self-report, acquaintance report, and behavior. Journal of Personality and Social Psychology 94 2 334346 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fraser, V. J., & Martin, J. G. (2009). Marketing data: Has the rise of impact factor led to the fall of objective language in the scientific article? Respiratory Research, 10. doi: .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Harnad, S. (2000). The invisible hand of peer review. Exploit Interactive (5). http://www.exploit-lib.org/issue5/peer-review/.

  • Hartley, J, Betts, L 2009 Common weaknesses in traditional abstracts in the social sciences. Journal of the American Society for Information Science and Technology 60 10 20102018 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hartley, J, Pennebaker, JW, Fox, C 2003 Abstracts, introductions and discussions: How far do they differ in style?. Scientometrics 57 3 389398 .

  • Kahn, JH, Tobin, RM, Massey, AE, Anderson, JA 2007 Measuring emotional expression with the Linguistic Inquiry and Word Count. American Journal of Psychology 120 2 263286.

    • Search Google Scholar
    • Export Citation
  • Koonin, E., & Lipman, D. (2006). Systems: Reviving a culture of scientific debate. Can ‘open peer review’ work for biologists? Biology Direct is hopeful. Retrieved 21 June 2006, from http://www.nature.com/nature/peerreview/debate/nature05005.html.

    • Search Google Scholar
    • Export Citation
  • Mungra, P, Webber, P 2010 Peer review process in medical research publications: Language and content comments. English for Specific Purposes 29 1 4353 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Oberlander, J, Gill, AJ 2006 Language with character: A stratified corpus comparison of individual differences in e-mail communication. Discourse Processes 42 3 239270 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pennebaker, JW, Chung, CK, Ireland, M, Gonzales, A, Booth, RJ 2007 The development and psychometric properties of LIWC 2007 University of Texas/University of Auckland Austin, TX/Auckland.

    • Search Google Scholar
    • Export Citation
  • Pennebaker, JW, King, LA 1999 Linguistic styles: Language use as an individual difference. Journal of Personality and Social Psychology 77 6 12961312 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pennebaker, JW, Mehl, MR, Niederhoffer, KG 2003 Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology 54:547577 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Popping, R 2000 Computer-assisted text analysis Sage London.

  • Pöschl, U 2004 Interactive journal concept for improved scientific publishing and quality assurance. Learned Publishing 17 2 105113 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pöschl, U 2010 Interactive open access publishing and peer review: The effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Liber Quarterly 19 3/4 293314.

    • Search Google Scholar
    • Export Citation
  • StataCorp. 2011 Stata statistical software: Release 12 Stata Corporation College Station, TX.

  • Tausczik, YR, Pennebaker, JW 2010 The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology 29 1 2454 .

    • Crossref
    • Search Google Scholar
    • Export Citation

Manuscript submission: http://www.editorialmanager.com/scim/

  • Impact Factor (2019): 2.867
  • Scimago Journal Rank (2019): 1.210
  • SJR Hirsch-Index (2019): 106
  • SJR Quartile Score (2019): Q1 Computer Science Apllications
  • SJR Quartile Score (2019): Q1 Library and Information Sciences
  • SJR Quartile Score (2019): Q1 Social Sciences (miscellaneous)
  • Impact Factor (2018): 2.770
  • Scimago Journal Rank (2018): 1.113
  • SJR Hirsch-Index (2018): 95
  • SJR Quartile Score (2018): Q1 Library and Information Sciences
  • SJR Quartile Score (2018): Q1 Social Sciences (miscellaneous)

For subscription options, please visit the website of Springer

Scientometrics
Language English
Size B5
Year of
Foundation
1978
Volumes
per Year
4
Issues
per Year
12
Founder Akadémiai Kiadó
Founder's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
Publisher Akadémiai Kiadó
Springer Nature Switzerland AG
Publisher's
Address
H-1117 Budapest, Hungary 1516 Budapest, PO Box 245.
CH-6330 Cham, Switzerland Gewerbestrasse 11.
Responsible
Publisher
Chief Executive Officer, Akadémiai Kiadó
ISSN 0138-9130 (Print)
ISSN 1588-2861 (Online)