View More View Less
  • 1 Statistical Cybermetrics Research Group, School of Technology, University of Wolverhampton, Wulfruna Street, Wolverhampton WV1 1LY, UK
Restricted access

Abstract

In theory, the web has the potential to provide information about the wider impact of academic research, beyond traditional scholarly impact. This is because the web can reflect non-scholarly uses of research, such as in online government documents, press coverage or public discussions. Nevertheless, there are practical problems with creating metrics for journals based on web data: principally that most such metrics should be easy for journal editors or publishers to manipulate. Nevertheless, two alternatives seem to have both promise and value: citations derived from digitised books and download counts for journals within specific delivery platforms.

  • Aguillo, IF 2009 Measuring the institution's footprint in the web. Library Hi Tech 27 4 540556 .

  • Bar-Ilan, J. (2011). Articles tagged by ‘bibliometrics’ on Mendeley and CiteULike. Paper presented at the Metrics 2011 Symposium on Informetric and Scientometric Research.

    • Search Google Scholar
    • Export Citation
  • Barjak, F, Thelwall, M 2008 A statistical analysis of the web presences of European life sciences research teams. Journal of the American Society for Information Science and Technology 59 4 628643 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Becher, T, Trowler, P 2001 Academic tribes and territories 2 Open University press Milton Keynes.

  • Bogers, T., Bosch, A. V. D. (2008). Recommending scientific articles using citeulike. In: Proceedings of the 2008 ACM conference on Recommender Systems (RecSys ‘08), New York, pp. 287290.

    • Search Google Scholar
    • Export Citation
  • Bollen, J H Van de Sompel 2008 Usage impact factor: the effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and Technology 59 1 136149 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, H-D 2008 What do citation counts measure? A review of studies on citing behaviour. Journal of Documentation 64 1 4580 .

  • Brin, S, Page, L 1998 The anatomy of a large scale hyper textual web search engine. Computer Networks and ISDN Systems 30 1–7 107117 .

  • Brody, T, Harnad, S, Carr, L 2006 Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Information Science and Technology 57 8 10601072 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brooks, TA 1986 Evidence of complex citer motivations. Journal of the American Society for Information Science 37:3436.

  • Chakrabarti, S., Dom, B., Raghavan, P., Rajagonpalan, S., Gibson, D., Kleinberg, J. M. (1998). Automatic resource compilation by analyzing hyperlink structure and associated text. Paper presented at the 7th International World Wide Web Conference, April 1998.

    • Search Google Scholar
    • Export Citation
  • Chakrabarti, S., Joshi, M. M., Punera, K., Pennock, D. M. (2002). The structure of broad topics on the web. http://www2002.org/CDROM/refereed/338.

    • Search Google Scholar
    • Export Citation
  • Chakrabarti, S., van den Berg, M., Dom, B. (1999). Focused crawling: a new approach to topic-specific web resource discovery. Paper presented at the 8th International World Wide Web Conference, May 1999.

    • Search Google Scholar
    • Export Citation
  • Cole, S, Cole, JR 1967 Scientific output and recognition: a study in the operation of the reward system in science. American Sociological Review 32 3 377390 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Coleman, A 2007 Assessing the value of a journal beyond the impact factor. Journal of the American Society for Information Science and Technology 58 8 11481161 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • COUNTER. (2011a). Counting online usage of networked electronic resources. http://www.projectcounter.org/. Accessed 14 December 2011.

  • COUNTER. (2011b). Register of vendors providing usage reports compliant with release three of the code of practice for journals and databases. http://www.projectcounter.org/r2013/R2013CV_Dec2011.pdf. Accessed 14 December 2011.

    • Search Google Scholar
    • Export Citation
  • Cronin, B 1984 The citation process: the role and significance of citations in scientific communication Taylor Graham London.

  • Davis, P. (2011). Gaming the impact factor puts journal in time-out. http://scholarlykitchen.sspnet.org/2011/2010/2017/gaming-the-impact-factor-puts-journal-in-time-out/. Accessed 14 December 2011.

    • Search Google Scholar
    • Export Citation
  • CM De Costa 2002 “The contagiousness of childbed fever”: a short history of puerperal sepsis and its treatment. The Medical Journal of Australia 177 11/12 668671.

    • Search Google Scholar
    • Export Citation
  • Eysenbach, G 2011 Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research 13 4 e123 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Garfield, E 1972 Citation analysis as a tool in journal evaluation. Science 178 4060 471479 .

  • Garfield, E. (2005). The agony and the ecstasy: the history and the meaning of the journal impact factor. Fifth International Congress on Peer Review in Biomedical Publication, Chicago. USA. http://garfield.library.upenn.edu/papers/jifchicago2005.pdf. Accessed 27 September 2007.

    • Search Google Scholar
    • Export Citation
  • Gyongyi, Z., Garcia-Molina, H., Pedersen, J. (2004). Combating web spam with TrustRank. Proceedings of the thirtieth international conference on very large data bases, 30, 576-587.

    • Search Google Scholar
    • Export Citation
  • Han, S., Ahn, Y.-Y., Moon, S., Jeong, H. (2006). Collaborative blog spam filtering using adaptive percolation search, WWW2006 Workshop. http://www.blogpulse.com/www2006-workshop/papers/collaborative-blogspam-filtering.pdf. Accessed 5 May 2006.

    • Search Google Scholar
    • Export Citation
  • Haustein, S, Siebenlist, T 2011 Applying social bookmarking data to evaluate journal usage. Journal of Informetrics 5 3 446457.

  • Henning, V., Reichelt, J. (2008). Mendeley—A last.fm for research? In: IEEE Fourth International Conference on eScience (eScience ‘08), Los Alamitos, pp. 327328.

    • Search Google Scholar
    • Export Citation
  • Huizenga, JR 1992 Cold fusion: the scientific fiasco of the century University of Rochester press Rochester.

  • Ingwersen, P 1998 The calculation of web impact factors. Journal of Documentation 54 2 236243 .

  • Jacsó, P 2005 Google Scholar: the pros and the cons. Online Information Review 29 2 208214 .

  • Kleinberg, JM 1999 Authoritative sources in a hyperlinked environment. Journal of the ACM 46 5 604632 .

  • Kousha, K, Horri, A 2004 The relationship between scholarly publishing and the counts of academic in links to Iranian university web sites: exploring academic link creation motivations. Journal of Information Management and Scientometrics 1 2 1322.

    • Search Google Scholar
    • Export Citation
  • Kousha, K, Thelwall, M 2007 Google Scholar citations and Google Web/URL citations: a multi-discipline exploratory analysis. Journal of the American Society for Information Science and Technology 58 7 10551065 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kousha, K, Thelwall, M 2008 Assessing the impact of disciplinary research on teaching: an automatic analysis of online syllabuses. Journal of the American Society for Information Science and Technology 59 13 20602069 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kousha, K, Thelwall, M 2009 Google Book search: citation analysis for social science and the humanities. Journal of the American Society for Information Science and Technology 60 8 15371549 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kousha, K, Thelwall, M, Rezaie, S 2010 Using the web for research evaluation: the integrated online impact indicator. Journal of Informetrics 4 1 124135 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Larson, R. R. (1996). Bibliometrics of the world wide web: an exploratory analysis of the intellectual structure of cyberspace. ASIS 59th annual meeting, Baltimore.

    • Search Google Scholar
    • Export Citation
  • Li, X., Thelwall, M., Giustini, D. (2011). Validating online reference managers for scholarly impact measurement. Scientometrics.

  • Li, X, Thelwall, M, Wilkinson, D, Musgrove, PB 2005 National and international university departmental web site interlinking, part 2: link patterns. Scientometrics 64 2 187208 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Maxmen, A 2010 Science networking gets serious. Cell 141 3 387389 .

  • Mayr, P, Walter, AK 2007 An exploratory study of Google Scholar. Online Information Review 31 6 814830 .

  • Meho, LI, Yang, K 2007 Impact of data sources on citation counts and rankings of LIS faculty: web of science vs. scopus and Google Scholar. Journal of the American Society for Information Science and Technology 58 13 21052125 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Merton, RK 1973 The sociology of science. Theoretical and empirical investigations University of Chicago press Chicago.

  • Meyer, M 2003 Academic patents as an indicator of useful research? A new approach to measure academic inventiveness. Research Evaluation 12 1 1727 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moed, HF 2005 Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society for Information Science and Technology 56 10 10881097 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moed, HF 2010 Measuring contextual citation impact of scientific journals. Journal of Informetrics 4 3 265277 .

  • Newton, I. (1676; 1992 republication). Letter from Isaac Newton to Robert Hooke, 5 February 1676. In: J.-P. Maury (ed.), Newton: Understanding the Cosmos. New York: New Horizons.

    • Search Google Scholar
    • Export Citation
  • NISO. (2011). ANSI/NISO Z39.93-2007 the standardized usage statistics harvesting initiative (SUSHI) protocol. http://www.niso.org/apps/group_public/project/details.php?project_id=2021. Accessed 14 December 2011.

    • Search Google Scholar
    • Export Citation
  • Oppenheim, C 2000 Do patent citations count? B Cronin HB Atkins eds. The web of knowledge: a festschrift in honor of Eugene Garfield Information today Inc. ASIS monograph series Metford 405432.

    • Search Google Scholar
    • Export Citation
  • Priem, J., Costello, K. L. (2010). How and why scholars cite on Twitter. In: Proceedings of the American Society for Information Science and Technology (ASIST 2010), vol 47. pp. 14).

    • Search Google Scholar
    • Export Citation
  • Priem, J., Hemminger, B. M. (2010). Scientometrics 2.0: toward new metrics of scholarly impact on the social web. http://www.uic.edu/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/2570. First Monday, 15(7), Accessed 7 December 2011.

    • Search Google Scholar
    • Export Citation
  • Priem, J., Taraborelli, D., Groth, P., Neylon, C. (2011). Altmetrics: a manifesto. http://altmetrics.org/manifesto/. Accessed 14 December 2011.

    • Search Google Scholar
    • Export Citation
  • Provos, N., Mavrommatis, P., Rajab, M. A., Monrose, F. (2008). All your iFRAMEs point to us. http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//archive/provos-2008a.pdf. Accessed 13 December 2011.

    • Search Google Scholar
    • Export Citation
  • Qiu, JP, Chen, JQ, Wang, Z 2004 An analysis of back link counts and web impact factors for Chinese university websites. Scientometrics 60 3 463473 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • RCUK. (2011). Types of impact. http://www.rcuk.ac.uk/documents/impacts/TypologyofResearchImpacts.pdf. Accessed 12 December 2011.

  • Rodríguez, I, Gairín, JM 1997 Valorando el impacto de la información en Internet: AltaVista, el “Citation Index” de la Red (Evaluating the impact of internet information: Altavista, the “Citation Index” of the web). Revista Española de Documentación Científica 20 2 175181 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rousseau, R. (1997). Sitations: an exploratory study. http://www.cindoc.csic.es/cybermetrics/articles/v2001i2001p2001.html. Cybermetrics, 1(1), Accessed 25 July 2006.

    • Search Google Scholar
    • Export Citation
  • Shepherd, P. (2011). Journal usage factor: results, recommendations and next steps. http://www.uksg.org/sites/uksg.org/files/JournalUsageFactorReport080711.pdf. Accessed 14 December 2011.

    • Search Google Scholar
    • Export Citation
  • Smith, AG 1999 A tale of two web spaces; comparing sites using web impact factors. Journal of Documentation 55 5 577592.

  • Thelwall, M 2001 Extracting macroscopic information from web links. Journal of American Society for Information Science and Technology 52 13 11571168 .

  • Thelwall, M 2002 Conceptualizing documentation on the web: an evaluation of different heuristic-based models for counting links between university web sites. Journal of American Society for Information Science and Technology 53 12 9951005 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Thelwall, M, Harries, G 2004 Do the web sites of higher rated scholars have significantly more online impact?. Journal of the American Society for Information Science and Technology 55 2 149159 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Thelwall, M, Kousha, K 2008 Online presentations as a source of scientific impact?: an analysis of powerpoint files citing academic journals. Journal of the American Society for Information Science and Technology 59 5 805815 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Thomas, O, Willett, P 2000 Webometric analysis of departments of librarianship and information science. Journal of Information Science 26 6 421428 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • AFJ van Raan 2001 Bibliometrics and internet: some observations and expectations. Scientometrics 50 1 5963 .

  • Vanclay, J. K. (2012). Impact factor: outdated artefact or stepping-stone to journal certification? Scientometrics.

  • Vaughan, L, Hysen, K 2002 Relationship between links to journal web sites and impact factors. ASLIB Proceedings 54 6 356361 .

  • Vaughan, L, Shaw, D 2003 Bibliographic and web citations: what is the difference?. Journal of the American Society for Information Science and Technology 54 14 13131322 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vaughan, L, Shaw, D 2005 Web citation data for impact assessment: a comparison of four science disciplines. Journal of the American Society for Information Science and Technology 56 10 10751087 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vaughan, L, Shaw, D 2008 A new look at evidence of scholarly citation in citation indexes and from web sources. Scientometrics 74 2 317330 .

  • Vaughan, L, Thelwall, M 2003 Scholarly use of the web: what are the key inducers of links to journal web sites?. Journal of American Society for Information Science and Technology 54 1 2938 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vaughan, L, Thelwall, M 2005 A modeling approach to uncover hyperlink patterns: the case of Canadian universities. Information Processing and Management 41 2 347359 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weller, K., Dröge, E., Puschmann, C. (2011). Citation analysis in Twitter: approaches for defining and measuring information flows within tweets during scientific conferences. 1st Workshop on Making Sense of Microposts. http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-2718/paper_2004.pdf. Accessed 16 June 2011.

    • Search Google Scholar
    • Export Citation
  • Weller, K., Puschmann, C. (2011). Twitter for scientific communication: how can citations/references be identified and measured? In: Proceedings of the poster session at the Web Science Conference 2011, Koblenz, Germany. http://journal.webscience.org/2500/). Accessed 21 October 2011.

    • Search Google Scholar
    • Export Citation
  • Whitley, R 2000 The intellectual and social organization of the sciences 2 Oxford University press Oxford.