View More View Less
  • 1 Health Economics Research Group, Brunel University, Uxbridge, Middlesex UB8 3PH, UK
Restricted access

Abstract

There is an increasing need both to understand the translation of biomedical research into improved healthcare and to assess the range of wider impacts from health research such as improved health policies, health practices and healthcare. Conducting such assessments is complex and new methods are being sought. Our new approach involves several steps. First, we developed a qualitative citation analysis technique to apply to biomedical research in order to assess the contribution that individual papers made to further research. Second, using this method, we then proposed to trace the citations to the original research through a series of generations of citing papers. Third, we aimed eventually to assess the wider impacts of the various generations. This article describes our comprehensive literature search to inform the new technique. We searched various databases, specific bibliometrics journals and the bibliographies of key papers. After excluding irrelevant papers we reviewed those remaining for either general or specific details that could inform development of our new technique. Various characteristics of citations were identified that had been found to predict their importance to the citing paper including the citation's location; number of citation occasions and whether the author(s) of the cited paper were named within the citing paper. We combined these objective characteristics with subjective approaches also identified from the literature search to develop a citation categorisation technique that would allow us to achieve the first of the steps above, i.e., being able routinely to assess the contribution that individual papers make to further research.

  • Allen, L., Jones, C., Dolby, K., Lynn, D., & Walport, M. (2009). Looking for landmarks: The role of expert review and bibliometric analysis in evaluating scientific publication outputs. PLoS ONE, 4 (6), e5910.

    • Search Google Scholar
    • Export Citation
  • Bornmann, L, Daniel, HD. What do citation counts measure? A review of studies on citing behavior. Journal of Documentation 2008 64 1 4580 .

  • Buxton, M, Hanney, S. How can payback from health services research be assessed?. Journal of Health Services Research & Policy 1996 1 1 3543.

    • Search Google Scholar
    • Export Citation
  • Cano, V. Citation behavior: Classification, utility, and location. Journal of the American Society for Information Science 1989 40 4 284290 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Case, DO, Higgins, GM. How can we investigate citation behavior? A study of reasons for citing literature in communication. Journal of the American Society for Information Science 2000 51 7 635645 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cave, M, Hanney, S, Kogan, M, Trevett, G 1988 The use of performance indicators in higher education: A critical analysis of developing practice Jessica Kingsley Publishers Ltd. London.

    • Search Google Scholar
    • Export Citation
  • Cooksey, D 2006 A review of UK health research funding HM Treasury London.

  • Garfield, E. When to cite. Library Quarterly 1996 66 4 449 .

  • Gilbert, G. Referencing as persuasion. Social Studies of Science 1977 7:113122 .

  • Grant, J, Cotterell, R, Cluzeau, F, Fawcett, G. Evaluating “payback” on biomedical research from papers cited in clinical guidelines: Applied bibliometric study. BMJ 2000 320:11071111 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hanney, S, Buxton, M, Green, CR, Coulson, D, Raftery, J. An assessment of the impact of the health technology assessment programme. Health Technology Assessment 2007 11 53 1180.

    • Search Google Scholar
    • Export Citation
  • Hanney, S, Frame, I, Grant, J, Buxton, M, Young, T, Lewison, G. Using categorisations of citations when assessing the outcomes from health research. Scientometrics 2005 65 3 357379 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hanney, SR, Home, PD, Frame, I, Grant, J, Green, P, Buxton, MJ. Identifying the impact of diabetes research. Diabetic Medicine 2006 23 2 176184 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Harwood, N. An interview-based study of the functions of citations in academic writing across two disciplines. Journal of Pragmatics 2009 41 3 497518 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Haslam, N, Ban, L, Kaufmann, L, Loughnan, S, Peters, K, Whelan, J et al. 2008 What makes an article influential? Predicting impact in social and personality psychology. Scientometrics 76 1 169185 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hu, X, Rousseau, R, Chen, J. On the definition of forward and backward citation generations. Journal of Informetrics 2011 5 1 2736 .

  • Jones, T, Hanney, S, Buxton, M, Burns, T. What British psychiatrists read. Questionnaire survey of journal usage among clinicians. British Journal of Psychiatry 2004 2004 185 251257 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kostoff, RN. The use and misuse of citation analysis in research evaluation - Comments on theories of citation?. Scientometrics 1998 43 1 2743 .

  • Lewison, G 2004 Citations to papers from other documents H Moed W Glanzel U Schmoch eds. Handbook of Quantitative Science and Technology Research. The Use of Publication Statistics in Studies of S&T Systems Kluwer Academic Publishers Dordrecht 457472.

    • Search Google Scholar
    • Export Citation
  • McCain, KW, Turner, K. Citation context analysis and aging patterns of journal articles in molecular-genetics. Scientometrics 1989 17 1–2 127163 .

  • Moed, H 2005 Citation analysis in research evaluation Springer Dordrecht.

  • Moravcsik, MJ, Murugesan, P. Some results on function and quality of citations. Social Studies of Science 1975 5 1 8692 .

  • Moriarty, C., Jensen, J., & Stryker, J. (2009). Frequently cited sources of cancer news coverage: A content analysis examining the relationship between cancer news content and source citation. Cancer Causes Control. doi: .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Oppenheim, C, Renn, SP. Highly cited old papers and the reasons why they continue to be cited. Journal of the American Society for Information Science 1978 29 5 225231 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Paul, D. In citing chaos: A study of the rhetorical use of citations. Journal of Business and Technical Communication 2000 14 2 185222 .

  • Peritz, BC. A classification of citation roles for the social sciences and related fields. Scientometrics 1983 5 5 303312 .

  • Prabha, CG. Some aspects of citation behavior: A pilot-study in business administration. Journal of the American Society for Information Science 1983 34 3 202206 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Research evaluation and policy project (REPP). (2005). Quantitative Indicators for Research Assessment: A Literature Review (No. Discussion paper 05/1). Canberra: REPP, Research School of Social Sciences, The Australian National University.

    • Search Google Scholar
    • Export Citation
  • Safer, MA, Tang, R. The psychology of referencing in psychology journal articles. Perspectives on Psychological Science 2009 4 1 5153 .

  • Shadish, WR, Tolliver, D, Gray, M, Sengupta, SK. Author judgments about works they cite: 3 Studies from psychology journals. Social Studies of Science 1995 25 3 477498 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Small, HG. Cited documents as concept symbols. Social Studies of Science 1978 8:327 .

  • Snyder, H. W., & Bonzi, S. (1989). An enquiry into the behavior of author self citation, Proceedings of the 52nd Annual Meeting of the American Society for Information Science (pp. 147151).

    • Search Google Scholar
    • Export Citation
  • Sombatsompop, N, Kositchaiyong, A, Markpin, T, Inrit, S. Scientific evaluations of citation quality of international research articles in the SCI database: Thailand case study. Scientometrics 2006 66 3 521535 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tang, R, Safer, MA. Author-rated importance of cited references in biology and psychology publications. Journal of Documentation 2008 64 2 246272 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • UK Evaluation Forum. (2006). Medical research: Assessing the benefits to society. London: MRC and Wellcome Trust.

  • White, HD. Citation analysis and discourse analysis revisited. Applied Linguistics 2004 25 1 89116 .

  • White, MD, Wang, PL. A qualitative study of citing behavior: Contributions, criteria, and metalevel documentation concerns. Library Quarterly 1997 67 2 122154 .

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wooding, S, Hanney, S, Buxton, M, Grant, J. Payback arising from research funding: Evaluation of the arthritis research campaign. Rheumatology (Oxford) 2005 44 9 11451156 .

    • Crossref
    • Search Google Scholar
    • Export Citation