The article deals with the statistical problem of the difference between the mean citation frequencies of two sets of papers required to be significantly different. An analysis of citation data indicated that, as a first-order approximation, (1) The relative spread due to a short observation interval is independent of the long-term citation frequency and (2) the relative spread in long-term citation frequencies of different papers from the same author is independent of the mean citation score for the papers by that author. As a rule-of-thumb, these two sources of variance can be characterized by standard deviations of a ratio (factor) of 2 and 3, respectively. By applying these results to citation data published in the literature, it is shown that sometimes statistically unjustified conclusions have been drawn in the past.
After presenting arguments that the number of highly cited papers (HCPs, 25 or more citations) has some advantages as an indicator of an author's scientific impact, the paper discusses citation data of 338 university professors in departments of medicine in the Netherlands. An analysis of the distribution of HCPs over the years provides support for the following conclusions: (1) prolific researchers with a large number of HCPs usually manifest themselves already in their Ph.D. work, apparently almost independent of the scientific setting; (2) it cannot be taken for granted that a successful Ph.D. student with some HCPs connected with his/her doctoral thesis will become a prolific successful researcher; (3) it is unlikely that an unsuccessful Ph.D. student without HCPs connected with his/her doctoral thesis will turn out to be a prolific successful researcher; and (4) for researchers, just as for artists, sportsmen, etc., talent is the most decisive factor in being successful.
In the first part of the paper the citations in 1986 and 1987 of 3938 papers published in 1985 by 324 research groups in the faculties of science and of medicine of eight universities in the Netherlands are analyzed. Because of the large statistical spread of (1) the number of short-term citations of papers cited equally frequently over a long period, and (2) the number of citations over a long period of papers by the same author, short-term citation scores appear to be an unreliable indicator of a research group's contribution to science. In the second part of the paper an alternative approach is presented, based on a subdivision of the 3938 papers in papers authored by professors with 0–2, 3–8, or 9 highly cited papers (HCPs, 25 citations) to their name. Very large citation score differences were found for the three categories. For example: for papers first-authored by a professor, the average number of citations per person in 1986 and 1987 for 1985 papers was for 161 professors with 9 HCPs a factor 14 larger than for 575 professors with only 0–2 HCPs; for papers co-authored by professors, this factor was 6.6. These findings justify the conclusion that the number of HCPs scored by the professors (and other senior scientists) during their entire career is a much more reliable predictor of the performance of a research group than the number of short-term citations of the articles published by the group within a short period. A research group's contribution to science is primarily determined by the individual scientifictalents of its members.