J.K. Vanclay's article is a bold attempt to review recent works on the journal impact factor (JIF) and to call for alternative certifications of journals. The too broad scope did not allow the author to fulfill all his purposes. Attempting after many others to organize the various forms of criticism, with targets often broader than the JIF, we shall try to comment on a few points. This will hopefully enable us to infer in which cases the JIF is an angel, a devil, or a scapegoat. We shall also expand on a crucial question that Vanclay could not really develop in the reduced article format: the field-normalization. After a short recall on classical cited-side or ex post normalization and of the powerful influence measures, we will devote some attention to the novel way of citing-side or ex ante normalization, not only for its own interest, but because it directly proceeds from the disassembling of the JIF clockwork.
Among classical bibliometric indicators, direct and relative impact measures for countries or other players in science are
appealing and standard. Yet, as shown in this article, they may exhibit undesirable statistical properties, or at least ones
that pose questions of interpretation in evaluation and benchmarking contexts. In this article, we address two such properties
namely sensitivity to the Yule-Simpson effect, and a problem related to convexity. The Yule-Simpson effect can occur for direct
impacts and, in a variant form, for relative impact, causing an apparent incoherence between field values and the aggregate
(all-fields) value. For relative impacts, it may result in a severe form of ‘out-range’ of aggregate values, where a player’s
relative impact shifts from ‘good’ to ‘bad’, or conversely. Out-range and lack of convexity in general are typical of relative
impact indicators. Using empirical data, we suggest that, for relative impact measures, ‘out-range’ due to lack of convexity
is not exceptional. The Yule-Simpson effect is less frequent, and especially occurs for small players with particular specialisation
Summary As citation practices strongly depend on fields, field normalisation is recognised as necessary for fair comparison of figures in bibliometrics and evaluation studies. However fields may be defined at various levels, from small research areas to broad academic disciplines, and thus normalisation values are expected to vary. The aim of this project was to test the stability of citation ratings of articles as the level of observation - hence the basis of normalisation - changes. A conventional classification of science based on ISI subject categories and their aggregates at various scales was used, namely at five levels: all science, large academic discipline, sub-discipline, speciality and journal. Among various normalisation methods, we selected a simple ranking method (quantiles), based on the citation score of the article in each particular aggregate (journal, speciality, etc.) it belonged to at each level. The study was conducted on articles in the full SCI range, for publication year 1998 with a four-year citation window. Stability is measured in three ways: overall comparison of article rankings; individual trajectory of articles; survival of the top-cited class across levels. Overall rank correlations on the observed empirical structure are benchmarked against two fictitious sets that keep the same embedded structure of articles but reassign citation scores either in a totally ordered or in a totally random distribution. These sets act respectively as a 'worst case' and 'best case' for the stability of citation ratings. The results show that: (a) the average citation rankings of articles substantially change with the level of observation (b) observation at the journal level is very particular, and the results differ greatly in all test circumstances from all the other levels of observation (c) the lack of cross-scale stability is confirmed when looking at the distribution of individual trajectories of articles across the levels; (d) when considering the top-cited fractions, a standard measure of excellence, it is found that the contents of the 'top-cited' set is completely dependent on the level of observation. The instability of impact measures should not be interpreted in terms of lack of robustness but rather as the co-existence of various perspectives each having their own form of legitimacy. A follow-up study will focus on the micro levels of observation and will be based on a structure built around bibliometric groupings rather than conventional groupings based on ISI subject categories.
In advanced methods of delineation and mapping of scientific fields, hybrid methods open a promising path to the capitalisation
of advantages of approaches based on words and citations. One way to validate the hybrid approaches is to work in cooperation
with experts of the fields under scrutiny. We report here an experiment in the field of genomics, where a corpus of documents
has been built by a hybrid citation-lexical method, and then clustered into research themes. Experts of the field were associated
in the various stages of the process: lexical queries for building the initial set of documents, the seed; citation-based
extension aiming at reducing silence; final clustering to identify noise and allow discussion on border areas. The analysis
of experts’ advices show a high level of validation of the process, which combines a high-precision and low-recall seed, obtained
by journal and lexical queries, and a citation-based extension enhancing the recall. This findings on the genomics field suggest
that hybrid methods can efficiently retrieve a corpus of relevant literature, even in complex and emerging fields.
This article presents a citation-based mapping exercise in the nanosciences field and a first sketch of citation transactions
(a measure of cognitive dependences). Nanosciences are considered to be one of the “convergent” components shaping the future
of science and technology. Recurrent questions about the structure of the field concern its diversity and multi- or inter-disciplinarity.
Observations made from various points of view confirm a strong differentiation of the field, which is scattered in multiple
galaxies with moderate level of exchanges. The multi-disciplinarity of themes and super-themes detected by mapping also appears
moderate, most of the super-themes being based on physics and chemistry in various proportions. Structural analysis of the
list of references in articles suggests that the moderate multi-disciplinarity observed at the aggregate level partly stems
from an actual inter-disciplinarity at the article level.
The increasing use of bibliometric indicators in science policy calls for a reassessment of their robustness and limits. The
perimeter of journal inclusion within ISI databases will determine variations in the classic bibliometric indicators used
for international comparison, such as world shares of publications or relative impacts. We show in this article that when
this perimeter is adjusted using a natural criterion for inclusion of journals, the journal impact, the variation of the most
common country indicators (publication and citation shares; relative impacts) with the perimeter chosen depends on two phenomena.
The first one is a bibliometric regularity rooted in the main features of competition in the open space of science, that can
be modeled by bibliometric laws, the parameters of which are “coverage-independent” indicators. But this regularity is obscured
for many countries by a second phenomenon, the presence of a sub-population of journals that does not reflect the same international
openness, the nationally-oriented journals. As a result indicators based on standard SCI or SCISearch perimeters are jeopardized
to a certain extent by this sub-population which creates large irregularities. These irregularities often lead to an over-estimation
of share and an under-estimation of the impact, for countries with national editorial tradition, while the impact of a few
mainstream countries arguably benefits from the presence of this sub-population.
Like the citation network of scientific publications, the Web is also a graph where pages are connected together by hypertext
links or “sitations”. In the new research field Webometrics, scholars have investigated equivalencies between citationist
concepts established in bibliometrics and hyperlinks networks. This paper focuses on the possible analogy between co-citation
and co-sitation to structure Web universes. It reports an experiment in the field of bibliometrics and scientific indicators.
Several technical aspects that must be dealt with are reviewed. Co-sitation seems a promising way to delineate topics on the
Web. However, the analogy with traditional co-citation is deeply misleading: many precautions must be taken in the interpretation
of the results.
This article aims at a characterization of the cooperation behavior among five large scientific countries (France, Germany, Japan, United Kingdom and United States of America) from 1986 to 1996. It looks at the cooperation profiles of these countries using classical measures such as the Probabilistic Affinity. The results show the major influence which historical, cultural and linguistic proximities may have on patterns of cooperation, with few changes over the period of time studied.A lack of specific affinities among the three largest European countries is revealed, and this contrasts with the strong linkage demonstrated between United States and Japan. The ensuing discussion raises some questions as to the process of Europeanization in science. The intensity of bilateral cooperation linkages is then studied with regard to field specialization by country, and this analysis yields no general patterns at the scale studied. Specific bilateral behaviors are also analyzed.
This article depicts some features of the geography of science and technology outputs in the EU, with a particular attention
to regional “co-location” of these two pillars of the “knowledge-based society”. Economists have, for a decade, paid great
attention to local “spillovers” stating that industrial firms often draw advantages from the presence of nearby academic centres.
The presence in the same areas of strong academic and technological resources is both a condition and a result of science-technology
interactions. Concentrating on publications and patents as proxies of the science and technology level in regions, we built
a typology of regions according to their commitment to the two knowledge-base activities and then analysed the co-locations
of science and technology from several points of view. A fine-grain lattice, mainly based on standard Nuts3 level, was used.
Co-location, at the EU level, is not a general rule. A strong potential for spillover/ interaction does exist in the top-class
regions which concentrate a high proportion of European S and T output. But for regions with a small/medium level of S&T activity,
a divergence of orientations appears between a science-oriented family and a technology-oriented family, indicating an imbalance
between local S and T resources. If we look at the S-oriented regions, whilst controlling for underlying factors, such as
population and regional economic product, a significant geographic linkage between T and S appears. This suggests a trajectory
of science-based technological development. A careful examination of S&T thematic alignments and specialisation is necessary
to develop the hypothesis that fostering academic resources could increase the technological power along a growth path.