Scientific impact is a difficult thing to measure. Nevertheless, a variety of indexes and metrics have been created that attempt to quantify it.

Many of these measures are based on citations: the number of times a scientific paper is referenced by another scientific paper. Unfortunately, it is not entirely clear what these citation-based metrics actually reflect, nor is it clear whether they should be used (or misused) as a measure of impact or excellence [1].

There are many citation metrics and it is not clear which metric should be used. Researchers include such metrics in their curriculum vitae, but they are often inaccurate and not professionally calculated. Moreover, metrics do not systematically account for self-citation (incidental or strategic) or allow for comparisons between fields with different citation densities.

Ioannidis to the rescue: A composite citation metric

John Ioannidis is an influential researcher in the field of of open science and research reproducibility. Recently, he and his colleagues published a paper in PLOS Biology that tries to overcome many of the problems with individual citation metrics [2]. Specifically, they provide a comprehensive database of over 100,000 of the most-cited scientists across all scientific fields, as well normative values for over 6 million researchers who have authored at least 5 scientific papers.

Researchers are ranked based on a composite indicator that considers six citation metrics [3]. See below for a description of these metrics

One version of the database reflects data over 22 years (1996 to 2018) and provides a measure of long-term performance. Another version of the database reflects citations from 2017 and provides a measure of performance in that single recent year. This latter database may be better suited for junior scientists who have not had as much time to accrue citations.

Calculating your own composite citation score

In their paper, Ioannidis et al. [2] provide a simple formula that we can use to calculate our own composite citation score. Why would we do this? Well, if you are like me and you are not in the top 100,000 scientists that made it into the downloadable database, you can calculate your score and determine how well you rank in your field.

Interpreting the composite citation metric: an example

Rather than calculating my own (humiliating) composite citation score, I thought it would be interesting to compare a few of the scientists from the research institute where I work who actually made it to the prestigious top 100,000 list.

Name	rank	tc	h	h2	ncp	ncsf	ncsfl	composite metric
Gandevia, SC	1,397	20,417	76	44.7534	1,979	5,023	14,174	4.5984
Lord, S	4,562	19,049	75	36.8646	194	5,888	11,064	4.3458
Anstey, K	18,964	8,491	48	25.7105	110	3,345	5,508	3.9695
Herbert, R	25,111	10,195	51	25.2397	153	1,335	2,830	3.8863
Schofield, P	33,414	14,414	60	21.0989	45	722	4,684	3.7950
Taylor, J	44,266	5,418	43	18.994	68	1,417	3,420	3.7013
Weickert, C	99,276	7, 705	50	18.829	0	1,347	3,730	3.3969

• tc: total citations
• h: Hirsch h-index
• h2: coauthorship-adjusted Schreiber hm-index
• ncp: number of citation to papers as single author
• ncsf: number of citation to papers as single or first author
• ncsfl: number of citations to papers as single, first or last author

The database also includes an estimate of the rate of self-citation. Among the top 100,000 scientists, the median percentage of self-citations was 12.7% [IQR: 8.6% to 17.7%]. If we again look at the scientists from my own research institute, most are within this rage.

Name	self-citation
Gandevia, S	13.58%
Lord, S	14.27%
Anstey, K	16.00%
Herbert, R	14.55%
Schofield, P	18.33%
Taylor, J	16.72%
Weickert, C	14.44%

Ioannidis et al. [2] make the point that is that care should be taken when comparing scientists across very different fields (e.g. historical studies, physics & astronomy, clinical medicine). However, given that my example researchers all work in a medical research institute, we see that their identified scientific fields are sensible and, for the most part, comparable. As such, their composite citation score may serve as an indicator of their scientific impact and excellence.

Name	field 1	field 2	field 3
Gandevia, S	Physiology	Neurology & Neurosurgery	Biomedical Research
Lord, S	Geriatrics	Gerontology	Clinical Medicine
Anstey, K	Geriatrics	Experimental Psychology	Clinical Medicine
Herbert R	Rehabilitation	Orthopedics	Clinical Medicine
Schofield P	Neurology & Neurosurgery	Psychiatry	Clinical Medicine
Taylor, J	Physiology	Neurology & Neurosurgery	Biomedical Research
Weickert, C	Psychiatry	Neurology & Neurosurgery	Clinical Medicine

Summary

Quantifying scientific impact, excellence and quality will always be difficult. However, Ioannidis et al. [2] have provided us with a rich database and a standardized citation metric that accounts for issues of self-citation.

Interestingly, rankings are generally similar on the long-term and one year version of the database for more senior scientists, but they do change somewhat for more junior scientists. Moreover, although the current version of the database only includes data up to 2017, it takes dramatic changes in citations and citation metric scores to cause a large jump in rankings. Thus, it is proposed that the the current database and the information it provides will be valid for a few years.

Will this standardized citation metric be the one to rule them all? Only time will tell.

References

[1] Hicks D, Wouters P, Waltman L, de Rijcke S, Rafols I (2015). Bibliometrics: The Leiden Manifesto for research metrics.
Nature. 520:429-31.

[2] Ioannidis JPA, Baas J, Klavans R, Boyack KW (2019). A standardized citation metrics author database annotated for scientific field. PLoS Biol. 17:e3000384.

[3] Ioannidis JP, Klavans R, Boyack KW (2016). Multiple citation indicators and their composite across scientific disciplines. PLoS Biol. 14:e1002501.