COUNTING OR MEASUREMENT: THE LIMITS OF TRADITIONAL BIBLIOMETRICS AND AN INTERSUBJECTIVE ALTERNATIVE
DOI:
https://doi.org/10.62405/osiconf.2026.20Keywords:
research assessment; responsible metrics; intersubjective indicators; working time structure; Goodhart's law; open science.Abstract
The purpose of the paper is to substantiate an instrument for assessing the state of a research system that does not rely on quantitative indicators of publication activity. The methods are conceptual analysis of the literature, metrological analysis of the notion of measurement, analysis of the allocation of researchers' working time, and descriptive quantitative analysis of retraction dynamics based on Retraction Watch and SCImago data for 1996–2025. Heterogeneous papers and citations are shown not to form a quantity that can be measured, while the share of articles later retracted has grown roughly tenfold over two decades. A categorisation of working time into actual scientific, useful non-scientific and meaningless activity is proposed, together with a system of intersubjective indices built on the diffusion-index methodology of economic sentiment indicators. Effectiveness is redefined as the share of time preserved for actual scientific activity.
References
Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. SAGE Open, 9(1). https://doi.org/10.1177/2158244019829575
Becker, G. S. (1965). A theory of the allocation of time. The Economic Journal, 75(299), 493–517. https://doi.org/10.2307/2228949
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. https://doi.org/10.1016/0149-7189(79)90048-X
CoARA. (2022). Agreement on reforming research assessment. Coalition for Advancing Research Assessment. https://coara.eu/agreement/the-agreement-full-text/
Curtin, R. (2007). Consumer sentiment surveys: Worldwide review and assessment. Journal of Business Cycle Measurement and Analysis, 2007(1), 7–42. https://doi.org/10.1787/jbcma-v2007-art2-en
Else, H., & Van Noorden, R. (2021). The fight against fake-paper factories that churn out sham science. Nature, 591(7851), 516–519. https://doi.org/10.1038/d41586-021-00733-5
Fire, M., & Guestrin, C. (2019). Over-optimization of academic publishing metrics: Observing Goodhart's law in action. GigaScience, 8(6). https://doi.org/10.1093/gigascience/giz053
Graeber, D. (2018). Bullshit jobs: A theory. Simon & Schuster. https://openlibrary.org/works/OL20153626W
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a
Koenig, E. F. (2002). Using the Purchasing Managers' Index to assess the economy's strength and the likely direction of monetary policy. Economic and Financial Policy Review, 1(6). https://ideas.repec.org/a/fip/fedder/y2002nv.1no.6.html
Naddaf, M. (2025). Europe's largest paper mill? 1,500 research articles linked to Ukrainian network. Nature. https://doi.org/10.1038/d41586-025-02809-y
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
The Retraction Watch, (n.d.). The Retraction Watch Database [Data set]. Retrieved August 9, 2026. http://retractiondatabase.org/
SCImago. (n.d.). SJR — SCImago Journal & Country Rank [Portal]. Retrieved August 9, 2026. https://www.scimagojr.com/
Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), 160384. https://doi.org/10.1098/rsos.160384
