Bibliometrics-based heuristics: What is their definition and how can they be studied? - Research note


When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research are being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publication and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Recently, however Bornmann and Marewski (2019) proposed such a framework. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach (Gigerenzer; Todd; ABC Research Group, 1999) to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.


Binmore, Ken (2007). “Rational decisions in large worlds”. Annales d’economie et de statistique, n. 86, pp. 25-41.

Binmore, Ken (2009). Rational decisions. Princeton, NJ, USA: Princeton University Press. ISBN: 978 0 691149899

Bishop, Doroty (2018). “Luck of the draw. Funders should assign research grants via a lottery system to reduce human bias”. Nature index, May 7.

Bornmann, Lutz (2019). “Bibliometric indicators”. In: Atkinson, P.; Delamont, S.; Cernat, A.; Sakshaug, J. W.; Williams, R. SAGE Research methods foundations. Thousand Oaks, CA, USA: Sage. ISBN: 978 1 526421036

Bornmann, Lutz; Ganser, Christian; Tekles, Alexander; Leydesdorff, Loet (2020). “Does the hα index reinforce the Matthew effect in science? The introduction of agent-based simulations into scientometrics”. Quantitative science studies, v. 1, n. 1, pp. 331-346.

Bornmann, Lutz; Marewski, Julian N. (2019). “Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation”. Scientometrics, n. 120, pp. 419-459.

Bornmann, Lutz; Mutz, Rüdiger; Daniel, Hans Dieter (2008). “Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine”. Journal of the American Society for Information Science and Technology, v. 59, n. 5, pp. 830-837.

Brighton, Henry (2006). “Robust inference with simple cognitive models”. In: Lebiere, C.; Wray, B. Between a rock and a hard place: Cognitive science principles meet Al-hard problems. Papers from the AAAI spring symposium (AAAI Tech Rep No SS-06-03). Menlo Park, CA, USA: AAAI Press, pp. 17-22.

Czerlinski, Jean; Gigerenzer, Gerd; Goldstein, Daniel G. (1999). “How good are simple heuristics?”. In: Gigerenzer, G.; Todd, P. M.; ABC Research Group. Simple heuristics that make us smart. Oxford, UK: Oxford University Press, pp. 97-118. ISBN: 978 0 195143812

De-Rijcke, Sarah; Wouters, Paul F.; Rushforth, Alex D.; Franssen, Thomas P.; Hammarfelt, Björn (2016). “Evaluation practices and effects of indicator use - a literature review”. Research evaluation, v. 25, n. 2, pp. 161-169.

EMBO (2018). EMBO Young Investigator Programme. Application guidelines. Heidelberg, Germany: EMBO.

Evans, Jonathan St. B. T. (2008). “Dual-processing accounts of reasoning, judgment, and social cognition”. Annual review of psychology, v. 59, pp. 255-278.

Gigerenzer, Gerd (2008). Rationality for mortals: How people cope with uncertainty. Oxford, UK: Oxford University Press. ISBN: 978 0 195328981
Gigerenzer, Gerd; Gaissmaier, Wolfgang (2011). “Heuristic decision making”. Annual review of psychology, v. 62, pp. 451-482.

Gigerenzer, Gerd; Goldstein, Daniel G. (1996). “Reasoning the fast and frugal way: Models of bounded rationality”. Psychological review, v. 103, n. 4, pp. 650-669.

Gigerenzer, Gerd; Goldstein, Daniel G. (1999). “Betting on one good reason: The take the best heuristic”. In: Gigerenzer, G.; Todd, P. M.; ABC Research Group. Simple heuristics that make us smart. Oxford, UK: Oxford University Press, pp. 75-95. ISBN: 978 0 195143812

Gigerenzer, Gerd; Hertwig, Ralph; Pachur, Thorsten (eds.) (2011). Heuristics: The foundations of adaptive behavior. Oxford, UK: Oxford University Press. ISBN: 978 0 199744282

Gigerenzer, Gerd; Marewski, Julian N. (2015). “Surrogate science: The idol of a universal method for scientific inference”. Journal of management, v. 41, n. 2, pp. 421-440.

Gigerenzer, Gerd; Todd, Peter M.; ABC Research Group (1999). Simple heuristics that make us smart. Oxford, UK: Oxford University Press. ISBN: 978 0 195143812

Goldstein, Daniel G.; Gigerenzer, Gerd (2009). “Fast and frugal forecasting”. International journal of forecasting, v. 25, pp. 760-772.

Hafenbrädl, Sebastian; Waeger, Daniel; Marewski, Julian N.; Gigerenzer, Gerd (2016). “Applied decision making with fast-and-frugal heuristics”. Journal of applied research in memory and cognition, v. 5, n. 2, pp. 215-231.

Hauser, John R.; Wernerfelt, Birger (1990). “An evaluation cost model of consideration sets”. Journal of consumer research, v. 16, pp. 393-408.

Hertwig, Ralph; Hoffrage, Ulrich; ABC Research Group (2013). Simple heuristics in a social world. New York, NY, USA: Oxford University Press. ISBN: 978 0 195388435

Hertwig, Ralph; Todd, Peter M. (2003). “More is not always better: The benefits of cognitive limits”. In: Hardman, D.; Macchi, L. Thinking: Psychological perspectives on reasoning, judgment and decision making. Hoboken, NY; USA: Wiley, pp. 213-231. ISBN: 0 471 49457 7

Hicks, Diana; Wouters, Paul; Waltman, Ludo; De-Rijcke, Sarah; Ràfols, Ismael (2015). “Bibliometrics: The Leiden Manifesto for research metrics”. Nature, v. 520, n. 7548, pp. 429-431.

Hirsch, Jorge E. (2019). “hα: An index to quantify an individual’s scientific leadership”. Scientometrics, n. 118, pp. 673-686.

Kahnemann, Daniel; Slovic, Paul; Tversky, Amos (eds.) (1982). Judgment under uncertainty: Heuristics and biases. New York, NY, USA: Cambridge University Press. ISBN: 978 0 521 28414 1

Katsikopoulos, Konstantinos V. (2011). “Psychological heuristics for making inferences: Definition, performance, and the emerging theory and practice”. Decision analysis, v. 8, n. 1, pp. 10-29.

Kelman, Mark (2011). The heuristics debate. Oxford, UK: Oxford University Press. ISBN: 978 0 199755608

Knight, Frank H. (1921). Risk, uncertainty and profit. New York, NY, USA: Houghton Mifflin.

MacRoberts, Michael H.; MacRoberts, Barbara R. (2017). “The mismeasure of science: Citation analysis”. Journal of the Association for Information Science and Technology, v. 69, n. 3, pp. 474-482.

Marewski, Julian N.; Gaissmaier, Wolfgang; Gigerenzer, Gerd (2010). “Good judgments do not require complex cognition”. Cognitive processing, v. 11, pp. 103-121.

Marewski, Julian N.; Gaissmaier, Wolfgang; Schooler, Lael J.; Goldstein, Daniel G.; Gigerenzer, Gerd (2010). “From recognition to decisions: Extending and testing recognition-based models for multialternative inference”. Psychonomic bulletin & review, v. 17, pp. 287-309.

Marewski, Julian N.; Schooler, Lael J. (2011). “Cognitive niches: An ecological model of strategy selection”. Psychological review, v. 118, n. 3, pp. 393-437.

Marx, Werner (2014). “The Shockley-Queisser paper - A notable example of a scientific sleeping beauty”. Annalen der Physik, v. 526, n. 5-6, pp. A41-A45.

Merton, Robert K. (1968). “The Matthew effect in science”. Science, v. 159, n. 3810, pp. 56-63.

Raab, Markus; Gigerenzer, Gerd (2015). “The power of simplicity: A fast-and-frugal heuristics approach to performance science”. Frontiers in psychology, 6, 1672.

Research Excellence Framework (2014). Research Excellence Framework.

Rothenfluh, Tomas; Daniel, Hans-Dieter (2009). “Qualitätssicherung in Forschung, Lehre und Studium am Beispiel der Universität Zürich [Quality assurance in research, teaching, and studies exemplified by the University of Zurich]”. In: Webler, W-D. Universitäten am Scheideweg?! - Chancen und Gefahren des gegenwärtigen historischen Wandels in Verfassung, Selbstverständnis und Aufgabenwahrnehmung [Universities at the crossroads?! Opportunities and dangers of current historical change in the constitution, self-conception and task performance]. Bielefeld, Germany: UniversitätsVerlagWebler, pp. 259-278. ISBN: 978 3 937026 64 0

Savage, Leonard J. (1954). The foundations of statistics. New York, NY, USA: Wiley. ISBN: 978 0 486623498

Simon, Herbert A. (1956). “Rational choice and the structure of the environment”. Psychological review, v. 63, n. 2, pp. 129-138.

Simon, Herbert A. (1990). “Invariants of human-behavior.” Annual review of psychology, v. 41, pp. 1-19.

Todd, Peter M.; Gigerenzer, Gerd; ABC Research Group (2012). Ecological rationality: Intelligence in the world. New York, NY, USA: Oxford University Press. ISBN: 978 0 195315448

Waltman, Ludo (2018). “Responsible metrics: One size doesn’t fit all”. In: Wouters P. Proceedings of the Science and Technology Indicators Conference 2018, Leiden. Science, Technology and Innovation indicators in transition. Leiden, The Netherlands: University of Leiden, pp. 526-531.