Bibliometrics-based heuristics: What is their definition and how can they be studied? - Research note

Resumen

When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research are being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publication and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Recently, however Bornmann and Marewski (2019) proposed such a framework. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach (Gigerenzer; Todd; ABC Research Group, 1999) to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.

Citas

Binmore, Ken (2007). “Rational decisions in large worlds”. Annales d’economie et de statistique, n. 86, pp. 25-41. https://doi.org/10.2307/20079192

Binmore, Ken (2009). Rational decisions. Princeton, NJ, USA: Princeton University Press. ISBN: 978 0 691149899

Bishop, Doroty (2018). “Luck of the draw. Funders should assign research grants via a lottery system to reduce human bias”. Nature index, May 7. https://www.natureindex.com/news-blog/luck-of-the-draw

Bornmann, Lutz (2019). “Bibliometric indicators”. In: Atkinson, P.; Delamont, S.; Cernat, A.; Sakshaug, J. W.; Williams, R. SAGE Research methods foundations. Thousand Oaks, CA, USA: Sage. ISBN: 978 1 526421036

Bornmann, Lutz; Ganser, Christian; Tekles, Alexander; Leydesdorff, Loet (2020). “Does the hα index reinforce the Matthew effect in science? The introduction of agent-based simulations into scientometrics”. Quantitative science studies, v. 1, n. 1, pp. 331-346. https://doi.org/10.1162/qss_a_00008

Bornmann, Lutz; Marewski, Julian N. (2019). “Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation”. Scientometrics, n. 120, pp. 419-459. https://doi.org/10.1007/s11192-019-03018-x

Bornmann, Lutz; Mutz, Rüdiger; Daniel, Hans Dieter (2008). “Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine”. Journal of the American Society for Information Science and Technology, v. 59, n. 5, pp. 830-837. https://doi.org/10.1002/asi.20806

Brighton, Henry (2006). “Robust inference with simple cognitive models”. In: Lebiere, C.; Wray, B. Between a rock and a hard place: Cognitive science principles meet Al-hard problems. Papers from the AAAI spring symposium (AAAI Tech Rep No SS-06-03). Menlo Park, CA, USA: AAAI Press, pp. 17-22. http://www.aaai.org/Papers/Symposia/Spring/2006/SS-06-02/SS06-02-006.pdf

Czerlinski, Jean; Gigerenzer, Gerd; Goldstein, Daniel G. (1999). “How good are simple heuristics?”. In: Gigerenzer, G.; Todd, P. M.; ABC Research Group. Simple heuristics that make us smart. Oxford, UK: Oxford University Press, pp. 97-118. ISBN: 978 0 195143812

De-Rijcke, Sarah; Wouters, Paul F.; Rushforth, Alex D.; Franssen, Thomas P.; Hammarfelt, Björn (2016). “Evaluation practices and effects of indicator use - a literature review”. Research evaluation, v. 25, n. 2, pp. 161-169. https://doi.org/10.1093/reseval/rvv038

EMBO (2018). EMBO Young Investigator Programme. Application guidelines. Heidelberg, Germany: EMBO. https://www.embo.org/documents/YIP/application_guidelines.pdf

Evans, Jonathan St. B. T. (2008). “Dual-processing accounts of reasoning, judgment, and social cognition”. Annual review of psychology, v. 59, pp. 255-278. https://doi.org/10.1146/annurev.psych.59.103006.093629

Gigerenzer, Gerd (2008). Rationality for mortals: How people cope with uncertainty. Oxford, UK: Oxford University Press. ISBN: 978 0 195328981
Gigerenzer, Gerd; Gaissmaier, Wolfgang (2011). “Heuristic decision making”. Annual review of psychology, v. 62, pp. 451-482. https://doi.org/10.1146/annurev-psych-120709-145346

Gigerenzer, Gerd; Goldstein, Daniel G. (1996). “Reasoning the fast and frugal way: Models of bounded rationality”. Psychological review, v. 103, n. 4, pp. 650-669. https://doi.org/10.1037/0033-295X.103.4.650

Gigerenzer, Gerd; Goldstein, Daniel G. (1999). “Betting on one good reason: The take the best heuristic”. In: Gigerenzer, G.; Todd, P. M.; ABC Research Group. Simple heuristics that make us smart. Oxford, UK: Oxford University Press, pp. 75-95. ISBN: 978 0 195143812

Gigerenzer, Gerd; Hertwig, Ralph; Pachur, Thorsten (eds.) (2011). Heuristics: The foundations of adaptive behavior. Oxford, UK: Oxford University Press. ISBN: 978 0 199744282

Gigerenzer, Gerd; Marewski, Julian N. (2015). “Surrogate science: The idol of a universal method for scientific inference”. Journal of management, v. 41, n. 2, pp. 421-440. https://doi.org/10.1177/0149206314547522

Gigerenzer, Gerd; Todd, Peter M.; ABC Research Group (1999). Simple heuristics that make us smart. Oxford, UK: Oxford University Press. ISBN: 978 0 195143812 https://www.researchgate.net/profile/Ulrich_Hoffrage/publication/227466812_Simple_Heuristics_That_Make_Us_Smart/links/00b7d517f86ec6844d000000/Simple-Heuristics-That-Make-Us-Smart.pdf

Goldstein, Daniel G.; Gigerenzer, Gerd (2009). “Fast and frugal forecasting”. International journal of forecasting, v. 25, pp. 760-772. https://pure.mpg.de/rest/items/item_2099944_4/component/file_2099943/content

Hafenbrädl, Sebastian; Waeger, Daniel; Marewski, Julian N.; Gigerenzer, Gerd (2016). “Applied decision making with fast-and-frugal heuristics”. Journal of applied research in memory and cognition, v. 5, n. 2, pp. 215-231. https://doi.org/10.1016/j.jarmac.2016.04.011

Hauser, John R.; Wernerfelt, Birger (1990). “An evaluation cost model of consideration sets”. Journal of consumer research, v. 16, pp. 393-408. https://doi.org/10.1086/209225

Hertwig, Ralph; Hoffrage, Ulrich; ABC Research Group (2013). Simple heuristics in a social world. New York, NY, USA: Oxford University Press. ISBN: 978 0 195388435 https://doi.org/10.1093/acprof:oso/9780195388435.001.0001

Hertwig, Ralph; Todd, Peter M. (2003). “More is not always better: The benefits of cognitive limits”. In: Hardman, D.; Macchi, L. Thinking: Psychological perspectives on reasoning, judgment and decision making. Hoboken, NY; USA: Wiley, pp. 213-231. ISBN: 0 471 49457 7 http://www.al-edu.com/wp-content/uploads/2014/05/HarmanMacchi-eds-Thinking-Psychological-Perspectives-on-Reasoning-Judgement-and-Decision-Making.pdf

Hicks, Diana; Wouters, Paul; Waltman, Ludo; De-Rijcke, Sarah; Ràfols, Ismael (2015). “Bibliometrics: The Leiden Manifesto for research metrics”. Nature, v. 520, n. 7548, pp. 429-431. https://doi.org/10.1038/520429a

Hirsch, Jorge E. (2019). “hα: An index to quantify an individual’s scientific leadership”. Scientometrics, n. 118, pp. 673-686. https://doi.org/10.1007/s11192-018-2994-1

Kahnemann, Daniel; Slovic, Paul; Tversky, Amos (eds.) (1982). Judgment under uncertainty: Heuristics and biases. New York, NY, USA: Cambridge University Press. ISBN: 978 0 521 28414 1

Katsikopoulos, Konstantinos V. (2011). “Psychological heuristics for making inferences: Definition, performance, and the emerging theory and practice”. Decision analysis, v. 8, n. 1, pp. 10-29. https://doi.org/10.1287/deca.1100.0191

Kelman, Mark (2011). The heuristics debate. Oxford, UK: Oxford University Press. ISBN: 978 0 199755608

Knight, Frank H. (1921). Risk, uncertainty and profit. New York, NY, USA: Houghton Mifflin.

MacRoberts, Michael H.; MacRoberts, Barbara R. (2017). “The mismeasure of science: Citation analysis”. Journal of the Association for Information Science and Technology, v. 69, n. 3, pp. 474-482. https://doi.org/10.1002/asi.23970

Marewski, Julian N.; Gaissmaier, Wolfgang; Gigerenzer, Gerd (2010). “Good judgments do not require complex cognition”. Cognitive processing, v. 11, pp. 103-121. https://doi.org/10.1007/s10339-009-0337-0

Marewski, Julian N.; Gaissmaier, Wolfgang; Schooler, Lael J.; Goldstein, Daniel G.; Gigerenzer, Gerd (2010). “From recognition to decisions: Extending and testing recognition-based models for multialternative inference”. Psychonomic bulletin & review, v. 17, pp. 287-309. https://doi.org/10.3758/PBR.17.3.287

Marewski, Julian N.; Schooler, Lael J. (2011). “Cognitive niches: An ecological model of strategy selection”. Psychological review, v. 118, n. 3, pp. 393-437. https://doi.org/10.1037/a0024143

Marx, Werner (2014). “The Shockley-Queisser paper - A notable example of a scientific sleeping beauty”. Annalen der Physik, v. 526, n. 5-6, pp. A41-A45. https://doi.org/10.1002/andp.201400806

Merton, Robert K. (1968). “The Matthew effect in science”. Science, v. 159, n. 3810, pp. 56-63. https://doi.org/10.1126/science.159.3810.56

Raab, Markus; Gigerenzer, Gerd (2015). “The power of simplicity: A fast-and-frugal heuristics approach to performance science”. Frontiers in psychology, 6, 1672. https://doi.org/10.3389/fpsyg.2015.01672

Research Excellence Framework (2014). Research Excellence Framework. http://www.ref.ac.uk/2014

Rothenfluh, Tomas; Daniel, Hans-Dieter (2009). “Qualitätssicherung in Forschung, Lehre und Studium am Beispiel der Universität Zürich [Quality assurance in research, teaching, and studies exemplified by the University of Zurich]”. In: Webler, W-D. Universitäten am Scheideweg?! - Chancen und Gefahren des gegenwärtigen historischen Wandels in Verfassung, Selbstverständnis und Aufgabenwahrnehmung [Universities at the crossroads?! Opportunities and dangers of current historical change in the constitution, self-conception and task performance]. Bielefeld, Germany: UniversitätsVerlagWebler, pp. 259-278. ISBN: 978 3 937026 64 0

Savage, Leonard J. (1954). The foundations of statistics. New York, NY, USA: Wiley. ISBN: 978 0 486623498

Simon, Herbert A. (1956). “Rational choice and the structure of the environment”. Psychological review, v. 63, n. 2, pp. 129-138. https://doi.org/10.1037/h0042769

Simon, Herbert A. (1990). “Invariants of human-behavior.” Annual review of psychology, v. 41, pp. 1-19. https://doi.org/10.1146/annurev.ps.41.020190.000245

Todd, Peter M.; Gigerenzer, Gerd; ABC Research Group (2012). Ecological rationality: Intelligence in the world. New York, NY, USA: Oxford University Press. ISBN: 978 0 195315448 https://doi.org/10.1093/acprof:oso/9780195315448.001.0001

Waltman, Ludo (2018). “Responsible metrics: One size doesn’t fit all”. In: Wouters P. Proceedings of the Science and Technology Indicators Conference 2018, Leiden. Science, Technology and Innovation indicators in transition. Leiden, The Netherlands: University of Leiden, pp. 526-531. http://sti2018.cwts.nl/proceedings
Publicado
2020-05-22