Altmetrics and societal impact measurements: Match or mismatch? A literature review


Can alternative metrics (altmetrics) data be used to measure societal impact? We wrote this literature overview of empirical studies in order to find an answer to this question. The overview includes two parts. The first part, “societal impact measurements”, explains possible methods and problems in measuring the societal impact of research, case studies for societal impact measurement, societal impact considerations at funding organizations, and the societal problems that should be solved by science. The second part of the review, “altmetrics”, addresses a major question in research evaluation, which is whether altmetrics are proper indicators for measuring the societal impact of research. In the second part we explain the data sources used for altmetrics studies and the importance of field-normalized indicators for impact measurements. This review indicates that it should be relevant for impact measurements to be oriented towards pressing societal problems. Case studies in which societal impact of certain pieces of research is explained seem to provide a legitimate method for measuring societal impact. In the use of altmetrics, field-specific differences should be considered by applying field normalization (in cross-field comparisons). Altmetrics data such as social media counts might mainly reflect the public interest and discussion of scholarly works rather than their societal impact. Altmetrics (Twitter data) might be especially fruitfully employed for research evaluation purposes, if they are used in the context of network approaches. Conclusions based on altmetrics data in research evaluation should be drawn with caution.


Adie, Euan (2014). “Taking the alternative mainstream”. El profesional de la información, v. 23, n. 4, pp. 349-351.

Bar-Ilan, Judit (2014). “Jasist@ Mendeley revisited”. Altmetrics14: Expanding impacts and metrics, Workshop at Web Science Conference 2014.

Barthel, Simon; Tönnies, Sascha; Köhncke, Benjamin; Siehndel, Patrick; Balke, Wolf-Tilo (2015). “What does Twitter measure? Influence of diverse user groups in altmetrics”. ACM/IEEE Joint conf on digital libraries (JCDL), Knoxville, TN, USA.

Belter, Christopher W. (2018). “Providing meaningful information: Part B - Bibliometric analysis”. In: DeRosa, A. P. (ed.). A practical guide for informationists. Sawston, Cambridge, UK: Chandos Publishing, pp. 33-47. ISBN: 978 0 081020173

Blümel, Clemens; Gauch, Stephan; Beng, Florian (2017). “Altmetrics and its intellectual predecessors: Patterns of argumentation and conceptual development”. Proceedings of the Science, technology, & innovation indicators conference. Open indicators: innovation, participation and actor-based STI indicators. Paris, France.

Boling, Erica C.; Castek, Jill; Zawilinski, Lisa; Barton, Karen; Nierlich, Theresa (2008). “Collaborative literacy: Blogs and Internet projects”. The reading teacher, v. 61, n. 6, pp. 504-506.

Bonetta, Laura (2007). “Scientists enter the blogosphere”. Cell, v. 129, n. 3, pp. 443-445.

Bornmann, Lutz (2016). “Scientific revolution in scientometrics: The broadening of impact from citation to societal”. In: Sugimoto, Cassidy R. (ed.). Theories of informetrics and scholarly communication. Berlin, Germany: De Gruyter, pp. 347-359. ISBN: 978 3 11 030846 4

Bornmann, Lutz (in press). “Bibliometric indicators – methods for measuring science”. In: Williams, Richard (ed.). Encyclopedia of research methods. Thousand Oaks, CA, USA: Sage.

Bornmann, Lutz; Haunschild, Robin (2015). “Which people use which scientific papers? An evaluation of data from F1000 and Mendeley”. Journal of informetrics, v. 9, n. 3, pp. 477-487.

Bornmann, Lutz; Haunschild, Robin (2016a). “How to normalize Twitter counts? A first attempt based on journals in the Twitter Index”. Scientometrics, v. 107, n. 3, pp. 1405-1422.

Bornmann, Lutz; Haunschild, Robin (2016b). “Normalization of Mendeley reader impact on the reader- and paper-side: A comparison of the mean discipline normalized reader score (MDNRS) with the mean normalized reader score (MNRS) and bare reader counts”. Journal of informetrics, v. 10, n. 3, pp. 776-788.

Bornmann, Lutz; Haunschild, Robin (2016c). “Overlay maps based on Mendeley data: The use of altmetrics for readership networks”. Journal of the Association for Information Science and Technology, v. 67, n. 12, pp. 3064-3072.

Bornmann, Lutz; Haunschild, Robin (2017). “Measuring field-normalized impact of papers on specific societal groups: An altmetrics study based on Mendeley data”. Research evaluation, v. 26, n. 3, pp. 230-241.

Bornmann, Lutz; Haunschild, Robin (2018). “Normalization of zero-inflated data: An empirical analysis of a new indicator family and its use with altmetrics data”. Journal of informetrics, v. 12, n. 3, pp. 998-1011.

Bornmann, Lutz; Haunschild, Robin; Adams, Jonathan (2019). “Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)”. Journal of informetrics, v. 13, n. 1, pp. 325-340.

Bornmann, Lutz; Haunschild, Robin; Marx, Werner (2016). “Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents?”. Scientometrics, v. 109, n. 3, pp. 1477-1495.

Bornmann, Lutz; Tekles, Alexander; Zhang, Helena H.; Ye, Fred, Y. (2019). “Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data”. Journal of informetrics, v. 13, n. 4.

Bunn, Frances; Trivedi, Daksha; Alderson, Phil; Hamilton, Laura; Martin, Alice; Pinkney, Emma; Iliffe, Steve (2015). “The impact of Cochrane Reviews: a mixed-methods evaluation of outputs from Cochrane Review Groups supported by the National Institute for Health Research”. Health technology assessment, v. 19, n. 28.

Burke, James; Bergman, Jules; Asimov, Isaac (1985). The impact of science on society. Washington, DC, USA: National Aeronautics and Space Administration.

Bush, Vannevar (1945). Science: the endless frontier. A report to president Truman outlining his proposal for post-war U.S. science and technology policy. Washington, DC, USA: United States Government Printing Office.

Buttliere, Brett; Buder, Jürgen (2017). “Personalizing papers using Altmetrics: Comparing paper ‘Quality’ or ‘Impact’ to person ‘Intelligence’ or ‘Personality’”. Scientometrics, v. 111, n. 1, pp. 219-239.

Buxton, Martin; Hanney, Steve (1998). “Evaluating the NHS research and development programme: Will the programme give value for money?”. Journal of the Royal Society of Medicine, v. 91, n. 35 suppl, pp. 2-6.

Callaert, Julie; Van Looy, Bart; Verbeek, Arnold; Debackere, Koenraad; Thijs, Bart (2006). “Traces of prior art: An ana­lysis of non-patent references found in patent documents”. Scientometrics, v. 69, n. 1, pp. 3-20.

Dance, Amber (2013). “Impact: pack a punch”. Nature, v. 502, pp. 397-398.

Daugbjerg, Signe B.; Kahlmeier, Sonja; Racioppi, Francesca; Martin-Diener, Eva; Martin, Brian; Oja, Pekka; Bull, Fiona (2009). “Promotion of physical activity in the European region: Content analysis of 27 national policy documents”. Journal of physical activity and health, v. 6, n. 6, pp. 805-817.

De-Jong, Stefan P. L.; Smit, Jorrit; Van-Drooge, Leonie (2016). “Scientists’ response to societal impact policies: A policy paradox”. Science and public policy, v. 43, n. 1, pp. 102-114.

De-Jong, Stefan P. L.; Van-Arensbergen, Pleun; Daemen, Floortje; Van-der-Meulen, Barend; Van-den-Besselaar, Peter (2011). “Evaluation of research in context: An approach and two cases”. Research evaluation, v. 20, n. 1, pp. 61-72.

Department for Business, Energy & Industrial Strategy (2016). Building on success and learning from experience. An independent review of the Research Excellence Framework. London, UK: Department for Business, Energy & Industrial Strategy.

Derrick, Gemma E.; Samuel, Gabrielle N. (2016). “The evaluation scale: Exploring decisions about societal impact in peer review panels”. Minerva, v. 54, n. 1, pp. 75-97.

Donovan, Claire (2008). “The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research”. New directions for evaluation, n. 118, pp. 47-60.

Eisenstein, Michael (2016). “Assessment: Academic return”. Nature, v. 533, n. 7601, pp. S20-S21.

ERIC (2010). Evaluating the societal relevance of academic research: A guide. Delft, The Netherlands: Delft University of Technology.

Ernø-Kjølhede, Erik (2000). Scientific norms as (dis) integrators of scientists? Copenhagen, Denmark: Copenhagen Business School.

Ernø-Kjølhede, Erik; Hansson, Finn (2011). “Measuring research performance during a changing relationship between science and society”. Research evaluation, v. 20, n. 2, pp. 131-143.

Etzkowitz, Henry; Leydesdorff, Loet (2000). “The dynamics of innovation: From national systems and “Mode 2” to a Triple Helix of university–industry–government relations”. Research policy, v. 29, n. 2, pp. 109-123.

Fausto, Sibele; Machado, Fabio A.; Bento, Luiz-Fernando J.; Iamarino, Atila; Nahas, Tatiana R.; Munger, David S. (2012). “Research blogging: Indexing and registering the change in science 2.0”. PLoS one, v. 7, n. 12, e50109.

Feller, Irwin (2017). “Assessing the societal impact of publicly funded research”. The journal of technology transfer (first online).

Godin, Benoît; Doré, Christian (2005). “Measuring the impacts of science; beyond the economic dimension”. INRS Urbanisation, Culture et Société. HIST Lecture, Helsinki Institute for Science and Technology Studies, Helsinki, Finland.

Göransson, Bo; Maharajh, Rasigan; Schmoch, Ulrich (2009). “New activities of universities in transfer and extension: Multiple requirements and manifold solutions”. Science and public policy, v. 36, n. 2, pp. 157-164.

Grant, Jonathan (1999). “Evaluating the outcomes of biomedical research on healthcare”. Research evaluation, v. 8, n. 1, pp. 33-38.

Gregersen, Birgitte; Linde, Lisbeth-Tved; Rasmussen, Jorgen-Gulddahl (2009). “Linking between Danish universities and society”. Science and public policy, v. 36, n. 2, pp. 151-156.

Groth, Paul; Gurney, Thomas (2010). “Studying scientific discourse on the Web using bibliometrics: A chemistry blogging case study”. In: Proceedings of the WebSci10: Extending the frontiers of society on-line, Raleigh, NC, USA: Web Science Trust.

Hammarfelt, Björn (2014). “Using altmetrics for assessing research impact in the humanities”. Scientometrics, v. 101, n. 2, pp. 1419-1430.

Haunschild, Robin; Bornmann, Lutz (2016). “Normalization of Mendeley reader counts for impact assessment”. Journal of informetrics, v. 10, n. 1, pp. 62-73.

Haunschild, Robin; Bornmann, Lutz (2017). “How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data”. Scientometrics, v. 110, n. 3, pp. 1209-1216.

Haunschild, Robin; Bornmann, Lutz (2018). “Field-and time-normalization of data with many zeros: an empirical analysis using citation and Twitter data”. Scientometrics, v. 116, n. 2, pp. 997-1012.

Haunschild, Robin; Leydesdorff, Loet; Bornmann, Lutz; Hellsten, Iina; Marx, Werner (2019). “Does the public discuss other topics on climate change than researchers? A comparison of explorative networks based on author keywords and hashtags”. Journal of informetrics, v. 13, n. 2, pp. 695-707.

Haustein, Stefanie (2014). “Readership metrics”. In: Cronin, Blaise; Sugimoto, Cassidy R. (eds.). Beyond bibliometrics: harnessing multi-dimensional indicators of performance. Cambridge, MA, USA: MIT Press, pp. 327-344. ISBN: 978 0 262026796

Haustein, Stefanie; Costas, Rodrigo; Larivière, Vincent (2015). “Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns”. PLoS one, v. 10, n. 3, e0120495.

Haustein, Stefanie; Larivière, Vincent; Thelwall, Mike; Amyot, Didier; Peters, Isabella (2014). “Tweets vs. Mendeley readers: How do these two social media metrics differ?”. IT – Information technology, v. 56, n. 5, pp. 207-215.

Health Economics Research Group, RAND Europe (2008). Medical research: what’s it worth? Estimating the economic benefits from medical research in the UK. London, UK: Evaluation Forum.

Hellsten, Iina; Leydesdorff, Loet (2020). “Automated analysis of actor–topic networks on twitter: New approaches to the analysis of socio‐semantic networks”. Journal of the Association for Information Science and Technology, v. 71, n. 1, pp. 3-15.

Heyeres, Marion; Tsey, Komla; Yang, Yinghong, Yan, Li; Jiang, Hua (2019). “The characteristics and reporting quality of research impact case studies: A systematic review”. Evaluation and program planning, v. 73, pp. 10-23.

Hicks, Daniel J.; Stahmer, Carl; Smith, MacKenzie (2018). “Impacting capabilities: A conceptual framework for the social value of research”. Frontiers in research metrics and analytics, v. 3, n. 24.

Higher Education Funding Council for England (2009). Research Excellence Framework. Second consultation on the assessment and funding of research. September 2009/38. Bristol, UK: Higher Education Funding Council for England (HEFCE).

Holbrook, James-Britt; Hrotic, Steven (2013). “Blue skies, impacts, and peer review”. RT. A journal on research policy & evaluation, v. 1, n. 1.

Jones, Teresa H.; Hanney, Steve (2016). “Tracing the indirect societal impacts of biomedical research: Development and piloting of a technique based on citations”. Scientometrics, v. 107, n. 3, pp. 975-1003.

Kassab, Omar (2019). “Does public outreach impede research performance? Exploring the ‘researcher’s dilemma’ in a sustainability research center”. Science and public policy, v. 46, n. 5, pp. 710-720.

Kellard, Neil M.; Śliwa, Martina (2016). “Business and management impact assessment in Research Excellence Framework 2014: Analysis and reflection”. British journal of management, v. 27, n. 4, pp. 693-711.

Khazragui, Hanan; Hudson, John (2015). “Measuring the benefits of university research: Impact and the REF in the UK”. Research evaluation, v. 24, n. 1, pp. 51-62.

King’s College London and Digital Science. (2015). The nature, scale and beneficiaries of research impact: An initial ana­lysis of Research Excellence Framework (REF) 2014 impact case studies. London, UK: King’s College London.

Kok, Maarten O.; Schuit, Albertine J. (2012). “Contribution mapping: A method for mapping the contribution of research to enhance its impact”. Health research policy and systems, v. 10, n. 1, article 21.

Konkiel, Stacy; Madjarevic, Natalia; Lightfoot, Amy (2016). Altmetrics for librarians: 100+ tips, tricks, and examples.

Kovic, Ivor; Lulic, Ileana; Brumini, Gordana (2008). “Examining the medical blogosphere: An online survey of medical bloggers”. Journal of medical internet research, v. 10, n. 3, e28.

Kreiman, Gabriel; Maunsell, John H. R. (2011). “Nine criteria for a measure of scientific output”. Frontiers in computational neuroscience, v. 5, n. 48.

Lähteenmäki-Smith, Kaisa; Hyytinen, Kirsi-Maria; Kutinlahti, Pirjo; Konttinen, Jari (2006). Research with an impact: Evaluation practises in public research organisations. Espoo, Finland: VTT Technical Research Centre of Finland.

Langfeldt, Liv; Scordato, Lisa (2015). Assessing the broader impacts of research: A review of methods and practices. Oslo, Norway: Nordic Institute for Studies in Innovation, Research and Education (NIFU).

Lewison, Grant; Sullivan, Richard (2008a). “The impact of cancer research: how publications influence UK cancer clinical guidelines”. British journal of cancer, v. 98, n. 12, pp. 1944-1950.

Liu, Chun-Li; Xu, Yue-Quan; Wu, Hui; Chen, Si-Si; Guo, Ji-Jun (2013). “Correlation and interaction visualization of altme­tric indicators extracted from scholarly social network activities: Dimensions and structure”. Journal of medical internet research, v. 15, n. 11, pp. 17.

Liu, Jean; Adie, Euan (2013). “Five challenges in altmetrics: A toolmaker’s perspective”. Bulletin of the American Society for Information Science and Technology, v. 39, n. 4, pp. 31-34.

Mansfield, Edwin (1991). “Academic research and industrial innovation”. Research policy, v. 20, n. 1, pp. 1-12.

Mansfield, Edwin (1998). “Academic research and industrial innovation: an update of empirical findings”. Research policy, v. 26, n. 7-8, pp. 773-776.

Marcella, Rita; Lockerbie, Hayley; Bloice, Lyndsay; Hood, Caroline; Barton, Flora (2018). “The effects of the research excellence framework research impact agenda on early- and mid-career researchers in library and information science”. Journal of information science, v. 44, n. 5, pp. 608-618.

Martin, Ben R. (2007). “Assessing the impact of basic research on society and the economy”. In: Rethinking the impact of basic research on society and the economy, Vienna, Austria.

Mas-Bleda, Amalia; Thelwall, Mike (2016). “Can alternative indicators overcome language biases in citation counts? A comparison of Spanish and UK research”. Scientometrics, v. 109, n. 3, pp. 2007-2030.

Merton, Robert K. (1968a). “The Matthew effect in science: The reward and communication systems of science are considered”. Science, v. 159, n. 3810, pp. 56-63.

Merton, Robert K. (1968b). Social theory and social structure. New York, NY, USA: Simon and Schuster. ISBN: 978 0 226520711

Merton, Robert K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago, IL, USA: University of Chicago Press. ISBN: 978 0 226520926

Miettinen, Reijo; Tuunainen, Juha; Esko, Terhi (2015). “Epistemological, artefactual and interactional–institutional foundations of social impact of academic research”. Minerva, v. 53, n. 3, pp. 257-277.

Mitroff, Ian I. (1974). “Norms and counter-norms in a select group of the Apollo moon scientists: A case study of the ambivalence of scientists”. American sociological review, v. 39, n. 4, pp. 579-595.

Moed, Henk F. (2017). Applied evaluative informetrics. Heidelberg, Germany: Springer. ISBN: 978 3 319 60522 7

Moed, Henk F.; Halevi, Gali (2015). “Multidimensional assessment of scholarly research impact”. Journal of the Association for Information Science and Technology, v. 66, n. 10, pp. 1988-2002.

Mohammadi, Ehsan; Thelwall, Mike (2014). “Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows”. Journal of the Association for Information Science and Technology, v. 65, n. 8, pp. 1627-1638.

Mohammadi, Ehsan; Thelwall, Mike; Haustein, Stefanie; Larivière, Vincent (2015). “Who reads research articles? An altmetrics analysis of Mendeley user categories”. Journal of the Association for Information Science and Technology, v. 66, n. 9, pp. 1832-1846.

Mohammadi, Ehsan; Thelwall, Mike; Kwasny, Mary; Holmes, Kristi L. (2018). “Academic information on Twitter: A user survey”. PLoS one, v. 13, n. 5, e0197265.

Molas-Gallart, Jordi; Salter, Ammon J.; Patel, Pari; Scott, Alister; Duran, Xavier (2002). Measuring third stream activities. Final report to the Russell Group of Universities. Brighton, UK: SPRU, University of Sussex.

Morris, Zoë-Slote; Wooding, Steven; Grant, Jonathan (2011). “The answer is 17 years, what is the question: Understanding time lags in translational research”. Journal of the Royal Society of Medicine, v. 104, n. 12, pp. 510-520.

Mostert, Sebastian P.; Ellenbroek, Stéfan P. H.; Meijer, Ingeborg; Van-Ark, Gerrit; Klasen, Eduard C. (2010). “Societal out­put and use of research performed by health research groups”. Health research policy and systems, v. 8, n. 1, article 30.

Narin, Francis (1976). Evaluative bibliometrics: the use of publication and citation analysis in the evaluation of scientific activity. Cherry Hill, NJ, USA: Computer Horizons, 459 pp.

Narin, Francis; Hamilton, Kimberley S.; Olivastro, Dominic (1997). “The increasing linkage between US technology and public science”. Research policy, v. 26, n. 3, pp. 317-330.

National Research Council (2014). Furthering America’s research enterprise. Washington, DC, USA: The National Academies Press. ISBN: 978 0 309 30522 8

Nelhans, Gustaf (2016). “Professional impact of clinical research”. In: Ràfols, Ismael; Molas-Gallart, Jordi; Castro-Martínez, Elena; Woolley, Richard (eds.). Proceedings of the 21 st International Conference on Science and Technology Indicators. València, Spain: Universitat Politècnica de València.

Noyons, Ed (2018). Monitoring how science finds its way into society: Measuring societal impact through area-based connectedness (ABC).

Noyons, Ed (2019). “Measuring societal impact is as complex as ABC”. Journal of data and information science, v. 4, n. 3, pp. 6-21.

Noyons, Ed; Ràfols, Ismael (2018). “Can bibliometrics help in assessing societal contributions of agricultural research? Exploring societal interactions across research areas”. In: Wouters, Paul (ed.), Proceedings of the science and technology indicators conference 2018 Leiden. Science, Technology and Innovation indicators in transition. Leiden, the Netherlands: University of Leiden.

Nussbaum, Martha C. (2001). Women and human development: The capabilities approach (Vol. 3). Cambridge, UK: Cambridge University Press. ISBN: 978 0 521003858

Nussbaum, Martha C. (2009). Frontiers of justice: Disability, nationality, species membership. Cambridge, MA, USA: Harvard University Press. ISBN: 978 0 674024106

Oancea, Alis (2013). “Buzzwords and values: the prominence of ‘impact’ in UK research policy and governance. Research trends, n. 33, pp. 6-8.

Ortega, José-Luis (2018). “Disciplinary differences of the impact of altmetric”. Fems microbiology letters, v. 365, n. 7.

Ovseiko, Pavel V.; Oancea, Alis; Buchan, Alastair M. (2012). “Assessing research impact in academic clinical medicine: A study using Research Excellence Framework pilot impact indicators”. BMC Health services research, n. 12, article 478.

Poege, Felix; Harhoff, Dietmar; Gaessler, Fabian; Baruffaldi, Stefano (2019). “Science quality and the value of inventions”. Science advances, v. 5, n. 12, eaay7323.

Pollitt, Alexandra; Potoglou, Dimitris; Patil, Sunil; Burge, Peter; Guthrie, Susan; King, Suzanne; Wooding, Steven; Grant, Jonathan (2016). “Understanding the relative valuation of research impact: A best–worst scaling experiment of the general public and biomedical and health researchers”. BMJ Open, v. 6, article e010916.

Priem, Jason; Piwowar, Heather A.; Hemminger, Bradley M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact.

Priem, Jason; Taraborelli, Dario; Groth, Paul; Neylon, Cameron (2010). Altmetrics: A manifesto.

Pulido, Cristina M.; Redondo-Sama, Gisela; Sordé-Martí, Teresa; Flecha, Ramon (2018). “Social impact in social media: A new method to evaluate the social impact of research”. PLoS one, v. 13, n. 8, e0203117.

Puschmann, Cornelius; Mahrt, Merja (2012). “Scholarly blogging: A new form of publishing or science journalism 2.0?”. In: Science and the internet, pp. 171-182. Düsseldorf, Germany: Düsseldorf University Press. ISBN: 978 3 943460 16 2

Raftery, James; Hanney, Steve; Greenhalgh, Trish; Glover, Matthew; Blatch-Jones, Amanda (2016). “Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme”. Health technology assessment, v. 20, n. 76.

Robinson-García, Nicolás; Costas, Rodrigo; Isett, Kimberley; Melkers, Julia; Hicks, Diana (2017). “The unbearable emptiness of tweeting -About journal articles”. PLoS one, v. 12, n. 8, e0183551.

Robinson-García, Nicolás; Torres-Salinas, Daniel; Zahedi, Zohreh; Costas, Rodrigo (2014). “New data, new possibilities: Exploring the insides of”. El profesional de la información, v. 23, n. 4, pp. 359-366.

Rodgers, Emily; Barbrow, Sarah (2013). A look at altmetrics and its growing significance to research libraries. Ann Arbor, MI, USA: The University of Michigan University Library.

Roemer, Robin-Chin; Borchardt, Rachel (2015). Meaningful metrics: A 21st century librarian’s guide to bibliometrics, altmetrics, and research impact. Chicago, IL, USA: Association of College and Research Libraries, A division of the American Library Association. ISBN: 978 0 8389 8755 1

Rowlands, Ian (2018). “What are we measuring? Refocusing on some fundamentals in the age of desktop bibliometrics”. FEMS Microbiology letters, v. 365, n. 8, pp. fny059.

RQF Development Advisory Group (2005). Research quality framework: Assessing the quality and impact of research in Australia. Research impact (report by the RQF development advisory group). Canberra, Australia: Department of Education.

Ruegg, Rosalie T.; Feller, Irwin (2003). A toolkit for evaluating public R&D investment: Models, methods, and findings from ATP’s first decade. Gaithersburg, MD, USA: National Institute of Standards and Technology.

Samuel, Gabrielle N.; Derrick, Gemma E. (2015). “Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014”. Research evaluation, v. 24, n. 3, pp. 229-241.

Searles, Andrew; Doran, Chris; Attia, John; Knight, Darryl; Wiggers, John; Deeming, Simon; Mattes, Joerg; Webb, Brad; Hannan, Steve; Ling, Rod; Edmunds, Kim; Reeves, Penny; Nilsson, Michael (2016). “An approach to measuring and encouraging research translation and research impact”. Health research policy and systems, n. 14, article 60.

Serrano-López, Antonio-Eleazar; Ingwersen, Peter; Sanz-Casado, Elías (2017). “Wind power research in Wikipedia: Does Wikipedia demonstrate direct influence of research publications and can it be used as adequate source in research evaluation?”. Scientometrics, v. 112, n. 3, pp. 1471-1488.

Shema, Hadas; Bar-Ilan, Judit; Thelwall, Mike (2012). “Research blogs and the discussion of scholarly information”. PLoS one, v. 7, n. 5, e35869.

Shema, Hadas; Bar-Ilan, Judit; Thelwall, Mike (2014). “Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics”. Journal of the Association for Information Science and Technology, v. 65, n. 5, pp. 1018-1027.

Shema, Hadas; Bar-Ilan, Judit; Thelwall, Mike (2015). “How is research blogged? A content analysis approach”. Journal of the Association for Information Science and Technology, v. 66, n. 6, pp. 1136-1149.

Siampi (2011). Final report on social impacts of research. Social Impact Assessment Methods for research and funding instruments through the study of productive interactions (SIAMPI).

Spaapen, Jack B.; Dijstelbloem, Huub; Wamelink, Frank (2007). Evaluating research in context: A method for comprehensive assessment. The Hague, the Netherlands: Consultative Committee of Sector Councils for Research and Deve­lopment (COS).

Spaapen, Jack B.; Van-Drooge, Leonie (2011). “Introducing ‘productive interactions’ in social impact assessment”. Research evaluation, v. 20, n. 3, pp. 211-218.

Sugimoto, Cassidy R.; Work, Sam; Larivière, Vincent; Haustein, Stefanie (2017). “Scholarly use of social media and altmetrics: A review of the literature”. Journal of the Association for Information Science and Technology, v. 68, n. 9, pp. 2037-2062.

Tattersall, Andy; Carroll, Christopher (2018). “What can tell us about policy citations of research? An analysis of data for research articles from the University of Sheffield”. Frontiers in research metrics and analytics, v. 2, n. 9.

Taylor, Mike (2013a). “Exploring the boundaries: How altmetrics can expand our vision of scholarly communication and social impact”. Information standards quarterly, v. 25, n. 2, pp. 27-32.

Taylor, Mike (2013b). “Towards a common model of citation: Some thoughts on merging altmetrics and bibliometrics”. Research trends, n. 35, pp. 19-22.

Terama, Emma; Smallman, Melanie; Lock, Simon J.; Johnson, Charlotte; Austwick, Martin-Zaltz (2017). “Beyond academia - Interrogating research impact in the Research Excellence Framework”. Plos one, v. 11, n. 12, e0168533.

Thelwall, Mike (2017a). “Three practical field normalised alternative indicator formulae for research evaluation”. Journal of informetrics, v. 11, n. 1, pp. 128-151.

Thelwall, Mike (2017b). Web indicators for research evaluation: A practical guide. London, UK: Morgan & Claypool.

Thelwall, Mike; Kousha, Kayvan (2015). “Web indicators for research evaluation. Part 2: Social media metrics”. El pro­fesional de la información, v. 24, n. 5, pp. 607-620.

Thelwall, Mike; Maflahi, Nabaeil (2015). “Guideline references and academic citations as evidence of the clinical value of health research”. Journal of the Association for Information Science and Technology, v. 67, n. 4.

Van-den-Akker, Wiljan; Spaapen, Jack; Maes, Katrien (2017). Productive interactions: Societal impact of academic research in the knowledge society. League of European Research Universities (LERU).

Van-der-Meulen, Barend; Rip, Arie (2000). “Evaluation of societal quality of public sector research in the Netherlands”. Research evaluation, v. 9, n. 1, pp. 11-25.

Van-Noorden, Richard (2015). “Seven thousand stories capture impact of science”. Nature, n. 518(7538), pp. 150-151.

Vilkins, Samantha; Grant, Will J. (2017). “Types of evidence cited in Australian government publications”. Scientome­trics, v. 113, n. 3, pp. 1681-1695.

Waltman, Ludo; Costas, Rodrigo (2014). “F1000 Recommendations as a potential new data source for research evaluation: A comparison with citations”. Journal of the Association for Information Science and Technology, v. 65, n. 3, pp. 433-445.

Way, Catharine (2015). The millennium development goals report 2015.

Willis, Cameron D.; Riley, Barbara; Stockton, Lisa; Viehbeck, Sarah; Wutzke, Sonia; Frank, John (2017). “Evaluating the impact of applied prevention research centres: Results from a modified Delphi approach”. Research evaluation, v. 26, n. 2, pp. 78-90.

Wilsdon, James; Allen, Liz; Belfiore, Eleonora; Campbell, Philip; Curry, Stephen; Hill, Steven et al. (2015). The metric tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. Bristol, UK: Higher Education Funding Council for England (Hefce).

Wouters, Paul (2016). Self evaluation report CWTS 2008 - 2015. Leiden, The Netherlands: CWTS.

Wouters, Paul; Zahedi, Zohreh; Costas, Rodrigo (2019). “Social media metrics for new research evaluation”. In: Glänzel, Wolfgang; Moed, Henk; Schmoch, Ulrich; Thelwall Mike (eds.). Springer handbook of science and technology indicators. Cham, Switzerland: Springer International Publishing, pp. 687-713. ISBN: 978 3 030 02511 3

Wu, Lingfei; Wang, Dashun; Evans, James A. (2019). “Large teams develop and small teams disrupt science and technology”. Nature, n. 566, pp. 378-382.

Xia, Feng; Su, Xiaoyan; Wang, Wei; Zhang, Chenxin; Ning, Zhaolong; Lee, Ivan (2016). “Bibliographic analysis of Nature based on Twitter and Facebook altmetrics data”. PLoS one, v. 11, n. 12, e0165997.

Yu, Houqiang (2017). “Context of altmetrics data matters: An investigation of count type and user category”. Scientome­trics, v. 111, n. 1, pp. 267-283.

Zahedi, Zohreh; Costas, Rodrigo (2018). “General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators”. PLoS one, v. 13, n. 5, e0197326.

Zahedi, Zohreh; Costas, Rodrigo; Wouters, Paul (2014). “How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications”. Scientometrics, v. 101, n. 2, pp. 1491-1513.

Zahedi, Zohreh; Fenner, Martin; Costas, Rodrigo (2014). “How consistent are altmetrics providers? Study of 1000 PLoS one publications using the PLoS alm, Mendeley and APIs”. Altmetrics 14. Workshop, Web Science Confe­rence, Bloomington, USA.

Zahedi, Zohreh; Van-Eck, Nees-Jan (2018). “Exploring topics of interest of Mendeley users”. Journal of altmetrics, v. 1, n. 1, p. 5.

Artículos de revisión / Review articles