El uso de la evaluación en un contexto de institucionalización débil: una aproximación desde el qualitative comparative analysis
DOI:
https://doi.org/10.21308/recp.62.04Palabras clave:
usos instrumentales, institucionalización de la evaluación, policymaking, cultura de la evaluación, Qualitative Comparative Analysis (QCA)Resumen
El auge de las políticas basadas en evidencias ha revivido el interés por los usos instrumentales de las evaluaciones. Sin embargo, la evidencia disponible sobre usos instrumentales se ha generado, en su mayoría, en países con una cultura de la evaluación consolidada, de forma que se dispone de poca información sobre cómo se alcanzan este tipo de usos en contextos donde la evaluación no está completamente institucionalizada. En esta investigación se utiliza un Qualitative Comparative Analysis para identificar cuáles son los factores que explican el uso instrumental de las evaluaciones realizadas por el Instituto Catalán de Evaluación de Políticas Públicas (Ivàlua) en Cataluña, un contexto de institucionalización de la evaluación todavía débil. Estos se contrastan con aquellos encontrados en otras investigaciones realizadas en entornos de institucionalización avanzada para identificar diferencias. Tras constatar la importancia explicativa de los tres tipos de factores recogidos en la literatura (metodológicos, institucionales y relativos al contexto político) y la naturaleza coyuntural del uso de la evidencia, los resultados confirman que los factores que influyen en el uso instrumental de las evaluaciones cambian según el grado de institucionalización de la evaluación. En entornos con una cultura de la evaluación menos desarrollada, cobran importancia factores explicativos orientados, justamente, a paliar esa falta de institucionalización.
Descargas
Citas
Albæk, Erik (1995). Between knowledge and power: Utilization of social science in public policy making. Policy Sciences, 28(1), 79-100. https://doi.org/10.1007/BF
Alkin, Marvin C. y Daillak, Richard H. (1979). A Study of Evaluation Utilization. Educational Evaluation and Policy Analysis, 1(4), 41-49. https://doi.org/10.3102/01623737001004041
Alkin, Marvin C. y Taut, Sandy M. (2002). Unbundling evaluation use. Studies in Educational Evaluation, 29(1), 1-12. https://doi.org/10.1016/S0191-491X(03)90001-0
Balthasar, Andreas (2006). The Effects of Institutional Design on the Utilization of Evaluation: Evidenced Using Qualitative Comparative Analysis (QCA). Evaluation, 12(3), 353-371. https://doi.org/10.1177/1356389006069139
Barberà, Oscar, Doria, Emilio, Ntutumu-Sanchís, Fernando y Sanchis, Pau (2020). La institucionalización de la evaluación de políticas públicas. La Comunitat Valenciana en perspectiva comparada. Publicacions de la Universitat de València.
Barbier, Jean-Claude y Hawkins, Penny (2012). Evaluation Cultures: Sense-making in Complex Times (N.o 2). Transaction Publishers. https://doi.org/10.4324/97813
Baron, Jon (2018). A Brief History of Evidence-Based Policy. The ANNALS of the American Academy of Political and Social Science, 678(1), 40-50. https://doi.org/10.1177/0002716218763128
Basurto, Xavier y Speer, Johanna (2012). Structuring the Calibration of Qualitative Data as Sets for Qualitative Comparative Analysis (QCA). Field Methods, 24(2), 155-174. https://doi.org/10.1177/1525822X11433998
Befani, Barbara (2016). Choosing Appropiate Evaluation Methods. A Tool for Assessment and Selection. Bond. https://www.bond.org.uk/resources/evaluation-methods-tool
Befani, Barbara, Barnett, Chris y Stern, Elliot (2014). Introduction. Rethinking Impact Evaluation for Development. IDS Bulletin, 45 (6). https://doi.org/10.1111/1759-5436.12108
Bustelo, María (2020). Spain. En Wolfgang Meyer, Reinhard Stockmann y Lena Taube, (ed.), The Institutionalisation of Evaluation in Europe, 303-327. Springer International Publishing. https://doi.org/10.1007/978-3-030-32284-7_12
Cairney, Paul y Oliver, Kathryn (2017). Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy? Health Research Policy and Systems, 15(1), 35. https://doi.org/10.1186/s12961-017-0192-x
Caplan, Nathan (1979). The Two-Communities Theory and Knowledge Utilization. American Behavioral Scientist, 22(3), 459-470. https://doi.org/10.1177/000276
Comas, Núria y Vílchez, David (2018). L’ús de les avaluacions d’Ivàlua (2008-2016). Una anàlisi de com les avaluacions enforteixen les polítiques públiques. Institut Català d’Avaluació de Polítiques Públiques (Ivàlua). https://ivalua.cat/sites/default/files/inline-files/13_12_2018_16_10_18_Informe_Us_avaluacions_1.pdf
Cousins, J. Bradley y Leithwood, Kenneth A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56(3), 331-364. https://doi.org/10.2307/1170319
Daviter, Falk (2015). The political use of knowledge in the policy process. Policy Sciences, 48, 491-505. https://doi.org/10.1007/s11077-015-9232-y
Deaton, Angus y Cartwright, Nancy (2017). Understanding and misunderstanding randomized controlled trials. Social Science & Medicine, 210. https://doi.org/10.1016/j.socscimed.2017.12.005
Derlien, Hans-Ulrich (1990). Genesis and Structure of Evaluation Efforts in Comparative Perspective. En Program evaluation and the management of government (Rist, R.C., pp. 147-175). Transaction Publishers. https://doi.org/10.4324/9780429338380-14
European Evaluation Society (2007). EES Statement: The importance of a methodologically diverse approach to impact evaluation - Specifically with respecto to development aid and development interventions. https://europeanevaluation.org/wp-content/uploads/2020/03/EES-Statement-on-methodological-diversity.pdf
Furubo, Jan-Eric y Sandahl, Rolf. (2002). Introduction. A diffusion perspective on Global Developments in Evaluation, en Jan-Eric Furubo, Ray Rist y Rolf Sandahl, (ed.), International Atlas of Evaluation, 1-23. Transaction Publishers.
Furubo, Jan-Eric, Rist, Ray y Sandahl, Rolf (2002). International Atlas of Evaluation. Transaction Publishers.
García Sánchez, Ester (2015). Sistemas de evaluación en España: Análisis y propuestas. Studia Politicae, 34, 117-134.
Henry, Gary y Mark, Melvin (2003). Beyond Use: Understanding Evaluation’s Influence on Attitudes and Actions. The American Journal of Evaluation, 24, 293-314. https://doi.org/10.1016/S1098-2140(03)00056-0
Hornby, P. y Perera, H. S. R. (2002). A development framework for promoting evidence-based policy action: Drawing on experiences in Sri Lanka. The International Journal of Health Planning and Management, 17(2), 165-183. https://doi.org/10.1002/hpm.660
Huberman, Michael y Gather, Monica (1991). De la recherche à la pratique: Éléments de base. Recherche & formation, 12(1), 174-176.
Jacob, Steve (2005). Institutionnaliser l’évaluation des politiques publiques: Études comparée des dispositifs en Belgique, en France, en Suisse et aux Pays-Bas. PIE Peter Lang. https://doi.org/10.4000/sdt.22564
Jacob, Steve, Speer, Sandra y Furubo, Jan-Eric (2015). The institutionalization of evaluation matters: Updating the International Atlas of Evaluation 10 years later. Evaluation, 21, 6-31. https://doi.org/10.1177/1356389014564248
Johnson, Kelli, Greenseid, Lija O., Toal, Stacie A., King, Jean A., Lawrenz, Frances y Volkov, Boris (2009). Research on Evaluation Use: A Review of the Empirical Literature From 1986 to 2005. American Journal of Evaluation, 30(3), 377-410. https://doi.org/10.1177/1098214009341660
Kirkhart, Karen E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 2000(88), 5-23. https://doi.org/10.1002/ev.1188
Knorr, Karin D. (1977). Policymakers’ Use of Social Science Knowledge: Symbolic or Instrumental?, en Carol Weiss, (ed.), Using Social Research in Public Policy Making, 165-182. Lexington: Lexington Books.
Lázaro, Blanca (2015). Estudio comparado sobre institucionalización de la evaluación en Europa y en América Latina. Colección Estudios N.o 15. Programa EUROsociAL.
Ledermann, Simone (2012). Exploring the Necessary Conditions for Evaluation Use in Program Change. American Journal of Evaluation, 33(2), 159-178. https://doi.org/10.1177/1098214011411573
Lehne, Richard y Fisk, Ddonald M. (1974). The Impact of Urban Policy Analysis. Urban Affairs Quarterly, 10(2), 115-138. https://doi.org/10.1177/107808747401000201
Lester, James. P. y Wilds, Leah J. (1990). The utilization of public policy analysis: A conceptual framework. Evaluation and Program Planning, 13(3), 313-319. https://doi.org/10.1016/0149-7189(90)90062-2
Mahoney, James. (2010). After KKV: The New Methodology of Qualitative Research’. World Politics 62(1): 120-147. https://doi.org/10.1017/S0043887109990220
Mark, Melvin y Henry, Gary (2004). The Mechanisms and Outcomes of Evaluation Influence. Evaluation, 10, 35-57. https://doi.org/10.1177/1356389004042326
Marra, Mita (2004). The Contribution of Evaluation to Socialization and Externalization of Tacit Knowledge. https://doi.org/10.1177/1356389004048278
Marsh, David D. y Glassick, Judith M. (1988). Knowledge Utilization in Evaluation Efforts: The Role of Recommendations. Knowledge, 9(3), 323-341. https://doi.org/10.1177/107554708800900301
Mead, Lawrence M. (2015). Only connect: Why government often ignores research. Policy Sciences, 48(2), 257-272. https://doi.org/10.1007/s11077-015-9216-y
Mello, Patrick (2021). Qualitative Comparative Analysis: An Introduction to Research Design and Application. Georgetown University Press.
Meyer, Wolfgang, Stockmann, Reinhard y Taube, Lena (2020). «The Institutionalisation of Evaluation Theoretical Background, Analytical Concept and Methods», en Reinard Stockmann, Wolfgang Meyer y Laszlo Szentmarjay, (ed.), The institutionalization of Evaluation in the Americas, 3-34. https://doi.org/10.1007/978-3-030-32284-7_1
Owen, John M. y McDonald, Diane E. (1999). Creating an Evaluation Culture in International Development Cooperation Agencies. Journal of International Cooperation in Education, 2(2), 41-53.
Parkhurst, Justin O. (2017). The politics of evidence: From evidence-based policy to the good governance of evidence. Routledge. https://doi.org/10.4324/9781315675008
Patton, Michael Q. (1997). Utilization-focused evaluation: The new century text. Sage Publications, Inc. https://doi.org/10.1007/978-94-010-0309-4_15
Patton, Michael Q., Grimes, Patricia S., Guthrie, Kathryn M., Brennan, Nancy J., French, Barbara D. y Blyth, Dale A. (1977). In Search of Impact: An Analysis of the Utilization of Federal Health Evaluation Research, en Carol Weiss, (ed.), Using social research in public policy making, 141-164. Lexington: Lexington Books.
Pattyn, Valérie y Bouterse, Marjolein (2020). Explaining use and non-use of policy evaluations in a mature evaluation setting. Humanities and Social Sciences Communications, 7(1), 1-9. https://doi.org/10.1057/s41599-020-00575-y
Preskill, Hallie, Zuckerman, Barbara y Matthews, Bonya (2003). An Exploratory Study of Process Use: Findings and Implications for Future Research. The American Journal of Evaluation, 24(4), 423-442. https://doi.org/10.1016/j.ameval.2003.09.001
Ragin, Charles C. (1989). The Comparative Method. Moving Beyond Qualitative and Quantitative Strategies. University of California Press.
Ragin, Charles C. y Fiss, Peer C. (2008). Net Effects Analysis Versus Configurational Analysis: An Empirical Demonstration. En Charles Ragin, (ed.), Redesigning Social Inquiry: Set Relations in Social Research, 190-212. University of Chicago Press.
Rein, Martin y White, Sheldon H. (1977). Policy Research: Belief and Doubt. Policy Analysis, 3(2), 239-271.
Rihoux, Benoit y Ragin, Charles C. (2008). Configurational Comparative Methods. Qualitative Comparative Analysis (QCA) and Related Techniques (Vol. 51). Sage Publications, Inc. https://doi.org/10.4135/9781452226569
Schneider, Carsten y Wagemann, Claudius (2012). Set-Theoretic Methods for the Social Sciences. Cambridge University Press. https://doi.org/10.1017/CBO9781139004244
Shulha, Lyn M. y Cousins, Bradley (1997). Evaluation Use: Theory, Research, and Practice Since 1986. Evaluation Practice, 18 (3), 195-208. https://doi.org/10.1177/
Stern, Elliot, Stame, Nicoletta, Mayne, John, Forss, Kim, Davies, Rick y Befani, Barbara (2012). Broadening the Rang of Designs and Methods for Impact Evaluation. https://www.oecd.org/derec/50399683.pdf
Stockmann, Reinhard, Meyer, Wolfgang y Taube, Lena (2020). The Institutionalization of Evaluation in Europe. Palgrave Macmillan Cham.
Viñas, Verónica (2009). The European Union’s Drive towards Public Policy Evaluation: The Case of Spain. Evaluation, 15(4), 459-472. https://doi.org/10.1177/135
Webber, David J. (1984). Political condicions motivating legislators’ use of information. Policy Studies Review, 4, 110-118. https://doi.org/10.1111/J.1541-1338.1984.TB00165.X
Weiss, Carol (1979). The many meanings of research utilization. Public Administration Review, 426-431. https://doi.org/10.2307/3109916
Weiss, Carol (1988). If Program Decisions Hinged Only on Information: A Response to Patton. Evaluation Practice, 9(3), 15-28. https://doi.org/10.1177/1098214088
Weiss, Carol, Murphy-Graham, Erin y Birkeland, Sarah (2005). An Alternate Route to Policy Influence: How Evaluations Affect D.A.R.E. American Journal of Evaluation - AM J EVAL, 26, 12-30. https://doi.org/10.1177/1098214004273337
Wilkins, Peter (2012). Cultures of Ombudsman and Audit Institutions: Effects on Their Evaluative Activities, en Jean-Claude Barbier y Penny Hawkins, (ed.), Evaluation Cultures: Sense-Making in Complex Times. New Bruswick: Transaction Publishers.
Williams, Kevin, Laat, Bastiaan D. y Stern, Elliot (2002). The Use of Evaluation in the Commission Services Final Report. European Commission, DG Budget.
Publicado
Cómo citar
Número
Sección
Licencia
Derechos de autor 2023 Carla Cordoncillo
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial-SinDerivadas 4.0.
