Humans in automated decision-making under the GDPR and AI Act
Keywords:
human oversight, General Data Protection Regulation (GDPR), artificial intelligence (AI), automated decision-making (ADM), accountability, European Unión (EU)Abstract
Revista CIDOB d’Afers Internacionals, nº 138, pp. 121-144
Quadrimestral (October-December 2024)
ISSN:1133-6595 | E-ISSN:2013-035X
DOI: https://doi.org/10.24241/rcai.2024.138.3.121
Human oversight is a fundamental safeguard against inappropriate judgments made by machines about people who are the targets of these decisions. Although the General Data Protection Regulation (GDPR) and the AI Act address human oversight to some extent, they fall short in addressing qualitative aspects and the integration of human overseers into governance frameworks. This paper examines the legal requirements for human oversight, investigating how these intersect with the accountability obligations of the automated decision-making (ADM) deployers and individual rights. It argues for a more comprehensive approach that not only includes human oversight, but also a continuous and rigorous assessment of the effectiveness of human control. Without that, human oversight may fail to protect adequately and could even worsen the impact on individuals affected by ADM.
>> The full text articles of this issue are available only in Spanish language
References
AEPD-Agencia Española de Protección de Datos. «Evaluating human intervention in automated decisions», (4 de marzo de 2024) (en línea) [Fecha de consulta: 01.09.2024] https://www.aepd.es/en/press-and-communications/blog/evaluating-human-intervention-in-automated-decisions
Alexander, Veronika: Blinder, Collin y Zak, Paul J. «Why trust an algorithm? Performance, cognition, and neurophysiology». Computers in Human Behavior, vol. 89, (2018), p. 279-288. DOI: https://doi.org/10.1016/j.chb.2018.07.026
Almada, Marco. «Human Intervention in Automated Decision-Making». ICAIL '19: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, (2019), p. 2-11. DOI: https://doi.org/10.1145/3322640.3326699
Almada, Marco. «Automated Decision-Making as a Data Protection Issue». SSRN, (2021), p. 1-23 (en línea) [Fecha de consulta: 01.09.2024] https://dx.doi.org/10.2139/ssrn.3817472
Bauer, Kevin; Hinz, Oliver; van der Aalst, Wil y Weinhardt, Christof. «Expl(AI)n It to Me – Explainable AI and Information Systems Research». Business and Information Systems Engineering, vol. 63, (2021), p. 79-82. DOI: https://doi.org/10.1007/s12599-021-00683-2
Berger, Armin; Hillebrand, Lars; Leonhard, David; Deußer, Tobias; Feliz de Oliveira, Thiago B. y Dilmaghani, Tim. «Towards Automated Regulatory Compliance Verification in Financial Auditing with Large Language Models». 2023 IEEE International Conference on Big Data (BigData), (15-18 de diciembre de 2023), p. 4.626-4.635. DOI: https://doi.org/10.1109/BigData59044.2023.10386518
Berger, Benedikt; Adam, Martin; Rühr, Alexander y Benlian, Alexander. «Watch Me Improve: Algorithm Aversion and Demonstrating the Ability to Learn». Business and Information Systems Engineering, vol. 63, (2021), p. 55-68. DOI: https://doi.org/10.1007/s12599-020-00678-5
Binns, Reuben y Veale, Michael. «Is that your final decision? Multi-stage profiling, selective effects, and Article 22 of the GDPR». International Data Privacy Law, vol. 11, n.º 4 (2021), p. 319-332. DOI: https://doi.org/10.1093/idpl/ipab020
Burton, Jason W.; Stein, Mari-Klara y Jensen, Tina Blegind. «A systematic review of algorithm aversion in augmented decision making». Journal of Behavioral Decision Making, vol. 33, n.º 2 (2019), p. 220-239. DOI: https://doi.org/10.1002/bdm.2155
Bygrave, Lee A. «Article 22: Automated individual decision-making, including profiling», en Kuner, Christopher; Bygrave, Lee A.; Docksey, Christopher y Dreachsler, Laura (eds.) The EU General Data Protection Regulation (GDPR) – A Commentary. Oxford: Oxford University Press, 2020, p. 531.
Custers, Bart y Heijne, Anne-Sophie. «The Right of Access in Automated Decision-Making: The Scope of Article 15(1)(h) GDPR in theory and practice». Computer Law and Security Review, vol. 46 (setembre de 2022) DOI:https://doi.org/10.1016/j.clsr.2022.105727
Christofi, Athena; Breuer, Jonas; Wauters, Ellen; Valcke, Peggy y Pierson, Jo. «Data protection, control and participation beyond consent - Seeking the views of data subjects in data protection impact assessments», en Kosta, Eleni; Leenes, Roland y Kamara, Irene (eds.) Research Handbook on EU Data Protection Law. Cheltenham: Edward Elgar, 2022, p. 503-529.
Cobbe, Jennifer, Seng Ah Lee, Michelle y Singh, Jatinder. «Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems». FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, (2021), p. 598-609. DOI: https://doi.org/10.1145/3442188.3445921
Comité Europeo de Protección de Datos. «Coordinated Enforcement Action, Designation and Position of Data Protection Officers», (17 de enero de 2024) (en línea) [Fecha de consulta: 01.09.2024] https://www.edpb.europa.eu/our-work-tools/our-documents/other/coordinated-enforcement-action-designation-and-position-data_en
Crootof, Rebecca; Kaminski, Margot E. y Price II, William Nicholson. «Humans in the Loop». Vanderbilt Law Review, vol. 76, n.º 2 (2023), p. 429-510.
Custers, Bart y Heijne, Anne-Sophie. «The Right of Access in Automated Decision-Making: The Scope of Article 15(1)(h) GDPR in theory and practice». Computer Law and Security Review, (2022) DOI: https://doi.org/10.1016/j.clsr.2022.105727
Dreyer, Stephan y Schulz, Wolfgang. «The General Data Protection Regulation and Automated Decision-making: Will it deliver?». Discussion Paper Ethics of Algorithms, n.º 5, Bertelsmann Stiftung, (2019) (en línea) DOI: https://doi.org/10.11586/2018018
Edwards, Lilian y Veale, Michael. «Enslaving the Algorithm: From a ‘Right to an Explanation’ to a ‘Right to Better Decisions’?». IEEE Security & Privacy, vol. 16, n.º 3 (2018), p. 46-54. DOI: https://doi.org/10.1109/MSP.2018.2701152
Enarsson, Therese; Enqvist, Lena y Naarttijärvi, Markus. «Approaching the human in the loop – legal perspectives on hybrid human/algorithmic decision-making in three contexts». Information & Communications Technology Law, vol. 31, n.º 1 (2021), p. 123-153. DOI: https://doi.org/10.1080/13600834.2021.1958860
Enqvist, Lena. «Human oversight in the EU artificial intelligence act: what, when and by whom?». Law, Innovation & Technology, vol. 15, n.º 2 (2023), p. 508-535. DOI: https://doi.org/10.1080/17579961.2023.2245683
Grant, David Gray; Behrends, Jeff y Basl, John. «What we owe to decision-subjects: beyond transparency and explanation in automated decision-making». Philosophical Studies, (2023). DOI: https://doi.org/10.1007/s11098-023-02013-6 (en línea) [Fecha de consulta: 01.09.2024] https://link.springer.com/article/10.1007/s11098-023-02013-6
Green, Ben. «The flaws of policies requiring human oversight of government algorithms». Computer Law & Security Review, vol. 45, (2022). DOI: https://doi.org/10.1016/j.clsr.2022.105681 (en línea) [Fecha de consulta: 01.09.2024] https://www.sciencedirect.com/science/article/pii/S0267364922000292?via%3Dihub
Kaminski, Margot E. y Malgieri, Gianclaudio. «Algorithmic impact assessments under the GDPR: Producing multi-layered explanations». International Data Privacy Law, vol. 11, n.º 2 (2021), p. 125-144. DOI: https://doi.org/10.1093/idpl/ipaa020
Kern, Christoph; Gerdon, Frederic; Bach, Ruben L.; Keusch, Florian y Kreuter, Frauke. «Humans versus machines: Who is perceived to decide fairer? Experimental evidence on attitudes toward automated decision-making». Patterns, vol. 3, n.º 10 (2022), p. 1-12. DOI: https://doi.org/10.1016/j.patter.2022.100591
Koivisto, Ida; Koulu, Riikka y Larsson, Stefan. «User accounts: How technological concepts permeate public law through the EU’s AI Act». Maastricht Journal of European and Comparative Law, vol. 0, n.º 0 (2024). DOI: https://doi.org/10.1177/1023263X241248469 [Fecha de consulta: 01.09.2024] https://journals.sagepub.com/doi/10.1177/1023263X241248469
Koulu, Riikka. «Proceduralizing control and discretion: Human oversight in artificial intelligence policy». Maastricht Journal of European and Comparative Law, vol. 27, n.º 6 (2020), p. 720-735. DOI: https://doi.org/10.1177/1023263X20978649
Kyriakou, Kyriakos y Otterbacher, Jahna. «In humans, we trust». Discover Artificial Intelligence, vol. 3, n.º 44 (2023), p. 1-18. DOI: https://doi.org/10.1007/s44163-023-00092-2
Lazcoz, Guillermo y De Hert, Paul. «Humans in the GDPR and AIA governance of automated and algorithmic systems. Essential pre-requisites against abdicating responsibilities». VUB Brussels Privacy Hub Working Paper, vol. 8, n.º 32 (2022), p. 1-28. DOI: https://doi.org/10.2139/ssrn.4016502
Logg, Jennifer M.; Minson, Julia A. y Moore Don A. «Algorithm appreciation: people prefer algorithmic to human judgment». Organizational Behavior & Human Decision Processes, vol. 151, (2019), p. 90-103. DOI: https://doi.org/10.1016/j.obhdp.2018.12.005
Lynskey, Orla. «Regulating for the Future: The Law’s Enforcement Deficit». Studies: An Irish Quarterly Review, vol. 112, n.º 445 (2023), p. 104-119. DOI: https://doi.org/10.1353/stu.2023.0007
Mahieu, René L. P. y Ausloos, Jef. «Recognising and Enabling the Collective Dimension of the GDPR and the Right of Access». Law Archive Papers, (29 de abril de 2020a), p. 3-38. DOI: https://doi.org/10.31228/osf.io/b5dwm
Mahieu, René L. P. y Ausloos, Jef. «Harnessing the collective potential of GDPR access rights: towards an ecology of transparency». Internet Policy Review, (6 de julio de 2020b) (en línea) [Fecha de acceso: 01.09.2024] https://policyreview.info/articles/news/harnessing-collective-potential-gdpr-access-rights-towards-ecology-transparency/1487
Malgieri, Gianclaudio. «‘Just’ Algorithms: Justification (Beyond Explanation) of Automated Decisions Under the General Data Protection Regulation». Law and Business, vol. 1, n.º 1 (2021), p. 16-28. DOI: https://doi.org/10.2478/law-2021-0003
Mantelero, Alexander. Beyond Data Human Rights, Ethical and Social Impact Assessment in AI. La Haya: T.M.C Asser Press, 2022.
Mendoza, Isak y Bygrave, Lee A. «The Right not to be Subject to Automated Decisions based on Profiling», en: Synodinou, Tatiana-Eleni; Jougleux, Philippe; Markou, Christina y Prastitou, Thalia (eds.) EU Internet Law: Regulation and Enforcement. Cham: Springer, 2017, p. 77-98.
Misuraca, Gianluca y van Noordt, Colin. «AI Watch: Artificial Intelligence in public services, EUR 30255 EN». Publications Office of the European Union, (2020). DOI: https://doi.org/10.2760/039619, JRC120399
Novelli, Claudio; Taddeo, Mariarosaria y Floridi, Luciano. «Accountability in artificial intelligence: what it is and how it works». AI & Society, vol. 39, (2024), p. 1.871-1.882. DOI: https://doi.org/10.1007/s00146-023-01635-y
Parlamento Europeo. «Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance)». Official Journal of the European Union, L series, 2024/1689, (12 de julio de 2024) (en línea) http://data.europa.eu/eli/reg/2024/1689/oj
Roig, Antoni. «Safeguards for the right not to be subject to a decision based solely on automated processing (Article 22 GDPR)». European Journal of Law and Technology, vol. 8, n.º 3 (2017), p. 1-17 (en línea) https://ejlt.org/index.php/ejlt/article/view/570
Roig, Antoni. Las garantías frente a las decisiones automatizadas. Del Reglamento General de Protección de Datos a la gobernanza algorítmica. Barcelona: J.M. Bosch, 2020.
Rovatsos, Michael; Mittelstadt, Brent y Koene, Ansgar. «Landscape Summary: Bias in Algorithmic Decision-Making: What is bias in algorithmic decision-making, how can we identify it, and how can we mitigate it?». UK Government, Research and analysis, (19 de julio de 2019) (en línea) [Fecha de consulta: 01.09.2024] https://www.gov.uk/government/publications/landscape-summaries-commissioned-by-the-centre-for-data-ethics-and-innovation
Sartor, Giovanni y Lagioia, Francesca. «The impact of the General Data Protection Regulation (GDPR) on artificial intelligence». Panel for the Future of Science and Technology (STOA), (25 de junio de 2020) (en línea) [Fecha de consulta: 01.09.2024] https://www.europarl.europa.eu/thinktank/en/document/EPRS_STU(2020)641530
Schmidt, Philipp; Biessmann, Felix y Teubner, Timm. «Transparency and trust in artificial intelligence systems». Journal of Decision Systems, vol. 29, n.º 4 (2020), p. 260-278. DOI: https://doi.org/10.1080/12460125.2020.1819094
Selbst, Andrew D. y Powles, Julia. «Meaningful information and the right to explanation». International Data Privacy Law, vol. 7, n.º 4 (2017), p. 233-242. DOI: https://doi.org/10.1093/idpl/ipx022
Sivan-Sevilla, Ido. «Varieties of enforcement strategies post-GDPR: a fuzzy-set qualitative comparative analysis (fsQCA) across data protection authorities». Journal of European Public Policy, vol. 31, n.º 2 (2024), p. 552-585. DOI: https://doi.org/10.1080/13501763.2022.2147578
Sterz, Sarah; Baum, Kevin; Biewer, Sebastian; Hermanns, Holger; Lauber-Rönsberg, Anne; Meinel, Philip y Langer, Markus. «On the quest for effectiveness in human oversight: Interdisciplinary perspectives». FAccT '24: Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, (2024), p. 2.495-2.507. DOI: https://doi.org/10.1145/3630106.3659051
Tsamados, Andreas; Floridi, Luciano y Taddeo, Mariarosaria. «Human control of AI systems: from supervision to teaming». AI Ethics, (2024). DOI: https://doi.org/10.1007/s43681-024-00489-4
Wagner, Ben. «Liable, but Not in Control? Ensuring Meaningful Human Agency in Automated Decision‐Making Systems». Policy and Internet, vol. 11, n.º 1 (2019), p. 104-122. DOI: https://doi.org/10.1002/poi3.198
Wieringa Maranke. «“Hey SyRI, tell me about algorithmic accountability”: Lessons from a landmark case». Data & Policy, vol. 5, (2023), p. 1-24. DOI: https://doi.org/10.1017/dap.2022.39
WP251, European Data Protection Board. «Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679», 17/EN WP251rev.01, (3 de octubre de 2017) (en línea) [Fecha de acceso: 01.09.2024] https://ec.europa.eu/newsroom/article29/items/612053/en
Yeung, Karen. «Algorithmic Regulation: A Critical Interrogation». Regulation and Governance, vol. 12, n.º 4 (2018), p. 505-523. DOI: https://doi.org/10.1111/rego.12158
