Human involvement in autonomous decision-making systems. Lessons learned from three case studies in aviation, social care and road vehicles - Salvini, Pericle; Reinmund, Tyler; Hardin, Benjamin; Grieman, Keri; Ten Holter, Carolyn; Johnson, Aaron; Kunze, Lars; Winfield, Alan; Jirotka, Marina (Frontiers in Political Science, 2023)
This paper draws on three case studies to examine some of the challenges and tensions involved in the use of Autonomous Decision-Making Systems (ADMS). In particular, the paper highlights: i) challenges around the shifting ‘locale’ of the decision, and the associated consequences for stakeholders; ii) potential implications for stakeholders from regulation such as the General Data Protection Regulation (GDPR); iii) the different values that stakeholder groups bring to the ‘decision’ question; iv) how complex pre-existing webs of stakeholders and decision-making authorities may be disrupted or disempowered by the use of an automated system and the lack of evaluation of possible consequences; v) how ADMS for non-technical users can lead to circumvention of the boundaries of intended system use. We illustrate these challenges through case studies in three domains: adult social care, aviation, and vehicle driver monitoring systems. The paper closes with recommendations for both practice and policy in the deployment of ADMS.