Some calls are yours

AI can help with almost any task, but some decisions should stay with a person: ones about people’s jobs, grades, health or rights, where someone has to answer for the outcome and explain it.

1. An old warning. In 1976 Joseph Weizenbaum, who wrote ELIZA (Chapter 4), argued that some decisions should never be left to computers, whatever they can be made to do.¹
2. “A person checks it” is harder than it sounds. People working with automated aids can over-trust them and miss their mistakes: automation bias (Chapter 14).² A person who only rubber-stamps a ranking hasn’t really decided.
3. Models carry old patterns. They learn from text people wrote, and NIST lists harmful bias among the risks of generative AI: the “amplification and exacerbation of historical, societal, and systemic biases.”³
4. Lawmakers have drawn some lines. The European Union’s AI Act treats AI used to recruit, promote, fire, assign or monitor workers, and to admit students or evaluate their learning, as “high-risk,” with extra requirements; one is that such systems be built so people can effectively oversee them and stay aware of automation bias.⁴ Europe’s data-protection law, the GDPR, gives people a right, with exceptions, not to be subject to decisions based solely on automated processing that significantly affect them.⁵