Models carry old patterns

There is a second reason a person must stay responsible: a tool can carry unfairness that its user cannot see. A language model learns from text people wrote, and that text carries the patterns of the past, fair and unfair. NIST lists “harmful bias” as one of twelve risks of generative AI, describing it as the “amplification and exacerbation of historical, societal, and systemic biases.”¹

A 2024 study in the journal Nature showed how hidden such patterns can be. Valentin Hofmann and three colleagues gave language models statements written either in African American English or in Standardized American English, in one part of the study the same statement in each dialect. They never mentioned race, and asked the models to judge the speakers.² What the models said directly about African Americans was more positive. What they assumed about a speaker from dialect alone was more negative.²

The researchers also staged hypothetical trials in which the only evidence was a statement from the defendant. The models chose conviction more often when it was written in African American English: 68.7 percent of the time, against 62.1 percent. In a first-degree murder case, overall, they chose the death penalty more often too: 27.7 percent against 22.8 percent.² These were experiments, not real trials. They were run on the models available at the time, including versions of GPT-3.5 and GPT-4, and newer models may score differently.²

Two lessons follow. First, you can’t check a tool for bias by asking it whether it is biased, because what a model says about a group and how it treats people’s words can differ. The study found that training models on human feedback, a common fix, hid the racism on the surface while the stereotypes underneath remained.² Second, the detectors and the dialect study point the same way. Both judged people by how they write.

One such complaint about hiring is before a U.S. court. In February 2023 a job applicant, Derek Mobley, sued Workday,³ whose platform, his complaint says, employers use to collect and screen job applications.⁴ He alleged that its AI-based recommendation system discriminated against applicants by race, age and disability. In May 2025 the court let his age claim go forward as a collective action, so that other applicants aged 40 and over could be notified and join. The court called that ruling preliminary, and it is not a finding that Workday discriminated.⁴ As of October 2026 the case was still pending, with a hearing on class certification set for March 2027.³ Chapter 15 gathers more about bias and about AI detectors.

Where lawmakers have drawn lines

Some governments have turned these concerns into rules. The European Union’s AI Act, a regulation of June 2024, sorts uses of AI by risk. It treats as “high-risk” AI used to recruit or select people, to make decisions about promotion or firing, to assign work, or to monitor and evaluate workers. It does the same for AI used to decide who is admitted to schools and to evaluate learning outcomes.⁵ High-risk systems carry extra requirements, including the human oversight described above, and the Act allows some exceptions.⁵

Europe’s data-protection law, the GDPR, adopted in 2016, already reached part of the way. It gives people a right, with exceptions, not to be subject to a decision “based solely on automated processing” that has legal or similarly significant effects on them.⁶

As of October 2026, the dates are still moving. The AI Act’s rules for these high-risk uses were first set to apply from August 2026.⁵ An amendment adopted in July 2026 moved that date to December 2027.⁷ Laws differ from country to country and change often, so take this as the idea, not legal advice. Spot it today: find out whether the rules where you work cover AI used in hiring, grading or health care, and what they ask of the person who signs off.