Chapter 14Check before you trust
Fluent isn’t the same as true
In this chapter Why text that reads well can still be wrong, and how to check what an AI tool gives you.
Words to know
- Hallucination
- When a model states something false or made up as if it were true. NIST calls it confabulation.
- Automation bias
- The habit of trusting an automated aid too much, and missing its mistakes.
A model writes what sounds right, not what it has checked. Checking is your job, and the hard part is noticing what needs it.
- It sounds the same when it’s guessing. A model predicts likely text (Chapter 2), so a made-up detail comes out as smoothly as a real one. Researchers argue that training and testing reward guessing over admitting uncertainty.1 NIST, the U.S. standards agency, lists “confabulation” among the risks of generative AI: “confidently stated but erroneous or false content.”2
- People trust machines too much. Long before chatbots, a review of research on people working with automated decision aids found that when the aid is imperfect, people miss its mistakes or follow its wrong advice, experts as well as beginners. Researchers call it automation bias.3
- The cost can be real. In 2023 a federal court in New York sanctioned two lawyers and their firm, $5,000, after they filed “non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT” and stood by them when the court asked.4
Go deeper: made-up citations
In a 2023 study, ChatGPT wrote 84 short literature reviews, and the researchers checked the 636 works they cited: 55 percent of the citations from GPT-3.5 and 18 percent from GPT-4 were to works that didn’t exist.5 Those figures are for 2023 models. The lesson lasts: a made-up citation is formatted just like a real one, so open it.
The court was careful about what it punished. It wrote that “there is nothing inherently improper about using a reliable artificial intelligence tool for assistance”; the problem was filing fake opinions and standing by them after the court questioned them.4
Notes (5)
- 1Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang, “Why Language Models Hallucinate” (preprint, arXiv, September 4, 2025), https://doi.org/10.48550/arXiv.2509.04664. Cited at abstract; p. 1.
- 2National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) (NIST, July 26, 2024), https://doi.org/10.6028/NIST.AI.600-1. Cited at p. 4.
- 3Raja Parasuraman and Dietrich H. Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration,” Human Factors 52, no. 3 (June 2010): 381–410, https://doi.org/10.1177/0018720810376055. Cited at p. 381.
- 4U.S. District Court for the Southern District of New York (P. Kevin Castel, U.S. District Judge), “Mata v. Avianca, Inc., No. 22-cv-1461 (PKC): Opinion and Order on Sanctions (ECF No. 54),” No. 22-cv-1461 (PKC), ECF No. 54, June 22, 2023, https://www.courtlistener.com/docket/63107798/mata-v-avianca-inc/ (accessed October 27, 2026). Cited at pp. 1; page 34; p. 1.
- 5William H. Walters and Esther Isabelle Wilder, “Fabrication and Errors in the Bibliographic Citations Generated by ChatGPT,” Scientific Reports 13, no. 1 (September 7, 2023): 14045, https://doi.org/10.1038/s41598-023-41032-5. Cited at abstract; p. 1.
Chapter 14 · page 1 of 2
Turn pages with the ← and → keys.