In May 2023, a lawyer named Peter LoDuca submitted a legal brief to a federal court in New York. The brief cited six court cases. None of those cases existed. They were fabricated by a chatbot, and the lawyer — a practicing attorney, not a first-year law student — had not checked.
The opposing side noticed. The judge, P. Kevin Castel, asked LoDuca to produce the cases. LoDuca asked ChatGPT whether the cases were real. ChatGPT said yes. The judge called the brief “unprecedented.” He fined the lawyers $5,000, dismissed the case, and required them to take legal education on AI use.
That was three and a half years ago. It was supposed to be the cautionary tale that made everyone careful.
It wasn’t.
The problem didn’t go away. It multiplied.
A Bloomberg Law analysis, cited in a 2026 overview by the Illinois courts system, found that since 2023, more than 280 court filings have included hallucinated citations generated by AI tools. In 2025 alone, the number of such cases surged sevenfold. A database maintained by French lawyer and data scientist Damien Charlotin has cataloged over 490 global instances of AI hallucinations in court cases, including fake citations.
This is not a story about bad lawyers. The lawyers in Mata v. Avianca were not frauds. They were competent professionals who used a tool the way most people use tools: they typed a prompt, got an answer that sounded right, and moved on. The tool was confident. The answer was plausible. The cases looked real — complete with case names, citation formats, and even a real judge’s name attached to one of them.
That’s the thing about AI-generated errors. They don’t look like errors. They look like work.
The draft was never the hard part
Here’s what most people get wrong about AI and professional work: they assume the hard part is producing the answer. It isn’t. The hard part was never the draft. The hard part is the check.
For decades, the skill that separated a competent professional from an incompetent one was the ability to produce a good first draft. A lawyer who could write a clean brief. A journalist who could turn a messy interview into a readable story. An accountant who could build a model that made sense. The draft was the product. The check was a formality.
AI changes that equation. Now anyone can produce a plausible-sounding first draft in seconds. The draft is no longer the scarce resource. The check is. And most people, most organizations, and most institutions have not adjusted. They are still running the old workflow — produce, review lightly, ship — on a problem that now requires a fundamentally different workflow.
The lawyer in Mata didn’t need to be smarter. He needed to be more suspicious. The answer the tool gave him was not the answer. It was a draft. And a draft is not a fact until someone with access to the actual record has verified it against that record.
The accountability gap
Here’s where it gets uncomfortable, because the problem isn’t just about lawyers. It’s about everyone who uses a tool that can produce a confident-sounding answer and then acts on that answer without checking.
A marketing team uses AI to draft a press release. The numbers in the release are plausible. Nobody checks them against the actual data. The release goes out. A client sees a figure that doesn’t match the contract. The conversation that follows is not about the AI. It’s about the team that published it.
A student uses AI to draft a research paper. The citations look right. The professor asks for the sources. The student can’t find them, because they don’t exist. The student’s grade takes the hit, not the tool’s.
A small business owner uses AI to draft a contract clause. It sounds reasonable. The other party’s lawyer reads it and spots a liability exposure that would have been obvious to a human reviewer with a law degree. The business owner signs it. The AI is not in the room when the dispute arises. The business owner is.
The pattern is the same in every case: the tool produces a plausible answer, the human trusts the plausibility, and the consequences land on the human. The AI is not accountable. It has no license to lose, no reputation to damage, no malpractice insurance to pay out. The person who publishes, the organization that relies on the output, and the reader who trusts it all inherit the risk. That distribution has changed, and most people haven’t adjusted.
This is not a job-elimination argument. AI is not going to replace the lawyer, the journalist, or the accountant. What it is going to replace is the unexamined first draft. And the professionals who survive the transition are not the ones who use AI the most. They’re the ones who know how to check it.
The skill that actually matters now
The core competency has shifted. It used to be “can you produce a good draft?” Now it’s “can you check one?”
That sounds simple. It isn’t. Checking an AI-generated answer is not the same as checking a human-written draft, because the failure mode is different. A human draft has errors of reasoning, of judgment, of care. You can usually tell where the human got confused, because the confusion has a shape. An AI draft has errors of confidence. The wrong answer sounds exactly as authoritative as the right one. The fake case citation looks exactly like a real one. The plausible number looks exactly like the actual number.
So the checking skill is not “read it carefully.” It’s “go to the source.” For a legal citation, that means pulling the actual case from the court’s database and confirming it exists and says what the brief claims it says. For a statistic, it means finding the original dataset or the original report and confirming the number is there. For a quote, it means finding the person who said it and confirming they said it in that context.
This takes longer than trusting the draft. It always has. The difference is that now it’s not optional. The cost of skipping it is no longer a bad grade or an awkward conversation. It’s a dismissed case, a retracted paper, a contract that exposes you to liability you didn’t know you had.
What to do this week
You don’t need to become a prompt engineer. You don’t need to build an AI workflow. You don’t need to decide whether AI is a threat or a gift. You need to do one small thing.
Pick one recurring task where you currently trust a first draft. It could be a report you write every month. A summary you send to a client. A research note you prepare for a meeting. A social media post you draft before publishing. Whatever it is, take the next one you produce and actually check it. Not skim it. Check it. Go to the source. Verify the numbers. Confirm the citations. Read the thing you’re quoting and make sure it says what you think it says.
Then notice what you find.
You will find something. Not every time, but often enough. And the thing you find will be more useful than any prediction about what AI will or won’t do to your career, because it will be a concrete example of the gap between a plausible answer and a verified one. That gap is where the risk lives. And it’s the only part of the problem you can actually control.
The model isn’t the problem. The missing check is.