Why the Machine Makes Things Up

In 2023, a New York lawyer submitted a federal court filing that cited more than half a dozen judicial opinions supporting his client's position. The citations were formatted impeccably. The quotes were apt. The cases did not exist. He had asked a chatbot for research, the chatbot had obliged with inventions, and when challenged, he had asked the chatbot whether the cases were real. It assured him they were. The judge sanctioned the lawyers involved, and the episode entered the permanent folklore of this technology.

It is tempting to file that story under user error, and the judge largely did. But the more useful filing is under product behavior, because what happened to that lawyer is the single most important operational fact about language models, and it will happen, at some scale, inside your business too. The question is whether it happens inside a process designed for it.

Plausibility Is the Product

Recall what the machine actually does: it produces the most plausible continuation of the text in front of it. Read that sentence as an engineer would and the so-called hallucination problem dissolves into something more clarifying. The machine is doing its job. Plausibility and truth overlap most of the time, because the model's map was drawn from mostly true writing about a mostly consistent world. But when the map has a gap, where the facts are sparse, obscure, or simply absent, the prediction process does the only thing it can do. It fills the gap with whatever the patterns say should be there. A case name shaped exactly like real case names. A statistic with the right number of decimal places. A biography assembled from the lives of similar people.

There is no internal alarm that distinguishes the filled gap from the genuine recall, which is why the tone never changes. The confidence is constant; only the grounding varies. And the failure concentrates precisely where you would least want it: on specifics. Names, numbers, dates, citations, prices, the load-bearing details of business life. Ask for the general shape of contract law and you will do fine. Ask for the exact clause, the exact case, the exact figure, and you have entered the territory where the map runs out and the pattern fills in.

Now, the question every owner asks next: will the next version fix it? The honest answer is that the rates fall and the property remains. Newer models hallucinate measurably less, decline to answer more gracefully, and improve dramatically when connected to real documents and live search, a technique covered later in this series. But a system whose core operation is plausible continuation will always be capable of producing plausible fiction. Waiting for the version that never makes things up is like waiting for the table saw that cannot cut you. The professionals stopped waiting and built the guard.

Building the Guard

Here is what the guard looks like in practice, drawn from systems that survive contact with real operations.

Sort your work by stakes before anything else. Brainstorming, internal drafts, and summaries you will personally review tolerate errors cheaply. Anything with a number, a name, a legal claim, or a customer's trust attached gets a verification step. The sorting takes one meeting and prevents most disasters.

Ground the machine in your documents. When the model is made to answer from your actual contracts, policies, and records, rather than from its general map, fabrication drops sharply. Most modern business tools work this way for exactly this reason.

Demand sources, then spot-check one. Asking the model to cite where each claim came from improves its behavior, and checking a single citation at random tells you quickly whether the rest deserve trust.

Keep a person accountable for anything that leaves the building. The lawyer's failure was never that he used the tool; it was that no human owned the output before a judge read it. Every external document needs a name attached, and the name needs to have read it.

Treat each caught fabrication as data. Log where it happened. Patterns emerge fast, usually clustered around specific tasks, and those are the tasks that get redesigned or retired.

None of this is exotic. It is quality control, the discipline owners already apply to every supplier whose work varies. You inspect the lumber. You proof the print run. The machine joined the supply chain; it gets inspected like the rest of it.

The Honest Frame

I will give you the sentence I use with clients on day one, because the framing does more work than any checklist. The machine is a brilliant colleague who never says “I don’t know.” Everything else follows from that one trait. You would happily work with such a colleague, would profit enormously from them, and would never, ever file their research unread.

What separates the businesses that get burned from the ones that compound the gains has nothing to do with which model they bought. It is whether they treated fluency as a finished product or as a first draft awaiting a signature. The fluency belongs to the machine. The signature, and everything it implies, stays with you.