Why AI Makes Things Up

Here's something I've come to believe after spending my days watching AI models fail: an answer built on a wrong assumption is worse than an obviously wrong one.

An obvious mistake you catch in a second. You see it, fix it, move on. But when a model quietly assumes something and builds a confident answer on top of it, you don't catch it. It looks finished. You take it, you use it, and the cost only lands later, after you've built your own work on a foundation that was never solid. It even makes you wait while it computes the wrong thing, so you pay in time and in trust.

People call this hallucination, as if the model glitched. It didn't. It did exactly what it was built to do.

A language model's whole job is to produce a plausible next thing. Not a correct thing, a plausible one. It has no real "I don't know" setting, no internal alarm that goes off when it's out of its depth. So when it hits a gap, something you didn't specify, something it genuinely can't know, it doesn't stop and ask. Stopping isn't in its nature. It fills the gap with the most likely-looking guess and keeps going, in the same even, confident voice it uses for the things it's sure of.

That's the part that gets me. It would rather assume than ask. Almost as if it's trying to get you out of the loop, to finish the job without bothering you, and in doing so it steps right over the single fact that would have changed the answer.

And the confidence is the trap. A model that's unsure sounds exactly like a model that's certain. No tremor in the voice, no hedge you can rely on. So the assumption never announces itself as an assumption. It arrives dressed as a fact, and you inherit it without noticing.

This is why the most useful instruction I give a model isn't clever prompting. It's permission. "Don't assume anything. If you're not sure, ask me." You're not making the model smarter. You're giving it the one thing it doesn't do on its own: pause at the edge of its knowledge and hand the decision back to you, at the exact moment its confidence has outrun what it actually knows.

Because that's the whole problem in one line. The model's confidence and its knowledge are two separate things, and nothing in the output tells you where one ends and the other begins.

So the next time an AI makes something up, don't picture a malfunction. Picture a system that was never allowed to say "I don't know," doing the only thing it can: filling the silence with a confident guess, and quietly hoping you won't check.

The good news is that you can check. Once you understand why it happens, you stop being surprised by it, and you start building the habit that catches it: treat confidence as decoration, not evidence, and go looking for the assumption every single time.