Don't Trust the Answer

An AI I was using added up twenty numbers for me and got it wrong. Then I asked again, and it gave the same wrong total, just as confidently. Only when I handed it the correct number did it recompute, agree, and apologize.

Summing twenty numbers is supposed to be the easiest thing a computer does. That's the part that stuck with me. Not that it failed at something hard, but that it failed at something trivial, and never once sounded unsure.

I spend a lot of my time evaluating AI output, judging whether a model's answer is actually right. And the thing I've learned is that trusting AI has almost nothing to do with how good the answer looks. Here's what I actually watch for.

The first red flag is the clean, direct answer.

When I ask a model for something and it hands back a single result, the number, the finished document, the done thing, with no working shown, that's exactly when I trust it least. A bare answer gives you nothing to check. It looks finished, and that's the trap: finished-looking is not the same as correct, and models are very good at looking finished.

So the first thing I do is refuse the bare answer. I ask it to show me the steps it took to get there. Not because I'll read every line, but because the reasoning is where the mistake lives, and you can't catch a mistake you can't see.

For anything with real steps, I go one step at a time.

If a task takes ten or twenty steps, I make the model do one step, then check with me before moving on. Here's why that matters more than it sounds. If a twenty-step solution goes wrong at step four, every step after it is built on the wrong thing. It doesn't matter that steps five through twenty are each done perfectly, they're perfectly extending a mistake. You end up with a confident, detailed, completely wrong answer and a pile of wasted work to show for it.

Catching the wrong turn at step four costs seconds. Catching it at the end costs the whole thing. Going step by step feels slower. It's actually faster, because you never build twenty floors on a cracked foundation.

Then I forbid assumptions, out loud, every time.

Because they will assume. Models are in a hurry to hand you a solution, and when something is missing or ambiguous, they don't stop and ask, they guess, and then they treat the guess as fact. I don't think the model is asking itself whether its answer is really a solution. It's producing something that looks like one. The assumption feels true to it, so it never flags it, and you inherit a wrong answer wearing the face of a right one.

"Don't assume anything, ask me if you're not sure" is the most useful instruction I give. It turns silent guesses into questions, which is exactly the trade you want.

Which brings me back to the twenty numbers.

The reason that one rattled me isn't that models are bad. Most days they're genuinely great. It's that the failure showed up in the last place I'd look. We brace for AI to be wrong about hard, fuzzy things. We don't brace for it to be wrong about arithmetic. But that's exactly where the dangerous errors hide, in the small, obvious, "surely it got this right" places you stop checking. The tone is identical whether it's right or wrong, so the tone tells you nothing.

So the honest answer to "when can you trust an AI's answer" is: not when it looks right. You trust it when you can see how it got there, when you've watched it work in steps small enough to catch a wrong turn, and when you've made it surface its assumptions instead of burying them.

Trust the process you can see, not the answer you were handed. The model's job is to produce something. Deciding whether that something is actually right is still yours.