When AI is confidently wrong

When people worry about AI getting things wrong, they usually picture something obvious. The chatbot says something absurd. The summary is nonsense. The generated photo has six fingers. It is embarrassing, everyone shares a screenshot, and you switch the thing off.

That is not the failure to worry about. It announces itself, and anything that announces itself gets fixed.

The expensive failure is quiet. The output is fluent, specific, plausible and wrong. It arrives in exactly the same confident tone as everything else the system produces, and there is nothing on the surface to separate it from the correct output sitting next to it. Nobody laughs, because nobody notices. It goes to a customer, or into a report, or onto an invoice, and the first person to find out is someone you would rather had not.

I think this is the most under-priced risk when a business puts AI anywhere near its customers or its numbers, and I want to describe what it actually looks like rather than talk about it in the abstract.

The version that has nothing to do with AI

Start with an example from my own company that involves no AI at all, because it shows the shape of the problem more clearly than any chatbot story.

I run a trading-infrastructure business. Customers' accounts report their trading activity, and those reports roll up into a profit-and-loss figure people rely on. For a long period, that number was wrong.

Not obviously wrong. Not a blank screen or an error message. The figures were specific, they were formatted correctly, they moved in believable ways, and they sat in a system that had never given anyone a reason to distrust it. When I built a reconciliation process that compared records across platforms, it found $540,000 of phantom data spread across seven accounts, along with about $4,000 of straightforward accounting drift. Duplicate records, counted more than once, presented as fact.

There is no visual difference. You cannot spot them by being careful, or by being experienced, or by looking harder. The only way I found them was by building something that checked those numbers against an independent source of truth, in that case the broker, rather than trusting the system to grade its own homework.

That failure predates AI entirely. It is a very old problem. What is new is the speed and volume at which a language model can produce this exact category of output.

Why AI makes this worse rather than different

A traditional bug tends to be consistent. It breaks the same way every time, so it is discoverable: you find it once and you have found it forever.

A language model is not like that. It produces something different each time, most of it good, and the wrong output is not flagged as different in any way the system can see. It is generated with the same fluency as the right answer. It does not know it is wrong, so it cannot tell you.

This is why "we tested it and it worked" is such a weak assurance with these systems. You did not test it. You sampled it. And you sampled it at a moment when someone was paying attention, which is precisely the condition that will not hold once it is running unattended in your business.

What I did about it in my own business

When I built a system to audit my customer-facing emails, this was the problem I designed around. The pipeline read 126 emails and flagged compliance and consistency issues across them. Corrections went out to 38 active customers, with no failures.

The part that mattered was not the model. It was the shape of the thing. The system's job was to flag, not to send. It narrowed 126 emails down to a set worth looking at, and a human decided what actually reached a customer. The AI did the work that is tedious and mechanical, reading everything, comparing everything, never getting bored on email 94. The judgement about what left the building stayed with a person.

That is not a limitation I grudgingly accepted.

It is the design.

If I had let it send, the failure mode would not have been a rude email. It would have been a plausible email, correctly formatted, addressed to a real customer, saying something subtly untrue. I would have found out about it weeks later, from them.

What this means if you are running a small business

Large companies solve this with people. They have a compliance function, a QA team, an analyst whose job is to notice that a number moved oddly. Most businesses do not have any of that, and are not about to hire it.

So the check has to be designed in rather than staffed. Four things I would think about before putting AI anywhere it can affect a customer or a number:

• Decide what a wrong answer costs, before you deploy. Not in the abstract, for this specific task. An AI that drafts internal meeting notes and gets one wrong is an inconvenience. An AI that touches pricing, invoices, medical or legal wording, or anything a customer will act on is a different category of risk, and deserves a different level of checking. Most businesses apply the same casual standard to both.
• Put the check at the boundary. The moment that matters is where output leaves your control and reaches a customer, a ledger, or a decision. That is where a person should be, and it is the one place worth spending the human attention you have.
• Sample the output on a schedule, not when someone complains. A complaint is not a monitoring system; it is the absence of one. Read ten outputs a week yourself. It takes fifteen minutes and it is the cheapest insurance available to you.
• Anchor to something outside the system. My reconciliation only worked because it compared against the broker, not against another part of my own platform. If a system is the only witness to its own accuracy, you do not have a check, you have an assumption.

The question worth asking

There is a question I ask before any business I advise spends money on AI, and it is not "is this model any good?" It is:

When this is confidently wrong,

who catches it before a customer does?

If the answer is a name, and that person has both the time and the authority to stop something going out, you are in reasonable shape. If the answer is "well, we'd notice," you do not have a check. You have a hope. And the entire character of this failure mode is that it is specifically designed, by accident, to defeat noticing.

None of this is an argument against using AI. I use it across my own business, in production, on systems that customers pay for. It is an argument for being honest about which failure you are actually guarding against, because most people are guarding against the embarrassing one, and it is the invisible one that sends the invoice.

I build production AI systems, and I advise business owners on where AI fits and where it does not, in plain English, with no vendor pitch.

If you want the ten questions I ask before any business spends a penny on AI, they are here: the AI Reality Check.

And if you would rather just talk it through, including whether AI is worth it for your business at all, book a 15-minute call. No pitch: cal.eu/eaymon.