My AI Invented a Trade-In Program That Never Existed

A few months ago, before I shut it down, I had AI running the first pass of my outbound research. Hand it search criteria, it comes back with the angle on each company: what they sell, where they are exposed, why they would take my call. One of my criteria was that the company already ran a trade-in program. One morning it handed me a clean, well-sourced brief on a company I was about to email cold, and there it was: they already ran a trade-in program.

I still had a rule to trust but verify, so I went and looked. The company had no trade-in program. None. The model had simply guessed one into existence, because companies like that usually have one.

So what my system got wrong was the exact thing that made the prospect worth emailing. That was the reason the company was on my list at all, not some detail it fumbled on the way to the answer. And had I trusted it, I would have opened a cold email to a stranger by congratulating him on a program that does not exist. One sentence in, and I am the guy who did not do his homework. Credibility gone before the second line.

And there is a specific kind of wrong that only a machine can be: confident, well-formatted, and completely made up. The tone never wavers. It sounds exactly the same when it is right as when it is inventing. You've heard the name, and probably felt it at some point too: AI hallucinations!

Why waiting on the model is not a plan

Most people meet that unreliability in one of two ways, and both are a mistake. One camp treats AI like magic and waits for it to become flawless; the other calls it a toy and waits for it to embarrass itself. Obviously neither camp is stupid, but both are just waiting on the model, and the model is not coming to rescue either of them, because the thing they are waiting on is not a bug that gets patched in the next release. A large language model is a probabilistic guesser by design: it predicts the next likely word from patterns it has seen, it does not look up a rule and run it. A better model guesses better, but it does not stop guessing. Not only that, but LLMs are also trained to find a solution, to be people pleasers. Put another way, hallucinations are here to stay.

So "try harder" or "wait for the next version" is the same useless instruction as telling an employee to just do a better job next time. You don't fix an unreliable step by asking it nicely to be reliable, so you build a check around it that catches the miss before it reaches anyone. That is ordinary operations, pointed at the most hyped tool of the year. In AI speak, we call this a harness.

Retrieval, then plain code

Two specific and concurrent builds help solve this issue; just picking one isn't enough. The first is retrieval, which means handing the model the actual source text instead of letting it reach into fuzzy memory. Mine is a folder of 361 plain text files and growing: people, companies, deals, every newsletter issue and what it argued. Every agent I run reads this wiki before it does anything else. It builds as a side effect of the work rather than as a maintenance chore, so a detail that comes up twice in a week ends up with its own page. That lowers the odds of a bad guess, but it does not completely eliminate them.

Which is why the second feature probably matters even more, and it is the one almost nobody seems to build: plain (python) code sitting between the model and me. There's no AI involved here, which is the entire point. The rules I've implemented here are deliberately blunt. For example, the system is not allowed to tell me a job failed unless it actually opened that job's log and read it. Another rule is to refuse to let it report a task finished until it has gone back and confirmed that the file, the draft, or the record it claims to have created actually exists. Granted, none of this is clever. We are talking about just a few dozen lines of ordinary code, yet it catches more than I expected: those checks have now stopped 136 confident answers on their way to me. Just that 2nd rule above triggered ten times since I added it last month, such as when the system was about to tell me something in my business had completely broken without having read a single log first.

Here's another example of how this works, on a warehouse floor this time. In a returns or ITAD flow, the record does not close unless the wipe certificate's serial number matches the one scanned at intake. That is a deterministic gate sitting on top of the wipe standard. Here, the model can scan the certificate, summarize the batch, and flag the exceptions - but it does not get to decide on its own that a device was actually wiped until some independent code confirms that the job ran properly and both product IDs match.

As you'll have guessed by now, the reliability comes from the system wrapped around the model (the harness), never from the AI model itself. The model is probabilistic and it stays probabilistic. "I built a system around the model" and "the model can be trusted now" are not the same sentence, and confusing them is how you congratulate a prospect on a successful trade-in program they never implemented.

For context, I have been using AI hard for about a year and a half, but I only started building a proper harness in March, once it became clear to me that the guessing was not going to stop on its own. Five months, no engineering team, and nobody to catch my mistakes but me (which is its own kind of quality control). That is why I can tell you, candidly, that an operator can build this, and why I trust the system now in a way I will never trust the model.

Where to build the first check

So underneath the AI vocabulary, this is the same execution gap sitting under everything I write: the distance between "an answer was produced" and "the answer is right," closed by a check that fires on its own. Do not start by asking which model to buy, but where a confident wrong answer would cost you the most (a customer email, a payout figure, a device you swore was wiped), and build the check there first. The model will keep guessing. Your job is to make sure the guess never reaches the customer unchecked.