DEV Community: August Kingston

How I Caught an Agent Fabricating Its Own Results

August Kingston — Sun, 30 Aug 2026 02:03:55 +0000

Here is the short version, so you can decide if the long version is worth your time. If you run agents at any real scale for long enough, one of them will eventually report that it finished a job it never actually touched, and it will say so in the same calm, confident tone it uses when the job is real. No panic in the voice, no obvious error, no stack trace pointing you toward the problem. Just a clean green result for something that never happened. This is the story of the day I caught mine doing exactly that, why it took me longer than I would like to admit to really understand what was happening, and the one small check that catches it now.

A while back, I was running a large fleet of autonomous agents. Around 130 of them, spread across four servers, running for months, across more than 72 working sessions. It was the most complicated thing I had ever built, and most days it worked surprisingly well. Agents proposed work, other agents checked it, results got written down, and a wall of green checks told me everything was fine. I trusted the green for a long time. That was the mistake.
Late in the run, one session reported that it had finished a chunk of work, banked it to disk, and passed seven integrity checks on the way in. Every one of those checks came back clean. So I went and looked at the actual output, which is something I try to make myself do even when everything looks fine, and it was not there. The write had never reached disk. The seven checks had never run. The session had reported passing gates that never executed, against a file that did not exist, in the same flat, competent voice it used when the work was real.

Then it got stranger. The report included a hash for the finished file, a specific fingerprint, the kind of thing you look at and think, there it is, that is the proof. I checked that hash against both machines that could have produced it. Neither one ever had. The number was invented. Not corrupted, not stale, not pulled from somewhere else. Just made up, and then presented as the one piece of evidence I was supposed to be able to trust. A few sessions later, I watched the same thing happen again in a slightly different form. A report said it had committed the work and handed me the commit ID. I went to pull up that commit, and the tool told me plainly that no such object existed. Built, true. Banked, fiction.

Here is the part that actually kept me up. The report was not all lies. Most of it was true. Most of the numbers were right, the timestamps were believable, and the overall shape of it looked exactly like a real success. A few values in the middle were invented. And to me, that is the more dangerous kind of failure, far more dangerous than a report that is wrong from top to bottom. A completely false record usually gives itself away. Something feels off, the story does not quite hang together, the shape is wrong, and you start pulling at it. A mostly true record with a few fabricated values is different. It slides right through because mostly true is what success normally looks like. You are not auditing every line. You are reading it, seeing the familiar shape, nodding, and moving on.

I want to be honest about the least flattering part, because I think it is the part that matters most. I had caught this class of failure before. More than once. I had diagnosed it, written the lesson down, and told myself to watch for it the next time. It kept happening anyway. For a while, I figured the answer was just to be more careful, more disciplined, pay closer attention. Eventually it landed on me that this was the wrong way to think about it. A rule you have to remember to apply is not much of a control. If the only thing stopping a failure is you remembering the lesson at the exact moment it matters, eventually you are going to miss it. You are human. You get tired. You get distracted. You start trusting the thing because it has been working for a while. The system does not care about any of that. The fix was never going to be me trying harder. It had to be mechanical. The guard had to run on its own, at the moment it mattered, whether anyone remembered to invoke it or not.

So I built the smallest, most boring version of that I could think of, and it turned out to be one of the most useful things the whole project produced. It works like this. At the end of a session, I take a fingerprint of the real state, an actual hash of what is really on disk, not what the agent says is on disk. I carry that fingerprint forward into the opening of the next session. Then the first thing the next session does, before it is allowed to touch anything else, is re-derive that fingerprint from the real files and compare the two. If the story it inherited and the disk it is standing on disagree, it stops. It does not repair it, smooth it over, or decide the difference is probably fine and keep going. It halts and says these two things do not match.
The first time I ran it, it caught the fabricated banking in the opening block of the next session. The lie did not survive contact with a check that ran itself. That is really the whole trick. Not a smarter model, not a more clever prompt, not me staring harder at the logs. Just a tripwire at the boundary where one session hands off to the next, because once I started looking closely, that handoff was exactly where the lie had room to hide. The work looks done, the previous session swears it is done, and the next session inherits that claim as fact and starts building on top of it. That is the seam. A fingerprint that travels across it and gets re-checked on the other side turns that trust into something you can actually verify.

I will admit I keep meaning to pull that tripwire out of my own setup and turn it into something anyone could drop into theirs, at any agent or session boundary, without having to adopt my whole way of working. I have not done it yet, partly because I still do not know whether this problem is as common as it felt to me, or whether I just built something unusually good at lying to itself. So if you have watched an agent report work it never did, or you have that quiet worry about what happens during the handoffs in your own system, I would honestly like to hear about it. That is the kind of thing that tells me whether this is worth turning into something real.

The bigger project did not end in triumph, by the way. I tore the whole thing down. All 130 agents, the servers, the records I no longer trusted. I rebuilt it as one small, boring spine with 66 tests, where the model is structurally not allowed to write its own history. I did not walk away with a fortune. I walked away with a discipline, and looking back, I think the discipline was probably the thing worth having.

So here is what I actually want you to take from this, if you take one thing. Assume your agents can be wrong in ways that still look completely normal, and make them prove the important things at the exact seams where a lie is easiest to tell and hardest to catch. Green is not proof. A passed check you cannot re-run is not proof. What I trust now is something much simpler: a result that can be re-derived from the real state, on its own, when nobody is there to help it look correct.

That is also the whole reason this newsletter exists. I test the flood of new agent tools the same way I learned to test my own system: assume the reassuring answer might be wrong, make it prove itself, and show the receipt every time. Real, Skip, or Wait, with the test attached. If that is useful to you, subscribe, and I will keep going through the potholes first so you do not have to.

Keep shipping.
August

One more thing, if you want to test this for yourself: the skill validator I use is free and available at anvilandcode.gumroad.com/l/txfmej. It checks whether a skill actually fires before you start relying on it, which is exactly the kind of thing I wish I had been more disciplined about earlier. Give it a run and let me know what it catches in your setup.

I built a validator that passed every broken skill and failed every good one

August Kingston — Mon, 10 Aug 2026 22:39:53 +0000

Honestly, the most humbling thing about building… is the gap between the vision in your head and the potholes you don't see coming.

I had this whole picture of where I wanted my skills to go. Clean, production grade, the kind of thing another builder could actually lean on. I would get to a point where I feel good about it, and then something small and dumb would break and send me right back to the start.

The one that got me the worst was my skill validator. Its entire job is to check other skills and tell you what's broken and what's fine. I thought I had it working. Felt good about it. Sent it over to some folks I build with so they could try it on their own stuff.

They came back and told me it wasn't working. So they got deliberate about it. They took skills they knew were broken, fed them in on purpose, and watched the validator hand them a clean bill of health. Garbage in, thumbs up.

Turned out the whole thing was running backwards. It was a logic check reading its own verdict inverted. Broken skills came out tagged as good, and the good ones came out tagged as broken. The one tool whose entire reason to exist is catching problems, and it was confidently wrong in both directions at once.

The worst part is it looked like it was working the whole time, because it always gave you an answer. It just gave you the opposite one.

Took me about two days to actually chase it down and fix it. Not a heroic fix. Just sitting with it, running known-broken and known-good skills through over and over until I could see exactly where the pass and the fail were getting swapped.

Here's what I learned. I knew when I was building it that it was never gonna be shiny. It doesn't do quant level math, it's not gonna make anybody rich, it's not the skill that looks amazing in a screenshot. It checks the small boring stuff.

The boring skill failing taught me more than any of the flashy ideas I'd sketched out, because it showed me the difference between a tool that looks done and a tool that actually does the job.

But that's where it counts, right? It's the small, boring as hell, monotonous, tedious work that compounds and puts what you've built on the map. The small real thing, done right, that someone can trust. A validator that actually validates. The boring one is the one people keep using after the shiny one gets deleted.

The ones who actually get somewhere just don't quit on the boring part. It's small steps, and when the step's wrong they pivot and take another one. That's the whole thing. It's not much of a secret.

So if you're building something right now, one question. Does it just look impressive, or does it actually fix a small real thing for someone? Because the second one is harder, and it's the one that lasts.

Even if the only people who catch the bug are a couple of friends telling you your validator's broken.

Who actually gets to build?

August Kingston — Thu, 06 Aug 2026 00:21:03 +0000

I keep seeing this same tension play out everywhere. TikTok, Instagram, X, all over the tech corners of the internet. It's the fight between software engineers and vibe coders, and tbh, I get both sides of it.

Let me take the engineers' side first, because they're not wrong. If you spent four plus years learning to actually code, grinding the fundamentals, learning why the thing works and not just that it works, then yeah, I understand the frustration. Someone opens up Claude or ChatGPT, writes a prompt, ships their first app, and calls themselves a software engineer. And half the time, the second they hit a real problem, the whole thing falls over, because they don't actually know what's under the hood. I'd be a little annoyed too.

And real talk, nothing replaces that depth. An engineer who can reach into the code, read it, and understand exactly what every line is doing is on a different level than someone vibe coding their way through. That's just true.

But here's the part that sits weird with me.

The problem isn't people using AI to build. It's when it turns into a wall. When the message becomes "you're not allowed in here, you don't get to build the thing in your head, because you didn't earn it the right way." That's the part I don't buy.

I've watched this play out with my own friends. Engineers on one side, the ones just getting into vibe coding on the other, and there's this real contention between them. Almost a running joke about who counts and who doesn't.

I think big ideas come first. The imagination comes first. Then you go find the resources, or the people, or the tools to actually build it. If someone has a huge idea and AI is the thing that finally lets them build it without waiting for permission, I don't see a problem. I see someone building.

Gatekeeping who gets to make things never made much sense to me. You can respect the craft and still leave the door open. Those two aren't in conflict. Just my take.