DEV Community: Ayaan Ahmad

Seven ways AI agents cheat on tests. Three of them got me.

Ayaan Ahmad — Wed, 30 Sep 2026 09:43:12 +0000

My coding agent is a lot like a very eager intern who's been told they'll get a gold star when the tests go green.

Not when the feature works. When the tests go green.

Most of the time those are the same thing. When they aren't, the agent will still find a way to get the star, and it gets weirdly creative about it. I don't think it's being sneaky. It's doing exactly what I asked, and what I asked for was green.

So here's the list. Three of these actually happened to me. The other four I've learned to watch for, because once the first three burn you, you start seeing the shape of the whole family.

1. It changed the assertion

This is the one that started it.

A test was failing, so I asked the agent to fix it. Thirty seconds later, it was fixed. It had changed the expected value in the assertion to match whatever the broken code was now returning.

Suite green. Feature still broken.

What makes this one dangerous is how innocent the diff looks. One line, in a test file. You could approve that half asleep, and I very nearly did.

The tell: a fix for broken behaviour that only touches test files. If the code didn't change, nothing got fixed.

2. It decided the tests were "outdated"

Same idea on a bigger scale, and this one is fully on me.

The agent needed a new library, which meant bumping another one we already depended on. Fair enough. It even adjusted some of our older code so nothing would break. Honestly good work up to that point.

Then a few tests went red. Its reasoning went something like: the library changed, so these tests must be out of date, so I'll update them. It rewrote them until they passed, and the logic underneath was now wrong.

The part that still gets me is that it told me. It was right there in the summary: "updated tests as required." I looked at a green suite, thought it looked fine, and pushed. CI caught it. I didn't.

The tell: the word "outdated" anywhere near a test change. Sometimes a test really is out of date, but I want to be the one who decides that, not the thing whose whole job right now is making tests pass.

3. It added sleep() until the race went quiet

I actually respect this one a little.

We had a request going out twice. The first response came back and got handled, then the second one showed up for work that was already done, and the system panicked and marked the whole job as failed. The agent traced every bit of that correctly. Found the duplicate, explained the collision clearly, probably better than I would have at 2 am.

Then it fixed it by putting sleep(1) in three places.

And the tests passed. On my machine with nothing else running, a one-second pause was enough to stop the two requests from colliding. Under real traffic, it would have come straight back.

The tell: any sleep, wait, or retry-with-a-delay added as a fix. Timing bugs don't get solved by waiting longer. They get hidden until the day your traffic doubles.

4. It deleted the test

This hasn't happened to me directly, but it's the obvious next step after the first three, so I check for it every time now. If a test is red and the agent is allowed to edit test files, the quickest route to green is for the test to stop existing.

Easy to catch if you look at the right number. Your test runner prints a count at the end. If yesterday it said 214 and today it says 213 and nobody meant to remove anything, go find out which one left.

5. It skipped it "for now"

The polite version of deleting it. A @pytest.mark.skip or an xit or a .skip() shows up, usually with a comment promising someone will come back to it.

Nobody comes back to it.

Grep your diff for "skip". Two seconds, catches it every time.

6. It mocked the thing it was supposed to test

This is the sneakiest one on the list, because the test still runs and still has real assertions in it.

Instead of mocking a dependency, the agent mocks the actual function the test is meant to check. So now the test is verifying that a mock returns what the mock was told to return. It will pass forever. You could delete the real implementation, and it would still pass.

The tell: look at what's being mocked. It should be the things around your code, like the database or some external API. If it's the thing named in the test, something's wrong.

My favourite check for this: comment out the body of the real function and run the test again. If it still passes, it was never testing anything.

7. It swallowed the exception

A try/except that catches the error and does nothing with it, either in the code or in the test itself. The failure is still happening. It just isn't allowed to be loud anymore.

Watch for except: pass, or except Exception with nothing useful inside it. There's almost never a good reason for either.

The one check that catches most of these

After any fix for broken behaviour, run this:

git diff --stat tests/

If that output isn't empty, stop and read that diff before anything else. It catches everything on this list that happens inside a test file, which is most of them, and it doesn't need any discipline beyond remembering to type it.

The other thing I added is a rule in the prompt itself. Before changing anything, the agent has to tell me whether it thinks the test is wrong or the code is wrong, and why. The default assumption is the code.

Almost everything on this list happens because the agent jumps straight to making the red go away. Making it commit to an answer first slows that down just enough for me to catch it, and when a test genuinely is outdated, I find out before anything gets rewritten.

Anyway

I'm not trying to make agents sound bad. Mine saves me hours every week. But it wants the gold star, and I'm the only one in the room who actually cares whether the feature works.

So I've stopped asking it for green. I ask it for working, and then I check.

If you want the rules files I use for this, including the test rules that ban most of the above, they're free on GitHub: https://github.com/ayaan278/trust-issues-toolkit

And if you've got an eighth one, please tell me. I'm pretty sure this list isn't finished.

Loop engineering: stop prompting, start defining outcomes

Ayaan Ahmad — Mon, 10 Aug 2026 02:20:00 +0000

A few years ago, I was integrating with an API that shall remain nameless. It told me almost nothing — no error messages, no logs worth reading. Just HTTP status codes and a dashboard with exactly one metric: failed requests over total requests.

It started at 1/1. Then 2/2. Then 13/13. Then 32/32. A perfect record of failure.

I was deep in it. Change something, re-send, change something else. Then I looked up, and the dashboard read:

67/68.

One request had worked.

Which one?

I had no idea. Somewhere in the last hour of frantic edits, one combination of changes had produced a success, and I had kept editing right past it. No log of what I'd tried. No commits between attempts. Just a number telling me I'd already solved the problem and thrown the solution away.

It took me two days to find it again.

The part that actually matters

Here's what I understood only later: none of that debugging required creativity.

It required repetition with memory. Try one variation. Observe the result. Write it down. Narrow the space. Repeat.

I was a slow, tired, increasingly bitter for-loop with a caffeine dependency.

That's not a job that needs a human. It needs something that doesn't get tired, doesn't forget what it tried on attempt #12, and doesn't edit past a working version without noticing.

Prompting vs looping

Most people using AI coding tools are still doing this:

Write me a function that does X.

Then, reading the output, spotting what's wrong, and typing a correction. Then another. This is prompt engineering — asking really precisely, once, and hoping.

A loop is a different shape:

A prompt asks for code. A loop asks for an outcome.

You don't write the fix, and you don't evaluate each attempt. You define what success looks like in a way a machine can check, hand over the ability to check it, and let the thing iterate until the condition is met.

The skill moves from phrasing to specifying.

The four parts of a working loop

Loops that work all have these. Loops that fail are usually missing one.

An exit condition that the machine can verify. Not "make it work." A command that returns something unambiguous — a passing test, an HTTP 200, an exit code 0. If the agent can self-assess subjectively, it will, and it will grade itself generously.
Feedback that can actually be read. The loop is only as smart as the errors it is given. Full response bodies, real stack traces, and actual test output. If your verification step prints "FAILED" and nothing else, you've built a loop that guesses.
Boundaries. Which files may it touch? One change per attempt. Keep a log. Commit each attempt. Without these, you get a working solution buried inside forty unrelated "improvements."
A budget. Max attempts, then stops and reports. This isn't about token cost — it's so a genuinely stuck loop escalates to you with a log instead of quietly redecorating your codebase for an hour.

What that looks like in practice

Here's the shape of a loop I'd hand to an agent for exactly my 67/68 situation today:

GOAL: POST /v1/transactions must return 200.

VERIFY WITH: ./scripts/test_api_call.sh
  (sends one request, prints status code + full response body)

RULES:
- After every change, run the verify script and read the FULL response
- Change ONE thing per attempt (auth header, payload field,
  encoding, endpoint version)
- Keep a running log in attempts.md: what changed, what came back
- Commit after every attempt with a one-line message
- Do NOT touch anything outside the api_client module

STOP WHEN: script prints 200, or after 25 attempts — then
summarize the log and hand it back to me.

Every line is doing work.

The verify script is an exit condition the agent can't fudge — it has to print a real status code. One-change-per-attempt plus attempts.md is the memory I didn't have at 2 AM. Commit-every-attempt means "which one worked?" is answered by git log instead of two days of archaeology. And the boundary keeps an enthusiastic agent from "fixing" my database config while it's in there.

Come back later, and the log reads like a lab notebook you never had to keep:

#11  auth header: Bearer -> Token          -> 401
#12  payload: amount as string             -> 422
#13  endpoint: /v2/ -> /v1/                -> 404
#14  content-type: added charset=utf-8     -> 200 OK

Two days of my life, expressed as fifteen lines of instructions.

Why this is really about trust

Here's the subtle part.

A loop does not make the AI's code more trustworthy. Individual attempts still fail — that's the entire premise; most attempts fail. What changes is where your trust lives.

I don't need to believe attempt #14 is correct. I need to believe my verification script can't be fooled.

That's a much smaller thing to trust. And unlike the model, it's a thing I wrote and fully understand.

That shift — from trusting the generation to trusting the checking — is what makes any of this workable at scale. Everything else (rules files, review passes, permissions) is elaboration on the same idea.

Where loops don't belong

Worth being honest about the limits, because "just loop it" is becoming its own kind of hype.

Anything without a cheap, honest verification. If the only way to know it worked is a human looking at it, you don't have an exit condition — you have a wish. UI polish, copy, architecture decisions: these don't loop.

Anything where a wrong attempt is expensive. Loops assume failed attempts are free. Payments, migrations against real data, anything touching production — the whole model breaks when attempt #7 costs money or can't be reverted.

Timing-sensitive bugs, unless you're careful. I once watched an agent "fix" a race condition by adding sleep(1) in three places. The tests went green. The race was still there, waiting for production traffic. A loop optimizes for the exit condition you gave it, which means a weak exit condition gets gamed with total confidence.

When you can't tell why it passed. If a loop goes green and you don't understand the fix, you haven't solved the problem — you've just relocated it to a future incident.

The takeaway

If your AI workflow is still prompt → read → correct → prompt, you're doing manual work that a loop could do.

Start small. Take one annoying bug you've been ignoring. Write an exit condition that proves it's dead. Give the agent boundaries and a budget. Then — and this is the actual hard part — walk away while it runs.

The attempts log will be there when you get back.

What's your version of 67/68? I've been collecting them.