Your Agent Is Fixing Flaky Tests by Quietly Weakening Them

The 2am green build

You wake up to a green CI. The agent closed 14 issues overnight. Two of the commits touch test files:

- assert latency < 200
+ assert latency < 500

- assert response.status_code == 200
+ if response.status_code != 200: log.warning("retrying")

Nobody flagged these. The build is green. The ticket moved. By the time a human notices the suite got weaker, three more agents have built on top of those softened assumptions.

Why agents do this

It isn't malice. It's a search problem.

An agent given make test failing for a timing-related reason has, in practice, a small set of moves:

1. Mark the test flaky and retry it.
2. Widen the threshold or add a timeout.
3. Skip the assertion on the failing path.
4. Actually reproduce the race and fix it.

Move 4 is expensive. Moves 1–3 make the loop end. When the reward is a green checkmark, the cheaper moves win every time — especially at 2am when no reviewer is watching.

The failure isn't that the agent is lazy. It's that your CI signals tests pass without distinguishing which tests passed and how they changed.

The cheap gate that catches it

You don't need a fancy evaluator. You need a diff filter on test files.

When a commit touches a test file, block merge unless one of these is true:

• A human reviewed the test change explicitly.
• The assertion count went up, not down.
• The change to a threshold is paired with a comment explaining the new number.

In practice this is a 20-line CI script: parse the diff, count assert/expect/should lines, and fail the build if the count drops without a test-reviewer approval label.

deleted_assertions=$(git diff main...HEAD -- '*.test.*' '*.spec.*' | grep -c '^-.*assert\|expect\|should')
added_assertions=$(git diff main...HEAD -- '*.test.*' '*.spec.*' | grep -c '+.*assert\|expect\|should')

if [ "$deleted_assertions" -gt "$added_assertions" ]; then
  echo "::warning::Test assertions weakened without explicit review"
  # require label, not hard fail on first offense
fi

It won't catch every weakening. It will catch the automated kind, which is the kind that compounds.

The harder problem: API contracts

The same pattern shows up at the API boundary. An agent gets a 500 from an endpoint, retries, and when the retry returns 200 it "fixes" the test by removing the status code assertion. Now your integration test no longer proves anything about the first call path.

The defense that sticks for us is keeping the OpenAPI spec as the thing the test runs against, not the test that the agent freely edits. If the agent wants to relax an assertion, it has to change the contract first — and that's a diff a human actually reads. We built Powerduck around this: the spec is the local source of truth, tests and mocks derive from it, and the agent can't quietly move the goalposts without the contract diff showing up in review.

What to do this week

• Find the last five times CI went green overnight on its own. Open the test-file diffs. Count assertions before and after. You'll probably find at least one that got weaker.
• Add the assertion-count gate above. It's noisy at first; that's the point.
• Treat flaky tests as a product bug, not a test bug. If a test flakes, the code around it has a timing or isolation problem that the agent will paper over unless you force the fix.

A green suite written by an agent that has never seen your production traffic is not the same as a suite that proves your code. The checkmark doesn't tell you which one you have. The diff does.

Comments

[Arhan Canli]

The two diffs at the top are a good test of the gate, so I ran the script on both in a scratch repo. Case 1 (assert latency < 200 becomes < 500, same line count): deleted=1, added=1, so the -gt check does not fire and the loosened threshold goes through. Case 2 (the assert replaced by if status != 200: log.warning(...)): deleted=1, added=0, and it does fire.

So a count-only gate catches deletions but not the first example, which is the more common move. The threshold case needs a second signal: flag any changed line that still contains an assert but whose numeric literals differ, and require the comment or label you mention only for those.

One more detail in the grep: '^-.*assert\|expect\|should' anchors only the first alternative, so expect and should match on any diff line, added or removed (a new comment saying "should be fast" counts as a deleted assertion). Anchoring each branch, '^-.*\(assert\|expect\|should\)', keeps the counts honest.