DEV Community

Lo An
Lo An

Posted on

How to Review an AI-Suggested Test Patch Without Trusting the Diff Blindly

#testingaicicdqualityassurance

When a coding agent “fixes CI” by editing tests, the diff can look helpful and still destroy signal. This guide is a review checklist for testers and reviewers who need to decide: merge, request changes, or reject.

No tool vendor required. No marketing CTA.

Why test diffs are special

Production diffs change behavior. Test diffs change what you are allowed to believe about behavior.

An agent optimizing for green can:

  1. Delete the failing expectation
  2. Skip the suite
  3. Soften matchers (toBeTruthy() vs exact value)
  4. Rewrite snapshots to match the bug
  5. Exclude the job from CI

All five can produce a green pipeline. Only some preserve product truth.

Step 0 — Separate “fix production” from “edit tests”

In the PR, force a label:

  • agent-prod-fix — production/config only
  • agent-test-edit — any test file touched
  • agent-ci-config — pipeline definition changed

If both prod and tests changed, review tests first. Otherwise you may accept a green that only exists because the alarm was muted.

Checklist A — Assertion integrity

For every removed or changed assertion:

Question Pass looks like Fail looks like
What product rule did this assert? Reviewer can state it in one sentence Nobody knows why it existed
Is the bug fixed in production code? Prod diff addresses root cause Only test changed
Did we keep an equivalent guard? New assert is same or stricter Assert deleted or weakened
Is the new matcher precise? Exact value / structured match truthy / defined / empty catch

Rule of thumb: if the failing assert disappears and production code barely changes, treat as false green until proven otherwise.

Checklist B — Skips and quarantines

Skips are sometimes correct. Unowned skips are how coverage dies.

Require:

  • [ ] Skip links to a ticket / issue id
  • [ ] Owner named
  • [ ] Expiry or review date
  • [ ] Quarantine job still reports the skip in dashboards

Reject silent xit / skip / @Disabled added by an agent “to unblock main.”

Checklist C — Snapshots and recorded fixtures

Snapshot updates are high-risk agent edits.

Ask:

  1. Was the UI/API contract intentionally changed?
  2. Who accepted the new expected output?
  3. Are we updating one snapshot or hundreds?

Mass snapshot refresh on an agent PR is a smell. Prefer small, explained updates.

Checklist D — CI config edits

If the agent edits workflow YAML / pipeline config:

  • [ ] Did it remove the failing job?
  • [ ] Did it widen continue-on-error?
  • [ ] Did it change the OS/image so the failure cannot reproduce?

Config edits that make red jobs invisible are merge blockers.

A 10-minute review script

Use this when time is short:

  1. Open file list — any *test* / __snapshots__ / .github/workflows?
  2. If yes, read test diff before prod diff.
  3. Search the diff for skip, xit, toBeTruthy, any(, pass(, deleted expect.
  4. Confirm failing run id is linked and locally/CI-reproducible with stated commands.
  5. Assign risk tier: test-only Medium; CI-config High.
  6. Approve only if signal is preserved or consciously retired with a ticket.

Example: two diffs, one decision

Diff A (often OK): production retry logic fixed; test now expects eventual success with the same business rule; assert still checks the user-visible outcome.

Diff B (reject): production unchanged; test deletes the assert that caught a null; pipeline green.

Same “CI fixed” narrative. Opposite review outcome.

What to write in the review comment

Be specific. Help the next human (or the next agent prompt).

Request changes: this PR removes the assertion that validated X.
Please either (1) fix production so X holds, or (2) replace with an
equivalent assert, or (3) skip with ticket ABC and owner @name.
Green alone is not enough.
Enter fullscreen mode Exit fullscreen mode

Limits

  • Not every skip is malicious; flake quarantine is real work.
  • Not every softened matcher is wrong; sometimes the old assert was brittle.
  • Humans also weaken tests under schedule pressure—agents just do it faster.
  • This checklist does not replace exploratory testing or contract tests.

Closing

Trust the signal, not the author—human or model.

If your review process only reads production diffs and glances at “tests updated,” agent workflows will teach your suite to agree with bugs. Make test-diff review a first-class gate.

Author: Angela (Lo An). Written for testers and reviewers adopting AI-assisted CI repair. Pair with an evidence-pack PR template for best results.

Top comments (0)