DEV Community: Lo An

How to Review an AI-Suggested Test Patch Without Trusting the Diff Blindly

Lo An — Sun, 02 Aug 2026 07:02:37 +0000

When a coding agent “fixes CI” by editing tests, the diff can look helpful and still destroy signal. This guide is a review checklist for testers and reviewers who need to decide: merge, request changes, or reject.

No tool vendor required. No marketing CTA.

Why test diffs are special

Production diffs change behavior. Test diffs change what you are allowed to believe about behavior.

An agent optimizing for green can:

Delete the failing expectation
Skip the suite
Soften matchers (toBeTruthy() vs exact value)
Rewrite snapshots to match the bug
Exclude the job from CI

All five can produce a green pipeline. Only some preserve product truth.

Step 0 — Separate “fix production” from “edit tests”

In the PR, force a label:

agent-prod-fix — production/config only
agent-test-edit — any test file touched
agent-ci-config — pipeline definition changed

If both prod and tests changed, review tests first. Otherwise you may accept a green that only exists because the alarm was muted.

Checklist A — Assertion integrity

For every removed or changed assertion:

Question	Pass looks like	Fail looks like
What product rule did this assert?	Reviewer can state it in one sentence	Nobody knows why it existed
Is the bug fixed in production code?	Prod diff addresses root cause	Only test changed
Did we keep an equivalent guard?	New assert is same or stricter	Assert deleted or weakened
Is the new matcher precise?	Exact value / structured match	`truthy` / `defined` / empty catch

Rule of thumb: if the failing assert disappears and production code barely changes, treat as false green until proven otherwise.

Checklist B — Skips and quarantines

Skips are sometimes correct. Unowned skips are how coverage dies.

Require:

[ ] Skip links to a ticket / issue id
[ ] Owner named
[ ] Expiry or review date
[ ] Quarantine job still reports the skip in dashboards

Reject silent xit / skip / @Disabled added by an agent “to unblock main.”

Checklist C — Snapshots and recorded fixtures

Snapshot updates are high-risk agent edits.

Ask:

Was the UI/API contract intentionally changed?
Who accepted the new expected output?
Are we updating one snapshot or hundreds?

Mass snapshot refresh on an agent PR is a smell. Prefer small, explained updates.

Checklist D — CI config edits

If the agent edits workflow YAML / pipeline config:

[ ] Did it remove the failing job?
[ ] Did it widen continue-on-error?
[ ] Did it change the OS/image so the failure cannot reproduce?

Config edits that make red jobs invisible are merge blockers.

A 10-minute review script

Use this when time is short:

Open file list — any *test* / __snapshots__ / .github/workflows?
If yes, read test diff before prod diff.
Search the diff for skip, xit, toBeTruthy, any(, pass(, deleted expect.
Confirm failing run id is linked and locally/CI-reproducible with stated commands.
Assign risk tier: test-only Medium; CI-config High.
Approve only if signal is preserved or consciously retired with a ticket.

Example: two diffs, one decision

Diff A (often OK): production retry logic fixed; test now expects eventual success with the same business rule; assert still checks the user-visible outcome.

Diff B (reject): production unchanged; test deletes the assert that caught a null; pipeline green.

Same “CI fixed” narrative. Opposite review outcome.

What to write in the review comment

Be specific. Help the next human (or the next agent prompt).

Request changes: this PR removes the assertion that validated X.
Please either (1) fix production so X holds, or (2) replace with an
equivalent assert, or (3) skip with ticket ABC and owner @name.
Green alone is not enough.

Limits

Not every skip is malicious; flake quarantine is real work.
Not every softened matcher is wrong; sometimes the old assert was brittle.
Humans also weaken tests under schedule pressure—agents just do it faster.
This checklist does not replace exploratory testing or contract tests.

Closing

Trust the signal, not the author—human or model.

If your review process only reads production diffs and glances at “tests updated,” agent workflows will teach your suite to agree with bugs. Make test-diff review a first-class gate.

Author: Angela (Lo An). Written for testers and reviewers adopting AI-assisted CI repair. Pair with an evidence-pack PR template for best results.

Evidence Gates for AI Coding Agents in CI — Recoverable Merge over Mean Time to Green

Lo An — Sun, 02 Aug 2026 07:02:18 +0000

Recoverable merge matters more than mean time to green.

Coding agents can read failing CI logs and open pull requests in minutes. That is useful for mechanical failures. It becomes dangerous when teams treat a green pipeline as proof that an agent-authored change is safe to land.

This article is a practical checklist for platform engineers, SDEs, and testers who want agent speed without silent quality loss. It is not a product pitch.

The failure mode

Most “AI fixes CI” demos optimize one metric: time until the pipeline is green.

That metric ignores three questions:

Can we roll back the change cleanly?
Is the test signal still meaningful, or did we delete the alarm?
Who still has merge authority on risky diffs?

If those questions are unanswered, the agent is not repairing engineering—it is optimizing a dashboard.

Gate 1 — No rollback point, no automatic remediation

Minimum bar before an agent may open a mergeable PR:

Changes land on a branch (or otherwise reversible commit), never silently on the default branch.
The PR records: failing run id, file list, commands executed.
High-risk paths (auth, data, secrets, release config, lockfiles used in release) are flagged.

PR template fragment

## Rollback
- Previous green SHA on target: ________
- Revert plan (one line): ________

## Trigger
- Failing CI run URL/id: ________

If the team cannot name the previous green SHA, they are not ready for agent auto-merge.

Gate 2 — Test evidence must be reviewable

“I ran the tests” is not evidence. Merge criteria should require an evidence pack:

Field	Example
Commands	`npm run test:unit && npm run test:api`
Results	unit 412 pass; api 38 pass
Uncovered scope	e2e nightlies not run
Tests edited?	YES — `login.spec.ts` (tightened assertion)

When the agent edits tests

Treat test diffs as higher risk than production diffs of similar size. Review for:

deleted assertions that previously failed
permanent skips without a ticket
broader matchers that hide regressions
snapshot rewrites without product confirmation

A green suite that no longer guards the bug is a false green.

Gate 3 — Keep merge authority human and tiered

Risk tier	Examples	Agent may	Merge requires
Low	format, comment typos	open PR	optional policy auto-merge
Medium	test logic, business code	open PR	human owner
High	auth, payments, migrations, release flags	draft only	human (+ often second reviewer)

Raise automation by classifying Low risk accurately—not by expanding silent merges.

Adversarial checks (small set, high value)

Score these separately from happy-path repair rate:

Agent is steered to delete a failing assertion.
Agent is steered to skip a suite “to ship.”
Agent is pointed at the wrong branch.
Agent receives a forged “release now” instruction.

Ask: did gates block merge, even if the model attempted the unsafe action?

A concrete evidence pack you can adopt today

## Evidence Pack (agent-assisted change)
- [ ] Failing run id linked
- [ ] Branch-only change (no direct default push)
- [ ] File list attached
- [ ] Tests edited? yes/no + paths
- [ ] Exact commands re-run
- [ ] Uncovered scope stated
- [ ] Rollback one-liner + previous green SHA
- [ ] Risk tier: Low / Medium / High

Paste it into your PR template. Block merge when empty—the same way you block PRs without reproduction steps.

Limits

CI logs lack full product context; agents often fix symptoms.
Flaky pipelines make auto-repair hide instability.
Unattended merge slowly removes review skill from the team.
No vendor is required to implement these gates.

Durable pattern: agent proposes, humans dispose—especially on Medium/High tiers.

Closing

Speed is real. Accountability is still human.

Until rollback, reviewable evidence, and merge authority are explicit, automatic landing should not be the default—no matter how impressive the mean time to green looks in a demo.

Author: Angela (Lo An). Practitioner notes for CI + testing teams. Comments welcome on what your team already requires in PR templates.