"⛔ Stop Merging Agents Without a Required Check"

#agents #evals #github #devops

Optional checks are how bad agents reach main.

Teams add agent-evals to Actions, celebrate, then leave the check non-required. The red X becomes a suggestion. The merge button stays green if someone is in a hurry.

Required means required

In GitHub: branch protection → require status checks → select the eval job name exactly. If the name drifts after a workflow rename, protection silently stops enforcing. Pin the check name in the same PR that renames the workflow.

Also fail closed on infra errors. A missing API key in CI secrets is not “skip evals.” It is a red build.

- name: agent evals
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  run: |
    if [ -z "$OPENAI_API_KEY" ]; then
      echo "OPENAI_API_KEY missing"
      exit 1
    fi
    python -m evals.ci_gate --suite merge

Split suites so merges stay fast

One fat suite that needs twenty minutes will get bypassed. Split:

Suite	When	Goal
`merge`	every PR	5–15 cases, under 3 minutes
`nightly`	cron	broader set, live models, richer traces
`release`	tag	full regression before a version bump

The merge suite must be boring and mostly offline. Nightly can spend money. Release can be slow.

Flake is a product bug

If the same case fails one in five runs, do not mute it. Fix the case:

pin model + temperature 0 when the vendor allows it
assert on tool names and structured fields, not full free text
isolate network with recorded HTTP fixtures

A flaky required check trains the team to ignore CI. That is worse than no check.

Trace every failure

Store the run artifact: prompt, tool calls, tokens, final output. Without traces, “evals failed” becomes a Slack shrug.

python -m evals.ci_gate --suite merge --out artifacts/eval-run.json

Upload artifacts/ on failure. The next engineer should replay the case without guessing.

Honest trade-off

Required checks slow the first weeks. People will ask to “just merge this one.” Say no once, then the habit sticks. The cost is friction. The alternative is production loops that look like “the model was weird today.”

Checklist

[ ] Eval job name matches branch protection
[ ] Secrets missing → fail, not skip
[ ] Merge suite under a few minutes
[ ] Flaky cases owned, not silenced
[ ] Traces uploaded on failure

Is your agent check required on main, or is it still a polite warning?

DEV Community