"🚫 Your Agent Demo Is Not a Release Gate"

The demo looked green. The merge still broke a customer workflow.

That is the whole story for most agent work I see in 2025–2026. A notebook shows “pass rate 92%.” A slide says “evals.” Nobody can point to the GitHub check that would have blocked the bad prompt.

What broke

We shipped a tool-calling agent behind a feature flag. The notebook suite used a frozen fixture set from two weeks earlier. Production traffic hit a new tool schema the same day. The agent retried, burned tokens, and wrote partial side effects.

The notebook never saw that schema. CI never asked.

The smallest CI gate that would have caught it

You do not need a research harness on day one. You need three checks that fail the PR:

1. Contract check — tool JSON schemas in the repo must match what the agent loads at runtime (hash or OpenAPI snapshot).
2. Golden path — 5–20 fixed user tasks with expected tool names and final answers. Deterministic fixtures only.
3. Budget check — max tokens / max steps per task. A loop that “eventually works” is a fail.

Wire them as one job:

# .github/workflows/agent-evals.yml
name: agent-evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-evals.txt
      - run: python -m evals.ci_gate --suite golden --max-tokens 4000

Keep the suite offline when you can. If a live model is required, pin the model id and store traces as artifacts.

How the gate looks in code

# evals/ci_gate.py
import json
import sys
from pathlib import Path

def load_suite(path):
    return json.loads(Path(path).read_text())

def run_case(case, agent):
    result = agent.run(case["input"], max_steps=case.get("max_steps", 8))
    ok_tools = result.tool_names == case["expect_tools"]
    ok_answer = case["expect_substr"] in (result.final or "")
    ok_budget = result.tokens <= case["max_tokens"]
    return ok_tools and ok_answer and ok_budget, result

def main():
    suite = load_suite("evals/golden.json")
    failed = []
    for case in suite["cases"]:
        ok, result = run_case(case, build_agent())
        if not ok:
            failed.append((case["id"], result.summary()))
    if failed:
        for cid, summary in failed:
            print(f"FAIL {cid}: {summary}")
        sys.exit(1)
    print(f"PASS {len(suite['cases'])} cases")

if __name__ == "__main__":
    main()

golden.json is boring on purpose: ids, inputs, expected tool sequence, substring checks, token caps. Boring is what merge gates need.

Honest trade-off

A CI gate will reject useful prompt experiments that still need exploration. That is the point. Exploration stays on a branch or a notebook. Mainline gets the harness.

You will also spend time babysitting fixtures when tools change. That ownership cost is real. Paying it in CI is cheaper than paying it in incident reviews.

What to do Monday

1. Pick 10 real failures from the last month. Turn them into golden cases.
2. Add schema hash check for every tool the agent can call.
3. Make the workflow required on the PR.

If the check is optional, it is still a demo.

Has your agent stack blocked a merge yet, or is “evals” still a slide?