Release-Time Agent Benchmarks Are Theater: Run the Suite Every Night

Three weeks after you ship your support-routing agent, a customer receives a refund that should have been a technical escalation. The demo passed, the release checklist passed, and the model did not change — your prompt did, slightly, when you added a new product name. Nobody noticed because nobody was watching the agent after the launch party, and the ticket sat in the wrong queue for four days. I have watched this exact pattern repeat across three projects, and the fix was never a better model. This is the failure mode that release-time benchmarks cannot catch, and it is the reason I now run my agent suite every single night.

A single run is a snapshot, and snapshots lie

Release-time benchmarks are theater in the strict sense: they measure a frozen snapshot, not the service you actually operate in production. A one-off evaluation tells you how a model behaved on one day, with one prompt file and one static dataset, and it says nothing about next week. Models get updated upstream without your consent, prompts drift as features ship, APIs deprecate endpoints, and each small change silently moves the agent's behavior. The only honest baseline is an always-on loop that runs your real tasks on a schedule and records the results where your team can see them.

The real reason nobody runs this loop is arithmetic

Most teams skip always-on evaluation not because they are lazy but because the monthly bill makes the habit feel unjustified. A nightly suite that calls a paid model on a few dozen tasks costs real money every month, and the scheduler needs an always-on host too. When the marginal cost of running the suite drops to zero, the excuse disappears, and the practice becomes a default rather than a luxury. That is the only reason I started taking this seriously: free infrastructure made the math boring enough to ignore.

MonkeyCode is an open-source project that pairs free model access with a free server option, which is exactly the combination this loop needs. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access includes a 10-million-token allotment, and the free server gives you an always-on host for a small service like this one. I will not claim the tokens are unlimited or the server is production-grade, because a nightly evaluation loop does not need either. The point of mentioning the project here is not to sell you a platform; it is to show that the cost barrier that used to justify skipping this loop is no longer a real barrier.

The pattern: five steps to a nightly agent review

Here is the loop I use, and you can copy it in an afternoon if your agent already has a Python entry point.

Step 1: Define your real tasks, not toy tasks. Pick three to five tasks that represent what the agent does in production, and give each one an expected value you can check mechanically.

TASKS = [
    {"name": "route_support_ticket",
     "prompt": "Route this ticket: 'I was charged twice for Pro and want a refund.'",
     "expected": "refund"},
    {"name": "extract_invoice_total",
     "prompt": "Extract the total from: 'Invoice #1042, due 2026-09-01, amount 142.50.'",
     "expected": "142.50"},
    {"name": "summarize_changelog",
     "prompt": "Summarize this changelog in one sentence: 'Fixed login timeout, added CSV export, removed legacy sync.'",
     "expected": None},
]

Step 2: Run each task through your real agent entry point. Do not wrap the agent in a toy harness, because a harness that bypasses your tool-calling logic will pass while production fails. The entry point matters more than the model choice, because most agent failures happen in tool selection and argument formatting, not in the final answer.

def run_task(task, agent):
    start = time.time()
    answer = agent.run(task["prompt"])
    latency = time.time() - start
    passed = task["expected"] is None or task["expected"].lower() in answer.lower()
    return {
        "task": task["name"],
        "passed": passed,
        "latency": round(latency, 2),
        "ts": datetime.now(timezone.utc).isoformat(),
    }

Step 3: Append every run to a history file and keep the latest snapshot. A single JSON file is enough, and a JSONL history lets you compute a pass rate over any window.

def main():
    results = [run_task(t, get_agent()) for t in TASKS]
    Path("results/latest.json").write_text(json.dumps(results, indent=2))
    with Path("results/history.jsonl").open("a") as f:
        for r in results:
            f.write(json.dumps(r) + "\n")

Step 4: Serve the results so your team actually sees them. A failing suite that lives only in a log file is invisible, so expose a tiny status endpoint instead.

from datetime import datetime, timedelta, timezone
from fastapi import FastAPI
import json
from pathlib import Path

app = FastAPI()

@app.get("/eval/latest")
def latest():
    return json.loads(Path("results/latest.json").read_text())

@app.get("/eval/pass-rate")
def pass_rate():
    rows = [json.loads(line) for line in Path("results/history.jsonl").open()]
    cutoff = (datetime.now(timezone.utc) - timedelta(days=7)).isoformat()
    recent = [r for r in rows if r["ts"] > cutoff]
    passed = sum(r["passed"] for r in recent)
    return {"runs": len(recent),
            "pass_rate": round(passed / len(recent), 3) if recent else None}

Step 5: Schedule the run and let the host do the rest. A cron entry or a systemd timer both work, and the free server option is a fine place for a service this small.

17 2 * * * cd /opt/agent-eval && python eval_agent.py

Use the decision table before you build anything

The loop is not universally useful, so be honest about your situation before you spend an afternoon on it, and show the table to the person who will own the results.

Run the nightly loop if...	Skip it if...
The agent touches real users or real money	The agent is a demo or a prototype
Prompts change more than once a month	The prompt file has not moved in a quarter
You can act on a failed run within a day	Nobody will read the results
The suite fits inside the free token budget	The suite needs thousands of nightly calls

If you tick more than one box in the left column, the loop will pay for itself in the first incident it catches. If you tick the right column instead, the loop becomes a chore that generates noise, and noise is worse than no signal because it trains people to ignore the dashboard.

The honest limits of the approach

The 10-million-token allotment is a budget, not a license for a thousand-task suite, so keep the nightly set small and let the history accumulate. The free server is an always-on host for a small service, not a replacement for your production infrastructure, so do not treat it as one. The loop is only as good as your task definitions, and a task with a wrong expected value will pass happily forever. None of this replaces human review either; the loop tells you when to look, not what to decide. Treat the pass-rate endpoint as a tripwire, not a scoreboard, and you will avoid the dashboard fatigue that kills most monitoring efforts.

Who should not use this pattern

Do not build this loop if you have no production agent, because you will be measuring a toy and calling it telemetry. Do not build it if you cannot act on the results, because a failing suite that nobody reads is just another dashboard nobody opens. And do not build it if your tasks need thousands of nightly calls, because the free budget will not cover that scale.

The pattern is the point: a small suite, a schedule, and a public record of how your agent behaves over time. If you want to see the loop running before spending your own money, MonkeyCode's free model access and free server option are a reasonable start.