DEV Community

Sam Rivera
Sam Rivera

Posted on

Build Fail-Closed Evidence Gates Before the Agent Runs

#ai #python #cli #productivity

The agent invented a timeout. I never set one.

It picked thirty seconds because that felt standard. My cron job then overlapped itself. Have you watched a helpful default chew a queue?

I needed a stop, not another prompt.

Goal: block the model call until evidence exists. Budget: forty minutes, free-tier tokens, no extra hosts. Abandon the approach if a gate needs more than a file check.

What a gate is

A gate is not a checkbox in a wiki. It is a file on disk, or the run dies.

No evidence? No model call. Why would I feed a guess?

I want fail-closed behavior. Missing proof means exit code 2. The agent does not get a vote.

This is the opposite of a chatty helper. The helper already assumed retry_count. I paid for that assumption.

The five gates

Copy these. Rename them if your task differs.

  1. Schema gate. evidence/input.schema.json must exist and parse.
  2. Required-fields gate. evidence/required_fields.txt lists keys the agent may not invent.
  3. Bound gate. evidence/cost_bound.txt states a token or time cap.
  4. Rollback gate. evidence/rollback.sh must be present and executable.
  5. Failure-fixture gate. evidence/failure.json shows one bad output you already caught.

If any file is missing, skip the model. Would you ship a patch with no fixture?

Scaffold in five commands

I keep this under a single directory. Nothing fancy. No cluster.

mkdir -p evidence
printf '%s\n' '{"type":"object","required":["task_id","timeout_ms"],"additionalProperties":false}' > evidence/input.schema.json
printf '%s\n' 'task_id' 'timeout_ms' > evidence/required_fields.txt
printf '%s\n' 'max_tokens=4000' 'max_seconds=20' > evidence/cost_bound.txt
printf '%s\n' '#!/bin/sh' 'git checkout -- .' 'rm -f /tmp/agent-out.json' > evidence/rollback.sh
chmod +x evidence/rollback.sh
Enter fullscreen mode Exit fullscreen mode

That is the whole contract. Can you see a gate without a file? I cannot.

One failure fixture

This is the exact shape that burned me. The agent filled a field I never declared.

{
  "task_id": "job-442",
  "timeout_ms": 30000,
  "retry_count": 3
}
Enter fullscreen mode Exit fullscreen mode

Save it as evidence/failure.json. That extra retry_count is the bug.

The schema set additionalProperties to false. The agent still added a retry. Do you trust a second prompt to forget that habit?

I do not. The gate file has to refuse the output.

The checker

This script is the artifact. It never calls a model. It only decides whether a call is allowed.

#!/usr/bin/env python3
"""Fail-closed gates for one-task agent output."""
from __future__ import annotations

import json
import sys
from pathlib import Path

EVIDENCE = Path("evidence")
REQUIRED_FILES = [
    EVIDENCE / "input.schema.json",
    EVIDENCE / "required_fields.txt",
    EVIDENCE / "cost_bound.txt",
    EVIDENCE / "rollback.sh",
    EVIDENCE / "failure.json",
]


def die(msg: str, code: int = 2) -> None:
    print(f"GATE FAIL: {msg}", file=sys.stderr)
    raise SystemExit(code)


def load_schema() -> dict:
    raw = (EVIDENCE / "input.schema.json").read_text()
    try:
        schema = json.loads(raw)
    except json.JSONDecodeError as exc:
        die(f"schema is not JSON: {exc}")
    if schema.get("additionalProperties") is not False:
        die("schema must set additionalProperties to false")
    if "required" not in schema:
        die("schema missing required keys")
    return schema


def load_required() -> list[str]:
    keys = [
        line.strip()
        for line in (EVIDENCE / "required_fields.txt").read_text().splitlines()
        if line.strip()
    ]
    if not keys:
        die("required_fields.txt is empty")
    return keys


def check_bound() -> None:
    text = (EVIDENCE / "cost_bound.txt").read_text().strip()
    if "max_tokens=" not in text or "max_seconds=" not in text:
        die("cost_bound.txt needs max_tokens and max_seconds")


def check_rollback() -> None:
    path = EVIDENCE / "rollback.sh"
    if not path.stat().st_mode & 0o111:
        die("rollback.sh is not executable")


def check_failure_fixture(required: list[str]) -> None:
    fixture = json.loads((EVIDENCE / "failure.json").read_text())
    extra = [k for k in fixture if k not in required]
    if not extra:
        die("failure.json must include at least one invented key")
    print(f"fixture extra keys: {extra}")


def check_output(path: Path, required: list[str], schema: dict) -> None:
    payload = json.loads(path.read_text())
    missing = [k for k in required if k not in payload]
    extra = [k for k in payload if k not in required]
    if missing:
        die(f"output missing {missing}")
    if extra:
        die(f"output invented {extra}")
    for key in schema.get("required", []):
        if key not in payload:
            die(f"schema required key missing: {key}")


def main() -> None:
    for path in REQUIRED_FILES:
        if not path.exists() or path.stat().st_size == 0:
            die(f"missing evidence: {path}")
    schema = load_schema()
    required = load_required()
    check_bound()
    check_rollback()
    check_failure_fixture(required)
    if len(sys.argv) == 2:
        check_output(Path(sys.argv[1]), required, schema)
        print("GATES PASS: output matches evidence")
        return
    print("GATES PASS: model call is allowed")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it before any generate step. Then run it against the model output.

python3 gates.py
python3 gates.py /tmp/agent-out.json
echo $?
Enter fullscreen mode Exit fullscreen mode

Exit 2 means stop. Do not retry the same prompt. Fix the evidence or drop the task.

Walk the failure through

Drop the fixture in as if it were live output. Watch the script refuse it.

cp evidence/failure.json /tmp/agent-out.json
python3 gates.py /tmp/agent-out.json; echo exit:$?
Enter fullscreen mode Exit fullscreen mode

You should see GATE FAIL: output invented ['retry_count']. That is the whole product.

Now strip the invented key and rerun.

printf '%s\n' '{"task_id":"job-442","timeout_ms":30000}' > /tmp/agent-out.json
python3 gates.py /tmp/agent-out.json; echo exit:$?
Enter fullscreen mode Exit fullscreen mode

Exit 0 is the only green light. Anything else stays local.

What if the schema file goes missing? Delete it and run python3 gates.py. The model never starts. That is fail-closed.

Where a free model actually fits

I only call a model after gates.py prints GATES PASS.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option. I treat those two facts as the budget, not as a benchmark. I am not naming models, hardware, or uptime. The gates stay on my machine either way.

The free server is useful when I do not want agent code on a laptop. The gate script still runs first. If the sandbox is gone tomorrow, the checklist still works. That is the rollback for the vendor itself.

A soft try is enough: one gated loop, then stop. Do not burn the token pool proving a slogan.

Time, cost, and when to quit

Forty minutes is the cap. If the checker grows a policy engine, I quit.

Cost bound lives in evidence/cost_bound.txt. I set max_tokens=4000 and max_seconds=20 for this one-task loop. Cross either line and I run evidence/rollback.sh. No second model call.

Abandon criteria, written down:

  1. A gate needs network I/O to decide.
  2. Someone wants a dashboard instead of exit codes.
  3. The task needs more than one tool call.
  4. additionalProperties: false feels too strict for the job.

If item four is true, this method is wrong for you. Say that out loud.

Who should not use this

Do not use this as a platform safety story. It is a file check.

Skip it if you already have an eval harness. Skip it for customer chat. Skip it when legal needs a vendor SLA. Skip it if your agent must keep talking after a missing field.

This is for a solo builder shipping one CLI task. It is not a mesh. It is not MCP. It is not an agent framework.

Most so-called agents are still if-statements. Fine. Make the if-statement loud and local.

Rollback is a file too

People forget the undo. I did.

cat evidence/rollback.sh
./evidence/rollback.sh
Enter fullscreen mode Exit fullscreen mode

If rollback cannot run in one command, the gate fails. That is the point. Would you keep a patch you cannot revert?

I also keep a dry-run wrapper so a bad night cannot skip the checker.

#!/bin/sh
set -e
python3 gates.py
# only then: send the prompt in your usual CLI
python3 gates.py /tmp/agent-out.json
Enter fullscreen mode Exit fullscreen mode

No wrapper, no run. Yes, that is another if-statement. I want that if-statement.

What this does not prove

It does not prove the model is right. It proves the model did not invent a key.

It does not prove the timeout is wise. It proves you chose the timeout.

It does not prove the free server is healthy. Your two-host canary is a different article. This file only answers one question: did we guess?

If you need latency charts, stop here. If you need red-team coverage, stop here. If you need ten tools in a loop, stop here.

Ship rule I actually keep

No evidence directory, no generate. No fixture, no generate. No rollback, no generate.

The community keeps arguing that agents assume too much. I believe that. I still will not write a glossary about it.

I will keep a gate file. Will you?

Which assumed field has burned your CLI, and what evidence file would have blocked that run?

Top comments (0)