If an agent wrote the patch, a green pipeline is not a merge signal. You need an evidence packet, and any missing field should fail closed.
Agents fill silence with defaults. That is what they are good at. Your job is to refuse the default until someone can show the receipt.
Green is not a gate
You already test the diff. Necessary. Not sufficient.
A tool-using agent can invent a region, a retry count, a bucket name, or a "temporary" public ACL. Tests still pass when they were generated against the same fiction. Production then meets a different world.
Stop asking "did CI pass?" Ask "what did the agent assume, and where is the proof?"
This article is a copy-paste gate list, not a war story. Treat the files as a starter you must adapt. They are unlabeled against any production traffic.
What the packet is
Every agent-touched PR is incomplete until a packet exists in the repo. Keep it at .ci/evidence-packet.yml, or have a bot attach the same fields as a comment. Format is cheaper than the rule: missing key means the gate stays closed.
# .ci/evidence-packet.yml
# Proposal: fail closed if any required field is empty or "unknown".
schema_version: 1
pr:
id: "${PR_NUMBER}"
agent_assisted: true
human_owner: "" # required
rollback_owner: "" # required, must differ from human_owner on sev-1 paths
assumptions: [] # required list; empty list fails
egress_allowlist: [] # host:port only; empty fails if the diff touches network code
secrets:
new_credential_files: [] # must be empty
vault_refs: [] # required if the diff reads a new env key
replay:
commands: [] # exact, pinned, copy-pasteable
fixture_sha256: ""
seed: ""
tools:
calls: [] # name, idempotency_key, input_hash
data_class: "internal" # public | internal | restricted
kill_switch:
flag_name: ""
default_closed: true
If a field is unknown, you do not guess. You fail the PR.
Eight gates you can copy
Each gate has three parts: requirement, evidence, fail-closed criterion. Paste them into your team doc. Then wire the checker so humans cannot "agree in Slack" past a red job.
1. Assumption log
Require: every default the agent chose is written down. Region, timeout neighbor values, queue name, feature-flag default, "I assumed us-east-1" — all of it.
Evidence: assumptions[] with field, value, source (human | config | agent-guess).
Fail closed: any source: agent-guess without a human acked_by on the same line. Empty assumptions on an agent-assisted PR.
2. Egress allowlist
Require: if the diff can open a socket, the destinations are listed as host:port. No glob that matches the internet.
Evidence: egress_allowlist plus a static scan of new URLs, SDK clients, and webhook targets.
Fail closed: a new host in the diff that is not in the list. A list that contains * or 0.0.0.0.
3. Secret negative proof
Require: the diff adds no live credentials. New env keys must point at a vault reference, not a value.
Evidence: scanner output (even a cheap regex job) and secrets.vault_refs.
Fail closed: a high-entropy assignment in a committed file. A new AWS_SECRET_* style key with a literal. A packet that says new_credential_files is non-empty.
4. Replay recipe
Require: a stranger on the team can reproduce the agent's claimed result from the PR body alone.
Evidence: replay.commands with pinned tool versions, a fixture hash, and a seed if anything is stochastic.
Fail closed: commands that include latest, curl | sh, or unpinned images. A missing fixture_sha256 when tests use checked-in payloads.
5. Split ownership
Require: a human owner for the change and a different rollback owner for anything that can page.
Evidence: two GitHub handles in the packet. Not a bot. Not the agent.
Fail closed: identical names on a path tagged sev-1 or data_class: restricted. Empty owner fields.
6. Data class
Require: the PR declares what may leave the box. Public docs are one class. Restricted customer payloads are another.
Evidence: data_class plus a redaction note if logs or prompts can contain user text.
Fail closed: restricted data combined with any egress host that is not your own telemetry path. public declared on a diff that reads production dumps.
7. Tool-call receipts
Require: every tool the agent invoked has a name, an idempotency key, and a hash of the input.
Evidence: tools.calls[]. Re-runs with the same key must not create a second side effect.
Fail closed: a mutating tool call without an idempotency key. Two calls with the same key and different input_hash.
8. Kill switch default
Require: new agent-driven behavior ships behind a flag that defaults closed.
Evidence: kill_switch.flag_name and a test that boots with the flag off.
Fail closed: default_closed: false on a first merge. A missing flag on a new outbound path.
A checker that refuses missing proof
Do not wait for a perfect platform. Run a small fail-closed script in CI. The script below is a proposal. It reads the packet and the diff names. It does not claim to be a security product.
#!/usr/bin/env python3
"""fail_closed_packet.py — proposal, not a production audit."""
from __future__ import annotations
import os
import re
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError:
print("missing PyYAML", file=sys.stderr)
sys.exit = 2
PACKET = Path(".ci/evidence-packet.yml")
SECRET_HINT = re.compile(
r"(api[_-]?key|secret|passwd|private[_-]?key)\s*=\s*['\"][^'\"]+['\"]",
re.I,
)
HOST_HINT = re.compile(r"https?://([^/\s]+)", re.I)
def die(msg: str) -> None:
print(f"FAIL-CLOSED: {msg}")
raise SystemExit(1)
def load_packet() -> dict:
if not PACKET.exists():
die("missing .ci/evidence-packet.yml")
data = yaml.safe_load(PACKET.read_text()) or {}
if data.get("schema_version") != 1:
die("unsupported or missing schema_version")
return data
def changed_files() -> list[str]:
base = os.environ.get("EVIDENCE_BASE", "origin/main")
out = subprocess.check_output(
["git", "diff", "--name-only", f"{base}...HEAD"], text=True
)
return [line.strip() for line in out.splitlines() if line.strip()]
def main() -> None:
pkt = load_packet()
pr = pkt.get("pr") or {}
if pr.get("agent_assisted") and not pr.get("human_owner"):
die("agent PR has no human_owner")
if not pr.get("rollback_owner"):
die("rollback_owner required")
if pr.get("human_owner") == pr.get("rollback_owner") and (
pkt.get("data_class") == "restricted"
):
die("restricted changes need a distinct rollback_owner")
assumptions = pkt.get("assumptions") or []
if pr.get("agent_assisted") and not assumptions:
die("agent PR needs a non-empty assumption log")
for row in assumptions:
if row.get("source") == "agent-guess" and not row.get("acked_by"):
die(f"unacked agent-guess: {row.get('field')}")
if pkt.get("secrets", {}).get("new_credential_files"):
die("new_credential_files must be empty")
files = changed_files()
blobs = []
for rel in files:
path = Path(rel)
if path.is_file():
blobs.append(path.read_text(errors="ignore"))
joined = "\n".join(blobs)
if SECRET_HINT.search(joined):
die("literal secret assignment in diff")
allow = set(pkt.get("egress_allowlist") or [])
hosts = set(HOST_HINT.findall(joined))
if hosts and not allow:
die("network strings in diff but egress_allowlist is empty")
for host in hosts:
# packet lists host:port; compare on host
if not any(entry.startswith(host) for entry in allow):
die(f"host {host} not in egress_allowlist")
replay = pkt.get("replay") or {}
if not replay.get("commands"):
die("replay.commands required")
for cmd in replay["commands"]:
if "latest" in cmd or "curl " in cmd and "| sh" in cmd:
die(f"unpinned or piped install in replay: {cmd}")
if any("fixture" in f for f in files) and not replay.get("fixture_sha256"):
die("fixture change without fixture_sha256")
calls = (pkt.get("tools") or {}).get("calls") or []
seen = {}
for call in calls:
if call.get("mutating") and not call.get("idempotency_key"):
die(f"mutating tool {call.get('name')} lacks idempotency_key")
key = call.get("idempotency_key")
if key:
prev = seen.get(key)
if prev and prev != call.get("input_hash"):
die(f"idempotency key {key} reused with different input")
seen[key] = call.get("input_hash")
ks = pkt.get("kill_switch") or {}
if not ks.get("flag_name") or ks.get("default_closed") is not True:
die("kill_switch must exist and default closed")
print("PASS: evidence packet present and fail-closed checks held")
if __name__ == "__main__":
main()
Wire it so a missing packet is an error, not a skip:
# .github/workflows/evidence-gates.yml
# Proposal: copy and pin your own action versions.
name: evidence-gates
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
fail-closed:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install pyyaml
- name: Require evidence packet
env:
EVIDENCE_BASE: origin/${{ github.base_ref }}
run: |
set -euo pipefail
test -f .ci/evidence-packet.yml
python3 fail_closed_packet.py
Add a PR template so humans do not discover the schema after the bot yells.
## Evidence packet
- [ ] `.ci/evidence-packet.yml` updated
- [ ] Every `agent-guess` has `acked_by`
- [ ] Replay commands are pinned
- [ ] Kill switch defaults closed
- [ ] Rollback owner is a person, not the agent
Run the same checker locally before you open the PR:
export EVIDENCE_BASE=origin/main
python3 fail_closed_packet.py
If it dies, you do not "fix CI." You add proof or you delete the guess.
Where free model access belongs
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Free model access is useful for drafting the patch and for filling a first packet. A free server option is useful for running fail_closed_packet.py the same way CI will. Neither is evidence. If you already use MonkeyCode for that free model access or free server path, point the replay commands at the same checker. Keep the gate in your repo so a vendor outage cannot skip it.
Do not put model names, token budgets, or hardware claims in the packet unless you can pin them from a primary dashboard on the day you merge. Unpinned capacity is another agent-guess.
Who should not use this
Skip this approach if you have no agent in the loop. A human-only typo fix does not need an assumption log.
Skip it if your org already has a change-management system that stores the same fields and blocks merge on them. Duplicate gates rot. Pick one source of truth.
Skip it for research branches that never touch shared credentials, customer data, or network egress. A lab notebook is enough there.
Do not use the regex secret check as your only control. It is a tripwire. Real secret scanning and vault policy still sit in front of it.
Limitations
The checker does not execute the agent. It cannot see a tool call that never landed in the packet. If the agent lies by omission, a human still has to read the diff.
YAML is not cryptography. Anyone with push access can write fiction into .ci/evidence-packet.yml. Pair the gate with CODEOWNERS on that path and on .github/workflows/evidence-gates.yml.
Host extraction from https:// strings misses SDK defaults that never appear as URLs. Extend the scan for your language, or fail closed whenever a new HTTP client is imported.
Idempotency keys only help if the downstream tool honors them. If the vendor API ignores the key, your packet is a comment.
This list will not make cheap generated code cheap to own. It only stops you from merging a story that nobody can replay.
When the packet is missing, the answer is no. Not "ship and backfill." No.
Top comments (0)