The ticket said "retry the capture call if the provider times out." That was the entire spec. An agent returned a 180-line patch: jittered backoff, a new payments.retry_budget key, a Redis set, and a 24-hour dead-letter table. Tests were green.
None of those four inventions lived in the ticket, the OpenAPI document, or the runbook. Two would have been harmless. The Redis set was not. The service cannot store PAN-adjacent tokens outside the vault path. The defect was not a syntax error. It was assumption debt.
This walkthrough treats that class of failure as a routing problem. Name the gap. Then choose a leaf. Do not start by asking a model to "just be careful."
Glossary
Use these terms as they are used in the tree. If a word is not in this list, it is not a branch.
Spec gap. A fact the ticket needs and does not contain: timeout budget, idempotency key scope, retry storm limits, error taxonomy.
Assumption debt. Code or config the model added to close a spec gap without a cited source. Debt is not "the model was creative." It is an untracked requirement.
Ungrounded default. A numeric or enum choice with no repo evidence. max_retries = 5 and backoff = exponential are ungrounded if no runbook states them.
Contract surface. Anything other services, auditors, or future diffs must honor: public routes, OpenAPI, authz, durable schemas, log fields that leave the box.
Environmental unknown. A fact that is not in git because it exists only at runtime: DNS, queue depth, sandbox credentials, clock skew.
Constraint card. A short, versioned note that freezes a gap as unknown instead of filling it. The next generator must read it.
Fail-closed test. A test that fails if the implementation invents a store, route, or header the contract does not list.
Assumption marker. Language in a commit message, PR body, or code comment that admits a guess: "assuming," "defaulting to," "probably," "for now."
The branching test
Run the questions in order. Stop at the first leaf. Skipping a question is how Redis sets appear in payment services.
- Recoverable? Can a deterministic search of the repo, OpenAPI, or runbook answer the gap without a model?
- Contract-touching? Would a wrong fill change a contract surface?
- Encodable? Can you write a failing test that forbids invention, even if you cannot yet write the happy path?
- Environmental? Is the remaining unknown a runtime fact rather than a product fact?
Routing:
- Q1 yes → Leaf A: recover from the repo
- Q1 no, Q2 yes, Q3 yes → Leaf B: fail-closed contract test
- Q1 no, Q2 yes, Q3 no → Leaf C: park the gap
- Q1 no, Q2 no, Q4 yes → Leaf D: replay the environment
- Q1 no, Q2 no, Q4 no → Leaf C: park the gap
Leaf C appears twice on purpose. "Not a contract" and "cannot encode" are different reasons to refuse generation. The action is the same: do not let the model mint a requirement.
Leaf A: Recover from the repo
Worked gap: "What is the capture timeout?"
The agent wanted to invent timeout_seconds = 30. Before any generation, search.
rg -n "capture.*timeout|TIMEOUT|read_timeout" apps/payments docs/runbooks openapi.yaml
rg -n "retry" docs/runbooks/*.md
Suppose docs/runbooks/payments.md already says the provider's documented capture timeout is 8 seconds, and apps/payments/client.py uses timeout=8. The leaf is finished. Copy the number. Cite the path in the PR.
Do not "confirm" the number with a model. Confirmation is how 8 becomes 30 with a confident paragraph. If two sources disagree, that is not Leaf A. Escalate to Leaf C with both citations on the constraint card.
Leaf B: Fail-closed contract test
Worked gap: "Where may retry state live?"
The ticket still does not say. Q1 found nothing. Q2 is yes: storage is a contract surface. Q3 is yes: you can forbid new stores without knowing the final design.
# tests/test_retry_surface.py
# Fail closed if a retry module introduces undeclared durable stores.
from pathlib import Path
ALLOWED_STORE_MARKERS = (
"vault.put",
"payments_db.execute",
)
FORBIDDEN_STORE_MARKERS = (
"redis.",
"StrictRedis",
"localStorage",
"open(",
)
def test_retry_patch_does_not_invent_a_store() -> None:
retry_source = Path("apps/payments/retry.py").read_text(encoding="utf-8")
lowered = retry_source.lower()
assert any(m.lower() in lowered for m in ALLOWED_STORE_MARKERS)
for marker in FORBIDDEN_STORE_MARKERS:
assert marker.lower() not in lowered, f"invented store marker: {marker}"
Run it before the model runs.
pytest tests/test_retry_surface.py -q
Then generate against the red test. The model may still write backoff math. It should not be able to merge a Redis set. If your generator ignores failing tests, this leaf does not apply; use Leaf C.
Leaf C: Park the gap
Worked gap: "May we retain provider correlation IDs for 24 hours?"
Q1 is no. Q2 is yes (retention plus identifiers). Q3 is no: a unit test cannot decide legal retention. Parking is the honest output.
# constraints/payments-retry-correlation.yaml
id: pay-retry-correlation-2026-09
status: unknown
surface: contract
question: >
May capture correlation IDs be retained, and for how long?
must_not:
- invent a dead-letter table
- log PAN-adjacent tokens
- default retention to 24h
evidence:
- ticket: PAY-4417
- search: no hit in docs/runbooks or openapi.yaml
owner: payments-platform
Attach the card to the PR and stop. A parked gap is cheaper than a table you will spend a quarter deleting. Generation against an unknown card is out of scope for this tree.
Leaf D: Replay the environment
Worked gap: "Does the provider return 408 or close the socket?"
That is not in git. It is also not a product policy. It is environmental. Inventing except TimeoutError or if status == 408 is still assumption debt, just a different kind.
Replay it. Use a disposable host, a recorded fixture if you have one, and a client pointed at the sandbox—not at production.
# scripts/replay_capture_timeout.py
# Unexecuted sketch: classify the provider's timeout behavior once, then freeze it.
import json
import os
import urllib.error
import urllib.request
def classify(url: str) -> dict:
req = urllib.request.Request(url, method="POST", data=b"{}")
try:
with urllib.request.urlopen(req, timeout=2) as resp:
return {"kind": "http", "status": resp.status}
except urllib.error.HTTPError as exc:
return {"kind": "http", "status": exc.code}
except TimeoutError:
return {"kind": "socket_timeout"}
except Exception as exc:
return {"kind": "other", "type": type(exc).__name__}
if __name__ == "__main__":
result = classify(os.environ["PAYMENTS_SANDBOX_URL"])
print(json.dumps(result, indent=2))
Freeze the JSON next to the constraint card. After that, generation is Leaf A: the fact is now in the repo.
This is the only leaf where a spare server earns its keep. A local laptop often cannot reach the sandbox, hold the fixture process, or keep secrets off a laptop disk. A short-lived host can.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is a fit after Leaf B has a red test, not before Q1. The free server option is a fit for Leaf D replays when the unknown is environmental and the command is already written. Neither replaces the tree. If the card says unknown, do not send the ticket to a model to "take a pass."
A scanner for unmarked debt
Markers are the easy case. The costly case is a patch that never admits it guessed. The script below is lexical. It does not prove correctness. It ranks diffs for human review.
#!/usr/bin/env python3
"""Scan a unified diff for assumption markers and undeclared config keys.
Label: reproducible helper, not a production linter.
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
MARKER_RE = re.compile(
r"\b(assuming|assumed|defaulting to|for now|probably|i guessed|placeholder)\b",
re.I,
)
ADDED_ASSIGN_RE = re.compile(r"^\+\s*([A-Z][A-Z0-9_]+)\s*=\s*.+$")
def added_lines(diff: str) -> list[str]:
return [
ln[1:]
for ln in diff.splitlines()
if ln.startswith("+") and not ln.startswith("+++")
]
def repo_constants(root: Path) -> set[str]:
chunks: list[str] = []
for path in root.rglob("*"):
if path.suffix in {".py", ".yml", ".yaml", ".md"} and path.is_file():
try:
chunks.append(path.read_text(encoding="utf-8", errors="ignore"))
except OSError:
continue
blob = "\n".join(chunks)
return set(re.findall(r"[A-Z][A-Z0-9_]{3,}", blob))
def scan(diff: str, root: Path) -> dict:
added = added_lines(diff)
markers = [ln for ln in added if MARKER_RE.search(ln)]
new_consts = []
for ln in added:
match = ADDED_ASSIGN_RE.match("+" + ln)
if match:
new_consts.append(match.group(1))
known = repo_constants(root)
invented = [name for name in new_consts if name not in known]
return {"markers": markers, "invented_constants": invented}
if __name__ == "__main__":
report = scan(sys.stdin.read(), Path("."))
print("markers:", len(report["markers"]))
for ln in report["markers"]:
print(" ", ln)
print("invented_constants:", report["invented_constants"])
Usage:
git diff main | python3 scripts/scan_assumption_debt.py
Read the invented constants first. Markers without invented constants are noise. Invented constants without markers are the billing-service case.
Limits, and who should skip this
The tree does not estimate model quality. It routes work around generation until the gap is named. Lexical scans miss renamed Redis wrappers and dynamic getattr stores. Fail-closed tests encode only the forbids you remembered to type.
Do not use this approach if you need a fully autonomous merge bot. Do not use it as a substitute for threat modeling on payment, identity, or medical paths. Do not send production secrets to any hosted model, free or not, to "inspect" a sandbox. If your org cannot allow generated code onto a shared host, skip Leaf D and park the environmental gap until a blessed runner exists.
Cheap generation makes missing specs look like missing code. They are not the same. Fill from git, fail closed, park, or replay. Inventing a requirement is the one leaf this tree does not have.
If you run the scanner on one recent agent PR, keep the invented-constant list. That list is the backlog the ticket never wrote.
Top comments (0)