DEV Community: Sam Chen

You Closed the Chat. Five Anti-Patterns Still Merged.

Sam Chen — Sun, 20 Sep 2026 11:21:13 +0000

The chat went quiet, so the work felt finished. That feeling is the bug, not the model.

I keep a five-row anti-pattern catalog for this moment. Each row lists symptoms, root cause, and a local replacement.

A free session can draft code without any shame. Your repo still owns the merge gate, always.

Why this catalog exists

AI coding sessions now end in minutes, not days. Speed is not the same thing as a merge contract.

Have you ever pasted a green snippet as review? I have done that, and the next outage was local.

Trending posts ask if models outgrew our tests. Wrong question for this catalog on your machine. Those tests never ran against your actual checkout.

How to read each row

Use the catalog during the last ten minutes. Do not use it as a vibe check.

For every row I want three things only:

Symptoms you can spot in the diff or logs
Root cause stated in one blunt sentence
A replacement pattern with a real command

No score. No leaderboard. Just fail-closed checks you can rerun.

Anti-pattern 1: Clipboard as CI

You copy the patch into the tree. You never run a command that can fail.

Symptoms

The PR quotes the chat, not a failing command
Generator output never touches a scripts/ gate
Review says "looks good" with zero pasted output

Root cause

The session produced text. You treated that text as an execution receipt.

Replacement: a local fail-closed gate

Write one script the model cannot mark green by talking.

#!/usr/bin/env bash
# scripts/local-gate.sh
set -euo pipefail
cd "$(dirname "$0")/.."

test -f reproduce.sh
test -f decisions.md

if command -v rg >/dev/null; then
  if rg -n "AKIA|BEGIN PRIVATE|api_key\s*=" --glob '!decisions.md' .; then
    echo "secret-shaped text in the tree" >&2
    exit 1
  fi
fi

./reproduce.sh
python3 scripts/check_defaults.py
pytest -q tests/gate

Did the chat approve the patch in prose? Fine. Did local-gate.sh exit zero on your laptop?

Anti-pattern 2: Anonymous runtime

You ran the demo on a borrowed box. Nobody on the team can replay it.

Symptoms

The README says it worked on "the server"
No commit SHA, seed, or interpreter version is recorded
A teammate cannot hit the same path on a laptop

Root cause

A shared runtime became the source of truth. Your laptop never signed the result.

Replacement: a reproduce file that pins the command

#!/usr/bin/env bash
# reproduce.sh
set -euo pipefail
echo "git=$(git rev-parse HEAD)"
echo "python=$(python3 --version)"
python3 -m venv .venv
# shellcheck disable=SC1091
. .venv/bin/activate
pip install -r requirements-gate.txt
python3 -m app.probe --fixture tests/fixtures/min.json

The box can be disposable and even free. The script cannot vanish with the session.

Anti-pattern 3: Model-chosen defaults

The generator picked timeouts, pool sizes, and log levels. You shipped those numbers unchanged.

Symptoms

timeout=30, retries=3, workers=4 with no comment
Config keys appear only inside generated files
No test asserts a deadline or a fail-closed limit

Root cause

Defaults look like expertise in a hurry. They are the model's prior, frozen into your repo.

Replacement: an explicit defaults review

# scripts/check_defaults.py
from pathlib import Path
import re
import sys

ALLOWED = {
    "timeout_s": {5, 10, 15},
    "retries": {0, 1},
    "workers": {1, 2},
}

text = Path("app/config.py").read_text(encoding="utf-8")
fail = False
for key, allowed in ALLOWED.items():
    match = re.search(rf"{key}\s*=\s*(\d+)", text)
    if not match or int(match.group(1)) not in allowed:
        print(f"unreviewed default: {key}")
        fail = True
if fail:
    sys.exit(1)
print("defaults reviewed")

Would you defend retries=3 in an incident channel? If not, do not merge that line.

Anti-pattern 4: Secrets in the prompt

You pasted .env so the model could "see the real shape." That thread now outlives the key rotation.

Symptoms

Chat history contains hostnames, tokens, or customer ids
Generated fixtures still point at production URLs
.gitignore looks clean, but the session log is not

Root cause

Context dumping felt faster than making a fake fixture. The leak is now immortal in the thread.

Replacement: a fixture that cannot authenticate

{
  "base_url": "http://127.0.0.1:9",
  "token": "test-token-not-secret",
  "user_id": "fixture-user"
}

If the model needs shape, give it a dead socket. Never give it a live key.

Scan the tree before the gate returns green:

# last line of local-gate.sh before pytest
python3 - <<'PY'
from pathlib import Path
bad = ("AKIA", "BEGIN PRIVATE", "sk-live", "xoxb-")
root = Path(".")
for path in root.rglob("*"):
    if path.suffix in {".png", ".pyc"} or ".venv" in path.parts:
        continue
    try:
        text = path.read_text(encoding="utf-8", errors="ignore")
    except OSError:
        continue
    if any(token in text for token in bad):
        raise SystemExit(f"secret-shaped text: {path}")
print("no secret-shaped text")
PY

Did the fixture still need a routable host? Then it is not a fixture.

Anti-pattern 5: Forty-turn design doc

The architecture lives in a scrolling chat. The repo only has a shrug.

Symptoms

The PR links a transcript instead of a table
Two engineers recount different "final" decisions
Re-asking the model quietly changes the story

Root cause

Conversation is cheap and feels like design. A twelve-line table is the actual design.

Replacement: a decision table in the repo

Keep decisions.md tiny. Blank cells mean the chat is not done.

Decision	Options	Chosen	Why	Revisit when
Auth	session cookie / static token / none	static token in tests	local gate must not hit IdP	we add a real user
Runtime	laptop / borrowed server / CI	laptop owns merge	borrowed boxes lie about paths	CI is green twice
Timeouts	5s / 30s / none	5s	fail closed on hang	p99 is measured
Retries	0 / 3 / infinite	0	retries hide probe bugs	idempotency is proven
Fixtures	prod dump / synthetic	synthetic	secrets stay out of prompts	schema changes

Print this table in the PR body. If a cell is blank, stop generating more code.

The artifact: one gate, one table

Here is the workflow I want on the last commit. Treat it as a proposal, not a published study.

Fill decisions.md before asking for more code.
Generate against fixtures, never against live secrets.
Run ./scripts/local-gate.sh on your laptop.
Treat any borrowed server as a scratch lane only.
Merge only when the gate exits 0 and the table is complete.

I am not attaching latency charts on purpose. Charts without a pinned reproduce.sh are theater.

A tiny pytest file keeps the happy-path lie from shipping:

# tests/gate/test_fail_closed.py
from app.probe import probe
from app import config

def test_dead_socket_fails_closed():
    result = probe("tests/fixtures/min.json")
    assert result.ok is False
    assert result.error in {"timeout", "connection_refused"}

def test_retries_are_zero():
    assert config.retries == 0

def test_timeout_is_reviewed():
    assert config.timeout_s in {5, 10, 15}

Does a dead socket fail closed on the first try? If that probe "succeeds," your gate is theater.

Wire a stub module so the example is runnable:

# app/config.py
timeout_s = 5
retries = 0
workers = 1

# app/probe.py
from dataclasses import dataclass
from pathlib import Path
import json
import socket

@dataclass
class Result:
    ok: bool
    error: str | None

def probe(fixture_path: str) -> Result:
    data = json.loads(Path(fixture_path).read_text(encoding="utf-8"))
    host, port = "127.0.0.1", 9
    if ":" in data["base_url"].split("//", 1)[-1]:
        hostport = data["base_url"].split("//", 1)[-1]
        host, port_s = hostport.split(":")[0], hostport.split(":")[1].split("/")[0]
        port = int(port_s)
    try:
        socket.create_connection((host, port), timeout=0.2).close()
        return Result(ok=True, error=None)
    except OSError as exc:
        name = "timeout" if isinstance(exc, socket.timeout) else "connection_refused"
        return Result(ok=False, error=name)

Run it. Watch the dead port refuse you. That refusal is the point.

Where a free model and free server fit

I still use a throwaway lane for drafts. Scratch generation is fine. Merge authority is not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. I treat that pair as the scratch lane: draft the probe, try the fixture, throw the box away. The local gate above still decides the merge. I am not claiming model names, quotas, hardware, or benchmarks. Those claims go stale. These five anti-patterns do not.

If you want a disposable place to practice the gate, that scratch lane is enough. Do not promote it into CI.

Limitations

This catalog will annoy you on tiny scripts. Good. Friction on a one-line typo is a smell, not a virtue.

It will not catch semantic bugs the tests never named. It will not replace a threat model. It will not make a weak fixture honest.

Skip this approach when:

You are pairing live and already running the suite
The change is a comment, rename, or typo
You cannot run anything locally at all
You need a signed production deploy, not a laptop gate

Cannot run locally? Then you have a bigger problem than chat quality. Borrowed servers drift. Free sessions forget. Your reproduce.sh should not.

What I will not do

I will not call the chat a review. I will not paste live keys into a prompt. I will not ship retries=3 because it looked standard.

Would you bet an incident channel on a transcript? I would not.

Close the tab after the gate is green. Not before.

The Chat Approved It. Git Still Has No Receipt.

Sam Chen — Fri, 18 Sep 2026 07:48:44 +0000

Cheap inference does not replace a human review gate.
It makes five skip patterns cheaper to repeat.
I still catch them in my own branches.

What this piece is not

This is not another vibe-coding culture rant.
I will not grade your editor or your chat model.
I will show symptoms, root causes, and replacements.

The public debate this week is still noisy.
People argue about whether AI writes real software.
The useful question is smaller and much meaner.

Did this diff actually get a review receipt?
Or did a cheap run just feel like shipping?

The catalog at a glance

I use the same three columns every time.

Symptom: what shows up in git or chat.
Root cause: why cheap inference encourages the skip.
Replacement: the pattern I want instead.

The proposed scripts and tables live right below.
Treat the scripts as templates, not production gospel.

Anti-pattern 1: Chat as the source of truth

Symptoms

The pull request says "details in the thread."
Commits read like "update stuff" and nothing else.
Reviewers paste screenshots instead of file paths.

Root cause

The sidebar answered faster than the commit template.
The answer felt documented because it was long.
Git never stored the actual product decision.

Replacement: put the decision in git

I put the tradeoff in the commit body.
I link the chat only as appendix, never as proof.
If the chat dies, the repo must still explain the change.

git commit -m "$(cat <<'EOF'
reject unsigned webhook payloads

Why: the agent added a debug bypass on staging.
Decision: verify signatures in the wrapper, not the prompt.
Chat transcript is appendix only, not source.
EOF
)"

Ask yourself one rude and practical question.
Could a new teammate review this without the chat?
If the answer is no, the chat still owns your history.

Anti-pattern 2: Secrets riding the free prompt

Symptoms

Env fragments appear inside a pasted stack trace.
The prompt file contains hostnames from production.
The agent reprints a token in the session log.

Root cause

A free box still feels disposable and unofficial.
Disposable does not mean the prompt is private.
Cheap inference can still persist that text somewhere.

Replacement: redact, then hash

I redact the raw prompt first, without exceptions.
I hash the cleaned prompt file second.
I never paste live credentials into a sandbox prompt.

# Proposed local redaction before any remote prompt.
sed -E \
  -e 's/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+/[redacted-email]/g' \
  -e 's/(api[_-]?key|token|secret)[=:][^[:space:]]+/\1=[redacted]/Ig' \
  -e 's/[0-9]{1,3}(\.[0-9]{1,3}){3}/[redacted-ip]/g' \
  prompt.raw.txt > prompt.clean.txt

sha256sum prompt.clean.txt > prompt.clean.sha256

Run that before the model sees the file.
If redaction fails, the session does not start.

Would you paste that prompt into a public gist?
If not, do not send it to a free endpoint either.

Anti-pattern 3: The agent owns the lockfile

Symptoms

The lockfile moves with a one-line behavior fix.
The model tidies dependencies during a simple rename.
Nobody can explain the version jump in review.

Root cause

Cheap retries invite extra cleanup turns from the agent.
Those extra cleanup turns always love the lockfile.
Lockfiles are easy to generate and hard to audit.

Replacement: lockfile needs a human ACK

I split lockfile diffs from the behavior diffs.
I require an explicit ACK file in the pull request.
No ACK means the CI job fails closed.

# proposed_lockfile_ack.py
# Example only. Adapt paths before you trust it.
from pathlib import Path
import subprocess
import sys

LOCK_NAMES = {"package-lock.json", "poetry.lock", "Cargo.lock", "go.sum"}
ACK = Path("LOCKFILE_ACK.md")


def changed_files() -> set[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", "origin/main...HEAD"],
        text=True,
    )
    return {Path(p).name for p in out.splitlines() if p}


changed = changed_files()
touched_lock = sorted(changed & LOCK_NAMES)
if not touched_lock:
    print("no lockfile change")
    sys.exit(0)
if not ACK.exists():
    print("lockfile changed without LOCKFILE_ACK.md")
    sys.exit(1)
text = ACK.read_text(encoding="utf-8").lower()
if "i reviewed the lockfile diff" not in text:
    print("ACK is missing the review sentence")
    sys.exit(1)
print("lockfile ACK present for:", ", ".join(touched_lock))

Put that ACK in the same pull request.
Name every package version that actually moved.
Name the risk in one blunt sentence.

Did the agent bump a compiler to make tests pass?
That compiler bump is not just a tidy.
That bump is a silent product change too.

Anti-pattern 4: Policy living in the system prompt

Symptoms

"Do not call production" exists only in a prompt.
The tool wrapper still accepts the prod hostname.
Safety is a paragraph, not a check.

Root cause

Prompts are easy to type on a free model.
Guards in code take an extra hour.
Cheap inference rewards the hour you skip.

Replacement: policy in code, prompt as hint

I keep the system prompt short and polite.
I keep the policy in a denylist.
The model cannot override a failed hostname check.

# proposed_tool_guard.py
from urllib.parse import urlparse

BLOCKED_HOST_SUFFIXES = (".prod.internal", ".corp.example")
BLOCKED_HOSTS = {"127.0.0.1"}


def assert_safe_url(url: str) -> None:
    host = (urlparse(url).hostname or "").lower()
    if not host:
        raise ValueError("missing host")
    if host in BLOCKED_HOSTS:
        raise ValueError(f"blocked host: {host}")
    if any(host.endswith(sfx) for sfx in BLOCKED_HOST_SUFFIXES):
        raise ValueError(f"blocked suffix: {host}")

The prompt may still say "stay in staging."
The wrapper must still refuse the URL.
Which layer do you trust when the model is tired?

Anti-pattern 5: Idle free capacity as a release signal

Symptoms

The merge happened because the server was free tonight.
The reviewer is whoever is still online in chat.
Monday on-call inherits an unexplained diff.

Root cause

Cheap idle time feels like a wasted gift.
That waste anxiety beats review patience every time.
The review calendar never becomes a real gate.

Replacement: capacity is not a criterion

I keep capacity out of the ship list.
A free server can wait until review exists.
Idle machines do not approve the architecture change.

Would you merge this if the box were busy?
If the answer flips, capacity was the reviewer.
Ship criteria belong in the receipt, not the queue.

The artifact: a review receipt

I want one file per pull request.
I call that file REVIEW_RECEIPT.md on purpose.
This file is not a model freeze file.

A freeze file only pins the model inputs.
This receipt pins the review skips I just named.

# REVIEW_RECEIPT.md

- Decision lives in commit: yes/no
- Prompt redacted and hashed: yes/no (sha256: ...)
- Lockfile ACK present: yes/no/n/a
- Tool policy enforced in code: yes/no
- Capacity was not a ship criterion: yes/no
- Reviewer: name
- Date: ISO-8601

Use this table when the sandbox looks too green.

Claim you want	Cheap sandbox can show	Cheap sandbox cannot show
Script parses on a clean clone	Yes	Your dirty production working tree
Unit tests pass with fixtures	Yes	Real traffic shape
Tool guard blocks fixture URLs	Yes	A hostname someone adds later
Prompt hash matches redacted text	Yes	Extra secrets pasted only in chat
Feature is safe to ship	No	Data, auth, clocks, or load

If the table cell says no, the merge still needs a human.

Proposed checker:

# proposed_receipt_check.py
from pathlib import Path
import re
import sys

receipt = Path("REVIEW_RECEIPT.md")
if not receipt.exists():
    print("missing REVIEW_RECEIPT.md")
    sys.exit(1)

text = receipt.read_text(encoding="utf-8")
required = [
    "Decision lives in commit:",
    "Prompt redacted and hashed:",
    "Lockfile ACK present:",
    "Tool policy enforced in code:",
    "Capacity was not a ship criterion:",
    "Reviewer:",
]
missing = [line for line in required if line.lower() not in text.lower()]
if missing:
    print("receipt missing fields:", ", ".join(missing))
    sys.exit(1)

if re.search(r"sha256:\s*$", text, re.I | re.M):
    print("hash field is empty")
    sys.exit(1)

print("review receipt fields present")

Then you wire the checker into CI.

# proposed .github/workflows/review-receipt.yml
name: review-receipt
on: [pull_request]
jobs:
  receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python proposed_receipt_check.py

I have not executed this workflow on your org.
Copy it only after you change the names.

A cheap rehearsal, not a release

I still want a cheap place to practice the gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option.
I treat those as a rehearsal room for the receipt checker.
I do not treat them as production capacity, an SLA, or a benchmark.

If you rehearse there, run the redaction step first.
Then run the receipt checker on a throwaway branch.
Skip the demo prompt that never touches git.

Who should not use this approach

Do not use a free server for regulated customer data.
Do not use prompt text as your only access control.
Do not skip this catalog because the tokens were cheap.

This receipt is only a minimum bar.
It will not catch a clever exfil path.
It will not replace a real threat model.

Teams with no git history should not start here.
You should fix basic source control first here.
Then you add the review receipt file.

Limitations

The checker only proves that a file exists.
A bored reviewer can type "yes" everywhere.
That is still a process failure, not a model failure.

Cheap inference will keep getting even cheaper.
Your gates will not appear by accident.
Build them while the diffs are still small.

Would I merge a green sandbox without the receipt?
I would not merge that green branch yet.
Would I trust the receipt without reading the diff?
I would also refuse that lazy shortcut.

The Agent Wrote the Tests. They Still Echoed the Patch.

Sam Chen — Thu, 17 Sep 2026 05:44:43 +0000

Passing tests do not prove the agent understood the contract.
They often prove the agent quoted its own patch.
That gap is the merge risk, not the model.

I keep reviewing pull requests that look fully engineered.
The prose stays calm while the checklist stays green.
So why does production still surprise the on-call engineer?

We launder chat-driven patches through review theater.
Fluency becomes evidence and echoes become fake oracles.
This catalog names four anti-patterns I keep seeing.

Each entry has symptoms, a root cause, and a replacement.
I also include a copyable oracle check and a command pin.
Treat the snippets as labeled examples, not production history.

What this is not

This is not a ranking of coding models today.
This is not a claim that agents cannot write code.
This is a review workflow you can run tomorrow.

I am not publishing timings, winners, or customer stories.
Those claims go stale and I will not invent them.
If you need signed capacity data, use your own lab.

Anti-pattern 1: Confidence as a merge gate

Symptoms

The summary reads like a senior engineer wrote every line.
Reviewers praise tone, structure, and seemingly clear reasoning.
Nobody can name the invariant that actually changed here.

Root cause

Fluency is cheap while true invariants stay expensive.
A model can narrate a patch without owning its failure.
Your process then confuses narration with actual technical review.

Replacement

Require one written invariant before anyone opens the diff.
If the author cannot state it, the PR remains a spike.
Spikes get a ticket, a date, and no merge button.

INVARIANT: refunds never exceed the captured amount
PROOF: test_refund_cap in tests/billing_oracle.py
OWNER: @oncall-owner

Ask one rude question during every model-authored review.
What still fails if we delete the generated comments now?
If the answer is nothing, you reviewed a blog post.

Anti-pattern 2: Echo tests

Symptoms

Tests arrived in the same chat as the implementation.
Assertions repeat literals copied from the generated body.
Renaming a constant makes the test fail before production.

Root cause

The agent tested its story, not the external contract.
The test file is a mirror sitting beside the patch.
Green then only means I agreed with myself again.

Did the test exist before the implementation existed?
If not, you probably bought an echo, not an oracle.
Oracles come from specs. Echoes come from stdout.

Replacement

Write the oracle before the implementation is allowed.
Pin expected values from the spec, never from chat output.
If you lack a spec, you still lack a real test.

Here is a labeled example, not a measured suite.

# example_oracle.py — proposal, not a production run
from decimal import Decimal

SPEC_CAPTURED = Decimal("40.00")
SPEC_REQUESTED = Decimal("99.99")
SPEC_REFUND = Decimal("40.00")  # from the billing spec, not stdout


def refund_cap(captured: Decimal, requested: Decimal) -> Decimal:
    if requested < 0:
        raise ValueError("requested must be >= 0")
    return min(captured, requested)


def test_refund_never_exceeds_capture():
    assert refund_cap(SPEC_CAPTURED, SPEC_REQUESTED) == SPEC_REFUND


def test_oracle_does_not_quote_the_function():
    source = open(__file__, encoding="utf-8").read()
    banned = "assert refund_cap(SPEC_CAPTURED, SPEC_REQUESTED) == refund_cap"
    assert banned not in source

Run the example with a boring, repeatable command.

python -m pytest example_oracle.py -q

If your only assertion is output equals output, stop.
That check is a quote. It is not a contract.

Echo versus oracle, on one page

Signal	Echo test	Oracle test
Source of expected value	Generated code or chat	Spec, RFC, or ticket
Same session as the patch	Usually yes	Must be no
Rename a literal	Test dies first	Behavior still pinned
Delete the implementation comments	Still green	Still meaningful
Who can explain a failure	The chat log	A named owner

Print that table in the PR template if you must.
I would rather see one filled row than ten adjectives.
Which column did your last AI patch actually occupy?

Anti-pattern 3: Prompt until green

Symptoms

The chat log is longer than the resulting diff.
Failures vanish after another polite try again.
Nobody saved the first command that actually failed.

Root cause

You optimized the prompt instead of pinning the system.
The failure never became a hashed, repeatable command.
The next model, or the next morning, revives it.

Is the fix a new behavior, or a new pep talk?
If you cannot tell, you do not have a fix.
You have a mood swing with a green badge.

Replacement

Freeze the failing command, not the motivational thread.
Re-run that exact command after every prompt change.
If the command drifts, the supposed fix is theater.

# pin the failure; do not paraphrase it in chat
cat > /tmp/failing_cmd.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
python -m pytest tests/billing_oracle.py -q
EOF
chmod +x /tmp/failing_cmd.sh
sha256sum /tmp/failing_cmd.sh
/tmp/failing_cmd.sh; echo EXIT:$?

Paste the hash into the pull request body.
Did this command change, or did the behavior change?
Only the second answer counts as an actual fix.

Keep the script in the repo when the check matters.
A gist in chat will rot before the next on-call shift.
CI should run the same bytes your laptop ran.

install -d scripts
cp /tmp/failing_cmd.sh scripts/oracle_refund.sh
git add scripts/oracle_refund.sh tests/billing_oracle.py
git commit -m "pin refund oracle and the failing command"

Anti-pattern 4: Sandbox as production proof

Symptoms

It ran on a free box, so we called it done.
No owner, no data class, no rollback, no page path.
The demo path is the only path anyone executed.

Root cause

A disposable environment proves one thing: reproducibility.
It does not prove capacity, identity, or blast radius.
People collapse those proofs because the demo felt good.

Would you page this sandbox at three in the morning?
If the answer is no, it never signed production.
Stop letting a clean demo launder an unsigned change.

Replacement

Split the checks. Keep them boring and explicitly named.

Check	Sandbox may answer	Production still needs
Does the oracle fail closed?	Yes	Same oracle in CI
Can a stranger reproduce it?	Yes	README command plus hash
Will it hold real load?	No	Your own capacity test
Who gets paged at 03:00?	No	A named owner
Is this data class allowed?	No	Policy, not a chat

I use a free coding environment only as a reproduction box.
MonkeyCode is an open source option with free model access and a free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That box can rerun scripts/oracle_refund.sh from a clean tree.
It cannot sign your incident, your identity, or customer data.
If you need those signatures, you already left the sandbox.

A workflow you can copy tomorrow

Do this on the next model-authored pull request.
Do not skip the writing. The writing is the review.
Empty fields mean you are still in chat-driven mode.

State the invariant in one plain sentence.
Add one oracle that never imports patch literals.
Pin the failing command and store its sha256.
Mark sandbox results as reproduced, never shipped.
Name an owner who can revert without the chat log.

PR template
- Invariant:
- Oracle test path:
- Failing command hash:
- Reproduced on (sandbox/CI):
- Ship check (yes/no + owner):

Fill it in public comments, not in a private sidebar.
If any field is empty, the merge is still a spike.
Call it a spike. Do not call it engineering.

Want a tiny gate before humans even look?
This hook only checks that the template fields exist.
It will not prove the oracle is wise. Nothing local can.

# scripts/check_pr_template.sh — proposal
set -euo pipefail
file="${1:-pr_body.md}"
for key in Invariant "Oracle test path" "Failing command hash" "Ship check"; do
  grep -q "^- ${key}:" "$file"
  grep -vq "^- ${key}: *$" "$file"
done

Who should not use this

Do not use this catalog as a reason to ban models.
Do not use a free server as a load or soak test.
Do not paste secrets, prod dumps, or customer records there.

Do not treat my examples as measured benchmarks either.
I did not name models, quotas, hardware, or durations.
Those numbers rot. These anti-patterns do not rot.

Skip this sermon when the change is already specified.
A owned RFC with golden tests does not need this ritual.
You already have an oracle. Keep feeding it in CI.

Also skip it for throwaway spikes you will delete today.
Just do not merge the spike because the prose sounded sure.
Sounding sure is the cheapest part of the whole stack.

Limitations

Short review questions will not catch every failure mode.
Oracles are wrong when the spec itself is wrong.
Pinned commands rot if nobody runs them inside CI.

This workflow also slows a true exploratory spike.
That cost is the point, not an accident of process.
If the slowdown hurts, you were shipping narration.

I cannot prove your billing rules from a blog example.
Your domain invariants have to come from your own spec.
Copy the method. Do not copy my refund numbers blindly.

What I actually want in review

I want a sentence a stranger can falsify later.
I want a test that would fail if the spec changed.
I want a command hash that does not depend on charm.

Did you review behavior, or did you only review tone?
Did the tests quote the patch, or pin a contract?
If you cannot tell, the agent did not fail. Review did.

Pin the command. Name the owner. Then consider merge.
That is the whole method, and it still fits in a template.
The model can draft. It still cannot sign the incident.

The Model Replied. You Still Have No Freeze File.

Sam Chen — Wed, 16 Sep 2026 04:03:43 +0000

A green chat is not a green build.
The missing freeze file is the real outage.
Free remote inference does not pin a compiler.

The public feed keeps arguing about model skill.
Did AI already beat most working developers?
That question skips the only check I run.

Did you pin the run at all?
Or did you screenshot a lucky chat?
I keep a blunt catalog for this failure.

Each entry lists symptoms and a root cause.
Each one also names a replacement pattern.
I am not ranking vendors in this post.

I am not posting latency or quality charts.
I am talking about unpinned AI coding runs.
Folklore patches still look brilliant in a thread.

What I mean by a freeze file

A freeze file is a local contract.
It records what the model actually saw.
It also records the patch you accepted.

The proposed schema sits below.
Treat it as an example until you execute it.
Do not paste it into production CI untested.

{
  "prompt_sha256": "replace-me",
  "patch_sha256": "replace-me",
  "test_command": "pytest -q tests/test_billing.py",
  "created_at": "2026-09-16T12:00:00Z",
  "runtime_kind": "scratch-pad",
  "reproduced_locally": false,
  "secrets_present": false
}

I only trust prompt, patch, and test hashes.
Drop one hash and the run becomes folklore.
Git then stores a story, not a replay.

I gate merges on that file.
No freeze file, no merge.
The validator is boring on purpose.

#!/usr/bin/env python3
"""Refuse an AI coding run with no freeze file.

Local gate only. It never calls a model.
"""
from __future__ import annotations

import json
import sys
from pathlib import Path

REQUIRED = (
    "prompt_sha256",
    "patch_sha256",
    "test_command",
    "created_at",
    "runtime_kind",
)
ALLOWED_RUNTIME = {"local", "scratch-pad", "ci-pinned"}


def load(path: Path) -> dict:
    data = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict):
        raise ValueError("freeze file must be an object")
    return data


def validate(data: dict) -> list[str]:
    errors: list[str] = []
    for key in REQUIRED:
        if not data.get(key):
            errors.append(f"missing {key}")
    runtime = data.get("runtime_kind")
    if runtime not in ALLOWED_RUNTIME:
        errors.append(f"runtime_kind {runtime!r} is not pinable")
    if runtime == "scratch-pad" and not data.get("reproduced_locally"):
        errors.append("scratch-pad run was never reproduced locally")
    if data.get("secrets_present") is True:
        errors.append("freeze file claims secrets in the prompt")
    return errors


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: validate_freeze.py path/to/freeze.json", file=sys.stderr)
        return 2
    path = Path(sys.argv[1])
    if not path.is_file():
        print(f"no freeze file at {path}", file=sys.stderr)
        return 1
    errors = validate(load(path))
    if errors:
        print("unpinned run:")
        for item in errors:
            print(f"- {item}")
        return 1
    print("freeze file ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run the local gate like this.

sha256sum prompt.txt patch.diff
python3 validate_freeze.py freeze.json
pytest -q tests/test_billing.py

Want a shorter rule of thumb?
If validate_freeze.py cannot run, you vibe-coded.
The chat UI will not argue with that.

Anti-pattern 1: Unpinned inference

You asked the box. It answered.
You shipped the vibe anyway.
You never named the runtime kind.

Symptoms

Tomorrow's answer flips on the same prompt.
Nobody can name the runtime kind.
"Works on my machine" now means "works in chat."

Root cause

You treated a remote completion like gcc -v.
Compilers publish versions you can pin.
Many chat boxes do not.

Replacement pattern

Record runtime_kind before you accept a patch.
Reproduce the patch with local tests.
Refuse to merge folklore from a thread.

jq -r .runtime_kind freeze.json
# expected: local | scratch-pad | ci-pinned

Did the model get smarter overnight?
Or did the unnamed runtime drift?
I assume drift until the freeze says otherwise.

Anti-pattern 2: Chat log as the audit trail

The thread looks complete and friendly.
It even has jokes in the margins.
Auditors cannot replay jokes later.

Symptoms

The only spec lives in a vendor UI.
Patch files have no prompt hash.
New hires cannot reconstruct the change.

Root cause

You stored conversation, not evidence.
A chat log is not an artifact.
It is a diary with extra tokens.

Replacement pattern

Export three files, always, in one commit.
Keep prompt.txt, patch.diff, and freeze.json.
Then hash them before anyone reviews the patch.

git add prompt.txt patch.diff freeze.json
git commit -m "pin billing patch to freeze file"

Did the assistant summarize the diff for you?
I do not care about that summary.
I care whether patch_sha256 matches git diff.

python3 - <<'PY'
import hashlib, pathlib, json, sys
patch = pathlib.Path("patch.diff").read_bytes()
digest = hashlib.sha256(patch).hexdigest()
frozen = json.loads(pathlib.Path("freeze.json").read_text())
sys.exit(0 if frozen.get("patch_sha256") == digest else 1)
PY

Anti-pattern 3: Free server as CI

Exploration is cheap, and that is fine.
CI is a different machine with different trust.
Do not blur those two on purpose.

I still want a scratch pad for messy prompts.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option.

I map that pair to runtime_kind: scratch-pad only.
Draft there if you need a pad.
Then copy the patch home and run pytest locally.

Symptoms

Shared scratch boxes absorb flaky tests.
Failures get blamed on "the model."
Secrets leak into a communal prompt.

Root cause

You confused availability with isolation.
A free server is a shared whiteboard.
Whiteboards do not sign releases.

Replacement pattern

Keep the scratch pad in the draft lane.
Never point test_command at a shared box.
Set reproduced_locally only after local pytest.

{
  "runtime_kind": "scratch-pad",
  "reproduced_locally": true,
  "test_command": "pytest -q tests/test_billing.py"
}

If reproduced_locally is still false, stop.
You have a sketch, not a candidate patch.
The validator will say the same thing.

Anti-pattern 4: Tests written after the patch

The model "solved" the ticket first.
Then you asked it for tests.
The tests learned the first answer.

Symptoms

Tests assert the first returned string.
Edge cases never appear in the file.
Refactors break tests that never encoded rules.

Root cause

You used tests as applause.
Applause is not a specification.
The model taught the oracle.

Replacement pattern

Write the failing test first.
Commit it on main before any prompt.
Only then paste the prompt that cites it.

# tests/test_billing.py — proposed example, not executed here
def test_prorate_ignores_trailing_partial_day():
    assert prorate("2026-09-01", "2026-09-16", 3100) == 1500

Run this loop without skipping step one.

pytest must fail on main.
Freeze the prompt that cites that test.
Accept a patch only if the same test passes.
Re-run python3 validate_freeze.py freeze.json.

If step one was already green, you cheated.
The model did not invent your spec.
You let it grade its own homework.

Decision table I actually use

What I saw	Story I tell myself	Anti-pattern	Replacement
Same prompt, new answer	The model got worse	Unpinned inference	Freeze runtime, replay locally
Long chat history	We have an audit	Chat log as audit	Commit prompt, patch, freeze
Tests on a shared box	CI is covered	Free server as CI	Local pytest gate
Tests after the patch	We are thorough	Post-hoc tests	Failing test first

Print that table next to the PR.
If a row matches, name the anti-pattern.
Do not argue about model IQ instead.

A concrete debugging workflow

I use this when a "smart" patch rots.
It is a proposed workflow, not a benchmark.
I have not executed it against your repo.

Print the freeze file. Missing file means stop.
Check secrets_present. True means rotate keys.
Confirm reproduced_locally is actually true.
Replay test_command on a dirty checkout.
If replay fails, blame the freeze, not the model.

test -f freeze.json || { echo "unpinned"; exit 1; }
python3 validate_freeze.py freeze.json
git stash push -u -m "temp"
bash -lc "$(jq -r .test_command freeze.json)"

Why stash first?
Because dirty trees hide lucky passes.
I want the freeze to survive a clean tree.

Limitations

A freeze file is not magic.
Hashing a prompt does not freeze a remote model.
The validator cannot see server-side drift.

scratch-pad is an honesty label.
It does not claim isolation or uptime.
It does not claim a service level.

Do not use this approach when you need these.

Production traffic on the same box
Secret-bearing prompts on a shared server
A contractual SLA for inference
A merge that skips local replay

Regulated workflows need a runtime you control.
This catalog will not give you that.
It will only stop you lying to git.

Solo explorers can skip the freeze for toys.
Throwaway katas do not need freeze.json.
Anything you might merge tomorrow does.

What I ship

I ship a patch, a test, and a freeze file.
I do not ship a chat screenshot.
A green chat is still an unpinned run.

If the freeze is missing, I failed.
The model did not fail me first.
Pin the run or admit you vibe-coded.

Your Free Endpoint Is Fine. Your Retry Loop Is the Anti-Pattern.

Sam Chen — Mon, 14 Sep 2026 21:41:55 +0000

Free model access and a free server remove the billing argument. They do not remove the engineering. Your loop is still the part that breaks at 3 a.m.

Here is my anti-pattern catalog for that loop. Four symptoms, four root causes, four replacements. Then a harness you can actually run.

The setup I am assuming

You call a hosted model over an OpenAI-compatible HTTP API.
You run a worker on a small free server instance.
Your job is a loop: plan, call, check, repeat.

MonkeyCode is one option in that space. The operator markets free model access and a free server option. The operator also advertises a promotional token allowance; the figure I was given is 10M tokens. I have not verified that number, and quotas move. Check the current page before you plan around it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Anti-pattern 1: The Retry Storm

Symptom

One bad response triggers five retries. Each retry spawns its own retries. Your worker spends an hour asking the same question.

Root cause

Retry logic lives at three layers. The HTTP client, the SDK, and your loop each think they own the problem. Nobody caps the total.

Replacement

One owner, one budget, full jitter. Count attempts once, across everything.

import random
import time

MAX_ATTEMPTS = 4  # total budget, not per layer

def call_with_budget(fn, attempts=MAX_ATTEMPTS):
    for i in range(attempts):
        try:
            return fn()
        except Exception as err:  # narrow this in real code
            if i == attempts - 1:
                raise
            # Full jitter beats fixed backoff on a shared pool.
            time.sleep(random.uniform(0, min(2 ** i, 8)))
    raise RuntimeError("unreachable")

Add a circuit breaker on top. After N consecutive failures, stop for 60 seconds. Mark the job failed loudly instead of quietly.

Anti-pattern 2: The Sample of One

Symptom

You run the prompt once. It looks right. You call it validated.

Root cause

You measured an output, not a distribution. Shared free capacity is noisy by design.

Replacement

Repeat the call N times and record what actually came back.

import csv, json, os, statistics, time
from urllib import request

BASE   = os.environ["LLM_BASE_URL"].rstrip("/")  # any OpenAI-compatible endpoint
KEY    = os.environ["LLM_API_KEY"]
MODEL  = os.environ["LLM_MODEL"]
N      = int(os.environ.get("N", "20"))
PROMPT = "Return one JSON object with the single key 'ok'."

def one_call(prompt):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.2,
    }).encode()
    req = request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Authorization": f"Bearer {KEY}",
                 "Content-Type": "application/json"},
    )
    t0 = time.perf_counter()
    try:
        with request.urlopen(req, timeout=60) as resp:
            payload = json.loads(resp.read())
            status = resp.status
    except Exception as err:  # 429, timeouts, 5xx all land here
        payload, status = {}, f"error:{type(err).__name__}"
    latency_ms = (time.perf_counter() - t0) * 1000
    text = ""
    if payload.get("choices"):
        text = payload["choices"][0]["message"]["content"] or ""
    return status, latency_ms, text

def schema_ok(text):
    try:
        return isinstance(json.loads(text), dict)
    except Exception:
        return False

def main():
    latencies = []
    with open("runs.csv", "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["i", "status", "latency_ms", "chars", "schema_ok"])
        for i in range(N):
            status, latency_ms, text = one_call(PROMPT)
            latencies.append(latency_ms)
            writer.writerow([i, status, round(latency_ms), len(text),
                             schema_ok(text)])
    latencies.sort()
    print("p50_ms", round(statistics.median(latencies)))
    print("p95_ms", round(latencies[int(len(latencies) * 0.95) - 1]))
    print("max_ms", round(latencies[-1]))

if __name__ == "__main__":
    main()

Run it twice: once in your morning, once in your evening. Compare the two CSVs. If p95 shifts a lot, you learned something the demo never showed you. This snippet uses only the standard library, so you can paste it and go.

Anti-pattern 3: The Pet Server

Symptom

Your loop keeps progress in memory. A restart loses three hours of work.

Root cause

You treat the free instance like a pet. Free capacity is not a promise of uptime, and preemption is normal.

Replacement

Idempotent steps plus external checkpoints. Design for kill -9 at any moment.

import json, os, tempfile

CKPT = "checkpoint.json"

def load_state():
    if not os.path.exists(CKPT):
        return {"step": 0, "done": []}
    with open(CKPT) as f:
        return json.load(f)

def save_state(state):
    fd, tmp = tempfile.mkstemp(dir=".")
    with os.fdopen(fd, "w") as f:
        json.dump(state, f)
    os.replace(tmp, CKPT)  # rename is atomic on the same filesystem

The rule that matters: every step must be safe to run twice. If step 7 charges a card, step 7 needs a key.

Anti-pattern 4: Context Assumed, Not Measured

Symptom

Long input gets truncated. Your agent then answers confidently about text it never saw.

Root cause

You assumed a context size instead of measuring your own payload.

Replacement

Estimate before you send, then fail loudly.

Estimate tokens with len(text) / 4 as a rough heuristic, not a truth.
Refuse requests above your budget. Do not let the server pick for you.
Log the estimate next to the request ID so you can debug later.
Split long documents yourself, and track which chunk produced which claim.

Decision table

Situation	Free model + free server	Paid endpoint	Self-host
Prototype a loop, one developer	Good fit	Fine	Overkill
20-run evaluation harness	Good fit, expect noise	Better for tight p95	Fine
Overnight batch with retries	Workable with checkpoints	Simpler	Fine
Latency-sensitive user traffic	Wrong tool	Usually yes	Possible
Data that cannot leave your network	No	Depends on contract	Yes

Read the left column honestly. Free is excellent for learning a loop's shape. It is a bad place to hide a missing circuit breaker.

Who should not use this approach

You need a written SLA or a fixed quota. Free tiers change without notice.
You process regulated data that cannot go to a third party.
You need guaranteed concurrency for user-facing traffic.
You want a benchmark number you can quote. Run your own, on your workload.

What I would measure before committing

p50 and p95 latency, from your own runs.
Error rate split by status code, not one blended number.
Schema failure rate on structured output.
Recovery time after a forced restart.
Cost per completed job, priced at paid rates for comparison.

Numbers 1 through 4 tell you whether the loop is sound. Number 5 tells you whether the free tier was ever the point.

Where MonkeyCode fits, and where it does not

MonkeyCode is relevant at two steps in this workflow. Free model access lets you build the harness without a payment method. A free server option lets you test checkpoint recovery under real restarts. Both are the cheap part of the exercise, which is exactly why they are useful.

Neither one fixes a retry storm. Neither one makes a single sample meaningful. If you want a place to run the harness above, the free model access and free server option are the two things I would test first.

Keep the loop boring. Count attempts once. Checkpoint every step. Measure twenty runs before you trust one.

Your Free Probe Passed. You Still Measured the Wrong Box

Sam Chen — Sun, 13 Sep 2026 20:13:18 +0000

Your free probe passed on a scratch box. That green badge still measured the wrong machine. A green free endpoint is a cheap probe. It is not a production ship gate. Treat it like a probe, or you ship luck.

I keep watching agents pass on free compute. Then the same patch dies on the real model. Does that gap sound familiar to you? That gap is not a dumb model.

It is an eval-transfer bug in the harness. You measured the wrong machine the whole time.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I treat MonkeyCode as a scratch probe box here. Free model access plus a free server option cover that job. They do not issue a production verdict today. Below is the catalog I now run first.

What I measure instead of vibes

I no longer ask if the agent finished. I ask which frozen identity actually finished running.

An identity is a frozen tuple of fields. It is not a feeling at all.

pinned model endpoint plus exact id
server image plus worktree path
tool schema hash after freeze
fixture input plus named oracle
seed, temperature, and timeout budget

Miss one field and the folklore starts immediately. Would you sign a release with no build sha?

Anti-pattern 1: Model-swap theater

Symptom

The free endpoint returns a clean looking patch. You paste that patch into another sampler. CI goes red on the second machine. You blame the second model without evidence.

Root cause

You treated two samplers as one agent. They share a prompt file on disk. They do not share a decision surface though. Sampling noise is not a stable personality trait.

Replacement

Pin the eval identity before any replay. Replay the same fixture on that identity. Refuse transcript contests across two different endpoints.

# example sketch: freeze identity before the probe
export EVAL_ENDPOINT="$FREE_MODEL_URL"
export EVAL_MODEL_ID="${EVAL_MODEL_ID:?pin this; do not guess}"
export EVAL_IMAGE="agent-sandbox:replay-3"
printf '%s %s %s\n' "$EVAL_ENDPOINT" "$EVAL_MODEL_ID" "$EVAL_IMAGE" >> eval-identity.log

Did you actually log the model id anywhere? If not, you only have a story. Those stories should not merge to main.

Anti-pattern 2: Shared-server amnesia

Symptom

Two agents share one free scratch server. The second run knows files it never wrote. You call that leftover disk real skill.

Root cause

The disk was dirty from the last fixture. Yesterday's files leaked into today's context window. Shared scratch is not true process isolation. Leftover env files feel like sudden genius.

Replacement

Boot a clean worktree per replay run. Destroy that worktree on the way out. No leftovers means no haunted context files.

# example: one worktree, one eval, then burn it
RUN_ID="$(date +%s)-$RANDOM"
git worktree add "/tmp/eval-$RUN_ID" HEAD --detach
trap 'git worktree remove --force "/tmp/eval-$RUN_ID"' EXIT
cd "/tmp/eval-$RUN_ID"
python replay_eval.py --fixture tests/fixtures/refund_timeout.json

Could a leftover lockfile explain the smart call? If yes, you measured the disk instead. You did not measure the agent at all.

Anti-pattern 3: Happy-path burn

Symptom

You spend the free run on the demo. The refund flow looks gorgeous in the screenshot. Production then dies on a conflict response. Nobody saved the ugly case for replay.

Root cause

Free compute feels scarce inside your head. You protect it for polished demo screenshots. Failure cases look ungrateful next to demos. So the probe never sees those cases.

Replacement

Spend the free probe on the ugly path first. Demos are optional during a real eval. Contracts are not optional during a real eval. Invert the queue before you burn the box.

Attack order I actually use:

Timeout from the tool
Duplicate id on retry
Partial JSON in the stream
Empty file that still exists
Only then the happy path

# proposal: order fixtures by how often they lie
FIXTURES = [
    "timeout_on_tool.json",
    "duplicate_idempotency_key.json",
    "partial_json_stream.json",
    "empty_but_present_file.json",
    "happy_refund.json",  # last on purpose
]

def run_probe_queue(runner):
    for name in FIXTURES:
        result = runner.replay(name)
        if not result.oracle_ok:
            raise SystemExit(f"ugly path failed first: {name}")

Would you still ship if step two failed? That is the only question that matters here.

Anti-pattern 4: Latency theater

Symptom

The free endpoint answers faster than production does. You call the agent snappy without an oracle. You skip timeouts in the harness code. Slow production then looks like a model regression.

Root cause

You scored user vibe, not the contract. Fast and correct are different measurement axes. Mixing them hides stalls and silent retries.

Replacement

Record duration as telemetry beside the result. Never treat that clock as a grade. Grade the oracle against the world state. Log the clock next to that grade.

# example: clock is telemetry, not a pass condition
import json, time, urllib.request

def probe(payload: bytes, url: str, timeout_s: float) -> dict:
    started = time.monotonic()
    req = urllib.request.Request(url, data=payload, method="POST")
    try:
        with urllib.request.urlopen(req, timeout=timeout_s) as resp:
            body = json.loads(resp.read().decode())
            status = "ok"
    except TimeoutError:
        body, status = {}, "timeout"
    latency_ms = int((time.monotonic() - started) * 1000)
    return {"status": status, "latency_ms": latency_ms, "body": body}

Did the tool finish, or did you enjoy speed? Separate those answers on paper before merge.

Anti-pattern 5: Transcript-as-proof

Symptom

You read the chat and it sounds sure. You merge the patch after that reading. Tests never saw the side effect land. The ledger still double-refunds the same cents.

Root cause

Language is cheap on a free model. Side effects are not cheap at all. Confidence is not a checksum of state. Prose cannot close a money path safely.

Replacement

Oracle the world after the agent run. Check files, HTTP calls, and ledger rows. Do not grade the essay in CI.

# example: the transcript cannot pass this test
def test_refund_is_idempotent(tmp_path):
    world = World(tmp_path)
    agent_run(world, fixture="refund_twice")
    events = world.ledger.read()
    refunds = [e for e in events if e["type"] == "refund"]
    assert len(refunds) == 1
    assert refunds[0]["cents"] == 4099

If the ledger lies, the speech does not matter. Why are we still grading essays in CI?

The artifact: an Eval Identity Card

I keep one JSON card per probe run. No card means no claim worth publishing. This card is the whole method I use.

{
  "eval_id": "refund-timeout-2026-09-14",
  "endpoint": "free-model",
  "model_id": "pin-me-do-not-guess",
  "server": "free-scratch",
  "worktree": "/tmp/eval-detach",
  "tool_schema_sha256": "replace-with-real-hash",
  "fixture": "tests/fixtures/refund_timeout.json",
  "oracle": "ledger_refund_count_eq_1",
  "seed": 17,
  "temperature": 0,
  "timeout_s": 30,
  "result": null
}

Those nulls are honest before the probe runs. Fill them after the oracle actually returns. Do not prettify empty results to look done.

Here is the checker I want in CI. Label it a proposal, not a benchmark number. It is not a latency claim either.

# proposal: fail the job if the identity drifted
import hashlib, json, pathlib, sys

def schema_hash(path: str) -> str:
    data = pathlib.Path(path).read_bytes()
    return hashlib.sha256(data).hexdigest()

card = json.loads(pathlib.Path("eval_card.json").read_text())
actual = schema_hash("tools/schema.json")
if card["tool_schema_sha256"] != actual:
    sys.exit("schema drifted; this green run is from another world")
if card["result"] is None:
    sys.exit("no oracle result; a transcript is not a gate")

One probe, start to finish

Write the identity card with result left null.
Create a detached worktree and nothing else.
Replay ugly fixtures before the happy path.
Fill the card from the world oracle.
Run the drift checker on that card.
Only then open the chat transcript.

python check_eval_card.py
python replay_eval.py --card eval_card.json

Run both commands from a clean worktree. Two commands pin one identity with no folklore. Can your current log survive that bar today?

Decision table

Use this table before you trust a free run.

Symptom	What you tell yourself	What actually broke	Replacement
Shared files across runs	The agent remembered	Dirty worktree	Detach, then destroy
Green on free, red later	Prod model is worse	Unpinned identity	Replay the same card
Demo looks perfect	We covered the flow	Happy-path burn	Failures first
Fast replies	The agent is tight	Latency theater	Log ms, grade oracles
Confident chat	It understood the spec	Transcript-as-proof	Assert the ledger

Print the table and tape it beside the terminal. Keep it there until the folklore stops.

What this does not prove

A free endpoint can catch cheap obvious lies. It cannot certify a vendor model family. It cannot certify tail latency under load. It cannot certify your IAM or tenancy story.

I am not claiming transfer across model endpoints. I am claiming a cleaner negative test only. If the probe fails, you should not ship. If the probe passes, you still need the real gate. Do not confuse a found bug with a proved release.

Who should skip this

Skip this if you already pin models in CI. Skip this if every run is a throwaway VM. Skip this if your oracle is a real ledger test.

Also skip this if you need a production SLA. A free scratch server is not that SLA. Do not pretend otherwise on a launch checklist. This catalog is for people mixing probes and gates.

The habit I want

Start with the ugly fixture every single time. Pin the identity before the first token. Wipe the disk after the oracle returns. Grade the world, not the chat transcript.

Then, and only then, read the transcript aloud. Does that extra friction feel slower to you? Good, because slower is the entire point here.

Need a disposable box for that replay loop? MonkeyCode's free model access and free server option can host that practice. Treat the run as a probe, not a gate. Keep the ship gate somewhere honest and pinned.

Your Agent Called the Tool. The Contract Still Failed

Sam Chen — Sat, 12 Sep 2026 18:14:56 +0000

A tool call is not a passing test. I refuse that substitution on every trace. Did the payload match a contract you actually own?

Most demos celebrate the call itself. I watch the next turn swallow raw JSON. Then someone ships the session as evidence.

This catalog is about the tool boundary. Not the prompt pack. Not the CI oracle.

The rule I use

If a tool result is unchecked, the run is theater. I want a fixture, an envelope, and a replay. Cheap runs still need that triangle.

Ask a harder question after every call. What field would falsify this turn?

Anti-pattern 1: Status-code theater

Transport success is not semantic success. Agents still treat them as twins. Why do we keep buying that story?

Symptoms

The trace shows 200 and a victory line.
The agent claims the file was written.
Nobody parsed the body before the next tool.
Exit code 0 becomes a patch summary.

Root cause

HTTP status and process exit are transport. Meaning lives in the envelope. The loop never named that envelope.

Replacement

Assert a tiny contract before the next thought. Status, error, and typed result. Fail closed on unknown shapes.

# labeled example: envelope check, not a live service
REQUIRED = ("ok", "error", "result")

def accept_tool_body(body: dict) -> dict:
    missing = [k for k in REQUIRED if k not in body]
    if missing:
        raise ValueError(f"missing {missing}")
    if body["ok"] is True and body["error"] is not None:
        raise ValueError("ok/error conflict")
    if body["ok"] is False and body["result"] is not None:
        raise ValueError("failed call still carried result")
    return body["result"]

I run that check in the harness. The model never sees a raw 200.

Anti-pattern 2: Raw dump into the next turn

This is not the app schema bug. This is the tool envelope bug. Huge difference. Did your last article already cover database fields? This one covers the wire.

Symptoms

The next prompt is a wall of stdout.
The model summarizes noise as facts.
Later tools receive invented ids.
Logs look rich and still prove nothing.

Root cause

The loop concatenates because concatenation is easy. Easy context is not true context. Unknown keys become future arguments.

Replacement

Normalize to a struct. Drop unknown keys. Hash what remains. Pass only fields the next tool may legally consume.

# labeled example: allowlist, then hash
import hashlib, json

ALLOW = {"path", "sha256", "bytes"}

def project_result(result: dict) -> dict:
    slim = {k: result[k] for k in ALLOW if k in result}
    if set(slim) != ALLOW:
        raise ValueError(f"incomplete result {slim.keys()}")
    blob = json.dumps(slim, sort_keys=True).encode()
    slim["digest"] = hashlib.sha256(blob).hexdigest()[:12]
    return slim

I paste digest into the next turn. Not the novel. The model can still reason. It cannot launder junk keys.

Anti-pattern 3: Same-args retry theater

Identical calls look like persistence. They are usually a stuck loop. Does four search_repo hits mean the model thought harder?

Symptoms

Four identical tool names and payloads.
Latency looks like careful thinking.
The writeup says the agent persisted.
The working tree never changed.

Root cause

The loop has no call fingerprint. Retry policy is missing. Skill gets the credit for a spinner.

Replacement

Fingerprint name + canonical args. A second identical call is a bug. Allow it only when the contract says the tool is volatile.

# labeled example: fingerprint gate
import json

class CallGate:
    def __init__(self):
        self.seen = {}

    def check(self, name, args, volatile=False):
        key = name + json.dumps(args, sort_keys=True)
        n = self.seen.get(key, 0) + 1
        self.seen[key] = n
        if n > 1 and not volatile:
            raise RuntimeError(f"duplicate call {name} x{n}")
        return n

I log duplicate call as a harness fail. I do not log it as extra effort.

Anti-pattern 4: Side-effect amnesia

Tools mutate the world. Context is a cache. Did anyone invalidate the named artifact?

Symptoms

A tool wrote a file.
A later read uses an old snapshot.
The agent "fixes" the stale text again.
Diffs oscillate across two versions.

Root cause

The prompt still holds the pre-mutation copy. The model reconciles against a ghost. Your eyes follow the ghost too.

Replacement

After every mutating tool, refresh the named artifact. Version it. Feed the new digest, not the memory.

# labeled example: refresh after a mutating tool
set -euo pipefail
TARGET="src/bill.py"
BEFORE=$(sha256sum "$TARGET" | awk '{print $1}')
# agent tool would edit here
AFTER=$(sha256sum "$TARGET" | awk '{print $1}')
if [ "$BEFORE" = "$AFTER" ]; then
  echo "mutation claimed, digest unchanged" >&2
  exit 1
fi
echo "artifact $TARGET digest $AFTER"

I treat an unchanged digest as a failed mutation. The model does not get a second speech about it.

Anti-pattern 5: Single-session folklore

One chat is a story. An eval is a replay. Did you freeze the tool server first?

Symptoms

One model session becomes the report.
No fixture server exists.
"It worked on my run" is the metric.
Tomorrow the same prompt drifts.

Root cause

People confuse availability with evidence. A free lane is useful. It is not a regression suite by itself.

Replacement

Replay against a frozen tool server. Same args. Same expected envelope. Store the transcript as a fixture, not as folklore.

# labeled example: replay one frozen turn
import json
from pathlib import Path

def replay_turn(path: str, handler):
    rec = json.loads(Path(path).read_text())
    got = handler(rec["name"], rec["args"])
    if got != rec["expected"]:
        raise AssertionError(f"{path}: {got} != {rec['expected']}")
    return got

I keep those JSON files next to the tests. A green chat window does not land in that folder.

Artifact: a contract fixture you can run

Here is a tiny fake tool server. It is local. It is boring. Boring is the point.

# labeled example: fixture_server.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

TOOLS = {
    "write_note": {
        "ok": True,
        "error": None,
        "result": {"path": "notes/a.md", "sha256": "abc", "bytes": 12},
    }
}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        payload = json.loads(self.rfile.read(n) or b"{}")
        name = payload.get("name")
        body = TOOLS.get(name)
        code = 200 if body else 404
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(body or {"ok": False, "error": "unknown", "result": None}).encode())

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run the server. Then hit it with a contract client. Do not skip the accept function.

python fixture_server.py &
curl -sS -X POST http://127.0.0.1:8765 \
  -H 'Content-Type: application/json' \
  -d '{"name":"write_note","args":{"path":"notes/a.md"}}'

I also keep a decision table beside the fixture. Use it before you trust a run.

Signal in the trace	What I still do not know	Gate that must pass
HTTP 200	Envelope validity	`accept_tool_body`
Long stdout	Which keys are legal	`project_result`
Repeated tool name	Whether args changed	`CallGate.check`
"I wrote the file"	Whether bytes moved	digest before/after
One pretty session	Whether it replays	frozen `replay_turn`

Where a free lane actually helps

I use a scratch lane for the fixture, not for folklore. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are a place to exercise that triangle when I do not want a production box in the loop.

The product does not replace the contract. It only hosts a cheap run. If the envelope check is missing, a free server still lies politely.

Limitations

This catalog will not save a bad problem statement. Contract tests need an owner. They rot when tools change and fixtures do not.

Volatile tools break the duplicate-call gate. You must mark them volatile on purpose. Networked third-party APIs need recorded cassettes, not live hope.

I am not claiming model rankings here. I am not citing quotas, hardware, or duration. Those numbers go stale overnight. The envelope does not.

Who should not use this

Skip this if you only generate one-off snippets. Skip it if no tool can mutate state. Skip it if you already replay frozen envelopes in CI.

Do not use a shared scratch server for secrets. Do not point a fixture at production data. Do not treat my labeled examples as a security boundary.

What I do on the next trace

I ask five questions in order. Did transport get confused with meaning? Did raw stdout enter the next turn? Did identical args retry? Did a mutation keep a stale snapshot? Did one session become the report?

If any answer is yes, I stop blaming the model. I fix the tool boundary first. Then I replay. Then I talk about the patch.

Five Workspace Anti-Patterns That Fake Agent Skill

Sam Chen — Fri, 11 Sep 2026 15:48:11 +0000

Most so-called coding-agent wins are workspace luck. The model looks sharp because the tree was dirty. Did the agent reason well, or did leftover files help?

I stop blaming the model on first contact. I audit the working tree before I touch prompts. This catalog is about workspace contracts, not model lore.

Each entry lists symptoms, a root cause, and a replacement. Steal the gates. Skip the slogans. Ask one question before you swap vendors.

Was the workspace even bounded this time?

How to read each entry

Use one anti-pattern per incident, not five. Stacking fixes hides the real leak. I treat these gates like tests, not vibes.

If a demo only works on a warm directory, it is not a demo. Cold clone it. Then talk about models.

Anti-pattern 1: Live-tree edits

The agent writes straight into your daily checkout. No branch. No worktree. No disposable clone. Failure and success share the same files.

Symptoms

git status turns noisy after a "small" task
Demo files mix with your real local changes
A failed run leaves half a refactor behind

Root cause

You treated the repo as a chat scratchpad. The agent had no isolation boundary. Every tool call could own main.

Why should a bad patch keep your uncommitted work hostage?

Replacement

Give every run a throwaway worktree. Apply patches only after a gate passes. Keep your daily checkout read-only for the agent.

RUN_ID=$(date +%s)
git worktree add -b "agent/run-$RUN_ID" "/tmp/wt-$RUN_ID" HEAD
cd "/tmp/wt-$RUN_ID"
# agent may edit only this path

I refuse runs that cannot name their worktree path. If you cannot delete the directory, you do not own the experiment.

Anti-pattern 2: Unbounded glob

The agent searches the tree like a disk dump. It reads secrets, lockfiles, and vendor junk. Context fills with files no human would open.

Symptoms

Tool logs show .env, node_modules, or .git
The model "discovers" keys you never pasted
Answers cite minified files you do not maintain

Root cause

Read tools default to convenience. Convenience is not an access policy. A shared machine does not make that safer.

Would you let an intern cat the whole laptop?

Replacement

Deny by default. Allow one path prefix. Reject reads that match a deny list. Log every blocked path, not only the allowed ones.

# proposal: a path gate, not a sandbox claim
from pathlib import Path

ALLOW_PREFIX = Path("src").resolve()
DENY_PARTS = (".env", ".git", "node_modules", "id_rsa")
DENY_SUFFIX = (".pem", ".p12", ".key")

def allowed(raw: str) -> bool:
    path = Path(raw).resolve()
    try:
        path.relative_to(ALLOW_PREFIX)
    except ValueError:
        return False
    text = path.as_posix()
    if any(part in text.split("/") for part in DENY_PARTS):
        return False
    return not text.endswith(DENY_SUFFIX)

I want the blocked-path list in the transcript. Hidden denies teach nothing when the next run fails.

Anti-pattern 3: Partial writes as progress

The agent writes file A, then dies on file B. You keep A because tests looked "almost green." Drift becomes the new baseline.

Symptoms

One test file changed, the production file did not
Imports point at functions that do not exist yet
Re-runs stack more half-patches on the wreckage

Root cause

You scored steps, not transactions. A coding agent is a patch applicator. Half a patch is a fresh bug with extra confidence.

Is a half-merge a feature? It is not.

Replacement

Stage one unified diff. Apply it atomically. Reject the whole patch when any gate fails. Do not bargain with leftover hunks.

# proposal workflow: check, then apply, else restore
git diff --binary > /tmp/agent.patch
git checkout -- .
git apply --check /tmp/agent.patch
git apply /tmp/agent.patch

I also keep the patch file beside the logs. If apply fails, the artifact is the patch, not a chat recap.

Anti-pattern 4: No revert oracle

The run ends. You eyeball the diff and shrug. Nobody can restore the pre-run tree without another model call.

Symptoms

"Undo that" means more agent chat, not git
You cannot name the parent commit for the run
CI looks green on a dirty local state

Root cause

You logged prompts. You did not log trees. Transcripts are not recovery tools. Porcelain status is.

Can you rewind without begging the model to remember?

Replacement

Snapshot before tools. Revert on red. Record HEAD and git status --porcelain as first-class outputs.

BEFORE=$(git rev-parse HEAD)
git status --porcelain > /tmp/status.before
# ... agent run ...
if ! ./gates.sh; then
  git reset --hard "$BEFORE"
  git clean -fd
  echo "reverted to $BEFORE"
fi

Read that reset --hard line twice. It will destroy uncommitted work if you point it at the wrong tree.

Anti-pattern 5: Shared eval workspace

Several runs share one server directory. Yesterday's fixtures become today's context. You think the model improved. It remembered a file.

Symptoms

The agent "knows" files you never fetched
Wins vanish on a cold clone of the same commit
Two tasks overwrite the same fixture path

Root cause

A cheap remote directory feels like a playground. Playgrounds get contaminated. You evaluated residue, not the agent.

Did it pass because leftover fixtures whispered answers?

Replacement

One run, one directory, one wipe. Cold clone or a fresh worktree every time. Delete on exit, even after green.

RUN_DIR=$(mktemp -d)
git clone --depth 1 "$REPO" "$RUN_DIR"
trap 'rm -rf "$RUN_DIR"' EXIT
cd "$RUN_DIR"

I do not trust a win I cannot delete and recreate. If step six in the plan still "remembers" files, stop scoring the model.

Artifact: gates, a table, and a test plan

This is a proposed checklist, not a published benchmark. I am not attaching timings, hardware, or token counts. Run it as a human-labeled test plan.

Decision table

Signal you actually saw	Likely anti-pattern	Do this first
Dirty `main` after a chat	Live-tree edits	New worktree
Secrets in tool traces	Unbounded glob	Prefix plus deny list
Missing symbols after a "win"	Partial writes	Atomic `git apply`
Cannot restore files	No revert oracle	Hard reset on red
Win disappears on a cold clone	Shared eval workspace	Fresh directory

Minimal `gates.sh`

#!/usr/bin/env bash
set -euo pipefail

if [[ -n "$(git status --porcelain)" ]]; then
  echo "refuse: dirty tree"
  exit 1
fi

git diff --check
[[ -f /tmp/status.before ]] || {
  echo "refuse: missing preflight status"
  exit 1
}

Wire this after apply, not after a vibe check. A green model reply is not a gate.

Test plan I actually run

Start from a clean clone of a known commit.
Inject a fake .env outside src/.
Confirm the path gate blocks that read.
Build a two-file patch with one bad hunk.
Confirm git apply --check rejects the whole patch.
Confirm the tree matches HEAD after the reject.
Re-run the same task in a second directory.
Confirm leftover fixtures cannot leak into that run.

If step 7 still passes only on the warm directory, you tested contamination. You did not test the agent.

Where cheap inference actually helps

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use cheap inference to rerun gates, not to skip them. MonkeyCode's free model access and free server option are useful as a disposable place to repeat the checklist. The server is another workspace. It is not a policy engine.

Do not treat free compute as a license to share trees. These anti-patterns still apply on a free server. If the path gate fails on your laptop, it fails remotely too.

The only check I care about is boring. Can a stranger clone reproduce the win without your leftovers?

If you try the gates there, paste the porcelain log. Do not paste a model nickname.

Limitations

This catalog does not rank models. It does not claim quotas, uptime, or hardware. It will not save a repo that has no tests.

Skip this approach if you need any of the following.

Formal multi-tenant isolation with a real sandbox
A workflow that cannot use git worktrees
In-place edits on production checkouts
The model itself acting as the security boundary

A path prefix is not a seccomp jail. git reset --hard destroys work you forgot to commit. Atomic apply still needs tests that mean something.

What I run instead of another prompt tweak

Bound the tree with a worktree or clone.
Deny secret paths before the first read.
Apply one atomic patch, or apply nothing.
Revert on red using a recorded HEAD.
Throw the directory away after the score.

Then, and only then, judge the model. Was the last "agent failure" a model failure? Or did you hand it main and a glob?

Your Agent Assumed the Schema. That Was the Bug

Sam Chen — Thu, 10 Sep 2026 04:23:04 +0000

The model did not invent that broken payload. Your tool contract left a hole. I still watch teams debug the wrong layer first. They swap models before they freeze a single field. They add system prompts instead of required keys. Freeze the schema first. Then judge the model.

Why start with contracts, not prompts? Free models fail in a loud way. They will not hide a sloppy tool contract. That loud failure is a useful feature today. Use it before you spend a paid token.

What this piece actually covers

This is a catalog, not a launch post. Four anti-patterns. Symptoms. Root cause. A replacement you can ship. Then a tiny contract suite you can run tonight.

I mention a free model server once, late. It is only a cheap place to fail. Skip that part if you already own an endpoint.

Anti-pattern 1: Optional everything

Symptom

The tool schema marks every field optional. The agent still reports a clean success. Your logs show three shapes for one tool. Downstream code then parses a maybe-object.

Ask yourself a blunt question here. Did the model pick a field? Or did your schema refuse to choose?

Root cause

You copied a fat OpenAPI dump into the tool list. Optional fields feel safer than required ones. The model fills that vacuum with a confident guess. Your harness accepts the guess because JSON still parses.

Replacement

Mark the real inputs required, not polite. Split rare fields into a second tool. Reject extra keys. Do not coerce them into defaults.

{
  "name": "repo_search",
  "parameters": {
    "type": "object",
    "additionalProperties": false,
    "required": ["query", "path"],
    "properties": {
      "query": { "type": "string", "minLength": 2, "maxLength": 200 },
      "path": { "type": "string", "pattern": "^src/" },
      "limit": { "type": "integer", "minimum": 1, "maximum": 20, "default": 8 }
    }
  }
}

See additionalProperties: false in that block? That one line kills a whole drift class. If a field is truly rare, it is another tool.

Anti-pattern 2: Stringly-typed tools

Symptom

One parameter. Type string. You named it payload or json. The model pastes a novel into it. Your parser throws on the second brace. Then you blame the weights.

Sound familiar in your last trace?

Root cause

Nested objects felt annoying inside the harness. A string always “validates” at the edge. Real validation moved to runtime, after the call. The model never saw the shape you actually needed.

Replacement

Declare the object in the tool list. Validate arguments before any HTTP call. If you need an escape hatch, version that hatch.

# labeled: proposal for your own harness
REQUIRED = {"query", "path"}
ALLOWED = {"query", "path", "limit"}

def assert_tool_args(name: str, args: dict) -> None:
    if name != "repo_search":
        raise AssertionError(f"unknown tool: {name}")
    missing = REQUIRED - set(args)
    if missing:
        raise AssertionError(f"missing: {sorted(missing)}")
    if not isinstance(args["query"], str):
        raise AssertionError("query must be str")
    extra = set(args) - ALLOWED
    if extra:
        raise AssertionError(f"extra keys: {sorted(extra)}")

Run this guard before the network hop. Not after the model “looks right” in a screenshot.

Anti-pattern 3: HTTP 200 with a buried error

Symptom

The tool returns status 200 every time. The body still contains "error": "not found". The agent writes a patch against empty hits. Your diff looks busy. Your tests stay red.

Why would the loop stop there? You labeled the call a success.

Root cause

You wrapped every exception in a friendly JSON blob. The model reads tone, not protocol. A 200 is a green light with extra words. Retries then amplify the wrong branch.

Replacement

Map failures to explicit tool errors. Keep a tiny error taxonomy. Four codes are enough for file tools.

Code	Meaning	Agent should
`INVALID_ARGS`	Schema failed	Fix args, retry once
`NOT_FOUND`	Path or query missed	Ask, do not invent files
`FORBIDDEN`	Path outside allowlist	Stop the loop
`TRANSIENT`	Timeout or lock	Retry with backoff

class ToolError(Exception):
    def __init__(self, code: str, message: str):
        self.code = code
        self.message = message

def run_repo_search(args: dict) -> dict:
    try:
        assert_tool_args("repo_search", args)
    except AssertionError as exc:
        raise ToolError("INVALID_ARGS", str(exc)) from exc
    path = args["path"]
    if not path.startswith("src/"):
        raise ToolError("FORBIDDEN", "path must start with src/")
    hits = search(args["query"], path, args.get("limit", 8))
    if not hits:
        raise ToolError("NOT_FOUND", "no files matched")
    return {"hits": hits, "count": len(hits)}

Return data or raise a coded error. Never both in one body.

Anti-pattern 4: Unbounded tool dumps

Symptom

One read_file call returns twelve thousand lines. The next model turn is garbage. You call it context rot in the standup. Is it rot, or did you hose the window?

Root cause

The tool has no byte budget at all. The model cannot refuse a huge result. Your loop then “summarizes” the damage with another call. Tokens vanish. The schema still looks fine.

Replacement

Cap bytes on every read. Return a cursor. Force a second, smaller call.

MAX_BYTES = 8_000

def read_slice(path: str, offset: int = 0) -> dict:
    if ".." in path or path.startswith("/"):
        raise ToolError("FORBIDDEN", "path not allowed")
    with open(path, "r", encoding="utf-8") as handle:
        data = handle.read()
    chunk = data[offset: offset + MAX_BYTES]
    return {
        "path": path,
        "offset": offset,
        "bytes": len(chunk.encode("utf-8")),
        "truncated": (offset + MAX_BYTES) < len(data),
        "next_offset": offset + len(chunk),
        "text": chunk,
    }

Truncation is a signal the agent can use. Hide it, and the agent keeps guessing missing lines.

The artifact: contract tests, not a vibe check

I do not “try a prompt” to prove tools work. I freeze three fixtures and fail the build. No model sits in this loop. If this suite is red, stop blaming tokens.

# labeled: reproducible contract checks
CASES = [
    {
        "name": "missing_path",
        "args": {"query": "ToolError"},
        "expect": "INVALID_ARGS",
    },
    {
        "name": "path_escape",
        "args": {"query": "foo", "path": "../secrets"},
        "expect": "FORBIDDEN",
    },
    {
        "name": "empty_hits",
        "args": {"query": "zzz-no-hit", "path": "src/"},
        "expect": "NOT_FOUND",
    },
]

def classify(exc):
    if isinstance(exc, ToolError):
        return exc.code
    raise

def test_tool_contracts():
    for case in CASES:
        try:
            run_repo_search(case["args"])
        except Exception as exc:
            got = classify(exc)
            assert got == case["expect"], (case["name"], got)
            continue
        raise AssertionError(f"{case['name']} should fail")

Three cases cover the lies I see most. Missing required keys. Path escape. Empty result treated as success. Add your own poison next.

Want a command, not a story?

python -m pytest test_tool_contracts.py -q

Green means the runtime tells the truth. Red means the agent never had a chance.

Where a free model server actually helps

Need a live model after the suite is green? I use a free endpoint for that next layer.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. I treat that box as a schema furnace, not a demo stage. Same tools. Same byte caps. Same error codes. If the free model invents keys, the contract is still too loose. If it stops cleanly, the schema is doing real work.

I am not claiming a quota, a ranking, or a hardware spec. I am claiming a workflow you can copy. Fail on a free server first. Pay only after traces stay boring.

# labeled: local workflow; swap in your endpoint
python -m pytest test_tool_contracts.py -q
python replay_traces.py --endpoint "$FREE_SERVER_URL" --limit 20

Replay frozen traces, not vibes from chat. Twenty pinned tasks beat one flashy gist.

Decision table I keep beside the harness

If you see	Do not	Do instead
Extra JSON keys	Add a prompt rule	Set `additionalProperties: false`
Parse errors on `payload`	Switch models	Kill stringly-typed tools
Patches on 200 plus error	Write “be careful”	Raise `ToolError` codes
Context explosions	Summarize harder	Cap bytes and return a cursor
Flaky retries	Raise temperature	Split `INVALID_ARGS` from `TRANSIENT`

Print the table. Stick it on the harness. Argue with the row, not the model card.

Limitations

This will not fix a bad retrieval index. This will not catch prompt injection. This will not make a weak model design a new architecture. Contract tests only prove the tool told the truth.

The suite assumes you own the tool runtime. It assumes a single-process harness on disk. It assumes file tools, not billing APIs with side effects. Schemas drift across pull requests. Re-run the suite on every tool change. A green run yesterday is not a green run today.

Who should not use this

Do not use this if you cannot pin tool versions. Do not use this if every tool is a raw shell. Do not use this if you need multi-tenant isolation tomorrow. Those are product problems, not schema nits.

If your agent is a cron job with two branches, stop. You do not have a schema problem. You have a script. Keep the script. Skip the catalog.

What I want you to run tonight

Pick one tool. Only one. Make three fields required. Add additionalProperties: false. Write three fixtures that must fail. Then, if you want a cheap live check, point the same harness at a free MonkeyCode server and replay traces.

That is the whole move. The model was never the first bug.

Green CI, Bad Patch: Five Oracle Anti-Patterns

Sam Chen — Wed, 09 Sep 2026 16:02:28 +0000

Your agent did not succeed, and your oracle lied. I keep watching green checks land on broken diffs. The suite never asked the right question, did it?

This is not another agent-loop post from me. This post is about the judge you trust. If the judge is cheap, every model looks brilliant.

The point

Stop swapping endpoints. Fix the success check first.

I want a judge I can rerun tomorrow. Same input. Same fail. No vibes allowed.

What I mean by an oracle

An oracle decides this coding run worked. It is not the model. It is not the prompt either.

It is the gate you actually trust. Most teams skip this gate. Then they argue about models. Wrong fight, every single time.

How to read this catalog

I use three fields. Every time. No extra theater.

Symptom: what the log shows
Root cause: why the gate is lying
Replacement: what I run instead

Steal the replacements. Leave the excuses on the floor.

1. The exit-code oracle

Symptom: pytest returns zero. You merge.

Root cause: zero only means the process lived. It does not mean tests ran. Collection can fail open. Markers can drop every case.

Skips look like wins. Empty suites look like wins. That is the trap.

Replacement: assert collected, failed, and skipped counts. Empty collection is a red gate.

# proposed gate — unexecuted example
import json, subprocess

def collect_pytest():
    p = subprocess.run(
        ["pytest", "-q", "--collect-only", "-p", "no:cacheprovider"],
        capture_output=True, text=True,
    )
    return p.returncode, p.stdout

Collection is lock one. I still run the suite.

def assert_pytest_really_ran(report_path="report.json"):
    data = json.loads(open(report_path).read())
    summary = data["summary"]
    collected = summary.get("collected", 0)
    skipped = summary.get("skipped", 0)
    failed = summary.get("failed", 0)
    if collected == 0:
        raise SystemExit("oracle-fail: zero tests collected")
    if skipped == collected:
        raise SystemExit("oracle-fail: every test skipped")
    if failed:
        raise SystemExit(f"oracle-fail: {failed} failed")

Did your last green run collect anything real? Check the counts before you brag.

2. The snapshot the model wrote

Symptom: golden files match. Reviewers relax.

Root cause: the agent wrote the snapshot too. The test now certifies the bug. You froze the hallucination in git.

Replacement: freeze expected output from humans. The agent may read fixtures. It may not rewrite them in the same turn.

Who wrote the expected file	Merge oracle?
Human, before the agent ran	Yes
Agent, same turn as the patch	No
Agent, later, after a failing test	Only with a human ack

I keep fixtures in testdata/frozen/. That directory is read-only in the tool schema. Simple rule. Hard to dodge on purpose.

3. The compile oracle

Symptom: tsc --noEmit is clean. Ship it?

Root cause: types do not encode behavior. A function can typecheck and still drop rows. Generated casts hide the hole.

Replacement: one behavioral probe per claimed contract. Compile is lint. It is not a verdict.

# proposed probes — fill with your real domain
CONTRACTS = [
    ("refund_total", {"items": [10, 2], "tax": 0}, 12),
    ("refund_total", {"items": [], "tax": 1}, 0),
]

If I cannot write that table, I do not have a task. I have a vibe. Would you merge a vibe from a stranger?

4. The small-diff oracle

Symptom: twelve lines changed. Feels safe.

Root cause: blast radius is not line count. One line can swap auth. One line can widen a query.

Agents love tiny lethal patches. Reviewers love tiny diffs. That pairing is how bugs ship.

Replacement: classify the diff. Fail on auth, money, delete, and schema paths. Unless a human flag is set.

# proposed pre-merge classify
git diff --name-only origin/main...HEAD \
  | rg "auth|payment|migration|schema|delete" && echo "needs-human"

Ask yourself one question. Would I accept this from a stranger on the internet?

5. The self-reported done oracle

Symptom: the last message says "fixed." You stop.

Root cause: the model grades its own homework. Completion text is not evidence. Confident prose is still prose.

Replacement: ignore the narration. Read artifacts only. This is the only success packet I accept.

{
  "tests_collected": 14,
  "tests_failed": 0,
  "tests_skipped": 0,
  "forbidden_paths_touched": [],
  "behavior_probes_failed": 0,
  "agent_said_done": true
}

agent_said_done is telemetry. It is never the gate. Ever. Why are you still parsing the chat?

A reproducible oracle pack

I want one command. Laptop and remote shell. Same fail.

Proposed layout:

oracle_pack/
  gate.py
  contracts.json
  frozen/
  report.json

gate.py fails closed. Missing keys fail. Missing files fail.

#!/usr/bin/env python3
"""Proposed oracle pack. Wire it to your runner."""
from pathlib import Path
import json

FORBIDDEN = ("auth", "payment", "migration", "schema")

def load_report(path: Path) -> dict:
    if not path.exists():
        raise SystemExit("oracle-fail: missing report.json")
    return json.loads(path.read_text())

def main() -> None:
    report = load_report(Path("report.json"))
    required = (
        "tests_collected",
        "tests_failed",
        "tests_skipped",
        "touched_paths",
        "behavior_probes_failed",
    )
    for key in required:
        if key not in report:
            raise SystemExit(f"oracle-fail: missing {key}")
    if report["tests_collected"] < 1:
        raise SystemExit("oracle-fail: nothing collected")
    if report["tests_failed"] != 0:
        raise SystemExit("oracle-fail: tests failed")
    if report["tests_skipped"] == report["tests_collected"]:
        raise SystemExit("oracle-fail: all skipped")
    if report["behavior_probes_failed"] != 0:
        raise SystemExit("oracle-fail: behavior probe")
    touched = report["touched_paths"]
    hits = [p for p in touched if any(f in p.lower() for f in FORBIDDEN)]
    if hits:
        raise SystemExit(f"oracle-fail: human-required paths {hits}")
    print("oracle-ok")

if __name__ == "__main__":
    main()

Run it like this.

python3 gate.py
echo $?

I keep report.json as the only input. The agent may emit logs. The gate does not parse vibes.

Debug the lying oracle first

When a run "succeeds" and the product still breaks, I do not swap models. I interrogate the judge.

Print collected, failed, skipped. Not the exit code.
Diff testdata/frozen/ against git HEAD.
Re-run probes on a clean checkout of the patch.
List touched paths. Mark forbidden ones.
Drop the assistant transcript from the packet.

If step two shows the agent edited goldens, I already have my bug. Why would a new endpoint fix that?

Why free endpoints make lying oracles louder

Paid models hide a bad judge with extra luck. Free endpoints usually do not.

You see more refusals. You see more partial patches. You see more "done" messages with empty diffs. That noise is useful. It is not a model verdict.

If I change the model and the same gate.py flips, then I may have a model issue. If I change the model and the gate never ran, I learned nothing. Which one were you doing last week?

Where a free coding server actually helps

I still need a cheap place to exercise the gate. Not to skip it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode shows up here for two operator-supplied facts only. Free model access. A free server option. I do not treat those as a quality claim.

The workflow stays boring on purpose.

Freeze contracts and fixtures on my machine.
Point the agent at the free server.
Let it patch a throwaway branch.
Copy report.json back.
Run gate.py locally. Same rules.

Why split the judge? Because the agent can rewrite an in-band oracle. Call that judge-in-band if you want a sixth name. Keep the gate off the model's host.

Does this prove a free model is good? No. It proves my definition of good is stable.

If you want to rehearse that split, MonkeyCode's free model access and free server are enough. That is the only invite.

Test plan I write down

No dashboard. One table. Fill it per task.

Step	Command or check	Pass means
Collect	`pytest --collect-only`	count > 0
Execute	`pytest` with a JSON report	failed = 0, skipped < collected
Behavior	run `contracts.json` probes	every row matches
Diff class	path allow-list	no forbidden path without a flag
Judge host	`gate.py` on my laptop	exit 0

If a cell is blank, I do not have a result. I have a story. Which cell is blank on your last AI PR?

Limitations

This pack will annoy you. Good.

It blocks refactors that touch auth by design. It misses logic bugs outside the probe table. It assumes you can write fixtures. It assumes a shell and pytest.

Who should not use this approach?

People generating sketches with no merge path
Teams with zero tests to freeze
Folks who need formal proofs, not gates
Anyone hoping a free endpoint replaces review

A free server does not make a weak oracle honest. It only makes an honest oracle cheap to rerun.

What I will not claim

I will not quote latency numbers. I will not quote token caps. I will not name models I did not pin.

Those figures rot by next week. These anti-patterns do not.

If your last merge was "the model said done," stop. Replace the oracle first. Then pick an endpoint.

Debug the Context Pack Before You Debug the Model

Sam Chen — Tue, 08 Sep 2026 03:35:43 +0000

The model did not fail your last coding task. Your context pack failed that task instead. I keep seeing the same six packing mistakes. They still look like ordinary model quality problems. They are packing errors and not weight errors.

Want the real failure point right now? Watch the files you actually send. Then watch the files you quietly skip. The miss is rarely sitting in the weights. The miss is sitting in the pack.

What I mean by a context pack

A context pack is the snapshot the model sees. It holds files, diffs, commands, and constraints. It is not your entire laptop state. It is not last week's chat transcript. It is only that frozen snapshot.

Did you record that snapshot at all? Most debugging sessions never do. Then the postmortem blames the model again. That blame is usually the first leak.

How to read this catalog

Each anti-pattern has three tight parts.

Symptom: what you observe during the live session
Root cause: the packing mistake hiding underneath
Replacement: the pattern I actually run instead

I am not ranking hosted models in this piece. I am ranking the packs we feed them.

Anti-pattern 1: The Whole-Repo Dump

You archive the repository and paste the tree. You hope the model finally has everything. That completeness still feels like real engineering discipline.

Symptom: The answer cites dead folders without shame. Latency climbs while the useful patch misses. The edit lands in the wrong package.

Root cause: You treated complete as helpful input. Complete is mostly noise in this loop. Noise drowns the one sibling file that matters.

Replacement: Pack by blast radius around the failure. Start from the failing test file. Add the implementation file it imports. Add one neighbor, then stop packing.

Can you name every file in the pack? If you cannot, the pack is already too fat.

Anti-pattern 2: The Lucky File

You send only app.py because it compiles today. The regression actually lives in app_utils.py. You never packed the file that scared you.

Symptom: The model rewrites a function that is innocent. The original failing test still fails hard. You call the model weak and stop.

Root cause: You packed the file you already understand. You skipped the file you did not want. Comfort is not a serious retrieval strategy here.

Replacement: Pack the failing assertion before any implementation. Then pack every import that assertion touches. Then stop packing again on purpose.

Which file are you avoiding right now? Send that feared file next.

Anti-pattern 3: Chat as Source of Truth

The model proposed a patch ten turns ago. You kept talking without freezing the tree. Nobody wrote down the bytes that were sent.

Symptom: Fresh advice targets symbols you already deleted. Imports point at names from an old branch. You feel gaslit by a helpful tone.

Root cause: The chat drifted while the pack stayed stale. You never wrote a pack manifest. Debugging then becomes folklore instead of honest engineering.

Replacement: Write a manifest before every model call. Hash each file and record the command. Refuse to debug a session without that file.

Anti-pattern 4: The Orphan Diff

The model returns a clean looking unified diff. You never apply it to a fresh worktree. You eyeball the patch inside the chat bubble.

Symptom: The patch looks right beside the explanation. git apply --check fails on first contact. Hidden context never made the pack.

Root cause: Review happened on prose, not on git. Chat remains a terrible merge tool here. The orphan diff never paid rent in git.

Replacement: Apply the patch on a throwaway branch. Re-run the exact failing command. Review git diff and nothing else.

Did you apply the diff before arguing quality? If not, you reviewed a story.

Anti-pattern 5: Token Theatre

You pad the pack with README, LICENSE, and lockfiles. The prompt looks thorough and strangely official. That padding is theatre and not signal.

Symptom: The model recites slogans from your README. It never touches the flaky test line. You burned a turn on branding copy.

Root cause: You optimized for looking complete and serious. You did not optimize for the failing line. Ceremony is not a substitute for context.

Replacement: Keep an explicit deny list of junk paths. Fail the packer when those paths appear. Make that failure loud in local CI.

Why did the model quote your license header? Because you packed that header.

Anti-pattern 6: Review in the Bubble

You ask the same model if the patch is safe. It agrees with its own earlier story. You ship on that warm agreement.

Symptom: Production hits a case the chat never saw. The postmortem blames "AI generated code" again. The bubble never contained the real users.

Root cause: Author and reviewer shared one context window. That window cannot surprise itself honestly. Agreement inside one window is not evidence.

Replacement: Split generation and review into two packs. Keep the review pack smaller and meaner. Use a checklist the model did not write.

Who reviews the reviewer in your loop? If nobody, the bubble ships.

The replacement workflow I actually run

I run one boring loop on purpose. That boredom is the entire point of the loop.

Capture the exact failing command output.
Build a pack from that failure only.
Write pack_manifest.json with file hashes.
Call the model with only that pack.
Apply the patch on a clean branch.
Re-run the same failing command.
Keep the patch or revert immediately.

Need a remote model for step four? A free model endpoint belongs there when you do not want a local GPU. MonkeyCode fits that narrow slot because it offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I only need those two things from that stack. I need free model access for the generation call. I also need a free server option for the endpoint. I do not need a story about permanence or quotas. I need a repeatable call against a frozen pack.

Artifact: a packer that can fail

The script below is a worked example. I am not claiming production metrics. Run it in a throwaway clone only.

#!/usr/bin/env python3
"""Blast-radius context packer. Worked example, not a benchmark."""

from __future__ import annotations

import argparse
import hashlib
import json
from pathlib import Path

DENY_NAMES = {
    "license",
    "license.md",
    "copying",
    "readme.md",
    "package-lock.json",
    "yarn.lock",
    "pnpm-lock.yaml",
    "poetry.lock",
    "cargo.lock",
}
SKIP_DIRS = {
    ".git",
    ".hg",
    ".svn",
    "node_modules",
    "__pycache__",
    ".venv",
    "venv",
}


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def is_denied(path: Path) -> bool:
    return path.name.lower() in DENY_NAMES


def iter_files(root: Path):
    for path in root.rglob("*"):
        if not path.is_file():
            continue
        if any(part in SKIP_DIRS for part in path.parts):
            continue
        if is_denied(path):
            continue
        yield path


def score_file(path: Path, failing: Path) -> int:
    score = 0
    if path == failing:
        score += 100
    if path.parent == failing.parent:
        score += 20
    if path.suffix == failing.suffix:
        score += 5
    left = path.stem.replace("test_", "")
    right = failing.stem.replace("test_", "")
    if left == right:
        score += 40
    return score


def build_pack(root: Path, failing: Path, limit: int = 6) -> dict:
    ranked = sorted(
        iter_files(root),
        key=lambda p: score_file(p, failing),
        reverse=True,
    )
    chosen = []
    for path in ranked:
        if len(chosen) >= limit:
            break
        if score_file(path, failing) <= 0:
            continue
        chosen.append(
            {
                "path": str(path.relative_to(root)),
                "sha256": sha256_bytes(path.read_bytes()),
                "bytes": path.stat().st_size,
            }
        )
    return {
        "root": str(root.resolve()),
        "failing": str(failing.relative_to(root)),
        "files": chosen,
    }


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", type=Path, required=True)
    parser.add_argument("--failing", type=Path, required=True)
    parser.add_argument("--out", type=Path, required=True)
    parser.add_argument("--limit", type=int, default=6)
    args = parser.parse_args()
    root = args.root.resolve()
    failing = (root / args.failing).resolve()
    pack = build_pack(root, failing, limit=args.limit)
    args.out.write_text(json.dumps(pack, indent=2), encoding="utf-8")
    print(f"wrote {args.out} with {len(pack['files'])} files")


if __name__ == "__main__":
    main()

A tiny test belongs beside the packer. The test should fail closed.

from pathlib import Path
from pack_context import build_pack


def test_deny_list_blocks_license(tmp_path: Path):
    (tmp_path / "LICENSE").write_text("MIT", encoding="utf-8")
    test_file = tmp_path / "test_app.py"
    test_file.write_text("def test_ok():\n    assert True\n", encoding="utf-8")
    (tmp_path / "app.py").write_text("x = 1\n", encoding="utf-8")
    pack = build_pack(tmp_path, test_file, limit=6)
    paths = {item["path"].lower() for item in pack["files"]}
    assert "license" not in paths


def test_failing_test_is_included(tmp_path: Path):
    test_file = tmp_path / "test_app.py"
    test_file.write_text("def test_ok():\n    assert True\n", encoding="utf-8")
    (tmp_path / "app.py").write_text("x = 1\n", encoding="utf-8")
    pack = build_pack(tmp_path, test_file, limit=6)
    paths = {item["path"] for item in pack["files"]}
    assert "test_app.py" in paths

Run the example like this.

python pack_context.py --root . --failing tests/test_app.py --out pack_manifest.json
python -m pytest test_pack_context.py -q
git checkout -b pack-scratch
# apply the model patch here, then:
git apply --check proposed.patch
python -m pytest tests/test_app.py -q

If LICENSE lands in the manifest, your packer is lying. Fix the packer before you touch the model. The manifest is the test you forgot to write.

Decision table I keep on the desk

Observation	Likely anti-pattern	First move
Whole tree cited in the answer	Whole-Repo Dump	Cut the pack to blast radius
Innocent function rewritten first	Lucky File	Pack the feared sibling file
Advice uses deleted symbol names	Chat as Source of Truth	Rewrite the pack manifest
`git apply --check` fails immediately	Orphan Diff	Apply on a throwaway branch
README slogans quoted back to you	Token Theatre	Enforce the deny list in CI
Model blesses its own earlier patch	Review in the Bubble	Split generation and review packs

Print that table and tape it near the terminal. Use it before you change model providers. Provider hopping will not fix a lying pack.

Limitations

This loop will not catch security issues for you. It will not design your architecture either. A deny list is not a retrieval system. Hashes prove what you sent. They do not prove the patch is right.

Free model access is not an availability contract. A free server option is not an SLO. Do not paste secrets into any remote pack. Do not treat this catalog as a legal audit trail.

Large monorepos need path indexes I did not ship here. This packer is a seatbelt around the call. It is not an agent, and it will not steer.

Who should not use this

Skip this if policy forbids third-party model hosts. Skip this if your tree is mostly secrets. Skip this if you need guaranteed latency numbers.

Skip this if you want the model to pick architecture. This catalog assumes you already have a failing test. No failing test means no honest pack. No honest pack means you are back to folklore.

What I want you to try tomorrow

Pick one failing test and nothing else. Build a pack you can name out loud. Write the manifest before the model call. Apply the diff on a clean branch.

Keep the manifest even if the patch is ugly. The manifest is the actual product of the session. Blame the pack first. Only then blame the model.

Six Harness Anti-Patterns That Fake Model Failure

Sam Chen — Sun, 06 Sep 2026 21:17:35 +0000

Stop blaming the model for a dirty harness. Six request-shaping habits waste free inference and hide real bugs. I name each habit, then I replace it.

The catalog is for coding agents on cheap endpoints. It is not a vendor scorecard. If your traces look clean, skip the product talk entirely.

What I mean by harness

The harness is everything around the weights. Tools, retries, logs, and context packing all count. The model is the last suspect, not the first.

You rewrote the prompt three times last night. Did the request shape change at all? If not, you only shuffled adjectives.

How to read each item

Every anti-pattern uses the same four beats. Symptoms. Root cause. Replacement. A cheap check. Keep a JSONL request log or this catalog cannot help you.

I am not claiming a published benchmark here. I am claiming a repeatable inspection habit.

1. Whole-Repo Paste

Symptoms

One user message dwarfs the failing test.
The model restates files you never named.
Latency climbs while the diff stays noisy.

Root cause

You treat context like a dump truck. Retrieval never happened. Neighbors of the bug drown the invariant.

Replacement: slice, then cite

Send the failing test, the target file, and the contract. Cite paths in the prompt. Leave the rest on disk.

# allowed pack for one coding turn
- tests/test_invoice.py   # failing example
- invoice/totals.py       # target
- invoice/types.py        # types only

Check

Cap a single coding turn with a character budget. Start at twelve thousand characters, then tune. If you need the whole repo, you need search, not a bigger paste.

Why did the agent need node_modules in chat?

2. Happy-Path Tool Schema

Symptoms

Tool payloads look successful with empty bodies.
The agent repeats the same call without new evidence.
Chat says "done" while the test still fails.

Root cause

Your schema has ok and stdout. It has no error_class. Failure looks like silence, so the model invents a story.

Replacement: errors as data

from typing import Literal, TypedDict

ErrorClass = Literal["", "timeout", "not_found", "assert", "schema"]

class ToolResult(TypedDict):
    ok: bool
    error_class: ErrorClass
    stdout: str
    stderr: str
    elapsed_ms: int

Return structured failure from every tool. Do not let the model guess why a subprocess went quiet.

Check

Grep traces for tool messages with empty stderr and ok: true. If tests failed in that turn, your schema lied.

3. Session That Never Dies

Symptoms

Turn fourteen still mentions a deleted file.
The model "remembers" an API you reverted.
Token use grows while the task shrinks.

Root cause

You never reset the thread. Stale assistant text becomes fake source. Cheap inference makes that habit feel free. It is not free in correctness.

Replacement: ephemeral tasks, durable facts

New bug, new session. Persist the spec file and the failing test. Drop the chat history on purpose.

# proposed reset, labeled as a workflow
cp task.md /tmp/task.md
: > session.jsonl

Why is a week-old thread still the workspace?

Check

Flag any session past eight model turns without a new failing test. That bound is a starting heuristic, not physics.

4. Spec Written After the Diff

Symptoms

The PR body is a chat export.
Reviewers argue about intent, not code.
You cannot replay the task next month.

Root cause

Chat felt faster than a contract. The contract still exists. It is just scattered across bubbles.

Replacement: spec file first

# task.md
## invariant
`totals()` never returns negative cents.
## fail
pytest tests/test_invoice.py::test_refund_rounding
## done
That test passes. No other test is deleted.

The model may edit code. The spec does not live in scrollback. If the spec changes, that is a new task.

Check

Refuse to start an agent turn without task.md in the pack. No file, no request.

5. Retry Until the Tests Blur

Symptoms

Five regenerations, same stack trace.
You cannot name the delta between tries.
Someone says the free model is just weak.

Root cause

Retry is not a search strategy. You mutated hope, not evidence. The harness hid that you asked the same question.

Replacement: one change per retry

Change only one axis: context slice, tool result, or instruction. Log the delta in a file. Stop after a hard bound.

# proposed loop, not a magic fixer
for i in 1 2 3; do
  python harness_lint.py --log last.jsonl || exit 1
  echo "attempt=$i" >> delta.md
  run_agent --attempt "$i" --spec task.md
done

Check

If two adjacent requests hash to the same pack, you did not retry. You refreshed.

6. Shared Endpoint, Shared Secrets

Symptoms

.env fragments appear in prompts.
Every toy agent points at one public URL.
Nobody can say who stored the last trace.

Root cause

Free inference feels like a scratch pad. Scratch pads leak. A shared server is not your staging cluster.

Replacement: redact, isolate, then call

Strip secrets before the request leaves the machine. Use a throwaway project for public endpoints. Keep production traffic off the shared box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention MonkeyCode only because its free model access and free server option are a convenient place to re-run a cleaned request, not a place to paste production env files. The linter below still matters if you never touch that product.

Check

Scan the JSONL for token-shaped strings before you send anything. Fail the turn if one matches.

Artifact: lint the request log

This is a small, local checker. Point it at JSONL traces you already store. It does not call a model. Label it as a harness test, not a quality score.

Fixture: `sample.jsonl`

{"turn":1,"chars":18000,"tools":[{"ok":true,"stderr":"","stdout":""}],"has_spec":false,"session_turns":12,"same_pack_as_prev":true,"prompt":"API_KEY=sk-demo-not-real\nplease fix invoice totals"}

Checker: `harness_lint.py`

#!/usr/bin/env python3
"""Flag request-shaping anti-patterns in JSONL agent traces."""
from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

SECRET = re.compile(r"(api[_-]?key|secret|token)\s*=\s*\S+", re.I)
CHAR_BUDGET = 12_000
SESSION_BUDGET = 8


def findings(row: dict) -> list[str]:
    out: list[str] = []
    if int(row.get("chars", 0)) > CHAR_BUDGET:
        out.append("whole_repo_paste")
    tools = row.get("tools") or []
    for tool in tools:
        if tool.get("ok") and not tool.get("stderr") and not tool.get("stdout"):
            out.append("happy_path_tool_schema")
            break
    if int(row.get("session_turns", 0)) > SESSION_BUDGET:
        out.append("session_that_never_dies")
    if not row.get("has_spec"):
        out.append("spec_written_after_diff")
    if row.get("same_pack_as_prev"):
        out.append("retry_until_tests_blur")
    text = str(row.get("prompt", ""))
    if SECRET.search(text):
        out.append("shared_endpoint_shared_secrets")
    return out


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--log", required=True)
    args = parser.parse_args()
    path = Path(args.log)
    failed = 0
    for line_no, line in enumerate(path.read_text().splitlines(), 1):
        if not line.strip():
            continue
        row = json.loads(line)
        hits = findings(row)
        if not hits:
            continue
        failed += 1
        print(f"line {line_no}: {', '.join(hits)}")
    print(f"flagged_turns={failed}")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Command

python harness_lint.py --log sample.jsonl

Expected output

line 1: whole_repo_paste, happy_path_tool_schema, session_that_never_dies, spec_written_after_diff, retry_until_tests_blur, shared_endpoint_shared_secrets
flagged_turns=1

If that fixture does not fail, the script drifted. Fix the checker before you lecture the model.

Map each trace field on purpose. chars is packed prompt size. has_spec is whether task.md was attached. same_pack_as_prev is your own hash compare. Do not invent those fields after the run.

Decision table

Signal in the log	Do not do this	Do this instead
Prompt larger than the failing test	Paste the repo	Pack test + target + types
Tool `ok` with empty streams	Ask the model why	Return `error_class`
Session longer than the bug	Keep chatting	Reset, keep `task.md`
PR description is a transcript	Merge on vibes	Write invariants first
Adjacent requests are identical	Hit regenerate	Change one axis, log it
Secrets in the prompt	Use a shared demo box	Redact, then consider any free server

What this does not prove

The linter does not grade model quality. It grades your request shape. False positives happen on generated fixtures and on legitimately large generated files.

It is not a security scanner. The secret regex is a tripwire. Rotate anything that already left your disk.

I also do not claim these bounds are universal. Twelve thousand characters and eight turns are starting knobs. Publish your own knobs if your stack disagrees.

Who should skip this

Skip it if you have no request logs. You cannot lint a vibe.

Skip it if you need a contractual SLA from a shared free endpoint. This workflow assumes throwaway tasks and local tests.

Skip it if your agent already ships with retrieval, structured tool errors, and a spec file. You are not the audience. You already left the anti-patterns.

Do not point a dirty harness at any free server and call that an evaluation. You will measure paste quality, not model quality.

The move I want

Run the fixture first. Then point the same script at one real trace. Fix the highest finding before you switch models.

If you already have a clean pack and still want a second cheap run, MonkeyCode's free server option is one place to try that cleaned request. Keep secrets out of it either way.