The Model Agreed. That Is Not a Contract.

Did the agent just bless your API change?
A chat transcript is not a design review.
Fluent text still needs a failing test command.

I keep hearing the same mix-up on calls.
A free model drafts a neat plan.
Then someone treats that plan as law.

Does that loop sound familiar to you?

What this article is

This FAQ hunts claims that I still hear.
It is not a product comparison piece.
It is not a latency or quota story.

I name four myths that fake a contract.
Then I show checks that puncture each one.
Then I give a mental model you can reuse.

After the myths I include a claim file.
You hash the prompt and you hash the output.
You refuse to merge on vibes alone.

Why these myths spread so fast

These agents speak in complete confident sentences.
Those complete sentences look like real decisions.
Real decisions belong in tests you actually run.

A remote scratch box adds another cheap trick.
The run happened on a machine somewhere else.
Somewhere else still is not your production.

I use a free model as a hypothesis mill.
I sometimes draft those hypotheses with MonkeyCode nearby.
MonkeyCode offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That scratch pad is useful for first drafts.
The contract still lives inside my repository.

Myth 1: A fluent plan is reviewed architecture

The claim developers repeat

The model explained the system, so it is sound.
People paste a folder tree into a prompt.
They get a layered diagram back in seconds.

Then they schedule a full rewrite anyway.

Evidence you can check today

Ask the same architecture question twice in a row.
Change one filename and watch the story drift.
Architecture that moves with a rename is fiction.

Read the plan against one real entrypoint.
Does that plan name the actual queue?
Does it mention the real auth middleware file?

If you cannot grep the claim, drop it now.

# proposed check: claims must point at files that exist
rg -n "apps/api|middleware|queue" outputs/plan.md
rg -n "class |def |router" apps/api | head

Corrected mental model

A fluent plan is only a hypothesis.
Reviewed architecture is a set of constraints.
Version those constraints and back them with tests.

Myth 2: HTTP 200 means the write actually stuck

The claim developers repeat

The agent called the API, so it worked.
A wrapper returns a friendly status string.
The model then thanks itself in prose.

Did anyone read the row back after that?

Evidence you can check today

Replay the side effect from your own machine.
Query the datastore. Do not query the chat.
If the row is missing, that 200 was theater.

Check idempotency keys on the write path.
A second rehearsal should not double apply anything.
You own the ledger. The model does not own it.

# proposed check: prove state from a client you control
curl -sS "$LOCAL_API/v1/orders/$ORDER_ID" | jq .
# if this object is missing, the tool log is not evidence

Corrected mental model

A tool call is only a request.
A transaction is a durable state change.
Prove that change with a read you control.

Myth 3: A saved prompt is a reproducible build

The claim developers repeat

We saved the prompt, so we can rebuild this.
That sentence sounds like gcc plus a lockfile.
It is not a compiler. It is a sampler.

Evidence you can check today

Hash the prompt. Hash the output. Compare tomorrow.
Routing and sampling still move under your feet.
Your prompt file did not freeze the whole world.

Pin whatever you can honestly pin in git.
Then assume the generated text still drifts.
Treat that text like a patch from a stranger.

python tools/hash_claim.py prompts/auth.md outputs/auth.md
git rev-parse HEAD
# store both digests; do not call this a rebuild

Corrected mental model

Saved prompts are only inputs to a sampler.
Builds are functions of pinned tools and tests.
Model text is an untrusted patch, every time.

Myth 4: A free-model review replaces a merge gate

The claim developers repeat

The agent reviewed it, so we can ship.
This one is tempting on a Friday afternoon.
It is also how secrets leak into stories.

Evidence you can check today

Does that review actually run make test?
Does it refuse to read a local .env file?
Does it fail closed when the scratch box dies?

If the answer is "it usually catches things," stop.
That is a comment. That is not a merge gate.
Gates are boring scripts with nonzero exits.

test -f .env && echo "never upload this file" 
make test
# if make is red, the polite review does not matter

Corrected mental model

Reviews from a model are extra eyes only.
Merge gates are scripts that exit nonzero.
Keep the extra eyes. Do not fire the script.

The artifact: a claim file plus a verifier

I keep agent output under claims/, not only Slack.
Each claim starts guilty until commands pass.
Status stays hypothesis until a human flips it.

Here is a schema I actually type by hand.

# claims/2026-09-07-auth-migration.yaml
id: 2026-09-07-auth-migration
status: hypothesis   # hypothesis | rejected | accepted
summary: "Move session checks into middleware."
prompt_sha256: ""
output_sha256: ""
repo_head: ""
must_pass:
  - "make test"
  - "python tools/check_no_secrets.py"
  - "rg -n 'session' apps/api -g '*.py'"
must_not:
  - "curl a production URL from the agent"
  - "paste .env into a remote prompt"
acceptance:
  - "Auth tests fail closed on a missing cookie."
  - "No new global mutable store appears."

Empty hashes are intentional at draft time here.
The local verifier fills those hashes for me.
I still do not type sixty-four hex characters.

Hash the untrusted text

# tools/hash_claim.py
from hashlib import sha256
from pathlib import Path
import sys

def digest(path: Path) -> str:
    return sha256(path.read_bytes()).hexdigest()

if __name__ == "__main__":
    for raw in sys.argv[1:]:
        p = Path(raw)
        print(f"{digest(p)}  {p}")

Run it like this on the two files.

python tools/hash_claim.py prompts/auth.md outputs/auth.md
git rev-parse HEAD

Paste those digests into the YAML claim.
Now that chat log finally has a name.
It still has no merge authority at all.

Fail closed before anyone merges

Treat this verifier as a proposed local workflow.
It is for YAML you wrote, not random internet files.
eval is sharp. Keep the directory private.

#!/usr/bin/env bash
# tools/verify-claim.sh
set -euo pipefail

claim="${1:?usage: verify-claim.sh claims/file.yaml}"
test -f "$claim"

if grep -q '^status: accepted' "$claim"; then
  grep -Eq '^prompt_sha256: "[a-f0-9]{64}"' "$claim" \
    || { echo "accepted claims need prompt_sha256"; exit 1; }
fi

echo "== must_pass =="
awk '/^must_pass:/{p=1;next} /^[a-z]/{p=0} p && /^- /{
  s=$0; sub(/^- "/,"",s); sub(/"$/,"",s); print s
}' "$claim" | while IFS= read -r cmd; do
  echo "+ $cmd"
  eval "$cmd"
done

echo "Leave status=hypothesis until a human flips it."

Yes, that eval path is local and explicit.
Do not point it at untrusted YAML from strangers.
This script is for your claims directory only.

Commands that keep me honest

I run a tiny loop on purpose.
The loop is ugly and that is fine.
Each step leaves a file I can grep.

# 1. Capture the prompt you actually sent
cat > prompts/auth.md

# 2. Save the model output without editing it
cat > outputs/auth.md

# 3. Name the git tree you thought you described
git rev-parse HEAD > claims/HEAD.txt

# 4. Hash both sides before anyone edits them
python tools/hash_claim.py prompts/auth.md outputs/auth.md

# 5. Run the real suite, not the model's summary
make test

# 6. Search for fake universals in the draft
rg -n "will always|cannot fail|production ready" outputs/auth.md

That last search is petty on purpose.
It catches fake universals in the draft.
I like petty checks that cost nothing.

Decision table

If you hear this	Do not do this	Do this instead
"The model designed it"	Rewrite the whole service	Write one constraint test
"The tool returned 200"	Assume the write stuck	Re-read from your datastore
"We saved the prompt"	Call that a rebuild	Hash output and expect drift
"The agent approved it"	Skip CI for speed	Keep CI and attach the claim
"The free server ran it"	Call the box staging	Call it a scratch rehearsal

Print this table for your team wall.
Then tape it near the merge button.
I am not joking about the tape.

What this does not solve

This workflow does not make a model deterministic.
It does not turn a scratch box into prod.
It does not score quality behind your back.

I am not listing model names or hardware here.
Those vendor details will change without notice often.
Your tests should not depend on that catalog.

If you need bit-identical generation, just stop.
Use templates plus checked-in code for that.
If you need a compliance sign-off, get a human.

Do not send secrets to any remote endpoint.
A free server is still someone else's memory.
Scratch compute is not a vault or a witness.

Who should skip this approach

Skip this if you do not have tests yet.
A claim file without tests is extra paperwork.
Fix the suite before you hash any prose.

Skip this if the agent can mutate production.
Hypothesis mills do not get production credentials.
That rule is not negotiable in this workflow.

Skip this if you wanted a single-click oracle.
There is no honest oracle in this setup.
There is a patch, a hash, and a command.

The mental model I want stolen

In this workflow free models only draft patches.
A free server only rehearses those patches.
Your repo still decides what actually merges.

Ask one question before you hit merge.
What command fails if this claim is wrong?
If you cannot name it, the claim is still chat.

Would you merge a stranger patch with no tests?
You can keep the intern and the politeness.
Then add the tests before anyone merges.