I do not trust a nodding chat model.
I trust a contract file that fails closed.
If your free session only agrees, you lack a design.
That sounds harsh to a busy shipping team.
It is also a cheap bug that hides inside speed.
A green chat log is not a release gate.
The conclusion up front
Free drafts are cheap, and cheap agreement is dangerous.
A scratch host can show a page, not own your data.
Your contract file is the only gate that stays put.
You still need a schema, nasty cases, and a host role.
You still need a checker that exits non-zero.
You still need a human who can read a diff.
Name the tool, then get back to work
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
This outreach brief presents MonkeyCode as an open-source project.
It mentions free model access and a free server option.
I will not invent quotas, model names, hardware, or duration.
Those numbers were not verified for this draft.
Would you bet a sprint on a sentence I refused to source?
Why this showed up again
Developers are arguing about models that agree too fast.
Some posts blame the model for soft, flattering answers.
I blame the workflow that treats agreement as review.
I am not citing a lab study from this week.
I am using that public argument as a warning, not a dataset.
Your repo can fail the same way without any leaderboard.
What a contract means here
A contract is a tiny set of files in git.
It lists required keys, reject cases, and forbidden scans.
It also says whether the host is scratch or durable.
I am describing a proposal, not a personal benchmark.
I did not execute these samples against a live host.
Label them unrun until you try them on a toy repo.
Anti-pattern 1: The nod review
Symptom
The model restates your plan with cleaner words.
You feel reviewed, yet nothing was really falsified.
Does the pull request still hide the original blind spot?
Root cause
You asked the model for confidence, not a counterexample.
A yes-leaning model will happily take that bait.
Your deadline then mistakes polite tone for evidence.
Replacement
Demand three inputs that should make the code fail.
Save those inputs before you accept any praise.
Would you merge a human note that only said nice?
Anti-pattern 2: Schema roulette
Symptom
Field names drift across otherwise similar model runs.
One run says userId, the next says a bare id.
Clients then parse whichever shape arrived last night.
Root cause
The prompt described a vibe, not a locked schema.
Fresh retries feel like progress, so drift hides.
Who froze the response shape in git, if anyone?
Replacement
Pin the required keys and reject unknown fields.
Fail the local check when a required key is missing.
Do not repair drift by rewriting the client in chat.
Anti-pattern 3: Secret bleed
Symptom
A token, cookie, or customer row sits in the prompt.
That same text can land in a host log later.
Can you pull a secret back out of history?
Root cause
The scratch path felt private because it was free.
Free access can still retain prompts or logs.
A free server is a convenience, not a vault.
Replacement
Redact every secret before you write the prompt.
Keep real credentials in a local env file.
Refuse the push when a secret-like pattern matches.
Anti-pattern 4: Host role swap
Symptom
The free server starts life as a short demo.
A week later it holds the only copy of state.
No commit records that quiet change of role.
Root cause
A live page felt like real ownership to you.
A free option can pause, reset, or disappear.
What breaks tonight if that host forgets your data?
Replacement
Write the host role down as scratch or durable.
Block the deploy when scratch would store user data.
Promote the role only through a human commit.
Anti-pattern 5: Theater tests
Symptom
Tests assert one happy string the model just wrote.
They pass on the same tree that generated them.
Where are the empty, huge, and badly typed inputs?
Root cause
The model graded a story it had just told.
You wanted speed, so the adversarial cases slipped.
The contract never listed those cases at all.
Replacement
You write the failing cases before the implementation.
The model may still draft the handler code.
It must not be the only author of the tests.
The replacement workflow
Work in this order, even when the chat feels done.
Speed is fine after the contract exists, not before.
- Write a schema file that lists the required keys.
- Write a cases file with reject inputs you invented.
- Write a host file with the role and scan paths.
- Run the local checker and fix every printed error.
- Only then ask a model to draft the handler.
- Re-run the checker before you upload anything at all.
- Use a free server only while the role says scratch.
MonkeyCode can help on the draft step and the scratch host.
Feed it the schema, not a dump of customer rows.
If you remove the product name, the order still holds.
Proposed checker
Treat this script as an unexecuted local example.
It uses only the Python standard library here.
It does not call any network API at all.
#!/usr/bin/env python3
'''Local contract check. Proposal only, not a vendor SDK.'''
import json
import re
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SECRET = re.compile(
r'(?i)(api[_-]?key|secret|token|password)\s*[:=]\s*\S+'
)
def load(name):
path = ROOT / 'contract' / name
return json.loads(path.read_text())
def main():
schema = load('schema.json')
cases = load('cases.json')
host = load('host.json')
errors = []
required = schema.get('required', [])
if not required:
errors.append('schema has no required keys')
for case in cases.get('cases', []):
if 'input' not in case or 'expect' not in case:
errors.append('case missing input or expect')
role = host.get('role')
if role not in {'scratch', 'system-of-record'}:
errors.append('host role is missing or unknown')
if role == 'scratch' and host.get('stores_user_data'):
errors.append('scratch host must not store user data')
for rel in host.get('scan_paths', []):
path = ROOT / rel
if not path.is_file():
errors.append('scan path missing: %s' % rel)
continue
text = path.read_text(errors='ignore')
if SECRET.search(text):
errors.append('secret-like text in %s' % rel)
if errors:
print('\n'.join(errors))
return 1
print('contract ok')
return 0
if __name__ == '__main__':
sys.exit(main())
Put the file at tools/check_contract.py in the repo.
The parent lookup assumes that exact file path.
Change ROOT if your repo layout actually differs.
Sample files
These files are illustrations, not a production capture.
Copy them, then break them, then fix the breaks.
Would a silent pass on broken files fool you?
contract/host.json:
{
"role": "scratch",
"stores_user_data": false,
"scan_paths": ["README.md", "src/handler.py"]
}
contract/schema.json:
{
"required": ["id", "status"],
"additionalProperties": false
}
contract/cases.json:
{
"cases": [
{"input": {}, "expect": "reject"},
{"input": {"id": ""}, "expect": "reject"},
{"input": {"id": "a", "status": "nope"}, "expect": "reject"}
]
}
Run the checker from the repository root directory.
A non-zero exit means you do not upload yet.
A zero exit only means the files agree internally.
python3 tools/check_contract.py
echo "exit=$?"
A local pre-push hint
Treat this hook as a proposal, not a team policy.
It will annoy you on purpose during rushed pushes.
Annoyance is cheaper than a secret on a public log.
#!/bin/sh
python3 tools/check_contract.py || exit 1
Save that snippet as a local pre-push hook.
Make the hook executable, then document the command.
A teammate without the hook still needs the same check.
Decision table
| Symptom | Root cause | Replacement |
|---|---|---|
| Chat only praises the plan | Prompt asked for agreement | Require three breaking inputs |
| Keys rename between runs | No pinned response schema | Reject missing or extra keys |
| Secret text reaches a log | Scratch path felt private | Scan files and redact prompts |
| Demo host holds real state | Role was never recorded | Label host and block scratch writes |
| Tests cheer one happy string | Model wrote its own exam | You own the failing cases |
Read the row that matches your last real incident.
If two rows match, fix the secret row first.
A leaked token outruns a messy schema every time.
Failure analysis you can rehearse
Break the sample on purpose before you trust the script.
A checker you never broke is just another nod.
- Delete the required list and expect a schema error.
- Set stores_user_data to true while the role is scratch.
- Paste a fake token line into the scanned handler.
- Drop the expect field from one case object.
- Point scan_paths at a file that does not exist.
Each break should print a reason and exit non-zero.
If a break stays silent, your checker is theater too.
Fix the checker before you blame the model again.
Limitations
This checker never calls the model by itself.
It cannot prove the handler matches the cases.
You still need a real test runner for that gap.
The secret regex misses encoded blobs and split strings.
It can also flag toy examples if you scan them.
Point scan_paths at source files, not at this writeup.
I did not verify quotas, regions, hardware, or retention.
Do not quote this piece as a pricing or status page.
Vendor pages change, and this draft will not follow.
Short sentences can sand off real edge cases.
The table and the script carry the precise part.
If they disagree with your policy, keep your policy.
Who should skip this
Skip it when the data is regulated or customer confidential.
Skip it when you need an audit sign-off this week.
Skip it when nobody on the team will read diffs.
Skip it if the free server must be your only database.
That wish is the host-role anti-pattern in a new coat.
Should a disposable box be your system of record?
Skip it if the contract will not live in version control.
A checker with stale files is another nodding reviewer.
You already know how that particular story ends.
Questions worth asking out loud
What input would make this function lie to you?
Where does the prompt go if the vendor logs it?
Is this host allowed to remember a single user?
If you cannot answer those, stop the upload.
Ask the model for a counterexample, not a compliment.
Write that counterexample into the cases file first.
One next step
Pick a tiny handler you already understand well.
Write three reject cases before you open a session.
Run the checker, then decide if a scratch host helps.
If MonkeyCode is already your tool, feed it the schema only.
Keep the free server labeled scratch until a human promotes it.
The same gate still works if you later change vendors.
Top comments (0)