Does a green sentence in chat close the ticket?
I do not think that should close it.
The model writes a confident recap of tests.
You accept the patch and then merge it.
Nothing executable ever ran against your real fixtures.
That pattern shows up in every free-model session.
The loop feels fast and still skips evidence.
The ticket is not done without a runner.
This is a myth FAQ about acceptance checks.
I will name five claims developers still repeat.
Then I replace each claim with a runnable check.
Why this keeps biting people
Free models make drafting both cheap and loud.
A free server makes a scratch run feel official.
Neither thing is an acceptance gate by itself.
Does that sequence sound familiar to you?
Have you merged a patch because the recap looked complete?
I have watched that movie too many times.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I mention the free model and free server only as workflow lanes.
They are a scratch lane, not a release manager.
Myth 1: The recap is the test report
The claim
If the model says the tests passed, they passed.
Why people repeat it
The recap is formatted like a CI log.
It uses bullet points, checkmarks, and file names.
It even copies the shape of exit language.
What actually happened
The model only predicted a plausible test report.
It did not necessarily execute your real runner.
It may have invented fixtures you never committed.
Corrected mental model
Treat chat as a hypothesis, never as evidence.
The runner output is the only evidence.
No runner means you do not have a pass.
Ask this out loud before you merge anything.
Where is the process that printed the summary?
If you cannot point to a command, you do not have a report.
# proposal: pin the command, then read its exit code
python -m pytest tests/test_accept.py -q
echo "exit=$?"
If that command never ran, the recap is fiction.
Fiction can still look extremely professional.
Myth 2: A quiet linter is acceptance
The claim
If lint is clean, the change is acceptable.
Why people repeat it
Linters are fast and they feel adult.
Teams already trust them on human patches.
What actually happened
Lint encodes style and a few bug patterns.
It does not encode the product ticket.
It will not notice a missing refund path.
Corrected mental model
Lint is a filter, not a definition of done.
Acceptance is an invariant with an exit code.
Keep those two concerns in different files.
Would a formatter catch a wrong account balance?
A formatter will never catch that money invariant.
The recap would still claim success quite often.
# proposal: one invariant, one exit code
def test_refund_restores_available_balance():
ledger = Ledger(balance=50)
ledger.charge(20)
ledger.refund(20)
assert ledger.available == 50
Run lint after the invariant, not instead of it.
Lint remains useful, but it is not done.
Myth 3: A free server run is the gate of record
The claim
I ran it on the free server, so it is signed off.
Why people repeat it
Remote execution feels more real than local chat.
A log appears and a process existed.
People relax after they see remote stdout.
What actually happened
A scratch runner proved a process could start.
It did not pin the image, seed, or fixtures.
It is closer to a rehearsal than a release.
This is not another sandbox versus staging argument.
I am talking about the signature of the run.
I am not talking about the environment nickname.
Corrected mental model
Treat the free server as a rehearsal room.
Promote only what git can replay later.
Save the exact command inside the repo.
# proposal: record the invocation, not the vibe
cat > .accept-cmd <<'EOF'
#!/bin/sh
set -eu
python -m pytest tests/test_accept.py -q
EOF
chmod +x .accept-cmd
# proposal: Makefile target so humans stop improvising
.PHONY: accept
accept:
sh ./.accept-cmd
MonkeyCode's free model access can draft that command list.
The free server option can execute it as a scratch check.
Neither step replaces a pinned pipeline you own.
Is that rehearsal still useful for you?
Yes, if you refuse to treat it as signed.
Myth 4: The last prompt is the spec
The claim
Acceptance is whatever I just asked for.
Why people repeat it
Prompts sit close to the actual work.
Tickets feel heavy and files feel slow.
What actually happened
Prompts drift hard inside a long session.
The latest turn adds one extra null rule.
The earlier rule falls out without a diff.
The model optimizes for the latest sentence.
Corrected mental model
Specs live in files with stable names.
Prompts can propose edits to those files.
They should not replace those spec files.
<!-- proposal: tests/accept.md lives in git -->
Given a user with balance 50
When they refund a 20 charge
Then available returns to 50
And the audit log has one refund row
Then bind that document to a test, not to memory.
Can a teammate run it tomorrow without you?
If not, that text was never a spec.
Myth 5: The model's scorecard is a review
The claim
The summary table means a second pair of eyes.
Why people repeat it
Those tables look like serious review checklists.
They list risk, files, and tests together.
What actually happened
The same generator graded its own homework.
That is not independence of any kind.
That is a formatted guess with borders.
Corrected mental model
Reviews need a different source of truth.
Use a harness, a teammate, or a recorded command.
Pick at least one that cannot flatter the patch.
Would you accept a PR that only contains a selfie of the tests?
Then you should not accept a scorecard either.
# proposal: fail closed if the scorecard is the only artifact
import json
import sys
from pathlib import Path
def main(path: str) -> None:
data = json.loads(Path(path).read_text())
if data.get("source") == "model_summary":
print("reject: summary is not evidence")
sys.exit(2)
if data.get("exit_code") != 0:
print("reject: harness failed")
sys.exit(1)
print("accept")
if __name__ == "__main__":
main(sys.argv[1])
Save harness JSON instead of a chat table.
{
"source": "harness",
"command": "sh .accept-cmd",
"exit_code": 0
}
This JSON is a proposal fixture, not a captured log.
Do not treat a chat table as the same object.
Artifact: a one-file acceptance loop
Here is a proposed loop you can copy.
It is not a benchmark or a leaderboard.
It is a workflow you can run this afternoon.
Step 1: freeze the invariant
Write one test that encodes the ticket.
Do not start with a twelve-file suite.
One failing test is enough to begin.
Step 2: freeze the command
Put the exact runner command into .accept-cmd.
Commit that file with the production change.
Do not keep the runner inside the prompt.
Step 3: draft with the free model
Ask the model to change production code only.
Tell it the harness file is off limits.
If it edits the test to pass, reject the patch.
Why reject a test edit so quickly then?
Because the invariant moved with the code.
Step 4: rehearse on the free server
Run .accept-cmd there as a scratch execution.
Read stdout and then read the exit code.
Ignore the chat recap if those disagree.
Step 5: promote only a replayable result
If you cannot replay the command from git, stop.
Do not paste a screenshot into the ticket.
A screenshot is not a gate you can rerun.
# proposal: local replay after the scratch run
git diff --stat
sh .accept-cmd
test "$?" -eq 0
python check_evidence.py evidence.json
Decision table
Use this when you feel tempted to merge from chat.
| Signal in chat | What it proves | What you still run | Merge now? |
|---|---|---|---|
| "Tests passed" | A sentence exists | sh .accept-cmd |
No |
| Clean lint output | Style gate moved | Invariant test | No |
| Free server log | A process ran | Pinned command from git | Only if replay matches |
| Scorecard table | Formatting | Independent harness | No |
| Patch plus now-green invariant | The invariant moved | Replay once more | Maybe |
Read the maybe row twice before merging.
A passing harness is necessary, not sufficient.
You still need a human on product risk.
Limitations
This loop does not replace CI you control.
It does not pin compilers, images, or model versions.
I am not claiming durability of any free lane.
It will not help if you have no invariant.
It will not help if the ticket says make it nicer.
It will not help if secrets live in the prompt.
The free server is only a rehearsal.
It is not a compliance boundary of any kind.
Do not store production credentials in that lane.
Who should skip this
Skip this if your org already blocks merges on a real pipeline.
You do not need a second scratch ritual.
Skip this if you cannot write even one invariant test.
Fix that skill before you tune prompts.
Skip this for safety-critical or regulated releases.
Those need signed runners, not a myth FAQ.
Skip this if the work is an exploratory spike.
Spikes can stay messy on purpose here.
Just do not ship them as accepted work.
A better default question
Stop asking whether the model said it passed.
Ask which pinned command failed closed instead.
Where is that command stored in git?
That question kills most of these myths.
It also keeps the free tools in a useful box.
Draft in the free lane, then rehearse there.
Accept the change only after git can replay it.
If you try this, keep the checks in git.
Chat is a draft, and git is the gate.
Top comments (0)