The issue still wore a green good first issue label, even though the reproduction steps named a CLI flag the project had removed. A volunteer cloned the repository on a quiet weekday evening, ran the default test target, and watched every check pass. Nothing in the suite described the behavior the reporter had lost, so the failure lived only in a stale comment thread. The volunteer almost opened a speculative pull request that matched today's source rather than the original contract.
That near-miss is now a common open source pattern, not a personal confession. Cheap drafting tools make a plausible diff easy to produce before anyone can name what must stay true. Maintainers then inherit patches that compile, satisfy existing tests, and still change a public promise the issue never stated clearly. The missing artifact is not another review persona. It is a written invariant that can fail on purpose.
Passing tests can still hide a broken contract
Open source bug reports are narratives, screenshots, version pins, and guesses about root cause. They are not specifications, and they age faster than the default branch. A label such as good first issue only means a maintainer once hoped the work was bounded. It does not mean the steps still execute, or that the suite encodes the lost behavior.
Industry conversation in 2026 has leaned on generated patches and agent workflows that fill gaps with confident assumptions. Those assumptions are expensive in a public repository, because the next release ships them to strangers. A contributor who starts by asking a model for a fix is asking it to invent the contract. The safer order is to freeze the claim, encode it as a test, and only then touch production code.
The gate below is meant for functional bugfixes and small behavioral repairs. It is not a substitute for security review, license diligence, or maintainer judgment. It exists to stop a volunteer from submitting a patch whose success criteria live only in chat history.
A four-step invariant gate
The working unit is a small packet on disk, not a thread in a chat product. Each step writes a file that another person can replay without the original volunteer. The model is invited late, and only to attack the written claim.
1. Freeze the issue, not the rumor
Capture the issue, the base commit, and the exact command that is supposed to fail. Do not summarize from memory after reading related pull requests. Related discussion often describes a later design that the reporter never had.
# Proposed packet layout; adjust IDs and commands per project.
mkdir -p .invariant-gate
gh issue view 4821 --json number,title,body,labels,createdAt,url \
> .invariant-gate/issue.json
git rev-parse HEAD > .invariant-gate/base-sha.txt
git status --porcelain > .invariant-gate/dirty-tree.txt
printf '%s\n' 'cargo test -p cli -- parse_config -- --nocapture' \
> .invariant-gate/repro-command.txt
If the tracked tree is dirty, stop and stash. An invariant extracted against mixed local edits cannot be replayed by a maintainer. If gh is unavailable, save the issue HTML or API JSON by hand and record the retrieval time beside the base SHA.
2. Write the invariant in one sentence
The sentence must name an actor, an input, and an observable that should hold after the patch. Cause language is optional and often wrong. Prefer a statement a test harness can contradict.
Use this worksheet as a committed markdown file, not as a chat prompt:
# Invariant worksheet (fill before any production diff)
- Issue: https://github.com/example/cli/issues/4821
- Base SHA: 9f3c1a2
- Actor: `cli parse-config`
- Input: a TOML file with duplicate `[env]` tables
- Observable: process exits 2 and prints the first duplicate key on stderr
- Non-goals: formatter output, help text wrapping, Windows path canonicalization
- Invariant: Given duplicate `[env]` tables, parse-config exits 2 and names the first duplicate key.
- Falsifier I will accept: exit 0, exit 1 with no key name, or a passing suite with no new test.
If the sentence needs a paragraph of caveats, the issue is not ready for a first-time patch. Split the work or move it back to discussion. Vague words such as handle correctly or make it more robust are not invariants, because they cannot fail a test.
3. Encode it as a characterization test
A characterization test records current or intended behavior with the smallest public API that the issue already mentions. It should fail on the base SHA if the bug is still present. If it passes, the report is stale, already fixed, or aimed at an untested surface.
# tests/test_parse_config_duplicate_env.py
# Characterization test: expected to FAIL on the base SHA when issue 4821 is live.
from pathlib import Path
import subprocess
import sys
CLI = Path(__file__).resolve().parents[1] / "target" / "debug" / "cli"
def test_duplicate_env_tables_exit_two_and_name_key(tmp_path: Path) -> None:
config = tmp_path / "app.toml"
config.write_text(
"[env]\nFOO = \"1\"\n\n[env]\nFOO = \"2\"\n",
encoding="utf-8",
)
result = subprocess.run(
[str(CLI), "parse-config", str(config)],
capture_output=True,
text=True,
check=False,
)
assert result.returncode == 2
assert "FOO" in result.stderr
Run the recorded command and store both the exit code and a short log excerpt. The packet should make the failure obvious without a screen recording.
chmod +x target/debug/cli || cargo build -p cli
pytest -q tests/test_parse_config_duplicate_env.py | tee .invariant-gate/test-before.txt
When the new test passes on the base SHA, do not invent a patch anyway. Comment on the issue with the SHA, the command, and the passing output. Stale-issue archaeology is a valid contribution, and it prevents a decorative diff.
4. Invite a model to attack the sentence
Only after the worksheet and the failing test exist should a model see the packet. The task is not to draft the patch first. The task is to list ways the invariant is false, incomplete, platform-specific, or already covered elsewhere.
Label the following prompt as a proposal, and keep it in the packet so a maintainer can rerun it:
You are reviewing an invariant, not writing a patch.
Read issue.json, the worksheet, the characterization test, and the surrounding parser module.
Return only:
1. Ways the invariant could be wrong given the public CLI contract.
2. Inputs the test does not cover that would still satisfy a sloppy patch.
3. Existing tests that already encode this observable, with file paths.
4. A verdict: KEEP, NARROW, or ABANDON the invariant.
Do not propose implementation code unless the verdict is KEEP and the test is still failing.
Treat every model claim as unverified until a command or a file path backs it. If the model cites a helper that does not exist, that is evidence the packet lacks context, not evidence the helper is real. Narrow the worksheet when the attack is valid. Abandon the issue when the observable cannot be stated without private maintainer knowledge.
Running the loop on a constrained machine
Many volunteers cannot keep a heavy toolchain on a personal laptop for a weekend reproduction. The packet is small enough to copy onto a throwaway workspace: issue JSON, worksheet, one test, and the attack prompt. Contributors who need a machine that is not their daily driver can run that same packet on a free server that also exposes free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is one place that currently pairs those two pieces; this workflow still holds if the packet is replayed on any other ephemeral box, and readers should check live product pages rather than this article for what remains free.
Keep secrets out of the packet. Do not paste tokens, private fixtures, or production dumps into a shared workspace. Prefer public fixtures and synthetic configs that match the issue's input shape. If the project needs network services, record that requirement in the worksheet as a reason to stop, because a free server will not magically supply them.
A minimal remote replay looks like the following commands, once the repository clone is already allowed by the project's contribution guide:
git fetch origin
git checkout --detach "$(cat .invariant-gate/base-sha.txt)"
# build step from CONTRIBUTING.md, then:
pytest -q tests/test_parse_config_duplicate_env.py
# only then paste the attack prompt with the worksheet attached
The model output belongs in .invariant-gate/attack.md beside the test log. A later reviewer should see the failing assertion, the attack, and the eventual diff as one timeline. If the attack file is missing, the patch is still a vibe, even when a model wrote a tidy commit message.
A decision table before the first production diff
Use the table as a hard gate. If the row says stop, do not open a pull request for the original issue.
| Observation on base SHA | Invariant quality | Action |
|---|---|---|
| New test fails as designed | One observable, few caveats | Patch only code the test names |
| New test passes | Sentence still sounds right | Report stale issue with SHA and logs |
| New test fails for a different reason | Sentence names two behaviors | Split into two worksheets |
| Build or tests cannot run | Unknown | Do not speculate with a generated diff |
| Model attack finds an existing test | Duplicate | Close the loop and link the test |
| Model attack requires private context | Not a starter issue | Comment and wait for maintainers |
| Observable is a security boundary | Out of scope here | Follow the project's security process |
After a KEEP verdict, write the smallest patch that turns the characterization test green without expanding non-goals. Re-run the full suite, then re-read the worksheet aloud against the diff. If the diff touches formatter internals while the invariant is an exit code, the patch has drifted.
git diff --stat origin/HEAD
pytest -q
git add tests/test_parse_config_duplicate_env.py src/parse_config.rs
# Commit message should restate the invariant, not the implementation guess.
A useful commit subject restates the observable: parse-config: exit 2 on duplicate [env] keys. A weak subject restates the tool: fix issue 4821 with clearer errors. Maintainers search the former years later. They cannot search the latter without the issue tracker.
What this method will not catch
The gate does not prove performance, accessibility, or backward compatibility beyond the written observable. A green characterization test can still miss encoding, locale, and path-separator bugs. Free-tier models also miss architectural context when the packet omits the module that defines the contract.
The gate can freeze a wrong invariant if the issue reporter was mistaken and the volunteer never checked adjacent docs. That failure is still cheaper than a generated patch with no invariant, because the wrong sentence is visible and can be rejected in review. It is not cheaper than talking to a maintainer on an ambiguous public API.
Ephemeral servers may differ from maintainer laptops in architecture, case-sensitive filesystems, and available compilers. Record uname -a, toolchain versions, and the exact test command in the packet. If those details are missing, a passing remote run does not transfer.
Who should skip this workflow
Coordinated disclosure, cryptography changes, and packaging backports need specialist processes, not a starter invariant file. Contributors who cannot run any project tests should not use a model to simulate a suite they never executed. Drive-by patches that only rephrase comments or rename symbols also gain little, because there is no lost observable to encode.
Teams that already keep executable specifications for every public flag may find the worksheet redundant. They can still steal the attack step, because generated patches tend to widen types and swallow errors around even well-tested edges. Everyone else should treat the bug report as evidence, write the invariant first, and refuse a diff that cannot name what must remain true.
Top comments (0)