Declare the Behavior Delta Before Maintainers Open an OSS Bugfix

A contributor cloned a popular CLI, reproduced a parser crash, and typed a confident patch in one sitting. The pull request touched a helper, two tests, and a changelog line that promised a complete fix. Maintainers opened the diff and still could not tell which captured outputs were supposed to change. Screenshots arrived later, then a second patch, then a request for logs the contributor had already discarded.

That stall is common when a bugfix is treated as a narrative instead of a measurable delta. Reviewers need the command, the before output, the after output, and a short allowlist of intended change. Without those four pieces, a green local run is only a private memory of one machine. The workflow below turns that memory into files a distant maintainer can replay without extra archaeology.

Treat the patch as a declared behavior delta

An OSS bugfix is not finished when the process exits zero on one laptop. The finished unit is a declared delta: old behavior, new behavior, and the exact invocation that produces both. Code review then checks whether the git hunks are necessary for that delta, rather than guessing intent from renamed helpers. This framing also limits accidental drive-by refactors that hide inside a supposedly small fix.

A useful delta statement is boring, local, and specific enough to fail in public. It names the binary or test target, the fixture, and the lines that must differ after the patch. It also names the outputs that must stay byte-identical so collateral churn cannot hide. Maintainers can reject a patch that cannot produce that statement in a few quiet minutes.

The oracle packet

Keep the packet beside the worktree, not inside a chat log that will rot. A simple directory is enough for most CLI tools and library bugs that fail in user space.

oracle/
  README.md
  invoke.sh
  before.txt
  after.txt
  before.exit
  after.exit
  allowlist.txt
  observed.diff
  check_allowlist.py
  check_oracle.sh

Each file has one job, and mixing those jobs is how packets become essays. invoke.sh is the only entry point a stranger should need to execute. before.txt and after.txt are raw captured bytes, not edited summaries of what the contributor remembers. allowlist.txt lists the fragments that are allowed to differ, and check_oracle.sh fails when the live diff escapes that list.

README.md as the human index

# Oracle packet: parse crash on empty --config

- Upstream issue: #4182
- Failing commit used for the before capture: 9f3c1aa
- Invocation: ./oracle/invoke.sh
- Intended delta: stderr gains a one-line usage error; process status becomes 2
- Forbidden delta: stdout of `--help` and the JSON fixture under tests/data/

The README is an index so a maintainer can skip archaeology, not a second design document. Link the issue, pin the commit, and state the delta in one plain sentence. Leave motivation and design opinions in the pull request body, where discussion belongs.

Step 1: freeze the failing invocation

Do not start in the editor. Write the invocation that currently fails, including working directory and environment variables that the tool actually reads. If the bug needs a fixture, generate that fixture in the same script so the packet stays closed.

#!/usr/bin/env bash
# oracle/invoke.sh
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"

export PYTHONUNBUFFERED=1
# Status is recorded below; do not abort the script on the expected failure.
set +e
python -m tools.parser --config tests/data/empty.toml \
  >oracle/stdout.txt 2>oracle/stderr.txt
echo "$?" > oracle/exit_code.txt
set -e

Record status, stdout, and stderr as separate streams when those streams matter to the bug. Mixing them in a terminal screenshot is how reviews lose the only signal they can replay. If a test runner can target one case, prefer that invocation over a manual CLI session with hidden shell history.

Step 2: capture the before side without editing it

Run the frozen invocation on the parent commit that still contains the bug. Save the streams before any local patch exists in that tree. Resist the urge to trim timestamps or absolute paths by hand, because silent edits make the after file incomparable.

git switch --detach 9f3c1aa
./oracle/invoke.sh
cp oracle/stderr.txt oracle/before.txt
cp oracle/exit_code.txt oracle/before.exit

If logs embed clocks, hostnames, or random request ids, freeze those sources in invoke.sh before treating the files as oracles. A packet that cannot be replayed tomorrow is a diary entry, not evidence. Characterization of the failure belongs in the captured bytes, not in a paraphrase of a stack trace.

Step 3: apply the smallest patch that should move the delta

Copy before.txt and before.exit to a safe path if the working tree will rerun the same script. Then restore a writable branch and change only what the issue asked to change. Do not fold formatter noise, comment rewrites, or unrelated helper cleanups into the same commit, because those extra hunks make the allowlist dishonest even when the crash is gone.

git switch --create fix/empty-config 9f3c1aa
# edit tools/parser.py and the focused test only
git diff --stat

A short self-check before capture keeps the later allowlist honest:

• Every edited path is named in the issue or required by the failing invocation.
• No lockfiles, generated docs, or IDE metadata slipped into git status.
• The changelog line, if required by the project, matches the declared delta and nothing else.

Step 4: capture the after side with the same invocation

Use the identical invoke.sh. If the command line changes, the packet is invalid and the README must be rewritten first. Copy the new streams into after.txt and after.exit, then produce a unified diff the checker can read.

./oracle/invoke.sh
cp oracle/stderr.txt oracle/after.txt
cp oracle/exit_code.txt oracle/after.exit
diff -u oracle/before.txt oracle/after.txt > oracle/observed.diff || true

Read observed.diff as a product artifact, not as debug scrap from a local terminal. If the diff is empty, the patch did not change the failing behavior and review is premature. If the diff is huge, the patch changed more than the issue asked for and the allowlist will not save it.

Step 5: publish an allowlist, not a vibe

The allowlist is the original technical artifact of this workflow. It states which observed changes are load-bearing for the issue. Everything else is a defect in the packet, the invocation, or the patch itself.

# oracle/allowlist.txt
# One fragment per line. Lines starting with # are comments.
# EXIT: lines record an intended status change.
+error: empty --config is not a valid document
-Traceback (most recent call last):
EXIT: 1 -> 2

The checker below is a proposal for local use, not a harvested suite from production CI. It treats uncovered unified-diff lines as a failed packet, which is stricter than a glance at git diff.

#!/usr/bin/env python3
"""Fail if oracle/observed.diff contains lines not covered by allowlist.txt."""
from pathlib import Path
import sys

root = Path(__file__).resolve().parent
observed = (root / "observed.diff").read_text(encoding="utf-8", errors="replace")
allow = []
exit_rule = None
for raw_line in (root / "allowlist.txt").read_text(encoding="utf-8").splitlines():
    line = raw_line.strip()
    if not line or line.startswith("#"):
        continue
    if line.startswith("EXIT:"):
        exit_rule = line
        continue
    allow.append(line)

uncovered = []
for raw in observed.splitlines():
    if not raw.startswith(("+", "-")) or raw.startswith(("+++", "---")):
        continue
    text = raw[1:]
    if not any(pat in text for pat in allow):
        uncovered.append(raw)

before_exit = (root / "before.exit").read_text(encoding="utf-8").strip()
after_exit = (root / "after.exit").read_text(encoding="utf-8").strip()
if exit_rule:
    expected = exit_rule.split(":", 1)[1].strip()  # e.g. "1 -> 2"
    actual = f"{before_exit} -> {after_exit}"
    if expected != actual:
        print(f"exit allowlist expected {expected!r}, got {actual!r}")
        sys.exit(1)
elif before_exit != after_exit:
    print("exit code changed but allowlist.txt has no EXIT: rule")
    sys.exit(1)

if uncovered:
    print("oracle allowlist missed these diff lines:")
    print("\n".join(uncovered))
    sys.exit(1)
print("oracle allowlist covered the observed diff")

Wrap the checker so a maintainer runs one script after cloning the branch.

#!/usr/bin/env bash
# oracle/check_oracle.sh
set -euo pipefail
cd "$(dirname "$0")"
test -s before.txt
test -s after.txt
python3 check_allowlist.py

Adjust the exit-code rule to the issue rather than copying the sample. Some fixes must keep the same status and only change a message, and the allowlist should say so in words the checker can enforce.

Step 6: review the packet against the git hunks

Once the files exist, a second reader can compare three artifacts that humans often keep in separate windows. The first is observed.diff. The second is allowlist.txt. The third is git diff for the actual source change. Contradictions among those three are the review, not whether the patch looks tidy in isolation.

A narrow prompt keeps that second reader on the packet. The block below is a template, not a transcript of a run performed for this article.

You are reviewing an OSS bugfix packet, not writing new code.
Inputs: issue excerpt, git diff, oracle/before.txt, oracle/after.txt,
oracle/allowlist.txt, oracle/observed.diff, oracle/invoke.sh.
Report only:
1. Allowlist claims that do not appear in observed.diff
2. Observed.diff lines that the allowlist does not mention
3. Git hunks that cannot be tied to the declared delta
4. Steps in invoke.sh that look non-deterministic (time, net, random)
Do not suggest extra features. Do not rewrite the patch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access can run that constrained pass over the packet, and the free server option can replay invoke.sh plus check_oracle.sh on a machine that is not already loaded with the contributor's daemons. The packet remains the useful artifact if both of those options are skipped.

Local caches, language shims, and half-installed extras are a frequent source of unreproducible fixes that only exist on one laptop. A clean clone that reproduces before.txt and then after.txt is part of the proof. If a shared server is used for that clone, scrub host paths, tokens, and customer fixtures from the oracle files before they leave the machine.

Decision table for the packet

Signal in the packet	Meaning	Action before opening a PR
observed.diff is empty	The patch did not move the failing behavior	Keep debugging; do not request review
Diff is large, allowlist is short	Hidden refactors or noisy logs	Split the commit or quiet the invocation
Allowlist mentions lines absent from the diff	The declaration is aspirational	Rewrite the allowlist against real bytes
Exit code unchanged when the issue needs a status change	Incomplete fix	Add an EXIT: rule and recapture
Invocation embeds timestamps or absolute home paths	Replay will rot on another host	Fix invoke.sh before any model pass

Limitations

This workflow assumes a deterministic invocation. Flaky tests, live network calls, and clocks inside log lines will poison before.txt and after.txt on the second run. Binary fixtures and huge snapshot dumps also make the packet hard to review, so a focused assertion beats megabytes of pretty-printed JSON.

The allowlist checker is string oriented on purpose. It can miss semantic changes that keep the same text, and it can flag harmless path reordering that still confuses a reviewer. Contributors still need a human pass on the git hunks, especially around public API and error-code stability. A model will not see uncommitted files, private credentials in environment dumps, or project policy on breaking changes.

Do not paste secrets into a shared server or a model prompt. Oracle files often contain host paths, tokens from failed auth, or fixtures that should never be public. If the bug only reproduces with proprietary data, a public before-and-after packet is the wrong shape for the proof.

Who should skip this approach

Maintainers of one-line typo fixes do not need an oracle directory sitting in the review. Security patches that should not advertise exploit details in before and after logs should use a private disclosure channel instead of this packet. Contributors without a failing invocation, including design debates and API proposals, should write a decision record rather than invent a fake output pair.

Teams that already have a golden-test harness covering the same invocation can point the pull request at that harness and skip the extra files. The principle remains even then: declare the delta, prove it on both sides of the patch, and keep the proof next to the diff. Chat summaries disappear; replayable bytes do not.