48-Hour Field Notes: The Patch Was Plausible Because Half My Prompt Never Arrived

Last weekend I spent two days on a task that should have taken an hour: getting a generated patch to actually fix the failing test it was written for. The patch applied cleanly, read well in review, and even tightened an unrelated assertion. It also never touched the file that mattered, because that file never made it into the prompt I thought I had sent. These are my field notes, written the way I wish somebody had written them for me on Friday afternoon.

The setup: one failing test, a manifest, and a free tier

I picked a small Python service with a single red test, then wired a loop where a model proposes a diff and a runner applies it on a clean machine. I treated the two availability claims I started from as availability and nothing more: free model access and a free server option on MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I actually controlled, before any request left the box:

• A manifest listing every file I intended to paste into the prompt.
• A recorded base commit SHA, so a patch could only apply to the tree it was written for.
• A wrapper that ran preflight, sent the request, validated the stream, and only then wrote a patch.
• Two cheap assertions: the payload stays under budget, and the stream ends on purpose.

Notice how much of that list is my own plumbing and how little of it is the model. That ratio is the entire lesson of this post, and it took me roughly thirty hours to internalize it.

Hour 3: an HTTP 200 says nothing about whether the model finished

The first patch came back with a successful status code, and I believed it. Why wouldn't I? The response body parsed, the diff hunks looked coherent, and the runner exited zero. What I had actually received was a stream that stopped mid-sentence after the connection dropped, and my client treated the last complete chunk as the final answer.

Streaming responses usually end with an explicit terminator line. If your client never checks for it, truncation is indistinguishable from completion, and a half-written JSON object just looks like a shorter patch. Here is the validator I now pipe every stream through, and it fails closed instead of guessing:

#!/usr/bin/env python3
"""validate_stream.py - read an SSE stream, fail loudly on truncation.

Usage:
    curl -sN "$ENDPOINT" -d @prompt.txt | python3 validate_stream.py > reply.txt
"""
import json
import sys

TERMINATORS = {"[DONE]", "done"}


def extract_text(obj):
    """Adjust to your provider's chunk shape. Return '' for keep-alives."""
    if isinstance(obj, dict):
        return obj.get("delta") or obj.get("text") or ""
    return ""


def main() -> int:
    saw_terminator = False
    chunks = []
    for raw in sys.stdin:
        line = raw.rstrip("\n")
        if not line.startswith("data:"):
            continue
        payload = line[len("data:"):].strip()
        if payload in TERMINATORS:
            saw_terminator = True
            continue
        try:
            obj = json.loads(payload)
        except json.JSONDecodeError:
            print(f"malformed chunk: {payload[:80]!r}", file=sys.stderr)
            return 2
        chunk = extract_text(obj)
        if chunk:
            chunks.append(chunk)
    if not saw_terminator:
        print("stream ended without a terminator: treating as incomplete", file=sys.stderr)
        return 3
    sys.stdout.write("".join(chunks))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The exit codes matter more than the parsing here. Code 2 means the wire protocol broke, code 3 means the answer is incomplete, and both of them stop the pipeline before a diff is ever written to disk.

Hour 9: my prompt receipt was hashing the wrong thing

This one hurt, because I thought I had already solved it. I kept a receipt for every request, and the receipt said the payload had a certain hash and a certain length. What the receipt did not know was that the assembled payload was larger than what I had budgeted for, and my assembler was silently slicing the tail to fit. Guess which file lived at the tail? The failing test.

So the model produced a plausible patch for the files it could see, and my receipt cheerfully certified a payload that never shipped. The fix is to build the payload, measure it, print the receipt, and only then decide whether to send it:

#!/usr/bin/env python3
"""preflight_prompt.py - refuse to send a prompt we cannot account for.

Reads a manifest of files plus a question, prints a receipt, and exits
nonzero when the payload exceeds our operator-set budget.
"""
import hashlib
import json
import sys
from pathlib import Path

BUDGET_CHARS = 24000  # our own ceiling, not a provider-imposed limit


def build(manifest_path: str) -> str:
    manifest = json.loads(Path(manifest_path).read_text())
    parts = [f"--- {name} ---\n{Path(name).read_text()}" for name in manifest["files"]]
    parts.append(manifest["question"])
    return "\n\n".join(parts)


def main() -> int:
    manifest = json.loads(Path(sys.argv[1]).read_text())
    payload = build(sys.argv[1])
    final_hand = payload.rstrip()[-80:]
    receipt = {
        "files": manifest["files"],
        "chars": len(payload),
        "sha256": hashlib.sha256(payload.encode()).hexdigest(),
        "est_tokens_chars_over_4": len(payload) // 4,
        "tail_preview": final_hand,
    }
    print(json.dumps(receipt, indent=2), file=sys.stderr)
    if len(payload) > BUDGET_CHARS:
        print("payload over budget: drop files or split the task", file=sys.stderr)
        return 1
    sys.stdout.write(payload)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The tail_preview field is the part I would keep even if I deleted everything else. Read the last eighty characters of your own prompt and ask yourself whether the thing you care about is still in there.

Hour 26: I was logging the reply, not the run

By the second evening I had two validators and still no reliable signal, because I was saving reply.txt and calling that the result. The reply is an input to a process, not the process. What I needed was a wrapper whose only job is to make failure loud and early:

#!/usr/bin/env bash
set -euo pipefail

BASE_SHA="$(git rev-parse HEAD)"
echo "base: $BASE_SHA" >&2

python3 preflight_prompt.py prompt.json > prompt.txt

curl -sN "$ENDPOINT" -d @prompt.txt | python3 validate_stream.py > reply.txt

# your own extraction step turns reply.txt into reply.patch
git apply --check reply.patch

With set -euo pipefail, an over-budget prompt, an incomplete stream, or a patch that will not apply all end the run before anything else happens. That is three gates, maybe forty lines of shell and Python, and they caught every single failure mode I hit over the weekend.

Decision table: when the free path fits my work

Situation	Free model access plus free server?	Gate I add first
One failing test, one repo, human reviews the diff	Yes	Payload budget and stream terminator
Wide refactor across many files	Only in batches	One manifest and receipt per batch
Hard deadline with nobody reviewing	No	Nothing saves you here
Code that cannot leave your network	No	Data policy decides, not tooling
Long unattended pipeline	Not as the only stage	Idempotent replay plus alerting
Learning an unfamiliar codebase	Yes	Read the receipt before the diff

Limitations I am not going to paper over

• The chars // 4 estimate is a crude heuristic. Real tokenizers diverge on code, unicode, and long identifiers, so treat the number as a smell test, not a measurement.
• My validator assumes server-sent events. A non-streaming response needs a different completeness check, such as inspecting the finish reason on the final object.
• Availability claims are operator-supplied and can change. Free access is convenient, not a capacity guarantee, and I would not build an SLO on top of it.
• Every gate costs latency and maintenance. For a throwaway script you run once, this is over-engineering, and I would skip it.

Who should not use this approach? Anyone with data residency constraints, anyone who cannot afford a validation stage in the loop, and any team whose pipeline pages a human at 3 a.m. when a stage returns a nonzero exit code.

What I would repeat on Monday

I would keep the manifest, the receipt with a tail preview, the terminator check, and the git apply --check against a recorded SHA. I would drop the twenty hours I spent reading patches that were fine and asking the wrong question about them. If you want to rehearse the same loop cheaply, this is the workflow I ran against MonkeyCode's free model access and free server option, and the scripts above are the part that actually did the work.

The patch was plausible for exactly one reason: I never verified what left my machine. What is the last thing you check before you trust a generated diff?