The Prompt Is Not a Lockfile: A Provenance FAQ

Did your last free model run leave a paper trail? Or did the answer vanish into a chat tab? That gap is how unowned code ships.

I keep a short FAQ for this mess. You will hear these claims in standups. Each one sounds reasonable. Each one hides a broken mental model.

Why this FAQ exists

Free model access is useful. A free server is useful too. Neither one is a build system.

So what actually gets committed after the chat? A blob of text with no parent. Then someone asks why production drifted.

Sound familiar?

This is not a latency piece. This is not a quota piece. This is about ownership of drafts.

Myth 1: Same prompt, same output

The claim: "I saved the prompt. I can recreate it."

What people mean: The prompt file is the whole story.

The problem: Sampling is not a compiler. Wrappers inject hidden system text. Routing changes. Defaults move. Your "same prompt" is a different experiment.

Corrected model: A prompt is an input, not a lockfile.

Ask yourself one rude question. Can you name the wrapper around that prompt? If you cannot, you cannot replay the run.

# labeled example: fields a replay actually needs
prompt_path: prompts/extract_errors.md
prompt_sha256: (fill after hashing the file)
wrapper: chat-completions + unnamed system text
temperature: unknown   # this is the bug
seed: unset

No hash? No flags? Then you have folklore, not an input bundle.

Myth 2: I can just re-run it later

The claim: "It is free. I will hit the endpoint again."

What people mean: Cost is the only scarce resource.

The problem: Free lanes change under you. Capacity shifts. Defaults shift. Your future re-run is not a checkout from git.

Corrected model: A re-run is a new trial. Store the bytes now.

Do you keep the output bytes? Or only the story of the output?

# proposed local capture; wire this to your own client
mkdir -p artifacts/runs
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
# save whatever your client prints, then freeze it
# cp raw.out "artifacts/runs/${STAMP}.out"
sha256sum "artifacts/runs/${STAMP}.out"

If the file can mutate without a new hash, you do not have evidence. You have a sticky note.

Myth 3: The model already knows my repo

The claim: "I pasted the stack trace. It has context."

What people mean: Chat memory equals project memory.

The problem: Context windows forget. Tabs close. The next volunteer sees a patch with no files. Who owns that helper then?

Corrected model: If it is not in git, it is not context.

I ask a blunt question in review. Where is the before and after patch?

# proposed: never accept "the model said" without a diff
git status --short
git diff --stat
git diff > "artifacts/runs/${STAMP}.patch"

No patch file means no change you can name. A screenshot of a chat is not a source tree.

Myth 4: Cheap generation means skip the gate

The claim: "It compiled. Ship the helper."

What people mean: Review does not scale with token cost.

The problem: Debt is not priced in tokens. Ownership is. A free function still needs a test. It still needs a name on the pull request.

Corrected model: Generation cost and review cost are different ledgers.

Would you merge a stranger's gist because the download was free?

# proposed ownership gate, not a full eval suite
def test_helper_rejects_empty_payload():
    from app.helpers import parse_event

    try:
        parse_event({})
    except ValueError:
        return
    raise AssertionError("empty payload must fail")

Green on that test does not prove the model was wise. It proves a human picked a contract. That is the point.

Myth 5: Chat success equals CI success

The claim: "It worked in the playground."

What people mean: One lucky interactive run is a pipeline.

The problem: Playgrounds hide retries. They hide extra tools. They hide warm caches. CI is cold, scripted, and impatient.

Corrected model: The playground is a sketch. CI is the product.

If you cannot invoke it from a script, you do not have a workflow. You have a demo with good lighting.

# proposed: the smallest scripted invocation check
# replace the client with yours; fail closed if it is missing
command -v your-model-client >/dev/null || {
  echo "no scripted client; playground-only is a demo" >&2
  exit 1
}

The corrected mental model

Stop treating a free model run like a commit. Treat it like an untrusted draft with a receipt.

The receipt is a run manifest. The draft becomes owned only after a human and a test say so.

Three objects, always:

1. Input bundle: prompt, files, flags.
2. Output bundle: raw bytes, patch, logs.
3. Decision: keep, rewrite, or throw away.

Miss one object and you are storytelling. Storytelling does not survive the next on-call.

Artifact: a run manifest

This is a proposed template. It is unexecuted glue. I have not tied it to one vendor API. Wire it to your own client after you save a prompt file and an output file.

#!/usr/bin/env python3
"""Proposed run manifest helper. Not a vendor SDK."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()


def git_head() -> str:
    try:
        out = subprocess.check_output(
            ["git", "rev-parse", "HEAD"],
            stderr=subprocess.DEVNULL,
        )
        return out.decode().strip()
    except (OSError, subprocess.CalledProcessError):
        return "unknown"


def main() -> int:
    if len(sys.argv) != 3:
        print("usage: manifest.py PROMPT_FILE OUTPUT_FILE", file=sys.stderr)
        return 2
    prompt = Path(sys.argv[1])
    output = Path(sys.argv[2])
    if not prompt.is_file() or not output.is_file():
        print("prompt and output must exist", file=sys.stderr)
        return 2
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    record = {
        "recorded_at": stamp,
        "git_head": git_head(),
        "prompt_path": str(prompt),
        "prompt_sha256": sha256_file(prompt),
        "output_path": str(output),
        "output_sha256": sha256_file(output),
        "cwd": os.getcwd(),
        "status": "draft_until_tests_and_review",
    }
    dest = Path("artifacts/runs") / f"{stamp}.manifest.json"
    dest.parent.mkdir(parents=True, exist_ok=True)
    dest.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")
    print(dest)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

How you use it:

mkdir -p artifacts/runs prompts
# 1. put the prompt in git
# 2. run your model client however you already do
# 3. save stdout to artifacts/runs/latest.out
python3 manifest.py prompts/extract_errors.md artifacts/runs/latest.out

That is the whole trick. Boring on purpose. Boring is how receipts survive.

Paste this checklist on the PR:

• Prompt file path and hash recorded?
• Raw output file stored, not paraphrased?
• git diff captured if code changed?
• One failing test turned into a passing test?
• Named reviewer, not "the model"?

If any box stays empty, the draft is still a chat. Do not merge a chat.

Decision table

Claim you hear	Evidence to demand	Mental model to keep
Same prompt, same answer	Wrapper, flags, raw output hash	Prompt is not a lockfile
I will re-run later for free	Stored output bytes	Re-run is not checkout
The model knows the repo	A git patch	Chat is not VCS
It compiled, ship it	Test plus named owner	Tokens are not review
Playground said yes	Scripted CI invocation	Sketch is not product

Print that table. Drop it in the PR template. Then watch the myths shrink.

Where a free model and a free server fit

Need a place to generate the draft without burning paid quota? Fine. Use it as a sketch bench, not as source control.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. I mention that pair because this workflow needs cheap drafts. It does not need a new religion. You still write the prompt as a file. You still capture output. You still run manifest.py. The product you keep is the receipt, not the tab.

If you already generate on a free endpoint, wrap the next draft with a manifest before you argue about quality.

What this does not do

• It does not make sampling deterministic.
• It does not pin a hidden system prompt.
• It does not replace code review.
• It does not prove next month's route is identical.
• It does not bless secrets on a shared free server.

If your org handles regulated data, stop. Do not paste customer payloads into a free lane. This FAQ is about provenance of drafts. It is not a compliance program.

Who should skip this

Skip this approach if you need bit-identical replay for audits. Skip it if your only interface is a chat you cannot script. Skip it if you will not store outputs.

Also skip it if you wanted a leaderboard. This is plumbing. Plumbing is supposed to be dull.

Close the loop on one PR

Pick one merged change that started as a model draft. Can you find the prompt hash? Can you find the raw output? Can you find the test that accepted it?

If any answer is no, you do not have a model workflow. You have a myth with a green build.

Fix the next one. Leave the last fifty alone.