Don't let an unattended model rewrite its own prompt

Swaraj Shekhar — Wed, 04 Nov 2026 03:03:22 +0000

Unattended LLM pipelines fail in ways a person would normally fix by editing the prompt. If the system rewrites that prompt itself, the rewrite is new logic that nobody has reviewed. In a chat window that is tolerable, because a person reads the next turn. In a batch job it is not. The consumer of the output is code, the document in the window is often large, and the person shows up later, on an exception queue.

This note describes a narrower rule. At runtime the system may only choose among plans a person approved once for that version of the task. When an attempt fails, the feedback it carries forward is a short list of typed template messages, not a growing transcript of the failure. The note also says what a small receipt-extraction pilot did and did not show.

This is a design to build on. It is not a finished system, and it is not a claim that the outputs got better.

The rule

Call the approved package an envelope. It holds the invariants, the allowed step graph, the list of plans the compiler may pick, the library of failure templates, and the numeric limits: attempts, time, prompt length, and how many times the same failure may repeat. People sign that package once per version. They do not sign each prompt the job emits.

A run on one record looks like this.

Assemble the prompt from the envelope: invariants, the unchanged document, one approved plan, the current failure notes, the task, and a short reminder of the critical rules.
Call the model. A deterministic validator returns pass, or a set of failing check ids.
On a pass, accept the output and log the prompt hash.
On a fail, write one template directive per failing group. A new failure in a group replaces the old directive. The ledger does not grow with the number of attempts.
A fuse opens if the attempt limit is hit, the same failure signature or the same plan has repeated enough times, the plan is not in the approved list, or the prompt would exceed its caps. The record then tries a short fallback menu, at most once per item. If that is exhausted, the record goes to a person.

A separate breaker watches the batch. A rising dead-letter rate, or too many silent failures in a sample of accepted outputs, pauses the batch until a person decides to resume.

The prompt layout is deliberate. Stable instructions sit in the system role. The document stays byte-for-byte the same on every attempt. Only the plan, the ledger, and a short note change, and each of those is drawn from the envelope.

What holds even if the model misbehaves

Three properties follow from the controller, provided the envelope signature checks out and the deployment assumptions hold. They say nothing about whether the answer is good.

The prompt length is capped. Each block has a token cap fixed in the envelope. The sum does not depend on the attempt number. A conversational retry that appends the last output and the last error gets longer every turn. This design refuses to send a prompt that would break the cap.

Every record ends. There is a finite attempt limit, a timeout on every call, and a finite fallback menu that cannot re-enter the main loop. The record is accepted or it is dead-lettered. The number of model calls is bounded.

The plan, the note, and the invariants stay inside the envelope. A plan must be one of the approved ones. A note must be an instance of an approved template. The invariant block is the one that was signed. Content inside the document is a different problem: the validator and the data-marking rules have to deal with it, and this design does not claim to have solved prompt injection.

Humans still have work. They approve envelope versions, they clear dead letters, and they sample accepted outputs. That sample is the only way to see failures the validator is too weak to catch.

What we ran

The pilot is receipt field extraction on CORD v1, a public dataset under CC BY 4.0 (Park et al., NeurIPS 2019 workshop;). Each record is a photographed retail receipt. The model saw text reconstructed from the word boxes, not the image. The task was nine amount fields plus the menu lines. A record was “clean” when every reference value appeared verbatim in that text.

The validator was intentionally loose. An amount passed if, after stripping currency marks and thousands separators, the number occurred somewhere in the receipt. A menu name passed if it occurred as a substring. A record can pass that check and still put the tax into the total. Arithmetic checks were not implemented.

The model was a local qwen3-8k quant at temperature 0.3, with an 8192-token context. Three conditions shared the envelope, the model, and the temperature: the controller, a single call, and a naive retry that repeated the same prompt with no failure directive. Seeds were 0, 1, and 2.

Envelope v2 told the model to copy values as printed, and it capped generation at 512 tokens. On the first 100 clean training receipts the controller dead-lettered the same six records on every seed, and rescued none of them. The directive added on the second attempt said the same thing as the first prompt, so the model did the same thing again.

We then read those six failures and wrote envelope v3. Two were truncated JSON: the receipt’s menu did not fit in 512 tokens. The other four were ambiguous instructions: a unit price marked with @, a line that prints both unit price and line total, or a value copied with its label (Total Rp 5.000 instead of 5.000). v3 raised the generation cap to 1024 and tightened the templates. That change was a new version, with a new hash, approved before the v3 runs.

What the pilot actually showed

On the same 100 training receipts, v3 accepted every record. Five of the six old failures were accepted on the second attempt, on every seed, after the typed directive was injected. The sixth, a truncated menu, passed on the first attempt once the generation cap was raised; no retry was required. One of the five, train-00103, is a cascade: the larger cap exposed a verbatim menu error that the truncation had hidden, and the directive then addressed that error on the second attempt. Naive retry, using the same v3 envelope and no directive, did not rescue those five records. A repeated identical prompt was not enough.

That pattern is consistent with a simple lesson. The ledger helps when the approved template tells the model something the first attempt did not. It does not show that the typed wording is what matters, as opposed to any extra hint. The two comparisons that would show that — a retry that carries only the check id, and a retry that appends the full failing output — were not run.

On 84 held-out clean test receipts, which were not used to write v3, the controller’s dead-letter rate was 2.4–3.6 percent (2 or 3 records out of 84). Single-shot and naive retry were both 7.1–9.5 percent (6 to 8 records). Naive retry’s dead-letter set matched single-shot on every seed. A paired exact test of controller against naive retry gave p-values from 0.0625 to 0.125. None of them is significant at 0.05, and with only four or five discordant pairs the test is too small to rule a real difference in or out. Two test records stayed dead letters on every controller seed. They are candidates for another envelope version, not evidence that v3 generalized cleanly.

Prompt length stayed small: about 388 tokens on the first attempt, and about 506 to 653 on the second, inside an 8192-token window. The length cap was not under pressure. A conversational baseline that appends the whole failure was not run, so this pilot does not show that the envelope wins a length contest against that style of retry. The receipts are short relative to the problem the design is aimed at.

Acceptance is not accuracy. On the training sample, a reference comparison disagreed with the CORD annotation on 11 of 14 accepted records for v2 (Wilson 95 percent interval about 52–92 percent) and on 10 of 14 for v3 (about 45–88 percent). A person reviewing the same 14 v3 records marked 8 wrong (about 33–79 percent). On the test set, the first 14 accepted records from one controller seed, taken in run order and not as a random sample, all disagreed with the annotation on at least one field (the interval for 14 of 14 still starts near 78 percent). No person reviewed that test sample. The intervals are wide. A permissive validator will still report a high accept rate on outputs that a reference, or a person, rejects.

What is safe to say

The controller can be built so that length, termination, and envelope membership do not depend on the model behaving. On these receipts, repeating the same prompt did not repair the systematic failures. A more specific approved template did repair the ones we had already inspected, and the held-out difference pointed the same way without clearing a significance bar.

The accept rates are validator passes, not extraction quality. The adherence comparison and the pre-set cap on human effort were never run as defined. Mean inputs were a few hundred tokens, so the long-document case is still open.

What would make this worth arguing with

The expensive part is writing the envelope. A narrow one blocks a legitimate fix. A loose one permits behavior the task owner did not intend. If a fixed prompt plus the same validator does as well as plan-switching inside the envelope, the runtime choice of plan is not the method. We did not time envelope authoring, and we did not freeze the plan and vary only the ledger.

The question left is practical. When is an approved envelope worth more than a fixed prompt and a strict validator? The pilot says the retry is only as good as the template a person was willing to sign. That is the part I trust enough to publish, and the part I most want checked before it goes out.

DEV Community: Swaraj Shekhar