What to Do About In-Flight Requests When You Roll Back a Prompt

#ai #machinelearning #llm #programming

A rollback changes which prompt new requests get. It does nothing to the requests already running, and on an LLM path those live much longer than the ones you are used to. Deciding what happens to them is a real design decision, and defaulting to “whatever the code happens to do” produces the confusing half-states people remember about an incident.

The middle state is longer than you think

Add up the durations honestly. A single completion may run for many seconds. A streamed response holds a connection open for the whole generation. An agent loop can run for a minute across several model calls and tool executions. A queued batch job may not start for minutes or hours after it was accepted. And a multi-turn conversation can span days while remaining, from the user’s point of view, one continuous thing.

So the honest statement of the recovery time is not “the rollback took 15 seconds”. It is 15 seconds plus the longest unit of work that had already started, and for a queue-backed feature that can be the dominant term. If your incident review reports the first number, it is reporting the wrong thing.

There are four cases and they need different answers.

Pin the version at entry, not per call

The first thing to get right is where in your code the prompt version is resolved. If it is resolved once per model call, a single agent loop can straddle the switch: round one uses version 7, round three uses version 6, and the conversation the model is reasoning over now contains instructions from two different prompts. That is a state neither version was tested in, and it is strictly worse than either.

Resolve once, at the boundary of the unit of work, and carry the resolved version through:

# One resolution per unit of work, carried explicitly.
async def handle_request(req):
    selected = active_prompt("support.triage")     # resolved exactly once
    ctx = RequestContext(
        request_id=req.id,
        prompt_version=selected["version"],
        prompt_body=selected["body"],
    )
    return await run_agent_loop(ctx, req.message)  # every call inside uses ctx

# For queued work, pin at enqueue time and store it with the job, so a job
# that waits an hour runs the version that was current when it was accepted
# -- or deliberately re-resolve at dequeue. Both are defensible; silently
# doing whichever the code happens to do is not.
job = queue.enqueue(
    "summarise",
    payload=payload,
    prompt_version=active_prompt("billing.summary")["version"],
)

For queued work the two policies genuinely differ. Pinning at enqueue gives consistency — the job behaves as the user was told it would when they submitted it — but it means a rollback does not reach the backlog, and if the backlog is an hour deep you will keep producing bad output for an hour. Re-resolving at dequeue makes the rollback effective immediately at the cost of jobs behaving differently from their siblings submitted in the same batch. Pick per queue: for anything a human is waiting on, pin; for a large backfill, re-resolve, and add a way to purge the backlog entirely.

Streams that are already half-delivered

An open stream is the hardest case because the user has already read part of the answer. Three policies, and the choice depends on what the bad prompt was doing.

Let it finish. The default and usually right for a quality regression. The user gets one more mediocre answer; interrupting them mid-sentence is a worse experience than that.
Cut and restart. Terminate the stream, clear the rendered text, re-run on the old prompt. Justified when the output is actively harmful — leaked internal instructions, unsafe content, exposed data from another tenant. This needs client support for a replace-not-append event, which means it has to be built before the incident, not during it.
Cut and apologise. Terminate, show a clear error with a retry. Cheapest to implement and honest. If you have no replace mechanism, this is the correct fallback for the harmful case; silently truncating is not, because a truncated answer looks like a complete one.

Whichever you pick, make sure a terminated stream is distinguishable from a completed one in your records. A stream that ends without a terminal event and without a finish_reason is the signature of an abort, and if your client treats a closed connection as a normal end, your metrics will report those as successes.

Conversations that span the switch

Multi-turn is the case that persists longest after the incident is declared over. A conversation started under the bad prompt has that prompt’s output in its own history, and every later turn reads it. Rolling back changes the system prompt; it does not remove the three assistant messages already in the transcript that followed the old instructions.

This matters concretely when the bad prompt changed output format. If version 7 started producing a different structure and your later turns rely on the earlier ones, the reverted prompt is now reading history that does not match what it expects. Behaviour in that state was never tested by either version’s evaluation run.

Practical options, in ascending order of effort: accept the mixed history and rely on the model handling it, which is usually fine for conversational text and unreliable for structured output; start a new conversation for affected users, which is heavy-handed but unambiguous; or, where you can identify a small number of affected conversations from your logs, drop the affected turns from the history you send while leaving them visible to the user. The last is the most surgical and requires that you can find the affected turns — which you can, if you recorded the prompt version on every response, as in tracing a prompt version to a response.

What to tell users, and what to record

Most of these requests will complete normally with slightly worse output, and that needs no announcement. The cases that do need something said are the ones you actively interrupted, and the message should be specific enough to be actionable: that the answer was interrupted, that retrying is expected to work, and that no charge or quota was consumed if that is true. “Something went wrong” on a request you deliberately cancelled is a lie you will pay for in support volume.

For the record, count these separately rather than folding them into the incident’s error total:

Requests that started on the rolled-back version, which is the real blast radius — not requests that started after the switch.
Streams terminated deliberately, tagged as such so they are not counted as provider failures.
Queued jobs that ran on the old version after the rollback, and how long the tail lasted. This is the number that argues for changing the queue policy next time.
Conversations containing turns from both versions, which is your list of accounts worth checking by hand.

The last one is the one that gets skipped and the one that produces a support ticket three days later, long after everyone has stopped thinking about the incident.