Conversation Summarization Breaks After a Provider Switch

#ai #debugging #llm #machinelearning

The report usually arrives as a size complaint. “Our conversation summaries were about 120 words and after the switch they are 400.” Or the reverse: they collapsed to two sentences and the agent started forgetting what the user asked for three turns ago. Nothing in the prompt changed. The prompt was never specifying a length in the first place.

The symptom

Two distinct failures present the same way, and they need opposite fixes, so separate them before you touch the prompt.

Length drift. The summary is coherent and complete but systematically longer or shorter than it was. Downstream, that shows up as context pressure, a higher bill, or lost detail.
Structural drift. The summary changed shape — it became a bulleted list where it was prose, it started including preamble (“Here is a summary of the conversation:”), or it dropped a section your parser expects. If anything downstream reads the summary with a regex or a schema, this is the one that breaks loudly.

Check the raw stored summaries for both before deciding. A length histogram over a few hundred stored summaries from each model, plotted side by side, answers the question in one look and costs nothing but a query.

Concise is a relative instruction

The overwhelmingly common summarization prompt says something like “summarise the conversation so far concisely, preserving any decisions and open questions”. There is no length in that sentence. “Concise” is resolved by the model against a prior it learned during post-training, from data where somebody decided what concise looked like. Different post-training, different prior.

The same applies to every other relative word people put in these prompts: brief, detailed, high-level, thorough. Each is a request to sample from the model’s idea of a distribution, and you tuned yours by trial and error against one model until the output looked right. That tuning was a calibration against a specific prior, and it does not transfer. This is the same mechanism that makes prompt portability hard in general, concentrated into one word.

Asking for a word count directly helps but only partly, because models cannot count their own output tokens while producing them. A request for “about 100 words” is another prior — a better anchored one — not a constraint. Expect it to land in a range, and plan to enforce the actual limit outside the model.

Why a rolling summary diverges

If your agent keeps a running summary — summarise, append the next few turns, summarise again — a small change in compression behaviour is not a small change in outcome. Model the loop. Let L(n) be the summary length after the nth compaction, a the fraction of the previous summary that survives into the next one, and b the tokens the new turns contribute:

L(n+1) = a * L(n) + b

fixed point:   L* = b / (1 - a)      when a < 1
divergent:     L(n) grows without bound when a >= 1

With a = 0.6 and b = 200, the summary settles at 500 tokens and stays there forever. With a = 0.9 and the same b, it settles at 2,000. With a = 1.0 — a model that faithfully preserves everything it is given and adds the new material, which is exactly what a diligent instruction-follower does when told to “preserve all decisions” — it never settles at all. It grows linearly until it hits your context limit, at which point the symptom stops being “summaries are long” and becomes a request failure or a truncation.

The migration changed a. That is the entire story, and it is why a summarization regression can look fine for the first dozen conversations in staging and then break on the long-running threads in production. Instrument a directly: log the token length of every summary alongside the thread id and the compaction index, and the ratio between consecutive entries is your a per model.

The other direction: silent truncation

Before concluding the model got terser, rule out the boring explanation. If the summarization call sets an output cap and the new model is more verbose, the response is being cut off mid-sentence and your storage layer is saving the fragment. The completion will report a length-limited termination rather than a normal stop, so check the field: an OpenAI-shaped response gives finish_reason and an Anthropic-shaped one gives stop_reason, and a cap hit is a distinct value in both. If you never inspected that field, you have been storing truncated summaries and calling it terseness.

Also check whether your cap parameter survived the migration at all. The field that limits output length is not spelled the same across APIs and has been renamed within one of them; a request whose cap silently reverted to a provider default is a common cause of the opposite symptom, summaries that ballooned. The distinction between that cap and the model’s total window is covered in context window versus max tokens.

Re-anchoring the prompt

Replace the relative word with a structure. Give the summary a fixed skeleton with a stated slot count — three bullets of decisions, up to five open questions, one paragraph of context — rather than an adjective. A structural constraint is checkable and it transfers between families far better than “concise” does, because it does not depend on the model’s idea of a good length.
State the budget in the units the model can see. Characters or words, not tokens. Then measure the real distribution of what you get and pick the wording that centres it where you want.
Enforce outside the model. Count tokens on the produced summary. If it exceeds the budget, run one repair pass (“this is too long, cut it to N words, keep every decision and every open question”). One pass, then accept and log — a repair loop that can run forever is a cost incident waiting for a pathological thread.
Pin a below 1 explicitly. Tell the summarizer that older material may be compressed further as the thread grows, and that only decisions and unresolved questions are preserved verbatim. Without a licence to forget, a well-behaved model will not forget, and the loop diverges.
Regression-test on facts, not on text. Summaries will not be string-identical between models and should not have to be. Take a set of real threads, write down the facts each summary must contain, and assert containment. That is the test described in testing that a summarization keeps its facts, and it is the only summarization test that survives a model swap.
Re-tune the wording against the new model, once. With the structure fixed and a length gate in place, the remaining adjustment is small: usually a single sentence about verbosity. Do it against your stored thread set so the change is visible as a distribution, not as an impression from three examples.