DEV Community

Cover image for The optimizer made the prompt bigger
RLM Labs
RLM Labs

Posted on Originally published at rlmlabs.ai

The optimizer made the prompt bigger

In July 2026 I found the component built to shrink prompts making them bigger. One manifest line from a long run:

mode=compiler_pack  stage=execution  budget=47616
before=26461  after=26798  saved=-337
Enter fullscreen mode Exit fullscreen mode

26,461 tokens went in and 26,798 came out. In session 7b3256c2, 21 of 69 packing passes had negative savings. The optimizer had been reporting a minus sign for days, and nothing in the house was reading the sign.

The bigger cost sat somewhere else. A packing pass sorts the conversation into history and state, then rebuilds the window. My classifier pinned two messages: the system prompt and the latest user message. In a chat, those are the right two. In a tool loop, the latest user message is usually the injected task checklist. The model's working state, the file the model read and the result of the last operation, sits one or two messages earlier. Under budget pressure the packer dropped the result the model was acting on.

So the model read the file again. The pressure stayed, and the packer dropped the file again. The same 5.6 KB file was fetched more than ten times, at about 25,000 tokens per iteration of the crawl. The run used 3.4 million tokens against a 2 million cap. A check of the fleet afterwards found the same crawl in every long run between 5 and 8 July. Four days of a model rereading the same page, like a commuter who keeps losing his place.

Two rules came out of the incident on 8 July. The last four messages are now must-keep. In a tool loop those four are the working set: the last program, the program's result and the injected context. The packer compacts history, never state. And the optimizer is allowed to give up. A pass which fails to shrink the prompt at a self-reported confidence under 0.25 is discarded, and the unoptimized window goes out instead. Over those four days the optimizer's confidence never once exceeded 0.57. A tool at 0.57 confidence should not be touching the furniture.

The test: for every component reporting a saving, alert on the sign. Mine reported saved=-337 and carried on, and nothing was watching for a negative number.

Source: session 7b3256c2 from my logs, and Spec 007, AC-007-008-06 and AC-007-008-07, written 8 July 2026.

Top comments (2)

Collapse
 
ahmetozel profile image
Ahmet Özel •

That incident log—saved=-337 going unnoticed while the model reread the same 5.6 KB file ten times—is one of the best production postmortems on context compaction I have read. Chat heuristics that pin only the system prompt and latest user turn break immediately in agent loops where the active working set sits in the preceding tool output.

Compacting history while treating the last four tool-loop messages as active state, and alerting whenever a purported savings metric flips negative, are two invariants every prompt optimizer should enforce as hard assertions.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

Alerting on the sign of the saving is a sharp rule. A negative number in a field called saved should never pass quietly.