On 2026-08-22 the agent harness I work on shipped two "smarter context" features within five hours of each other. Both were reverted before the day was over. This is the honest version of that afternoon, because the same-day revert is the less told half of the fast-feedback story.
Fast-fix stories get the attention: "issue to merged fix in 50 minutes" is a good headline. But the mechanism only works if you are equally fast at the other direction. A bet that loses is cheap to fold at hour five and expensive at week five. This is the story of two bets, both folded in one afternoon.
09:33 - the tool-result sliding window
The first feature started as a host complaint at 11:33 local: long agent sessions were drowning the LLM in tool outputs. Read a file, list a directory, run a test - each result is a fat blob of text, and after ten rounds the context is mostly yesterday's tool output.
The fix (#936, built 16:33): a sliding window. At send time, tool-result groups older than the most recent N rounds are folded into a placeholder:
[Tool results omitted - older than recent 7 rounds]
executed: read_file x3, bash x2
tool_call_ids: ...
full results: history.jsonl
The full outputs stayed on disk. The model could in principle backtrack to the file. On paper it is a clean token economy: keep the recent context, compress the old, leave an audit trail. It even had a config knob (tool_window_rounds, 0 disables).
13:25 - the reasoning pass-back
Six hours later a second complaint landed at 17:25: DeepSeek's thinking-mode reasoning was being discarded. The model spends tokens thinking, then only the final answer comes back. Why not persist the reasoning and pass it back into context on the next round? (#937, built 19:46.) More context for free - the model's own reasoning as memory.
21:12 and 21:23 - both folded
By 21:12 the reasoning pass-back was reverted. By 21:23 the sliding window was reverted too (#939, released as v0.2.70 at 21:47). Two features, built and shipped over the afternoon, both gone before dinner.
What was the problem? The commits carry the trigger but not the full autopsy, so here is what we can verify from the code and from what the features did:
The sliding window failed SILENT. Folding is a bet that "the model will not need those older outputs again". When the bet lost - a round 12 decision that needed the round 3 file listing - there was no error. The model simply had less evidence, and no mechanism anywhere would tell us that a specific omission changed an answer. The placeholder was a breadcrumb, but a breadcrumb only helps if you know to look. Loss without a signal is the worst failure mode for an agent harness, because the output still looks fine.
The reasoning pass-back failed UNMEASURED. It made nothing break, so nobody could tell it was working. Re-consuming hidden chain-of-thought as context costs tokens and changes the model's distribution; without a counter or an experiment, "pass it back" was a belief, not a mechanism.
The lesson: know which direction your guard fails
The general shape is what stuck. Both features TRANSFORMED context - compressed it or re-injected it - and neither had a countable signal for when the transformation was hurting. A reader on our token-counter post said it better than we did: a guard that fails safe (does nothing, keeps everything) trains people to ignore it, and a guard that fails silent (quietly drops or changes something) cannot be audited at all. Overcount and undercount are the same disease in mirror: you only watch the direction that is easy to watch.
So the rule we took from that afternoon: context management must be anchored to a number from outside the system, and it must fire a countable event in both directions.
What we do now
The day after the revert (08-23) the auto-compact gate was rebuilt on that rule (#946): the decision to compact keys off the provider-reported usage number, not our estimate and not a transformation. Three reader-driven iterations since then have made that anchor falsifiable instead of trusted:
- #995 (reader heinrichneb: "does it land somewhere countable, a metric not just a line?"): anchor losses and drift are now countable events on disk, not log lines.
- #1003 (reader vinhnguyenthanhdn: "the anchor is keyed only by session - what if the model switches mid-session?"): a real model switch invalidates the anchor; the switch round re-anchors from the new provider's real number.
- #1029 (reader heinrichneb again, issue #1027: "the anchor can silently drift when the provider changes under an unchanged base_url"): a silent-drift detector re-checks the anchor against the provider on every use and fires an anchor_provider_drift event when the bias shifts past a threshold - direction-agnostic, over- or under-count - then re-anchors automatically. The anchor is now falsifiable by its own number.
And #1030 made the meta-loop measurable: the repo now carries a script that measures reader-feedback to merged-fix latency. Median so far: 38 minutes across the first three.
The pattern
Three pieces, in order of importance:
- Do not transform context. Verify against reality. The compact gate reads the provider's number; the anchor is re-checked on every use; nothing is silently compressed away.
- Know the failure direction of every guard you build, and give it a counter. Fails-safe trains people to ignore it; fails-silent cannot be audited; both are cheaper to find at hour five than week five.
- Revert fast. The same-day revert is not a failure of engineering discipline - it is the discipline. A bet that loses at hour five costs one afternoon. The same bet at week five costs a migration.
The features we killed that day were clever. The system that replaced them is dumber and better: it keeps everything, reads the real number, and counts when it drifts.
Top comments (0)