You know that moment — you're three hours into an AI session and everything is clicking, the code is clean, the architecture is sound. You're a genius. You could refactor the universe!
Then you type one more prompt, the exact same kind that worked two hours ago, and what comes back is… off.
You rephrase, nudge it, add context, speak encouraging words to yourself as you retype it all. Yet... the response gets worse.
That's context window degradation. If you're using AI for anything beyond one-shot queries, it's happening to you right now.
The problem
I built a governance system for AI-assisted development — structured workflows, knowledge files loaded into Claude Projects, quality gates, convention enforcement. It worked. My waste rate dropped dramatically.
But the governance system itself was quietly consuming the same scarce resource it was supposed to manage — context window attention.
The research backs this up. Liu et al. (2024) found that model accuracy drops 30%+ when critical information sits in the middle of the context window. On the MRCR v2 benchmark, even the best model only retrieves 76% at full context length. Some models advertising million-token windows score below 25%.
A bigger desk doesn't help if your brain ignores the middle of it.
Last week, Microsoft's VS Code team published results from an A/B test on their AI coding agent. A compact prompt nudge barely moved the needle. A structurally rewritten prompt — with explicit phases and boundaries — won decisively. Structure beats cleverness, even at Microsoft scale.
The measurement
I use Claude, and I build my workflows inside Claude Projects — Anthropic's way of giving the AI persistent context across sessions. You load knowledge files into the project — how you name things, how you structure files, which libraries you use, what "done" looks like. Your project knowledge files are the rules you've learned the hard way that you don't want to re-explain every session.
I pulled the word count of every knowledge file and estimated token footprint (word count × 1.3).
Seventeen files. Convention guides, formatting rules, library references, workflow definitions, schema files, code skeleton templates.
The total: ~46,000 tokens of governance material loaded before I typed a single prompt.
And that's just the knowledge files — not the system prompt, not the model's instructions, not a single word of conversation history. 46K tokens is the floor — the cost of opening the door.
How big is the window? It depends on the model:
Other models (ChatGPT, Gemini, etc.) have their own context window sizes and handle long-context differently. This post focuses on Claude because that's what I use — the economics apply universally, but the specific numbers don't.
The fix: three strategies, zero new tools
For each knowledge file, I asked one question: what are the use cases where this material actually gets referenced?
Once you frame it as utilization rate — "How often does this asset justify its cost for the use case at hand?" — the decisions almost make themselves.
Migrate to on-demand. Four library reference files moved off the desk. They live on disk now, pulled in only when the session's use case involves that library. Savings: ~6,300 tokens freed from ~80% of sessions.
Split and compress. Two large convention files got split into compact indexes (always loaded) and full skeleton templates (loaded only when generating new files). A cost analysis dashboard got replaced with a compact summary. Savings: ~10,100 tokens removed.
Compact in place. A lint rule registry got reduced from full constraint details to just ID, summary, and severity. Full details stay in the canonical registry, fetched when needed. Savings: ~2,400 tokens.
Result: ~46K → ~27K tokens in a typical session. A 41% reduction in fixed governance overhead.
No retrieval pipeline needed. No vector database required. For this problem, just better allocation decisions.
The bridge between the tiers is a routing index — a file that says "if the current task involves library X, load reference Y." The index itself costs almost nothing. It's a table of contents, not a library.
What else helped
Control response verbosity. Every AI response stays in the conversation history, which means it stays on the context window for the rest of the session. A three-paragraph status update when a sentence would do is a carrying cost you'll pay for two hours. The AI is happy to be concise, it's usually verbose because nobody told it not to be!
Shorter sessions. The research has a practical implication most developers resist: shorter, focused sessions with clean handoffs outperform marathon sessions almost every time. Five minutes writing a handoff saves thirty minutes of degraded work.
Build a gauge. A simple green/yellow/red indicator that estimates context pressure based on exchange count, topic shifts, and loaded material. It's not a separate tool — it's a set of rules built into the AI's own instructions. The AI self-assesses and reports. Now I see yellow and start wrapping up on my own terms.
The deeper lesson
There's a principle from search engineering that applies here: the quality of your results depends more on how you organize the data going in than on how cleverly you query it coming out. Knowledge files, workflow definitions, routing indexes — all of that is shaping the input. You're making interpretation predictable by structuring what the model sees, not by crafting the perfect prompt.
There's an emerging cottage industry around prompt optimization. That work matters. But treating it as the whole solution is like bailing with a thimble while ignoring the hole in the hull. Both matter. But one of them stops the flooding more consistently over time.
Every mature engineering discipline eventually discovers that its control systems have carrying costs. Logging consumes storage. Monitoring consumes compute. Security introduces latency.
AI governance consumes attention.
The fix isn't to abandon governance. Unstructured AI development is still far more wasteful. The fix is to treat governance like the investment it is — worth making, but only when the carrying cost stays below the waste it prevents.
My dad went to Navy destroyer school in 1966. He was the only one who passed the damage control exercise on the first try — because he ignored the big dramatic leak and went after the small ones first.
"The big leak gets all the attention. The small ones sink the ship."
Keep your eye on the water level.
References
- Liu et al. (2024) — Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.
- Microsoft VS Code Team (2026) — Optimizing VS Code's Coding Harness for Model Providers. code.visualstudio.com.
- Palomares Carrascosa, I. (2026) — Context Window Management for Long-Running Agents. MachineLearningMastery.com.
- Palomares Carrascosa, I. (2026) — Context vs. Memory Engineering in Agentic AI Systems. MachineLearningMastery.com.
- Anthropic — What Are Projects?. support.claude.com.
- Anthropic — Context Windows Documentation. platform.claude.com.
This is the second post in a series on the economics of AI-assisted development. The first post covers the waste problem that led to building a governance system in the first place. Full version with all graphics and the complete measurement methodology on Substack.
The views and opinions expressed here are my own and do not represent those of my employer or any other organization, entity, or group I am affiliated with. This post reflects my personal experience on a personal project.
Top comments (0)