A few months ago, I was running long AI coding sessions that would just... stop working well after a couple of hours. Not crash. Just get worse. Slower, more expensive, and weirdly forgetful — like the model had too much on its mind.
Turns out, it did.
I dug into what was actually happening under the hood, and the answer surprised me: it wasn't the AI model that was the problem. It was everything I was feeding it.
The junk drawer problem
Think about how most AI tools handle memory. Every message, every file you opened, every decision you made gets shoved into context. Nothing gets cleaned up. Nothing gets organized. It's like a junk drawer that keeps growing — except every time the AI needs to find something in that drawer, you're paying for it. In tokens. In money. In slower responses.
And here's the part that really got me: this mess doesn't just cost more. It actually makes the AI's answers worse. Buried under redundant, outdated information, the model starts missing what actually matters.
What I found when I actually measured it
I decided to stop guessing and start measuring. I built a system to track exactly what was useful in a long AI session versus what was just noise — repeated file reads, decisions that got reversed three messages later, errors that were already fixed but kept getting mentioned again.
The results were honestly kind of embarrassing. A huge chunk of what gets fed to AI models in long sessions is just... repetition. Same information, described five different ways, sitting in context, costing money every single time the model has to process it.
So I built something to fix it. Three ideas made the biggest difference:
Organize, don't accumulate. Instead of one long messy transcript, I split everything into categories — goals, decisions, files touched, errors hit. Suddenly the system could pull exactly what it needed instead of re-reading everything.
Track the current decision, not every decision ever made. If someone says "let's switch to Postgres" after saying "let's use SQLite," most systems keep both floating around in context. Mine tracks the chain and keeps only what's actually true right now.
Save a snapshot, don't replay the whole story. For long sessions, instead of reconstructing everything from scratch, the system checkpoints where things stand and picks up from there.
The numbers that mattered to me
After building this out, the system was holding onto 76% of important tasks and 85% of key decisions correctly — while using a fraction of the tokens a "keep everything" approach would need. Every one of the 67 tests I wrote to check accuracy kept passing through each round of changes, which mattered to me more than any single benchmark number. A cheaper system that gives wrong answers isn't actually cheaper. It's just wrong and cheap.
Why this matters beyond my one project
Here's the thing I keep coming back to: efficiency and accuracy aren't actually enemies. Most of what gets cut when you clean up context is genuinely useless information. You're not sacrificing quality to save money — you're removing noise that was quietly making things worse anyway.
If you're building anything with AI models — a chatbot, a coding assistant, an agent that runs for a while — the biggest cost lever probably isn't the model you're using. It's how much irrelevant history you're dragging along with every single request.
Most systems have 30-50% of token usage sitting around as pure waste. No model upgrade required to fix that. Just better housekeeping.
I write about building practical AI systems — memory, context, and the unglamorous engineering that makes AI tools actually work in production. If this was useful, I'd love to hear what context problems you're running into in your own projects.
Top comments (0)