The magic window
Every AI framework ships with a demo that looks incredible. Spin up an agent in five lines. Chain three tools together in ten. Build a chatbot with memory in fifteen minutes. The getting-started experience is deliberately frictionless. And it works. You feel productive immediately.
Then you try to do something real.
Where the abstractions become walls
The problem isn't that these frameworks are badly built. Many of them are impressively engineered. The problem is that they're designed to optimize for the first 10% of development at the cost of the remaining 90%.
That remaining 90% is where you need to manage raw API payloads when the abstraction doesn't expose the field you need. Where you need custom retry logic because the default exponential backoff doesn't account for the rate limit patterns your specific provider uses. Where you need to track token costs per user, per session, per feature, and the framework only gives you aggregate totals. Where an agent chain fails silently at step four and you're debugging through three layers of abstraction to figure out what state was passed between steps.
Heavy abstractions help when everything works. They become walls when something doesn't. And in production, something always doesn't.
The debugging tax
This is the part that hits hardest in practice. When your lightweight wrapper breaks, you read the error, look at your code, and fix it. The call stack is yours. The state is visible. The flow is explicit.
When a framework breaks, you're reading through someone else's abstractions trying to figure out what happened between your input and the error. The state got transformed somewhere inside a chain you didn't write. The retry logic fired but you can't tell how many times because it's handled internally. The error message references an internal class name that doesn't appear in the documentation.
The time you saved in the first 10% gets paid back with interest during debugging. And debugging is where production teams spend most of their time.
The cost tracking problem nobody mentions
This one is specific to AI infrastructure but it matters a lot at scale. Most frameworks treat API calls as an implementation detail that happens behind the scenes. That's fine for a prototype. In production, you need to know exactly how many tokens each feature consumes, per user, per session, broken down by model. You need to set cost ceilings per workflow. You need to alert when a chain enters a retry loop that's burning through your budget.
Frameworks that abstract away the API layer also abstract away the cost visibility. You find out what something costs when the invoice arrives, not when the code runs. By then the budget damage is done and the forensics to figure out which workflow caused it requires instrumenting the framework from the outside, which is harder than building the instrumentation into your own code from the start.
When to build your own
Not always. But more often than the ecosystem wants you to believe.
If your use case is straightforward and matches the framework's happy path, use the framework. If your use case involves custom retry logic, granular cost tracking, complex state management across agent chains, or any production concern that the framework treats as an edge case, you're going to fight the abstractions more than they help you.
A lightweight wrapper around the raw API that you wrote, you understand, and you can debug in five minutes is often faster and safer than inheriting someone else's opinionated framework and spending days working around the opinions that don't fit your use case.
The architecture principle
Keep your architecture simple, explicit, and transparent. Every layer of abstraction you add is a layer you have to debug through when something breaks. Every opinion a framework encodes is an opinion you either agree with or work around. The magic of the first ten minutes is not worth the friction of the next ten months if the abstractions don't match your production reality.
The best AI infrastructure I've seen in production is usually the simplest. Thin wrappers, explicit state, visible costs, debuggable chains. Not impressive to demo. Extremely effective to operate.
How much of your AI stack is framework vs custom? And which part causes more production incidents?
Top comments (19)
Honestly, I've had to build everything custom. Starts off as simple refinements, like adding tooling to save tokens, things like building a dependency graph, or relationship table are huge token savers. But then you start looking at the rest of the overheads and it just cascades... Even MCP-Server is bad. Even JSON is bad. So current iteration is a standalone rust ide, with a rust mcp-server, operating in a custom deterministic language that doesnt require serialization and is easier for the LLM to understand. Oh and it never uses more than 50mb of ram (fixed amount). Frameworks are great, but they are fundamentally bloated. Once you start digging, you see all the little taxes you pay add up to half your credit bill, whereas just optimizing it ground up makes it possible for you to improve well beyond what's otherwise possible.
Building a custom deterministic language to avoid serialization overhead is a level of optimization most people would call overkill until they see the token savings. The dependency graph approach for context management is smart too because most of the waste in AI workflows isn't the inference itself, it's shipping redundant context on every call because the framework doesn't know what the model already has. The 50mb fixed RAM is impressive and it proves the point from the post taken to its logical extreme: every abstraction you remove is overhead you stop paying for. The "half your credit bill" observation is the part that should scare people using heavy frameworks at scale. Most teams have no idea how much of their API spend is framework overhead vs actual useful inference because the abstraction hides it. You're the per-call cost meter guy's argument taken all the way to the metal.
Exactly. Take MCP-server. simple in theory, node.js server + JSON. Till you consider that means hosting a node server. Serializing and deserializing json every single call. While serializing and deserializing everything is halted. What was slapped together as a POC became a standard, because nobody thought of the inefficiency of it, they just saw it's better than raw. That's kind of been the way everything was handled with AI, take something standard and slap a sticker on it. Take every single agentic IDE sofar, all are just glorified browser apps with a full chromium started with it. If Chrome eats up your ram, why would you expect your IDE to do anything else? It's a bloated ecosystem built on 'the tool's reliable, lets re-use it' and 'everyone knows how to write JSON, so lets standardize', never considering the incredible overhead it adds. My IDE operates at 1/10th the ram of even an idling fresh vs-code instance. My MCP operates in nanoseconds, not milliseconds. When that's per-call overhead, it starts adding up over a year...
The Chromium-in-every-IDE point is the one that deserves more attention. The industry standardized on shipping a full browser engine as a desktop app runtime and then acts surprised when developer tools consume 2GB of RAM at idle. Your nanoseconds vs milliseconds on MCP calls is an interesting case study in how "negligible overhead per call" stops being negligible when you multiply it by every interaction across a year of production usage. The broader pattern you're describing is real: the ecosystem optimized for developer familiarity (everyone knows JSON, everyone knows Node, everyone knows Electron) rather than for runtime efficiency, and that tax compounds silently until someone like you actually measures it and builds the alternative. Whether most teams would benefit from going as far as you did is debatable, but the fact that the savings exist at that scale proves the overhead is there for everyone, most just never look.
The real cost-saver isnt at the user-scale, it's at the provider scale. How much overhead does google have using standard mcp servers?
There was another guy on here, who built (correct me if I'm wrong, OrinIDE), essentially an IDE as a browser extension, instead of electron with it's own full chromium. 99% of people have a browser open anyway, so it's just reusing what's already available. Which is another way people are trying to save some ram. Though the intention of mine is more to do with V.E.L.O.C.I.T.Y. OS and eventually as an ultra-efficient cloud server OS + agentic layer, so they can swarm a task, though I'll need to build a git equivalent to manage it better. The thought being, currently you waste 2-4gb of ram on a server instance just for an OS and the agentic layer, dropping that number down to 8mb for the OS and 50mb for the ide/agentic layer, even a 1gb ram system can run it at full performance, not to mention stripping away the OS tax that windows and linux subsystems add.
The provider-scale point is where the math gets absurd. If Google is running MCP with Node + JSON serialization on every call across millions of requests, the aggregate overhead in compute, memory, and latency is a line item that would justify an entire team to eliminate. At individual developer scale, the waste is annoying. At provider scale, it's infrastructure cost that gets passed to customers.
The 8mb OS + 50mb agentic layer for cloud instances is an ambitious target but the logic is sound. If you're spinning up hundreds of instances for swarm tasks, the difference between 4GB overhead and 58MB overhead per instance is the difference between needing a serious cluster and running it on hardware most people would call modest. Curious to see where the git equivalent goes because that's where swarm coordination either works or falls apart. Good thread, this one went way deeper than I expected.
I gave it some thought earlier, think I have a pretty solid strategy now. turn the codebase into a standard knowledge graph, except agent context is appended to it, everything is timestamped validity, so it's idempotent. Zeor aloc, means that having a shared master graph and individual agent sub-graphs and interference checks to trigger a discourse between agents with a mediator, would work quite well. That way any conflicting edits are caught at write-time and resolved before a commit. Everything is is auditable with context and with a history. Diff vs git, would be that the agent sees the reasoning for the change, so it can spot regression early on and flag it in the graph as a disapproved edit, so it doesnt try make the same mistake twice.
All that with Rust zero-allocation, means that it's no serialization, just pure sync over wire between nodes and the master graph. Unlike normal git, it would basically be storing the live-context and full edit history of the codebase. Stripping out the JSON aspect, would mean though that while it stores more data, it's more compact and easier to search.
The knowledge graph as version control with agent context baked in solves a problem that git was never designed for. Git tracks file diffs. It doesn't track why the change was made, what the agent's reasoning was, or whether this change conflicts with an earlier design decision at the intent level rather than the line level. Your approach makes the reasoning a first-class artifact alongside the code, which means the conflict resolution happens at the decision layer not just the merge layer. The disapproved edit pattern is the part that interests me most because it's basically an ADR system that's machine-readable and enforced at write time. If the agent can see "this approach was tried before and rejected because X" before it proposes the same thing again, you've eliminated the re-litigation loop that eats hours in long agent sessions. Curious how the discourse between agents works in practice when two agents have legitimately conflicting valid approaches and the mediator has to pick one. That's where the hard design decisions live.
Functionally, the best solution is more performant. If both solutions are valid (eg. 1 agent making it async to make it high concurrency and the other just prevents locking threads), the logical choice is the one that's other benefits are better. Eg. Non-locking synchronous would still be lower concurrency, while async allows it to operate as fast as possible, better for higher concurrency. The logical choice is Async. But if the Async pattern creates an error, eg. some values calculated synchronously end up being garbled as a result, then the discourse becomes a resolution cycle for, do we switch pattern to synchronous for that 1 function, or do we switch the rest to async too. Performance, stability, scalability are the key factors, at which point async all would be the choice, unless the refactor is too large to handle effectively, or the user is afraid the shift would make it fragile, then the pick is to make it synchronous.
No 2 solutions are ever really identical, there's always a performance, stability, scalability tradeoff. That's why discourse needs to be focused on those points.
Framing the mediator's decision as a performance/stability/scalability tradeoff matrix is the right constraint. Without those three axes the mediator is just picking favorites. With them, the decision is derivable from measurable criteria rather than arbitrary preference. The "refactor is too large" escape hatch is important too because it introduces a practical cost dimension that pure technical analysis ignores. The theoretically optimal solution that requires touching 200 files is often worse than the slightly suboptimal solution that touches 3. That's engineering judgment, not just performance benchmarking, and encoding that into an automated mediator is the hard part. Good thread, genuinely one of the most interesting technical discussions I've had on here.
If it wasnt for the 'too big' hatch, everything would end up in Rust, because otherwise there's always a faster way to do it. I'm gunna integrate it over the weekend and give it a try, the IDE is looking and running pretty good sofar, but I'm busy adding multi-model support and multi-provider support (I saw OpenRouter has HY3 for free till the 21st!)
Ha the Rust gravity is real, everything looks like it should be rewritten in Rust if you let the optimizer in you run unchecked. Good luck with the integration this weekend, curious to hear how the knowledge graph conflict resolution plays out with real agent sessions. Drop a link when you've got something to show.
Will do, been scrambling to integrate OpenRouter quick into the ide as a provider, I learned yesterday that they have free models... And not tiny ones, Nemotron 3 ultra (550b MoE), for free? So got that wired up now, so I can have it expand on the IDE, then I'll integrate the Git 2.0, give it a thorough test and release the binary. I'll be sure to let you know once it's live
Nemotron 3 Ultra for free is a solid find for testing. Looking forward to seeing it, ping me when it's live.
Answering your closing question: my production incidents cluster on the parts I couldn't put a meter or a test on, which is exactly your four. So the line I draw isn't framework versus custom, it's keep custom whatever you'll need to instrument, measure, or assert on, retries, cost, state transitions, and let a framework own only what you'll genuinely never need to see inside. Point 3 hit hardest for me. I ended up writing a per-call cost meter, because aggregate totals hide the exact thing you're trying to control, and once cost is a number you see per request instead of per invoice, half the "why is this so expensive" incidents just vanish. Frameworks break you where they're opaque; the fix isn't more framework, it's moving the opaque part into code you can watch.
"Keep custom whatever you need to instrument" is a better decision rule than anything I wrote in the post. It cuts straight to the actual boundary instead of debating framework vs custom in the abstract. And the per-call cost meter point is exactly right. Aggregate totals are a budget management tool for finance. Per-request costs are an engineering tool for the team. They answer completely different questions and only the second one prevents the incident instead of explaining it after the invoice arrives. The framing at the end nails it too. The fix for opacity isn't a better framework, it's moving the opaque parts into code you own. That's the whole post in one sentence.
That's the split worth keeping: aggregate explains the invoice, per-request prevents it. Same number, opposite jobs. Enjoyed thinking against this one.
Same here. Good thread.