The magic window
Every AI framework ships with a demo that looks incredible. Spin up an agent in five lines. Chain three tools together ...
For further actions, you may consider blocking this person and/or reporting abuse
Honestly, I've had to build everything custom. Starts off as simple refinements, like adding tooling to save tokens, things like building a dependency graph, or relationship table are huge token savers. But then you start looking at the rest of the overheads and it just cascades... Even MCP-Server is bad. Even JSON is bad. So current iteration is a standalone rust ide, with a rust mcp-server, operating in a custom deterministic language that doesnt require serialization and is easier for the LLM to understand. Oh and it never uses more than 50mb of ram (fixed amount). Frameworks are great, but they are fundamentally bloated. Once you start digging, you see all the little taxes you pay add up to half your credit bill, whereas just optimizing it ground up makes it possible for you to improve well beyond what's otherwise possible.
Building a custom deterministic language to avoid serialization overhead is a level of optimization most people would call overkill until they see the token savings. The dependency graph approach for context management is smart too because most of the waste in AI workflows isn't the inference itself, it's shipping redundant context on every call because the framework doesn't know what the model already has. The 50mb fixed RAM is impressive and it proves the point from the post taken to its logical extreme: every abstraction you remove is overhead you stop paying for. The "half your credit bill" observation is the part that should scare people using heavy frameworks at scale. Most teams have no idea how much of their API spend is framework overhead vs actual useful inference because the abstraction hides it. You're the per-call cost meter guy's argument taken all the way to the metal.
Exactly. Take MCP-server. simple in theory, node.js server + JSON. Till you consider that means hosting a node server. Serializing and deserializing json every single call. While serializing and deserializing everything is halted. What was slapped together as a POC became a standard, because nobody thought of the inefficiency of it, they just saw it's better than raw. That's kind of been the way everything was handled with AI, take something standard and slap a sticker on it. Take every single agentic IDE sofar, all are just glorified browser apps with a full chromium started with it. If Chrome eats up your ram, why would you expect your IDE to do anything else? It's a bloated ecosystem built on 'the tool's reliable, lets re-use it' and 'everyone knows how to write JSON, so lets standardize', never considering the incredible overhead it adds. My IDE operates at 1/10th the ram of even an idling fresh vs-code instance. My MCP operates in nanoseconds, not milliseconds. When that's per-call overhead, it starts adding up over a year...
The Chromium-in-every-IDE point is the one that deserves more attention. The industry standardized on shipping a full browser engine as a desktop app runtime and then acts surprised when developer tools consume 2GB of RAM at idle. Your nanoseconds vs milliseconds on MCP calls is an interesting case study in how "negligible overhead per call" stops being negligible when you multiply it by every interaction across a year of production usage. The broader pattern you're describing is real: the ecosystem optimized for developer familiarity (everyone knows JSON, everyone knows Node, everyone knows Electron) rather than for runtime efficiency, and that tax compounds silently until someone like you actually measures it and builds the alternative. Whether most teams would benefit from going as far as you did is debatable, but the fact that the savings exist at that scale proves the overhead is there for everyone, most just never look.
The real cost-saver isnt at the user-scale, it's at the provider scale. How much overhead does google have using standard mcp servers?
There was another guy on here, who built (correct me if I'm wrong, OrinIDE), essentially an IDE as a browser extension, instead of electron with it's own full chromium. 99% of people have a browser open anyway, so it's just reusing what's already available. Which is another way people are trying to save some ram. Though the intention of mine is more to do with V.E.L.O.C.I.T.Y. OS and eventually as an ultra-efficient cloud server OS + agentic layer, so they can swarm a task, though I'll need to build a git equivalent to manage it better. The thought being, currently you waste 2-4gb of ram on a server instance just for an OS and the agentic layer, dropping that number down to 8mb for the OS and 50mb for the ide/agentic layer, even a 1gb ram system can run it at full performance, not to mention stripping away the OS tax that windows and linux subsystems add.
Answering your closing question: my production incidents cluster on the parts I couldn't put a meter or a test on, which is exactly your four. So the line I draw isn't framework versus custom, it's keep custom whatever you'll need to instrument, measure, or assert on, retries, cost, state transitions, and let a framework own only what you'll genuinely never need to see inside. Point 3 hit hardest for me. I ended up writing a per-call cost meter, because aggregate totals hide the exact thing you're trying to control, and once cost is a number you see per request instead of per invoice, half the "why is this so expensive" incidents just vanish. Frameworks break you where they're opaque; the fix isn't more framework, it's moving the opaque part into code you can watch.
"Keep custom whatever you need to instrument" is a better decision rule than anything I wrote in the post. It cuts straight to the actual boundary instead of debating framework vs custom in the abstract. And the per-call cost meter point is exactly right. Aggregate totals are a budget management tool for finance. Per-request costs are an engineering tool for the team. They answer completely different questions and only the second one prevents the incident instead of explaining it after the invoice arrives. The framing at the end nails it too. The fix for opacity isn't a better framework, it's moving the opaque parts into code you own. That's the whole post in one sentence.
That's the split worth keeping: aggregate explains the invoice, per-request prevents it. Same number, opposite jobs. Enjoyed thinking against this one.
Same here. Good thread.