The magic window
Every AI framework ships with a demo that looks incredible. Spin up an agent in five lines. Chain three tools together in ten. Build a chatbot with memory in fifteen minutes. The getting-started experience is deliberately frictionless. And it works. You feel productive immediately.
Then you try to do something real.
Where the abstractions become walls
The problem isn't that these frameworks are badly built. Many of them are impressively engineered. The problem is that they're designed to optimize for the first 10% of development at the cost of the remaining 90%.
That remaining 90% is where you need to manage raw API payloads when the abstraction doesn't expose the field you need. Where you need custom retry logic because the default exponential backoff doesn't account for the rate limit patterns your specific provider uses. Where you need to track token costs per user, per session, per feature, and the framework only gives you aggregate totals. Where an agent chain fails silently at step four and you're debugging through three layers of abstraction to figure out what state was passed between steps.
Heavy abstractions help when everything works. They become walls when something doesn't. And in production, something always doesn't.
The debugging tax
This is the part that hits hardest in practice. When your lightweight wrapper breaks, you read the error, look at your code, and fix it. The call stack is yours. The state is visible. The flow is explicit.
When a framework breaks, you're reading through someone else's abstractions trying to figure out what happened between your input and the error. The state got transformed somewhere inside a chain you didn't write. The retry logic fired but you can't tell how many times because it's handled internally. The error message references an internal class name that doesn't appear in the documentation.
The time you saved in the first 10% gets paid back with interest during debugging. And debugging is where production teams spend most of their time.
The cost tracking problem nobody mentions
This one is specific to AI infrastructure but it matters a lot at scale. Most frameworks treat API calls as an implementation detail that happens behind the scenes. That's fine for a prototype. In production, you need to know exactly how many tokens each feature consumes, per user, per session, broken down by model. You need to set cost ceilings per workflow. You need to alert when a chain enters a retry loop that's burning through your budget.
Frameworks that abstract away the API layer also abstract away the cost visibility. You find out what something costs when the invoice arrives, not when the code runs. By then the budget damage is done and the forensics to figure out which workflow caused it requires instrumenting the framework from the outside, which is harder than building the instrumentation into your own code from the start.
When to build your own
Not always. But more often than the ecosystem wants you to believe.
If your use case is straightforward and matches the framework's happy path, use the framework. If your use case involves custom retry logic, granular cost tracking, complex state management across agent chains, or any production concern that the framework treats as an edge case, you're going to fight the abstractions more than they help you.
A lightweight wrapper around the raw API that you wrote, you understand, and you can debug in five minutes is often faster and safer than inheriting someone else's opinionated framework and spending days working around the opinions that don't fit your use case.
The architecture principle
Keep your architecture simple, explicit, and transparent. Every layer of abstraction you add is a layer you have to debug through when something breaks. Every opinion a framework encodes is an opinion you either agree with or work around. The magic of the first ten minutes is not worth the friction of the next ten months if the abstractions don't match your production reality.
The best AI infrastructure I've seen in production is usually the simplest. Thin wrappers, explicit state, visible costs, debuggable chains. Not impressive to demo. Extremely effective to operate.
How much of your AI stack is framework vs custom? And which part causes more production incidents?
Top comments (1)
Answering your closing question: my production incidents cluster on the parts I couldn't put a meter or a test on, which is exactly your four. So the line I draw isn't framework versus custom, it's keep custom whatever you'll need to instrument, measure, or assert on, retries, cost, state transitions, and let a framework own only what you'll genuinely never need to see inside. Point 3 hit hardest for me. I ended up writing a per-call cost meter, because aggregate totals hide the exact thing you're trying to control, and once cost is a number you see per request instead of per invoice, half the "why is this so expensive" incidents just vanish. Frameworks break you where they're opaque; the fix isn't more framework, it's moving the opaque part into code you can watch.