Yes, build one, and keep it thin. One internal function that takes a typed request, handles retries, timeouts, logging, and routing, and returns a typed response, with the provider SDK hidden behind an adapter. Don't build a general framework, and don't adopt a heavy one blindly. The leaky parts (tool-call formats, streaming shapes, structured-output modes, error codes) will leak through anything, and a thick abstraction means you spend more time fighting it than you would have spent switching providers.
What belongs in the thin layer
Roughly three hundred lines you own outright: a typed request and response, two or three provider adapters, model aliases in config, the reliability layer, logging and cost accounting in one place, and flags for model and prompt version. Every item on that list is work you'd do with or without an abstraction.
- A typed request (messages, tools, output schema, an alias for the model tier) and a typed response (content, tool calls, token usage, finish reason).
- Two or three provider adapters, each translating that request into the vendor's SDK call and back.
- Model aliases in config:
fast,smart,long-context, mapping to actual model ids per environment. Application code never mentions a vendor's model name. - Timeouts, bounded retries with jitter, a circuit breaker, and an idle-stream watchdog.
- Feature flags for model and prompt version, so rollback is a config change.
What to leave out
Prompt DSLs. Chain and graph orchestrators baked into the core. "Memory" modules with opinions about your database. Vendor-specific "agents" that hide their loop from you. Prompts stay as versioned files, and orchestration stays plain code on a queue you already run, with state in the database you already have.
When a calling convention changes, and it changes a few times a year, a thin layer means editing one adapter. A framework means waiting for a release and then reading its changelog with your fingers crossed.
| Thin layer you own | Heavy framework | |
|---|---|---|
| Size | ~300 lines, all yours | Thousands of lines, none yours |
| Breaking upstream change | Edit one adapter | Wait for a release, then adapt |
| Debugging a bad response | Your own stack, top to bottom | Nine frames of library code first |
| Cost to start | About two days | An afternoon |
| Cost to leave | Nothing to leave | A week, in our case |
I keep seeing teams reach for the framework because it feels like buying instead of building. In practice this is one of the places where the build-vs-buy line has moved toward building the small thing, precisely because the small thing is small.
When a heavier framework earns its place
Four cases earn it: prototyping, where speed of trying ideas beats everything; teams that genuinely need many providers and many modalities on day one; teams without the capacity to maintain even three hundred lines; and regulated environments where a vendor's audited framework saves a compliance conversation. All four are legitimate, and all four still need guardrails.
Pin the version. Wrap it behind your own interface so application code never imports it directly. Write one integration test per callback you depend on. Keep an ejection plan.
Our own lesson: we adopted a popular orchestration framework early on one product. A minor version bump changed the semantics of a streaming callback, and partial responses started being treated as complete. It took most of a day to trace, because the stack was nine frames deep in library code before it reached anything we wrote. Ejecting took a week. Writing our own layer at the start would have taken two days, and we'd have understood every frame of every stack trace since.
The switching test
The whole point of the layer fits into one test you can run in an afternoon, without touching application code. If it passes, your abstraction is doing its job. If switching means a branch, a refactor, a week of QA, and a nervous release, the abstraction has quietly become the thing you're locked into.
- Change the model behind the
smartalias in config, in one file. - Rerun the eval set against old and new.
- Compare pass rate, latency, and cost per call side by side.
- Decide, then roll forward or back with the same config change.
At Shanti Infosoft that thin client is shared across projects, and it has quietly outlived four generations of "must-have" frameworks, which tells you most of what I think about the trade. It's the default we start from on custom software development work where a model sits somewhere in the stack.
If your provider doubled prices tomorrow, how many files would you have to touch to move?
Sagar Jain is the technical co-founder of Shanti Infosoft; his team of 80+ engineers works to a CMMI Level 5 standard and typically has a first working build in front of a client in about two weeks.
Top comments (0)