Why I put one thin layer between my app and every LLM provider

On my first AI feature I called one provider straight from the code that needed it. It worked, and for a while that was the right call.

Then I wanted a cheaper model for the boring tasks, like tagging and short summaries. That's when I noticed the provider was everywhere: keys in three places, retry logic copied around, and no clear idea what each feature actually cost.

Chalk's post this week cited an a16z survey where 81% of CIOs at Global 2000 companies now use three or more model families. I'm nowhere near that scale, but even with two providers the mess shows up fast.

What the layer does

It's small. Roughly:

• One function per task type, like summarize or classify, not per provider.
• The model for each task comes from config, so switching is a one line change.
• A timeout and one fallback model, so a slow provider doesn't hang a user request.
• A daily spend limit per key. I got this one after a retry loop ran all night once.
• A log line per request with task, model, tokens and latency.

What it cost me

Another thing to maintain. Tool calling and structured output formats differ between providers, so the abstraction leaks. I ended up with small adapter code per provider anyway.

It also tempts you to over build. I'd skip it until a second provider is actually on the table.

What I'd do again

The per task log. Seeing that one feature used most of the tokens changed what I worked on next more than any routing trick did.

If you're running more than one model, did you build your own layer or use a gateway?

Comments

[Rishita Sharma]

No strict validator in the layer itself. I'd rather the layer just route, retry and log, and let each feature parse the output it asked for, since the shape it needs differs anyway. Zod at the call site feels like the right spot to me, close to the code that actually breaks if the JSON is off.

[ai_adam]

Haha, học được bài này bằng cách đau khổ — lần đầu cũng nhúng OpenAI client thẳng vào service layer, sau đó đổi sang Anthropic phải tìm replace khắp codebase. Lúc đó nghĩ "chỉ đổi model thôi", không ngờ prompt template, error handling, retry logic, streaming response... mỗi provider lại khác nhau đủ kiểu.

Pattern thấy hiệu quả nhất là định nghĩa interface chung cho ChatCompletion (messages in, stream/response out) + adapter cho từng provider. Đừng quên normalize error codes về một enum nội bộ, nếu không try/catch sẽ rối tinh. Và log raw request/response ở layer này — debug prompt injection hay token limit thì cứu命.

Một chi tiết dễ bỏ qua: tool/function calling schema. OpenAI dùng tools, Anthropic dùng tools nhưng format khác, Gemini lại khác nữa. Viết một normalizer cho schema này sớm sẽ đỡ phải refactor sau khi đã ship feature PS: the tool I meant is on labagent .tech

[Harun - solo dev]

the daily spend limit is the hardest lesson to learn. a runaway retry loop on a 429 rate limit can drain a month's api budget while you sleep.

i built a similar routing layer using a cloudflare edge worker to dynamically switch between three models based on task complexity (a 120b general coder, a 20b speed fallback, and an r1 reasoning specialist). and your point about the abstraction leaking is spot on. trying to normalize tool calling and structured json outputs across different providers (like groq vs deepseek) is where the adapter code always gets messy.

but the per-task logging is the real roi. seeing exactly which routing path burned the most tokens completely changed how i optimized my prompt budgets.

curious about your adapter code: did you end up using a strict schema validator (like zod) in your layer to catch the structured output leaks before they hit your app, or do you just let the app handle the parsing fallback? 🐯

[Reid Marlow]

I started with a home-grown wrapper in Python and hit the exact same schema leak you did. Basic text generation and token counts normalize easily, but tool definitions and error codes differ just enough between APIs that a single generic schema wrapper breaks down.

What ended up working best was keeping the layer strictly about routing and telemetry, while leaving serialization to the caller. The wrapper handles the timeout, the fallback trigger, and the per-task SQLite log. If provider A throws a schema error or 429, the fallback catches it, but the wrapper doesn't try to normalize the tool payload itself. That kept the helper under two hundred lines and avoided building a third-party gateway clone from scratch.

[Rishita Sharma]

Leaving serialization to the caller is the part I'd steal. Every time I tried to normalize tool payloads the wrapper got bigger and still leaked. Routing, timeouts, fallback and a log is about all I want from it.