Add a New AI Provider Behind One Route Contract, Not Across Your Codebase

New model releases land fast. The first question on my team is never "is the model good?" — it's "how many files do we touch to try it?" If the honest answer is "the prompt builder, three handlers, the retry logic, and the billing page," then the model isn't the problem. The architecture is.

This article walks through a working vertical slice: one user action (submitting a code question from a UI), one API route, one provider seam, and a test that proves a brand-new provider — including a free-tier one you want to evaluate this week — is a config change, not a refactor.

The user action, and where it first breaks

A developer types a question into a panel, the frontend POSTs to /api/ask, and the backend forwards it to an LLM. The first place this breaks when a new model ships isn't the model call. It's the handoff: prompt shape, streaming format, error codes, and token accounting are usually hard-coded next to the provider's SDK call, duplicated anywhere the app talks to the model.

So when something like DeepSeek V4 Flash appears and you want a low-cost way to evaluate it, you're not evaluating the model — you're re-implementing your integration in three places and hoping they agree.

The contract

Everything provider-specific lives behind one interface. The route only knows this shape:

# providers/base.py
from dataclasses import dataclass
from typing import AsyncIterator, Protocol

@dataclass
class Usage:
    input_tokens: int
    output_tokens: int

@dataclass
class AnswerChunk:
    text: str
    usage: Usage | None = None  # only on the final chunk

class AnswerProvider(Protocol):
    name: str
    async def ask(self, question: str, *, max_output_tokens: int) -> AsyncIterator[AnswerChunk]:
        ...

The route enforces our error contract — never the provider's:

# routes/ask.py
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel

router = APIRouter()

class AskRequest(BaseModel):
    question: str
    max_output_tokens: int = 1024

@router.post("/api/ask")
async def ask(body: AskRequest, request: Request):
    provider = request.app.state.provider  # injected at startup
    try:
        chunks = []
        usage = None
        async for chunk in provider.ask(
            body.question, max_output_tokens=body.max_output_tokens
        ):
            chunks.append(chunk.text)
            if chunk.usage:
                usage = chunk.usage
    except ProviderRateLimited:
        # our contract: always 429 with Retry-After, regardless of provider
        raise HTTPException(
            429, detail="provider_rate_limited",
            headers={"Retry-After": "30"},
        )
    except ProviderAuthError:
        # never leak upstream auth detail to the client
        raise HTTPException(502, detail="provider_unavailable")
    return {
        "answer": "".join(chunks),
        "provider": provider.name,
        "usage": usage.__dict__ if usage else None,
    }

Two deliberate choices here:

1. Usage is part of the response contract. If you're evaluating a free tier with a daily token allowance, you need per-request token counts in one place, not scraped from logs. This is the same seam where a budget check would live if you later enforce one (I wrote about making budget exhaustion an API contract separately — the seam is identical).
2. Error codes are normalized at the boundary. Providers disagree on what "rate limited" looks like. Your frontend should never have to know.

The provider seam in practice

Each provider is one small adapter. Here's the shape for any OpenAI-compatible endpoint, which covers a lot of the current ecosystem:

# providers/openai_compatible.py
import httpx

class OpenAICompatibleProvider:
    def __init__(self, name: str, base_url: str, api_key: str, model: str):
        self.name = name
        self._client = httpx.AsyncClient(
            base_url=base_url,
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=60,
        )
        self._model = model

    async def ask(self, question, *, max_output_tokens):
        resp = await self._client.post("/chat/completions", json={
            "model": self._model,
            "messages": [{"role": "user", "content": question}],
            "max_tokens": max_output_tokens,
            "stream": False,
        })
        if resp.status_code == 429:
            raise ProviderRateLimited()
        if resp.status_code in (401, 403):
            raise ProviderAuthError()
        resp.raise_for_status()
        data = resp.json()
        yield AnswerChunk(
            text=data["choices"][0]["message"]["content"],
            usage=Usage(
                input_tokens=data["usage"]["prompt_tokens"],
                output_tokens=data["usage"]["completion_tokens"],
            ),
        )

Provider selection happens once, at startup, from environment config:

# main.py (startup)
app.state.provider = build_provider_from_env()
# PROVIDER=monkeycode
# MONKEYCODE_BASE_URL=...  MONKEYCODE_API_KEY=...  MODEL=deepseek-v4-flash

That last block is the entire point of the exercise. When DeepSeek V4 Flash released, I wanted to test it against my actual prompts without a card, a waitlist, or burning through a paid quota on an experiment. MonkeyCode had same-day access with a 30M free-token daily allowance, so pointing build_provider_from_env at it was the whole "integration." Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reason it fits this article naturally is that a free, same-day tier is exactly the workload a provider seam is for: evaluate first, commit later. If you want to try the same evaluation loop, MonkeyCode's free access is the lowest-friction way I've found to get a real endpoint for it.

Prove it with a cross-layer failure test

The artifact that matters isn't the adapter — it's the test proving the contract holds when the provider misbehaves:

# tests/test_ask_contract.py
import pytest
from httpx import ASGITransport, AsyncClient

@pytest.mark.asyncio
async def test_rate_limit_is_normalized(app, monkeypatch):
    async def boom(self, question, *, max_output_tokens):
        raise ProviderRateLimited()
        yield  # make it an async generator
    monkeypatch.setattr(type(app.state.provider), "ask", boom)

    async with AsyncClient(transport=ASGITransport(app=app), base_url="http://t") as c:
        resp = await c.post("/api/ask", json={"question": "hi"})

    assert resp.status_code == 429
    assert resp.json()["detail"] == "provider_rate_limited"
    assert "Retry-After" in resp.headers

@pytest.mark.asyncio
async def test_usage_is_returned(app):
    async with AsyncClient(transport=ASGITransport(app=app), base_url="http://t") as c:
        resp = await c.post("/api/ask", json={"question": "explain retries"})
    body = resp.json()
    assert body["usage"]["input_tokens"] > 0
    assert body["provider"] == app.state.provider.name

Run the suite once against a fake provider, once against the real free endpoint. If both pass, swapping providers forever after is PROVIDER=x plus a restart.

Decision table: when to add a provider vs. fix the seam

Situation	Right move
New model, same wire protocol	New row in env config; no code
New provider, OpenAI-compatible API	One new env block; adapter is reusable
New provider, non-standard streaming	One new adapter file; route untouched
Rate-limit test fails for new provider	Fix the adapter's error mapping, not the route
You need provider-specific prompt tuning	That's a smell — keep prompts in the route layer or accept per-provider prompt modules, explicitly

Limitations, and who shouldn't do this

• Free tiers are for evaluation, not SLAs. A daily token allowance is generous for testing prompts and latency, but don't put production traffic on it without a fallback provider configured — and test that fallback path, or it won't work when you need it.
• One abstraction can't hide real capability differences. If you depend on tool-calling, structured output, or long-context behavior, those belong in the contract too, or you'll get false confidence from a seam that only covers chat.
• If your app makes exactly one LLM call and will never change providers, this is over-engineering. The seam pays off at the second provider, not before.
• Usage numbers are provider-reported. For budget enforcement, reconcile them against your own accounting before treating them as billing truth.

Reusable checklist

• [ ] One AnswerProvider interface; route imports the interface, never an SDK
• [ ] Provider constructed once at startup from env config
• [ ] Error taxonomy normalized at the route (429/502 with stable detail strings)
• [ ] Token usage in every response body
• [ ] Contract tests run against a fake provider in CI and the real endpoint pre-merge
• [ ] Fallback provider configured and failure-tested before free-tier traffic matters
• [ ] Per-provider prompt hacks explicitly located, or removed

The model release cycle isn't slowing down. The teams that evaluate new models in an afternoon instead of a sprint aren't faster at integrating — they just made sure there was nothing to integrate.

Which layer handoff is least stable in your stack when a provider changes — the streaming format, the error codes, or the token accounting? If you've got a concrete failure state or response code that bit you, I'd like to see it in the comments.