Portability First: Token Cost, Caching, and Batch Before You Commit to One LLM API Gateway

#llm #api #architecture #python

Pick the LLM API gateway you can leave. For a private knowledge base inside a regulated fintech shop, the cheap per-token cost you compare on day one is a tiebreaker; what decides the bill two quarters later is whether moving the model behind your retrieval service is a config change or a rewrite.

That ordering isn't a preference. It falls out of how the three levers that actually move spend on a retrieval workload — token accounting, prompt caching, and batch — are specified, because each of them is defined by the provider, not by the standard, and every one of them leaks into your code if you let it.

Start from the constraint, not from the gateway

A private knowledge base in financial services has a shape that has nothing to do with which vendor is cheapest this month. The corpus is internal and boring: policy documents, procedure manuals, product terms, a decade of support macros. Answers have to cite the source document, because an answer without a citation is an unreviewable claim and compliance will treat it as one. The list of providers legal has cleared is shorter than the list on the market, it differs between the EU and US entities, and it changes without asking engineering when it's convenient.

So the binding constraint isn't price. It's the time between "this provider is no longer approved for this region" and "production is answering questions again on an approved one."

Write that down as a design rule and most of the gateway comparison answers itself: the retrieval service, the prompt assembly, and the evaluation harness must not import a provider SDK. One HTTP boundary, one request shape, provider and model chosen as data — a row in a config table, not a branch in code. Everything downstream of that boundary is allowed to be provider-specific, and nothing upstream of it is. If a gateway can't sit on that boundary without leaking its own vocabulary into your callers, it isn't a gateway; it's another SDK with a proxy in front.

The cheap option that violates the rule is not cheap. It's a deferred migration with interest.

How do I compare token cost, caching, and batch across gateways without a rewrite?

Start with token accounting, because it's the one people get wrong quietly. A local token estimate is a planning tool — it tells you whether a retrieved context is about to blow past a context window, and it's the right place to reject a request before you pay for it. It is not billing truth. Tokenizer vocabularies differ between model families, so the same 40 KB of policy text lands on different token counts depending on who answers, and a budget alarm calibrated against one vocabulary silently drifts the day you route to another. Read the usage object off the response and treat that as the ledger. Local estimates go in the admission-control path; measured usage goes in the invoice reconciliation path. Keep them in separate code, or you will eventually reconcile an estimate against itself and conclude everything is fine.

Caching is where the money is on a knowledge-base workload, and it's also where portability quietly dies. The three model families most fintech shortlists carry — OpenAI's GPT, Anthropic's Claude, Google's Gemini — all offer prompt caching, and all three define it differently: an automatic prefix cache, explicit cache breakpoints with a short TTL and a premium on the write, and an explicit cached-content object that you create, address by id, and pay to keep alive. The portability question isn't which is best. It's what a given gateway does with a cache directive it doesn't understand. Silent pass-through is the dangerous answer, because your responses stay correct while your bill goes back to full price, and nothing in your dashboards says so. Call that failure mode what it is — a silent cache pass-through — and put a metric on it: cached input tokens as a share of prompt tokens, per provider, per day. If that ratio falls off a cliff after a routing change, you found it in hours instead of at month end.

Batch is the easy one. Asynchronous job in, completion window measured in hours rather than seconds, discounted token price for tolerating the wait — that's the shape across the major providers, with a 24-hour target window being the common commitment. It's the right tool for re-embedding the corpus after a chunking change and for nightly evaluation runs over a golden set. It is the wrong tool for the interactive answer path, and no amount of cost pressure makes a 24-hour window acceptable to someone waiting for an answer about a chargeback.

The code boundary that makes all of this comparable is unremarkable, which is the point:

import json
import os

import requests

GATEWAY = os.environ["LLM_GATEWAY_URL"]        # e.g. https://gateway.internal/v1
API_KEY = os.environ["LLM_GATEWAY_KEY"]

# Prices are configuration, not code: one row per (route, model), USD per million tokens.
with open("prices.json", encoding="utf-8") as fh:
    PRICES = json.load(fh)


def answer(question: str, context: str, model: str) -> dict:
    """One HTTP call, one request shape. Provider selection lives in `model`."""
    body = {
        "model": model,
        "messages": [
            {"role": "system", "content": context},   # stable prefix: the cacheable part
            {"role": "user", "content": question},
        ],
    }
    response = requests.post(
        f"{GATEWAY}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json=body,
        timeout=60,
    )
    response.raise_for_status()
    payload = response.json()

    usage = payload.get("usage") or {}
    prompt_tokens = usage.get("prompt_tokens", 0)
    cached = (usage.get("prompt_tokens_details") or {}).get("cached_tokens", 0)
    price = PRICES[model]
    cost = (
        (prompt_tokens - cached) * price["input_per_mtok"]
        + cached * price["cached_input_per_mtok"]
        + usage.get("completion_tokens", 0) * price["output_per_mtok"]
    ) / 1_000_000

    return {
        "text": payload["choices"][0]["message"]["content"],
        "model": model,
        "prompt_tokens": prompt_tokens,
        "cached_tokens": cached,
        "cost_usd": cost,
    }

Two things in there are load-bearing. The price table is data, so a provider move is a pull request against a JSON file rather than a deploy of the answering service. And cached tokens are subtracted before pricing, so the cache hit ratio shows up as a number you can alarm on rather than as a vague sense that things got more expensive.

What breaks after the swap, rather than during it

Cutover is the part everyone tests. The interesting failures arrive a week later.

Streaming accounting is the first one. Answer UIs stream, so the streaming path carries most of your traffic and, if you're careless, none of your cost data — the usage totals arrive in a final event that a naive parser drops on the floor. Server-sent events are a plain-text framing over HTTP, and the browser's native EventSource only issues GET requests, which is why a POST-bodied model call ends up parsed by your own reader on the server side rather than by the platform. Ask for usage on the stream explicitly, and keep the frame:

def stream_answer(body: dict):
    """Yields ('token', str) and finally ('usage', dict). Cost accounting survives streaming."""
    body = {**body, "stream": True, "stream_options": {"include_usage": True}}
    with requests.post(
        f"{GATEWAY}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json=body,
        stream=True,
        timeout=120,
    ) as response:
        response.raise_for_status()
        for line in response.iter_lines(decode_unicode=True):
            if not line or not line.startswith("data:"):
                continue                       # comments and blank keep-alive frames
            chunk = line[len("data:"):].strip()
            if chunk == "[DONE]":
                break
            event = json.loads(chunk)
            if event.get("usage"):
                yield "usage", event["usage"]
                continue
            for choice in event.get("choices") or []:
                delta = (choice.get("delta") or {}).get("content")
                if delta:
                    yield "token", delta

Cache hit collapse is the second. Any edit to the stable prefix — a reworded system instruction, a reordered retrieval block, a timestamp someone helpfully added — invalidates the cached prefix, and on a knowledge-base workload the prefix is most of the prompt. Treat the prefix as a versioned artifact with its own change review, not as a string a product manager can tune on a Friday.

Rate limiting is the third, and it's the one that makes retries dangerous. Providers meter differently — requests per minute, tokens per minute, concurrent requests — so a retry policy tuned against a token-per-minute ceiling behaves badly against a concurrency ceiling, and a naive exponential backoff without jitter turns one throttled batch into a synchronized stampede. Cap concurrency at the client, honor the retry hint the response gives you, and add jitter.

Then there's answer quality, which nobody wants to own. A model change moves citation adherence and refusal behavior even when benchmark scores look flat, and on a regulated corpus an uncited answer is a defect regardless of how good the prose is. Keep a golden set of real questions with known source documents, run it as a batch job before every routing change, and gate the change on citation accuracy rather than on vibes.

A portability scorecard you can fill in during an evaluation

Run the candidates through the same seven rows. Fill it in from documentation and a two-day spike, not from a comparison page.

What you're buying	The portable form	Where it turns provider-specific
Request shape	One HTTP endpoint, OpenAI-compatible JSON body	Tool-calling and structured-output dialects diverge first
Token accounting	`usage` on every response, including streamed ones	Tokenizer vocabularies differ, so estimates aren't comparable
Prompt caching	Cached input reported as a separate token count	Automatic prefix vs explicit breakpoints vs cached-content objects
Batch	Async job with a stated completion window	Job payload formats and per-file limits
Streaming	SSE frames with a terminal usage event	Whether the gateway forwards the final usage frame at all
Data residency	Region-pinned endpoint named in the contract	Which regions each provider covers, and for which capability
Rate limits	Documented ceiling plus a retry hint on the response	RPM vs TPM vs concurrency, and per-model quotas

The row that will decide the outcome for a fintech knowledge base is residency, and it's the one that resists gateway abstraction hardest. A gateway can normalize a request body; it can't normalize a data processing agreement. If your EU entity has to keep retrieval traffic in-region, the honest test is a signed contract naming the region and an endpoint that provably answers from it, not a checkbox in a console.

Rolling this out without a big-bang migration

Do it in four moves, none of which need a maintenance window. Put a provider and model tag on every request log and every cost record first, so you have a baseline denominated in cost per answered question rather than in cost per million tokens. Then shadow: send a sampled copy of live questions to the candidate route, compare against the golden set offline, and look at cached-token ratios as well as answer quality. Then move a small share of real traffic behind a config flag — a row in a table, changed without a deploy, revertible in seconds. Then, and only then, negotiate.

The catch is real, and it's the reason this design isn't free. Holding a portable boundary means you're consistently one release behind whatever a single provider ships this quarter, and the newest capability — a specific reasoning control, a native tool protocol, a bespoke document format — is exactly what doesn't survive normalization. If your product's differentiation is one model's particular behavior, stick with a direct integration and budget for the migration when it comes; the abstraction will cost you more than it saves. I'm also not certain that the batch discount survives the next round of provider repricing, so treat batch savings as a variable input to the decision rather than a fixed one.

For a private knowledge base, though, the differentiator is the corpus and the retrieval, and both belong to you. Keep them that way.

DEV Community