Selecting a Compatible API Gateway: Token Cost, Batch Work, and Data Residency

#python #llm #api

Short answer: the cheapest compatible API gateway is the one with the lowest cost per accepted result on your own traffic after cache misses, batch eligibility, retries, and regional constraints are counted. A token price alone cannot answer the question. Build a small Python replay ledger, require every candidate to emit enough usage data for the ledger, and reject any comparison that silently changes the model, quality bar, or US/EU data path.

That decision rule matters more than a pricing-page winner. It also survives price changes.

How should a team compare compatible API gateway token caching and batch costs?

Start with one data flow. A Python application creates a model request, an adapter translates it into the gateway's compatible request shape, and the response is normalized into a local observation. The observation contains token counts, latency, outcome, execution mode, and region. An eval then marks the result accepted or rejected. Only after that mark exists does the cost calculation run.

This ordering prevents a common notebook-to-production mistake: optimizing the numerator while ignoring the denominator. If a cheaper route produces more answers that fail the same eval, its apparent saving becomes retry traffic or manual review. The useful comparison is therefore:

effective cost = total charged cost / accepted results

Keep the eval fixed across candidates. Keep the prompt, temperature, output limit, and test records fixed too. Provider-specific model names need not match, but the task and acceptance threshold must. If the evaluation set changes halfway through a run, throw away the comparison; it no longer answers which route handles the same work more efficiently.

"Compatible" also needs a testable definition. Request compatibility, streaming compatibility, error semantics, usage accounting, and cancellation are separate behaviors. A client may accept the same JSON fields while exposing cached tokens differently, omitting them, or reporting usage only at the end of a stream. Normalize those differences at the adapter boundary rather than scattering conditional logic through the application.

This is the contract I want before discussing price:

Evidence	What the harness records	Why it changes the decision
Fresh input	Non-cached prompt tokens	Establishes the normal input charge
Cached input	Cached prompt tokens, or `null` when unavailable	Separates a verified hit from an assumed hit
Output	Generated tokens	Prevents short prompts with long answers from looking artificially cheap
Batch mode	Explicit batch or interactive label	Stops an offline rate from being applied to live traffic
Quality	Eval acceptance and reason	Makes model or routing changes visible
Region	Intended processing region plus verification result	Keeps an invalid data path out of the price race
Transport	Status, attempts, and elapsed time	Exposes retry amplification and tail latency

A missing number is not zero. Treat it as unknown.

Build the ledger before touching production traffic

The runnable part can stay deliberately boring. Export one normalized JSON object per completed request from each adapter, then let a local Python program calculate comparable totals. This example performs no network calls and assumes no vendor-specific route. Its input schema belongs to the test harness, so an adapter can populate it from whichever supported response and billing records a candidate provides.

One JSONL row looks like this:

{"request_id":"eval-001","region":"eu","mode":"interactive","prompt_tokens":420,"cached_prompt_tokens":300,"output_tokens":85,"input_usd_per_million":2.0,"cached_input_usd_per_million":0.5,"output_usd_per_million":8.0,"attempts":1,"latency_ms":930,"accepted":true}

The values above are illustrative harness data, not a quote for any service. Replace every rate with the applicable published rate for the exact model, region, and execution mode being tested.

from __future__ import annotations

import json
import statistics
from collections import defaultdict
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class Observation:
    request_id: str
    region: str
    mode: str
    prompt_tokens: int
    cached_prompt_tokens: int | None
    output_tokens: int
    input_usd_per_million: float
    cached_input_usd_per_million: float
    output_usd_per_million: float
    attempts: int
    latency_ms: float
    accepted: bool

    @classmethod
    def from_json(cls, line: str) -> "Observation":
        return cls(**json.loads(line))

    def charged_usd(self) -> float | None:
        if self.cached_prompt_tokens is None:
            return None
        if not 0 <= self.cached_prompt_tokens <= self.prompt_tokens:
            raise ValueError(f"invalid cached token count: {self.request_id}")

        fresh = self.prompt_tokens - self.cached_prompt_tokens
        return (
            fresh * self.input_usd_per_million
            + self.cached_prompt_tokens * self.cached_input_usd_per_million
            + self.output_tokens * self.output_usd_per_million
        ) / 1_000_000


def percentile(values: list[float], fraction: float) -> float:
    ordered = sorted(values)
    index = min(len(ordered) - 1, int(len(ordered) * fraction))
    return ordered[index]


def summarize(rows: list[Observation]) -> dict[str, float | int | None]:
    known_costs = [cost for row in rows if (cost := row.charged_usd()) is not None]
    accepted = sum(row.accepted for row in rows)
    prompt = sum(row.prompt_tokens for row in rows)
    cached = sum(row.cached_prompt_tokens or 0 for row in rows)

    return {
        "requests": len(rows),
        "accepted": accepted,
        "acceptance_rate": accepted / len(rows),
        "known_cost_usd": sum(known_costs),
        "cost_per_accepted_usd": (
            sum(known_costs) / accepted
            if accepted and len(known_costs) == len(rows)
            else None
        ),
        "cache_token_rate": cached / prompt if prompt else 0.0,
        "mean_attempts": statistics.fmean(row.attempts for row in rows),
        "p95_latency_ms": percentile([row.latency_ms for row in rows], 0.95),
    }


rows = [
    Observation.from_json(line)
    for line in Path("gateway-observations.jsonl").read_text().splitlines()
    if line.strip()
]
if not rows:
    raise SystemExit("gateway-observations.jsonl contains no observations")

groups: dict[tuple[str, str], list[Observation]] = defaultdict(list)
for row in rows:
    groups[(row.region, row.mode)].append(row)

for key, group in sorted(groups.items()):
    print(json.dumps({"region": key[0], "mode": key[1], **summarize(group)}))

Notice the None path. When cached-token accounting is unavailable, the program refuses to manufacture a cost per accepted result from partial evidence. The same principle should apply to pricing metadata: snapshot the rate card used for each run, record its effective date outside the request log, and rerun the calculation when rates change. Don't edit old observations to make a new tariff look historical.

The notebook phase can use a small, scrubbed sample to verify schemas and eval behavior. The production candidate run needs representative prompt lengths, output lengths, languages, tools, and retrieval contexts. Split results by mode and region before aggregating them. A blended global average can hide the exact segment that decides the architecture.

Cache and batch discounts solve different shapes of work

Prompt caching rewards repeated leading content only when the selected API and model support that behavior. Design the experiment as paired traffic: send an identical static prefix with varied user content, compare cold and repeated observations, and verify the cached-token field rather than inferring a hit from lower latency. Then deliberately mutate an early part of the prefix. If the accounting does not distinguish the cases, the harness has not proved a usable cache contract.

Prompt layout is part of the cost model. Stable system instructions, tool definitions, and reusable examples should precede volatile retrieval results when the API's documented caching rules make prefix reuse relevant. Timestamps, request IDs, and user-specific material near the front can reduce reuse. The exact eligible length, retention behavior, and discount are provider facts, so copy them from current primary documentation into the test metadata instead of treating them as universal constants.

Batch processing makes a different trade. It fits eval sweeps, document enrichment, classification backlogs, and other work whose deadline permits queued completion. It is not suitable when a person is waiting for the first token, when jobs must finish in strict arrival order, or when the provider's completion window exceeds the product's service objective. Keep interactive and batch observations in separate cohorts; applying a batch price to an interactive workload is spreadsheet fiction.

There is another catch. A multi-provider gateway may offer one client shape while upstream capabilities still differ. Cache controls, batch lifecycle, usage fields, tool calling, and streaming events can require adapter logic and explicit capability checks. Stick with direct provider integrations when the application depends on a provider-specific feature that cannot be represented cleanly, or when an extra routing layer would complicate a tightly controlled latency path. Choose a self-hosted gateway when control of routing and telemetry justifies owning upgrades, scaling, policy, and on-call work. Neither choice wins automatically.

Can streaming, errors, and US/EU controls change the cheapest result?

Yes. Streaming affects perceived latency and failure handling even when it does not change the token rate. Server-Sent Events use the text/event-stream media type, and clients process a stream of named or unnamed events. A compatibility test should parse events incrementally, preserve UTF-8 content across chunk boundaries, handle cancellation, and verify how final usage arrives. Time to first token and time to completed answer belong beside total latency; a single average hides too much.

Retries can quietly multiply spend. Define retry policy by failure class, cap attempts, respect documented server guidance, and attach every attempt to one logical request ID. Never retry malformed input or an eval rejection as if it were transient transport trouble. For rate limiting, the HTTP specification defines status 429, and Retry-After communicates how long a client should wait. A gateway comparison should confirm the behavior it actually exposes and record attempts, because a nominally low rate paired with frequent duplicate work may lose on effective cost.

US and EU are constraints, not decorative dropdown values. Write down which data must remain where, which systems process prompts and outputs, where logs and backups live, and whether support access crosses the boundary. Then require evidence for the complete path: gateway, upstream model, telemetry, eval storage, and failure queues. I'm not sure a generic "EU endpoint" claim proves that whole chain; architecture documentation, contractual terms, and an audit trail would resolve the uncertainty for a specific deployment.

This is where the cheapest eligible option can change. Exclude a route that cannot meet the required residency or retention policy before comparing cost. Do not assign a dollar value to noncompliance and let the spreadsheet trade it away.

From notebook evidence to an operational decision

Run the harness in stages. First validate adapters with synthetic, non-sensitive prompts. Next replay a scrubbed evaluation set at low concurrency, using stable request IDs and a fixed grader. Then exercise warm and cold cache cases, interactive and batch cohorts, stream cancellation, rate limits, and the required regions. Before moving traffic, compare acceptance rate, cost per accepted result, p95 latency, retry amplification, cache observability, and missing usage records. A result with incomplete accounting remains incomplete, even if its visible subtotal is attractive.

In production, keep the normalized ledger as an observability surface rather than a one-off purchasing script. Alert on shifts in acceptance rate, missing usage, attempts per logical request, output length, and regional routing. Budget limits should act on measured spend and request volume, while an eval canary checks that a routing or model change has not bought lower token charges by reducing answer quality. This is prompt-cost work tied to software quality, where it belongs.

Revisit the decision when traffic shape, model mix, rate cards, cache behavior, or residency requirements change. Your mileage may vary because prompt repetition and batch eligibility vary sharply by application. The method remains stable: compare only eligible paths, price complete attempts, and divide by accepted work.