Auditable API Routing Beyond Cost per Token: Caching, Batch, and Regional Risk

#llmgateway #costaccounting #regionalrouting

Short answer: the cheapest gateway is the one that produces the lowest reconciled cost per successful business operation while preserving idempotency, regional policy, and an auditable record of caching, batch execution, retries, and final token usage; a quoted cost per token cannot establish that result.

For a Node.js service comparing OpenAI, Claude, and Gemini-compatible access, compatibility is only the entry constraint. The consequential question is whether the control plane can explain each charge after a retry, a broken stream, a cache decision, or deferred batch completion. Treat the gateway as a metering boundary first and a convenience proxy second.

Price tables age quickly. Ledgers shouldn't.

How should a Node.js API gateway account for caching and batch work?

Begin with the unit of account. Input tokens, cache-eligible tokens, cache-read tokens, output tokens, and batch status are separate quantities; collapsing them into one total_tokens field destroys information needed to replay a charge under a versioned rate card. Store the raw usage evidence or a controlled hash of it, the provider and model identifiers, the request's declared region, the observed processing region when that evidence is available, the business idempotency key, an attempt identifier, and the pricing-rule version. Currency amounts should be derived postings, never replacements for source quantities.

That separation matters because “compatible API” describes a request surface, not a universal accounting contract. Three upstreams can accept broadly similar chat input while reporting cache use, batch completion, and streaming usage through different fields or lifecycle events. The application-facing record can be normalized, but the provider-specific evidence must remain attached to it. Otherwise, a parser change can silently rewrite history.

Use two identities. The business idempotency key says that one user action may create at most one settled accounting effect; the attempt ID admits that the network may have performed more than one upstream attempt. Exactly-once delivery isn't a defensible assumption across processes and networks. An exactly-once ledger effect is achievable when inserts are idempotent, attempts are append-only, and corrections use compensating entries instead of updates.

The following Go fragment captures that narrow invariant. It doesn't calculate money and it doesn't pretend that a missing value is zero.

package meter

import (
    "errors"
    "fmt"
)

type Usage struct {
    InputTokens     *int64
    CacheReadTokens *int64
    OutputTokens    *int64
}

type Quantity struct {
    UncachedInput int64
    CacheRead     int64
    Output        int64
}

func Normalize(u Usage) (Quantity, error) {
    if u.InputTokens == nil || u.CacheReadTokens == nil || u.OutputTokens == nil {
        return Quantity{}, errors.New("usage remains unsettled: a token field is absent")
    }
    if *u.InputTokens < 0 || *u.CacheReadTokens < 0 || *u.OutputTokens < 0 {
        return Quantity{}, errors.New("usage remains unsettled: a token field is negative")
    }
    if *u.CacheReadTokens > *u.InputTokens {
        return Quantity{}, fmt.Errorf(
            "usage remains unsettled: cache read %d exceeds input %d",
            *u.CacheReadTokens,
            *u.InputTokens,
        )
    }

    return Quantity{
        UncachedInput: *u.InputTokens - *u.CacheReadTokens,
        CacheRead:     *u.CacheReadTokens,
        Output:        *u.OutputTokens,
    }, nil
}

Small rule. Large consequence.

Caching needs its own denominator. A useful report distinguishes eligible requests, attempted lookups, reported hits, and accepted cached quantities; a single hit-rate percentage hides prompt instability and tenant-specific prefixes. Batch accounting needs a state machine rather than a Boolean flag: eligible, submitted, accepted, completed, failed validation, cancelled, and returned to the interactive path are economically different outcomes. The settlement record should identify which path completed the business operation without counting the fallback as a second success.

The comparison belongs in a control matrix

There is no durable “cheapest” ranking without a workload trace and a common settlement rule. Construct a replay set that represents the actual distribution of prompt size, output size, cache eligibility, latency class, and region. Run the same logical operations through each candidate, preserve every attempt, and divide the reconciled amount by successful business operations. This exposes retry amplification and incomplete work that a cost-per-token column ignores.

I would reject a comparison that cannot answer one concrete fixture: business key invoice-7841 starts attempt a1, the client receives 23 streamed events and disconnects, then attempt a2 completes. The expected output is one settled business operation, two visible attempts, and no invented token quantity for a1 until authoritative usage evidence resolves it. This is a test scenario, not a production incident. Its value is precision: a dashboard that reports two “requests” or silently records zero usage for the first attempt is unsuitable for financial reconciliation.

Streaming sharpens the boundary. Server-Sent Events use a persistent HTTP connection and the text/event-stream format, with events delivered as blocks of text. Receiving some events proves neither application completion nor final billable usage. A gateway test therefore needs explicit states for started, partially observed, completed, cancelled, and unsettled; browser bytes are operational evidence, while authoritative usage is accounting evidence. Don't merge them.

The architectural options then become comparable without pretending that one always wins:

Shape	Best fit	Limitation to accept	Evidence to demand
Direct provider adapters	Few upstreams and a need to retain provider-specific controls	Policy, retry, and usage normalization are repeated	Per-provider raw evidence tied to one ledger schema
Self-hosted compatibility layer	A team prepared to own deployment and control testing	Upgrades, availability, and reconciliation remain internal duties	Immutable configuration and rate-card change history
Managed compatibility layer	A team seeking less proxy operation	Another contractual, retention, and processing boundary	Exportable attempt and usage records with regional evidence

LiteLLM is public evidence that the self-hosted, multi-provider proxy category exists; its repository describes a gateway and a common OpenAI-format interface across multiple model providers. That observation does not settle operating cost, feature equivalence, or compliance fitness. Those remain workload- and deployment-specific questions.

The catch is ownership. A self-hosted layer is not suitable when the team cannot staff upgrades, incident response, and recurring control tests. A managed layer is not suitable when its available evidence or contractual processing boundary cannot satisfy the organization's audit requirements. Stick with direct adapters when there are only a few upstreams and provider-specific semantics are more valuable than central policy. Conversely, a shared layer earns its place when centralized idempotency, routing controls, and metering remove more duplicated risk than the additional dependency creates.

Regional labels are policy inputs, not compliance conclusions

US and EU routing must be modeled as an enforceable policy with evidence, not as a string passed by a Node.js client. The decision record should identify which data classes may be sent, which processing locations are allowed, what retention terms apply, who may inspect payloads, and what happens when no permitted route is available. Enforcement belongs before outbound transmission. Audit records should preserve the policy version and route decision without copying prompt content into routine metrics.

No gateway selection, by itself, proves compliance with every privacy, financial-services, or record-retention obligation. Applicable limits depend on the organization's role, contracts, data, jurisdictions, and regulators; legal and security reviewers must define them, while engineering turns those decisions into testable controls. I'm not sure a generic regional badge can ever provide enough evidence for a regulated ledger workload, because the answer depends on subprocessors and contractual commitments that a benchmark cannot infer. A current data-flow review and contract set would resolve that uncertainty for a specific deployment.

Cost and residency also interact. A cache may improve unit economics while expanding the set of stored derivatives that need classification, access control, expiry, and deletion. Batch execution may reduce a quoted processing rate while violating an interactive latency objective or crossing a prohibited processing boundary. Neither capability is inherently beneficial. Caching is not suitable when reusable prefixes contain rapidly changing authorization context or when retention policy forbids the required stored representation; deferred batch work is not suitable for a user-blocking decision with a strict response deadline.

The observability design should follow these invariants. Alert on aging unsettled usage, duplicate business keys, attempts per settled operation, reconciliation variance, regional-policy denials, and cache quantities that violate normalization rules. Keep prompt and response bodies out of ordinary labels and logs. Correlation identifiers, policy versions, controlled hashes, and access-governed evidence stores provide a cleaner audit boundary than indiscriminate content capture.

Roll out with shadow settlement, then narrow authority

Start by copying usage evidence into a shadow ledger while the existing integration remains authoritative. Reconcile by upstream, model, region, and pricing-rule version; investigate unknown quantities instead of coercing them to zero. Next, enable one low-risk workload, preserve its business idempotency key across old and new paths, and verify that rollback cannot create a second settled effect.

Exercise the awkward cases before widening traffic: interrupt an SSE consumer after partial output, repeat the same business key, change a rate-card version at a settlement boundary, send cache-eligible and cache-ineligible variants, complete deferred work, cancel deferred work, and deny a disallowed regional route before transmission. The release criterion is explainability: every settled operation can be reconstructed from immutable quantities and a versioned rule, every duplicate attempt is visible, and every unsettled item has an owner and an age.

Only then compare the reconciled unit economics. A gateway may reduce duplicated integration work, but it also concentrates policy and accounting responsibility. The right architecture is conditional: choose the operating model whose failures the team can contain and whose records finance, security, and compliance can independently verify.