Should Compatible API Gateway Cost per Token Decide Node.js Caching and Batch Design?

#llm #architecture #caching

Short answer: no compatible API gateway is demonstrably the cheapest from advertised cost per token alone; compare completed operations under the application's actual caching, batch, streaming, and US/EU placement constraints.

This is a constraint problem before it is a shopping problem. A gateway can normalize an API surface, yet the application still owns request identity, retry policy, output acceptance, and proof that data stayed where policy required. Those details decide whether a nominally cheap route remains cheap under production traffic.

Compatibility is the least interesting checkbox.

What should a Node.js team verify about API caching, batch work, and US/EU routing?

Start with five written invariants. One application operation has a stable identifier across every attempt. A cache entry cannot cross a tenant or data-classification boundary. A batch item has its own terminal state rather than inheriting the state of its submission. A streamed answer is not marked complete until the consumer accepts the end of the stream. A regional label covers content, logs, cache entries, and backups, rather than merely the ingress address.

These invariants are deliberately independent of OpenAI, Claude, Gemini, or any compatible facade. They describe what the application must be able to prove. If a candidate cannot expose enough evidence to test them, don't fill the gap with assumptions — remove it from the shortlist or narrow the workloads routed through it.

Node.js does add one practical concern: many applications process generated text as a stream, while the protocol has its own framing and reconnection behavior. Server-Sent Events are a one-way server-to-client mechanism, and MDN documents named events, event data, IDs, retry intervals, and connection limits. A familiar endpoint shape doesn't establish what happens to billing, cancellation, or usage metadata when the client disconnects halfway through a response. Test that boundary with the same abort and timeout behavior used by the real service.

The same skepticism belongs in regional testing. “EU endpoint” is not a complete placement claim. Draw the data path: request content, gateway logs, provider request, cache key, cached value, metrics labels, support access, and backup copies. Then ask for evidence at every storage boundary. I'm not sure a single region badge can ever answer that design question; a data-flow record and a retention policy can.

Freeze the workload before opening a pricing page

Price comparisons drift because the workload moves underneath them. Freeze a sanitized replay corpus first. It should preserve message shape, prompt-prefix repetition, tool-schema size, expected output class, timeout budget, and region policy without retaining production secrets. Split it into cohorts instead of averaging everything together: interactive streams, ordinary synchronous requests, cache-eligible repeated prefixes, cache-ineligible requests, and offline batch items.

For each operation, retain an immutable request fingerprint and an attempt graph. The fingerprint must be derived from every field that can affect output or cache eligibility: ordered messages, model alias, tool definitions, sampling controls, safety configuration, tenant boundary, and any application prompt version. Volatile timestamps and random request IDs should stay outside the semantic prompt unless they are genuinely part of the requested answer. Otherwise two equivalent operations become unrelated cache candidates.

This can fail quietly. Imagine 10,000 evaluation items with a common policy prefix. A deployment adds the current timestamp to that prefix, the cache-key distribution suddenly becomes almost one key per request, and the application still reports that caching is “enabled.” Nothing has crashed; the economic assumption is simply false. To catch it, compare the number of distinct canonical fingerprints before and after the deployment, replay an unchanged cohort twice, and retain the prompt-version field beside the observed usage. If the second run does not show the reuse expected by the written rate model, stop there and inspect identity construction instead of compensating with a larger batch. At the other extreme, a cache key that omits tenant identity can produce reuse that looks excellent on a dashboard while violating isolation; rerun one repeated prompt under two synthetic tenants and require distinct cache identities even when every other byte matches. One case is waste, while the other is a security event, and an aggregate “hit rate” can conceal both.

Prove it.

Use a small, auditable record format before building a dashboard. The example below creates deterministic request identities and represents retries as separate attempts. It makes no network call, so the same schema can sit beside a Node.js transport without pretending that language-level SDK compatibility proves accounting compatibility.

import hashlib
import json
from dataclasses import dataclass
from decimal import Decimal
from typing import Any


@dataclass(frozen=True)
class Attempt:
    operation_id: str
    attempt_number: int
    request_fingerprint: str
    region_policy: str
    input_tokens: int
    output_tokens: int
    accepted: bool


def fingerprint(payload: dict[str, Any], tenant_id: str) -> str:
    cache_identity = {
        "tenant_id": tenant_id,
        "model": payload["model"],
        "messages": payload["messages"],
        "tools": payload.get("tools", []),
        "temperature": payload.get("temperature"),
    }
    encoded = json.dumps(
        cache_identity,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=True,
    ).encode("utf-8")
    return hashlib.sha256(encoded).hexdigest()


def attempt_cost(
    attempt: Attempt,
    input_rate: Decimal,
    output_rate: Decimal,
) -> Decimal:
    return (
        Decimal(attempt.input_tokens) * input_rate
        + Decimal(attempt.output_tokens) * output_rate
    )

The rates deliberately remain inputs. Published prices change, different models use different units, and cached input or batch execution may have separate rules. Preserve the original rate card and its effective date with every experiment, then calculate both total attempted cost and cost per accepted operation. Don't silently discard timed-out attempts; they are often where a gateway comparison changes direction.

Treat caching, batching, and streaming as different state machines

Caching has an identity problem. Batch has a lifecycle problem. Streaming has a partial-commit problem. Putting all three under a single “optimization” column hides distinct failure modes and makes the cheapest-looking candidate impossible to audit.

For caching, run paired cold and warm cohorts. Change one field at a time and observe whether reuse changes as expected. A useful test matrix varies the tenant, prompt version, tool schema, model alias, whitespace policy, and sampling controls. Record hits using evidence supplied by the system under test, then reconcile that evidence against measured input usage; latency alone is a weak proxy because queues and model execution vary.

For batch work, assign an operation ID to every item as well as an ID to the submission. Force mixed outcomes in a synthetic corpus: accepted items, intentionally invalid items, duplicate item IDs, and a client interruption after submission. The test question isn't merely “did the batch finish?” It is “can each item be retried without charging or accepting a second valid completion?” A local worker queue does not, by itself, prove that an upstream batch execution mode or its associated rate was used.

Streaming needs a commit rule. With an event stream, bytes may have reached the client even when the application never accepted a complete answer. Capture time to first event, time to accepted completion, disconnect point, final usage metadata when available, and whether a retry reused the same operation identity. Test an abort before the first event, an abort after several events, and a clean completion. A client timeout should create another attempt under the original operation, not a brand-new business operation that conceals retry amplification.

Here is the uncomfortable part: exact cross-provider equivalence may be unobtainable for features whose native semantics differ. Your mileage may vary with tools, structured output, safety controls, and usage reporting. Resolve that uncertainty by testing the subset the application actually sends, pinning the model alias and gateway configuration, and rejecting unknown-field behavior that cannot be observed. A broad compatibility label is not evidence that every field survives translation.

Compare control boundaries after the replay

Only now is a comparison table useful. It should compare ownership, evidence, and failure modes, not logos.

Control boundary	Suitable when	Failure mode to test	Cost the token rate omits
Direct provider adapters	The provider set is small and native semantics matter	Retry and usage fields diverge across adapters	Application maintenance and separate reconciliation
Self-hosted compatible gateway	The team can own proxy operations and policy	Configuration drift, retry fan-out, or telemetry loss	Capacity, upgrades, on-call work, and state storage
Managed compatible gateway	Delegated operations are more valuable than infrastructure control	Route resolution or usage export lacks required evidence	Contract review and control-plane dependency
Internal policy router over adapters	Placement or governance rules are unusually specific	Policy aliases drift from deployed model mappings	A second platform to build, secure, and operate

None of these is the cheapest by definition. Direct adapters can be the clearer choice for one or two providers, particularly when native features matter more than a common facade. A self-hosted gateway is not suitable when nobody owns upgrades, capacity, security patches, and telemetry. Stick with a managed boundary when those operational duties would otherwise be neglected, but only if its usage exports and regional evidence meet the written constraints. A managed boundary is a poor fit when infrastructure-level audit evidence is mandatory and unavailable. An internal router deserves its engineering cost only when custom policy is important enough to justify another production control plane.

LiteLLM is an open-source example showing that a self-hosted LLM gateway is an available control boundary. Its existence does not establish fitness for a particular workload, and a repository feature list cannot replace the replay, failure injection, regional review, or operational staffing decision described above.

Rank candidates with a scorecard fixed before results are revealed. Include invoiced or rate-card cost per accepted operation, retry amplification, cache reuse for the frozen corpus, item-level batch completion, streaming acceptance rate, latency distribution, regional evidence, export completeness, and engineering hours. Keep raw totals beside ratios. A 2% error rate can be made to disappear in an average while still creating duplicate work, retry traffic, and ugly tail latency.

Roll out the chosen boundary without betting the data layer

Begin by emitting operation IDs and request fingerprints without changing traffic. Replay sanitized traces offline, then canary one workload cohort in one region. Reconcile client-observed operations, gateway usage records, and the applicable rate card daily. Promote streaming, batch, and cache-eligible traffic separately because their rollback signals differ.

Version alias mappings and cache-key rules. Set explicit stop conditions for duplicate accepted operations, unexpected cross-region records, cache isolation violations, missing usage evidence, and cost per accepted operation outside the experiment budget. Keep the previous route available until delayed batch results and reconciliation records have arrived.

The catch is simple: a compatible gateway centralizes policy by creating another stateful control boundary. For a small, regionally uncomplicated workload tied to native provider behavior, that boundary may add more work than it removes. For a mixed workload, it can be justified, but the decision should rest on repeatable evidence about completed operations — not a token-price cell, a compatibility badge, or a smooth demo.