Short answer: The cheapest OpenAI-, Claude-, or Gemini-compatible API gateway is the one with the lowest verified cost per accepted result for your traffic, after caching, batch work, retries, quality, latency, and US or EU requirements are included.
A small input-token quote cannot settle that choice. Public price sheets don't reveal your cache-hit distribution, failed-request policy, prompt growth, or how much work your eval gate rejects. I'm not sure a static comparison can stay accurate for long; a current rate card plus a representative usage export would resolve the uncertainty. Build a replayable spend harness instead.
What should a compatible API gateway cost test cover for Node.js, caching, batch, US, and EU?
Treat the gateway as one stage in a data flow. A Node.js application emits a request with a workload label and an idempotency key. The gateway selects a model endpoint, while the application records timestamps and usage metadata. A separate ledger joins those records to the invoice and the eval result. Interactive responses return to the caller immediately, often as a stream; batch candidates enter a deferred queue. The final table has one row per logical task, even if retries created several billable attempts.
Start with constraints before price. Does the compatibility surface preserve the request fields, response shape, streaming events, tool-call arguments, usage counters, and error semantics your application depends on? Does the selected region satisfy the team's data-handling policy? Can a batch result still be traced to the originating eval case? A low quote is irrelevant when an adapter drops a required field or when a valid result arrives outside the product's latency budget.
Use a comparison frame that keeps unlike concerns apart:
| Decision axis | Evidence to collect | Disqualifying boundary |
|---|---|---|
| Compatibility | Contract tests for requests, streams, tools, and usage | Required semantics cannot be preserved |
| Quality | Accepted results from a frozen eval set | Acceptance falls below the product threshold |
| Cost | Reconciled spend per accepted task | Invoice and ledger cannot be reconciled |
| Interactive path | First-useful-token and completion latency | Tail latency misses the user-facing budget |
| Batch path | Queue, cancellation, and result traceability | Jobs cannot be replayed safely |
| Region | Documented processing and log locations | Deployment conflicts with data policy |
Keep the units explicit. Input, cached input, and output are token counts. Rates are money per token in the same currency. Batch and cache adjustments are coefficients supplied from current, attributable terms rather than constants copied into application code. The ledger should retain the rate-card version so a later change doesn't rewrite history.
Build the ledger before debating vendors
This Python example is provider-neutral. It performs no network request and assumes an ingestion layer has normalized usage metadata and current contract rates. That makes it useful beside a Node.js service without pretending every compatible API reports billing details identically. Save it as cost_ledger.py; the invocation is runnable with values from your own export.
from __future__ import annotations
import argparse
from dataclasses import dataclass
from decimal import Decimal
@dataclass(frozen=True)
class Usage:
input_tokens: int
cached_input_tokens: int
output_tokens: int
@dataclass(frozen=True)
class Rates:
input_per_token: Decimal
cached_input_per_token: Decimal
output_per_token: Decimal
batch_multiplier: Decimal
def task_cost(usage: Usage, rates: Rates, is_batch: bool) -> Decimal:
if usage.cached_input_tokens > usage.input_tokens:
raise ValueError("cached input cannot exceed total input")
fresh_input = usage.input_tokens - usage.cached_input_tokens
subtotal = (
Decimal(fresh_input) * rates.input_per_token
+ Decimal(usage.cached_input_tokens) * rates.cached_input_per_token
+ Decimal(usage.output_tokens) * rates.output_per_token
)
multiplier = rates.batch_multiplier if is_batch else Decimal("1")
return subtotal * multiplier
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--input-tokens", type=int, required=True)
parser.add_argument("--cached-input-tokens", type=int, required=True)
parser.add_argument("--output-tokens", type=int, required=True)
parser.add_argument("--input-rate", type=Decimal, required=True)
parser.add_argument("--cached-input-rate", type=Decimal, required=True)
parser.add_argument("--output-rate", type=Decimal, required=True)
parser.add_argument("--batch-multiplier", type=Decimal, default=Decimal("1"))
parser.add_argument("--batch", action="store_true")
args = parser.parse_args()
usage = Usage(
input_tokens=args.input_tokens,
cached_input_tokens=args.cached_input_tokens,
output_tokens=args.output_tokens,
)
rates = Rates(
input_per_token=args.input_rate,
cached_input_per_token=args.cached_input_rate,
output_per_token=args.output_rate,
batch_multiplier=args.batch_multiplier,
)
print(task_cost(usage, rates, args.batch))
if __name__ == "__main__":
main()
Run it with token counts and per-token rates from one reconciled task. The zero rates below are local input placeholders, not market claims.
python cost_ledger.py --input-tokens 1 --cached-input-tokens 0 --output-tokens 1 --input-rate 0 --cached-input-rate 0 --output-rate 0
In a production harness, fail validation when required usage fields are absent instead of quietly treating them as zero. Store raw usage beside normalized usage, because normalization logic may change as compatibility surfaces evolve.
Short code. Long audit trail.
Aggregate spend across every attempt, then divide by accepted eval results for the workload. Don't divide by raw request count. A gateway that retries aggressively can appear inexpensive in a per-request dashboard while charging multiple upstream attempts for one logical task. Record logical task ID and attempt ID separately so the ledger can expose that multiplication rather than smoothing it away.
Cache, batch, and streaming change different parts of the system
Caching is a workload property before it is a billing feature. Stable system instructions and repeated context may create reusable prefixes, while retrieved passages, tool output, timestamps, and personalized state may destroy reuse. Record the cache-eligible token count, the reported cache hit, and the charged class independently. Then perturb prompt order in an eval: if a harmless serialization change collapses reuse, the apparent saving was fragile.
Batching solves a scheduling problem. It fits offline evaluation, enrichment, indexing support, and other work with a relaxed completion deadline. It is not suitable when a person is waiting on the next token, when a tool loop has a tight control deadline, or when the job cannot be made idempotent. Stick with interactive requests for those paths, even if deferred processing has a better rate in a current contract. The catch is operational: a batch queue needs durable task identifiers, cancellation rules, result reconciliation, and a policy for replaying only failed logical items.
Streaming is different again. Server-Sent Events are a one-way server-to-client mechanism, and MDN documents the browser EventSource interface and its event-stream format. A compatibility test should parse the stream incrementally, preserve event boundaries, measure time to first useful token, and reconstruct the final response for the same semantic eval used on non-streaming calls. Fast first bytes do not guarantee a correct final tool call.
Don't merge these three columns into a single score.
A candidate can fit cached interactive chat and still be unsuitable for deferred bulk work, or the reverse. Route by workload class only after each class has enough samples to expose tail latency and rejected outputs.
Compare accepted-result cost, not a price-sheet headline
Use a replay set drawn from production-shaped prompts with sensitive content removed or synthesized under the same structural constraints. Freeze the prompt template and model parameters for the comparison. For every candidate and region, capture logical task ID, attempt ID, input classes, output tokens, cache status, batch status, first-token latency, completion latency, eval verdict, and invoiced amount. Reconcile totals at the billing boundary before trusting row-level estimates.
The primary metric is total reconciled spend divided by accepted results. Pair it with acceptance rate and latency percentiles so optimization cannot buy a lower number by returning shorter, worse, or late answers. For RAG, keep retrieval inputs fixed while testing the runtime layer; for agents, replay tool results so an external system does not add noise. This is notebook-to-prod discipline: the notebook explores distributions, while the checked-in eval fixture and ledger schema make the result repeatable in CI.
Token cost can still mislead. One model may need a longer prompt to satisfy the same rubric. Another may emit verbose answers that pass but consume more of a session's context window. A third may produce cheap individual turns yet trigger more tool-loop iterations. Track cost at the user-visible task boundary and, for multi-turn features, at the session boundary too.
Regional comparisons need the same rigor. Label where inference is requested, where gateway logs are stored, and where billing records are processed; a region selector by itself does not establish a complete data-residency story. Get the contractual and architectural details required by your policy, then run the same replay from representative clients. Your mileage may vary because network paths and workload mixes differ. Measure them.
A self-hosted gateway can make this experiment easier when a team needs control over routing and telemetry. LiteLLM is one open-source example of a proxy with a common interface across multiple model providers. Self-hosting is not automatically cheaper: the comparison must include compute, storage, upgrades, on-call work, and the failure domain introduced by another hop. A managed gateway trades some control for less platform ownership. Neither category wins without the team's traffic and operating constraints.
Put the harness on an operational clock
Before rollout, pin the compatibility tests, rate-card version, and eval dataset; verify interactive streaming and deferred jobs separately; and confirm that logs avoid prompt content unless retention is explicitly required. During a canary, compare ledger totals with invoices, alert on missing usage metadata, and watch accepted-result cost beside latency and quality. After rollout, rerun the fixed replay when a model, prompt, routing rule, region, or contract changes.
Keep a rollback path that changes routing without changing application code.
Keep it boring. The cheapest gateway can change as prompts, cache behavior, batch share, quality thresholds, and rates move, so the durable asset is the measurement loop, not a winner's name.
Top comments (0)