Short answer: don't choose an LLM API for a SaaS app from a public token-price table alone; replay a representative eval set through direct OpenAI, direct Claude API, and an OpenRouter-style routing path, then choose the smallest integration whose measured quality, billed usage, retry behavior, and recovery policy meet your release constraint.
The cheapest path is workload-dependent. A low input rate can hide engineering overhead, while a chatty agent can turn one user action into several model calls. A fallback can rescue an answer and still charge for an earlier attempt. A unified key can reduce credential sprawl, but it also changes the control plane you depend on. Those effects belong in the same experiment.
This is the notebook-to-prod move that matters: keep the comparison as a replayable ledger tied to an eval suite, not as a screenshot of three price pages. Prices and model catalogs can change after publication; the ledger remains useful because fresh quotes and fresh run records can be inserted without changing the decision method.
How should a Node.js SaaS compare token cost and fallback behavior?
Start with user-visible tasks, not providers. Sample the production shapes that drive spend: a short classification, a retrieval-augmented answer with a long context, a tool-using turn, and a regeneration after invalid structured output. Remove sensitive data, freeze the inputs, and attach an acceptance check to every case. If two paths don't clear the same quality bar, their token costs aren't comparable.
For each attempted call, record the request ID generated by your application, case ID, route, model label, input and output token counts reported by the API, elapsed time, outcome class, and whether another attempt followed. Keep quoted rates in a separate, dated table. That separation is deliberate — usage is an observation, while a rate is configuration that can become stale.
Use one normalized calculation:
from dataclasses import dataclass
from decimal import Decimal
@dataclass(frozen=True)
class Attempt:
case_id: str
route: str
input_tokens: int
output_tokens: int
accepted: bool
retried: bool
@dataclass(frozen=True)
class Rate:
input_per_million: Decimal
output_per_million: Decimal
def quoted_cost(attempt: Attempt, rate: Rate) -> Decimal:
million = Decimal(1_000_000)
input_cost = Decimal(attempt.input_tokens) * rate.input_per_million / million
output_cost = Decimal(attempt.output_tokens) * rate.output_per_million / million
return input_cost + output_cost
def summarize(attempts: list[Attempt], rates: dict[str, Rate]) -> dict[str, Decimal]:
attempted = sum(quoted_cost(item, rates[item.route]) for item in attempts)
accepted_ids = {item.case_id for item in attempts if item.accepted}
return {
"attempted_cost": attempted,
"accepted_cases": Decimal(len(accepted_ids)),
"retry_count": Decimal(sum(item.retried for item in attempts)),
}
The example refuses to fabricate rates. Populate Rate from the current official quote for each exact model and route at evaluation time, retain the retrieval date, and archive the raw usage response. Don't substitute a similarly named model. Also verify what each quote counts before comparing totals; a label such as "input" in your ledger needs a documented mapping to the billed fields returned by that route.
Then report cost per accepted case, not only cost per call. Suppose case rag_017 makes two attempts: the first consumes 8,200 input tokens and fails the application's schema check, while the second consumes 8,350 and passes. The ledger must retain both attempts. Dropping the rejected call makes fallback look free and rewards the path that needed extra work. This is a constructed test case, not a benchmark, but it exposes the accounting error quickly.
Quality comes first.
A practical acceptance function might require valid JSON, grounded citations found in the retrieved context, and a task-specific score over a frozen threshold. Keep that evaluator identical across routes. I'm not sure any single automated judge is sufficient for every product domain; disagreement sampling and periodic human review are what would resolve that uncertainty for a particular app.
The failed shortcut: comparing catalog prices
The simple approach is a spreadsheet with one row for OpenAI, one for the Claude API, and one for OpenRouter, followed by a minimum function. It answers a narrower question than the product team thinks it answers. Catalog rates don't capture prompt expansion, output length, invalid responses rejected by the app, extra tool turns, or duplicated work during recovery. The minimum cell can therefore identify the lowest quote while missing the more expensive execution path.
A fair comparison needs four layers:
| Layer | Measure | Why it changes the choice |
|---|---|---|
| Task quality | Acceptance rate on the same eval cases | A cheaper rejected answer has no product value |
| Variable usage | All attempted input and output tokens | Long context and retries alter the billable workload |
| Reliability policy | Retry and fallback attempts by outcome class | Recovery can duplicate work |
| Operating burden | Integration, observability, credential, and incident work | Runtime spend is only part of SaaS cost |
This also explains why there is no timeless winner for "cheapest." The result is a function of the chosen models, current quotes, token mix, acceptance threshold, and traffic distribution. Your mileage may vary — especially when a few long RAG requests dominate an otherwise small sample — so publish the distribution and the assumptions beside the mean.
Batchable work deserves a separate lane. The OpenAI Batch API guide documents a batch workflow, but interactive chat and offline enrichment have different latency constraints. Evaluate batch processing only for jobs that can wait, such as backfills or scheduled classification, and don't blend those results into the latency-sensitive cohort. The catch is straightforward: a lower-cost asynchronous option is not suitable when the user is waiting on the request.
Prompt-cost awareness is equally concrete. Store a prompt-template version and retrieved-context size with each attempt. When a notebook prompt graduates into production, a seemingly harmless instruction block or larger retrieval window can move every request. The ledger lets an eval run attribute that movement to a prompt revision rather than to the provider path.
Recovery is an HTTP and application-state problem
A fallback policy begins with classification. A timeout before the client observes a response is not proof that the upstream did no work. A 429 response is different from an application-level schema rejection, and neither should be treated like an authentication failure. RFC 9110 defines HTTP method semantics and explains idempotent methods; model-generation calls commonly use POST, so blindly replaying them can duplicate effects or usage unless the selected API documents an idempotency mechanism.
Be conservative.
Give every logical user operation an internal ID and every attempt a distinct child ID. Cap total attempts, enforce a deadline for the whole operation, add jitter to retry delays, and log the reason for each transition. Fallback should be a small state machine — not a catch-all except block — because the application must know whether it is retrying the same route, switching a model, or switching a provider path.
from dataclasses import dataclass
from enum import Enum
class Outcome(Enum):
ACCEPTED = "accepted"
RATE_LIMITED = "rate_limited"
TIMED_OUT = "timed_out"
INVALID_OUTPUT = "invalid_output"
AUTH_REJECTED = "auth_rejected"
@dataclass(frozen=True)
class Policy:
max_attempts: int = 2
allow_cross_route_fallback: bool = True
def next_action(outcome: Outcome, attempt_number: int, policy: Policy) -> str:
if outcome is Outcome.ACCEPTED:
return "finish"
if outcome is Outcome.AUTH_REJECTED:
return "stop_and_alert"
if attempt_number >= policy.max_attempts:
return "stop"
if outcome in {Outcome.RATE_LIMITED, Outcome.TIMED_OUT}:
return "fallback" if policy.allow_cross_route_fallback else "retry"
if outcome is Outcome.INVALID_OUTPUT:
return "repair_or_stop"
return "stop"
The code is intentionally policy-only. An adapter for a direct API and an adapter for a routing service should each translate their documented response into these internal outcomes. That keeps provider-specific fields at the edge and makes the fallback decision testable without live traffic. In a Node.js service, the same boundary can be represented with a TypeScript interface; the runtime language doesn't change the state model.
Test the uncomfortable sequence: the first attempt times out near the deadline, the fallback succeeds, and the original attempt may still have been processed. Confirm that one logical operation is returned to the caller, both attempts remain visible in telemetry, and the ledger counts all reported usage. Then inject a rate limit, malformed application output, and an authentication rejection. Each should take a deliberate branch.
Direct access or a unified key?
Direct provider access is the smaller dependency graph when one provider and its specific features are central to the product. It can expose provider-native controls without waiting for a normalization layer. Stick with direct access when you need a capability that a router doesn't expose, when contractual or data-location requirements demand a direct relationship, or when the extra network and control-plane dependency cannot fit the latency or risk budget. The cost is your own adapter surface, multiple credentials if you add providers, and more responsibility for cross-provider telemetry and fallback logic.
A routing layer is attractive when the application genuinely needs several model sources and the team values one credential boundary plus a normalized request surface. It can reduce integration duplication. It does not erase differences in model behavior, usage reporting, rate limits, or supported features, so the application still needs per-route evals and explicit capability checks. A unified key is also a wider trust boundary: key rotation, access policy, audit evidence, and outage planning now include that intermediary.
Neither shape wins by default. For a notebook exploring many models, a router can minimize setup friction. For a production feature bound tightly to one provider-specific behavior, direct access can be easier to reason about. A larger SaaS may deliberately support both behind its own thin adapter, but that choice is justified only if the eval results and recovery requirements repay the additional test matrix. Don't build a portability layer on speculation.
Before copying this choice, measure accepted cost per task, p50 and tail latency, acceptance rate, retry amplification, fallback frequency, token mix, and the share of traffic eligible for delayed batch processing. Review those measures by prompt version and task cohort. The final decision should name the losing scenarios too: which capability is unavailable, which dependency is added, and what operational work the team accepts. That record will age far better than a claim that one logo is always cheapest.
Top comments (0)