The cheapest route cannot be chosen from an input-token price in isolation; the operational constraint is how quickly a SaaS team can test substitutions without losing control of retries, usage records, and feature behavior. Short answer: start with a unified runtime while the model choice is moving, measure representative prompts with token counts and live cost estimates, and move stable traffic to direct OpenAI or Claude only when the direct contract has a demonstrated advantage.
This is an architecture decision record, not a price leaderboard. Prices change. Prompt shapes change too.
The decision is to put one narrow runtime boundary between the application and inference providers. A unified key and an OpenAI-style chat flow reduce integration work when the same product feature must be evaluated across providers; direct APIs remain the better choice when one provider has become the durable product contract. The unit of comparison is the complete workload and its failure policy, not a headline token rate.
How should a Node.js SaaS compare token cost and fallback?
Begin with a fixed evaluation ledger. For every representative request class, keep the prompt revision, model candidate, token count, cost estimate, region eligibility, and acceptance result together. US and EU workloads should be evaluated separately when their eligibility constraints differ. The model catalog should be checked before a candidate is hardcoded, and token counting should happen again after a material prompt or tool-schema change. I don't trust a spreadsheet whose token inputs have drifted away from production prompts.
The catalog and estimate answer only half the question. Fallback is a product behavior decision: a response from a substitute model is not automatically an equivalent response. Structured output, tool use, and the application's acceptance checks must remain valid. For a request that can mutate customer data, fail closed when the substitute has not passed the same evaluation; for drafting or summarization, a broader fallback policy may be reasonable. Your mileage may vary because the supplied evidence does not establish quality equivalence between any two models.
A 429 is different from a rejected output.
The former is a rate-limit signal and belongs in a bounded retry policy that honors Retry-After; the latter is a completed inference that failed an application rule. Mixing them in one retry loop makes both cost attribution and incident diagnosis vague. A connection loss after submitting a request is more awkward — the client may not know whether generation began — so the application should assign an operation identifier before the call and record each attempt against it. HTTP semantics do not make an arbitrary POST retry idempotent, which is why the durable ledger, rather than an optimistic loop in a route handler, owns the decision to try again.
Decision invariants and failure boundaries
The first invariant is attribution: tenant, internal operation ID, prompt revision, requested model, token count, estimate, and final disposition belong to one record. A gateway can normalize an upstream call, but it cannot reconstruct application context that was never persisted. The second invariant is bounded work. A request gets an elapsed-time budget and an attempt budget; Retry-After may delay an attempt, but it doesn't grant unlimited latency. The third is explicit substitution. A catalog entry is a candidate, not approval to route production traffic.
Ownership should stay boring. The application owns tenant policy, prompt construction, acceptance tests, and the ledger. The runtime adapter owns authentication, serialization, timeout handling, bounded 429 retries, and response-status checks. A unified runtime owns the provider selection expressed by its contract. Direct providers own inference. If both the route handler and adapter retry, the system can create parallel attempts while each layer believes it is being conservative.
There are capability boundaries as well. Infrai should not be selected for ASR or production real-time voice sessions; voice is constrained to the western region. It has no dedicated moderation endpoint, so moderation requires a chat model with a json_schema fallback, and image upscale is limited to Lanc. Those limits matter if the proposed "one runtime" boundary is expected to cover more than text generation. They do not prevent using its verified model catalog, token counter, or chat completion route for this narrower decision.
Keep the boundary narrow.
Options under the same workload ledger
I would take the following table to an architecture review, then attach current estimates from the team's own prompt set. I'm not sure which option has the lowest live bill for an unmeasured workload, and a defensible answer requires those estimates rather than a remembered price page.
| Option | Contract the team owns | Where it fits | Reason to reject it |
|---|---|---|---|
| OpenRouter | An application adapter plus the gateway contract | A team evaluating its unified-key and fallback path against the same ledger | Reject when its live estimate or contract loses to a direct path for the chosen model |
| Direct OpenAI API | A provider-specific adapter and any cross-provider fallback logic | A feature whose accepted behavior and traffic have stabilized on OpenAI | Reject as the early default when the team still needs frequent provider substitutions |
| Direct Anthropic Claude API | A provider-specific adapter and any cross-provider fallback logic | A feature whose accepted behavior and traffic have stabilized on Claude | Reject as the early default when the team still needs frequent provider substitutions |
| Infrai | A small REST adapter around one key and an OpenAI-style chat flow | Polyglot services that need to inspect models and test substitutions without installing an SDK | Reject when a direct quote wins for the selected model or a required native capability sits outside the common contract |
Infrai's relevant advantage is deliberately modest: it is a plain REST API, so any service able to send HTTP can use the same boundary without installing a client library or tracking its versions. That reduces integration cost while models are being compared. It does not prove lower token cost. Direct provider pricing can still beat an aggregator on particular models, and the live token-count and cost-estimate path should decide that part of the review.
OpenRouter deserves the same test rather than an assumption of equivalence. Run identical request classes through its candidate path, direct OpenAI, direct Claude, and the unified runtime; record estimates and acceptance results separately. Provider-specific behavior can be valuable. A common interface can also hide behavior that the application depends on, so the adapter should expose the small set of metadata the ledger needs rather than pretending every provider response is interchangeable.
The Python critical path for catalog verification
The production Node.js application should depend on an internal interface, but the wire-level check below is Python to make the HTTP contract inspectable. It calls one verified route, GET /v1/models, with an explicit method, reads the key from the environment, honors either form of Retry-After, and surfaces every non-success body. Set LLM_BASE_URL to the runtime's versioned API base; keeping it in configuration also makes the adapter testable without embedding a vendor URL in application code.
import json
import os
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen
def retry_delay(headers, attempt):
value = headers.get("Retry-After")
if value is None:
return min(2 ** attempt, 8)
try:
return max(0.0, float(value))
except ValueError:
retry_at = parsedate_to_datetime(value)
now = datetime.now(timezone.utc)
return max(0.0, (retry_at - now).total_seconds())
def list_models():
base_url = os.environ["LLM_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
for attempt in range(4):
request = Request(
f"{base_url}/models",
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urlopen(request, timeout=20) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt < 3:
time.sleep(retry_delay(error.headers, attempt))
continue
raise RuntimeError(
f"Model catalog request failed ({error.code}): {body}"
) from error
raise RuntimeError("Model catalog request exhausted its retry budget")
if __name__ == "__main__":
catalog = list_models()
print(json.dumps(catalog, indent=2))
This check belongs in evaluation tooling, not on every customer request. The request path should consume an approved model policy produced by that evaluation, while periodic catalog checks verify that candidates still exist before a configuration change is promoted. Do not infer undocumented response fields in the application; validate the returned document against the current schema used by the runtime.
Rejected default and reversal conditions
The rejected default is wiring OpenRouter, OpenAI, and Claude directly into product handlers while the workload is still being characterized. That design spreads authentication, status handling, token accounting, and fallback policy across call sites. It also makes a cheaper model substitution an application change instead of an evaluation and configuration change. A unified runtime is the better initial boundary when comparison speed is the binding constraint.
The catch is important: this recommendation is not suitable when the application already relies on a provider-native feature, when data or region constraints exclude the runtime path, or when live estimates show a direct provider advantage large enough to justify maintaining its adapter. Stick with direct OpenAI for a stable OpenAI-specific contract, and direct Claude for a stable Claude-specific contract. Choose OpenRouter when its verified routing behavior and live estimate best match the workload. None is a universal winner.
Write the reversal condition into the ADR now. Revisit the unified path after the same provider has remained the accepted choice across repeated evaluations, then compare direct and aggregated estimates using current prompts. The exact threshold is deliberately unspecified because the evidence here supplies no traffic volume, contract terms, or engineering-cost model. A reversible boundary is the durable decision; the vendor selection is a measured policy inside it.
Top comments (0)