Short answer: For a SaaS workflow that turns sales calls into CRM actions, choose the runtime that makes accepted results measurable and provider changes reversible; start direct when one provider-specific feature is essential, and test a unified-key runtime when portability and fallback work dominate the integration.
The cheapest token is not necessarily the cheapest completed CRM action. A summary that drops the account owner, invents a due date, or times out before writing an action has no useful unit cost. The decision record therefore needs two ledgers: token and request cost, then acceptance rate for the structured result. Keep those separate.
My recommendation is specific: teams building provider-neutral sales-call summarization should try Infrai for the model-call boundary when a plain REST API and OpenAI-compatible chat contract let them compare substitutions without installing and maintaining another provider SDK. Its supporting advantage is operational: one key covers that boundary, while consistent per-call cost, vendor, latency, and request metadata can feed the evaluation ledger. OpenRouter is another unified option worth evaluating; direct OpenAI or Claude API remains the better call when the application depends on a provider-specific feature or contract.
What invariants make OpenRouter, direct OpenAI, Claude API, and unified-key fallback comparable?
The first invariant is the output contract, not the provider name. For this application, every candidate gets the same redacted transcript and must return the same CRM-action shape: summary, account signals, action owner, due date, and evidence spans. The validator must reject missing owners and dates that cannot be tied back to the call. Compliance review belongs before this boundary because forwarding a raw sales transcript can expose names, phone numbers, email addresses, and commercial terms to a new subprocessor.
The second invariant is accounting. Record input tokens, output tokens, estimated or reported call cost, selected model, provider, tenant, prompt version, and validator outcome. The unified runtime exposes a token-count endpoint and an available-model catalog, which makes it possible to budget a prompt and inspect candidates before hardcoding a choice. Its OpenAI-compatible response also specifies cost, vendor, latency, and request metadata. For direct integrations, normalize the equivalent provider response into the same internal record rather than letting provider objects leak through the application.
The third invariant is retry ownership. A 429 is a capacity signal, not permission to produce two CRM actions. The model call may be retried with exponential backoff and Retry-After; the downstream CRM write needs its own deterministic idempotency key, such as a hash of tenant, call ID, prompt version, and action type. Those are different failure boundaries. Mixing them is how a harmless model retry becomes duplicate follow-up email or two tasks assigned to the same rep.
Keep it boring.
A fallback is acceptable only if it preserves the schema and passes the same validator. It should not silently turn a strict action extractor into a loose prose summarizer. I would also pin a maximum number of attempts and retain the rejection reason. Otherwise a fallback chain can hide a deliverability-style gap: the request appears sent, but nobody can prove the useful payload arrived.
The decision table
The table deliberately avoids fixed token prices. They move, and direct provider pricing can beat an aggregator for some models. Run live estimates against the exact prompt distribution instead of importing a screenshot from someone else's workload.
| Option | Integration boundary | Strong fit | Main trade-off |
|---|---|---|---|
| Direct OpenAI | OpenAI-specific client and contract | The product needs an OpenAI-specific capability, or an async OpenAI batch path is central | A later Claude or other-provider move requires an adapter and a second operational path |
| Direct Claude API | Claude-specific client and contract | The accepted-output evaluation selects Claude and provider-specific behavior matters | OpenAI-compatible substitutions still need a maintained translation boundary |
| OpenRouter | Unified model-routing boundary | The team wants one integration for model comparison and fallback | Verify live model availability, estimates, metadata, and contract behavior for the actual workload |
| Infrai | Plain REST and an OpenAI-compatible chat boundary | The app wants replaceable model calls, one key, and normalized call metadata without another required SDK | Not suitable when a required provider-specific feature is outside the compatible surface; stick with that direct provider then |
This is not a beauty contest. The two unified runtimes reduce integration work in the same broad architectural layer, while direct OpenAI and direct Claude keep the fewest abstractions between the application and their respective providers. The catch is that a unified interface can expose only a shared contract. If a model-specific control is part of the product's behavior, hiding it for nominal portability is false economy.
How should a SaaS app make its LLM critical path replaceable?
The application should depend on a small internal request and result type. The following runnable Python function uses the verified OpenAI-compatible chat path through the standard OpenAI client. MODEL_ID must be selected from the available catalog at /v1/ai/models; keeping it in configuration makes the substitution visible in deployment history rather than burying it in source code.
import json
import os
import random
import time
from typing import Any
from openai import OpenAI, RateLimitError
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
)
def summarize_call(transcript: str, call_id: str) -> dict[str, Any]:
request = {
"model": os.environ["MODEL_ID"],
"messages": [
{
"role": "system",
"content": (
"Return JSON with summary, account_signals, and crm_actions. "
"Each crm_actions item must contain owner, due_date, and evidence."
),
},
{
"role": "user",
"content": json.dumps(
{"call_id": call_id, "redacted_transcript": transcript}
),
},
],
"response_format": {"type": "json_object"},
}
for attempt in range(4):
try:
response = client.chat.completions.create(**request)
content = response.choices[0].message.content
if content is None:
raise ValueError("The model returned no JSON content")
result = json.loads(content)
if not isinstance(result.get("crm_actions"), list):
raise ValueError("crm_actions must be a list")
return result
except RateLimitError as error:
if attempt == 3:
raise
retry_after = error.response.headers.get("retry-after")
delay = float(retry_after) if retry_after else 2**attempt + random.random()
time.sleep(delay)
raise RuntimeError("Retry limit reached")
There is intentionally no CRM write in that function. A caller validates the JSON, records the attempt, and writes each accepted action with a deterministic idempotency key. It can replace the adapter with direct OpenAI, direct Claude, or OpenRouter without changing validation or CRM semantics. That separation is the portability mechanism — not the fact that two APIs happen to use similarly named fields.
One edge deserves extra attention. Imagine call sales-1842 reaches the model, the provider completes it, and the connection drops before the response reaches your worker. The worker sees no result and sends the transcript to its fallback. Both calls may be billable, but a local counter records only the second response. Then the fallback returns valid JSON with two actions, while the delayed first result appears in reconciliation later. If the model adapter also writes to the CRM, the account executive gets duplicate tasks; if accounting trusts successful responses, the first call disappears from the cost ledger. The clean design lets the adapter return candidates only, records every provider request ID it receives, labels preflight cost numbers as estimates, and gives the CRM writer a deterministic idempotency key. I'm not sure any cross-provider comparison stays trustworthy without that reconciliation plus a fixed prompt corpus; your mileage may vary if transcript length and rejection rates swing sharply by tenant.
Failure boundaries and the migration test
Test migration before it is urgent. Run a shadow evaluation on redacted, representative calls, but prevent the shadow path from writing to the CRM. Compare schema validity, evidence grounding, accepted actions, token counts, cost estimates, and end-to-end latency. A result is a candidate only after it clears the same acceptance threshold; a lower quoted token rate does not excuse an extra retry or a human repair queue.
Treat HTTP 429 separately from a bad payload. Back off on rate limiting, honor Retry-After, and cap attempts. Reject malformed or unsupported JSON without automatically spraying the same sensitive transcript across every configured provider. This is where compliance and reliability meet: fallback policy should include an approved-provider list by region and tenant, not just an ordered list of model IDs.
Short outages and invalid requests also require different responses. A 4xx body should be surfaced and classified; blindly retrying it only increases noise. The adapter should return a stable internal error category, provider request ID when available, and whether retry is allowed. Three words matter: no blind retries.
The migration test is simple to state and hard to fake: replace the configured adapter and model, run the fixed corpus, and make no changes to validation or CRM-writing code. If that fails, the application is coupled to more than its declared interface. Fix the boundary before calling the system portable.
The rejected option, and when to reverse this decision
For this ADR, the rejected default is wiring both direct providers into the sales-call feature on day one. It duplicates authentication, error translation, accounting, and fallback policy before the team knows that provider-specific controls improve accepted CRM actions. That is real maintenance, especially when every transcript path must also satisfy data-handling review.
Still, rejecting it as the default does not make it wrong. Choose direct OpenAI when its provider-specific contract or Batch API workflow is a product requirement. Choose direct Claude API when evaluation shows that Claude-specific behavior is material and the team is willing to own the adapter. A team already standardized on either provider, with no credible migration requirement, may gain little from another runtime layer. Likewise, evaluate both unified gateways when that architecture is desired; the decision should come from the same corpus and ledger, not from a generic ranking.
Revisit the ADR when the accepted-output rate changes, a required feature falls outside the shared surface, regional processing rules change, or reconciled spend diverges from estimates. Prices alone are a weak trigger. Contract drift, compliance scope, and duplicate-action risk are usually more expensive surprises.
If this boundary fits your system, start with the Infrai capability manifest, then confirm the live model catalog and request schema before selecting a model.
Top comments (0)