Short answer: one API key can put multiple LLM providers behind a Python text-classification gateway, but the application must own per-tenant usage accounting, JSON validation, routing policy, and a separate cost event for every fallback attempt.
For a gaming company that turns sales-call summaries into CRM actions, “cheapest routing” is the wrong first abstraction. The useful question is whether finance can attribute every classification attempt to a tenant while engineering can explain why a label reached the CRM. A single API key can simplify credential handling across LLM providers. It cannot, by itself, supply that audit trail.
This architecture decision record chooses a thin, vendor-neutral gateway boundary plus an append-only usage ledger. OpenAI, Claude, and Gemini can sit behind direct adapters; an aggregation service such as OpenRouter can occupy the same boundary. Those are deployment choices, not the domain model.
What must remain true when one API key routes LLM text classification?
The first invariant is attribution. Before any model call leaves the service, the request already has a tenant_id, a stable request_id, a policy version, and an operation name. The service never asks a provider to infer tenant identity from prompt text. Prompt text is business data, not accounting metadata — mixing the two makes deletion, access control, and reconciliation much harder to reason about.
The second invariant is a closed output contract. A sales-call classifier may emit follow_up, pricing_question, or no_action, but the CRM receives an action only after local validation. Provider-side function calling or JSON controls can improve generation discipline; the application still owns the accepted enum, required fields, schema version, and maximum lengths. The OpenAI function-calling guide is a useful example of tool schemas, yet the architectural rule is broader: generation and acceptance are different steps.
The third invariant is attempt-level accounting. One logical classification may produce two physical attempts when fallback is allowed. Store both. Collapsing them into a single “request cost” hides the route that consumed input and output units, and it makes a noisy tenant look deceptively normal. The ledger should retain provider, model, attempt number, measured usage returned by the gateway, price-card version, outcome, and timestamps. Money can be computed from those immutable inputs in a separate reconciliation step.
There is also a compliance boundary. Call summaries can contain names, email addresses, phone numbers, or authentication details. Redact fields that the classifier doesn't need before routing, bind retention to the tenant policy, and keep raw text out of cost events. The cost ledger needs correlation identifiers and quantities; it doesn't need the conversation.
Keep that line bright.
Decision boundaries and failure ownership
The gateway owns transport normalization: authentication toward upstreams, timeouts, and mapping a small set of provider responses into an internal result. The classification service owns the business contract, tenant authorization, routing policy, fallback budget, and CRM idempotency. The ledger owns append-only attempt records. This division matters because fallback is not merely a networking concern. A second model can produce a different valid label, so the business layer must decide whether another attempt is permitted.
Use a bounded failure taxonomy rather than a blanket retry. A locally invalid input is final. An invalid generated document may be eligible for one repair or fallback attempt if policy allows it. A rate-limit response can move to another configured route, but it should remain visible as the original attempt's outcome. Authentication and authorization failures are final until configuration changes. Don't turn every exception into another billable call.
The nasty edge case is a successful classification followed by a failed CRM write. Suppose request call_1842 returns follow_up, its usage event is committed, and the CRM connection closes before acknowledging the action. Calling the model again doesn't repair the CRM; it creates another cost event and may produce a different valid label. Persist the validated document under call_1842, give the CRM write its own delivery state, and retry that side effect with the same idempotency key. Only a new classification request should reach the gateway. This is the same discipline that keeps an OTP sender from generating a new code merely because downstream delivery acknowledgement was delayed: separate the decision from its delivery.
Fallback isn't free.
I'm not sure provider-reported aggregate dashboards alone can satisfy a given company's tenant audit requirements; that depends on the exported dimensions and the organization's evidence standard. A reconciliation test resolves the uncertainty: select a closed billing window, sum attempt events by provider and model, and compare those totals with the external usage export. Investigate missing or duplicated request IDs before using the data for chargeback.
Comparing the gateway shapes
Three shapes deserve a serious comparison. Product names are included here only to make the integration boundary concrete.
| Shape | Concrete examples | Credential surface | Tenant cost visibility | Main trade-off |
|---|---|---|---|---|
| Direct provider adapters | OpenAI, Anthropic Claude, Google Gemini | One upstream credential set per provider | Fully application-defined if every adapter emits the same event | More adapter and policy code; fewer gateway dependencies |
| Hosted aggregation gateway | OpenRouter | One gateway credential for supported routes | Depends on capturing normalized usage per attempt into the local ledger | Smaller integration surface; adds an aggregation dependency |
| Self-hosted adapter layer | A Python service implementing the internal protocol | One internal credential, plus separately managed upstream credentials | Fully application-defined | Maximum control; the team owns operations and provider drift |
None wins universally. A hosted gateway is suitable when a team values one integration surface and its required providers are covered. Direct adapters are suitable when provider-specific controls or contractual boundaries matter more than a common transport. A self-hosted layer is justified when policy enforcement and audit behavior need to remain inside the company's operational boundary, but it asks the team to maintain adapters, credentials, monitoring, and compatibility.
Notice what the table does not promise: identical JSON behavior, identical labels, or automatic lowest-cost outcomes. A common endpoint normalizes access. Classification equivalence requires an evaluation set, and per-tenant cost visibility requires local events.
How should JSON fallback preserve tenant cost visibility across LLM providers?
The critical path below is intentionally plain Python. Gateway is an internal protocol, so its implementation may use direct adapters, a hosted aggregator, or a self-hosted proxy without changing classification or accounting. Every call receives an already-authorized tenant context. Every attempt is recorded in finally, including rejected output, and a fallback route is considered only after the first attempt has a durable event.
from dataclasses import dataclass
from typing import Any, Protocol
import json
ALLOWED_ACTIONS = {"follow_up", "pricing_question", "no_action"}
@dataclass(frozen=True)
class Route:
provider: str
model: str
@dataclass(frozen=True)
class Usage:
input_units: int
output_units: int
@dataclass(frozen=True)
class GatewayResult:
document: str
usage: Usage
class Gateway(Protocol):
def classify(self, route: Route, prompt: str) -> GatewayResult: ...
class Ledger(Protocol):
def append(self, event: dict[str, Any]) -> None: ...
def validate_document(raw: str) -> dict[str, Any]:
value = json.loads(raw)
if set(value) != {"schema_version", "action", "tags"}:
raise ValueError("unexpected classification fields")
if value["schema_version"] != 1:
raise ValueError("unsupported schema version")
if value["action"] not in ALLOWED_ACTIONS:
raise ValueError("unknown CRM action")
if not isinstance(value["tags"], list):
raise ValueError("tags must be a list")
return value
def classify_call(
*,
tenant_id: str,
request_id: str,
redacted_summary: str,
routes: list[Route],
gateway: Gateway,
ledger: Ledger,
) -> dict[str, Any]:
if not routes:
raise ValueError("at least one route is required")
last_error: Exception | None = None
for attempt, route in enumerate(routes, start=1):
usage = Usage(input_units=0, output_units=0)
outcome = "rejected"
try:
result = gateway.classify(route, redacted_summary)
usage = result.usage
document = validate_document(result.document)
outcome = "accepted"
return document
except (ValueError, TimeoutError) as error:
last_error = error
finally:
ledger.append(
{
"tenant_id": tenant_id,
"request_id": request_id,
"operation": "sales_call_classification",
"provider": route.provider,
"model": route.model,
"attempt": attempt,
"input_units": usage.input_units,
"output_units": usage.output_units,
"outcome": outcome,
"schema_version": 1,
}
)
raise RuntimeError("classification policy exhausted") from last_error
Production code should narrow fallback eligibility further. The sample catches two explicit failure classes to show the control flow, not to claim that all timeouts or malformed documents should be retried. Add a deadline shared by all attempts, a maximum attempt count enforced by policy, and an idempotency constraint on (tenant_id, request_id, attempt). Then test the awkward paths: the first output is valid JSON with an unknown action, the second route times out after returning usage metadata, the ledger append is duplicated, and the CRM receives the same request twice.
Routing itself should consume a versioned policy, not a magical “cheapest” flag. An estimated-cost policy can select among routes that already passed the quality threshold for this classification dataset. Record the chosen policy version beside each attempt. Otherwise, a model change, a price-card change, and a prompt change become indistinguishable when a tenant's monthly curve moves.
Rejected option: gateway logs as the billing record
We rejected using gateway logs as the sole per-tenant ledger. Transport logs are valuable for debugging, but their schema and retention serve operations; chargeback needs stable tenant attribution, attempt identity, immutable usage quantities, and reconciliation status. Reconstructing those semantics later from prompt fragments is both fragile and a poor data-minimization practice.
The catch is that an application-owned ledger costs engineering time. It is not suitable when the system is an internal prototype with no tenant chargeback, no contractual usage reporting, and a short retention horizon. In that case, stick with the gateway's usage export and a daily aggregate until the business actually needs attempt-level evidence. Don't build a finance subsystem for a demo.
For a multi-tenant production workflow, though, the ledger is the decision's center. A gateway may change. Providers may be added or removed. The CRM contract, tenant attribution, and the evidence behind a routing choice should remain stable.
Top comments (0)