DEV Community

LunarBreeze4173085
LunarBreeze4173085

Posted on

One-Key Unified LLM API for Node.js Media Support: A Cost Ledger

#node #llm #architecture

Use a unified LLM API only behind an application-owned usage ledger for media-support ticket triage; the one-key convenience is secondary to proving which tenant, region, model, and retry produced each result. That is the practical answer for a Node.js backend serving US and EU customers that needs one credential across OpenAI, Claude, and Gemini without losing accountability.

This is an architecture decision record, not a vendor scorecard. The system classifies incoming support tickets, proposes urgency and topic, and may draft a response for an agent to approve. The storage question is more important than the routing slogan: can an operator reconstruct the customer-visible decision and its cost after a timeout, a model change, or a late bill?

What must a one-key LLM backend record before triage?

Start with invariants. Every ticket gets an application-owned request ID. Every tenant has an explicit region and budget policy. The selected model and capability profile are attached to the attempt before the outbound call. Usage, pricing-version, and outcome fields are immutable after settlement except through an auditable correction. A retry must not create a second approved reply.

The last invariant is where many “simple” designs become misleading. A timeout does not tell the application whether the upstream model completed. If the worker retries blindly, it may spend twice and produce two drafts. Inference and the customer-visible write should be separate state transitions: reserve an attempt, call the model, validate the structured result, then commit one approved draft under an idempotency key. An external call cannot be made exactly once by adding a gateway around it; the durable ledger is what lets reconciliation distinguish “never sent” from “sent, result unknown.”

Keep it boring.

For a media tenant, the minimum useful attempt record is tenant_id, ticket_id, request_id, region, model_id, input_tokens, output_tokens, pricing_version, estimated_cost, provider_request_id, status, and timestamps. Store a prompt digest and bounded metadata by default rather than copying an unreleased film synopsis or a customer contact record into every log sink. Retention and deletion rules must cover prompts, traces, cached context, and generated drafts, not only the primary ticket table.

I separate three failure boundaries:

  • Transport: rejection, rate limiting, timeout, or an unknown result. Reconcile before retrying.
  • Policy: a region, model, data class, or tenant budget is not allowed. Fail deterministically.
  • Semantic: the model returned valid text that violates the triage schema or workflow. Quarantine it for validation or human review.

Those failures have different operational owners. Treating them as one generic LLMError guarantees noisy retries and opaque tenant invoices.

How should a Node.js backend route models across US and EU tickets?

Routing should be an admission decision, not a string substitution. Resolve the tenant policy, choose an allowed model from a controlled catalogue, and persist that decision before the provider request. I don't infer residency from a browser locale or a friendly model name — region is a data-flow constraint that follows the prompt, logs, traces, cached context, and stored output.

The critical path can be represented with a provider-neutral adapter. The production service may be Node.js, but the contract matters more than the language: normalized input in, explicit usage and provenance out.

from dataclasses import dataclass
from typing import Protocol


@dataclass(frozen=True)
class TenantPolicy:
    tenant_id: str
    region: str
    model_id: str
    budget_cents: int


@dataclass(frozen=True)
class ModelResult:
    text: str
    input_tokens: int
    output_tokens: int
    provider_request_id: str


class ModelAdapter(Protocol):
    def complete(self, *, model_id: str, prompt: str, request_id: str) -> ModelResult:
        ...


def triage_ticket(*, policy: TenantPolicy, ticket_id: str,
                  prompt: str, adapter: ModelAdapter) -> dict:
    if policy.region not in {"US", "EU"}:
        raise ValueError("unsupported tenant region")
    if not policy.model_id:
        raise ValueError("model policy is missing")

    request_id = f"{policy.tenant_id}:{ticket_id}"
    # The real implementation inserts this row with a unique request_id.
    result = adapter.complete(
        model_id=policy.model_id,
        prompt=prompt,
        request_id=request_id,
    )
    if not result.text.strip():
        raise ValueError("empty model result")

    return {
        "tenant_id": policy.tenant_id,
        "ticket_id": ticket_id,
        "request_id": request_id,
        "region": policy.region,
        "model_id": policy.model_id,
        "input_tokens": result.input_tokens,
        "output_tokens": result.output_tokens,
        "provider_request_id": result.provider_request_id,
        "status": "awaiting_validation",
    }
Enter fullscreen mode Exit fullscreen mode

Notice what the function does not do: it does not pretend that OpenAI, Claude, and Gemini have identical capabilities, and it does not calculate a cost from a hard-coded rate. Keep provider capability declarations separate from the common request shape. If a workflow needs tool calls, vision, streaming, or a particular structured-output guarantee, the catalogue should say so explicitly and the router should reject an incompatible model before the call.

Which integration boundary makes tenant cost visible?

The comparison is about ownership and failure surface, not a universal ranking. A useful choice is the smallest boundary that leaves the ledger and policy under your control.

Integration shape Cost and provenance owner Appropriate use Trade-off
Direct provider clients Application team normalizes usage and credentials One provider or provider-specific features Each added model family multiplies adapters, policy paths, and billing tests
Self-hosted gateway Your team owns routing, logs, upgrades, and availability Custom routing with internal operational control Patching, capacity, and on-call become part of the product
Managed unified gateway Gateway supplies a common request surface; application keeps its ledger Fast evaluation across several text models A common surface may omit provider-specific semantics and does not replace regional review
Queue with worker adapters Durable attempt record precedes asynchronous calls Large ticket backlogs and controlled retries Latency rises, and operators must explain pending states

For per-tenant cost visibility, the common API is useful only if it returns enough usage and provenance to populate the ledger. Record request count, input and output tokens, rejected attempts, retries, model, region, and the pricing version used for an estimate. Keep estimated cost distinct from settled cost. A model migration should be visible as a dimension change, not disguised as a mysterious increase in traffic.

The dashboard should also show unknown-result attempts. Hiding them creates an attractive total that cannot be reconciled with provider records. I have seen enough accounting systems fail at the boundary between “request accepted” and “response observed” to distrust a green total without an exception queue.

When is one unified API the wrong choice?

The catch is that a unified contract is a portability aid, not a compliance certificate. It is not suitable when the workflow depends on a provider-specific tool protocol, real-time session behavior, a particular residency guarantee, or a new modality that the common surface has not modeled. Keep direct integration when that capability is the product. Choose an internally operated gateway when the organization must inspect every routing decision and can fund its operations.

It is also a poor fit for unbounded raw conversations. If retention, deletion, tenant isolation, and audit ownership are undefined, another routing layer adds review scope without solving the underlying storage problem. For a high-volume ticket queue, asynchronous workers may be preferable to synchronous requests, but that choice changes latency, user experience, and recovery semantics.

Do not classify HTTP 429, a timeout, and malformed structured output as the same error. The first may permit a delayed retry, the second requires reconciliation, and the third points to a contract problem. Your mileage may vary on retry budgets; the classification should not vary by whichever engineer happens to be on call.

The decision rule is modest: adopt one key when it reduces secret distribution and the common API preserves the evidence your ledger requires. Reject it when it hides capabilities, region behavior, usage, or failure state. The architecture is successful when a support lead can answer “which tenant paid for this draft, where did its data go, and why was this model selected?” without reading application logs by hand.

References

Top comments (0)