Cheap LLM API Gateway: One-Key Token Cost Signals for Moderation Queues

The least complex cheap LLM API gateway that works for logistics moderation is the one that makes every accepted report traceable to a tenant, a queue item, and an estimated cost before a human-review deadline is threatened. A single API key is convenient, but it is not the control plane. Per-tenant attribution is.

TL;DR: Record estimated input and output usage at admission, reconcile it with final usage, and join both to queue age, retries, cache disposition, region policy, and batch status. Page on review risk, not on a vague increase in tokens. A low advertised rate cannot rescue a system that hides which tenant created the spend or why reports missed review.

The page arrives at 02:17: moderation_review_deadline_risk{region="eu"} > 0. The on-call sees 38 logistics reports older than the 15-minute review objective, spread across three tenants. Requests are still completing, so an availability dashboard is green. The useful questions are narrower: Did one tenant burst? Did retries multiply accepted work? Did a cache miss change latency? Did a batch wait too long? Can the team prove that EU-bound reports stayed on their permitted path?

That is the test.

Can a cheap LLM API gateway make one key accountable?

The late page is a symptom. The earlier signal is a widening gap between admitted work and terminal outcomes, partitioned by tenant. Count reports when the system accepts them, then count exactly one final disposition: classified, sent to human review, permanently failed, or expired. A request timeout is not a disposition because a retry may still complete the same logical job. I treat duplicate delivery as normal queue behavior. Every report therefore needs a stable idempotency key that survives retries and execution-path changes. The worker may attempt the job more than once, but only one result can advance the report. Without that rule, token totals rise while the queue appears productive, and a gateway comparison rewards the wrong thing. For each tenant and region, watch three related values: oldest ready-job age, admitted-minus-finalized work, and estimated cost of unfinished work. The first protects the human-review objective. The second exposes throughput loss. The third makes the financial blast radius visible while jobs are still in flight. None requires a provider-specific model name in the alert. The one-key design works only when identity is supplied above the credential: accepting a shared key and reconstructing tenancy later is too late for admission control, and it leaves retries difficult to assign when the process stops between dispatch and ledger update.

A practical warning threshold might be oldest-job age above 8 minutes for two evaluation windows when the review objective is 15 minutes. Those numbers are an example, not a universal target. Set them from your own arrival pattern, worker concurrency, and time reserved for human handling. The warning must leave time to act.

No estimate is perfect.

Instrument admission, completion, and reconciliation

Cost visibility starts before the network call. At admission, store the tenant ID, report ID, idempotency key, policy region, input-token estimate, selected execution mode, and cache eligibility. At completion, append the attempt count, terminal status, reported usage when available, cache outcome, and elapsed time. Keep estimates and reported values separate; overwriting the estimate destroys the error signal used for capacity planning.

This Go sketch shows the boundary. It deliberately emits a logical-job record separately from attempt telemetry. The estimator is injected because tokenization and accounting rules belong to the selected runtime contract, not to a string-length shortcut.

package moderation

import (
    "context"
    "time"
)

type Job struct {
    TenantID       string
    ReportID       string
    IdempotencyKey string
    Region         string
    Payload        []byte
}

type Estimate struct {
    InputTokens  int64
    OutputTokens int64
    CostMicros   int64
}

type Usage struct {
    InputTokens  int64
    OutputTokens int64
    CostMicros   int64
    CacheHit     bool
}

type Recorder interface {
    Admitted(context.Context, Job, Estimate, time.Time) error
    AttemptFinished(context.Context, Job, Usage, time.Duration, error) error
    Finalized(context.Context, Job, string, time.Time) error
}

type Estimator interface {
    Estimate(context.Context, Job) (Estimate, error)
}

Use fixed-cardinality labels for metrics, such as region, execution mode, and terminal status. Put tenant and report identifiers in logs or traces where retention and access rules permit them; an unbounded tenant label can turn the monitoring system into the next incident. A separate ledger keyed by tenant and logical job is clearer for chargeback and reconciliation.

Reconcile on a schedule. Compare admission estimates with final reported usage and flag both absolute drift and persistent directional bias. A missing final usage record must remain visible as unknown cost, not silently become zero. For an asynchronous batch, retain the original admission time so delayed work cannot disappear from queue-age accounting.

Compare behavior, not a price column

A useful candidate exercise replays a fixed, redacted set of logistics moderation reports through the same gateway interface. Do not rank candidates by a transient unit price. Score the evidence each path returns and the failure semantics your workers must absorb.

Searches often name OpenAI, Claude, and Gemini because teams want one integration across those runtime families. Treat those names as test-matrix rows, not as a ranking. The same acceptance suite should run from a Node.js service or a Go worker, even though the instrumentation example here is Go; client language cannot be allowed to change tenant attribution.

Decision check	Evidence to capture	Operational reason
Tenant attribution	Stable tenant and logical-job keys on every ledger row	Shared credentials must not erase ownership
Estimate quality	Estimated versus final input and output usage	Admission controls need bounded error
Retry semantics	Attempt count plus one terminal disposition	Duplicate work must not become duplicate review
Cache behavior	Eligibility, hit or miss, and charged usage	A hit rate alone cannot explain spend
Batch behavior	Admission, dispatch, completion, and expiry times	Deferred work still consumes deadline budget
Region policy	Requested policy and observed execution path	EU and US handling must be auditable
Streaming	First-event time, terminal event, and disconnect outcome	Partial output needs an explicit state

Streaming deserves a small test of its own. Server-Sent Events deliver a one-way event stream over an HTTP connection. A client disconnect can happen after partial output, so the worker needs a rule for whether that attempt is retryable and how it records incomplete usage. The transport can improve time to first visible result, but it does not provide idempotency or a terminal business outcome.

Caching has a similar trap. Cache keys must include every input that can change classification, including the moderation policy version and relevant tenant configuration. Otherwise a technically successful hit can return a decision made under the wrong policy. Track avoided execution and stale-policy rejection separately.

Batching trades dispatch efficiency for waiting time. It fits reports with enough deadline budget and predictable cancellation semantics. Interactive and batch paths should converge on the same ledger and finalization rules; two accounting models create reconciliation gaps exactly when operators need a single answer.

The limitation is deliberate: a gateway adds another operational boundary. For a small service using one runtime in one region, a direct API integration plus the same tenant ledger may be easier to own. The gateway approach becomes useful when central policy and consistent evidence justify that extra failure domain. This trade-off belongs in the selection record.

Turn the trace into an on-call action

When the early warning fires, the runbook should identify a bounded action, not ask the on-call to browse dashboards until the queue recovers. Start with the tenant contribution to unfinished estimated cost and oldest-job age. Then split by region, mode, cache outcome, and attempt count. This sequence distinguishes a tenant burst from retry amplification, a regional capacity constraint, or batch delay.

The first protective action is admission control at the tenant boundary. Preserve reports already near their review deadline, slow new low-priority work, and keep the idempotency ledger authoritative. A global throttle is easier to implement but lets one noisy tenant degrade every logistics customer. Per-tenant limits cost more operational effort and produce a much smaller blast radius. That trade is usually justified when cost visibility is the primary decision axis.

Do not switch execution paths merely because one path looks cheaper. A path change is safe only if its output contract, region policy, idempotency behavior, and observability fields remain compatible. Record the reason as part of the job history. Otherwise a recovery action turns into an audit gap.

The post-incident question is concrete: which signal first diverged from normal, and how many minutes existed between that divergence and deadline risk? If the answer is unavailable, add the missing timestamp or disposition before tuning a threshold. More dashboards are not a substitute for a complete lifecycle.

Fix the lifecycle first.

How expensive is a sensitive warning?

A threshold set too low pages on ordinary tenant bursts. That cost is larger than an interrupted night: repeated false positives teach responders to wait, which consumes the very lead time the warning was meant to create. A threshold set too high protects attention but detects the problem only after the recovery window has narrowed.

Tune with historical queue distributions, but preserve a hard backstop tied to the review objective. Require persistence across two or more evaluation windows for a warning, while allowing a faster page when oldest-job age approaches the deadline or unfinished estimated cost crosses an explicit tenant budget guardrail. Route low-confidence estimate drift to a ticket; route imminent missed review to a page. Different urgency deserves different interruption.

Review alert outcomes by tenant and region. Count actionable pages, false positives, and cases detected first by the late deadline alarm. If a threshold changes, record the old value, new value, reason, and expected effect. This is configuration with production consequences.

Write it down.

The cheapest gateway is therefore not a product label or a single-key promise. It is the runtime path whose logical-job ledger lets the team control cost without losing deadlines, regional policy, or retry correctness. Measure that before signing a contract. Keep measuring it after deployment.

Further reading

• MDN: Using server-sent events