Short answer: for a property-management startup, the cheapest one-key router is the one that minimizes cost per correctly classified moderation report on your own replay set while preserving a provider-neutral request, response, and error contract. Raw token rates alone cannot make that choice.
| Choice | Best fit | Main catch | Measure first |
|---|---|---|---|
| Managed multi-provider router | Small team optimizing time-to-first-call | Another control plane owns the routing boundary | Valid classifications per dollar |
| Self-hosted gateway | Team that needs policy and telemetry under its control | You own upgrades, capacity, and incident response | Operator hours plus inference cost |
| Thin in-app adapters | Narrow model set and strict contract control | Every new capability adds adapter work | Change lead time and test burden |
My default for an early startup app is a managed router behind a tiny internal interface, with request fixtures stored outside the router. Choose the self-hosted runner-up when data-path control or custom routing policy is more important than low configuration overhead. Choose direct adapters when the application genuinely uses only a small, stable slice of each provider API.
This is a decision about portability, not a hunt for a permanent lowest price. OpenAI, Claude, and Gemini differ in message shapes, structured-output behavior, usage accounting, and model lifecycle. A shared API key removes credential sprawl; it doesn't erase those differences.
How should a startup app compare token cost across one-key routers?
Start with the unit of work: one moderation report reaching a human reviewer with a valid label, confidence, rationale, and trace ID. A property manager does not buy tokens for their own sake. They need reports such as harassment, fraud, safety, or noise triaged consistently enough that urgent cases rise and ambiguous cases stay in the human queue.
The useful equation is:
effective cost = inference charges + router charges + retries + invalid-output handling + operational labor
That denominator matters even more:
cost per accepted classification = effective cost / reports that pass automated validation
Build a frozen replay set before comparing anything. Keep the original report text, expected routing tier, allowed labels, and a human-reviewed acceptance outcome. Include short complaints, long email threads, multilingual text, copied lease clauses, empty submissions, and adversarial instructions embedded in a tenant message. Strip personal data or replace it with stable synthetic values before it reaches a test environment. The same 500-report set should run through every candidate under the same concurrency, timeout, maximum-output, and retry policy. Five hundred is an experiment size, not a universal minimum; use enough cases to cover the queues and languages your reviewers actually see.
Don't average everything into one pretty number. Report input tokens, output tokens, accepted classifications, schema failures, retry count, p50 and p95 latency, and human-review escalation rate by category. A low token cost can lose once verbose rationales, repair calls, or false escalation flood the review queue. Conversely, a higher per-token model can win if it produces short valid output on the first call.
This is where a static pricing page stops helping.
For a current shortlist, OpenRouter, Portkey, and LiteLLM represent different operating boundaries rather than a podium. OpenRouter documents a managed unified API and provider routing. Portkey documents an AI gateway with hosted and self-hosted deployment paths. LiteLLM documents a self-hosted proxy with an OpenAI-compatible interface. Those boundaries affect who operates the gateway and how much contract translation the application owns; they do not establish which option is cheapest for a particular report mix. Your mileage may vary, especially when prompts contain long property histories or the reviewer rubric changes.
Portability starts with the output contract
A single credential is convenient. It is not portability.
The portable asset is the application contract: a versioned input, a narrow output schema, normalized usage fields, explicit timeout rules, and errors your queue can act on. Keep provider model IDs and router-specific options in deployment configuration. Keep them out of domain code and persisted moderation records. If a report record says model: fast-cheap-vendor-x, a provider swap has already leaked into the business layer.
Structured output is worth enforcing because moderation labels feed a workflow, not a chat window. The OpenAI Structured Outputs guide distinguishes schema adherence from merely producing valid JSON. That distinction generalizes: JSON parsing answers "is this syntax?" while schema validation answers "can the review system safely use it?" Require additionalProperties: false, bound strings, enumerate labels, and reject confidence values outside the declared range. Preserve the raw response in restricted telemetry only when policy permits it; the queue should consume the normalized object.
Streaming deserves separate treatment. Server-Sent Events use the text/event-stream media type, and MDN notes the browser connection limit concern when SSE is not used over HTTP/2. Moderation jobs usually benefit more from a complete validated object than token-by-token display, so don't stream by default. If an operator UI needs progress, stream job state from your own service and validate the model result before publishing the terminal state.
Errors need the same discipline. Normalize timeout, rate-limit, authentication, invalid-output, and policy-rejection classes. Record the original provider status in protected diagnostics, but let retry policy depend on the normalized class. Authentication errors should stop. Rate limits may move to another allowed target. Invalid output may get one bounded repair attempt, after which the report goes to human review. No infinite retries.
The catch is real: a strict common contract can hide provider-specific capabilities. If a moderation workflow depends on a unique feature that cannot be represented without flattening its meaning, stick with a provider-native integration for that path and isolate it behind an explicit capability interface. Portability is not worth silent semantic loss.
A small TypeScript boundary beats a large configuration file
The application needs one function and a boring result type. The router can change behind it. This example deliberately excludes provider-specific request fields and validates the returned shape before the moderation queue sees it.
const labels = ["harassment", "fraud", "safety", "noise", "other"] as const;
type Label = (typeof labels)[number];
type Classification = {
label: Label;
confidence: number;
rationale: string;
traceId: string;
};
type Usage = {
inputTokens: number;
outputTokens: number;
};
type RuntimeResult = {
classification: Classification;
usage: Usage;
latencyMs: number;
};
type RuntimeRequest = {
reportId: string;
text: string;
schemaVersion: "moderation.v1";
signal?: AbortSignal;
};
interface ModerationRuntime {
classify(request: RuntimeRequest): Promise<RuntimeResult>;
}
function isClassification(value: unknown): value is Classification {
if (typeof value !== "object" || value === null) return false;
const item = value as Record<string, unknown>;
return (
typeof item.label === "string" &&
labels.includes(item.label as Label) &&
typeof item.confidence === "number" &&
item.confidence >= 0 &&
item.confidence <= 1 &&
typeof item.rationale === "string" &&
item.rationale.length <= 280 &&
typeof item.traceId === "string"
);
}
async function classifyForReview(
runtime: ModerationRuntime,
request: RuntimeRequest,
): Promise<RuntimeResult> {
const result = await runtime.classify(request);
if (!isClassification(result.classification)) {
throw new Error("INVALID_MODERATION_OUTPUT");
}
return result;
}
The adapter that implements ModerationRuntime can call a managed router, a self-hosted gateway, or a provider directly. Its tests should use recorded request and response fixtures with secrets removed. Contract tests then run against each enabled target in a non-production project and assert the same normalized behavior. This is less glamorous than a routing DSL. Good. Every knob creates another state to benchmark, document, and debug.
For deployment, pin the contract version and roll routing changes independently of application releases. Send a small percentage of sanitized replay traffic to a candidate, compare it with the current target, then promote only after the acceptance and latency thresholds hold. Production reports should not become an undeclared experiment. If live evaluation is permitted, require an audit trail and never let shadow output alter case priority.
Observability should join the application trace to the gateway request without storing the API key or raw tenant text in ordinary logs. Useful fields include report ID, contract version, target alias, attempt number, normalized outcome, input and output tokens, latency, and routing-policy version. Keep cardinality under control: trace IDs belong in traces or logs, not metric labels. Dashboards should show accepted classifications per dollar and escalation rate beside p95 latency. One number lies. Three numbers start a conversation.
When is the runner-up the better choice?
Use a self-hosted gateway when routing logic is a core capability, the data path must remain inside infrastructure you operate, or an existing platform team can absorb upgrades and on-call work. It is not suitable when one developer would become the permanent gateway operator. The saved glue code can reappear as deployment manifests, dashboards, capacity planning, and security patching.
Use thin direct adapters when only two or three stable model targets matter and native capabilities drive classification quality. This preserves exact provider semantics and removes a gateway dependency. The bill arrives in engineering time: authentication, retries, streaming, schemas, usage normalization, and model migrations must be implemented and tested for every adapter.
A managed router fits when time-to-first-call and a small integration surface dominate. Its limit is control. Policy expressiveness, telemetry export, data handling, model availability, and fallback semantics must be verified against requirements rather than inferred from an "OpenAI-compatible" label. Compatibility describes an interface shape, not identical behavior.
I'm not sure a universal cheapest answer can survive contact with a real moderation queue. The defensible decision is reproducible: freeze the replay set, declare the acceptance rule, calculate cost per accepted classification, and retain an exit test that proves another adapter can pass the same contract. Re-run it whenever the rubric, traffic mix, model, or router policy changes.
Then ship the boring boundary.
Top comments (0)