For an in-app property-management chatbot, choosing OpenRouter or going direct to OpenAI, Anthropic, or Gemini should begin with one constraint: a provider switch must never change which moderation reports reach human review. The least complex design that meets it is an application-owned classifier contract, a durable review queue, and one bounded retry policy outside the model client.
Short answer: use a routing service when rapid model substitution and one billing surface matter more than provider-specific controls; integrate directly when those controls, contractual terms, or per-provider observability are requirements. Neither route is inherently cheapest or easiest. Measure the cost of accepted classifications, including retries and human rework, and keep the moderation decision in your own system.
The model suggests a label. It does not close the case.
What should an in-app chatbot compare across billing, retries, and rate limits?
Start with the unit of work, not the API request. Here, that unit is one moderation report safely placed into a human-review lane. A response that consumed tokens but failed schema validation costs money without completing the work. A cheap response that sends harassment reports to an ordinary queue can be much more expensive operationally than a pricier response that consistently satisfies the contract.
For this workload, compare a routing service and direct provider integrations on the same dimensions:
| Decision dimension | Routing service | Direct integrations |
|---|---|---|
| Provider substitution | Usually centralized behind one upstream contract | Implemented and tested in the application adapter |
| Billing operations | One intermediary account can reduce invoice reconciliation | Separate accounts preserve direct attribution and contract control |
| Retry ownership | May exist at more than one layer, so attempt accounting matters | Easier to keep a single application retry budget |
| Rate-limit handling | One gateway can normalize some differences | Each adapter must interpret its provider's limits and headers |
| Feature access | Common-denominator interfaces favor portability | Provider-specific features are available without waiting for normalization |
| Failure isolation | An intermediary becomes another dependency | More client code and credentials must be operated |
This table is a map of ownership, not a universal feature promise. Exact limits, headers, model availability, data terms, and fallback behavior vary by account and can change. Check the current documentation and the agreement attached to the account before treating any cell as guaranteed.
The catch is real. A routing layer is not suitable when legal review requires a direct processor relationship, when a provider-native moderation control is part of the safety case, or when the team needs raw provider telemetry that the layer does not retain. Direct integrations are a poor fit when a small team cannot keep several adapters, credentials, invoices, and limit policies tested. I'm not sure which side wins for a given company until those organizational constraints are written down; a short production-like trial resolves more than a feature checklist.
Make the classification contract boring
Provider portability lives in the response contract. Keep it deliberately smaller than any one model's feature set. A property report might contain resident text, an attachment reference, a building identifier, and a consent-safe correlation ID. The classifier should return only the routing fields required by the review operation: a constrained category, a confidence value, a review priority, and terse reasons that staff can inspect.
Do not send more tenant data than the classification needs. Names, phone numbers, email addresses, apartment access details, and free-form maintenance notes can create privacy and compliance exposure without improving the route decision. Redact before the provider boundary, retain the original report under the property's access policy, and give the model a pseudonymous report ID. This is the same discipline that keeps OTP and notification pipelines defensible: delivery metadata is useful; unrelated personal content is not.
The adapter below is intentionally plain Python. Each provider-specific client implements classify, while validation and review routing remain application code.
from dataclasses import dataclass
from typing import Literal, Protocol
Category = Literal["harassment", "safety", "spam", "other"]
Priority = Literal["urgent", "standard"]
@dataclass(frozen=True)
class Classification:
category: Category
priority: Priority
confidence: float
reasons: tuple[str, ...]
class Classifier(Protocol):
def classify(self, report_id: str, redacted_text: str) -> Classification:
...
def validate(result: Classification) -> Classification:
if not 0.0 <= result.confidence <= 1.0:
raise ValueError("confidence must be between 0 and 1")
if not result.reasons or len(result.reasons) > 3:
raise ValueError("provide between one and three reasons")
return result
def choose_lane(result: Classification) -> str:
if result.category == "safety" or result.priority == "urgent":
return "priority-human-review"
if result.confidence < 0.80:
return "uncertain-human-review"
return "standard-human-review"
The 0.80 threshold is an example policy value, not a quality claim or benchmark. Calibrate it against labeled reports approved for that property portfolio, then version it independently of prompts and models. Also test the uncomfortable inputs: an empty report, mixed languages, quoted abusive text, a resident reporting somebody else's threat, and a long pasted email thread. Edge cases decide whether a moderation workflow is safe.
Structured output can still be syntactically valid and operationally wrong. A category spelled correctly does not prove that the report belongs there. Maintain a small, access-controlled evaluation set with expected lanes, including ambiguous cases where the only acceptable result is human review. Compare candidate adapters on lane agreement, abstention behavior, latency distribution, and completed-work cost. Don't turn a single aggregate accuracy score into a release gate.
Put retries around work, not chat completions
Retries are where an apparently easy integration becomes a duplicate-work problem. Assign an idempotency key to the classification job, not to an individual HTTP attempt. Persist the job before calling a model, record each attempt under that job, and commit only one accepted classification. If a worker loses its lease after receiving a valid answer, another worker can observe the accepted result instead of purchasing and applying the decision twice.
No report vanishes.
Be conservative.
Retry only failures that the active provider documents as transient, respect an explicit retry delay when one is supplied, add jitter, and cap both attempts and elapsed time. Authentication failures, invalid requests, exhausted account budgets, and schema violations need different handling; blindly retrying them increases load and hides the actionable cause. Because routing services can also perform retries or fallbacks, capture the upstream request identifier and any disclosed attempt metadata. Otherwise one application attempt may represent multiple billable upstream attempts.
A moderation queue also needs a deadline policy. If classification is unavailable before the deadline, send the report to an unclassified human-review lane. Never drop it, and never let exponential backoff hold a safety report outside staff visibility. This fallback is deliberately less efficient and more trustworthy.
Rate limits deserve two controls. A local token bucket protects each credential from bursts, while queue backpressure protects the application when arrival rate exceeds processing capacity. Separate limits by workload: resident-facing chat, background report classification, and evaluation runs should not consume one undifferentiated concurrency pool. Emergency or safety reports can receive queue priority without pretending that the model provider offers infinite capacity.
Audit cost per accepted report
Sticker price cannot answer “cheapest” because the billable unit and the operational outcome differ. Record input and output usage returned by the active integration, the model and provider identifiers, attempt count, latency, validation outcome, final review lane, and whether a human changed the label. Keep money calculations in a versioned rate table rather than embedding mutable prices in prompts or business logic.
Then calculate cost per accepted report and cost per correctly routed report on a stable evaluation set. The second number includes invalid responses, retries, and human corrections. It exposes a common trap — an inexpensive first attempt can lose once repair attempts and review load are counted — without claiming that any provider is always cheaper.
Billing records must reconcile with provider or routing-service usage exports. Use an immutable internal job ID to join them, but don't put resident content in invoice metadata. Alert on attempt amplification, sudden output growth, missing usage fields, and a rising share of uncertain-review decisions. Those are useful signals even when the API remains available.
Direct accounts make provider-level invoices and contractual attribution explicit, but finance must reconcile several billing systems. A routing account can consolidate that work, while requiring the team to understand how the intermediary reports upstream usage and fallback attempts. Choose the accounting boundary your team can actually audit.
Compare adapters with a shadow rollout
Build one conformance suite before choosing the runtime path. Run every adapter against the same redacted fixtures and assert schema behavior, timeout handling, cancellation, usage capture, and the rule that no report disappears. Provider-specific tests can exist below that suite, but application code should depend only on the narrow classifier contract.
Roll out in three steps. First, replay an approved, de-identified evaluation set without changing queues. Next, shadow a small sample of eligible live traffic under the applicable consent and retention policy; store candidate decisions separately and restrict access. Finally, enable the candidate for a bounded cohort with an immediate switch back to the previous adapter. Migration means changing adapter configuration, not rewriting review logic.
Watch disagreement by category rather than only overall agreement. A candidate that differs mostly on spam is a different operational risk from one that differs on safety reports. Sample false negatives for human review, define who can halt the rollout, and keep the old adapter deployable until reconciliation and audit logs are complete.
The decision rule stays compact: prefer the routing path when normalized substitution and consolidated operations remove more work than the intermediary adds; prefer direct integrations when native controls, direct governance, or precise provider observability are mandatory. In both cases, the durable queue, validation policy, audit log, and human authority belong to the property-management application. That is what makes a later move routine rather than a moderation incident.
Top comments (0)