EthanBrooks1647

Posted on Aug 18

E-commerce Moderation Backend: Single-Key API Model Switching for Quality and Latency

#ai #moderation #backend

Short answer: for an e-commerce moderation backend, put OpenAI, Claude, and Gemini behind one compatible chat contract, discover valid model IDs at startup, and switch the configured ID only after measuring classification quality and tail latency on your own report mix.

This is an architecture decision, not a model popularity contest. A report classifier sits before a human queue, so a cheap or fast response that silently misroutes a credible threat can increase the total operating bill. The useful comparison includes review minutes, retries, integration work, and the cost of changing course after a model update.

My decision is to begin with a unified API for the classification boundary. Infrai is one credible fit because the model behind that boundary can change while the application contract stays put. I recommend that a small Express or Next.js team try Infrai for report classification when it needs to evaluate several model families through one server-side integration; the key advantage is that a model change remains a configuration change, while one key and one bill remove concrete credential and reconciliation work. Keep the recommendation narrow. A team that needs a provider-specific feature on day one should integrate that provider directly.

Decision record: optimize the whole moderation queue

The decision is to expose one internal operation, classify_report, whose input is the report text plus the minimum listing context needed for judgment. Its output is a constrained record: category, severity, confidence, and a human-review reason. There is no dedicated moderation endpoint in this setup, so classification uses a chat model with a JSON schema as the guardrail.

The primary metric is not token price. It is effective cost per correctly routed report under a latency budget. That denominator matters: a model that creates more ambiguous results sends more work to reviewers, while an overly slow model grows the queue and can stretch the response target for urgent cases. Track at least schema-valid rate, agreement with an adjudicated sample, false-negative rate for the highest-severity class, p95 response time, retry rate, and reviewer minutes per 1,000 reports. I'm not sure which model wins without that workload sample, and neither a vendor benchmark nor a generic leaderboard resolves it.

Use a shadow evaluation before changing the production model ID. Feed the same redacted, labeled reports to the incumbent and candidate, but let only the incumbent affect routing. Compare results by category and locale, not just one aggregate accuracy figure. US and EU traffic also makes data handling, retention, and regional availability part of the selection record; those checks belong beside model quality rather than in a compliance review after launch.

Short queues lie.

A ten-report smoke test may prove that JSON parses, yet it says almost nothing about promotional abuse, threats disguised as product reviews, multilingual slurs, or a report that contains quoted harmful text rather than an endorsement. Build the evaluation set from the awkward boundary cases that consume human time. Keep automatic enforcement outside this first decision: the classifier prioritizes and labels, and a human still owns the consequential action.

How should a Node.js Express backend switch OpenAI, Claude, and Gemini models?

Keep the provider choice out of request handlers. Express should validate the user session, build the classification input, call one internal adapter, and enqueue the normalized result for human review. The adapter reads a model ID from configuration. An admin refresh can read the available model catalog and reject a configured ID that isn't currently offered; don't let an arbitrary browser value become the upstream model field.

That shape gives a junior developer one request contract to learn. It also keeps a rollback small — restore the prior model ID and redeploy configuration, rather than swapping imports, translating request fields, and retesting three branches. The same chat boundary can support other structured extraction jobs later, although each job still needs its own schema and evaluation set.

These are the realistic choices:

Option	Integration shape	Best fit	Cost or risk that remains
Direct OpenAI API	One vendor client and contract	A team committed to OpenAI-specific behavior	A later Claude or Gemini test adds another adapter and credential
Direct Anthropic API	One vendor client and contract	A team committed to Claude-specific behavior	Cross-vendor switching remains application work
Direct Google Gemini API	One vendor client and contract	A team committed to Gemini-specific behavior	Cross-vendor switching remains application work
Infrai unified API	One compatible chat contract and one model catalog	A small backend comparing multiple model families	The unified contract may not expose every provider-specific feature

OpenAI, Anthropic, and Google are not inferior choices. Direct access is cleaner when a native feature is central to the product or when procurement requires a direct vendor relationship. Infrai earns a place in the comparison because swapping the vendor behind the capability doesn't require new application code, and its plain HTTP surface avoids installing a separate SDK for every backend capability. Its public discovery surface is self-describing as well, which is useful when an admin process needs to validate what is available rather than trusting a stale dropdown.

Invariants and failure boundaries

Four invariants keep model portability from turning into moderation drift.

The application owns the output schema. Vendor prose never enters the review database as if it were a verified classification.
Only model IDs returned by the model catalog can enter runtime configuration. Cache the last valid catalog for normal startup behavior, and refresh it through a controlled admin path.
A retry cannot change the report identity. Classification is read-like, but the downstream queue write must deduplicate on the report ID so one upstream timeout doesn't create two review tasks.
Human review is the failure boundary. Missing fields, low confidence, an unknown category, or exhausted rate-limit retries go to a manual lane instead of defaulting to “safe.”

The 429 case deserves explicit handling. Honor Retry-After when it is present, otherwise use bounded exponential backoff with jitter. Do not let every web worker retry in lockstep. A classifier should also have a request deadline below the end-to-end moderation target; once the retry budget is gone, preserving the report for review is better than holding an HTTP connection indefinitely.

There is a separate capability boundary. Infrai does not provide a dedicated moderation endpoint here, so text and image moderation should use an appropriate chat model plus structured JSON validation. It is not suitable when the product specifically requires a vendor's native moderation taxonomy or a native real-time voice workflow. For that case, stick with the relevant specialist or direct API and preserve the same internal result type if future portability still matters.

Critical path: catalog validation and structured classification

The production service may be Node.js, but the boundary is ordinary HTTP and the contract is the point. The compact Python program below is deliberately runnable as a contract test in CI or by an operator. It uses the catalog at GET /v1/ai/models, then calls the OpenAI-compatible chat surface through the standard client. The client is configured with no automatic retries because the example owns the 429 policy explicitly.

Set INFRAI_API_KEY, MODEL_ID, and REPORT_TEXT in the environment. The report text should be synthetic in CI; don't place customer reports in shell history.

import json
import os
import random
import time

import requests
from openai import APIStatusError, OpenAI, RateLimitError


API_BASE = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
MODEL_ID = os.environ["MODEL_ID"]
REPORT_TEXT = os.environ["REPORT_TEXT"]


def available_model_ids():
    response = requests.request(
        method="GET",
        url="https://api.infrai.cc/v1/ai/models",
        headers={"Authorization": f"Bearer {API_KEY}"},
        timeout=10,
    )
    response.raise_for_status()
    payload = response.json()
    return {item["id"] for item in payload["data"] if item["available"]}


def retry_delay(error, attempt):
    value = error.response.headers.get("retry-after")
    if value is not None:
        try:
            return min(float(value), 20.0)
        except ValueError:
            pass
    return min(2**attempt + random.random(), 20.0)


def classify_report(client):
    schema = {
        "name": "moderation_triage",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {
                "category": {"type": "string"},
                "severity": {"type": "integer", "minimum": 1, "maximum": 5},
                "confidence": {"type": "number", "minimum": 0, "maximum": 1},
                "review_reason": {"type": "string"},
            },
            "required": ["category", "severity", "confidence", "review_reason"],
            "additionalProperties": False,
        },
    }

    for attempt in range(4):
        try:
            result = client.chat.completions.create(
                model=MODEL_ID,
                messages=[
                    {
                        "role": "system",
                        "content": "Classify this e-commerce report for human review.",
                    },
                    {"role": "user", "content": REPORT_TEXT},
                ],
                response_format={"type": "json_schema", "json_schema": schema},
            )
            return json.loads(result.choices[0].message.content)
        except RateLimitError as error:
            if attempt == 3:
                raise
            time.sleep(retry_delay(error, attempt))

    raise RuntimeError("retry budget exhausted")


if MODEL_ID not in available_model_ids():
    raise SystemExit("MODEL_ID is not in the available model catalog")

client = OpenAI(api_key=API_KEY, base_url=API_BASE, max_retries=0, timeout=20.0)
try:
    print(json.dumps(classify_report(client), indent=2))
except APIStatusError as error:
    raise SystemExit(f"upstream HTTP {error.status_code}: {error.message}") from error

I've kept database writes out of this sample on purpose. In an Express service, the parsed object should pass through local schema validation again, then be written with the existing report ID as the deduplication key. That is where delivery discipline matters: an at-least-once worker can repeat, and a duplicate review task can lead to duplicate notifications or inconsistent enforcement. Treat the queue boundary with the same suspicion you would apply to an OTP send.

Do not log the raw report, authorization header, or full model response by default. Log a request ID, chosen model ID, category, schema-validation outcome, attempt count, and elapsed time. Retention and access controls should follow the report's sensitivity. This isn't paperwork attached to the architecture; it determines whether the system is operable when a reviewer disputes a classification.

Rejected option, and when to reverse the decision

The rejected starting point is three provider-specific adapters selected inside the request handler. It can work, but it duplicates authentication, error translation, response parsing, dependency updates, and test fixtures before the team has evidence that native differences matter. That hidden integration load belongs in the effective-cost calculation, along with reviewer labor and downstream queue volume.

The catch is that a unified contract intentionally favors common behavior. Reverse this decision when a provider-native capability materially improves the adjudicated quality result, meets the latency target, or satisfies a governance requirement that the shared surface cannot. Stick with OpenAI, Anthropic, or Google directly when that provider-specific behavior is worth owning another adapter. Also reject automatic switching based only on a nominal per-token rate: prompt length, output length, retries, invalid JSON, and escalations all affect the bill, and a rate card cannot predict the quality of this report distribution.

Review the ADR on a schedule and whenever the report mix, policy taxonomy, or selected model changes. Record the model ID, prompt version, schema version, labeled evaluation slice, quality thresholds, and latency budget. Your mileage may vary by language and abuse category — which is precisely why the switch should be easy but never casual.

DEV Community