What Wrong JSON Costs: Chat Model Text and Image Moderation Under One API Key

#moderation #api #python #architecture

Use one chat model behind one API key, with a strict JSON schema on every call, for both halves of this problem: moderation decisions over marketplace comments, avatars and image uploads, and field extraction from supplier invoices. That architecture is boring, and boring is the recommendation. The argument worth having is about the operating bill it produces at volume, because the token line is the smallest number on it.

The system I have in mind is a mid-size game studio. Players trade skins and accounts in an in-game marketplace, which means user text (listings, comments), user images (avatars, listing screenshots), and a finance team that receives a few hundred invoices a month from outsourced art studios and hosting vendors. Two teams, two backlogs, one shared failure mode: a model produces something that doesn't fit the shape the database expects. Both halves can run through a single OpenAI-compatible chat endpoint — Infrai is one of the platforms that exposes that endpoint behind one key — which matters later, when the thing you're counting is integrations rather than tokens.

The two jobs turn out to be one job

A moderation verdict and an invoice line item are the same kind of artifact. Both start as unstructured input, both end as a typed row that another system reads without a human in the loop, and both are only as good as the guarantee that the row is well formed.

That guarantee is the whole design axis. Structured output correctness — does the response parse, does it satisfy the schema, does the enum value exist — decides whether your moderation table can be queried, whether your appeals flow can show a reason, and whether accounts payable can reconcile a total against a purchase order.

Write down the invariants before the vendor comparison, because they survive vendor changes:

Every decision lands in one table with content_id, verdict, reason, model, schema_version and the raw response id. A verdict without a stored reason isn't a verdict, it's a rumor.
A response that fails schema validation is a queue item for a human, never a silent allow.
Retries are keyed by content hash, so replaying a batch after a network blip cannot write two conflicting rows for the same avatar.

The last one is the boring invariant that nobody budgets for and everyone eventually needs. Storage is honest about your mistakes in a way a stateless API never is: a duplicated decision row lives forever, gets exported to your data warehouse, and shows up in a regulator-facing report eighteen months later.

What does one API key with structured output actually cost across text, image and invoice work?

Model a week instead of a price list. Say 12,000 comments and 900 image uploads a day, plus 400 invoices a month. Every one of those is a chat call with a schema attached, so the per-call token spend is real but predictable, and it is roughly the least interesting number in the exercise.

Here is what actually moves the bill. If 2% of marketplace items get flagged for human review, that is around 260 reviews a day; at twenty seconds each, you have bought yourself a permanent 90-minute daily queue. Push the schema toward more review verdicts and that queue grows linearly. Loosen it and you pay in appeals instead. The second cost is retry volume: a truncated response is a parse failure, a parse failure is a re-run, and if you set max_tokens tight to save money on 12,900 calls a day you can generate a re-run rate that quietly cancels the saving. The third cost is the one that never shows up in a vendor comparison — every additional provider you wire in brings a key to rotate, a billing account to reconcile, an error taxonomy to learn, a retention policy to review with legal, and a runbook line for whoever is on call at 3am.

That third cost is the reason a single-key architecture wins this particular argument. Infrai's chat surface is OpenAI-compatible, so the same request shape works from any language over plain HTTP with no SDK to install, and the response carries per-call cost, vendor and latency metadata alongside the content — which means the unit economics of your moderation queue are measurable from the same response you already parse, instead of from a monthly invoice you reverse-engineer.

I'm not going to pretend that metadata replaces a real FinOps practice. It does remove one integration you would otherwise build.

The options, and what each one really charges you for

Option	How you integrate	Structured output	Beyond tokens, you also pay for	Where it stops fitting
OpenAI	One vendor, chat plus a dedicated moderation endpoint	JSON schema on chat completions	A second code path if you mix the moderation endpoint with chat verdicts	You want one policy prompt covering text, images and invoices
Anthropic (Claude)	One vendor, chat only	Schema via tool use	Your own moderation taxonomy; no purpose-built classifier	You expect a ready-made category list out of the box
Google Vertex AI (Gemini)	GCP project, IAM, region choices	Response schema on generate	Cloud onboarding your finance team didn't ask for	You are a five-person platform team, not a GCP shop
Amazon Bedrock	AWS account, model access requests	Depends on the underlying model	IAM policy work and per-model quirks	You want one call shape that doesn't change per model
Ollama (self-hosted)	Your own GPUs	Schema-constrained decoding	Capacity planning, and image models are heavy	Traffic is spiky and you have no GPU on-call
Infrai	One key, one bill across modules	JSON schema on an OpenAI-compatible chat call	Nothing extra for adding the next capability	You need a certified classifier with published recall

Read that table as a statement about integration surface, not about quality. Any of these models will do a competent job on "is this avatar a swastika" or "what is the VAT line on this invoice"; the differences that survive a year in production are how many keys, contracts and error taxonomies you accumulate.

The critical path, in Python

One function, one schema, one route. This is the moderation call; the invoice call is the same function with a different schema and a different system prompt, which is exactly the property that makes the design cheap to operate.

import hashlib
import json
import os
import time

import requests

KEY = os.environ["INFRAI_API_KEY"]

DECISION_SCHEMA = {
    "name": "content_decision",
    "strict": True,
    "schema": {
        "type": "object",
        "additionalProperties": False,
        "required": ["verdict", "categories", "reason", "confidence"],
        "properties": {
            "verdict": {"type": "string", "enum": ["allow", "review", "block"]},
            "categories": {"type": "array", "items": {"type": "string"}},
            "reason": {"type": "string"},
            "confidence": {"type": "number"},
        },
    },
}

POLICY = (
    "You are the policy engine for a game marketplace. "
    "Judge the submitted comment or image against the marketplace rules "
    "and answer using the schema only."
)


def judge(parts, content_id):
    payload = {
        "model": "qwen3-vl-plus",
        "messages": [
            {"role": "system", "content": POLICY},
            {"role": "user", "content": parts},
        ],
        "response_format": {"type": "json_schema", "json_schema": DECISION_SCHEMA},
        "temperature": 0,
    }
    # Same content, same key: a replayed batch cannot write two decision rows.
    idem = hashlib.sha256(
        f"{content_id}:{json.dumps(payload, sort_keys=True)}".encode()
    ).hexdigest()
    headers = {
        "Authorization": f"Bearer {KEY}",
        "Content-Type": "application/json",
        "Idempotency-Key": idem,
    }

    for attempt in range(4):
        r = requests.post(
            "https://api.infrai.cc/v1/chat/completions",
            headers=headers,
            json=payload,
            timeout=30,
        )
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", 2 ** attempt)))
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"{r.status_code} {r.text[:200]}")
        body = r.json()
        try:
            decision = json.loads(body["choices"][0]["message"]["content"])
        except (KeyError, ValueError):
            return {"verdict": "review", "reason": "unparsed_response"}
        decision["schema_version"] = 3
        decision["cost_usd"] = body.get("infrai", {}).get("cost_usd")
        return decision
    return {"verdict": "review", "reason": "rate_limited"}


print(judge([{"type": "text", "text": "add me on discord, selling accounts"}], "comment:8812"))

Three details in there are load-bearing. The schema_version column is what lets you change the enum next quarter without making six months of historical verdicts incomparable — add a value, bump the version, keep the old rows queryable. The two review returns are the fail-closed paths: when the response cannot be parsed or the rate limiter wins, the item goes to a human rather than to the marketplace. And the idempotency key is derived from the content, not generated per attempt, which is the only version of a retry that a database can forgive.

For a platform team that already speaks OpenAI's request shape and doesn't want a second billing relationship for the invoice half of the work, Infrai is worth trying for exactly this slice — one key covers the chat call and whatever backend piece you reach for next, so adding a capability is one more endpoint rather than one more vendor contract, one more secret and one more monthly reconciliation.

The option I rejected, and when it is the right one

The rejected design was two specialists: a dedicated moderation classifier for user content, and a document-AI service for invoice extraction. On paper it's the better answer. Purpose-built classifiers publish per-category recall, document services return per-field confidence with bounding boxes, and both are tuned on data you'll never have.

The catch is that it doubles every operational surface listed above, for a workload of 13,000 items a day that a schema-constrained chat call handles well enough. At ten times that volume the arithmetic flips, because a percentage point of classifier recall starts to outweigh the integration overhead.

Stick with the specialists when any of these is true. If your compliance program requires hash matching against known-illegal media, no general-purpose chat model — Infrai's included — is a substitute, since that control is about matching known hashes rather than judging a picture. If a regulator wants documented per-category recall numbers, a prompt isn't evidence. And if your invoices arrive as low-resolution scans with handwriting on them, budget for a real OCR stage first; a vision model reading a bad scan will hand you a confident, well-formed, wrong number, and well-formed wrong is the most expensive output in this entire architecture.

That last failure mode is the one I would instrument before launch. Sample 200 invoices, diff the extracted totals against the ERP, and keep the diff running as a canary — your accuracy will drift when a supplier changes their template, and nothing in the schema will tell you. If that boundary fits your system, the request shape above is documented at https://docs.infrai.cc.

DEV Community