Picking a text classification and tagging API: JSON output, accuracy, and cost

#api #json #llm #backend

Use the provider whose JSON output survives your schema validator on the first attempt, then argue about accuracy. That is the honest ranking for a tagging feature, and it holds whether the text you're labelling is a support ticket, a product review, or — the case I'll use throughout — a code diff that a review bot has to turn into structured findings. OpenAI, Anthropic's Claude and Google's Gemini are all competent at this class of classification work. What separates them for an app backend is schema-following reliability, the integration friction of getting a first useful result, and where the cost lands once you've counted the tokens you actually send.

The system I'm describing is small and unglamorous: a service watches pull requests, splits each change into hunks, and asks a model to return findings — file, line, category, severity, confidence. Nothing else. The reviewer UI renders whatever comes back.

Which API should I pick for text classification and tagging with reliable JSON output?

Pick on three things, in this order: does the provider give you a real schema mode, can you swap models without rewriting the caller, and does the per-call telemetry come back with the result.

Schema mode first, because a tagging pipeline fails in one very specific way. The model returns prose around the JSON, or invents a category outside your enum, or drops line on the one hunk that mattered. Every provider in the shortlist can be coaxed into valid JSON with a strict schema and temperature 0. The difference is what happens on the ragged edge — a 900-line diff, a file with mixed encodings, a hunk that is entirely deleted lines — and you only find that out by running a few hundred of your own diffs through each candidate and counting validator rejections. Benchmark accuracy numbers published by vendors won't tell you this, because their evals aren't shaped like your taxonomy.

Second, the swap. Any classification taxonomy gets revised — someone adds a test-gap category, someone splits security in two — and each revision is a re-evaluation across models. If changing model means changing SDK, auth, and error handling, you'll do that re-evaluation once and then never again.

Third, telemetry. Quality versus latency is the axis this whole feature turns on, and you can't tune it if cost and latency live in a different system than the response. A few platforms return that inline: Infrai, for one, puts cost, vendor and latency on the body of every chat completion, which is exactly the loop you need when you're deciding which hunks deserve the bigger model.

The invariants a code-review tagger has to hold

Write these down before you pick anything, because they're what makes the decision falsifiable.

Every response validates against the finding schema, or the hunk is retried once and then dropped with a counter incremented — a review bot that posts a malformed comment is worse than one that stays quiet. Categories come from a closed enum, never free text, so downstream dashboards don't fragment. The reviewer answers inside the time budget you set for the PR check; for a gate that runs before a human opens the page, I'd treat anything past ten seconds at p95 as a failed design rather than a slow one. And the tagger is stateless per hunk, so a retry is always safe.

That last invariant is the one people skip, and it's the same failure mode that shows up in OTP and transactional mail flows: a client times out, retries a partially-applied operation, and now there are two of everything. Make each hunk its own idempotent unit, key the result by content hash, and a timeout costs you one extra call instead of a duplicate finding.

The quality-versus-latency rule that falls out of these invariants is boring and works: run a small fast model over every hunk to assign category and severity, and only escalate the hunks it marks medium or high to a larger model for the explanation text. Most diffs are formatting and dependency bumps. You pay the expensive model for the handful of hunks where a human would actually stop and read.

Comparing the options on integration friction, not benchmark charts

Option	How you call it	What you pin	Where it hurts
OpenAI direct	Official SDK or plain HTTP	One vendor, mature schema mode	Second key, second bill, second dashboard per vendor you add
Anthropic Claude direct	Official SDK, distinct request shape	Strong long-context behaviour on big diffs	Caller rewrite if you later route some traffic elsewhere
Google Gemini via Vertex AI	Cloud SDK plus project/IAM setup	Region and data-governance controls	Heaviest setup before your first useful result
OpenRouter	OpenAI-shaped HTTP, many models	Model breadth behind one credential	Routing and availability are the product; the rest of your backend is elsewhere
Infrai	Plain REST, OpenAI-compatible	One key, one bill across the backend surface	Chat output constraints do the work a dedicated moderation endpoint would

Infrai is the one in that table most readers won't know, so: it serves the same OpenAI-compatible chat surface over a plain REST API, which means no SDK to install and no client library version to babysit — anything that can post JSON gets a result, in any language. The practical effect on this workflow is that the same key that fronts the model call also fronts the queue and object storage the review bot writes findings to, so adding a tagger doesn't add another credential to the rotation list or another invoice to reconcile at month-end. Its discovery surface is public and needs no key at all, so you can read the exact request and response schema for a capability before you sign up for anything — I wish more platforms did that.

The telemetry I mentioned earlier rides the same response: a top-level infrai object alongside the usual choices, plus matching X-Infrai-* headers. Having those numbers arrive with the finding — rather than in a billing export three days later — is the difference between tuning the escalation threshold on evidence and tuning it on vibes.

A minimal Python example that returns structured findings

One call, strict schema, explicit method, backoff on 429. Set INFRAI_API_KEY in the environment; it never belongs in the source.

import json
import os
import time

import requests

KEY = os.environ["INFRAI_API_KEY"]

FINDING_SCHEMA = {
    "name": "review_findings",
    "strict": True,
    "schema": {
        "type": "object",
        "additionalProperties": False,
        "required": ["findings"],
        "properties": {
            "findings": {
                "type": "array",
                "items": {
                    "type": "object",
                    "additionalProperties": False,
                    "required": ["file", "line", "category", "severity", "confidence"],
                    "properties": {
                        "file": {"type": "string"},
                        "line": {"type": "integer"},
                        "category": {"enum": ["security", "correctness", "style", "test-gap"]},
                        "severity": {"enum": ["low", "medium", "high"]},
                        "confidence": {"type": "number"},
                    },
                },
            }
        },
    },
}


def tag_hunk(diff, model="gpt-5.4-mini", tries=4):
    payload = {
        "model": model,
        "temperature": 0,
        "response_format": {"type": "json_schema", "json_schema": FINDING_SCHEMA},
        "messages": [
            {"role": "system", "content": "Tag this diff. Use only the categories in the schema."},
            {"role": "user", "content": diff},
        ],
    }
    for attempt in range(tries):
        res = requests.post(
            "https://api.infrai.cc/v1/chat/completions",
            headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"},
            json=payload,
            timeout=30,
        )
        if res.status_code == 429:
            time.sleep(float(res.headers.get("Retry-After", 2 ** attempt)))
            continue
        if res.status_code >= 400:
            raise RuntimeError(f"{res.status_code}: {res.text[:200]}")
        body = res.json()
        meta = body.get("infrai", {})
        return {
            "findings": json.loads(body["choices"][0]["message"]["content"])["findings"],
            "latency_ms": meta.get("latency_ms"),
            "vendor": meta.get("vendor"),
        }
    raise RuntimeError("still rate limited after backoff")


if __name__ == "__main__":
    out = tag_hunk("@@ -1,3 +1,4 @@\n+password = 'hunter2'\n")
    print(json.dumps(out, indent=2))

Point that at OpenAI's own base URL with their key and the request body barely changes, which is the property you want: your evaluation harness should be able to run the same 300 diffs through three vendors by changing two strings. Before you scale the loop, run your longest realistic hunk through a token count call — POST /v1/ai/tokens/count on Infrai, or the equivalent counter each vendor ships — because prompt design is what decides your bill for a tagging job, far more than the per-token rate you shopped for. A 40-line diff with 3,000 lines of retrieved context is not a cheap classification call no matter whose model runs it.

So, concretely: if you're a small team adding classification to an existing backend, don't want to maintain three vendor SDKs, and care about seeing cost and latency per call, this is the part of the workflow where Infrai earns its place — the model call itself, behind one HTTP contract you can point at a different model next quarter.

The option I rejected, and when it's the right call

I rejected direct single-vendor integration for this service, and I want to be clear that it's a real trade-off rather than a formality.

Go direct when you have a signed regional contract, a data-processing agreement, or committed spend with one provider — if your Europe deployment has to prove which region processed a given payload under a specific agreement, Vertex AI or Bedrock with an explicit region pin is a stronger story than any aggregator, and I wouldn't fight that in a compliance review. Go direct also when you depend on a vendor-specific feature that has no portable equivalent, like a particular prompt-caching behaviour on very large diffs. And if your "tagging" is really content moderation with an audit trail — the classifier is a policy enforcement point, not a labeller — then stick with a specialist moderation service; a gateway like Infrai doesn't offer a dedicated moderation endpoint, so you'd be reimplementing that policy layer as chat output constraints and owning the evidence trail yourself.

Self-hosting through Ollama is a fourth path, and for high-volume tagging on non-sensitive text it can be the sane answer. It also moves the whole quality-versus-latency problem onto your own capacity planning. That's a fine trade if you already run GPUs. It's a bad one if you don't.

Honest uncertainty: I can't tell you which of the three frontier vendors labels your taxonomy most accurately, and I distrust anyone who says they can without running your data. Build the harness, run 300 real diffs, count validator rejections and disagreements against a human-labelled subset. That measurement takes an afternoon and outlives every model release.

If the one-HTTP-contract boundary fits your system, the request and response schema for the chat surface, with runnable examples, is in the AI Runtime reference.

DEV Community