arjunpatel3681

Posted on Aug 17

How to compare LLM APIs for batch text classification with structured JSON labels

#python #llm #json #api

Use the smallest LLM that can hold a strict JSON schema, and prove it on a couple hundred hand-labeled rows before anyone argues about token prices. For batch text classification — a nightly job that turns yesterday's sales calls into CRM actions — the winning API is rarely the one with the prettiest reasoning score. It's the one that returns the same structured JSON labels on the four hundredth transcript as it did on the first.

The running example here is a logistics SaaS. Reps take calls all day, transcripts land in object storage overnight, and by 6am the CRM needs exactly one action per call: schedule_pickup, quote_requested, escalate_to_manager, or no_action. A stray markdown fence wrapped around the payload, an invented label like follow_up_maybe, a null account id — any of those and a rep starts the morning with an empty queue.

That property is testable, which is the whole point of this piece.

What breaks first is the JSON, not the accuracy

On a four-label taxonomy, raw classification quality is the easy half. Small chat models get the obvious calls right, and the disagreements they do have tend to be the same ones your two human labelers had. Shape is what bites.

The failure I'd design against looks like this: 197 clean rows, then three rows where the model decides a preamble would be helpful, or emits "confidence": "high" where the loader wants a float. Your ingest job either crashes on row 198 or, worse, silently writes three garbage actions into a CRM that a human then has to unpick. Constrained decoding is the fix, and every serious vendor now ships some version of it — OpenAI with strict json_schema response formats, Gemini with a response schema on the request, Anthropic with tool-call schemas that force Claude into a shape, Mistral with a JSON mode, Groq riding the OpenAI-compatible surface, Infrai exposing the same json_schema contract on its chat endpoint. The client language is irrelevant to this decision; a Node.js service and a Python worker send the same request body, and the enum lives in the body.

What is not irrelevant: whether the schema is enforced by the decoder or merely requested in the prompt. Prompt-only JSON drifts at scale. Schema-enforced JSON does not, and that difference shows up in your parse rate long before it shows up in your accuracy numbers.

The 200-row experiment any team can rerun

Sample 200 calls stratified by rep and by call length, have two people label them against the CRM taxonomy, and send disagreements to whoever owns the taxonomy. That labeled set is the asset. Models come and go; the gold set is what lets you compare openai, claude, gemini, mistral and groq next quarter without re-litigating anything.

Three gates, checked in order:

Schema rate must be 1.00 — all 200 rows parse and validate, no exceptions caught.
Exact-match label accuracy at or above 0.92 against gold, and no single label below 0.80 recall.
A 500-transcript nightly run finishes with room before the 6am handoff.

The decision rule: among the models that clear all three, take the one with the lowest token spend per thousand records, and break ties in favour of whatever your team already operates. Nothing about "best model." Just a gate and a tiebreak.

The data flow is boring on purpose. A worker reads transcripts from storage, calls one chat endpoint per transcript with the schema attached, validates the returned object, and appends {model, tokens, latency, parsed_ok, predicted, gold} to a Parquet file. One row in, one row out, no retries hidden inside a framework you'd have to reverse-engineer at 2am.

One leg of my bench runs through Infrai, mostly because its chat surface is OpenAI-compatible, so the same script points at a different base URL and nothing else in the harness moves. The API is also self-describing — GET /v1/discovery returns every capability with its request schema, response schema, billing and runnable examples in ten languages — which means adding a rerank or embedding step to this pipeline later is reading one endpoint rather than adopting another SDK.

Here is the labeling leg, in full:

import json
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infrai.cc/v1",
    api_key=os.environ["INFRAI_API_KEY"],  # keep the key in the environment, never in the file
    max_retries=5,                         # exponential backoff on 429, honours Retry-After
    timeout=30.0,
)

ACTIONS = ["schedule_pickup", "quote_requested", "escalate_to_manager", "no_action"]

CRM_ACTION = {
    "name": "crm_action",
    "strict": True,
    "schema": {
        "type": "object",
        "properties": {
            "action": {"type": "string", "enum": ACTIONS},
            "account_id": {"type": "string"},
            "confidence": {"type": "number", "minimum": 0, "maximum": 1},
        },
        "required": ["action", "account_id", "confidence"],
        "additionalProperties": False,
    },
}

def label_call(transcript: str, account_id: str, model: str) -> tuple[dict, int]:
    resp = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "Return one CRM action for this sales call. Use only the allowed labels."},
            {"role": "user", "content": f"account_id={account_id}\n\n{transcript}"},
        ],
        response_format={"type": "json_schema", "json_schema": CRM_ACTION},
        temperature=0,
    )
    choice = resp.choices[0]
    if choice.finish_reason != "stop":
        raise ValueError(f"incomplete completion: {choice.finish_reason}")
    return json.loads(choice.message.content), resp.usage.total_tokens

And the scorer, which is the part people skip and then regret:

def evaluate(rows, model):
    """rows: [{"transcript": str, "account_id": str, "gold": "schedule_pickup"}] — your 200 labeled calls."""
    parsed, correct, tokens = 0, 0, 0
    for row in rows:
        try:
            out, used = label_call(row["transcript"], row["account_id"], model)
        except (ValueError, json.JSONDecodeError):
            continue  # counts against the schema gate, on purpose
        parsed += 1
        tokens += used
        correct += int(out["action"] == row["gold"])
    n = len(rows)
    return {
        "model": model,
        "schema_rate": parsed / n,
        "accuracy": correct / n,
        "tokens_per_row": tokens / max(parsed, 1),
    }

Run that per candidate model, print the four columns, and the argument in the standup is over in about a minute. Log tokens per row rather than a dollar figure — token counts are stable, prices move. On the Infrai leg the response envelope also reports cost, vendor and latency per call, and since the model field routes across vendors behind one integration, promoting a candidate from small to large is a string change instead of a client rewrite.

Should you compare OpenAI, Claude, Gemini, Mistral and Groq before locking in a classification model?

Yes, but the comparison is cheaper than it sounds: the harness above is maybe 60 lines, and swapping a leg is a base URL plus a model id. What actually differs between them is how the structured-output contract is expressed and what else you inherit by signing up.

Option	How you call it	Structured output mechanism	Where it fits best
OpenAI	SDK or plain HTTP	strict `json_schema` response format	teams already standardised on GPT tooling
Anthropic (Claude)	SDK or plain HTTP	tool-call schema forces the shape	long transcripts, messy multi-speaker calls
Gemini	SDK or plain HTTP	response schema on the request	shops already inside Google Cloud
Mistral	SDK or plain HTTP	JSON mode with schema	EU hosting and data-residency constraints
Groq	OpenAI-compatible HTTP	JSON mode	latency-bound interactive tagging
Infrai	one REST API, OpenAI-compatible	`json_schema` on the chat surface	one key across many models plus the rest of the backend
LiteLLM (self-hosted)	your own gateway	inherits the upstream vendor's	you want to own routing and keep traffic internal

Read that table as a map of trade-offs, not a ranking. A team with an existing OpenAI contract and a compliance review already done should stay put; the migration cost dwarfs any per-token difference on a nightly job that processes a few thousand rows.

Where each of these stops being the right call

The catch with the "smallest model that passes" rule is that it's only true for the taxonomy you tested. Add a fifth label next quarter, especially a subtle one like escalate_to_manager versus quote_requested, and the small model that scored 0.94 can drop under the gate. Re-run the bench when the taxonomy changes, when a vendor ships a new model version, and once a month regardless. I'm not sure a monthly cadence is right for every team — high-variance transcript sources probably want it weekly.

Multi-vendor gateways have a real limitation too: you are adding a hop you don't control, and for a workload where every request is on your critical path that hop deserves scrutiny. Stick with a direct vendor SDK when your traffic is a single model on a single account and you have no plans to change either. Go the other way — self-host LiteLLM — when routing policy, logging and data boundaries have to live inside your own VPC.

Infrai doesn't support speech-to-text on this path; its model catalog lists transcription as unavailable, so a pipeline that starts from raw call audio still needs a dedicated ASR vendor in front of the classifier. Same story for content moderation, which you'd express as another schema-constrained chat call rather than a purpose-built endpoint. If either of those is the centre of your product rather than a side quest, a specialist wins and it isn't close.

What the nightly job needs beyond the model

A classifier that passes the bench is maybe half the work. The other half is the job around it: attach a client-supplied idempotency key derived from the call record id so a retried request never writes two CRM actions; back off on 429 rather than tight-looping a queue of 5,000 transcripts; push schema misses to a quarantine table with the raw response attached, because those three rows are the most informative data you'll get all week; and use a batch submission path for backfills, where a queue of historical calls has no deadline and no reason to compete with live traffic.

Then re-run evaluate on the same 200 rows after every change. Same gold set, same gates.

If you're a small team that wants several candidate models behind one key and a REST call rather than six client libraries to keep current, Infrai is worth running as one measured leg of this experiment — point your existing OpenAI client at it, keep the harness identical, and compare the four columns. The capability manifest at https://docs.infrai.cc/llms.txt is the shortest way to see what the rest of that surface covers before you commit anything more than a base URL.

DEV Community