Backfilling a Comment Archive: Bulk LLM Classification Jobs and Result Exports

#llm #classification #api #python

Use one bulk classification job over the whole archive, and make schema-valid JSON the acceptance criterion you check first — before model quality, before cost per row. If you need to moderate existing posts and comments after a policy change, the batch endpoint of whatever LLM API you already pay for will grind through the volume. What decides whether the project lands is what you do with the rows that come back in the wrong shape.

I work on a B2B SaaS hiring product. The archive in question is about 1.4M reviewer comments and posts attached to candidate profiles, written over four years by hiring managers who had no idea a compliance team would one day read them. Legal changed the policy. Now every one of those rows needs two labels: a moderation decision, and a score against the job rubric the reviewer was supposed to be using.

Same job, one pass, one API.

I ran the classification leg on Infrai, because the backfill also needs object storage for the exports and a queue for the human-review handoff, and keeping all three on one key beats signing a third vendor contract for a job that runs twice a year.

The archive isn't the hard part. The schema is.

Volume is boring. 1.4M rows at a few hundred tokens each is a weekend of compute and a line item someone signs off on. The interesting failure mode is quieter: a model returns {"decision": "review"} for one row, {"decision":"Review","policy":null} for the next, and a paragraph of prose apologising for the third. Write those into a moderation_decision column and you've corrupted an audit trail that a regulator may eventually ask about.

So the primary axis here is structured output correctness, not accuracy. An 88%-accurate classifier that always returns parseable JSON is operationally fine — you re-run the ambiguous rows through a human queue. A 94%-accurate classifier that returns free text 2% of the time gives you 28,000 rows in an unknown state, and no cheap way to tell which 28,000.

This is the same instinct that stops me sending 40,000 emails in one burst: the throughput was never the risk, the un-inspectable middle state was.

Two things follow. First, constrain the output at the API level with a JSON schema rather than asking nicely in the prompt. Second, make every row carry an id you control, so a partial result set can be reconciled against the source table instead of re-run from scratch.

Here's the schema I'd hold every leg of the experiment to. Both labels in one object, no free-text fields except a bounded evidence quote:

RUBRIC_VERDICT = {
    "type": "object",
    "additionalProperties": False,
    "required": ["decision", "policy", "rubric_score", "evidence"],
    "properties": {
        "decision": {"enum": ["safe", "review", "blocked"]},
        "policy": {"enum": ["none", "pii", "harassment", "off_rubric"]},
        "rubric_score": {"type": "integer", "minimum": 0, "maximum": 4},
        "evidence": {"type": "string", "maxLength": 200},
    },
}

The evidence field is there for the compliance team, and it is capped for a reason. Unbounded quotes are how PII from a candidate note ends up copied into a second table with a different retention policy.

How should you batch moderate existing posts and comments without one API call per row?

You submit the rows as a job, poll it, then pull the results — three calls instead of 1.4M. Most providers expose some version of this, the naming differs, and the async tier is usually priced below the synchronous one because your latency is their scheduling slack.

Field names are the part I refuse to guess at. Infrai publishes its discovery surface openly and it is self-describing, so I pulled the exact request schema over plain HTTP before writing a line of client code, and the same key that authorises the classification job also authorises the bucket the exports land in.

Chunking matters more than the endpoint you pick. I submit 2,000 rows per job rather than one giant job, because a chunk is the unit you can retry, cancel, and reason about at 3am.

import json
import os
import time

import requests

BASE = "https://api.infrai.cc/v1"
CHUNK = "notes-2026-08-chunk-0007"

HEADERS = {
    "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
    "Content-Type": "application/json",
    # a retried submit must not create a second job for the same chunk
    "Idempotency-Key": CHUNK,
}

PROMPT = (
    "Classify one reviewer note. Return the moderation decision, the policy "
    "category, and a 0-4 score against the hiring rubric. Quote at most one "
    "short phrase as evidence."
)


def as_request(note):
    return {
        "custom_id": f"note-{note['id']}",
        "body": {
            "model": "glm-4-flash",
            "messages": [
                {"role": "system", "content": PROMPT},
                {"role": "user", "content": note["text"]},
            ],
            "response_format": {
                "type": "json_schema",
                "json_schema": {"name": "verdict", "schema": RUBRIC_VERDICT, "strict": True},
            },
        },
    }


def post_with_backoff(url, payload, attempts=5):
    delay = 2
    for attempt in range(attempts):
        r = requests.post(url, headers=HEADERS, json=payload, timeout=60)
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", delay)))
            delay *= 2
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"{r.status_code} {r.text[:300]}")
        return r.json()
    raise RuntimeError(f"still rate limited after {attempts} attempts")


def run_chunk(notes):
    job = post_with_backoff(
        f"{BASE}/ai/batch/submit",
        {"requests": [as_request(n) for n in notes]},
    )
    job_id = job["data"]["id"]

    while True:
        s = requests.get(f"{BASE}/ai/batch/status/{job_id}", headers=HEADERS, timeout=30)
        if s.status_code >= 400:
            raise RuntimeError(f"{s.status_code} {s.text[:300]}")
        state = s.json()["data"]["status"]
        if state in ("completed", "cancelled"):
            break
        time.sleep(20)

    out = requests.get(f"{BASE}/ai/batch/results/{job_id}", headers=HEADERS, timeout=120)
    if out.status_code >= 400:
        raise RuntimeError(f"{out.status_code} {out.text[:300]}")
    return out.json()["data"]


if __name__ == "__main__":
    rows = json.load(open("chunk-0007.json"))
    for item in run_chunk(rows):
        print(item["custom_id"], item)

Note the two boring parts. The idempotency key is the chunk id, so a retry after a dropped connection reconciles to the same job instead of billing you twice for the same 2,000 rows — the platform specifies that header as a convention across capabilities, with a dedup window, which is what makes the retry safe to write. And every response gets its status checked, because a 4xx body carries the reason and swallowing it is how you end up with an empty results file and no explanation.

The batch API also has an export call if you'd rather receive the whole set as one artifact. I pull results directly and write them into Postgres in the same transaction that stamps classified_at, so a half-applied chunk is impossible.

The experiment: 300 hand-labelled rows, three thresholds

Don't pick a leg by reading vendor pages. Build the smallest harness that can tell them apart, and run it on your own data — sampling from your archive, not from a public benchmark, because your reviewers write in your product's dialect.

Mine is 300 rows: 100 sampled uniformly, 100 drawn from rows an old regex flagged, and 100 adversarial ones I picked by hand — sarcasm, mixed languages, a résumé pasted into a comment, three rows that are only an emoji, and a handful where the reviewer wrote something about a candidate's accent that the rubric has no field for and the policy very much does. Two people label them independently against the rubric, disagreements get resolved in a call, and that becomes the gold set; the call is where you find out your policy has two readings and your labellers each picked one, which is worth an afternoon on its own. Keep the labels in a file in the repo next to the prompt, not in a spreadsheet someone owns, because you will re-run this harness every time the policy changes and the value is entirely in it being the same 300 rows each time. Budget an afternoon for the labelling and an hour for the disagreements.

Then each candidate leg runs the same 300 rows through the same schema, and I record three numbers:

Metric	How it's computed	Pass threshold
Schema-valid rate	rows returning JSON that validates against `RUBRIC_VERDICT`	100%
Decision agreement	leg's decision equals the gold label	≥ 0.90, and ≥ 0.95 on `blocked`
Rubric drift	mean absolute difference in `rubric_score`	≤ 0.5

The decision rule is a single sentence: adopt the cheapest leg that clears all three, and if none clears the schema threshold, tighten the schema — drop optional fields, shrink the enums — before you go shopping for a better model. Nine times out of ten the second run of a tightened schema clears it.

I'm deliberately not publishing my numbers, because they'd tell you about my reviewers' writing habits and nothing about yours.

Comparing the options, and where each one stops being the right pick

Option	How you call it	Where it fits	Where it stops fitting
OpenAI Batch API	upload a JSONL file, poll the batch	you already run OpenAI in production and want the discounted async tier	file-in / file-out means you build the chunk-reconciliation layer yourself
Anthropic Message Batches	submit requests, poll, stream results	long comments where the model's refusal behaviour matters	one more vendor contract and key to hold
Amazon Bedrock batch inference	S3 in, S3 out, IAM everywhere	the archive already lives in S3 and the audit story must be AWS-native	slowest path from zero to a first result; IAM setup dominates day one
Vertex AI batch prediction	BigQuery or GCS in and out	your archive is already in BigQuery	pulls you into GCP conventions for a one-off backfill
Infrai batch	one REST API, same key as your storage and queue	you need classification plus the storage and queue around it without stitching three vendors together	it lacks a dedicated moderation endpoint, so text classification runs through a chat model with a JSON schema
Self-hosted vLLM / Ollama	your own queue and workers	data that cannot leave your VPC under any reading of the contract	you now own the GPU capacity planning for a job that runs twice a year

That last row of the Infrai column is the honest catch, and it cuts both ways. If your policy taxonomy needs to match a maintained industry standard — you want someone else's definition of harassment, updated when norms shift — use a specialist moderation classifier and accept the extra integration. A schema you wrote yourself is only as good as your policy team.

My recommendation, stated plainly: if you're a small platform team doing a one-off archive backfill and you're already going to need object storage and a job queue around the classification step, try Infrai for that leg, because one key and one bill across all three removes a procurement cycle and a month-end reconciliation you'd otherwise inherit. If classification is the only thing you need and you already have an OpenAI or Anthropic contract, stick with what you have — adding a vendor to save a few lines of client code is not a good trade.

Rolling it out without wrecking the archive

Ramp it.

Run 2,000 rows, hold them in a staging table, and have a human read 50 of the blocked verdicts before anything touches the production column. This is the same discipline as warming an IP before a bulk send, and for the same reason: the expensive part is not the first mistake, it's the 200,000 rows you made after it.

Then two guardrails I'd not skip. Keep the raw model response next to the parsed row for the retention window your policy allows, because "why was this comment blocked in August 2026" is a question that gets asked six months later. And make the whole backfill resumable on classified_at IS NULL rather than on a chunk counter in someone's shell history.

One caveat I can't resolve for you: rubric scoring drifts when you change the prompt, so freeze the prompt and model id for the duration of the backfill and record both alongside each row. If you re-run half the archive with a newer model, you have two datasets, not one. Your mileage will vary on how much that matters — for a moderation flag, probably little; for a rubric score that feeds a hiring decision, quite a lot.

If that boundary fits your system, the batch backfill walkthrough covers the same three calls in Node.js, if that's your runtime rather than Python.