How to batch moderate existing posts and comments with an LLM classification API

SvenNilsson228 — Thu, 30 Jul 2026 13:58:43 +0000

Short answer: for a moderation backfill over existing posts and comments, submit the whole backlog as one bulk job and poll it, instead of firing one LLM classification request per row. Every batch API worth using is built for that shape, and the export step — the part that writes verdicts back into your database — is where the real work hides.

I've done this three times now, on three different products. The first two times I did it the slow way.

Why the one-request-per-row loop dies under real traffic

Last year I ran a policy re-check across 412,000 forum comments with a thread pool and a plain for loop. In the notebook it looked great: 20 sample rows, about 380 ms per classification, ship it. Then I pointed it at the production table on a Sunday night and watched p99 climb to 6.2 s inside the first ten minutes while the median barely moved. Cold start, mostly. My traffic pattern was 40 minutes of nothing, then 300 concurrent requests, then nothing again, so every burst paid the warm-up tax from scratch. The ETA I'd computed from notebook latency said four hours. The real curve said thirty-one. I killed the run at 2 a.m. and went to bed annoyed.

That's the trap nobody warns juniors about. A per-row loop makes your backfill's runtime a function of someone else's tail latency, and tail latency is the one number you can't influence from the client side. Polite retry handling makes it worse, since each 429 you back off from adds its own wait to the critical path.

A bulk job inverts that. You hand over the whole list once, the provider schedules it however it likes, and you poll a status field every fifteen seconds. Tail latency stops being a client problem and turns into throughput accounting, which is a much easier thing to reason about on a spreadsheet.

Same tokens. Roughly the same money. Far less of your evening.

How should you submit a bulk moderation job for existing posts and comments?

Three calls, in this order: submit the list, poll until the job settles, then pull the rows. The example below runs against Infrai — I reached for it here because it's a plain REST API with no SDK to install, so the same three calls port to Node.js, Go, or a shell script without anyone rewriting the logic. POST /v1/ai/batch/submit takes the list, GET /v1/ai/batch/status/{id} tells you where it is, and GET /v1/ai/batch/results/{id} hands back the classified rows.

import os
import time
import uuid
import requests

BASE = "https://api.infrai.cc/v1"
SESSION = requests.Session()
SESSION.headers.update({"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"})

VERDICT_SCHEMA = {
    "name": "moderation_verdict",
    "schema": {
        "type": "object",
        "properties": {
            "verdict": {"type": "string", "enum": ["safe", "review", "blocked"]},
            "category": {"type": "string"},
        },
        "required": ["verdict", "category"],
        "additionalProperties": False,
    },
}

POLICY = (
    "You moderate a marketplace forum. Return verdict=blocked for scams, doxxing or "
    "sexual content involving minors; verdict=review for borderline harassment or "
    "spam; verdict=safe otherwise. category is the policy rule you matched, or 'none'."
)


def call(method, path, **kwargs):
    delay = 1.0
    for _ in range(6):
        r = SESSION.request(method=method, url=BASE + path, timeout=30, **kwargs)
        if r.status_code == 429:
            time.sleep(float(r.headers.get("Retry-After", delay)))
            delay *= 2
            continue
        if r.status_code >= 400:
            raise RuntimeError(f"{method} {path} -> {r.status_code} {r.text[:200]}")
        return r.json()
    raise RuntimeError(f"{method} {path} -> still throttled after 6 attempts")


def as_request(row):
    return {
        "custom_id": f"comment-{row['id']}",
        "body": {
            "model": "glm-4-flashx",
            "messages": [
                {"role": "system", "content": POLICY},
                {"role": "user", "content": row["text"][:4000]},
            ],
            "response_format": {"type": "json_schema", "json_schema": VERDICT_SCHEMA},
        },
    }


def backfill(rows, chunk_name):
    # One stable key per chunk, so re-running the script never submits it twice.
    chunk_key = str(uuid.uuid5(uuid.NAMESPACE_URL, f"comment-backfill/{chunk_name}"))
    job = call(
        "POST",
        "/ai/batch/submit",
        headers={"Idempotency-Key": chunk_key},
        json={"requests": [as_request(r) for r in rows]},
    )
    job_id = job["id"]

    while call("GET", f"/ai/batch/status/{job_id}")["status"] not in ("completed", "cancelled"):
        time.sleep(15)

    return call("GET", f"/ai/batch/results/{job_id}")


if __name__ == "__main__":
    sample = [
        {"id": 1, "text": "great seller, shipped in two days"},
        {"id": 2, "text": "text this number to claim your free prize"},
    ]
    for item in backfill(sample, "chunk-0001")["data"]:
        print(item["custom_id"], item)

Three details in there are load-bearing. The idempotency key means a retried submit is the same submit rather than a second charge, which follows the retry semantics in RFC 9110 and matters a lot when you're chunking half a million rows and your laptop sleeps mid-run. The 429 branch honours Retry-After rather than tight-looping. And every response gets its status checked, because a 4xx body carries the reason and swallowing it costs you an hour of confusion later.

Getting the four labels right before you spend tokens on 400,000 rows

This is the part I'd argue about with anyone who wants to skip it. Pull 200 rows that human moderators have already ruled on, run them through your prompt, and diff the verdicts. Not vibes — an actual agreement number you can put in a commit message.

My last golden set came back at 91% agreement on the first prompt, which sounds decent until you notice the disagreements clustered entirely in sarcasm and quoted abuse. Two prompt revisions later it was 96%, and the fix was a single sentence telling the model that quoted content is attributed, not authored. That sentence was worth more than any model upgrade I could have bought.

The json_schema response format is what makes the export sane. You get verdict and category as typed fields, so writing back into your moderation_state column is a plain UPDATE keyed on custom_id rather than a regex against prose. Fetch or export the finished batch, stream it into a staging table, then flip the flags in one transaction so a half-applied backfill can't leave your forum in a weird state.

Cheap models are fine for this. Classification against an explicit rubric is not the task where a frontier model earns its keep, and running the whole backlog through a small one leaves budget for the eval loop that actually moves accuracy.

How the batch options compare

Option	How you call it	Setup before the first job	Where it fits	Main limit
OpenAI Batch API	upload a JSONL file, poll the batch id	account plus file upload flow	you already run OpenAI models	file-oriented; one vendor's catalogue
Anthropic Message Batches	REST call carrying an array of requests	an Anthropic key	you want Claude's read on nuanced policy	one vendor, separate bill
AWS Bedrock batch inference	S3 in, S3 out, job started via API	IAM roles, buckets, model access	your data already lives in AWS	heaviest setup of the five
Ollama on your own hardware	local HTTP, you build the queue	GPU and ops time	content that can't leave your network	you own throughput, retries and capacity
Infrai batch jobs	three REST calls, any language	one key	mixed models behind one contract	classification runs through chat models

If your org is already standardised on Azure OpenAI, use its batch deployment and stop reading — the integration tax of adding another vendor for a one-off cleanup is real, and nobody thanks you for it at review time.

Where a batch backfill is the wrong call

The catch is latency, and it's not subtle. Batch jobs are scheduled work; results land in minutes or hours depending on queue depth and how much you submitted. If you need a verdict before a comment appears on the page, that's an inline call at write time, and the two systems have to coexist — backfill the history, moderate the new stuff synchronously.

There's no dedicated text-moderation route in the Infrai surface, so classification goes through a chat model with a JSON schema, exactly as in the code above. For my policies I prefer that, since I control the rubric wording and can version it. If you need a pre-trained safety classifier with published category definitions and someone else's liability attached to them, a purpose-built moderation service is the more defensible pick.

Two more places I'd steer you away. If your rule is "no phone numbers", write a regex — an LLM is a silly way to find \d{10}. And if a blocked verdict triggers an account ban, you need a human review queue behind it regardless of how good your agreement number looks. Automated blocks without appeal paths age badly.

Your mileage may vary on the model choice. Everything else here I'd defend.

References

Infrai error codes and retry semantics — https://docs.infrai.cc/errors
RFC 9110, HTTP Semantics (idempotency and retries) — https://www.rfc-editor.org/rfc/rfc9110
OpenAI Batch API guide — https://platform.openai.com/docs/guides/batch
Anthropic Message Batches — https://docs.claude.com/en/docs/build-with-claude/batch-processing
Prompt Engineering Guide — https://www.promptingguide.ai