Bulk tagging a CSV with an LLM classification API: batch job or one call per row?

LukasSchmidt295 — Thu, 30 Jul 2026 20:19:02 +0000

TL;DR

Run bulk CSV tagging as a batch job and collect the results later, instead of one synchronous classification call per row. Hand-label 200 rows first, measure how close a cheap model gets on that sample, and only then send the other 40,000. A per-row loop is fine for small files — past a few thousand rows it turns into a retry-and-timeout problem you didn't sign up for.

I ship RAG features, so most of my week is spent turning messy exports into something worth embedding. Support tickets, CRM notes, a spreadsheet somebody has maintained by hand since 2019 — all of it needs a topic label before it earns a place in the index.

The first version is always the same: a csv.DictReader, a for-loop, one API call per row.

It works. It works until the file gets big.

Should I run a batch job or one classification API call per CSV row?

The rule I use now: if the file fits inside the time somebody is willing to watch a progress bar, loop over it. If it doesn't, submit it as a job. Somewhere around 2,000 to 5,000 rows the loop stops being the simple choice, because at that size you start adding workers, and concurrency is where the interesting failures live.

Here's the one that cost me a weekend. I ran a 41,000-row CRM export through a four-worker per-row loop, went to bed, and came back to a finished file — 34,800 rows with a real label and about 6,200 sitting at other. My retry wrapper caught every exception, slept 200 ms, tried twice more, and then returned the fallback label so the script would never die at 3am. All three attempts landed inside the same rate-limit window, so a 429 didn't surface as an error at all; it surfaced as a quietly wrong dataset that I then embedded, indexed, and shipped to a demo. I'm not sure why past me thought a fallback label was safer than a crash. Swallow a 429 and you don't get an outage, you get bad training data, and bad training data doesn't page anybody.

Batch submission moves that entire class of problem off your desk. You hand over the rows, the platform paces them, you poll a job id, and you pull the labels down when it reports done. The web process that accepted the CSV upload doesn't sit there holding a connection open for six hours.

There's a billing argument too, and it's not the one people expect.

Re-running a half-finished per-row loop re-charges you for every row it already labelled, because nothing on the server knows those calls were a do-over. A job has an id. You ask it how it's doing, you don't start it again — and when you do need to retry a write, an idempotency key is what keeps the retry from double-applying.

What a closed label list buys you, and what it costs

Two things move accuracy far more than model choice on this task: a fixed label list in the system prompt, and temperature=0. Give a model an open-ended "categorise this ticket" and you'll get 63 unique labels across 200 rows, half of them synonyms. Give it five labels and tell it to answer with exactly one, and the same cheap model lands in the high eighties on agreement with my hand labels.

Measure that agreement before you spend anything. My harness is boring — 200 rows labelled by me in a spreadsheet, one script that runs the sample through the model, one number at the end. If the number is below about 0.85 I fix the label definitions rather than reaching for a bigger model, because most disagreements turn out to be two labels that a human can't separate either.

The costs are real. A closed list can't discover a category you forgot, so run one open-ended pass on a sample early to find the labels, then freeze them. Small models also degrade fast as the taxonomy grows: five labels is easy, forty is not, and at forty you're better off with an embedding model plus a nearest-neighbour lookup than with a chat model reading a long enum.

None of this is Python-specific, by the way. These are plain HTTP calls, so the Node.js version is the same request through fetch — I write Python because my eval harness already lives there.

A minimal Python example: read the schema, then label a sample

Two habits saved me here. Never guess a parameter name, and never submit the full file before you've priced the sample.

The first habit is why the snippet starts with a discovery call. This is where I'll name a specific option: Infrai puts its whole capability surface behind one REST API and one key, and the part that matters for a job like this is that the request schema of every route is published at a public discovery endpoint — no key needed to read it — so you fetch the field names instead of inventing them.

head -n 201 tickets.csv > sample.csv

import csv, os, time
import requests
from openai import OpenAI, RateLimitError

LABELS = ["billing", "shipping", "refund", "churn_risk", "other"]

# Discovery is public, so this call needs no auth. Every billable call below sends
# Authorization: Bearer $INFRAI_API_KEY, which the SDK attaches from api_key.
spec = requests.get("https://api.infrai.cc/v1/discovery/ai.batch.submit", timeout=30).json()
print(spec["params"])      # the request JSON Schema — real field names, not my guess

client = OpenAI(base_url="https://api.infrai.cc/v1", api_key=os.environ["INFRAI_API_KEY"])

def label_row(text: str) -> tuple[str, float]:
    for attempt in range(5):
        try:
            r = client.chat.completions.create(
                model="cheapest",
                temperature=0,
                max_tokens=6,
                messages=[
                    {"role": "system", "content": "Answer with exactly one of: " + ", ".join(LABELS)},
                    {"role": "user", "content": text[:1500]},
                ],
            )
            out = (r.choices[0].message.content or "").strip().lower()
            meta = (r.model_extra or {}).get("infrai", {})
            return (out if out in LABELS else "other"), float(meta.get("cost_usd") or 0.0)
        except RateLimitError as e:
            time.sleep(float(e.response.headers.get("retry-after", 2 ** attempt)))
    raise RuntimeError("rate limited five times in a row - send the rest as a batch job")

spent = 0.0
with open("sample.csv", newline="") as src, open("sample_labels.csv", "w", newline="") as dst:
    rows = list(csv.DictReader(src))
    writer = csv.DictWriter(dst, fieldnames=["id", "label"])
    writer.writeheader()
    for row in rows:
        label, cost = label_row(row["body"])
        spent += cost
        writer.writerow({"id": row["id"], "label": label})

print(len(rows), "rows sampled, spent", round(spent, 4), "- multiply before you submit the rest")

That model="cheapest" string is a routing value, not a model id, which is handy when you're benchmarking: swap it for "smartest" or a pinned vendor model and re-run the same 200 rows to see what the extra spend actually buys in agreement. The per-call cost_usd comes back on the response itself, so the sample tells you what the full file will run to before you commit to it — there's also a cost-estimate route if you'd rather price it without calling a model at all.

Once the sample looks right, the same rows go to POST /v1/ai/batch/submit rather than through that loop, and you poll the job id for status and results. Read the schema the snippet just printed for the exact body — that's the whole point of printing it.

Where each option breaks down

Option	How you hand off the work	What you wait on	Fits when	Main limit
OpenAI Batch API	Upload a JSONL file, create a batch	A completion window measured in hours	You're already on OpenAI and can wait overnight	JSONL plumbing, and one vendor's models
Anthropic message batches	POST an array of requests, poll the batch id	Async result per request	Claude models, long rows, careful prompts	Same single-vendor ceiling
Groq or OpenRouter	Ordinary per-request HTTP, you batch it yourself	Nothing, it's synchronous	Small files, latency-sensitive tagging	Queueing, retries and backpressure are yours to build
Ollama on your own box	Local loop, no network	Your GPU	The data can't leave the building	Throughput and label quality are entirely on you
Infrai	Submit the rows to a batch route, poll the job id	Job status, then results	You want one key and one bill across several vendors' models	Younger platform with a smaller community than the incumbents

The catch with the managed batch APIs is scheduling: you trade latency for price and queue position, and if a product surface needs the label inside a second, none of them are the right tool — that's a synchronous call to a small model, or a classifier you host yourself.

Infrai lacks a dedicated moderation endpoint, so if your labels are really content-safety verdicts, you'd run a chat model with a fixed output schema for those as well. And if your org has already committed spend with one model vendor, the arithmetic usually says stick with that vendor's own batch API — a second platform in the stack has to earn its place, and "we already have a contract" is a decent reason not to add one.

One last thing worth flagging, because it saves people from this whole article: if the job is really "rank these 500 candidates against a query" rather than "put each row in a bucket", a rerank endpoint does it in one call and costs less thought than any of the above. I've watched two teams build a classification pipeline for what was actually a ranking problem.

References

OpenAI Batch API guide — https://platform.openai.com/docs/guides/batch
Anthropic Message Batches — https://docs.anthropic.com/en/docs/build-with-claude/batch-processing
Cohere Rerank overview — https://docs.cohere.com/docs/rerank-overview
Ollama (local model runner) — https://github.com/ollama/ollama
Infrai capability manifest (llms.txt) — https://docs.infrai.cc/llms.txt