A supplier-invoice pipeline has one constraint that changes the whole design: a plausible wrong field is worse than a slow field. Short answer: for cheap bulk text classification with an LLM API, parse the CSV once, classify bounded batches through a narrow adapter, validate every response, and send uncertain rows to review instead of chasing per-call latency.
That choice makes the LLM only one stage in a boring data job. Good. The import, schema checks, retry policy, measurements, and output format stay under application control. A provider swap then changes the adapter, not the job.
This is a build log for extracting invoice_number, invoice_date, currency, and total from supplier invoice text. The primary benchmark is useful records per minute: records that pass structural validation and an explicit confidence rule. Raw requests per second hides too much.
Set the failure budget before making an API call
The tempting design is one row, one request, one returned tag. It looks clean in a demo and turns ugly in a backfill. Request overhead is repeated for every row, concurrency becomes accidental, and partial progress is hard to identify after a process restart. A huge single request has the opposite failure mode: one malformed item can make an otherwise useful response difficult to reconcile.
So the unit of work is a bounded batch with stable row IDs. Every input row gets an ID before inference. Every output must return that same ID. Ordering is never treated as identity — a small detail that prevents a large class of silent joins. Consider a three-row batch in which the middle invoice has no total: an API may return two valid objects, and joining those objects by array position would attach the third supplier's total to the second supplier. Joining by ID instead leaves the missing row visibly incomplete. The job can review one invoice while accepting two; it never has to pretend that a partially useful response was either wholly successful or wholly lost.
Quality also needs a sharper definition than “the output looked right.” For invoice extraction, I would split it into three gates:
- Shape: the response has exactly the requested keys and supported value types.
- Evidence: each extracted value is present in the source text after conservative normalization.
- Decision: confidence clears the review threshold; otherwise the record is held back.
The evidence gate is deliberately strict. It will reject some useful normalization, such as converting a written month to an ISO date, but that is a visible false negative. Accepting a fabricated total creates a quiet false positive, and quiet errors are expensive to find later.
Latency gets measured at the batch boundary, not from a single warm request. Record queue time, API time, validation time, and end-to-end time separately. Use p50 and p95 for each, then count accepted records. I'm not sure which batch size wins for a given invoice mix; document length and review threshold can move the result. A small representative fixture set resolves that uncertainty faster than a vendor feature matrix.
IDs first.
How should a Node.js batch job balance cheap LLM API latency and classification quality?
Start with a fixed concurrency limit and test several batch sizes against the same labeled fixture set. Do not tune cost, speed, and quality in separate runs. The comparison row should include all three because an apparently cheap run can create more review work, while a fast run that drops valid records has poor useful throughput.
The benchmark table I use is a template, not a set of claimed results:
| Run | Batch size | Concurrency | p50 end-to-end | p95 end-to-end | Valid fields | Review rate | Usage units |
|---|---|---|---|---|---|---|---|
| A | measured | measured | measured | measured | measured | measured | measured |
| B | measured | measured | measured | measured | measured | measured | measured |
| C | measured | measured | measured | measured | measured | measured | measured |
“Usage units” is intentionally generic. Providers expose usage differently, and translating usage into currency belongs in a separate reporting function. Cost matters, but it is not the decision by itself. The useful ratio is accepted fields per usage unit, paired with p95 latency and review rate.
Keep the prompt contract small. Ask for JSON, enumerate the allowed fields, require source-preserving values, and define the uncertain case. Don't ask the model to explain its reasoning. Explanations add output that the batch job cannot safely join or validate. A compact evidence string is different: it can be checked against the source and shown to a reviewer.
A 429 is a scheduling signal. Retry that batch with backoff and jitter, preserve its IDs, and cap attempts. Authentication failures and invalid request shapes should stop the run because repeating them only creates noise. Invalid model output belongs in the review stream, not in an infinite retry loop. These categories should be explicit even if the first deployment uses one provider.
Put the extraction contract in a TypeScript worker
The vendor-specific code is the Classifier implementation. Everything below it is ordinary TypeScript: batching, bounded concurrency, validation, evidence checks, and stable output. The CSV parser is injected because production CSV can contain quoted commas, embedded newlines, and varying encodings; those rules should be handled by the parser already approved in the application, not by line.split(",").
type InvoiceRow = {
id: string;
text: string;
};
type ExtractedFields = {
invoice_number: string | null;
invoice_date: string | null;
currency: string | null;
total: string | null;
};
type Classification = {
id: string;
fields: ExtractedFields;
confidence: number;
};
type Classifier = (rows: readonly InvoiceRow[]) => Promise<unknown>;
type CsvParser = (source: string) => Array<Record<string, string>>;
const fieldNames: Array<keyof ExtractedFields> = [
"invoice_number",
"invoice_date",
"currency",
"total",
];
function normalizeEvidence(value: string): string {
return value.toLocaleLowerCase("en-US").replace(/\s+/g, " ").trim();
}
function validateResult(value: unknown, source: InvoiceRow): Classification | null {
if (typeof value !== "object" || value === null) return null;
const item = value as Record<string, unknown>;
if (item.id !== source.id) return null;
if (typeof item.confidence !== "number" || item.confidence < 0 || item.confidence > 1) {
return null;
}
if (typeof item.fields !== "object" || item.fields === null) return null;
const rawFields = item.fields as Record<string, unknown>;
const fields = {} as ExtractedFields;
const normalizedSource = normalizeEvidence(source.text);
for (const name of fieldNames) {
const field = rawFields[name];
if (field !== null && typeof field !== "string") return null;
if (typeof field === "string" && !normalizedSource.includes(normalizeEvidence(field))) {
return null;
}
fields[name] = field as string | null;
}
return { id: source.id, fields, confidence: item.confidence };
}
function chunks<T>(items: readonly T[], size: number): T[][] {
if (!Number.isInteger(size) || size < 1) throw new Error("batchSize must be positive");
const result: T[][] = [];
for (let index = 0; index < items.length; index += size) {
result.push(items.slice(index, index + size));
}
return result;
}
async function mapConcurrent<T, R>(
items: readonly T[],
limit: number,
fn: (item: T) => Promise<R>,
): Promise<R[]> {
if (!Number.isInteger(limit) || limit < 1) throw new Error("concurrency must be positive");
const output = new Array<R>(items.length);
let next = 0;
async function worker(): Promise<void> {
while (next < items.length) {
const index = next++;
output[index] = await fn(items[index]);
}
}
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
return output;
}
export async function extractInvoiceFields(
csv: string,
parseCsv: CsvParser,
classify: Classifier,
options = { batchSize: 20, concurrency: 3, reviewBelow: 0.85 },
): Promise<{ accepted: Classification[]; review: InvoiceRow[] }> {
const rows = parseCsv(csv).map((record, index) => ({
id: record.id || `row-${index + 1}`,
text: record.invoice_text || "",
}));
const batches = chunks(rows, options.batchSize);
const results = await mapConcurrent(batches, options.concurrency, async (batch) => {
const raw = await classify(batch);
if (!Array.isArray(raw)) return { accepted: [], review: batch };
const byId = new Map(batch.map((row) => [row.id, row]));
const accepted: Classification[] = [];
const acceptedIds = new Set<string>();
for (const candidate of raw) {
const id = typeof candidate === "object" && candidate !== null
? (candidate as Record<string, unknown>).id
: null;
const source = typeof id === "string" ? byId.get(id) : undefined;
if (!source) continue;
const valid = validateResult(candidate, source);
if (valid && valid.confidence >= options.reviewBelow) {
accepted.push(valid);
acceptedIds.add(valid.id);
}
}
return {
accepted,
review: batch.filter((row) => !acceptedIds.has(row.id)),
};
});
return {
accepted: results.flatMap((result) => result.accepted),
review: results.flatMap((result) => result.review),
};
}
The default numbers are starting parameters, not benchmark claims. Put them in one options object so a test matrix can vary them without config sprawl. I would also keep the adapter responsible for translating this contract to its API and returning decoded JSON. That boundary makes tests cheap: a fake classifier can return missing IDs, duplicate IDs, low confidence, or a value absent from the invoice text without making a network call.
Notice what the core refuses to do. It does not guess how to repair malformed output. It does not silently coerce numbers. It does not assume response order. Rejected rows remain attached to their original text, ready for review or a separately governed retry.
Instrument accepted fields before scaling the queue
The in-memory arrays are fine for a small backfill. They are not suitable when the CSV is larger than the process can comfortably hold, when jobs must survive restarts, or when several workers share a queue. At that point, stream parsed records into durable work units, store a content hash with each row, and checkpoint completed batch IDs. Keep the same classifier boundary.
I would add two fixture suites. The first is a small frozen set with expected fields, including missing totals, duplicate invoice numbers, comma-formatted amounts, and ambiguous dates. It runs on every adapter change. The second is a rotating sample from the actual supplier mix, reviewed by a human before it affects a threshold. The frozen suite catches regressions; the rotating suite catches drift in the input population.
Observability stays plain: batch ID, row count, attempt count, queue duration, API duration, validation duration, accepted count, and review count. Do not log full invoice text by default. Logs need identifiers for reconciliation, while access to source documents should remain in the system that already governs them.
Measure the tail.
There is another scale boundary. If the fields follow stable layouts and deterministic parsing reaches the required accuracy, stick with deterministic extraction. If documents demand visual layout understanding, a text-only classifier is the wrong component; use a document-processing path that preserves layout. And if every result must return interactively, a background CSV job is a poor fit no matter how low its average latency looks.
Stop using this design when the document demands more
This design favors auditability over maximum throughput. Evidence checks reject normalized values that are not literal source substrings. Human review adds latency. Bounded batches may leave provider capacity unused. Those are real costs, and loosening each rule can be reasonable after a labeled evaluation shows where the errors go.
The catch is that confidence is model output, not proof. Calibrate the review threshold against labeled invoices, then watch the accepted error rate as the supplier mix changes. Your mileage may vary — especially with short OCR fragments or locale-dependent totals — so publish the test corpus definition beside the benchmark instead of presenting one throughput number as universal.
The decision rule is compact: choose the configuration with the lowest p95 latency that still meets the accepted-field quality target and review capacity. Cheap usage is useful only inside that boundary. This keeps the batch job honest, the adapter replaceable, and the failure cases visible.
Top comments (0)