Short answer: use a schema-gated batch classifier for routine decisions, send ambiguous or invalid results to a bounded human review queue, and estimate cost from measured input and output tokens plus reviewer minutes. For a gaming marketplace that accepts supplier invoices as user uploads, structured output correctness is the deciding constraint: an inexpensive label is useless if the invoice ID, supplier, currency, or moderation reason lands in the wrong field.
This architecture decision record chooses a two-stage path. A deterministic intake layer rejects malformed envelopes and exact duplicates. An LLM then returns one small, versioned object that covers both upload safety and invoice-field extraction. Validation happens before any automated action. Nothing crosses that boundary on confidence alone.
The catch is operational: a review queue can absorb ambiguity, but it cannot be an unbounded fallback for every parser disagreement. Capacity, token spend, and schema failures need separate budgets.
Decision, invariants, and failure boundaries
The decision is to optimize for valid, traceable decisions rather than the lowest advertised token price. Batch execution is useful because this workload does not require an answer in the upload request, but batching is a scheduling choice, not a correctness mechanism. The same validation and escalation rules must apply to every item.
Four invariants define the boundary:
- Every accepted result matches one pinned schema version and a closed set of labels.
- Every automated block or approval points to the immutable source hash, prompt version, classifier version, and decision ID.
- A retry cannot create a second moderation action or a second review ticket.
- Missing fields, unknown labels, contradictory reasons, and uncertain invoice extraction go to review; they never become guessed values.
The third invariant is easy to underestimate. RFC 9110 distinguishes idempotent methods because automatic retry changes risk when an operation has side effects. A queue worker should carry that idea beyond HTTP method choice: claim work by a stable content hash, record the decision under an idempotency key, and make the final state transition conditional. If a worker loses its lease after classification, another worker may repeat computation, but it must not repeat the user-visible action.
Keep failure classes separate. Transport failure means retry with a cap and jitter. Schema failure means quarantine the response and create one review item. Policy ambiguity means normal review. Capacity exhaustion means apply backpressure at ingestion. Mixing these states into a generic failed bucket makes cost forecasting impossible and invites retry storms.
For invoice uploads, validation also protects downstream accounting data. currency: USD with no invoice total is incomplete extraction, while label: allow with a reason code reserved for blocked content is a contradictory moderation result. Both are structurally readable JSON. Neither is safe to automate.
No guesswork.
How should batch LLM classification feed a large-volume content review queue?
Use a durable item record between ingestion, classification, and review. The intake request should return after storing the source reference and hash; it should not wait for classification. Workers then form batches according to age and size limits, preserving a per-item identity even if the model request groups many items together. When results return, split them by identity, validate them independently, and commit each decision through an idempotent state transition.
The human queue needs explicit reasons, not a single needs_review label. At minimum, distinguish policy ambiguity, invalid structured output, extraction conflict, and manual sampling. Those lanes have different staffing consequences. A compliance specialist may need the policy lane, while a data operations reviewer can resolve an invoice-field conflict. Spam-filter work teaches a useful architectural lesson here — deliverability failures, rate limits, and content decisions look similar from far away, yet combining them in one retry loop destroys the signal needed to fix any of them.
Do not expose the model's free-form explanation as the reviewer record. Store a constrained reason code and, where policy permits, a short evidence excerpt tied to source offsets. The source remains authoritative. The explanation is supporting material. This also limits the chance that a supplier note containing instructions becomes an operational command in the review interface.
A practical queue policy uses three exits: automate a decision that satisfies every invariant, escalate an ambiguous but valid result, or quarantine an invalid result. The last two may share a user interface, but they should not share metrics. An ambiguity rate describes the policy boundary; an invalid-output rate describes the classification contract.
I'm not sure any fixed confidence threshold can transfer unchanged between game categories, languages, and supplier templates. A labeled evaluation set resolves that uncertainty. Measure results by policy slice, then set thresholds only for slices with enough representative examples; otherwise route the slice to review. Your mileage may vary as upload mix changes, so threshold versions belong beside prompt and schema versions.
Compare the operating options by structured output correctness
The cheapest design depends on how much bad automation costs and how much review capacity exists. This table treats price as one input rather than the verdict.
| Option | Structured-output boundary | Queue impact | Cost model | Suitable when | Main limitation |
|---|---|---|---|---|---|
| Deterministic rules only | Fixed parser and allow/block rules | Exceptions need manual routing | Compute plus reviewer minutes | Invoice formats and prohibited patterns are narrow and stable | Semantic ambiguity becomes rule sprawl |
| LLM for every item | Schema validation after classification | Ambiguous and invalid results enter review | All input/output tokens plus reviewer minutes | Content is varied and the evaluation set supports automation | Repeated boilerplate consumes tokens; a loose schema inflates review |
| Rules before an LLM | Exact duplicates and obvious envelope errors stop early; remaining items use the same schema gate | Review is reserved for semantic cases and contract failures | Rule compute, remaining tokens, and reviewer minutes | Large volume contains meaningful exact repetition or deterministic rejects | Two policy layers must stay aligned |
| Human review only | Reviewer form is the schema boundary | Every item enters the queue | Reviewer minutes plus queue operations | Volume is low or the policy is still being discovered | Throughput and response time scale with staffing |
For this gaming invoice case, choose rules before the LLM, then require schema validation before automation. This is not suitable when uploads are rare and policy changes daily; stick with human review while the team builds a labeled set and stabilizes the decision taxonomy. Rules-only remains valid when suppliers use a small set of contractual templates and moderation concerns are exact, inspectable patterns.
The comparison also exposes a rejected shortcut: using a permissive JSON parser as the correctness test. Parseable output proves syntax, not meaning. Closed enums, required fields, cross-field checks, and source-linked evidence define the useful contract.
Token counting and a cost estimate that survives queue growth
Build the estimate from a sample of the real upload mix. Count tokens with the exact tokenizer used for billing, after applying the actual prompt template and serialization. Character-count heuristics are acceptable for a rough capacity envelope, but they aren't a billing ledger, especially when supplier names, part codes, and multilingual notes change tokenization.
For a period, calculate:
model_cost = input_tokens * input_rate + output_tokens * output_rate
review_cost = reviewed_items * average_review_minutes * loaded_reviewer_rate_per_minute
total_cost = model_cost + review_cost + queue_infrastructure_cost
Keep rates in a dated configuration instead of article prose or worker code. The useful forecast reports ranges for token count, review rate, and review time. A single point estimate hides the expensive branch: a small shift in invalid or ambiguous results can move far more work into the human queue.
Suppose a planning sample contains 10,000 uploads. Do not multiply one average prompt by 10,000 and stop. Partition the sample by language, file type, supplier template, and policy label; measure input tokens, output tokens, schema-valid rate, automated-decision rate, and reviewer minutes for each slice. Then weight the slices using the expected production mix. These are planning quantities, not a claimed benchmark. Re-run the calculation when the mix, policy, prompt, schema, or classifier changes.
Token counting should happen twice. Before dispatch, it enforces a per-item and per-batch ceiling so one huge OCR payload cannot crowd out the rest. After completion, recorded usage reconciles the estimate and feeds the next forecast. If the runtime does not return authoritative usage, record that gap explicitly and reconcile against the billing export; don't silently treat the estimate as observed spend.
Review capacity needs its own equation: required_reviewer_minutes = arrival_count * review_rate * average_review_minutes. Compare that demand with staffed minutes over the same interval and include an arrival spike assumption. Batch discounts or lower token rates cannot rescue a queue whose review arrival rate exceeds its service capacity.
Cost per processed item is useful, but cost per correct final decision is the better comparison axis. It charges invalid output and false automation back to the architecture that produced them. Establishing that denominator requires a labeled audit sample, including automated approvals and blocks; reviewing only escalations cannot reveal errors that bypassed the queue.
The critical path in Python
The example below keeps commercial APIs behind a generic classifier interface. It validates a small result contract, applies cross-field rules, and derives an idempotency key from the policy version and source bytes. Production code would persist these transitions atomically; the important part here is where automation stops.
from dataclasses import dataclass
from decimal import Decimal
from hashlib import sha256
from typing import Any, Literal, Protocol
Decision = Literal["allow", "block", "review"]
REASONS = {"safe", "prohibited_content", "ambiguous", "invalid_invoice"}
@dataclass(frozen=True)
class ModerationResult:
decision: Decision
reason: str
invoice_id: str | None
supplier: str | None
currency: str | None
total: Decimal | None
class Classifier(Protocol):
def classify(self, payload: bytes, schema_version: str) -> dict[str, Any]: ...
def parse_result(raw: dict[str, Any]) -> ModerationResult:
required = {"decision", "reason", "invoice_id", "supplier", "currency", "total"}
if set(raw) != required:
raise ValueError("result keys do not match the pinned schema")
if raw["decision"] not in {"allow", "block", "review"}:
raise ValueError("unknown decision")
if raw["reason"] not in REASONS:
raise ValueError("unknown reason")
total = Decimal(str(raw["total"])) if raw["total"] is not None else None
result = ModerationResult(
decision=raw["decision"],
reason=raw["reason"],
invoice_id=raw["invoice_id"],
supplier=raw["supplier"],
currency=raw["currency"],
total=total,
)
if result.decision == "allow" and (not result.invoice_id or not result.currency or total is None):
raise ValueError("an allowed invoice requires complete accounting fields")
if result.decision == "allow" and result.reason != "safe":
raise ValueError("allow requires the safe reason")
return result
def classify_for_queue(
content: bytes, policy_version: str, schema_version: str, classifier: Classifier
) -> tuple[str, ModerationResult | None, str]:
item_key = sha256(policy_version.encode() + b":" + content).hexdigest()
try:
result = parse_result(classifier.classify(content, schema_version))
except (ValueError, ArithmeticError):
return item_key, None, "invalid_output_review"
if result.decision == "review":
return item_key, result, "policy_review"
return item_key, result, "automated_decision"
The code deliberately does not retry inside classify_for_queue. Retry policy belongs to the worker, where it can distinguish transport errors from contract errors and respect the item's attempt budget. The state store should enforce uniqueness on item_key; a conditional write then makes redelivery harmless at the decision boundary.
Prompt structure still matters. Define the policy, allowed labels, schema, and treatment of untrusted invoice text explicitly, and evaluate changes against labeled examples. The Prompt Engineering Guide is a useful catalog of prompting techniques, but no prompt instruction replaces validation after generation.
Rejected option and the case where it wins
The rejected option is synchronous classification inside the upload request. It appears simpler because there is no visible queue, but it couples user latency and request retries to classification latency, makes load spikes harder to absorb, and tempts the handler to repeat side effects after an uncertain timeout. For large-volume moderation, that is the wrong failure boundary.
It has a valid use case. Choose synchronous processing when volume is low, the caller truly needs a decision before proceeding, the request deadline comfortably covers the operation, and idempotent state transitions are still enforced. A pre-publication chat message may have that requirement; a supplier invoice that can display a processing state usually does not.
The broader limitation of the selected batch design is delayed feedback. It is also a poor fit when policy interpretation is changing faster than a labeled evaluation set can be maintained. In those conditions, route more items to humans and accept the higher review cost. Automation should earn its scope one policy slice at a time.
There is no universal cheapest runtime. The defensible choice is the one that minimizes cost per correct, auditable decision while staying inside queue capacity and the error budget. For invoice-bearing user content, schema validity and cross-field consistency come before token price; batching, token ceilings, and deterministic prefilters then reduce avoidable work without weakening that contract.
Top comments (0)