Short answer: use a JSON Schema as the boundary between an LLM classification and the logistics system that accounts for it. Prompt parsing asks a downstream service to infer structure from prose; schema-constrained output makes the accepted shape explicit, so validation, retries, metrics, and tenant billing operate on fields instead of punctuation. The schema does not make a label correct. It makes an unusable response visible at the right boundary.
For moderation reports, that distinction matters. A carrier, warehouse, or marketplace tenant may submit text about a damaged parcel, an unsafe handoff, or a suspicious status update. Human reviewers need a category and priority, while operations also needs to know how many model calls each tenant caused and how much review remains. A blob of text is a poor accounting record.
What is the bill actually made of?
The first cost is the model request and response, measured in tokens or provider-specific units. The second is operational: retries, queue time, validation failures, human review, and storage for evidence. Prompt parsing hides the second group because a response can look readable while failing the machine contract.
I would start with a ledger keyed by tenant and report, not with a parser. Record input size, classification attempt, validation result, final decision, and retention deadline. Keep raw report and model response only as long as policy requires. Retaining every retry makes investigation easier, but it makes deletion, access review, and tenant cost attribution harder. That is a deliberate trade.
The cheap parser is rarely cheap once it owns the failure.
A narrow schema can keep the durable record small.
from jsonschema import Draft202012Validator
REPORT_DECISION_SCHEMA = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"required": ["category", "priority", "needs_human_review"],
"properties": {
"category": {"type": "string", "enum": ["damage", "safety", "status", "other"]},
"priority": {"type": "string", "enum": ["low", "normal", "high"]},
"needs_human_review": {"type": "boolean"}
},
"additionalProperties": False
}
validator = Draft202012Validator(REPORT_DECISION_SCHEMA)
errors = sorted(validator.iter_errors(decision), key=lambda error: list(error.path))
if errors:
raise ValueError("classification did not satisfy the decision contract")
An enum prevents a dashboard from reconciling damaged, damage, and parcel_damage. additionalProperties rejects silently expanding the durable contract. It does not prove that the model selected the right category; that needs labeled evaluation and human review rules.
The storage decision follows. Keep the normalized decision, tenant identifier, schema version, timestamps, token accounting supplied by the runtime, and an evidence reference. Delete or separately protect raw text on its own schedule. What you stop keeping is convenient replay of every prompt and retry. When something goes wrong, you pay with less forensic context, so the decision must be paired with a useful audit record before deletion.
Why use JSON Schema structured output for LLM tagging?
A prompt can describe a format, but description is not validation. A parser built around Category: or a fenced JSON block has at least three failure modes: extra explanation, malformed JSON, and a valid-looking value outside the operational vocabulary. Whitespace is the easy case. The hard case is a response that parses successfully and still cannot be routed or counted.
A schema gives producer and consumer a shared vocabulary. The consumer can reject missing fields, wrong types, unknown properties, and invalid enum values before the result reaches a queue. JSON has an interoperability specification in RFC 8259, while JSON Schema supplies vocabulary for constraints; neither standard guarantees semantic truth, so evaluation remains part of the design.
The 2020-12 schema dialect is a contract version, not a promise of classification quality.
It is a boundary, not a guarantee.
I initially treated the parser as the cheap component and the model as the risky one. The better correction is that both are part of the classification system. A permissive parser moves ambiguity downstream, where it becomes a routing error or an unassigned tenant cost. A strict boundary turns ambiguity into a measured validation failure. That may increase retries. It also gives the team a number it can investigate.
The runtime contract needs a failure path
Do not send an invalid decision to human review as if it were a low-confidence label. Store the failure reason, increment a tenant-scoped metric, and apply an explicit retry limit. After that limit, route the report to a human or operational quarantine according to policy. The choice belongs in the audit trail.
def classify_report(report, tenant_id, runtime, ledger):
attempt = ledger.start_attempt(tenant_id=tenant_id, report_id=report["id"])
raw = runtime.classify(text=report["text"], schema=REPORT_DECISION_SCHEMA)
errors = sorted(
Draft202012Validator(REPORT_DECISION_SCHEMA).iter_errors(raw),
key=lambda error: list(error.path),
)
if errors:
ledger.record_failure(attempt, reason="schema_validation", error_count=len(errors))
raise ValueError("send to retry policy")
decision = {
"tenant_id": tenant_id,
"report_id": report["id"],
"schema_version": "2020-12",
**raw,
}
ledger.record_success(attempt, decision)
return decision
The important field is not a clever prompt. It is the attempt record. Without it, a tenant with unusually long reports may appear to have a classification problem when it actually generated more input and retries. Cost visibility requires dimensions that survive the request: tenant, report, attempt, outcome, and retention state.
Which details deserve retention?
Retention is an engineering choice with operational consequences. A minimal decision record supports cost reporting and queue routing. A hash or evidence reference can correlate it with protected source data. A full prompt and response support replay and adjudication, but expose more sensitive text and widen deletion obligations.
| Record | Use | Cost or risk |
|---|---|---|
| Normalized label and priority | Routing, reporting, tenant allocation | Context can be lost if vocabulary is too coarse |
| Validation and attempt metadata | Retry analysis and cost attribution | Fields need stable definitions |
| Evidence reference | Human review and audit correlation | Requires protected storage and deletion coordination |
| Raw prompt and response | Replay and investigation | More storage, access exposure, retention work |
A schema version belongs in the record because changing an enum changes historical aggregates. Versioning makes deployment tests concrete: old fixtures validate against the old contract, while new fixtures prove migration. Do not reinterpret last month's other values after changing the taxonomy.
How should teams test the boundary?
Test more than happy-path JSON. Include missing properties, wrong scalar types, unknown properties, invalid enum values, long report text, empty report text, and a response containing a valid object plus irrelevant prose. Test retries and the terminal human-review path separately. Then evaluate quality against a labeled set; schema compliance can reach 100 percent while labels remain wrong.
Operationally, watch validation-failure rate by tenant, schema version, report category, and model configuration. Watch retry counts and retained bytes as well. A global success rate can conceal one tenant whose reports systematically produce expensive failures. Alerts should lead to a decision: adjust taxonomy, improve the input contract, change retry policy, or send more cases to review.
The least complex design is a small, versioned schema, a validator at the queue boundary, a tenant-scoped attempt ledger, and a retention policy that distinguishes decisions from evidence. Prompt parsing can still help an exploratory script. It is a weak durable contract for a logistics workflow where every rejected, retried, and retained report affects human workload and tenant billing.
The limitation is that this approach is a poor fit for a one-off exploratory task where nobody needs to route, aggregate, or retain the result. A schema also becomes counterproductive when the taxonomy is unsettled: strict enums can turn an unknown real-world case into repeated validation failures. That is the trade-off. In those situations, keep the output provisional, measure the cases that do not fit, and revise the vocabulary before treating it as a durable contract.
Top comments (0)