To reduce LLM cost when systems summarize, classify, and extract JSON from moderation reports, treat a model result as untrusted until its structure and policy evidence pass an admission gate; otherwise, a cheap call that silently drops a category reaches the human queue late and without a trustworthy explanation.
Short answer: route each moderation report through the smallest model that passes a versioned JSON contract and replay test suite, count input and bounded output tokens before dispatch, and reserve a larger model for ambiguous or schema-invalid cases; use batch processing only where delayed review is operationally acceptable.
This is an architecture decision record, not a model leaderboard. The selected design is an admissibility protocol with deterministic validation, an evidence ledger, and escalation. Cost is evaluated only after that protocol decides which outputs may enter the review queue.
Governance begins with an admission rule
The concrete workload is media moderation triage. A report arrives with free-form reporter text, a content identifier, and policy context. The machine produces a compact summary, assigns one category from a closed set, extracts evidence into JSON, and sends the record to a human reviewer. The classifier does not make the final enforcement decision. That compliance boundary matters: automation may order the queue, but retained input, policy version, output, validator result, and human disposition form the audit trail.
The decision rests on four invariants. Every accepted output must validate against the exact schema version used for that request. Every request must have a stable idempotency key, so a retry cannot create a second review item. Every routing decision must be reconstructable from stored token counts, model-policy version, attempt number, and validation outcome. Finally, escalation must be explicit: uncertainty, an unknown category, missing evidence, or invalid JSON moves the item to another path rather than being coerced into a plausible value.
Exactly once is an accounting objective, not a property bestowed by an API. The practical implementation is at-least-once delivery plus an idempotent state transition in the system of record. A worker can crash after receiving a model response but before acknowledging its queue message; on replay, the same key must resolve to the existing attempt or advance it under a compare-and-swap rule. Don't let the model response itself become the transaction boundary.
The hardest failure is syntactically valid, semantically wrong JSON. A decoder will happily accept {"category":"spam"} even when the evidence describes impersonation, and a schema cannot prove that the label follows the policy. Contract validation is therefore the first gate, not the last one: follow it with policy fixtures, adversarial examples, and a sampled human reconciliation process that compares machine routing with reviewer disposition. This is where a payment-style ledger mindset earns its keep. Corrections are appended as new events; history isn't overwritten.
One rule is absolute.
Never infer a missing required field during persistence.
Reliability requires evidence that survives retries
Compare candidates on accepted work, not raw calls. The useful denominator is the number of reports that pass structural validation and the policy test set without escalation. A small model can have a lower per-call charge yet produce enough retries, larger-model fallbacks, or reviewer corrections to lose on effective cost. Conversely, a larger model can be wasteful on short, obvious reports that a constrained classifier handles consistently.
Prompt token counting belongs before dispatch because it is also admission control. Count the stable policy instructions, schema, report text, and any examples using the tokenizer associated with the candidate model; then enforce an output ceiling derived from the schema rather than granting a generic maximum. Token estimates from character counts are suitable only for coarse queue planning, since the actual tokenizer resolves the billable input. I'm not sure a single static threshold will remain optimal as policy text and the report mix change; weekly cohort data, separated by language and category, would resolve that uncertainty better than intuition.
Use an offline, frozen evaluation set with expected categories and required evidence fields. Keep a separate challenge set for malformed text, conflicting cues, prompt injection inside user content, empty descriptions, and content near the context limit. The promotion rule should state both quality and operations constraints before any candidate is tested, preventing a team from moving the goalposts after seeing an attractive price. For example, require zero schema acceptance bypasses, define a minimum category agreement chosen by the policy owner, and cap the escalation rate according to available reviewer capacity. Those thresholds are governance choices; they are not universal facts.
The comparison table records the architectural options without pretending that one wins every workload:
| Option | Correctness control | Cost behavior | Latency and operations | Suitable boundary |
|---|---|---|---|---|
| Small-model synchronous route | Strict schema, fixture suite, explicit fallback | Low work per ordinary report; retries and fallback count toward accepted-work cost | Immediate result; router and reconciliation required | High-volume, short reports with stable categories |
| Larger-model synchronous route | Same schema and tests; often fewer ambiguity escalations must still be measured | More capacity is purchased for every report | Simple first deployment; immediate result | Low volume, difficult language, or rapidly changing policy |
| Mixed router | Per-route schema plus recorded decision policy | Capacity follows measured difficulty | More state, monitoring, and audit work | Diverse traffic with enough volume to justify routing |
| Batch processing | Identical validation after results return | Can consolidate delay-tolerant work; pricing must be checked against the chosen service | Delayed completion and more reconciliation state | Backlogs, reclassification, and offline evaluation |
Batching is not the same thing as sending many reports in one prompt. Packing unrelated reports together couples their failure domains, complicates per-report idempotency, and can let one long item consume the shared context budget. A provider's asynchronous batch facility can preserve one logical request per report while changing scheduling and completion semantics. If no such facility is part of the deployment, an internal queue can still form operational batches, but it does not imply a billing discount. Check the applicable service terms rather than assuming one.
Developer experience depends on one executable contract
The critical path below deliberately hides vendor transport behind an interface. All durable writes go through a store that can enforce uniqueness on ReportID + PolicyVersion + Attempt; the model client receives bounded input and returns bytes plus measured usage. In production, the JSON decoder should also enforce the complete schema, including length bounds and whether additional properties are permitted. This abbreviated example keeps the decision logic visible.
package triage
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
)
type Report struct {
ID string
ContentID string
Description string
}
type Classification struct {
Summary string `json:"summary"`
Category string `json:"category"`
Evidence []string `json:"evidence"`
NeedsHuman bool `json:"needs_human"`
}
type Usage struct {
InputTokens int
OutputTokens int
}
type ModelClient interface {
Classify(ctx context.Context, model string, prompt string, maxOutputTokens int) ([]byte, Usage, error)
}
type AuditEvent struct {
IdempotencyKey string
ReportID string
PolicyVersion string
Model string
InputTokens int
OutputTokens int
Outcome string
}
type AuditStore interface {
AppendOnce(ctx context.Context, event AuditEvent) error
}
var allowed = map[string]bool{
"harassment": true,
"impersonation": true,
"spam": true,
"other": true,
}
func key(reportID, policyVersion string, attempt int) string {
sum := sha256.Sum256([]byte(fmt.Sprintf("%s:%s:%d", reportID, policyVersion, attempt)))
return hex.EncodeToString(sum[:])
}
func decodeStrict(raw []byte) (Classification, error) {
var out Classification
if err := json.Unmarshal(raw, &out); err != nil {
return out, err
}
if out.Summary == "" || len(out.Evidence) == 0 || !allowed[out.Category] {
return out, errors.New("output violates the moderation contract")
}
return out, nil
}
func Classify(
ctx context.Context,
client ModelClient,
store AuditStore,
report Report,
policyVersion string,
prompt string,
) (Classification, error) {
models := []string{"small-policy-candidate", "large-policy-fallback"}
for attempt, model := range models {
raw, usage, callErr := client.Classify(ctx, model, prompt, 220)
out, validationErr := decodeStrict(raw)
outcome := "accepted"
if callErr != nil || validationErr != nil || out.NeedsHuman {
outcome = "escalated"
}
event := AuditEvent{
IdempotencyKey: key(report.ID, policyVersion, attempt),
ReportID: report.ID, PolicyVersion: policyVersion,
Model: model, InputTokens: usage.InputTokens,
OutputTokens: usage.OutputTokens, Outcome: outcome,
}
if err := store.AppendOnce(ctx, event); err != nil {
return Classification{}, err
}
if outcome == "accepted" {
return out, nil
}
}
return Classification{}, errors.New("human classification required")
}
There is an intentional distinction between escalation and failure. Invalid output is expected input to the router's state machine; it becomes an operational failure only if the item loses its durable state, bypasses human review, or cannot be reconciled. The audit event should carry hashes or references to retained artifacts according to the organization's privacy policy, because moderation text may itself contain sensitive material. Retention duration, access control, regional processing, and the lawful basis for storing report content require review by the relevant compliance owner. This article cannot set those limits for a particular jurisdiction.
Observe three ratios by policy version and traffic cohort: structural acceptance, escalation, and later reviewer disagreement. Token totals are necessary, but they should sit beside queue age and duplicate-suppression counts. An aggregate acceptance rate can conceal a weak category or language cohort, so reconciliation needs stratified samples and an immutable link from the correction to the original decision.
Migration proceeds through reconciliation gates
Start in shadow mode: create predictions and audit events, but leave queue ordering unchanged. Once the frozen test set and live reconciliation sample satisfy the predeclared gates, enable the small-model route for one narrow category cohort. Rollback should change the versioned routing policy, not mutate old records. Clean boundaries make this boring, which is exactly what a moderation control plane should be.
Cost accounting should include input tokens, output tokens, retry attempts, fallback attempts, batch fees where applicable, and the human-review load caused by escalation or correction. Avoid collapsing these into one monthly average. A route may appear cheaper only because its difficult cases are charged to another team's review budget. The defensible measure is cost per accepted and reconciled report, with the raw components preserved for audit.
Cache only where policy allows it and where the key includes every input that can change the answer: normalized report content, policy version, schema version, and routing configuration. Moderation reports that look identical may refer to different content or actors, so semantic caching can be unsafe unless the decision is explicitly designed to ignore those distinctions. Exact request deduplication is the conservative first step.
Operationally, alert on stuck queue age, missing terminal audit events, duplicate key conflicts, sudden changes in schema rejection, and disagreement drift. Do not alert on every escalation; escalation is part of the design. Page when the reconciliation contract breaks.
How should media teams compare small models, token counting, and batch processing?
The rejected default is “send every report to the largest available model synchronously.” It provides a straightforward integration and may be the right starting point when traffic is low, categories change daily, reports are unusually nuanced, or the team cannot yet operate a router and reconciliation ledger. Stick with that design when its simplicity is worth more than the capacity it leaves unused. A small-model route is not suitable when the evaluation set cannot represent the policy boundary or when reviewer capacity cannot absorb its measured escalation tail.
The opposite extreme, “small model for everything,” is also rejected. It optimizes the visible unit price before proving structured output correctness and hides ambiguity behind forced labels. Batch processing is rejected for urgent safety reports because queue delay is part of the product behavior; it remains valid for historical reclassification, offline prompt evaluation, and other work with an explicit completion window.
No permanent model choice follows from this record. The durable choice is the testable contract: versioned schemas, prompt token counting, bounded output, deterministic acceptance, explicit escalation, idempotent writes, and reconciliation against human decisions. Models can then change without rewriting the system's definition of correctness.
Top comments (0)