Short answer: the best way to reduce LLM cost for summarize, classify, and JSON extraction is to make structured-output correctness the gate, then route only the records that need more reasoning to a larger model. Count prompt tokens before dispatch, keep a fixed evaluation set, and batch work that does not need an immediate response. A cheap model that emits invalid catalog data is not cheap.
For a fintech product catalog, the useful unit is not a chat request. It is a record that can be checked, retried, and explained. A messy description becomes a normalized task, passes through a token budget, receives a model assignment, returns one schema-shaped object, and gets validated before persistence. The same path can support a short summary, a category label, or extracted fields such as term, currency, and billing interval.
The notebook-to-prod move matters here. An eval harness should replay representative descriptions before a routing rule reaches a queue. I care about prompt cost because long instructions and retrieved examples are paid for on every row; I care about the schema because a syntactically valid JSON object can still put a number in the wrong field.
What should a small-model comparison prove for catalog JSON?
Start with a frozen set rather than a model leaderboard. Include clean descriptions, missing values, nested product details, multilingual text, and deliberately awkward punctuation. For each row, keep the expected category and a reference object. A summary can be checked for required facts and length; a classifier can be scored with a confusion matrix; an extractor needs type, required-field, allowed-value, and extra-key checks.
Run the identical task contract against a larger baseline and smaller candidates. Record exact-match fields separately from overall pass rate. A model can achieve a good category score while silently dropping a required currency field, which is a much more serious failure for downstream pricing logic than a slightly awkward summary.
The decision rule should be explicit: choose the smallest model that clears the acceptance threshold with room for ordinary input drift. Store the prompt version, schema version, model identifier, input tokens, output tokens, validation result, and fallback reason beside each eval result. That record turns a future regression into a comparison instead of a debate.
Measure it first.
How do prompt token counting and batch processing reduce LLM cost?
Token counting is a preflight check, not a dashboard decoration. Count the system instruction, schema description, few-shot examples, retrieved text, conversation history, and output allowance. Character counts are only a rough proxy. A verbose schema repeated on every request can be a material part of the input, especially when the source description itself is short.
When a request is over its ceiling, apply a deterministic policy: remove low-value retrieved passages, shorten repeated instructions, or split the source at a meaningful boundary. Do not silently truncate the text that contains the product's billing terms. The preflight should record the reason for a trim or rejection so the eval can measure its effect.
Batching changes scheduling, not correctness. Interactive catalog edits need a short response path. A historical backfill, nightly classification run, or reprocessing job after a schema revision can wait in a queue. Submit those records through the selected runtime's documented batch interface, persist the operation identifier, and let a resumable worker poll the documented status and result operations. Keep a small synchronous canary set: a bad prompt should be visible before a large batch is committed.
Here is the application-side part of the contract. The endpoint is deliberately generic; the important behavior is bounded retries, an explicit model choice, and validation before a write.
import json
import os
import time
import urllib.error
import urllib.request
BASE_URL = os.environ["LLM_BASE_URL"]
API_KEY = os.environ["LLM_API_KEY"]
def call_json(payload: dict, attempts: int = 3) -> dict:
body = json.dumps(payload).encode("utf-8")
for attempt in range(attempts):
request = urllib.request.Request(
BASE_URL + "/chat/completions",
data=body,
method="POST",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as error:
if error.code != 429 or attempt == attempts - 1:
raise
time.sleep(2 ** attempt)
raise RuntimeError("retry budget exhausted")
schema = {
"type": "object",
"required": ["category", "term", "currency", "billing_interval"],
"properties": {
"category": {"type": "string"},
"term": {"type": ["string", "null"]},
"currency": {"type": ["string", "null"]},
"billing_interval": {"type": ["string", "null"]},
},
"additionalProperties": False,
}
description = "Annual fraud-monitoring subscription, billed in EUR, renewable after 12 months."
response = call_json({
"model": os.environ.get("SMALL_MODEL", "configured-small-tier"),
"temperature": 0,
"messages": [
{"role": "system", "content": "Return only the requested catalog object."},
{"role": "user", "content": description},
],
"response_format": {"type": "json_schema", "json_schema": schema},
})
record = json.loads(response["choices"][0]["message"]["content"])
required = set(schema["required"])
if set(record) != required or not isinstance(record["category"], str):
raise ValueError(f"catalog contract violation: {record!r}")
The schema option in this example must be supported by the chosen API contract; otherwise, use the provider's documented structured-output mechanism and keep the application validator anyway. Parsing JSON proves only that the bytes form JSON. It does not prove that currency is allowed, that term has the expected meaning, or that a category belongs to the controlled vocabulary.
Consider a description with twelve expected fields: the model returns eleven, wraps the object in a Markdown fence, and spells the currency as EURO. A JSON parser may accept the first two problems in the wrong layer, while a schema and vocabulary check catches all three before the database sees them. The repair path can then send only that record to a larger model or to human review, preserving the original input and validation evidence. This is the important cost boundary: the expensive route is an exception, not the default for every catalog row.
Small models still need a contract.
Which comparison axis matters more than a lower token price?
Compare accepted records per total cost, not the advertised input rate in isolation. A useful report includes validation pass rate, field-level accuracy, fallback rate, input and output tokens, p95 latency for interactive work, queue age for batches, retry count, and human-review volume. For structured catalog output, the cost of a rejected record includes another request and the operational cost of repairing the data.
| Option | Plausible first workload | Engineering trade-off |
|---|---|---|
| Smaller hosted model | Stable labels and shallow fields | More sensitive to schema wording and unusual descriptions |
| Larger general model | Ambiguous descriptions and nested entities | Higher usage and latency; still requires validation |
| Self-hosted small model | Workloads needing local control | GPU operations, upgrades, and on-call ownership |
| Deterministic parser | Fixed phrases and controlled source formats | Brittle when descriptions change or omit key phrases |
The last row is easy to overlook. If a field can be extracted with a regular expression, a finite-state parser, or a lookup table without losing recall, use that route and reserve the model for the uncertain remainder. Retrieval can help classification, but it also adds tokens and noise; a reranking stage should earn its place on the frozen set instead of being included by habit. It doesn't make sense to pay for another model call when a controlled vocabulary and a deterministic parser already produce the same accepted record, yet it also doesn't make sense to force a parser to interpret an ambiguous renewal clause that needs context across sentences.
I'm not sure a single routing threshold will survive a prompt revision. Your mileage may vary across languages and catalog teams. Re-run the eval whenever the schema, examples, retrieval policy, or model changes, and compare the new report with the previous one.
When is this cost-control design the wrong fit?
The catch is that a queue and a fallback route add machinery. This design is not suitable when every request must stream immediately, when a specialized realtime transport is required, or when local inference and contractual controls cannot be represented by the selected API. Stick with a direct integration when it already gives your team the required transport, residency, audit, and support guarantees.
It is also a poor fit for a tiny, stable dataset where a deterministic parser is easier to test and maintain. A small model is not automatically the right answer for ambiguous financial terms, and batch processing cannot repair a weak schema or irrelevant retrieval. Choose the simpler path when its acceptance tests are already strong.
For the general case, the operational checklist is short but strict: enforce the token ceiling before generation, validate JSON before persistence, bound retries, make writes idempotent, preserve the batch operation identifier, and make workers resume without duplicate records. Log why a row used a fallback model. Keep the canary results next to the prompt and schema versions. Those details are what make cost reduction observable rather than aspirational.
Top comments (0)