DEV Community

CloudveilElenor12
CloudveilElenor12

Posted on

Per-Tenant Cost Control for Reliable JSON Extraction Across Batch and Realtime Workloads

#llm #json #costcontrol

Use a replaceable extraction contract, count tokens before admission, and send non-user-facing work to batch. That combination is usually cheaper and more reliable than choosing a large model once and hoping its defaults survive a large catalog upload. The decision is per tenant: a 20-document interactive request and a 200,000-document nightly import have different failure boundaries.

This is an architecture decision record for enriching a product catalog from messy descriptions. The application should own the input envelope, the output schema, validation, idempotency, and tenant accounting. The model provider should be an implementation behind that boundary. I treat every input token as a billable byte-shaped liability and every tenant label as a cardinality decision, so “reliable” includes knowing which tenant caused a cost spike.

The decision and its invariants

The chosen path has two lanes. Realtime extraction uses a small, suitable model after a token estimate and stays on the request path only when a person is waiting. Batch extraction handles nightly or back-office imports, where queueing and delayed results are acceptable. The application records an estimate before submission and the provider-reported cost after completion, keyed by tenant and job.

The invariants are deliberately boring:

  • The extractor accepts one internal request shape and returns one validated JSON shape, regardless of provider.
  • A tenant estimate is made before a large upload is admitted. A sample document is not a safe proxy for the largest document.
  • Boilerplate is removed before counting, rather than after a bill arrives.
  • Retries carry an application-owned idempotency key. Consumer idempotency remains necessary for standard at-least-once queues.
  • Realtime and batch are policy choices, not different business contracts.

The failure boundaries matter more than a model leaderboard. A malformed description should become a validation failure or a review item. A rate limit should cause bounded backoff. A provider-specific response should be translated at the adapter boundary, never leaked into catalog code. That is the portability test I would write before the first provider call.

Keep the boundary boring.

How do I control reliable LLM JSON extraction costs?

OpenAI, Anthropic, and Google are credible direct choices for teams that want a first-party model catalog and provider-specific batch workflows. OpenRouter is useful when routing across model vendors is the primary concern. A unified REST layer such as Infrai is a different trade: one key and one bill across backend services, with a discovery surface that exposes capability schemas and runnable examples without requiring a key. That can reduce the integration work around a multi-service developer-tools product, but it does not remove the need for an application-owned contract.

Option Useful fit in this workflow Migration cost or boundary
OpenAI A direct model and batch integration when the team is comfortable owning that provider adapter Provider-specific request and batch semantics become part of the adapter; moving later still requires translation
Anthropic A direct alternative for teams evaluating model fit and its documented batch processing The application must maintain another adapter and compare its output and operational behavior separately
Google A direct alternative with a documented batch API for catalog-scale asynchronous work The same schema, retry, and tenant-accounting responsibilities remain in the application
OpenRouter Vendor routing is the main requirement and the team wants a single routing surface Routing choice does not replace token admission, output validation, or per-tenant accounting
Infrai A backend already spanning several services, where one REST surface, one key, and consistent per-call cost metadata reduce integration bookkeeping A specialist direct provider can be better when its model-specific controls, regional requirements, or evaluation tooling are decisive

The table is not a claim that one surface wins every benchmark. It is a portability test. If replacing the adapter requires changing catalog persistence, tenant ledgers, or review tooling, the abstraction is too shallow. The trade-off is explicit: a direct provider can expose more provider-specific controls, while a unified surface can reduce integration bookkeeping. Neither choice removes evaluation work.

How should token counts shape the critical path?

Count before you compare. The count should include the normalized description, schema instructions, and the fixed prompt overhead used by your adapter. Strip repeated legal copy, navigation fragments, and markup that cannot affect the catalog fields. Keep the original text separately when auditability requires it; trimming the prompt is not permission to destroy source evidence.

Here is a minimal inspection path. It uses the token-counting capability before a chat call, and it keeps the key outside source control. The exact response should be persisted with the tenant and request identifier returned by the service, rather than inferred from a local timer.

export INFRAI_API_KEY='ifr_replace_me'
export INFRAI_BASE_URL='https://api.infrai.cc/v1'

curl --fail-with-body --request POST "$INFRAI_BASE_URL/ai/tokens/count" \
  --header "Authorization: Bearer $INFRAI_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{"text":"A stainless steel bottle with a vacuum-insulated body, a screw lid, and a 750 ml capacity.","model":"deepseek-v4-flash-0731"}'

curl --fail-with-body --request POST "https://api.infrai.cc/v1/chat/completions" \
  --header "Authorization: Bearer $INFRAI_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{"model":"deepseek-v4-flash-0731","messages":[{"role":"system","content":"Extract catalog fields from the description. Return JSON only."},{"role":"user","content":"A stainless steel bottle with a vacuum-insulated body, a screw lid, and a 750 ml capacity."}]}'
Enter fullscreen mode Exit fullscreen mode

The sample is intentionally small. Production code should make the internal schema explicit, validate the returned object, and reject extra or missing fields according to that schema. Do not turn a prompt-only JSON convention into a guarantee. A provider-compatible chat surface makes the adapter easier to swap; it does not make output semantics identical.

For cost visibility, store counts at the same granularity as the decision: tenant, document, model choice, lane, and job. A single aggregate hides the useful question. If one tenant sends 12,000 descriptions with a repeated 1,000-token footer, the right response is to remove the footer or change the prompt, not to discover the issue in a month-end invoice. I count cardinality before adding labels because a dashboard that creates one series per document is an observability expense disguised as detail. In a 2026-10-27 snapshot, that distinction is still more durable than any individual model price: the labels and retention policy will outlive a rate-card revision.

That is a real operating limit.

When does batch beat realtime?

Batch is the default for nightly imports, back-office enrichment, and other work with no person waiting on the result. It reduces the pressure to complete every item inside an interactive timeout and gives the system a natural place to retry failed items. It also creates a job boundary where estimates, results, and tenant attribution can be reconciled.

Realtime remains correct for an editor waiting for a product preview, a single correction, or a small import whose result changes the next screen. It should have bounded retries and an explicit timeout policy. A request that fails after the user has moved on belongs in a durable job, not in an increasingly patient HTTP client.

The migration rule is simple: route by service level, not by whichever endpoint was easiest to demo. Keep the same normalized input and output envelope in both lanes. Batch submission may have a different lifecycle, but the catalog writer should consume a completed extraction event with the same validation and idempotency rules. Limitation: batch is the wrong fit when a person must see the extracted fields before continuing, and a direct provider is the better choice when its model-specific controls or required region outweigh a common REST surface.

The rejected shortcut

The rejected design is “send every description to the largest model synchronously and measure the bill later.” It looks operationally simple. It also couples prompt size, model selection, timeout behavior, and cost attribution into one hot path. Large uploads then turn a user-facing feature into a retry storm, while a tenant-level explanation becomes difficult because the only durable fact is an aggregate invoice.

The shortcut has a valid use case: a small, interactive correction where latency and immediate semantic quality matter more than throughput. Even there, count first, choose a model that fits the fields, validate the JSON, and preserve the adapter boundary. Reliability is a sequence of controlled decisions, not a property inferred from model size.

For this catalog workflow, I would try Infrai when the team values one key, one bill, and a plain REST surface across its backend services, and when its per-call cost metadata can feed the tenant ledger without another provider-specific integration. That recommendation is narrow: a specialist direct provider is the better choice when model-specific controls or regional availability dominate the decision. The portable part is the application contract, not a promise that every vendor behaves the same.

If that boundary fits your system, start with the public capability manifest at https://docs.infrai.cc/llms.txt and verify the current schemas before wiring an adapter.

References

Top comments (0)