Reliable LLM JSON extraction for catalog enrichment needs token counting and cost control, but a product description can be processed twice while its catalog mutation cannot be allowed to happen twice. That operational constraint changes the design more than a model comparison does.
Short answer: put a provider-neutral extraction contract in front of the LLM, estimate tokens before admission, record actual usage after completion, and promote only schema-valid output through an idempotent catalog write. Use realtime processing for work with a real freshness deadline, batch processing for replayable backlog, and compare models with the same catalog fixture rather than a public score or an advertised input rate.
This is an accounting problem disguised as a prompt problem. The unit under control is not a request; it is one version of one product description moving through extraction, validation, and publication. Once that unit has an immutable identity, a team can change providers without changing the catalog workflow, reconcile estimates against billed usage, and prove why a particular attribute appeared.
Decision record: own the extraction ledger
The decision is to keep an internal request envelope, result schema, and append-only attempt record, while treating every model API as an adapter. Four controls sit on the critical path: a content-derived idempotency key, a model-specific token estimate, a hard admission budget, and post-response usage reconciliation. None requires the catalog domain to know a vendor request shape.
The tempting alternative is to pass each provider's JSON directly into the product service. It has less code on day one. The catch is that provider portability then becomes a data-migration problem: retry semantics, usage fields, model identifiers, and structured-output settings leak into catalog records and queue payloads. An adapter boundary costs some maintenance, but the cost is visible and testable.
The record should distinguish an extraction attempt from a catalog effect. A retry may create another attempt with its own timing and usage, while the eventual write remains guarded by the same idempotency key. This is an exactly-once mindset, not a claim that a distributed call occurs exactly once. Duplicate delivery is expected; duplicate business effect is rejected.
| Option | Portability | Cost evidence | Failure boundary | Appropriate use |
|---|---|---|---|---|
| Provider objects in catalog code | Low | Whatever the response exposes | Model and domain failures are coupled | A disposable experiment with no durable records |
| Internal contract with provider adapters | High | Estimate and actual usage can share one ledger | Adapter failure stops before catalog mutation | A production enrichment pipeline |
| Generic routing layer plus internal contract | High at the application edge | Normalized accounting still needs reconciliation | Router and model decisions remain separate from the domain | Many providers or centrally managed routing |
The middle row is the default here, though it isn't universally correct. A short-lived merchandising prototype with hundreds of throwaway descriptions may rationally keep the direct integration. Conversely, a team already operating a routing layer can retain it, provided the team owns the domain contract and audit trail rather than assuming a normalized API also normalizes business semantics.
What must remain invariant across model and provider changes?
Start with the output contract. For a messy product description, a useful envelope might permit brand, color, material, and a list of source spans, while prohibiting unknown fields. Each field needs an explicit null policy. A missing color and the literal string "unknown" are not interchangeable; allowing both creates reconciliation debt downstream.
The second invariant is provenance. Persist the source text hash, schema version, prompt version, requested model identifier, provider identifier, attempt number, token estimate, reported token usage when available, validation outcome, and the idempotency key. Keep the raw response under the retention and access rules that apply to the catalog data. Compliance can constrain what is retained and for how long, so an audit trail should be designed with deletion and access control in mind rather than treated as an excuse to store everything forever. A syntactically valid JSON object can still be wrong for the catalog: the validator must reject an unrecognized enum, an attribute unsupported by its quoted source span, or a schema version the writer no longer accepts. Failed validation produces an attempt record but no product mutation, while a retry can use a different model or provider under the same logical job, which keeps failover policy outside the catalog service. Token counting belongs at this boundary too. tiktoken is an official BPE tokenizer library, and its repository documents model-aware tokenization, making it useful when the selected model has a matching encoding; it does not justify assuming that one local count is exact for every provider, model, wrapper, or hidden request transformation. I'm not sure a local estimate can ever settle the invoice by itself - only the provider's accounting contract can resolve that. Store both numbers and investigate drift.
The boundary is sharp.
How should Node.js teams compare reliable LLM JSON extraction batch and realtime cost?
Compare complete, validated catalog outcomes, not raw calls. For each candidate path, freeze a representative fixture of descriptions, the schema, and the prompt version; then record admission estimates, actual usage, validation failures, retries, and accepted outputs. The denominator is the number of accepted, idempotently published products. This avoids declaring a small model inexpensive when its malformed or unsupported attributes cause repeated work.
Use a simple ledger equation for admission: estimated input tokens times the configured input rate, plus the maximum allowed output tokens times the configured output rate. Rates are configuration with an effective date and currency, never constants scattered through application code. The estimate reserves budget; reported usage settles it. If the provider doesn't return usable token accounting, mark the attempt unreconciled rather than manufacturing precision.
Batch and realtime are scheduling policies over the same state machine. Realtime is justified when the product cannot enter search or review until enrichment completes and that delay has a stated service objective. Batch is preferable when descriptions form a replayable backlog, catalog publication can proceed independently, and aggregate throughput matters more than per-item latency. Don't maintain two extraction implementations. Maintain two admission lanes that create the same envelope and produce the same audit events.
The comparison must include queue delay and deadline misses, but it should not invent a universal winner. Provider batch semantics and pricing can change, and the supplied workload shape matters. A catalog with large nightly imports will behave differently from one with sporadic seller edits. Your mileage may vary; decide with the fixture and the deadline, then retain the evidence used for the decision.
For a Node.js service, the interfaces map naturally to typed request and result objects, even though the critical-path listing below uses Go to make the state transitions explicit. The language is incidental. The ledger boundaries aren't.
Critical path: reserve, extract, validate, settle, publish
The following code deliberately omits an HTTP route. A route would be provider-specific, while the useful artifact is the contract around that call. TokenCounter can wrap an encoding appropriate to the configured model; Extractor owns the provider translation; Ledger and Catalog remain domain ports.
package enrichment
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
)
type Job struct {
ProductID string
Description string
SchemaVersion string
PromptVersion string
Model string
MaxOutputTokens int
}
type Attributes struct {
Brand *string `json:"brand"`
Color *string `json:"color"`
Material *string `json:"material"`
Sources []string `json:"sources"`
}
type Usage struct {
InputTokens int
OutputTokens int
}
type TokenCounter interface {
Count(model, text string) (int, error)
}
type Extractor interface {
Extract(ctx context.Context, job Job, key string) (json.RawMessage, Usage, error)
}
type Ledger interface {
Reserve(ctx context.Context, key string, estimatedInput, maxOutput int) error
Settle(ctx context.Context, key string, usage Usage, valid bool) error
}
type Catalog interface {
PublishOnce(ctx context.Context, productID, key string, value Attributes) error
}
func IdempotencyKey(job Job) string {
sum := sha256.Sum256([]byte(job.ProductID + "\x00" + job.Description + "\x00" +
job.SchemaVersion + "\x00" + job.PromptVersion))
return hex.EncodeToString(sum[:])
}
func Enrich(ctx context.Context, job Job, counter TokenCounter, extractor Extractor, ledger Ledger, catalog Catalog) error {
if job.ProductID == "" || job.Description == "" || job.MaxOutputTokens <= 0 {
return errors.New("invalid enrichment job")
}
key := IdempotencyKey(job)
estimatedInput, err := counter.Count(job.Model, job.Description)
if err != nil {
return err
}
if err := ledger.Reserve(ctx, key, estimatedInput, job.MaxOutputTokens); err != nil {
return err
}
raw, usage, err := extractor.Extract(ctx, job, key)
if err != nil {
return err
}
var attributes Attributes
valid := json.Unmarshal(raw, &attributes) == nil && len(attributes.Sources) > 0
if err := ledger.Settle(ctx, key, usage, valid); err != nil {
return err
}
if !valid {
return errors.New("extraction failed schema or provenance validation")
}
return catalog.PublishOnce(ctx, job.ProductID, key, attributes)
}
Production validation should use the complete schema rather than the compact condition shown here, and reservation must be atomic with the ledger's duplicate-key check. The important ordering is stable: reserve before spending, settle before publication, and publish through a uniqueness guard. If publication is retried after settlement, PublishOnce must return the already-recorded outcome for the same key rather than create a second catalog effect.
Observability follows the same identifiers. Emit latency and token metrics by model and prompt version, but keep high-cardinality product IDs in trace or audit storage rather than metric labels. Alert on unreconciled usage, validation-rate changes, reservation denials, and growing queue age. A model swap is then a controlled deployment: shadow it against the frozen fixture, compare accepted outcomes, canary one admission lane, and preserve the prior adapter for rollback.
Rejected option and the case where it still wins
The rejected option is choosing one globally cheapest advertised model and sending every description through it in realtime. It fails the decision record because an advertised unit rate says nothing about schema acceptance, retry volume, deadline fit, or the effort required to move the domain payload later. It also collapses interactive edits and replayable imports into one operational queue.
Still, direct realtime calls are suitable when the work is exploratory, results are not persisted, the schema is changing daily, and nobody needs reconciliation. Stick with that smaller design until durable catalog mutation, multiple providers, or a real spending boundary appears. Architecture should follow the obligation to account for effects.
For the production catalog, the decision rule is stricter: accept a model-and-lane combination only when it meets the validation contract and freshness deadline on the fixed fixture, exposes enough usage evidence for reconciliation, and can be replaced behind the adapter without rewriting catalog records. Cost breaks ties after those conditions. Reliability is the ability to explain and replay the decision, not merely the absence of an error on one request.
Top comments (0)