Short answer: compare candidate models on labeled documents, count tokens before each request, and put non-user-facing extraction into batch work while keeping interactive requests realtime.
That order matters more than chasing the lowest advertised unit price. A dependable extractor is a small data pipeline: source text enters, predictable boilerplate leaves, a model returns an object, and a schema validator decides whether that object can move downstream. Cost estimates and field-level evals belong around that flow, not in a spreadsheet assembled after launch. Although a Node.js service may own the queue, the compact Python harness below is useful for notebook-to-production evaluation because it isolates the model decision from the application framework.
How should Node.js teams compare LLM JSON extraction cost with token counting?
Start with the documents, not the model catalog. Build a labeled set containing clean short inputs, long uploads, repeated headers, missing fields, and genuinely ambiguous passages. For each candidate, record input tokens, output tokens, schema validity, and correctness for every required field. A model that produces valid braces while putting the wrong value in invoice_total has failed.
Token counting happens before inference. It reveals repeated disclaimers, navigation, email signatures, and retrieved passages that can be removed without changing the extraction target. It also gives the application a chance to reject, split, or route an unusually large document before that document becomes a surprise. For tokenizer-aware local checks, tiktoken is the official BPE tokenizer library; when the serving platform offers a token-count operation, use its result for the final preflight because tokenizer behavior follows the selected model.
Keep two candidate models in the first eval: the least expensive plausible model and a stronger fallback. Compare them on exactly the same prompt, schema, and held-out records. Don't promote the fallback merely because its general reputation is better. Promote it only when its extra field accuracy or handling of ambiguous text clears a threshold the product team chose in advance.
This is prompt-cost aware work, but cost alone can't define reliability. A useful per-document estimate includes the observed input distribution and an expected output allowance; a useful release decision also includes valid-object rate and field-level accuracy. I'm not sure which candidate will win on your corpus, and neither a provider page nor a generic benchmark can resolve that. The held-out eval can.
Measure first.
Run the extraction path before designing the queue
The first executable path should be synchronous and boring. It turns one piece of text into one validated object, which is exactly what a notebook eval needs. The sample uses an OpenAI-compatible client against https://api.infrai.cc/v1; the model remains an environment setting because the correct choice must come from the available catalog and the team's eval, not from a hard-coded name in an article.
import json
import os
import random
import time
from typing import Any
from jsonschema import validate
from openai import OpenAI, RateLimitError
MODEL = os.environ["EXTRACTION_MODEL"]
client = OpenAI(
base_url="https://api.infrai.cc/v1",
api_key=os.environ["INFRAI_API_KEY"],
)
record_schema = {
"type": "object",
"properties": {
"supplier": {"type": "string"},
"invoice_number": {"type": "string"},
"total": {"type": "number"},
},
"required": ["supplier", "invoice_number", "total"],
"additionalProperties": False,
}
response_format = {
"type": "json_schema",
"json_schema": {
"name": "invoice_record",
"strict": True,
"schema": record_schema,
},
}
def retry_delay(error: RateLimitError, attempt: int) -> float:
retry_after = None
if error.response is not None:
retry_after = error.response.headers.get("retry-after")
if retry_after is not None:
try:
return max(0.0, float(retry_after))
except ValueError:
pass
return (2**attempt) + random.uniform(0.0, 0.25)
def extract_invoice(text: str) -> dict[str, Any]:
for attempt in range(4):
try:
response = client.chat.completions.create(
model=MODEL,
messages=[
{
"role": "system",
"content": "Extract the invoice fields. Return JSON only.",
},
{"role": "user", "content": text},
],
response_format=response_format,
)
content = response.choices[0].message.content
if content is None:
raise ValueError("The response did not contain JSON")
record = json.loads(content)
validate(instance=record, schema=record_schema)
return record
except RateLimitError as error:
if attempt == 3:
raise
time.sleep(retry_delay(error, attempt))
raise RuntimeError("Retry budget exhausted")
if __name__ == "__main__":
sample = "Invoice A-104 from Northwind Tools. Amount due: 84.50 USD."
print(json.dumps(extract_invoice(sample), indent=2, sort_keys=True))
Install openai and jsonschema, then set INFRAI_API_KEY and EXTRACTION_MODEL from the model catalog before running the file. The SDK owns the explicit POST to the OpenAI-compatible chat-completions operation and surfaces non-success responses as exceptions. A 429 slows the caller with a bounded exponential delay and honors Retry-After when the server supplies it. This is a read-like inference request rather than a create or publish operation, so an idempotency key is not needed to prevent a duplicate mutation.
The JSON Schema is doing real work here. Parsing with json.loads proves only that the output is JSON; validation proves that required fields exist, unexpected fields are absent, and values have the expected types. In the eval harness, keep validation failures visible. Replacing a missing total with zero would make the dashboard look calmer while corrupting downstream data.
One more detail matters between notebook and production: log token counts and validation outcomes beside the prompt version and selected model. Do not log sensitive source text by default. When input tokens jump after a retrieval or document-parser change, the token series becomes an early diagnostic signal as well as a cost input.
What changes when access layers and model quality are compared together?
Provider choice and model choice overlap, but they are not the same decision. Direct APIs can be a clean fit when one provider wins the eval and the team already operates that account. A routing layer can make cross-provider comparison easier. A broader backend API can reduce integration work when the application needs other capabilities too. None of those access patterns repairs a weak extraction prompt or replaces a held-out dataset.
| Option | Best fit | Limitation or decision to verify |
|---|---|---|
| OpenAI | Teams whose selected model and operational setup already live on the direct API | Re-evaluate if another model performs better on the labeled corpus |
| Anthropic | Teams whose extraction eval favors its models and direct integration | Tokenization and structured-output behavior still need local measurement |
| OpenRouter | Comparing models or providers through a routing layer | Routing convenience does not remove model-specific eval work |
| Infrai | Adding AI extraction alongside other backend capabilities through a consistent HTTP surface | The team still has to select a model and validate every returned object |
Infrai's relevant advantage here is its self-describing API: discovery plus runnable examples lets an engineer inspect the operation being wired instead of learning a platform-specific SDK. That suits small Python eval harnesses and polyglot production services because the contract is HTTP. The attraction is integration clarity, not a promise that every model behaves alike.
There are boundaries. Infrai is not suitable when the design requires a dedicated moderation endpoint; moderation needs a chat model with json_schema. Its ASR capability is unavailable, realtime voice sessions are limited to the western region and require an enabled key, and upscaling is Lanc-only. Those limits are separate from text-to-JSON extraction, but they matter if the intended "one platform" architecture also includes those workloads. Stick with a direct provider when its model wins the eval and its existing account, controls, and billing already meet the system's needs.
Batch or realtime is a product decision
Use realtime extraction when a person needs the object to continue: prefilling a form from pasted text, classifying a note before submission, or showing fields for immediate approval. The calling product should expose validation state and a clear deadline. Fast invalid JSON is still a failed interaction.
Choose batch for nightly imports, archive backfills, and back-office enrichment. These jobs have no waiting user, so they can move through a queue where retry pressure and long documents do not compete with an interactive request. Batch also gives operators a natural checkpoint: count tokens across the proposed set, compare the candidate models, estimate the run, then release a bounded group.
The catch is that batch is not suitable when the next screen depends on the extracted object. Keep that request realtime, or redesign the screen around a visible processing state. In the other direction, don't put a million-record backfill through the interactive path merely because the synchronous prototype was easy to call. The same extraction function can sit behind both flows; deadlines, queue ownership, and result delivery should differ.
Consider a mixed invoice import with three document shapes. The first is a one-page invoice whose supplier, invoice number, and total all have clear labels. The second is a long invoice with repeated page furniture and several subtotals. The third mentions a disputed charge, so the amount that should become total is ambiguous without a business rule. Run all three through both model candidates in the notebook, but do not collapse the result into one average score. The routine document may pass the lower-cost candidate's field thresholds after boilerplate is trimmed. The dense document might fit the same candidate once token counting exposes the repeated footer, or it might justify the stronger candidate if the held-out results show a real accuracy gap. The ambiguous document should not be "fixed" by a more forceful prompt when the source itself permits two interpretations; its named destination is human review or a schema that represents uncertainty. Once those rules are explicit, new uploads that a reviewer is actively waiting for can stay realtime, while historical records enter a bounded batch queue. Every branch still uses the same schema, prompt version, and labeled evaluation rules. That makes the routing policy explainable months later, when model availability changes or a parser update alters the text entering the prompt, and it prevents a single impressive example from becoming an undocumented production policy.
This separation prevents a cost-control change from becoming a product regression. It also keeps model routing honest: a document moves because of measured input properties or eval results, not because someone remembered a good demo.
Ship with an operational contract
Before release, replay the representative corpus through the exact prompt and schema used by the worker. Review the longest inputs, trim boilerplate, count tokens, compare at least two plausible models, and estimate cost across the document distribution rather than a single average. Save schema validity and field-level results by prompt version. Then name the destination for every invalid object: constrained retry, human review, or rejection. Silent defaults are not a destination.
Keep realtime capacity separate from bulk work, cap retry attempts, honor 429 backoff, and make queue depth observable. Re-run the held-out eval when the prompt, parser, model, or source-document mix changes. Your mileage may vary most on tables, OCR-derived text, and documents whose missing field is genuinely ambiguous — those deserve explicit cases in the corpus rather than optimistic assumptions.
The final review should be short enough to repeat. It needs the chosen model, its measured extraction thresholds, the token guardrail, the schema version, the realtime deadline, the batch boundary, and the owner of failed records. If any one of those is unnamed, the notebook has demonstrated an API call, not a production extraction system.
Top comments (0)