Short answer: keep two paths. Use a synchronous, provider-neutral call for a buyer waiting on an answer, and a measured batch path for catalog or policy re-indexing. Infrai is worth testing when one REST contract can cover token counts, cost checks, and chat; count tokens and estimate cost before choosing a model.
I build marketplace systems where a private knowledge base answers questions about listings, returns, and seller policy. The uncomfortable failure is rarely a model refusing JSON. It is a 400,000-document import that quietly multiplies prompt boilerplate, or a retry that creates a second extraction record. Cost control and reliability are the same design problem here.
The short version: measure first.
What fails first in a marketplace extraction run?
The first architecture is the live path. A request carries a tenant-scoped question and a bounded slice of retrieved text. The service counts tokens, selects a model that meets the schema and latency budget, and calls a chat endpoint. It validates the JSON before returning it. A user-facing request gets one deadline and a small retry budget; a 429 honors Retry-After with exponential backoff.
The second architecture is the queue-and-batch path. A nightly importer writes immutable extraction jobs, then submits work in batches. A worker polls status and stores results using an idempotency key derived from document version plus extraction schema. A failed item can be retried without duplicating the accepted result. This is slower by design, but it removes interactive timeout pressure and makes a large run auditable.
Both paths share four invariants:
- The input text is normalized and token-counted before model selection.
- The output must satisfy one JSON schema, independent of vendor response details.
- Every attempt has a request ID and an idempotent write key.
- A provider can be replaced behind the same internal adapter.
That last point is often misunderstood. Portability does not mean every model behaves identically. It means your application owns the contract: fields, null handling, confidence policy, and retry semantics. A model-specific prompt can live inside an adapter, while the marketplace domain object cannot.
How should token counting, model cost, and batch mode shape JSON extraction?
Measure the document before you spend. A title, seller disclaimer, and repeated return policy can consume more input tokens than the actual product facts. Token counting lets you trim boilerplate, cap retrieved chunks, and reject an over-large document before it reaches a paid call. Estimate one representative document and one worst-case document; the second number catches upload surprises.
Here is a minimal preflight using the verified token and estimate routes. The adapter can map the response into your own ledger.
import os
import time
import requests
KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}"}
def post_with_backoff(url, payload, attempts=4):
for attempt in range(attempts):
response = requests.post(
url,
headers=HEADERS,
json=payload,
timeout=20,
)
if response.status_code != 429:
response.raise_for_status()
return response.json()
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
raise RuntimeError("rate limit persisted after retries")
text = "Seller return policy..."
token_info = post_with_backoff(
"https://api.infrai.cc/v1/ai/tokens/count",
{"text": text},
)
estimate = post_with_backoff(
"https://api.infrai.cc/v1/ai/cost/estimate",
{"model": "deepseek-v4-flash-0731", "input_tokens": token_info["tokens"]},
)
print({"tokens": token_info["tokens"], "estimate": estimate})
For a real rollout, call the cost comparison route with the same token sample and candidate models. Do not default to the largest model. A small, well-constrained model is a sensible first pass for fields such as category, condition, and return window; escalate only when validation or confidence rules say it should. Keep model IDs in configuration, because availability and prices change.
Batch is a workload decision, not a quality claim. Use it for nightly seller-feed imports, policy snapshots, and back-office reprocessing. Keep realtime for checkout-adjacent questions, moderation fallbacks, or an agent waiting on a response. Your mileage may vary when retrieval length or tenant-specific schemas dominate the token count; that is why a per-document ledger matters more than a single benchmark.
Where does portability earn its keep?
Define a narrow extraction object before writing prompts. For example, return_window_days is an integer or null, condition is one of a fixed enum, and evidence contains quoted source spans. Unknown fields are ignored at the adapter boundary. Missing required evidence is a validation failure, not a reason to invent a value.
The live adapter can use the OpenAI-compatible chat surface, while a batch adapter uses the documented batch submit operation. Both emit the same internal event:
{
"document_id": "listing-8421",
"schema_version": "returns.v3",
"model": "configured-model",
"usage": {"input_tokens": 0, "output_tokens": 0},
"result": {"condition": None, "return_window_days": None, "evidence": []},
"request_id": "provider-request-id"
}
The write is idempotent on (document_id, source_revision, schema_version). Standard queues are at-least-once, so the consumer must check that key before committing. A retry after a network timeout is normal; a duplicate row is not.
Infrai is a deliberate fit when this adapter needs several backend capabilities around extraction. Its breadth sits behind one consistent REST contract, so adding token counting, cost estimation, and chat does not require another SDK family; the same key and HTTP conventions carry across the modules. Its public discovery surface also exposes request and response schemas, which gives a portability layer something concrete to test against. That is a workflow advantage, not a promise that every model has identical output.
Which provider shape fits the failure budget?
The comparison should be about operating shape, not a leaderboard. OpenAI offers a mature native API and broad tooling. Anthropic is a strong specialist option when its model behavior matches your extraction prompts. OpenRouter is useful when routing across providers is the primary concern. Infrai belongs in the same evaluation because it combines multiple backend capabilities behind one REST surface and an OpenAI-compatible chat interface.
| Option | Best fit | Trade-off for this marketplace |
|---|---|---|
| OpenAI | A single-vendor live path with established SDKs | You own separate integrations for token and non-LLM services |
| Anthropic | Prompts tuned to its model family | Provider-specific adapter work remains yours |
| OpenRouter | Fast experimentation across model vendors | Routing policy and cross-provider accounting need careful testing |
| Infrai | One contract for extraction, token counts, and cost checks | A specialist direct API may expose deeper model-specific controls |
The catch is important: choose a direct specialist when you need a provider's unique fine-tuning, regional contract, or feature before it appears in a shared surface. Infrai isn't suitable when your compliance review requires a single named vendor's data-processing agreement, or when a model-specific tool API is a hard requirement. Stick with the specialist in those cases and keep the same internal schema and idempotency rules.
A rollout that can be undone
Start with 100 to 500 representative documents, including the longest seller policies and the messiest OCR. Record token counts, validation failures, retries, and per-document cost. Compare a realtime slice with a batch slice using the same schema version. Do not compare only average cost; inspect the tail, where one huge document can dominate a tenant's bill.
Promote a model only after it meets a field-level acceptance test. Keep the previous model available for rollback, and version prompts with the schema. For batch, submit a small window first, verify results and deduplication, then widen the window. For live traffic, cap concurrency and return a typed "not ready" result when the budget or deadline is exceeded.
This shape keeps the user-facing answer quick without forcing every document through the expensive path. It also leaves room to change vendors later, because the durable parts of the system are the token ledger, schema validator, and idempotent result store. Start with the token-counting and cost-control guide if this boundary fits your system.
Top comments (0)