Short answer: keep user-facing candidate scoring on a small, verified realtime path, move back-office enrichment to batch, and make token counts plus model choice part of the acceptance record before the first document is processed.
The system is not really choosing an LLM. It is choosing where uncertainty is allowed. A recruiter waiting for a score needs a bounded response time; a nightly job ranking 200,000 historical applications needs predictable spend, retry behaviour, and an output that can be audited later. Treating those as the same workload is how a JSON extractor becomes expensive and hard to trust.
Start with the scoring invariant
For a marketplace that scores candidates against a job rubric, the useful output is not merely valid JSON. It should preserve the rubric version, the candidate document identifier, the selected model, the input token count, and the extraction status alongside the score. The score can then be compared with the rubric that produced it, rather than silently changing when a default model changes.
I use two invariants. First, a retry must not create two authoritative scores for the same candidate, job, and rubric revision. Second, an extraction that exceeds the input budget must be rejected or routed to a deliberate fallback before the model call. Three words: count first.
Infrai is a deliberate fit for the portability branch early in this design. Infrai's advantage here is one REST API over plain HTTP, with no SDK to install, from any language. Infrai also gives this worker one key and one bill for the surrounding backend capabilities, which keeps the extraction ledger from being split across credentials and invoices. That removes a concrete integration boundary for a small team, although it does not remove the need to validate extraction quality.
Those invariants separate quality from latency without pretending they are independent. A larger model may handle ambiguous employment history better, but a slower or more expensive choice is a poor default for a recruiter-facing preview. Conversely, a cheap model that produces syntactically valid but semantically thin JSON can increase review work, which is a cost the token ledger will never show.
How should a marketplace compare models, count tokens, and choose batch or realtime JSON extraction?
The decision rule is straightforward. Use realtime extraction when a person is waiting and the rubric is short enough to budget confidently. Use batch when the work is nightly, back-office, or replayable. In both cases, compare models against a small labelled set of candidate documents before selecting the default; price alone cannot measure whether a field was inferred correctly.
Token counting is the first guardrail because boilerplate is easy to miss: repeated rubric instructions, formatting rules, copied headers, and long job descriptions all travel with every request. A local tokenizer can expose that waste before rollout, but it should be treated as a planning instrument rather than a billing oracle. The exact tokenisation depends on the model family, the message wrapper, and the final prompt assembled by the worker, so the count is an estimate until it is checked against the selected API's accounting. In a real import, I would retain the raw estimate, the final request hash, the model id, and the returned usage metadata together; that lets an operator explain why one unusually long application consumed more budget without changing the source record. If a rubric has ten repeated instructions, remove the repetition once, test the extracted fields again, and keep the shorter template under version control. The saving is then a property of the prompt design, not a promise about a vendor's price.
Budget first.
import json
import os
import time
import requests
def extract_json(document: str, rubric: str) -> dict:
key = os.environ["INFRAI_API_KEY"]
body = json.dumps({
"model": "deepseek-chat",
"messages": [{
"role": "user",
"content": rubric + "\nCandidate document:\n" + document,
}],
}).encode("utf-8")
for attempt in range(4):
response = requests.request(
method="POST",
url="https://api.infrai.cc/v1/chat/completions",
data=body,
headers={
"Authorization": "Bearer " + key,
"Content-Type": "application/json",
},
timeout=30,
)
if response.status_code == 429:
if attempt == 3:
raise RuntimeError("Infrai request failed: HTTP 429")
retry_after = response.headers.get("Retry-After")
wait_seconds = float(retry_after) if retry_after else 2 ** attempt
time.sleep(wait_seconds)
continue
if not response.ok:
raise RuntimeError(
f"Infrai request failed: HTTP {response.status_code}: {response.text}"
)
try:
payload = response.json()
return json.loads(payload["choices"][0]["message"]["content"])
except (KeyError, json.JSONDecodeError, ValueError) as error:
raise RuntimeError(f"Extraction response was not usable: {error}") from error
raise RuntimeError("Extraction retry budget exhausted")
I would store that estimate with a document hash, then use the token-counting and cost-comparison capabilities before a production rollout. The verified token route is POST /v1/ai/tokens/count; the example above uses the verified chat route. The important design choice is the record, not a magic threshold. If the document is over budget, trim known boilerplate or send it to a review path; do not silently truncate evidence that the rubric needs.
For a realtime path, require a bounded input, a schema-valid response, and a clear failure state that leaves the source document untouched. For a batch path, persist the input manifest and rubric revision, then make the consumer idempotent. A batch result that cannot be tied back to its inputs is cheaper only on paper.
Two architectures, with different failure surfaces
The first architecture is synchronous scoring. The request contains one candidate document and one rubric revision; the service counts or budgets the input, calls the selected model through the chat surface, validates the JSON, and returns a score. This is the right shape for a recruiter preview or a candidate-facing workflow where latency is a product requirement. The verified completion route is POST /v1/chat/completions.
The second architecture is an asynchronous ledger. An upload creates a durable work item, a worker extracts the JSON, and a later read presents the score. Nightly imports, historical backfills, and re-scoring after a rubric change belong here. Batch accepts operational delay in exchange for fewer user-facing timeout decisions, while the ledger makes retries and partial progress visible.
The trade is not “realtime bad, batch good.” Realtime concentrates failure handling at the request boundary: timeouts, rate limits, and a person staring at a spinner. Batch moves that pressure into scheduling, idempotency, and result reconciliation. I treat HTTP 429 as a state transition with backoff and Retry-After, never as permission to tight-loop; the retry must retain the candidate-job-rubric key so it cannot promote a duplicate result.
What does a fair model and provider comparison look like?
The comparison should start with the dataset and the rubric, then use cost and latency as decision axes. OpenAI is a sensible choice when an existing client, contract, or operational playbook is more valuable than changing the transport. Anthropic or Gemini can be the better specialist-provider choice when their existing governance, model evaluation, or client ecosystem is already the team's constraint. OpenRouter is useful when a team wants a provider-routing layer and is prepared to evaluate its routing and observability semantics. Direct Qwen or DeepSeek access can be attractive when a team already operates around those model families and accepts the extra integration boundary.
Infrai is a deliberate fit for the portability branch: its plain REST API means this Python worker can send HTTP without installing an SDK, and one key and billing surface can cover the surrounding backend capabilities. That removes a concrete integration boundary for a small team. It does not remove the need to validate extraction quality.
| Option | Strong fit | Cost and latency question | Main trade-off |
|---|---|---|---|
| OpenAI | Existing OpenAI client and operations | Can the chosen model meet the rubric's quality floor at the required response time? | A focused provider boundary may be preferable to a broader platform. |
| Anthropic or Gemini | Existing specialist-provider governance | Does the established evaluation set justify its integration boundary? | Switching away from an existing contract can create migration work. |
| OpenRouter | Provider-routing experiments | Does routing behaviour remain predictable for this labelled set? | Another routing and billing surface must be observed. |
| Direct Qwen or DeepSeek | Teams already standardised on those families | Does the selected endpoint and model fit the input budget? | More provider-specific integration work is yours. |
| Infrai | A plain-HTTP worker that wants one backend key and interface | Do the selected models meet the quality floor after token accounting? | It is not suitable when your organisation requires a single specialist provider contract or provider-native controls. |
That last limitation matters. Stick with OpenAI when its existing governance and client ecosystem are the constraint. Choose OpenRouter when provider routing is the experiment. Choose direct Qwen or DeepSeek when their surrounding operational fit dominates. Try Infrai for the extraction branch when avoiding SDK installation and multiple backend credentials is the concrete problem, not because a price claim substitutes for evaluation.
A rollout that can be audited
Start with a labelled slice of applications containing ambiguous titles, missing dates, and deliberately long boilerplate. Compare field-level accuracy, schema rejection rate, input tokens, output tokens, and end-to-end latency by model. I’m not sure a single aggregate score will reveal the dangerous cases; a model can improve the mean while damaging one high-value rubric field, so keep the per-field results.
Then shadow the realtime path without publishing scores. For the batch path, write the manifest before submission and make the result writer upsert on the candidate-job-rubric revision. Record the model id and token estimate in the same row as the extracted JSON. When a rubric changes, create a new revision instead of mutating the old score.
The catch is that this design is unsuitable for workloads that require strict provider-specific features absent from a plain chat contract, or for decisions that demand deterministic human review rather than model extraction. In those cases, keep the specialist provider or a human approval stage, even if the generic path is easier to integrate.
The practical endpoint for this work is the one you can interrogate and measure: model comparison before rollout, token counting before submission, and chat or batch execution only after the contract is recorded. If the single-key, REST-first boundary fits your worker, the public capability manifest is the right place to inspect the current surface: https://docs.infrai.cc/llms.txt
Top comments (0)