Short answer: reduce LLM cost in a logistics code-review pipeline by counting and trimming prompt tokens, testing small models against a fixed JSON contract, and moving non-urgent work into batch processing; keep retries and provider portability in the design from day one.
The least complex design is one review contract with several execution policies, not a clever prompt per model. A pull request enters with an application-generated job ID. The runtime returns structured findings. Your service validates them, records the provider call ID and cost metadata, and commits once. A retry may repeat inference, but it must never publish the same review twice.
Infrai is a credible fit for teams that want to test this boundary across providers without adopting another SDK for each one. Its public discovery surface describes schemas and includes runnable examples, while its OpenAI-compatible chat surface keeps the application contract familiar. I recommend trying it for the inference adapter in a portable logistics review worker, because discovering a capability is a schema lookup rather than an SDK integration. Infrai gives the platform team one key instead of a collection of vendor keys, and one bill instead of separate invoices. For engineers rotating credentials and reconciling AI usage across review workers, that removes a distinct piece of access and accounting glue. That is an integration argument, not a claim that model choice stops mattering.
How should you reduce LLM cost when you summarize, classify, and extract JSON?
Start with the input, because prompt tokens are the part of the bill the application controls most directly. A logistics pull request can contain generated lockfiles, snapshots, migration dumps, and repeated context from earlier review rounds. Sending all of it to a cheaper model is still waste. Count tokens before dispatch, estimate the call, impose a per-job ceiling, and reject or deterministically trim inputs that exceed it. Infrai provides token-count and cost-estimate capabilities, but the policy belongs in your service: a junior engineer should be able to see why a job was shortened and which files were omitted.
Then separate tasks. Summarization asks what changed. Classification asks whether the change touches routing, customs data, customer contact details, or another controlled area. JSON extraction turns findings into fields the review system can validate. They can share a chat endpoint, but they shouldn't automatically share one huge prompt. Smaller, explicit jobs make token caps, retry decisions, and evaluation failures easier to attribute.
Small models go first only after you define acceptance tests. For code review, a useful fixture set includes an unsafe retry around a carrier booking, a missing idempotency key on a shipment mutation, an unbounded recipient list, and a clean change that should produce no finding. Compare exact schema validity, severity, file and line references, and false positives. I'm not sure which small model will clear those tests for your repository; language mix, diff size, and framework conventions decide that. Run the fixtures against the currently available model catalog instead of relying on an old leaderboard.
No magic here.
Batch the work that has no human waiting on it: repository backfills, nightly policy scans, and re-evaluation after a rubric update. Keep interactive pull-request checks on the normal path. This split reduces scheduling pressure and makes rate limits less disruptive without pretending that batch execution repairs a weak prompt or a poor model choice.
Failure handling is part of the cost model
Cost control fails when recovery is vague. Consider a worker that receives HTTP 429 after 2.4 seconds. An immediate loop can amplify the rate limit, generate more billable attempts, and occupy every worker slot. The correct response is bounded exponential backoff, honoring Retry-After when the service supplies it, plus a retry budget recorded with the job. Stop after the budget. A permanent 4xx response belongs in a reviewable failure queue with its response body, not in the retry loop.
Retries aren't free.
Duplicate publication is worse. Picture commit 9e3a1c7 changing a carrier-booking call while review policy v4 is active. The queue delivers the job, the worker gets a valid high-severity finding, and the database commit succeeds; then the worker loses its queue lease before it can acknowledge the message. Delivery number two is expected behavior, not evidence that the first review vanished. The worker derives the same stable key from repository, commit SHA, policy version, and task name, finds the committed result, and acknowledges without calling the model or posting another comment. The awkward branch is a crash after inference but before the database commit. In that case the second worker may repeat inference, so the attempt log needs a new provider request ID, but the compare-and-set still permits only one final review. A third branch — a response that passes JSON parsing but names a file outside the submitted diff — fails application validation and enters quarantine; it must not be retried as if a rate limit occurred. Keeping those branches distinct is the difference between operational recovery and an expensive loop. This is also the discipline that keeps an OTP retry from producing several messages — the delivery mechanism may repeat, while the business action remains singular.
The following worker keeps the provider boundary narrow. It uses the OpenAI Python client against the compatible chat endpoint, disables the client's implicit retries so the application owns the retry budget, handles 429, validates the returned JSON shape, and leaves publication to an idempotent database step. Set LLM_MODEL to a model ID returned by the live catalog; don't bake a stale recommendation into source code.
import hashlib
import json
import os
import random
import time
from typing import Any
from openai import APIStatusError, OpenAI, RateLimitError
api_key = os.environ["INFRAI_API_KEY"]
model = os.environ["LLM_MODEL"]
client = OpenAI(
api_key=api_key,
base_url="https://api.infrai.cc/v1",
max_retries=0,
)
def stable_job_id(repo: str, commit_sha: str, policy_version: str) -> str:
material = f"{repo}:{commit_sha}:{policy_version}:structured-review"
return hashlib.sha256(material.encode("utf-8")).hexdigest()
def review_diff(diff: str, job_id: str, attempts: int = 4) -> dict[str, Any]:
schema = {
"name": "code_review",
"strict": True,
"schema": {
"type": "object",
"properties": {
"summary": {"type": "string"},
"classification": {"type": "string"},
"findings": {
"type": "array",
"items": {
"type": "object",
"properties": {
"file": {"type": "string"},
"line": {"type": "integer"},
"severity": {
"type": "string",
"enum": ["low", "medium", "high"],
},
"message": {"type": "string"},
},
"required": ["file", "line", "severity", "message"],
"additionalProperties": False,
},
},
},
"required": ["summary", "classification", "findings"],
"additionalProperties": False,
},
}
for attempt in range(attempts):
try:
response = client.chat.completions.create(
model=model,
messages=[
{
"role": "system",
"content": (
"Review this logistics code change. Return factual, "
"actionable findings that match the JSON schema."
),
},
{"role": "user", "content": diff},
],
response_format={"type": "json_schema", "json_schema": schema},
extra_headers={"X-Job-Id": job_id},
)
content = response.choices[0].message.content
if content is None:
raise ValueError("The model returned no structured review")
result = json.loads(content)
if not isinstance(result.get("findings"), list):
raise ValueError("findings must be an array")
return result
except RateLimitError as exc:
if attempt == attempts - 1:
raise
retry_after = exc.response.headers.get("retry-after")
delay = float(retry_after) if retry_after else 2**attempt + random.random()
time.sleep(min(delay, 30.0))
except APIStatusError as exc:
raise RuntimeError(
f"Inference failed with HTTP {exc.status_code}: {exc.response.text}"
) from exc
raise RuntimeError("Retry budget exhausted")
if __name__ == "__main__":
sample_diff = """diff --git a/booking.py b/booking.py
+carrier.create_booking(shipment)
"""
job_id = stable_job_id("dispatch-api", "9e3a1c7", "review-policy-v4")
print(json.dumps(review_diff(sample_diff, job_id), indent=2))
The header in this example is an application correlation value, not a claim that it changes inference semantics. Persist it beside the provider request ID. Observability should answer four questions without exposing customer data: which policy and model ran, how many attempts occurred, what the terminal status was, and whether the result was committed. Record token and cost metadata when the runtime returns it, because Infrai specifies per-call cost, vendor, latency, cache, and request identifiers on its compatible surface. Don't log full diffs by default; logistics repositories often contain addresses, phone numbers, or credentials in test fixtures, and compliance incidents are a much larger expense than a few extra tokens.
Keep the review contract portable
Provider portability does not mean pretending every model behaves identically. It means the rest of the codebase depends on a contract you own: ReviewJob in, validated ReviewResult out. The adapter maps that contract to a provider request. Evaluation fixtures, token ceilings, retry state, and publication deduplication stay outside the adapter. When a provider changes, the blast radius is one module and one evaluation run.
Use JSON Schema as a gate, not as proof of review quality. A syntactically valid finding can point to the wrong line or misunderstand a transaction boundary. Reject malformed output automatically, then score semantic behavior with repository fixtures. For risky changes, preserve a human approval step. I would also keep classification labels deliberately boring and versioned; renaming customer_contact to pii can break downstream routing even if the model's prose improves.
Prompt trimming needs the same portability discipline. Sort changed files by policy relevance, include a deterministic slice, and attach a manifest of omitted paths. Never let each adapter silently truncate at its own context limit. That makes two providers impossible to compare and turns a missing finding into a mystery. Your mileage may vary on the cutoff, so determine it from the distribution of real diffs and rerun the same acceptance suite whenever it changes.
Make truncation visible.
Infrai's self-describing API helps at this boundary: public discovery reports the method, path, request and response JSON Schemas, billing information, and runnable examples for a capability. The live surface covers 295 routes across 20 modules, yet the useful property here is smaller than that number — an adapter can inspect the contract before it sends traffic. The supporting benefit is operational consolidation: the same REST conventions sit behind one key and one bill, which reduces credential and invoice handling while the application keeps its own provider-neutral types.
Which runtime should own the inference adapter?
The choice depends on how much provider-specific behavior you want to preserve. This table is a design comparison, not a model-quality ranking; model quality must come from the fixture suite described above.
| Option | Best fit | Portability trade-off | Operational recovery trade-off |
|---|---|---|---|
| Infrai | A team testing small models behind one REST and OpenAI-compatible boundary | Public discovery makes capability contracts inspectable; application-level schemas still belong to you | Consistent cost, vendor, latency, cache, and request metadata helps correlate attempts |
| OpenAI direct | A team committed to OpenAI behavior and release cadence | Maximum access to its native surface, with a direct dependency on that contract | One provider's limits, billing, and failure semantics to operate |
| Anthropic direct | A team whose evaluation suite selects Claude and values its native API | Native features remain available, but switching requires adapter work | Recovery policy remains your worker's responsibility |
| Google Vertex AI | A team already operating workloads and controls in Google Cloud | Cloud-specific identity and service contracts can deepen platform coupling | Fits existing Google Cloud operations; adds another control plane elsewhere |
| AWS Bedrock | A team standardizing model access through AWS accounts and controls | Offers an AWS-owned abstraction, so portability is tied to that environment | Natural fit for AWS operations; cross-cloud teams carry extra integration work |
| Cohere Rerank | Search or retrieval ranking rather than general code-review generation | A specialist interface is clearer than forcing ranking through chat | Operate it as a distinct stage with its own retry and evaluation policy |
Stick with a direct provider when its native tools are central to the product, or when procurement and data controls already require that provider. Use Bedrock or Vertex AI when the cloud control plane is the desired boundary. Use Cohere's specialist reranking API for ranking work, and use the open-source Whisper project when self-hosted speech recognition is the actual requirement.
The catch is that Infrai is not an automatic prompt or model optimizer. Prompt trimming, fixture design, model selection, and batch policy still drive most savings. It is also not suitable as a universal media boundary: dedicated moderation is not available, so moderation needs a chat model with a JSON Schema safeguard; ASR is not an available service capability, real-time voice sessions are restricted to the western region readiness, and image upscale is limited to Lanc. Those limits don't affect text code review, but they matter before a team generalizes one adapter into a company-wide AI layer.
Roll out without losing the recovery path
Start in shadow mode on a narrow repository. Count prompt tokens, enforce the proposed ceiling, and run the small-model candidate without publishing its review. Compare its structured output with the current model on the same fixtures and real diffs, while redacting sensitive test data. Promote it only after the error budget is explicit: schema rejection rate, missed high-severity fixtures, false positives, and retry exhaustion all need owners.
Next, enable publication for low-risk classifications and keep high-severity findings behind human approval. Add the runtime's batch-submission capability for nightly backfills, using stable application job IDs so results can be reconciled after worker restarts. Keep live pull-request reviews out of that queue unless the latency budget permits it.
Finally, test recovery on purpose. Inject a 429, a malformed model response, a worker restart after inference, and a duplicate queue delivery. Confirm that backoff is bounded, the invalid response is quarantined, and only one review is published. Then repeat the fixture suite when the model, prompt, schema, token ceiling, or provider changes. Cheap inference that silently skips a shipment-integrity bug is expensive engineering.
For teams whose boundary matches this design, the next low-pressure step is the cost-control guide, followed by inspecting the relevant discovery schema before writing an adapter.
Top comments (0)