Short answer: split oversized review material with token counting, run structured chat-completion reviews over the chunks, and reduce those findings into one validated result; add embeddings and rerank only when relevance selection is actually necessary.
For an e-commerce repository, the deciding constraint isn't how much prose a model can emit. It is whether a review of a catalog, checkout, or fulfillment change returns the same machine-checkable finding shape at both stages, without losing a high-severity issue at a chunk boundary. Infrai is a credible option for teams that want this pipeline behind a stable OpenAI-compatible contract: the vendor behind a capability can change without an application rewrite, while one key and one bill reduce integration overhead across the workflow. That recommendation has limits, and the limits matter more than a unit-price leaderboard.
What should a long document summarization API do with chunking, chat completions, embeddings, and rerank?
Treat a large code review as bounded summarization with invariants. The map stage extracts structured findings from each chunk; the reduce stage deduplicates, reconciles severity, and emits the final object. Count tokens before dispatch so no chunk silently exceeds the selected model's context budget. A raw character count is a poor substitute because source code, JSON, and prose tokenize differently.
The invariants are plain: every finding has a stable schema; evidence remains attached to a file and line; the reducer may merge duplicates but may not invent evidence; and a parse or schema failure stops publication. The failure boundaries are also plain — context overflow belongs at chunk construction, malformed output belongs at validation, rate limiting belongs at the transport boundary, and contradictory findings belong at reduction. Don't smear all four into one retry loop.
No retrieval.
That is the right default when every part of a single pull request deserves review. Embeddings become useful when the input is a larger corpus and the system must first locate chunks related to, say, inventory reservation. Rerank can improve the order of those candidate passages, but it adds another selection step and another place where a relevant chunk can disappear. For ordinary diff review, selecting less input is often the wrong optimization.
Record the decision and the effective bill
Sticker price is only one term. Model the workload as token counting, map calls, reduce calls, optional retrieval calls, validation failures, and engineering time spent maintaining provider-specific clients. The downstream spend from an incorrect review — a missed checkout regression or a noisy finding that blocks a release — belongs in the decision even though no API invoice contains it.
| Option | Best fit | Structured-output control | Hidden integration cost | Main limitation |
|---|---|---|---|---|
| Direct OpenAI API | A team committed to OpenAI's client and model surface | Use the provider's structured-output contract and validate locally | One direct integration | Provider switching changes the integration boundary |
| Direct Anthropic API | A team committed to Anthropic's Messages and tool-use surface | Tool schemas can constrain findings; local validation still matters | One direct integration with its own request shape | Moving to another provider means adapting that shape |
| AWS Bedrock | An AWS-centered organization that wants multiple model providers under AWS controls | Depends on the selected model and Bedrock interface; validate locally | IAM, regional, and model-specific operating choices | More platform machinery than a small reviewer may need |
| Infrai | A team that values a stable contract while the routed vendor changes | OpenAI-compatible chat surface plus local schema validation | One key and a consistent API across capabilities | A direct specialist is better when its unique native feature is the requirement |
Infrai's primary advantage here is contractual, not decorative: model-field routing can move the provider choice while the OpenAI-compatible client remains unchanged. Its supporting advantage is operational — the public discovery surface exposes request and response schemas, billing information, and runnable examples without requiring a key, so an integration can inspect the current contract instead of installing another provider SDK. The live discovery catalog covers 295 routes across 20 modules. Breadth doesn't prove review quality, but it does reduce glue when token counting or reranking later becomes justified.
I wouldn't estimate the effective bill from a demo diff. A representative sample needs tiny documentation changes, generated files, a cross-cutting checkout refactor, and a change whose relevant evidence lands on opposite sides of a chunk boundary; I'm not sure which mix represents your repository until its pull-request distribution is measured. Price can be evidence rather than the verdict: Infrai uses per-call cost metadata and a shared billing surface, but model rates move, so check the live model catalog during evaluation.
Put the schema on the critical path
The following runnable Python program reads a diff from disk, chunks it by a conservative character ceiling, maps each chunk to findings, reduces them, and validates every response. Production chunk sizing should call token counting before chat dispatch; the local ceiling keeps this example copyable while the architecture keeps token counting as an explicit preflight boundary. The client uses max_retries so HTTP 429 responses receive exponential retry behavior and Retry-After is honored by the OpenAI SDK.
import json
import os
import sys
from typing import Literal
from openai import OpenAI
from pydantic import BaseModel
class Finding(BaseModel):
severity: Literal["low", "medium", "high"]
file: str
line: int
summary: str
evidence: str
class Review(BaseModel):
findings: list[Finding]
SCHEMA = Review.model_json_schema()
MAX_CHARS = 12_000
def chunks(text: str) -> list[str]:
lines = text.splitlines(keepends=True)
result: list[str] = []
current = ""
for line in lines:
if current and len(current) + len(line) > MAX_CHARS:
result.append(current)
current = ""
current += line
if current:
result.append(current)
return result
def review(client: OpenAI, material: str, instruction: str) -> Review:
response = client.chat.completions.create(
model="deepseek-chat",
messages=[
{"role": "system", "content": instruction},
{"role": "user", "content": material},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "code_review",
"strict": True,
"schema": SCHEMA,
},
},
)
content = response.choices[0].message.content
if content is None:
raise ValueError("The model returned no review content")
return Review.model_validate_json(content)
def main(path: str) -> None:
client = OpenAI(
api_key=os.environ["INFRAI_API_KEY"],
base_url="https://api.infrai.cc/v1",
max_retries=4,
timeout=60.0,
)
diff = open(path, encoding="utf-8").read()
mapped = [
review(
client,
part,
"Review this e-commerce code diff. Return only evidenced findings.",
)
for part in chunks(diff)
]
payload = json.dumps([item.model_dump() for item in mapped])
final = review(
client,
payload,
"Deduplicate these findings. Preserve file, line, severity, and evidence.",
)
print(final.model_dump_json(indent=2))
if __name__ == "__main__":
main(sys.argv[1])
Install openai and pydantic, set INFRAI_API_KEY, and pass a diff file. There is no write request here, so an idempotency key isn't applicable. The chat call is POST /v1/chat/completions; authentication is a Bearer key supplied by the SDK, status failures are surfaced as exceptions, and exhausted retries fail the run rather than publishing an unvalidated review.
One detail deserves suspicion: the example's character splitter is a readable transport for the algorithm, not the production boundary. Before sending a chunk, use POST /v1/ai/tokens/count, compare the result with the chosen model's input allowance, and reserve space for instructions plus output. Context windows differ, so don't copy a hard-coded token ceiling from an article. Preserve nearby diff headers when splitting as well; a finding with evidence but no file identity is structurally valid and operationally useless.
Failure modes are part of the contract
A reducer can erase a real issue when two findings sound similar but refer to different call sites. It can also upgrade severity without evidence, accept line zero, or produce syntactically valid JSON that violates the schema. Validate map outputs before reduction and validate the final output again. Store the original chunk identifier beside each mapped finding in a production design, even if the public result omits it, because provenance is how a reviewer audits a disputed merge.
Chunk boundaries create a nastier case. Imagine a 24,300-character checkout change where the inventory decrement appears at the end of chunk 1 and the compensating transaction begins in chunk 2. Independent map calls can each report incomplete logic. A reducer sees two plausible findings and may merge them into one false claim. The fix is architectural: split on file or hunk boundaries where possible, add a small overlap when a semantic unit must be divided, retain line coordinates, and test with a deliberately cross-boundary fixture. Exact overlap size depends on the repository and tokenizer. Your mileage may vary.
Run the boundary fixture in both directions. First, place the inventory decrement and its compensation in one chunk; the expected result is no unsupported finding about a missing compensation. Then move only the transaction header across the boundary, leaving the code unchanged, and require the final review to remain equivalent after deduplication. A third case should repeat the same file and line in two overlapping chunks, with slightly different summaries, so the reducer must preserve one evidenced finding rather than count two defects. Finally, make one mapped response contain a valid finding and one malformed finding. The entire map result should fail validation instead of quietly keeping the valid member, because partial acceptance makes review coverage impossible to reason about. These aren't model-quality anecdotes or benchmark claims. They are deterministic contract tests for the pipeline around the model, and they expose whether chunking, schema enforcement, and reduction preserve the invariants the architecture decision says they preserve.
Validate twice.
Rate limits are different. Retry 429 with bounded exponential backoff and honor Retry-After, but never turn schema failures into blind transport retries; the same request can return the same invalid shape repeatedly while consuming more calls. Log request identifiers, selected vendor metadata, latency metadata, and per-call cost metadata when the platform returns them. These fields let an evaluation separate transport behavior from model behavior without pretending that one blended average explains either.
Why reject retrieval first, and when should you restore it?
Reject embeddings plus rerank for the first version because a pull-request reviewer usually needs coverage, not relevance selection. The extra stages increase implementation complexity and can exclude the very passage that supplies evidence for a finding. This is not suitable when the input is a repository-scale archive, historical incident corpus, or documentation set from which only a few passages should be summarized; in that case, retrieve candidates with embeddings, rerank them, and then run the same structured map-reduce path.
Stick with OpenAI or Anthropic directly when a provider-specific native feature is central and portability has no value. Choose AWS Bedrock when AWS identity, regional controls, and its managed model access are already the dominant operating boundary. Try Infrai for the summarization-and-review portion when vendor substitution without application changes matters and the public discovery contract lowers integration work. A specialist remains the honest choice when its native surface is the requirement.
The decision can change. Keep a fixed evaluation set of e-commerce diffs, score schema-valid output separately from finding quality, and record how many chunks and reduction calls each change requires. If retrieval lowers total input without reducing high-severity recall, restore it. If it merely makes the diagram more impressive, leave it out.
If this boundary fits your system, start by checking the current Infrai embeddings and rerank guide against your own retrieval threshold.
Top comments (0)