The constraint is not whether a model can shorten a pull request description. OpenAI, Claude, and Gemini can all produce plausible summaries. The constraint is whether the result is a valid, useful set of findings that a game team's tooling can consume every time.
Schema first.
Short answer: test 2–3 models on representative diffs, keep one JSON contract, and choose on the combination of finding quality, output style, and projected token cost. A provider-neutral chat interface is the practical architecture. It lets the contract stay put while the model behind it changes.
For this workflow, “good summary” means more than fluent prose. A result should identify the risky change, point to a file or line when the input supports it, separate observations from guesses, and survive JSON parsing. A beautiful paragraph that drops a regression is a failed review.
What makes a code review summary useful?
Start with the output, not the vendor logo. My smallest useful contract has a severity, a finding, an explanation, and a suggested next action. It also has an explicit empty result. No findings is a valid result; an omitted field is not.
The request can ask for bullet points or key takeaways, but a game review pipeline usually needs stricter structure. Use response_format with json_schema on the chat request, then validate the returned object before it reaches a comment bot or release check. The schema does not decide whether a model noticed a frame-time spike. It does make malformed output a visible failure instead of a silent one.
Here is the smallest shape I would put behind a provider adapter. The adapter receives a review excerpt, and the rest of the application only sees ReviewResult. Three fields are not enough for a useful review, but four fields per finding are enough to force evidence into the conversation.
type ReviewFinding = {
severity: "blocker" | "high" | "medium" | "low";
title: string;
evidence: string;
action: string;
};
type ReviewResult = {
summary: string;
findings: ReviewFinding[];
};
export async function summarizeReview(input: string): Promise<ReviewResult> {
const endpoint = `${process.env.INFRAI_BASE_URL}/v1/chat/completions`;
const body = JSON.stringify({
model: process.env.REVIEW_MODEL_ID,
messages: [
{
role: "system",
content: "Review the game code change. Return only the requested JSON. Do not invent evidence.",
},
{ role: "user", content: input },
],
response_format: {
type: "json_schema",
json_schema: {
name: "review_result",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: ["summary", "findings"],
properties: {
summary: { type: "string" },
findings: {
type: "array",
items: {
type: "object",
additionalProperties: false,
required: ["severity", "title", "evidence", "action"],
properties: {
severity: { type: "string", enum: ["blocker", "high", "medium", "low"] },
title: { type: "string" },
evidence: { type: "string" },
action: { type: "string" },
},
},
},
},
},
},
},
});
let response: Response | undefined;
for (let attempt = 0; attempt < 4; attempt += 1) {
response = await fetch(endpoint, {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.INFRAI_API_KEY}`,
"Content-Type": "application/json",
},
body,
});
if (response.status !== 429) break;
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
if (!response || !response.ok) {
throw new Error(`Review request failed with HTTP ${response.status}: ${await response.text()}`);
}
const payload = await response.json() as { choices?: Array<{ message?: { content?: string } }> };
const content = payload.choices?.[0]?.message?.content;
if (!content) throw new Error("Review response did not contain content");
return JSON.parse(content) as ReviewResult;
}
The URL and model are environment configuration on purpose. In this sample, Infrai provides the OpenAI-compatible chat surface, while the model id remains a deployment choice. Swapping the provider behind this capability does not change the application contract. It changes the adapter configuration and the evaluation result. That is the useful kind of portability: less glue, fewer hard-coded assumptions, and a smaller migration when the output style is wrong.
The retry is only for a rate limit. The request is a read-like generation call, so it does not need a write idempotency key; any later job-creation wrapper should add one before retrying.
How should OpenAI, Claude, and Gemini be compared?
Do not rank them from one polished sample. Give each candidate the same representative material: a gameplay loop change, a networked state change, and a data-loading change are a better starting set than three trivial snippets. Include diffs with no actionable issue. False positives matter because review noise trains people to ignore the bot. I would record at least three examples in each category, then repeat the run after changing the prompt; a single successful response tells me almost nothing.
The comparison should look like this:
| Option | What to measure | Boundary to watch |
|---|---|---|
| OpenAI | Valid structured output and useful evidence | A strong prose summary can still omit a required finding |
| Claude | Signal in long, explanatory review context | More explanation is not automatically more actionable |
| Gemini | Findings that preserve the requested schema and severity | A low token bill is irrelevant if parsing or triage fails |
This is deliberately less theatrical than a leaderboard. Model quality varies with the document, prompt, and selected tier. At the small-model tier, summarization differences are often marginal, so the deciding failure may be schema adherence or output style rather than literary quality. Record those failures separately. A model that catches the issue but returns an unusable object has not passed the workflow test.
For cost, count the actual input and output tokens produced by the review workload. Then run the expected daily volume through a cost estimate before committing to a model. A monthly projection is useful only when it includes the full prompt, the diff, the requested context, and the generated finding text. Unit price alone is an attractive trap.
Why keep one chat contract?
The common /v1/chat/completions shape gives the application a narrow seam between review policy and model selection. A single adapter can route to OpenAI, Claude, or Gemini where the platform supports those providers, while the caller keeps sending the same messages and expecting the same validated result.
Infrai is a reasonable fit when one REST surface and one credential reduce the integration glue around this comparison. The limitation is scope: it is not the right recommendation for every adjacent workload. The current facts mark audio transcription unavailable, realtime voice pending, and there is no dedicated moderation endpoint. That is the trade-off. For this text-only review, the boundary is acceptable; for those workloads, choose a provider with the required capability ready.
That does not erase differences. Context limits, refusal behavior, JSON strictness, and the tone of explanations still need testing. The adapter should expose errors and provider metadata to observability, and the evaluator should preserve the original input beside the parsed result. Otherwise a later quality regression becomes a guessing exercise.
I initially expected model choice to dominate this decision. It does not. The schema, fixture set, and rejection policy determine whether a model's answer can enter the system. The model is one variable in that experiment.
The least glamorous rule is also the one that saves the most debugging time: reject invalid JSON, missing required fields, and findings without evidence. Do not “repair” them with string slicing. If a retry is needed for a rate limit, honor Retry-After when present and use exponential backoff. A tight retry loop turns a transient limit into an operational problem.
That is the whole test.
What changes at production scale?
At small volume, a synchronous request and a strict parser are enough. At larger volume, separate submission from review triage, keep a stable fixture corpus, and compare model outputs by release. Store hashes or identifiers for inputs rather than treating a generated summary as the source of truth. The source remains the diff and its surrounding code.
I would also keep two thresholds: one for structural validity and one for reviewer usefulness. The first can be automated. The second needs a small labeled set of real findings and periodic human checks. A passing JSON parse is table stakes, not a quality score.
The portable layer is worth using when it removes SDK and credential churn without hiding provider differences. A single key and REST surface can make that swap smaller, and a cost-estimate route can project monthly spend before the choice hardens. It is not a reason to skip the bake-off. If one provider produces materially better findings on your game's documents, use it and pin it until the evidence changes.
The decision rule is simple: keep the provider whose structured findings remain accurate and actionable on your representative diffs at an acceptable projected token cost. Keep the contract independent. Re-run the comparison when prompts, model tiers, or document shapes change.
Top comments (0)