A game code-review assistant has a stricter job than producing plausible prose: every finding must fit the consumer's contract and point to retrieved evidence. The choice follows from that constraint. Use semantic search to assemble a bounded evidence set, ask the chat completion for a structured answer, validate the answer against a closed schema, and reject any citation that cannot be resolved back to that set. A fluent answer is never allowed to repair a broken contract.
This is an architecture decision record for correctness, not a model comparison. The same boundary works in a Node.js service even though the critical-path example below is Python: retrieval, generation, validation, and evidence resolution remain separate ports.
What must stay true?
The primary invariant is mechanical: a successful review is a typed collection of findings, and each finding has a stable identifier, severity, explanation, and one or more citation identifiers. The model doesn't get to invent another severity, omit evidence, or return commentary beside the object. Downstream systems should be able to sort findings, annotate a pull request, or trigger a human review without scraping prose.
The evidence invariant is narrower. A citation identifies one retrieved chunk supplied to the completion request. It does not identify a URL guessed by the model, a document title copied from memory, or an offset that the application cannot reproduce. Keep chunk metadata outside the generated answer, then join citation identifiers against that trusted map after schema validation. That small separation matters — the answer can quote evidence, but it cannot define what counts as evidence.
The abstention invariant prevents a clean JSON object from becoming a cleanly packaged hallucination. The result has a status such as answered or insufficient_evidence; an answered result needs at least one valid finding, while an abstention carries no findings and states which evidence was missing. I'm not sure a universal retrieval score can decide that boundary across every game repository. Your mileage may vary because generated engine files, gameplay scripts, and security rules have very different vocabulary. Calibrate the threshold with labeled review cases, but keep the abstention shape fixed.
One more rule: untrusted repository text is data. A comment that says to ignore the review policy must not become an instruction. Delimit retrieved chunks, identify their source, and tell the completion layer to use them only as evidence. The prompting guide is useful background for designing that separation, while the enforcement still belongs in application code.
How should semantic search and chat completions produce structured answers with citations?
The critical path has four boundaries. Search selects candidate chunks; optional reranking reorders those candidates for the review question; the completion produces only the requested object; deterministic code validates both shape and evidence membership. Reranking is a relevance step, not a citation verifier. The Cohere overview describes reranking as sorting documents by relevance to a query, which is why the architecture keeps identity and authorization checks elsewhere.
For a concrete gaming example, imagine a change to an item-trading handler. The patch moves inventory before writing the transaction record, and the review question asks whether ownership checks and idempotency still hold. Search retrieves four candidates: the changed handler as ev_01, the current trading policy as ev_02, an idempotency test as ev_03, and a general inventory guide as ev_04. Their repository paths, revisions, and line ranges stay in a server-owned evidence map. The completion receives the opaque IDs and chunk text, then proposes a high-severity finding citing ev_01 and ev_03. Schema validation confirms the finding's fields, but that is only the first gate. The resolver proves both IDs belong to this exact retrieval bundle; a policy check can then require the ownership claim to cite ev_02 as well. If the answer instead cites ev_09, the entire candidate is rejected even if its explanation sounds right. If retrieval never found the trading policy, the correct result is insufficient_evidence, not a confident guess based on the handler's variable names. This example also shows why repository revision belongs in trusted metadata: a citation to yesterday's policy may resolve as an ID while still describing the wrong review snapshot. Pin the evidence bundle to the commit under review, preserve it for replay, and publish displayable paths only after every check succeeds.
No evidence, no finding.
The order is deliberate. Validate syntax and schema before resolving citations, because malformed output has no trustworthy fields. Resolve every citation before publishing any finding, because one dangling ID makes the result internally inconsistent. Then apply semantic policies that a basic schema cannot express cleanly: answered requires findings, insufficient_evidence forbids them, and duplicate findings should not create duplicate review comments. Don't silently drop the bad item and publish the rest.
Partial acceptance hides failure.
A Node.js implementation can express the same answer contract with its preferred JSON Schema validator. Keep the provider adapter behind a narrow complete(evidence, schema) interface, and keep validation in the caller. This prevents a chat completions SDK response type from leaking into the review domain. It also makes replay tests cheap: store the bounded input object and candidate output, then run the deterministic validators without calling search or generation again.
Decision options and failure boundaries
| Option | Structured output behavior | Citation behavior | Best fit | Main limitation |
|---|---|---|---|---|
| Free-form answer plus parsing | Parser infers fields from prose | Usually inferred from text or links | Internal exploration where no machine action follows | Formatting drift can become a runtime parsing failure |
| Schema-constrained answer only | Shape is explicit and closed | Citation strings may still be unresolvable | Workflows whose facts come entirely from trusted inputs | Valid JSON does not prove grounding |
| Schema plus evidence resolution | Shape is validated, then cross-field rules run | Every ID must join to the retrieved evidence map | Automated code-review findings and audit trails | More application code and stricter rejection behavior |
| Extractive spans only | Output stays close to source text | Offsets or chunk IDs are direct | Search interfaces that primarily surface passages | Weak fit for synthesized review reasoning |
The decision is the third option. Its extra application code buys a failure boundary that an operator can explain. Invalid JSON is a generation failure. A schema mismatch is a contract failure. An unknown citation is a grounding failure. Too little relevant evidence is an abstention. Those outcomes should have separate counters rather than one generic quality metric, much as SMS acceptance, carrier delivery, and OTP verification are different events. I've learned from OTP delivery gaps that an accepted request isn't proof of delivery; a schema-valid answer isn't proof of support either.
Use client errors precisely at the service boundary. For example, a caller request that fails its input contract can return 422, while an internally rejected model candidate should become a typed review outcome or a controlled retry, not a fabricated success. This distinction keeps malformed caller data out of quality dashboards and keeps model-contract failures visible to operators. Avoid logging full proprietary source chunks. Record evidence IDs, repository revision, validator codes, latency, token usage if available, and the final status; apply the same retention and access controls used for code-review artifacts.
Cost belongs in the decision, but not as a headline. Candidate count, chunk size, reranking, retry policy, and completion length all consume resources. Measure them per accepted finding and per abstention. A pipeline that retries malformed output three times may look accurate in a demo while behaving badly under pull-request bursts. Set a hard attempt limit and a total deadline, then send unresolved cases to human review. Slow down. A gaming release branch doesn't need an automated comment badly enough to justify duplicate or unsupported findings.
The critical path in Python
The code below concentrates on the part that must remain deterministic. search and complete are injected ports; neither is allowed to publish directly. The schema is represented as a Python dictionary so the example stays in one language. A production validator should implement the full contract rather than the intentionally small checks shown here.
from dataclasses import dataclass
from typing import Callable, Literal, TypedDict
class Finding(TypedDict):
finding_id: str
severity: Literal["low", "medium", "high"]
explanation: str
citations: list[str]
class ReviewAnswer(TypedDict):
status: Literal["answered", "insufficient_evidence"]
findings: list[Finding]
missing_evidence: list[str]
ANSWER_SCHEMA = {
"type": "object",
"additionalProperties": False,
"required": ["status", "findings", "missing_evidence"],
"properties": {
"status": {"enum": ["answered", "insufficient_evidence"]},
"findings": {"type": "array"},
"missing_evidence": {"type": "array", "items": {"type": "string"}},
},
}
@dataclass(frozen=True)
class Evidence:
evidence_id: str
path: str
revision: str
text: str
def validate_review(answer: ReviewAnswer, evidence: list[Evidence]) -> None:
allowed_ids = {item.evidence_id for item in evidence}
required_keys = {"status", "findings", "missing_evidence"}
if set(answer) != required_keys:
raise ValueError("review answer has unexpected or missing fields")
if answer["status"] == "answered" and not answer["findings"]:
raise ValueError("answered review must contain a finding")
if answer["status"] == "insufficient_evidence" and answer["findings"]:
raise ValueError("abstained review cannot contain findings")
for finding in answer["findings"]:
if not finding["citations"]:
raise ValueError("every finding needs evidence")
unknown = set(finding["citations"]) - allowed_ids
if unknown:
raise ValueError(f"unknown evidence IDs: {sorted(unknown)}")
def review_change(
question: str,
search: Callable[[str], list[Evidence]],
complete: Callable[[str, list[Evidence], dict], ReviewAnswer],
) -> ReviewAnswer:
evidence = search(question)
if not evidence:
return {
"status": "insufficient_evidence",
"findings": [],
"missing_evidence": ["repository policy or relevant changed code"],
}
candidate = complete(question, evidence, ANSWER_SCHEMA)
validate_review(candidate, evidence)
return candidate
Notice what the function does not do. It doesn't let the completion return repository paths as authority, doesn't coerce an unfamiliar severity into a familiar one, and doesn't convert a validation exception into an empty successful review. The adapter may retry once with the same immutable evidence bundle, but the final publication gate sees only a validated object or a declared failure outcome.
Test this path with a matrix, not a single golden response. Include valid findings, malformed top-level fields, extra properties, empty citations, unknown IDs, duplicate IDs, an answered result with no findings, an abstention that smuggles in a finding, Unicode paths, and retrieved text containing instruction-like prose. Property-based tests can vary identifiers and array sizes; replay tests can pin a repository revision and prove that the resolver never crosses evidence bundles. In deployment, shadow the pipeline on real changes before allowing it to post comments, and compare accepted findings, abstentions, validator failures, and human overrides.
Rejected option, and when it is still right
We rejected free-form generation followed by a forgiving parser for automated code review. The catch is that every repair rule expands the accepted language without strengthening evidence. If a parser maps critical-ish to high, extracts a file path from prose, and discards a citation it cannot recognize, the final object looks tidy while hiding three contract violations. That is not suitable when findings can block a game release, create security work, or become part of a compliance record.
Stick with free-form output when the result is a private brainstorming aid, a human reads the entire response, no downstream action depends on field stability, and losing citation precision is an accepted trade-off. Extractive search is also a better choice when the user mainly needs passages rather than synthesized findings. A schema-plus-resolution pipeline imposes more code, more rejection cases, and more operational metrics; for a low-risk internal search box, that overhead may be unjustified.
The final decision rule is plain: automate only the object you can validate and the evidence you can resolve. Everything else should abstain or wait for a person.
Top comments (0)