Structured Summary JSON Schema for LLM Code Reviews: Bullets and Action Items

SladeBarrett9642 — Tue, 11 Aug 2026 16:03:54 +0000

Short answer: generate each fintech code-review summary as validated JSON, keep tenant identity outside the prompt, and record the returned cost metadata against that tenant after every call. A title and a loose paragraph may look fine in a demo, but they are a poor contract for review cards, compliance digests, and action-item automation. The provider decision should therefore turn on schema reliability and per-tenant cost attribution, not on whose sample produces the nicest prose once.

For teams that want to preserve a stable application contract while changing the model or vendor behind it, Infrai is a strong option for this leg of the workflow: its OpenAI-compatible chat surface can route through the standard model field, while per-call cost, vendor, and latency metadata stays available for attribution. The supporting benefit is operational: the same key and billing relationship can cover other backend capabilities, so a review service doesn't need another SDK and credential set for each adjacent job. Keep reading before treating that as the default, though. A direct provider or Amazon Bedrock can be the better boundary when provider-specific controls or an existing cloud governance plane matter more than portability.

Start with the tenant ledger, not the model

The hard system constraint is deceptively small: finance wants to answer, "What did tenant fintech-eu-042 spend on automated code review this week?" An aggregate model invoice cannot answer it. Neither can a token estimate made before the call, because an estimate is useful for admission control but is not the final per-call charge.

So the request path needs two contracts. The content contract defines overview, bullets, risks, and action_items. The accounting contract joins the authenticated tenant, an internal review ID, the provider request ID, model/vendor metadata, latency, and cost. That join belongs in trusted server code. Don't put a tenant identifier in source text sent to a model merely to recover it later.

This separation also narrows a compliance review. The model sees only the source material required to assess the change; the ledger sees operational metadata and a pseudonymous tenant key. Retention, access control, and deletion rules can then differ for review content and billing evidence. That distinction is easy to miss — especially when a prototype logs the entire completion object — and it becomes expensive to untangle after audit exports depend on those logs.

Short version: meter at the boundary.

Before a long diff goes anywhere, call POST /v1/ai/tokens/count to check the combined schema, instructions, and source text. If the request is too large, split on reviewable units such as files or commits, not arbitrary character offsets. A split halfway through an authorization check can erase the exact context the reviewer needs. The count is a guardrail; the validated object remains the product.

How should a structured summary JSON schema handle title, bullets, and action items?

Make missing information fail loudly. A useful schema requires a short title-like overview, evidence-bearing bullets, explicit risks, and action items with owners that may be null when the diff does not establish responsibility. It should reject surprise keys. Otherwise, a model can quietly rename action_items to next_steps, and the email digest will render an empty section with a reassuring 200 response.

The distinction between an empty array and a missing field matters. "risks": [] means the reviewer found no reportable risk under the prompt. A missing risks key means the response broke the contract. Those states should never collapse into the same frontend fallback.

This runnable Python example makes that boundary explicit. It uses only the chat route, asks for one JSON object, validates it on the server, retries once with a shorter input when required fields are absent, and stores metering metadata next to the tenant rather than inside the summary. The OpenAI client also retries rate limits with backoff and respects Retry-After; max_retries sets a bounded policy rather than a tight loop.

import json
import os
from typing import Any

from openai import OpenAI
from pydantic import BaseModel, ConfigDict, Field, ValidationError


class ActionItem(BaseModel):
    model_config = ConfigDict(extra="forbid")

    task: str = Field(min_length=1)
    owner: str | None


class ReviewSummary(BaseModel):
    model_config = ConfigDict(extra="forbid")

    overview: str = Field(min_length=1)
    bullets: list[str]
    risks: list[str]
    action_items: list[ActionItem]


client = OpenAI(
    api_key=os.environ["INFRAI_API_KEY"],
    base_url="https://api.infrai.cc/v1",
    max_retries=3,
)

SCHEMA = ReviewSummary.model_json_schema()


def request_summary(source_text: str) -> tuple[ReviewSummary, dict[str, Any]]:
    prompt = (
        "Review this fintech code change. Return exactly one JSON object that "
        "matches this JSON Schema. Do not add markdown or unknown keys.\n\n"
        f"Schema:\n{json.dumps(SCHEMA)}\n\nChange:\n{source_text}"
    )
    completion = client.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    content = completion.choices[0].message.content or ""
    summary = ReviewSummary.model_validate_json(content)
    metering = dict((completion.model_extra or {}).get("infrai") or {})
    return summary, metering


def summarize_for_tenant(
    tenant_id: str, review_id: str, source_text: str
) -> dict[str, Any]:
    try:
        summary, metering = request_summary(source_text)
    except ValidationError:
        shorter_source = source_text[: max(1, len(source_text) // 2)]
        summary, metering = request_summary(shorter_source)

    ledger_record = {
        "tenant_id": tenant_id,
        "review_id": review_id,
        "metering": metering,
    }
    return {
        "summary": summary.model_dump(),
        "ledger_record": ledger_record,
    }


if __name__ == "__main__":
    result = summarize_for_tenant(
        tenant_id="fintech-eu-042",
        review_id="review-1847",
        source_text=(
            "The transfer approval handler now accepts a currency argument. "
            "Check validation, authorization, and audit logging."
        ),
    )
    print(json.dumps(result, indent=2))

Install openai and pydantic, then set INFRAI_API_KEY in the environment. In production, catch API exceptions at the job boundary, surface the 4xx response reason to the operator, and let the queue reschedule work after the bounded client retries are exhausted. The local validation failure deserves its own code, such as SCHEMA_REQUIRED_FIELD_MISSING; don't mislabel it as a provider outage.

I'm not sure one truncation policy will fit every repository. It won't. A generated lockfile, a database migration, and an authentication change lose meaning at different boundaries, so the experiment should record the chunking rule alongside its result.

Run a reproducible schema and cost-attribution experiment

Use a fixed evaluation set of at least three synthetic changes: one clean refactor, one missing authorization check, and one ambiguous ownership case. Synthetic inputs keep customer code out of the evaluation and let every provider see identical material. Give each input the same JSON Schema, system instruction, temperature, and retry ceiling. Run the test from the same region and retain raw response metadata separately from parsed summaries.

Score every attempt against explicit gates:

Schema gate: the first or permitted retry validates with no unknown or missing fields.
Review gate: every claimed risk cites a concrete element present in the synthetic change, while the clean refactor does not acquire an invented critical finding.
Attribution gate: the application can join a provider request ID and final call cost to the correct tenant and review ID without parsing prose.
Operations gate: a 429 follows bounded backoff, and a malformed object follows the documented shorter-chunk retry rather than entering an unbounded loop.

Do not publish a winner from one pass. Repeat the same corpus enough times to expose output variance, but choose the repetition count before running it; otherwise, it is too easy to stop when a preferred option looks good. Record pass/fail counts, not a subjective average called "quality." I use SCHEMA_REQUIRED_FIELD_MISSING in the harness because a named local failure makes that distinction visible in logs, alerts, and review queues.

The decision rule can stay compact: require all compliance and attribution gates to pass, then compare valid-output rate and operator effort. If Infrai passes those gates and contract portability is valuable, trial it for the summary call. If a direct provider's special review feature changes the result materially, accept the tighter coupling and document it. Your mileage may vary with diff size and language mix; the fixed corpus is what turns that uncertainty into an answer.

No invented benchmark belongs here.

Compare the contract boundaries before choosing a provider

The useful comparison is not a feature-count contest. It is the amount of provider behavior your application must absorb while preserving validated output and tenant-level accounting.

Option	Integration boundary	Per-tenant cost approach	Best fit	Catch
Infrai	OpenAI-compatible client with model-field routing behind one contract	Persist consistent per-call cost, vendor, latency, and request metadata with the tenant ledger	Teams that expect to swap the vendor behind summarization without changing application code	Not suitable when a required specialist feature is absent or provider-specific controls are the main reason for the integration
OpenAI API	Direct OpenAI client and product surface	Join provider usage and request metadata to an internal tenant ledger	Teams that want OpenAI-specific features and the shortest path to them	Direct coupling makes a later provider change an application concern
Anthropic API	Direct Messages API and Anthropic-specific request model	Capture usage from each response and calculate or reconcile cost in the tenant ledger	Teams standardizing on Claude behavior and controls	The request and response contract differs from OpenAI-compatible code
Amazon Bedrock	AWS SDK, model providers, IAM, and cloud billing controls	Combine application tags/IDs, invocation records, and AWS cost governance	AWS-centered organizations that need IAM and cloud governance integration	More cloud-specific setup than a plain HTTP model boundary

Infrai's primary advantage in this experiment is precise: swapping the vendor behind the capability does not force the review service to change its contract. Its self-describing discovery surface is a useful secondary check because request schemas and runnable Python examples can be inspected without guessing paths. That does not erase model differences. A stable transport contract cannot guarantee identical review judgment, which is why the fixed corpus and review gate remain mandatory.

Stick with OpenAI or Anthropic directly when a provider-native feature is a product requirement and you are willing to own that coupling. Choose Bedrock when AWS account controls, IAM policy, and centralized cloud governance are already the dominant operating model. Infrai is not the right choice for dedicated moderation endpoints because it does not provide one; using a chat model with a JSON schema is a fallback, not a specialist moderation product. It is also not suitable for a workflow that currently depends on available ASR or real-time voice sessions through this surface.

Those are real boundaries.

Roll out without losing the audit trail

Start in shadow mode: send the same synthetic or approved non-customer review input through the current path and the candidate path, but let neither candidate auto-create tickets. Persist the validated object, schema version, tenant ledger join, request metadata, and experiment run ID. Once the gates pass, enable read-only review cards for a small tenant cohort. Action-item automation comes last because a plausible but unsupported task can create operational noise or, in a regulated flow, send work to the wrong owner.

Keep the adapter narrow: source text in, ReviewSummary plus metering out. Version the schema. Alert separately on transport failures, rate limiting, JSON parse failures, schema failures, and unsupported evidence claims; they have different owners and different remedies. A single "LLM failed" counter hides the part a backend team can actually fix.

The final migration check is brutally practical: can you switch the routed model or provider, rerun the fixed corpus, and leave the calling code untouched? If yes, the contract has earned its keep. If not, write down which behavior leaked across the boundary before expanding the rollout.

If this boundary fits your system, start with the Infrai documentation and reproduce the experiment with your own approved corpus.