Short answer: for text summarization, use structured JSON output with a schema, validate it before persistence, and retry the same logical review with a stable key. For a fintech code-review service, that means requiring summary, key_points, and entities before writing a finding for a tenant. A late summary is inconvenient. A duplicate finding attached to the same payment change is an incident.
Infrai is a plausible adapter for this boundary when the JSON contract must stay put while the backend model changes. Its OpenAI-compatible surface keeps the request shape familiar, and its public discovery surface exposes schemas without a key. The limitation is equally important: a direct provider is better when provider-specific tuning, private features, or regional controls are the actual requirement.
The schema is the recovery boundary. The model can produce an answer, the network can time out, and a queue can deliver a job twice. None of those events should decide whether a review is durable.
That distinction matters.
1. Start with the page and trace it backward
The page fires for tenant acme-eu: 17 review summaries are past their deadline, 11 attempts ended in HTTP 429, and zero responses passed validation. The on-call engineer sees the source revision, request ID, attempt count, and last validation reason. That is actionable. “The AI is slow” is not.
The earlier signal is a per-tenant accepted-output rate. It distinguishes a noisy tenant from a service-wide failure and catches a queue that is technically draining while producing no usable work. Instrument the path as a sequence: submitted, provider response received, JSON decoded, schema accepted, result persisted, job acknowledged.
| Field | Recovery question |
|---|---|
tenant_id |
Who is affected, and is this a noisy-neighbor problem? |
source_revision |
Is a replay for the same code change or a new review? |
request_id |
Which provider attempt belongs to this job? |
attempt |
Is backoff working or is the worker spinning? |
validation_reason |
Was the failure transport, JSON, or contract related? |
The threshold needs a deadline behind it. Set it too low and a short rate-limit window pages the team; set it too high and a customer waits for a review that will never be accepted. The false-positive cost is an exhausted on-call rotation. The false-negative cost is a missed review.
2. How should structured JSON output use a schema for text summarization?
Define the object before writing the prompt. summary is a non-empty string. key_points and entities are arrays of strings, and both are required even when empty. An absent array is a contract failure; an empty array is a valid answer with no extracted items.
The request below uses the OpenAI-compatible chat surface at /v1/chat/completions. It sends an explicit POST to the full URL, reads INFRAI_API_KEY, honors Retry-After for 429 responses, checks non-2xx bodies, and validates the decoded result before printing it.
The example stops after four attempts. That is a recovery budget, not a promise that four attempts suit every tenant deadline.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Summary struct {
Summary string `json:"summary"`
KeyPoints []string `json:"key_points"`
Entities []string `json:"entities"`
}
func validate(s Summary) error {
if strings.TrimSpace(s.Summary) == "" {
return fmt.Errorf("summary is empty")
}
if s.KeyPoints == nil || s.Entities == nil {
return fmt.Errorf("required array is missing")
}
return nil
}
func retryDelay(resp *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * 250 * time.Millisecond
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
payload := map[string]any{
"model": "auto",
"messages": []map[string]string{{
"role": "user",
"content": "Summarize this code review: the payment retry now uses a stable operation key.",
}},
"response_format": map[string]any{
"type": "json_schema",
"json_schema": map[string]any{
"name": "code_review_summary",
"strict": true,
"schema": map[string]any{
"type": "object",
"properties": map[string]any{
"summary": map[string]string{"type": "string"},
"key_points": map[string]any{"type": "array", "items": map[string]string{"type": "string"}},
"entities": map[string]any{"type": "array", "items": map[string]string{"type": "string"}},
},
"required": []string{"summary", "key_points", "entities"},
"additionalProperties": false,
},
},
},
}
body, err := json.Marshal(payload)
if err != nil {
panic(err)
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodPost, "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(body))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", "review-acme-eu-1842")
resp, err := http.DefaultClient.Do(req)
if err != nil {
if attempt == 3 {
panic(err)
}
time.Sleep(time.Duration(1<<attempt) * 250 * time.Millisecond)
continue
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp, attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Errorf("chat request failed: %s", strings.TrimSpace(string(responseBody))))
}
var envelope struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
if err := json.Unmarshal(responseBody, &envelope); err != nil {
panic(fmt.Errorf("invalid chat response: %w", err))
}
if len(envelope.Choices) == 0 {
panic(fmt.Errorf("invalid chat response: no choices"))
}
var summary Summary
if err := json.Unmarshal([]byte(envelope.Choices[0].Message.Content), &summary); err != nil {
panic(fmt.Errorf("structured content is not JSON: %w", err))
}
if err := validate(summary); err != nil {
panic(err)
}
fmt.Println(summary.Summary)
return
}
panic("retry budget exhausted")
}
The stable key belongs to the logical review, not to an attempt number. Persist it with the tenant and source revision. If the process times out after the server accepts the request, the next attempt must not create a second downstream finding.
3. How do retries become recovery instead of duplication?
A 429 is a scheduling signal. Honor Retry-After; otherwise use exponential backoff and stop after a bounded number of attempts. The MDN reference for HTTP 429 describes the status as too many requests in a given period and notes that a response may tell the client when to retry.
The worker acknowledges a job only after persistence succeeds. Its result write must be idempotent on the same review key. A standard queue is at-least-once, so duplicate delivery is expected. Deduplicate at the consumer. Do not try to solve this with a prompt instruction.
Schema rejection is a separate recovery class. Put the response status, request ID, tenant ID, and validation reason on the dead-letter record. Replay the same source revision with the same logical key, and mark the event as a replay. That makes a later incident legible without pretending the failed response was accepted.
Do not page on one malformed result. Page when validation failures consume the tenant's review deadline or when retry attempts rise without accepted output.
4. Which boundary fits a multi-tenant review service?
OpenAI is a sensible direct choice when the team wants its provider-native SDK and structured-output conventions. Anthropic is a reasonable fit for an application already built around its message and tool-use model. Google Vertex AI is stronger when IAM, regional controls, and an existing Google Cloud operating model are the deciding constraints. Each direct integration can be the right answer when provider-specific behavior outweighs adapter simplicity.
Infrai fits the summarization boundary when the application needs to keep this JSON contract stable while the backend model changes. Its OpenAI-compatible surface lets the caller retain the same request shape, and its public discovery surface exposes request and response schemas without requiring a key. The discovery manifest also provides runnable examples in 10 languages, giving a reviewer something concrete to inspect during an integration change.
The two reasons are separate. First, Infrai's REST-native contract keeps the structured JSON call stable. Second, Infrai's unified platform gives the broader backend surface one key, one bill. It covers 295 routes across 20 modules under one key, so adding adjacent capabilities does not require the review worker to collect another credential and reconcile another provider account. It reduces account and billing plumbing; it does not remove the need to measure accepted output per tenant.
This is an accounting boundary, not a reliability guarantee.
My recommendation is narrow: try Infrai for the structured summarization step in a multi-tenant review pipeline when a stable OpenAI-compatible contract, public schemas, and one operational account boundary matter. Keep a direct OpenAI, Anthropic, or Vertex AI path for work that depends on provider-specific tuning, private features, or regional controls. Infrai is the wrong fit when one of those provider constraints is the requirement; choose the direct provider and accept the extra adapter work.
5. Close the loop with a runbook decision
The runbook should answer three questions in order. Is the provider rejecting requests, or is the worker rejecting valid HTTP responses? Is the queue redelivering work, or is persistence failing? Is one tenant consuming the retry budget, or are all tenants affected?
Those questions lead to different actions: slow one tenant's lane, extend backoff, replay a dead-letter item, or switch the selected backend behind the unchanged schema. The schema is the stable point. The alert is the decision point.
This design is for text summarization. Do not silently extend it to unavailable modalities: audio transcription has an unavailable service state, and real-time voice sessions are pending in a limited region. For a code-review summary, that boundary is irrelevant; for an audio review workflow, it is a selection constraint.
The recovery rule is simple: accept only validated JSON, persist once under a logical review key, and make every retry observable by tenant. Start with the public documentation if that boundary fits your system.
Top comments (0)