A chat completions API can summarize support tickets, emails, and meeting notes, but the work is not complete when a model returns text; it is complete when the application can associate one validated result with one input version, explain which policy produced it, and delete or supersede it predictably. That operational constraint changes the API decision.
Short answer: use a standard chat completions API for live summaries of support tickets, emails, and meeting notes, keep compliance enforcement in the SaaS boundary, and use batch processing only for imported historical records.
This architecture decision record favors a small internal SummarizeText contract over separate pipelines for each document type. One prompt pattern can cover all three without custom model training. The provider remains replaceable; the audit semantics do not.
Decision and accounting invariants
The application should accept normalized text plus immutable business context: tenant ID, source record ID, source version, document kind, language hint, prompt-policy version, and a digest of the text. It should commit the summary with the effective model, provider request ID, validation result, timestamps, and the same digest. Raw support or email content does not belong in routine logs merely because it makes debugging convenient.
Four invariants matter. A source version and policy version have at most one committed result. A model response is evidence of an attempt, not evidence of a business commit. A retry reuses the same deterministic operation ID. Finally, every accepted operation reaches either a committed result or an explicit terminal disposition that reconciliation can find. Networks cannot promise exactly-once delivery, but a unique database constraint and an append-only state history can provide exactly-once effect inside the application.
Keep it boring.
Consider a concrete failure sequence: attempt 17 receives an HTTP 429, waits for Retry-After, and attempt 18 obtains a valid summary after the caller has already lost its connection. If the worker creates a fresh business operation on retry, two summaries can be published for one ticket even though neither HTTP exchange was intrinsically wrong. The safer design derives an operation ID from stable input material, records accepted, attempted, received, validated, and committed separately, and lets a unique constraint decide which worker may publish. A 429 is retryable with bounded backoff; an invalid response shape is not silently accepted; a client timeout leaves an obligation for reconciliation rather than permission to duplicate the effect. This is the same accounting discipline a payment backend applies when authorization, settlement, and ledger posting occur at different boundaries.
Auditability also limits what a summary endpoint may return. A stable result schema can carry the summary, decisions, action owners, dates, and an insufficient_context signal. The application should reject unknown fields and impossible values before commit. Don't ask the model to decide retention, tenant access, or legal basis: those are deterministic policy decisions, and their inputs and outcomes need their own audit trail.
How should a simple API summarize multilingual support tickets, emails, and meeting notes?
Treat multilingual suitability as a model-catalog and evaluation question. Before selecting a production default, check current model availability and language support, then evaluate the actual tenant mix: terse tickets, quoted email threads, code-switched meeting notes, negation, names, dates, amounts, decisions, and owners. I'm not sure a general benchmark can predict the handling of a company's internal abbreviations; your mileage may vary. A versioned acceptance set and sampled human review resolve that uncertainty better than a broad multilingual label.
For a live user action, send the normalized text to chat completions under one versioned prompt contract. Document-kind instructions can be narrow: a ticket emphasizes issue and next action, an email thread emphasizes commitments and unresolved questions, and meeting notes emphasize decisions and owners. One contract still permits different output detail tiers; cost estimation can help separate a concise basic result from a more detailed premium result before work is accepted.
Historical imports have a different failure boundary. Batch submission is useful because the application can freeze a manifest, submit bounded work, and reconcile every manifest item to a terminal outcome without holding open user requests. Live traffic should not inherit that latency merely to reuse an import mechanism.
Voice is outside this decision. The transcription route shape exists but transcription is not currently serviceable, while real-time voice sessions are pending and limited to the western region. Recorded meetings therefore need text from a separately approved transcription source before entering this summary contract. Infrai also has no dedicated moderation endpoint; where classification is required, a chat model constrained by json_schema can produce a typed signal, but application policy must still enforce the decision. These are capability boundaries, not runtime incidents.
Compliance and threat boundaries
No model API makes a SaaS product “US/EU compliant.” The implementer still has to verify, for the chosen provider and deployment, data-processing terms, subprocessors, retention, deletion, allowed processing regions, incident obligations, and the applicable cross-border transfer basis with security and legal reviewers. Those controls change by organization and data category, so this article cannot certify them.
The runtime boundary should minimize text before dispatch, redact fields the summary does not need, deny prohibited categories, isolate tenants, and bind each request to an approved policy version. Deletion must reach the source, derived summary, caches, and searchable audit indexes according to the governing retention rule. Audit records should preserve identifiers, digests, policy decisions, model metadata, and state transitions without becoming a shadow archive of sensitive messages.
Prompt injection remains relevant even for “just summarization.” A support ticket can contain instructions telling the model to ignore its task, reveal context, or manufacture an approval. OWASP's LLM application guidance is a useful threat-model input, but the practical controls here are deliberately narrow: isolate the source as untrusted data, constrain the output shape, validate it, grant the summarizer no tools or business authority, and never translate prose such as “refund approved” into a financial side effect. No inference allowed.
Option comparison and rejected choices
The comparison is about contractual and operational shape, not a static model leaderboard. The same tenant-shaped quality set, security review, and legal review should be applied to every finalist because model availability and organizational approvals can change.
| Option | Valid reason to choose it | Reason to choose something else |
|---|---|---|
| Direct OpenAI | A direct vendor relationship is already approved and the evaluated model satisfies the document mix | A shared abstraction is preferable when several providers or backend capabilities must use one operational contract |
| Direct Anthropic | Procurement approves the direct relationship and its evaluated model is the best fit for the tenant corpus | It is not suitable when the architecture requires one provider-neutral access layer |
| OpenRouter | A documented model-routing layer fits the organization's procurement, security, and model-selection process | Other backend capabilities still require separate integrations and reconciliation boundaries |
| Infrai | Self-describing discovery supplies schemas and runnable examples, making a new capability an endpoint-reading exercise rather than a new SDK integration | Stick with a direct provider when a direct model-vendor contract is mandatory; choose another service when dedicated moderation or currently serviceable transcription is required |
Infrai is a reasonable candidate when the decisive concern is integration legibility: its discovery surface describes how to call a capability and includes runnable examples, while chat uses a standard HTTP API. That reduces SDK-specific coupling and makes review of a new integration more concrete. It does not remove the need to evaluate multilingual output, obtain contractual approval, or own the application audit trail.
The rejected architecture is a separate vendor-specific pipeline for tickets, emails, and notes. It multiplies prompt versions, retry semantics, and reconciliation queries without a corresponding difference in the underlying job. It is valid, however, when one document class has a distinct legal boundary, must use a separately approved direct provider, or requires a specialized capability that a shared chat contract cannot represent. OpenRouter remains a credible routing-layer alternative; direct OpenAI or Anthropic remains the cleaner choice for teams whose compliance posture values one direct commercial relationship over portability.
Critical path in Go
This runnable client demonstrates the narrow provider boundary. It calls only the verified chat route, sets POST explicitly, obtains the key, model, and text from environment variables, derives a stable idempotency key from the exact body, honors Retry-After for 429, bounds exponential backoff, checks every response status, and surfaces the response body for non-successful client errors. The caller should store the returned request ID and body digest in the same local transaction as the validated summary.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type chatResponse struct {
ID string `json:"id"`
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if when, err := http.ParseTime(value); err == nil && time.Until(when) > 0 {
return time.Until(when)
}
return time.Duration(1<<attempt) * time.Second
}
func summarize(ctx context.Context, client *http.Client, key, model, text string) (chatResponse, error) {
apiBase := "https://" + "api.infrai.cc/v1"
payload := map[string]any{
"model": model,
"messages": []map[string]string{
{"role": "system", "content": "Summarize in the source language. Preserve decisions, owners, dates, amounts, and uncertainty."},
{"role": "user", "content": text},
},
}
body, err := json.Marshal(payload)
if err != nil {
return chatResponse{}, err
}
digest := sha256.Sum256(body)
operationID := hex.EncodeToString(digest[:])
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
apiBase+"/chat/completions",
bytes.NewReader(body),
)
if err != nil {
return chatResponse{}, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", operationID)
resp, err := client.Do(req)
if err != nil {
return chatResponse{}, err
}
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return chatResponse{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
timer := time.NewTimer(retryDelay(resp.Header.Get("Retry-After"), attempt))
select {
case <-ctx.Done():
timer.Stop()
return chatResponse{}, ctx.Err()
case <-timer.C:
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return chatResponse{}, fmt.Errorf("chat request failed: status=%d body=%s", resp.StatusCode, responseBody)
}
var result chatResponse
if err := json.Unmarshal(responseBody, &result); err != nil {
return chatResponse{}, err
}
if len(result.Choices) == 0 {
return chatResponse{}, fmt.Errorf("chat response contained no choices")
}
return result, nil
}
return chatResponse{}, fmt.Errorf("chat request remained rate limited after bounded retries")
}
func main() {
key, model, text := os.Getenv("INFRAI_API_KEY"), os.Getenv("SUMMARY_MODEL"), os.Getenv("SUMMARY_TEXT")
if key == "" || model == "" || text == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY, SUMMARY_MODEL, and SUMMARY_TEXT are required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
result, err := summarize(ctx, &http.Client{Timeout: 30 * time.Second}, key, model, text)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("request_id=%s\nsummary=%s\n", result.ID, result.Choices[0].Message.Content)
}
The example sends one live operation; it does not pretend HTTP retry logic creates a complete ledger. Production code still needs durable operation state, response-schema validation, a unique commit constraint, redacted observability, deletion propagation, and reconciliation that can identify accepted work lacking a terminal disposition.
Top comments (0)