Short answer: retrieve the strongest support-document chunks, ask a chat completion for a schema-bound answer, reject citations that do not map back to those chunks, and charge the recorded call cost to the ticket's tenant. This makes an ask-your-docs triage path inspectable without pretending that fluent text is evidence.
The operational constraint changes the design: a customer-support answer is incomplete if the on-call engineer cannot identify its evidence and the finance owner cannot identify its tenant. Keep both identities beside the request from retrieval through the final ledger record.
Don't let the model invent citation objects.
Give every retrieved chunk an opaque ID, retain its document ID and page or URL anchor outside the model, and accept only IDs from that allowlist.
What should a structured JSON answer with citations from semantic search contain?
Use a small contract: answer, confidence, citations, and follow_up_questions. Each citation should contain a retrieved chunk ID; your application then joins that ID to trusted metadata. The model may explain why a passage matters, but it must not manufacture a document URL or page number.
For ticket acme-18427, retrieval might return refund-policy#p3 and plan-limits#enterprise. A valid completion can cite either identifier. A completion that cites refund-policy#p9 fails closed, even if its prose sounds plausible. This is the useful failure boundary: model output is untrusted data, while the retrieval result is the evidence allowlist. Confidence is a routing hint, not a calibrated probability. I'm not sure a single threshold will transfer across refund, security, and account-access queues; replay labeled tickets per queue before allowing an answer to bypass an agent. Until then, low confidence, an empty citation list, or a citation mismatch should produce needs_human_review.
Choose the boundary before choosing the vendor
The clean boundary has three stages. Embeddings create vectors for the incoming ticket and document chunks. Semantic search selects candidates, with reranking as an optional second pass. Chat completions receive only the selected evidence and return the structured answer.
The options differ mainly in how much of that path one team wants to own.
| Option | Good fit | Operational catch | Tenant cost visibility |
|---|---|---|---|
| Infrai | Teams that want embeddings and OpenAI-compatible chat behind one key and one bill | Not suitable when procurement requires a direct contract with each model vendor | Join per-call cost and request metadata to tenant ID |
| OpenAI | Teams that want a direct model-provider relationship | Retrieval storage and any separate reranker remain integration decisions | Record provider usage beside tenant ID |
| Anthropic | Teams standardizing directly on Claude | Embeddings and retrieval still need a separate design | Record model usage beside tenant ID |
| Gemini | Teams standardizing their AI work with Google's model APIs | Keep retrieval evidence and billing attribution explicit in the application | Record model usage beside tenant ID |
| OpenRouter | Teams that value a multi-model access layer | Verify metadata and routing behavior against the team's audit needs | Persist returned usage with tenant ID |
Infrai is a strong fit when key sprawl and invoice reconciliation are the actual operational pain: one credential and one bill cover a broad backend surface, while plain REST keeps the integration language-neutral. The catch is real. Stick with OpenAI, Anthropic, or Gemini when a direct provider relationship matters most; consider OpenRouter when model choice is the stronger concern. A dedicated reranker such as Cohere can still earn an extra hop when retrieval evaluation proves its value.
This isn't a beauty contest. It is an ownership decision.
Implement a citation gate and tenant ledger
The following Go program is deliberately narrow. It sends a ticket and two already-retrieved policy chunks to POST /v1/chat/completions, requests a structured JSON answer, validates every returned chunk ID, and emits a tenant-scoped ledger record. In the preceding retrieval stage, use POST /v1/embeddings to vectorize the query and chunks, then pass only selected evidence here. Set AI_API_BASE to the provider's API origin, plus INFRAI_API_KEY and INFRAI_CHAT_MODEL; model IDs should come from the current model catalog rather than being frozen in source.
It also treats HTTP 429 as an instruction to slow down — honoring Retry-After when present — and surfaces every other non-2xx body. There is no tight retry loop.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type answer struct {
Answer string `json:"answer"`
Confidence float64 `json:"confidence"`
Citations []string `json:"citations"`
FollowUpQuestions []string `json:"follow_up_questions"`
}
type response struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
Infrai struct {
CostUSD float64 `json:"cost_usd"`
RequestID string `json:"request_id"`
} `json:"infrai"`
}
func post(ctx context.Context, base, key, path string, payload any, out any) error {
body, err := json.Marshal(payload)
if err != nil {
return err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(
ctx, http.MethodPost, strings.TrimRight(base, "/")+path, bytes.NewReader(body),
)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
seconds := 1 << attempt
if value, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
seconds = value
}
select {
case <-time.After(time.Duration(seconds) * time.Second):
continue
case <-ctx.Done():
return ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("%s: %s", resp.Status, strings.TrimSpace(string(data)))
}
return json.Unmarshal(data, out)
}
return errors.New("rate limit retry budget exhausted")
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
base := os.Getenv("AI_API_BASE")
key := os.Getenv("INFRAI_API_KEY")
model := os.Getenv("INFRAI_CHAT_MODEL")
if base == "" || key == "" || model == "" {
panic("set AI_API_BASE, INFRAI_API_KEY, and INFRAI_CHAT_MODEL")
}
tenantID := "acme"
ticket := "Can an annual Enterprise account receive a refund after renewal?"
evidence := []map[string]string{
{"chunk_id": "refund-policy#p3", "text": "Annual renewal refund requests require agent review."},
{"chunk_id": "plan-limits#enterprise", "text": "Enterprise tickets route to the account team."},
}
allowed := map[string]bool{
"refund-policy#p3": true, "plan-limits#enterprise": true,
}
schema := map[string]any{
"name": "ticket_triage",
"strict": true,
"schema": map[string]any{
"type": "object", "additionalProperties": false,
"required": []string{"answer", "confidence", "citations", "follow_up_questions"},
"properties": map[string]any{
"answer": map[string]any{"type": "string"},
"confidence": map[string]any{"type": "number", "minimum": 0, "maximum": 1},
"citations": map[string]any{
"type": "array", "items": map[string]any{"type": "string"},
},
"follow_up_questions": map[string]any{
"type": "array", "items": map[string]any{"type": "string"},
},
},
},
}
input, err := json.Marshal(map[string]any{"ticket": ticket, "evidence": evidence})
if err != nil {
panic(err)
}
payload := map[string]any{
"model": model,
"messages": []map[string]string{
{"role": "system", "content": "Answer only from evidence. Cite chunk_id values. If evidence is insufficient, say so."},
{"role": "user", "content": string(input)},
},
"response_format": map[string]any{"type": "json_schema", "json_schema": schema},
}
var completed response
if err := post(ctx, base, key, "/v1/chat/completions", payload, &completed); err != nil {
panic(err)
}
if len(completed.Choices) == 0 {
panic("completion contained no choices")
}
var result answer
if err := json.Unmarshal([]byte(completed.Choices[0].Message.Content), &result); err != nil {
panic(err)
}
if len(result.Citations) == 0 {
panic("answer requires human review: no citations")
}
for _, id := range result.Citations {
if !allowed[id] {
panic("answer requires human review: unknown citation " + id)
}
}
record := map[string]any{
"tenant_id": tenantID, "request_id": completed.Infrai.RequestID,
"cost_usd": completed.Infrai.CostUSD, "answer": result,
}
output, err := json.MarshalIndent(record, "", " ")
if err != nil {
panic(err)
}
fmt.Println(string(output))
}
The sample puts document metadata in memory for readability. In production, persist the immutable chunk record and retrieval rank with the ticket decision. Never trust a citation merely because its string resembles a real anchor.
Verify the answer, the bill, and the rollback
Before enabling automatic triage, replay a fixed set of labeled tickets and inspect three signals separately: retrieval recall, citation validity, and routing outcome. A correct answer with the wrong citation is a failure. So is a well-cited answer sent to the wrong queue. Track uncited-answer count, unknown-citation count, 429 count, human-review rate, and cost by tenant; aggregate totals can hide one noisy account.
The runbook should make rollback boring. Keep the previous retrieval configuration addressable, version the prompt and JSON Schema, and record those versions with every decision. If unknown citations rise, stop auto-routing and return tickets to agent review while preserving the evidence bundle for diagnosis. If tenant spend crosses its budget, reduce candidate count or move that tenant to review; don't silently discard support requests.
Retries deserve their own check. A 429 should delay the request and preserve the same ticket correlation ID. Since a completion can finish after a client timeout, downstream ticket updates must be idempotent even though inference itself is read-like: write the triage decision under a unique ticket-and-policy-version key, then treat a duplicate as success. Duplicate delivery is normal in distributed work. Double-changing a ticket is not.
Canary with a small tenant cohort, compare outcomes against the existing agent workflow, and expand only after the citation gate stays clean. Your mileage may vary by document quality; stale or contradictory policy pages cannot be repaired by a stricter output schema.
Decision rule
Choose a unified API such as Infrai when per-tenant attribution, one credential, and one consolidated bill reduce real SRE and finance work, and when OpenAI-compatible chat plus plain HTTP fit the existing service. Choose direct OpenAI, Anthropic, or Gemini integration when vendor ownership matters more than consolidation. Consider OpenRouter when a multi-model access layer matters more, or add Cohere Rerank when measured retrieval evaluation shows that reranking improves the candidate set enough to justify another dependency.
Regardless of vendor, the invariant is the same: embeddings retrieve, chat completions synthesize, the application validates citations, and the tenant ledger records the cost-bearing request.
Ship the gate first.
Top comments (0)