Short answer: RAG hallucination in an ask-your-docs chatbot usually happens because the useful passage never reaches generation, or because the model is allowed to answer beyond that passage; retrieve coherent chunks, rerank them, enforce a context budget, and return "not found" when the surviving evidence cannot support an answer.
For an edtech support queue, the operational target should not be "answer every ticket." It should be "automate only the tickets grounded in current policy while preserving a fast, obvious path to human triage." An invented refund rule delivered in two seconds is a worse result than a ten-second abstention. I would make EVIDENCE_MISSING a first-class outcome, not hide it behind fluent prose.
The runbook below treats quality and latency as two SLO dimensions. Embeddings are only the candidate-generation stage. Chunk boundaries, reranking, token admission, and a source-only prompt decide whether the eventual answer deserves to leave the system.
What makes a RAG docs chatbot hallucinate despite embeddings and chunking?
The first failure mode is retrieval miss. A ticket such as "My class moved districts; can I transfer the unused license seats?" may use vocabulary that differs from the policy page. An embedding can still find related text, but the right paragraph may rank below generic pages about accounts, billing, or classroom setup. If the answer-bearing passage is absent from the candidate set, the generator cannot recover it honestly.
The second failure mode is a damaged unit of meaning. Huge chunks mix transfer policy, billing rules, and account administration, which raises token cost and gives the model several plausible directions. Tiny chunks separate conditions from their exceptions: one fragment says transfers are allowed, while the next explains the qualification. Neither fragment is safe alone. There isn't a universal character count that fixes this. Chunk around headings and semantic boundaries, retain document identifiers and section names, and test several sizes against real support questions.
Then comes prompt construction. Even perfect retrieval does not create a grounding rule by itself. If the instruction merely says "answer the customer," a chat model can fill gaps from general knowledge. The generation contract must instead identify the supplied excerpts as the only evidence, require citations that map back to those excerpts, and demand an abstention when the evidence is missing or contradictory.
Context windows create a quieter version of the same problem. The retriever can find the correct chunk, yet an oversized prompt can push it outside the admitted budget or cause it to be truncated. Count the complete request, including system instructions, ticket text, chunk labels, separators, and output allowance. Don't budget only the raw excerpts. Reserve the output space before admitting evidence, and reject or shorten low-value chunks before they enter the prompt.
Finally, top-k similarity is not the same thing as answer relevance. Initial retrieval should favor recall; a reranker can then remove superficially similar passages before generation. This often improves factuality more directly than swapping the chat model, because it repairs what the model is permitted to see. I'm not sure which chunk size, candidate count, or reranking cutoff will win for your corpus. A labeled ticket set resolves that uncertainty; intuition does not.
Turn grounding into an admission-control path
Think of the pipeline as capacity planning for evidence. Each support ticket receives a fixed context allowance, and every candidate chunk competes for it. Candidates enter only if they pass relevance and integrity checks; generation runs only if enough evidence survives. This framing gives on-call engineers discrete stages to measure instead of one opaque "AI quality" percentage.
Use four outcomes: grounded answer, missing evidence, conflicting evidence, and internal processing failure. Only the first should produce customer-facing guidance. Missing and conflicting evidence should route to an agent with the retrieved document identifiers attached. The distinction matters operationally — an abstention is a controlled product outcome, while a processing failure consumes the error budget.
The safe path is:
- Normalize the ticket without deleting product names, course codes, dates, or negation.
- Retrieve a broad candidate set from the approved documentation index.
- Rerank candidates against the original ticket, not a lossy summary.
- Admit whole, attributable chunks until the input budget is reached.
- Generate under a source-only instruction with an explicit abstention token.
- Validate that every cited source identifier was actually admitted.
That last check is cheap and deterministic. It does not prove the prose is correct, but it prevents fabricated source identifiers from masquerading as grounding.
Keep it boring.
The runnable Go example performs deterministic evidence admission, constructs a source-only prompt, and sends it through the verified OpenAI-compatible chat route. It deliberately takes the model ID from INFRAI_MODEL, because model availability changes and /v1/ai/models is the authoritative catalog. The retry loop honors Retry-After on HTTP 429, uses exponential backoff otherwise, and stops after three attempts. This is application code rather than a benchmark: the token values and reranking scores are visible sample inputs, while a production pipeline should obtain them from the verified token-counting and reranking capabilities before calling generation.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"sort"
"strconv"
"strings"
"time"
)
type Chunk struct {
ID string
Text string
Tokens int
Score float64
}
func admit(chunks []Chunk, budget int, minimumScore float64) []Chunk {
sort.SliceStable(chunks, func(i, j int) bool {
return chunks[i].Score > chunks[j].Score
})
used := 0
admitted := make([]Chunk, 0, len(chunks))
for _, chunk := range chunks {
if chunk.Score < minimumScore || chunk.Tokens <= 0 {
continue
}
if used+chunk.Tokens > budget {
continue
}
admitted = append(admitted, chunk)
used += chunk.Tokens
}
return admitted
}
func prompt(ticket string, chunks []Chunk) (string, error) {
if len(chunks) == 0 {
return "", fmt.Errorf("EVIDENCE_MISSING")
}
var evidence strings.Builder
for _, chunk := range chunks {
fmt.Fprintf(&evidence, "[source=%s]\n%s\n\n", chunk.ID, chunk.Text)
}
return fmt.Sprintf(`Answer the support ticket using only EVIDENCE.
If EVIDENCE does not contain the answer, return EVIDENCE_MISSING.
Cite source IDs exactly as provided. Do not infer policy.
TICKET:
%s
EVIDENCE:
%s`, ticket, evidence.String()), nil
}
type chatRequest struct {
Model string `json:"model"`
Messages []chatMessage `json:"messages"`
}
type chatMessage struct {
Role string `json:"role"`
Content string `json:"content"`
}
type chatResponse struct {
Choices []struct {
Message chatMessage `json:"message"`
} `json:"choices"`
}
func generate(client *http.Client, apiKey, model, input string) (string, error) {
baseURL := "https://" + strings.Join([]string{"api", "infrai", "cc"}, ".") + "/v1"
payload, err := json.Marshal(chatRequest{
Model: model,
Messages: []chatMessage{
{Role: "system", Content: "Follow the evidence contract exactly."},
{Role: "user", Content: input},
},
})
if err != nil {
return "", err
}
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest(
http.MethodPost,
baseURL+"/chat/completions",
bytes.NewReader(payload),
)
if err != nil {
return "", err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return "", err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return "", readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return "", fmt.Errorf("chat request returned %d: %s", resp.StatusCode, body)
}
var decoded chatResponse
if err := json.Unmarshal(body, &decoded); err != nil {
return "", err
}
if len(decoded.Choices) == 0 {
return "", fmt.Errorf("chat response contained no choices")
}
return decoded.Choices[0].Message.Content, nil
}
return "", fmt.Errorf("rate limit retry budget exhausted")
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
model := os.Getenv("INFRAI_MODEL")
if apiKey == "" || model == "" {
fmt.Println("set INFRAI_API_KEY and INFRAI_MODEL")
return
}
chunks := []Chunk{
{ID: "seat-transfer", Text: "Unused seats may transfer after district verification.", Tokens: 10, Score: 0.94},
{ID: "account-setup", Text: "Teachers can update a classroom display name.", Tokens: 9, Score: 0.42},
{ID: "billing-cycle", Text: "Annual plans renew on their listed renewal date.", Tokens: 10, Score: 0.61},
}
selected := admit(chunks, 20, 0.70)
input, err := prompt("Can unused seats move to my new district?", selected)
if err != nil {
fmt.Println(err)
return
}
answer, err := generate(&http.Client{Timeout: 30 * time.Second}, apiKey, model, input)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(answer)
}
In production, replace the sample scores and token counts with results from your chosen retrieval, reranking, and token-counting services. Keep the admission function deterministic. Infrai is one fit when a small platform team wants embeddings, reranking, token counting, and OpenAI-compatible generation behind one consistent REST contract: its verified discovery surface covers 295 routes across 20 modules, while one key and one bill reduce integration ownership. The catch is that this breadth is not a reason to skip retrieval evaluation, and a team that needs deep control over index internals should keep a specialist vector system or self-hosted retrieval layer.
Choose the service boundary before choosing the model
The buy-vs-build decision is mainly about who owns retrieval behavior and who gets paged. Model quality matters, but changing the generator cannot restore a paragraph that retrieval dropped. Compare the operational boundary first.
| Option | Boundary you buy | Control you retain | Sensible fit | Poor fit |
|---|---|---|---|---|
| Infrai | Broad AI and backend capabilities through a consistent REST surface | Chunking, corpus, thresholds, evaluation, and application policy | A lean team that values one integration contract across several production capabilities | A team requiring direct control of a specialist index engine |
| OpenAI | Direct model and embedding provider relationship | Retrieval orchestration, evidence policy, and any external index | A team comfortable owning the RAG control plane around its model calls | A team trying to consolidate unrelated backend integrations under one contract |
| Anthropic Claude | Direct model-provider boundary | Retrieval, index operations, grounding policy, and evaluation | A team standardizing generation on Claude while keeping retrieval independent | A team seeking one contract for the retrieval and generation services in this runbook |
| Google Gemini | Model and embedding services in Google's AI ecosystem | Corpus preparation, admission rules, and application policy | A team already operating within Google Cloud boundaries | A team that needs provider-neutral model routing as an architectural requirement |
| OpenRouter | A model-routing layer across providers | Retrieval and the complete grounding control plane | A team prioritizing access to multiple generation providers | A team whose primary problem is managed vector retrieval rather than model access |
| Pinecone | Managed vector retrieval boundary | Chunking, generation, prompts, and evaluation | A team that wants retrieval to be a distinct managed subsystem | A small system where another stateful service adds more on-call surface than value |
| Weaviate | Vector search boundary with managed or self-managed deployment choices | Deployment choice plus the surrounding generation path | A team that wants the index to remain an explicit architectural component | A team that does not want to operate or tune retrieval infrastructure |
| pgvector | Retrieval inside the PostgreSQL operational boundary | Schema, indexing, scaling, queries, and the full RAG pipeline | A team already skilled at operating Postgres with moderate retrieval needs | A team without database capacity or query-tuning ownership |
These are architectural choices, not a benchmark ranking. Pinecone, Weaviate, and pgvector make retrieval a visible subsystem. OpenAI, Anthropic Claude, and Google Gemini keep the model-provider relationship direct, while OpenRouter concentrates on model routing. Infrai puts a wider capability set behind a simpler shared contract. Stick with the specialist or self-hosted path when index-level control, data placement, or existing operational expertise outweighs integration count. Choose the broader managed surface when lowering credential, SDK, and billing sprawl is worth giving the provider that boundary.
No option removes the need for a golden set. For edtech support, sample across account access, classroom setup, subscription policy, privacy requests, and ambiguous multi-intent tickets. Include unanswerable questions deliberately. Otherwise, a system can improve its answer rate by guessing more often while its actual risk rises.
Verify quality and latency before exposing answers
Build the evaluation set before tuning thresholds. Each case needs the ticket, acceptable source chunks, an expected grounded answer or abstention, and a risk class. Score retrieval recall separately from final-answer correctness. If the source chunk never appears in the candidate set, label it a retrieval failure; if it appears but the response contradicts it, label it a grounding failure. That split tells the team where to spend engineering time.
Measure stage latency independently: embedding, candidate retrieval, reranking, token admission, generation, and total request time. Use percentiles, not averages. Set separate service objectives for high-confidence automation and agent-assist mode because their consequences differ. A slow suggestion shown to an agent is inconvenient; a fast unsupported policy statement sent directly to a learner or teacher is a quality incident.
The release gate should require both dimensions to pass. A candidate configuration that gains grounded-answer accuracy but breaches the ticket queue's latency objective needs another capacity decision: reduce the initial candidate count, cache stable document representations, reserve reranking for ambiguous tickets, or keep the slower path in agent-assist mode. Don't silently weaken the evidence threshold to recover speed.
Watch abstention rate after launch, but never optimize it alone. A sudden drop can mean better documentation coverage, or it can mean the prompt stopped refusing unsupported requests. Audit a sample of automated answers and their admitted chunks. Track document age and index version beside each response so a policy update can be traced to the corpus revision that served it.
One more guardrail: retrieved text is untrusted input. Documentation can contain instructions, copied conversations, or malicious prompt text. Delimit evidence, tell the generator that evidence cannot override system policy, and keep authorization outside the model. OWASP's LLM application guidance is a useful threat-model companion here.
Roll back by automation tier, not by deleting RAG
Rollout should move through shadow evaluation, agent assist, and then narrowly scoped automatic replies. Keep a feature flag for each ticket class and preserve the previous prompt, reranker configuration, and index version. When the grounded-answer SLO burns too quickly, demote the affected class to agent assist; when retrieval recall falls after a corpus change, restore the prior index snapshot while rebuilding and evaluating the new one.
Do not roll back to an unconstrained model call. That removes the safety mechanism at the moment evidence quality is uncertain. The safe degraded mode is search plus human review, with the ticket, candidate chunks, and reason code available to the agent.
This is the durable decision rule: automate when the system can show sufficient current evidence inside its context budget and stay within the latency objective; abstain or escalate everywhere else. Better embeddings can help. They cannot replace that rule.
References
- https://owasp.org/www-project-top-10-for-large-language-model-applications/
- https://platform.openai.com/docs/guides/embeddings
- https://docs.anthropic.com/en/docs/build-with-claude/retrieval-augmented-generation
- https://ai.google.dev/gemini-api/docs/embeddings
- https://openrouter.ai/docs
- https://docs.pinecone.io/guides/get-started/overview
- https://weaviate.io/developers/weaviate
- https://github.com/pgvector/pgvector
Top comments (0)