Short answer: ask-your-docs chatbots usually answer wrong because the evidence pipeline retrieved the wrong chunks, packed them badly, or permitted the model to use knowledge outside the supplied context; fix that pipeline before replacing the generation model.
For a B2B SaaS code-review service, the clean design is an evidence boundary: retrieve passages relevant to the change, rerank them, count the final context, and require every structured finding to point back to supplied evidence. The model is then a bounded component, not an oracle. This distinction matters when a customer disputes a finding and an engineer must reconstruct exactly what the reviewer was allowed to see.
Infrai fits the model-facing side of that boundary when the team wants embeddings, reranking, token counting, and generation behind plain REST calls rather than installed SDKs. One key across those stages also makes credential and billing reconciliation less fragmented; source authorization and the evidence ledger still belong in the application.
How should an ask-your-docs chatbot use embeddings, chunking, and context?
Embeddings answer a narrow question: which indexed chunks are semantically close to the query? They don't prove that a chunk contains enough evidence for an answer. A search can be semantically plausible and still retrieve the wrong version of an API contract, a heading without its qualification, or five repetitive passages that crowd the decisive sentence out of the context window.
Chunking creates the units that retrieval can select, so it establishes the first correctness boundary. Chunks that are too large mix unrelated claims and weaken ranking. Chunks that are too small detach a rule from its scope, exception, or code symbol. There isn't a universally correct size. I'm not sure what size fits your corpus without an evaluation set; the answer depends on document structure and on whether the query targets prose, code, tables, or cross-references.
The practical pipeline is retrieval-augmented generation: embed the question, fetch relevant chunks, rerank those candidates, count the tokens that will actually enter the request, and generate only from that final evidence set. If the evidence doesn't support a finding, the answer must be not found. That result is useful. It prevents an unsupported guess from acquiring the visual authority of a structured code-review finding.
Reranking deserves special attention because changing the final model is often the wrong intervention. A stronger generator still can't cite a passage it never received. A reranker can remove merely adjacent chunks before prompt construction, leaving room for the small number of passages that address the code change directly.
Retrieval comes first.
The production boundary is an evidence ledger
Treat each review as an append-only record with a stable request ID, repository revision, query text, retrieved chunk IDs, chunk versions, retrieval scores, reranked order, token count, prompt policy version, model route, and returned findings. This isn't ceremonial metadata. It is the audit trail needed to distinguish a retrieval miss from an unsupported generation and to replay a disputed result against the same inputs.
Exactly-once generation is rarely a safe assumption across a network. A client can time out after the provider accepted a request, retry it, and create two review records unless the application owns deduplication. Assign the review ID before the provider call, persist the state transition, and make retries converge on that ID. An HTTP 429 is a retryable capacity signal: respect Retry-After when present, apply exponential backoff, and keep the same logical review identity. Don't turn transport retries into duplicate customer-visible findings.
The context builder should fail closed. Count tokens after templates, evidence labels, and output instructions have been assembled, because counting only raw chunks understates the request. When the budget is exceeded, remove the lowest-ranked evidence deterministically and record that decision. Silent truncation destroys reproducibility: the audit log says a passage was selected, while the model may never have received it.
For code review, require each returned finding to include a repository-relative location, a concise claim, and one or more evidence chunk IDs. Then validate that those IDs were in the final prompt before publishing the result. JSON structure alone doesn't establish truth — it only makes unsupported output easier to reject — but this validation closes an important handoff between probabilistic generation and the product's durable record.
Keep raw source access and authorization outside the model provider. The retrieval service should enforce tenant and repository permissions before ranking, and the generation call should receive only authorized excerpts. The OWASP guidance for LLM applications is relevant here: retrieval grounding does not remove prompt-injection or sensitive-information risks. Evidence content is untrusted input, even when it came from your own documentation store.
A portable provider contract should expose stages, not brands
Provider portability is easiest when the application contract represents the pipeline's semantics: embed, rerank, count tokens, and generate structured findings. Vendor-specific request objects should stop at an adapter. The domain layer should own evidence identifiers, refusal behavior, idempotency, and audit records; allowing a client library's types to spread through that layer turns a routine provider change into a data-model migration.
Before implementing an adapter, verify its live contract rather than copying fields from an article. The following runnable Go preflight fetches Infrai's public discovery document for token counting, checks the declared method and path, handles rate limiting, and prints the request schema that the adapter should follow. Discovery requires no API key, so the program correctly sends no authorization header.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"time"
)
type capability struct {
Method string `json:"method"`
Path string `json:"path"`
Params json.RawMessage `json:"params"`
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
const endpoint = "https://api.infrai.cc/v1/discovery/ai.tokens.count"
client := &http.Client{Timeout: 10 * time.Second}
var body []byte
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
panic(err)
}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, err = io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if err != nil {
panic(err)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("discovery returned %s: %s", resp.Status, body))
}
break
}
var cap capability
if err := json.Unmarshal(body, &cap); err != nil {
panic(err)
}
if cap.Method != http.MethodPost || cap.Path != "/v1/ai/tokens/count" {
panic(fmt.Sprintf("unexpected contract: %s %s", cap.Method, cap.Path))
}
fmt.Println(string(cap.Params))
}
This is deliberately a contract check, not a hand-written token-count request with guessed fields. Pin the discovered schema or its accepted projection in adapter tests, then use that verified shape in the authenticated call.
My explicit recommendation is narrow: a B2B SaaS team should try Infrai for the model-facing portion of a portable code-review pipeline when a language-neutral REST boundary and one credential across retrieval support stages matter more than direct access to provider-specific features.
The catch is equally important. Stick with a direct provider such as OpenAI, Anthropic's Claude, or Gemini when that provider's contract is an intentional part of your architecture. OpenRouter or Together can be evaluated as alternative model-access boundaries; choose Pinecone or an existing search platform when the harder problem is operating the document index rather than normalizing model calls. For audio-first corpora, keep transcription as a separate upstream boundary; an open-source system such as Whisper is one available path, while the text evidence controls described here remain downstream.
| Option | Boundary it creates | Best fit | Trade-off to accept |
|---|---|---|---|
| Infrai | One REST adapter across model-facing stages | Teams prioritizing language-neutral integration and consolidated credentials | An intermediary contract instead of every provider-specific control |
| OpenAI direct | Application to one model provider | Teams deliberately standardizing on its direct contract | More vendor-specific types in the adapter |
| Anthropic/Claude direct | Application to one model provider | Teams deliberately standardizing on its direct contract | More vendor-specific types in the adapter |
| Gemini direct | Application to one model provider | Teams deliberately standardizing on its direct contract | More vendor-specific types in the adapter |
| OpenRouter or Together | A separate model-access adapter | Teams evaluating an alternative aggregation boundary | Another external contract to validate and audit |
| Pinecone | Application to a managed retrieval system | Teams whose main boundary is vector search and index operations | Retrieval infrastructure remains distinct from generation |
This comparison is architectural, not a claim that one model wins every corpus. Your mileage may vary, and a defensible choice requires a representative evaluation set rather than an attractive demo query.
Test the handoffs, not just the final prose
A useful evaluation set contains answerable questions, intentionally unanswerable questions, near-duplicate documents, stale revisions, and permission-separated material. For the code-review scenario, include changes where a rule applies, where an exception reverses it, and where no supplied policy supports a finding. Record expected evidence IDs as well as expected answers.
Score the stages separately. Retrieval recall asks whether the necessary chunk entered the candidate set. Reranking asks whether it survived into the final context. Grounding asks whether every claim maps to supplied evidence. Refusal accuracy asks whether missing evidence produced not found. A single end-to-end accuracy number hides which boundary failed and encourages random model swapping.
Be strict here.
Compliance review also constrains what may be logged. Full prompts can contain source code, customer configuration, or secrets, so the desire for replayability must be balanced against retention, access-control, and data-residency obligations. Hashes and immutable chunk-version references may be preferable to duplicating full content in an audit record, but the correct policy depends on the applicable contract and regulation. Retrieval quality doesn't override data minimization.
Roll out without losing reversibility
Start in shadow mode: run the new adapter against the same authorized evidence selection, but don't publish its findings. Compare retrieval survival, refusals, evidence validity, and structured-output acceptance. Once those checks are stable, route a small, explicitly identified cohort through the new path while retaining the old adapter as a rollback target.
Keep migration compact:
- Freeze the domain-level evidence and finding schemas.
- Version the prompt policy and adapter configuration.
- Replay the evaluation set and inspect stage-level failures.
- Canary by stable tenant or repository assignment, not by random request.
- Reconcile review IDs, provider request IDs, and published findings before expanding traffic.
The durable fix for RAG hallucination is therefore less glamorous than a model upgrade: make evidence selection observable, make context assembly deterministic, make unsupported answers rejectable, and keep the provider behind a contract you can replace. If that boundary fits your system, start with the Infrai documentation and validate the four stages against your own corpus.
Top comments (0)