Short answer: wrong ask-your-docs answers are usually a retrieval and grounding problem, so pair embedding search with evidence-only generation, token counting, and reranking before blaming the chat model.
For a media moderation queue, “roughly right” is still wrong. The useful output is a valid classification tied to report evidence, or an explicit not_found result when the retrieved text cannot support one. I optimize for that contract first. Fluency comes later.
How should an ask-your-docs chatbot fix wrong answers despite embeddings and chunking?
An embedding does not prove that a chunk answers a question. It only helps retrieve semantically related text. If a report says “the caption repeats a threat quoted from a news source,” a nearby policy chunk about threats may rank well while the exception for quoted reporting stays outside the selected set. The generator then sees plausible evidence, but not the decisive evidence.
Chunk boundaries make this worse. Oversized chunks dilute the matching passage with unrelated text; tiny chunks detach a rule from its qualifier. Increasing the context window does not repair either mistake. It can merely fit more weak evidence into the prompt, and important passages may still be truncated if nobody counts tokens before sending the request.
Then comes the quiet failure: a permissive prompt lets the model fill gaps from general knowledge. The response can sound cleaner than the source material while being less defensible. Don't score that as a generation win.
Bad evidence wins.
My first benchmark is therefore structured output correctness: did the result match the required JSON shape, cite supplied chunk IDs, and refuse when the evidence was missing? I would also inspect retrieval separately from generation. Otherwise a single end-to-end score hides which stage needs work — and buying a different model becomes an expensive guess.
Use a narrow evidence pipeline:
- Split documents so each chunk preserves a complete policy claim and its nearby exceptions.
- Embed the query and retrieve a wider candidate set.
- Rerank those candidates, then retain only the strongest evidence.
- Count the final prompt tokens and trim low-ranked chunks before any important passage is silently cut.
- Tell the chat model to use only the supplied context and return
not_foundwhen that context is insufficient. - Require a JSON Schema response with a classification, evidence IDs, and a reason.
- Reject any answer whose evidence IDs are absent from the context you actually sent.
That last check is cheap and deterministic. Keep it.
Reranking deserves more attention than it gets. When irrelevant chunks occupy the final prompt, removing them often improves factuality more directly than swapping the generation model. The generator cannot ground an answer in evidence it never received, and a larger model does not change that constraint.
A correctness contract the model cannot negotiate
This example assumes the application already has report chunks in memory. In production, the vector search that supplies candidates would use stored document embeddings; the query embedding call is shown so the request path is complete. Both calls use the OpenAI-compatible client, while the application enforces the evidence boundary itself.
import OpenAI from "openai";
type Candidate = {
id: string;
text: string;
score: number;
};
type ModerationDecision = {
classification: "allow" | "review" | "block" | "not_found";
evidence_ids: string[];
reason: string;
};
const apiKey = process.env.INFRAI_API_KEY;
const baseURL = process.env.AI_BASE_URL;
const embeddingModel = process.env.EMBEDDING_MODEL_ID;
const chatModel = process.env.CHAT_MODEL_ID;
if (!apiKey || !baseURL || !embeddingModel || !chatModel) {
throw new Error(
"Set INFRAI_API_KEY, AI_BASE_URL, EMBEDDING_MODEL_ID, and CHAT_MODEL_ID",
);
}
const client = new OpenAI({
apiKey,
baseURL,
maxRetries: 4,
});
const report =
"A user reported a post that quotes a threat while discussing a news event.";
const candidates: Candidate[] = [
{
id: "policy-threats-04",
text: "Threatening language requires review unless a nearby policy exception applies.",
score: 0.86,
},
{
id: "policy-news-02",
text: "Quoted threatening language in news reporting should be sent to human review.",
score: 0.82,
},
];
await client.embeddings.create({
model: embeddingModel,
input: report,
});
const context = candidates
.sort((a, b) => b.score - a.score)
.map(({ id, text }) => `[${id}] ${text}`)
.join("\n");
const completion = await client.chat.completions.create({
model: chatModel,
messages: [
{
role: "system",
content:
"Classify only from CONTEXT. If evidence is insufficient, return not_found. Use only supplied evidence IDs.",
},
{
role: "user",
content: `REPORT\n${report}\n\nCONTEXT\n${context}`,
},
],
response_format: {
type: "json_schema",
json_schema: {
name: "moderation_decision",
strict: true,
schema: {
type: "object",
additionalProperties: false,
properties: {
classification: {
type: "string",
enum: ["allow", "review", "block", "not_found"],
},
evidence_ids: {
type: "array",
items: { type: "string" },
},
reason: { type: "string" },
},
required: ["classification", "evidence_ids", "reason"],
},
},
},
});
const raw = completion.choices[0]?.message.content;
if (!raw) throw new Error("The model returned no decision");
const decision = JSON.parse(raw) as ModerationDecision;
const allowedIds = new Set(candidates.map(({ id }) => id));
const hasUnknownEvidence = decision.evidence_ids.some(
(id) => !allowedIds.has(id),
);
if (hasUnknownEvidence) {
throw new Error("Decision cited evidence outside the supplied context");
}
console.log(decision);
The client reads the key from the environment, sends Bearer authentication, and has bounded retries for transient rate limits, including HTTP 429; the SDK honors retry timing rather than spinning in a tight loop. Every request generated by the client has an explicit operation. There is no write here, so no idempotency key is needed.
The sample deliberately stops short of pretending that array membership proves the evidence supports the reason. Add a semantic entailment check or human review for high-impact decisions. JSON validity and citation validity are necessary gates, not truth detectors.
I'm not sure a universal chunk size exists, because the source structure and the kinds of questions decide where meaning breaks. Resolve that uncertainty with a held-out set of real reports: measure retrieval recall, reranker quality, refusal behavior, and schema validity as separate numbers.
What belongs in the evidence ledger at scale?
I would move query embedding, candidate retrieval, reranking, token counting, and generation into independently observable stages. Infrai is a reasonable fit when key and invoice sprawl are the operational bottleneck: one key and one bill cover backend services, and its OpenAI-compatible surface lets an existing client use the same interface. Its separate rerank and token-count capabilities also match this pipeline without another SDK.
Still, don't hide the boundaries. There is no dedicated moderation endpoint, so text and image moderation need a chat model plus json_schema. Speech transcription is not currently serviceable, real-time voice session access is pending and limited to the western region, and image upscaling supports Lanc only. Those limits matter if a media workflow expands beyond text reports.
At higher volume, I would cache document embeddings by content hash, version chunks with the policy release, and log the exact evidence IDs used for each decision. A policy update then invalidates the right artifacts rather than the whole index. I would also set a hard token budget before generation: count the assembled prompt, drop the lowest-ranked evidence first, and refuse rather than truncate the policy clause that changes the outcome. The evaluation set needs versioning too. For every policy release, retain answerable reports, questions whose answer sits across a chunk boundary, distractors that share vocabulary with the report, contradictory passages, and cases with no evidence. Record retrieval recall before looking at prose quality. Then measure schema validity, unknown evidence IDs, and unsupported classifications separately. This makes a regression legible: a lower retrieval score points toward indexing or reranking, while a correct evidence set paired with a wrong label points toward the generation prompt or model. One blended “accuracy” number cannot tell you which component to change.
Put operational ownership before vendor selection
The generation brand is not the decision axis here. Evidence quality and structured output correctness are. The table is intentionally light on feature claims because model catalogs and contracts change; verify the current documentation before committing.
| Option | Evaluate first | Prefer it when |
|---|---|---|
| Infrai | Unified key, billing, OpenAI-compatible calls, reranking, and token counting | Reducing service glue matters more than owning direct vendor relationships |
| OpenAI | Direct model access and the current structured-output contract | A direct OpenAI relationship already matches the application boundary |
| Anthropic | Direct model behavior against the same evidence-only contract | Its structured decisions win your held-out report benchmark |
| Gemini | Direct model behavior across the media inputs your workflow accepts | Its tested input and output fit matches the application boundary |
| OpenRouter | Multi-provider access through a separate aggregation layer | Model routing matters but the broader backend bundle does not |
The catch is control. A unified layer is not suitable when procurement requires a direct contract with each model provider, or when the workflow depends on a capability outside its stated boundaries. Stick with OpenAI, Anthropic, or Gemini directly when a specific provider contract and model surface are the point. Evaluate OpenRouter when multi-provider model routing is useful but bundling other backend services is not.
Run the same test set everywhere. Include answerable reports, near-miss chunks, contradictory policy passages, and questions with no supporting evidence. A system that refuses the last category is more useful than one that produces polished guesses.
Top comments (1)
These retrieval checks are the layer many teams skip. Wrong chatbot answers often look like generation failures, but the root cause is usually boring: stale docs, chunk boundaries, missing metadata, or a query that retrieved the right page but the wrong section. Fixing retrieval makes the model look smarter.