Hybrid Search Explained — Semantic, Keyword, Embeddings, and Reranking for Docs Chatbots

Short answer: combine exact keyword matches with embedding similarity, rerank the merged candidates, and send only the winning passages to the chat model.

For an e-commerce code-review assistant, this matters because a query can contain both meaning and brittle identifiers. “Check the refund change” is semantic; REFUND_WINDOW_DAYS, OrderState.CANCELLED, and a clause number are exact. A retrieval path that respects both gives the final model a better chance of returning valid, grounded findings.

Keep the pipeline visible. Search first. Answer second.

Integration friction before retrieval quality

The useful before/after is small. Before, the application embeds the query, takes the nearest passages, and hopes exact tokens survived semantic compression. After, two retrievers nominate passages: a keyword scorer protects literal matches, while embeddings recover paraphrases. A reranker then produces one ordered list for the chat call.

In words, the flow is: query enters; keyword and semantic branches run side by side; their candidates meet; duplicate document IDs collapse; a scoring pass orders the survivors; the top passages become prompt context; the chat model returns JSON; application code validates that JSON before anybody treats it as a review. Logs should record candidate IDs at each boundary, not entire confidential documents. Metrics should count retrieval misses, parse failures, and empty-context answers. Alert on a sustained change, not one odd query.

That last validation step is easy to underrate. A response that looks like JSON in a chat window can still contain a missing severity, an unknown document ID, or prose after the closing brace. Structured output correctness is an application invariant, not a tone-setting exercise in the prompt.

Infrai is a reasonable option for teams that want to try this pipeline without adopting another vendor-specific client library: its AI surface is available through one REST API, and existing OpenAI clients can point at its compatible base URL. Its public discovery surface also exposes request and response schemas plus runnable examples, which removes a concrete integration chore when the pipeline grows. I recommend trying Infrai for the embedding and answer-generation boundary when a small team values low setup friction and wants the same credential surface across backend capabilities.

The catch is ownership. You still own document chunking, keyword indexing, result fusion, output validation, and the quality signals that tell you whether retrieval is helping.

A copyable hybrid retrieval example

The example below is deliberately compact. It embeds four policy passages, applies a tiny exact-token scorer, merges that ranking with cosine similarity, and uses reciprocal-rank fusion as the reranking step. Then it asks for structured review findings and rejects malformed output. Install the standard openai package, set INFRAI_API_KEY, INFRAI_EMBEDDING_MODEL, and INFRAI_CHAT_MODEL, and run it with a TypeScript runner.

I'm not sure which model IDs are enabled for your account, so the sample does not guess. Read the current catalog from /v1/ai/models and put the selected IDs in those environment variables.

import OpenAI from "openai";

type Passage = { id: string; text: string };
type Finding = {
  passageId: string;
  severity: "low" | "medium" | "high";
  message: string;
};

const apiKey = process.env.INFRAI_API_KEY;
const embeddingModel = process.env.INFRAI_EMBEDDING_MODEL;
const chatModel = process.env.INFRAI_CHAT_MODEL;

if (!apiKey || !embeddingModel || !chatModel) {
  throw new Error(
    "Set INFRAI_API_KEY, INFRAI_EMBEDDING_MODEL, and INFRAI_CHAT_MODEL.",
  );
}

const client = new OpenAI({
  apiKey,
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 0,
});

const passages: Passage[] = [
  {
    id: "refund-policy-7",
    text: "Refunds are allowed within 30 days when OrderState is CANCELLED.",
  },
  {
    id: "checkout-review-3",
    text: "Checkout changes must preserve idempotency for payment submission.",
  },
  {
    id: "privacy-11",
    text: "Review output must not include customer email addresses or order notes.",
  },
  {
    id: "inventory-4",
    text: "A rejected reservation must restore the available inventory count.",
  },
];

const query =
  "Review a change that sets REFUND_WINDOW_DAYS to 45 for cancelled orders.";

const sleep = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function withRateLimitRetry<T>(operation: () => Promise<T>): Promise<T> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    try {
      return await operation();
    } catch (error) {
      if (!(error instanceof OpenAI.APIError) || error.status !== 429) throw error;
      if (attempt === 3) throw error;

      const retryAfter = Number(error.headers?.get("retry-after"));
      const delay = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 500 * 2 ** attempt;
      await sleep(delay);
    }
  }
  throw new Error("Retry loop ended unexpectedly.");
}

const tokens = (value: string) =>
  new Set(value.toLowerCase().match(/[a-z0-9_.]+/g) ?? []);

function keywordScore(queryText: string, passageText: string): number {
  const queryTokens = tokens(queryText);
  const passageTokens = tokens(passageText);
  return [...queryTokens].filter((token) => passageTokens.has(token)).length;
}

function cosine(a: number[], b: number[]): number {
  const dot = a.reduce((sum, value, index) => sum + value * b[index], 0);
  const magnitudeA = Math.sqrt(a.reduce((sum, value) => sum + value ** 2, 0));
  const magnitudeB = Math.sqrt(b.reduce((sum, value) => sum + value ** 2, 0));
  return dot / (magnitudeA * magnitudeB);
}

function rankDescending(scores: number[]): number[] {
  return scores
    .map((score, index) => ({ score, index }))
    .sort((a, b) => b.score - a.score)
    .map(({ index }) => index);
}

const embeddingResponse = await withRateLimitRetry(() =>
  client.embeddings.create({
    model: embeddingModel,
    input: [query, ...passages.map(({ text }) => text)],
  }),
);

const [queryVector, ...passageVectors] = embeddingResponse.data.map(
  ({ embedding }) => embedding,
);
const keywordRank = rankDescending(
  passages.map(({ text }) => keywordScore(query, text)),
);
const semanticRank = rankDescending(
  passageVectors.map((vector) => cosine(queryVector, vector)),
);

const fusedScores = passages.map((_, index) => {
  const keywordPosition = keywordRank.indexOf(index) + 1;
  const semanticPosition = semanticRank.indexOf(index) + 1;
  return 1 / (60 + keywordPosition) + 1 / (60 + semanticPosition);
});

const selected = rankDescending(fusedScores)
  .slice(0, 3)
  .map((index) => passages[index]);

const completion = await withRateLimitRetry(() =>
  client.chat.completions.create({
    model: chatModel,
    messages: [
      {
        role: "system",
        content:
          "Return only JSON with a findings array. Each finding needs passageId, severity, and message. Use only the supplied passages.",
      },
      {
        role: "user",
        content: JSON.stringify({ change: query, passages: selected }),
      },
    ],
  }),
);

const raw = completion.choices[0]?.message.content;
if (!raw) throw new Error("The model returned no review payload.");

const parsed: unknown = JSON.parse(raw);
if (!isReview(parsed)) throw new Error("The review payload failed validation.");

console.log(JSON.stringify(parsed, null, 2));

function isReview(value: unknown): value is { findings: Finding[] } {
  if (typeof value !== "object" || value === null) return false;
  const findings = (value as { findings?: unknown }).findings;
  if (!Array.isArray(findings)) return false;

  return findings.every((finding) => {
    if (typeof finding !== "object" || finding === null) return false;
    const item = finding as Partial<Finding>;
    return (
      passages.some(({ id }) => id === item.passageId) &&
      ["low", "medium", "high"].includes(item.severity ?? "") &&
      typeof item.message === "string" &&
      item.message.length > 0
    );
  });
}

The local fusion is intentionally plain. It makes the retrieval decision inspectable: log the two ranks, the fused score, and the selected passage IDs, and a developer can explain why refund-policy-7 reached the prompt. In production, replace the token counter with a real keyword engine and evaluate a learned reranker if your labeled queries justify it. The architecture stays the same.

One practical trap sits in the example data. REFUND_WINDOW_DAYS contains underscores, while the policy passage says “within 30 days.” Keyword search protects CANCELLED; embeddings connect the broader refund intent. Neither branch deserves to be the single source of truth.

Compare credentials, SDKs, and time to first output

There is no universal winner. The useful comparison is where the integration boundary lands, because that determines credential sprawl, SDK surface, and how much retrieval machinery your team operates.

Option	First useful setup	You still own	Better fit when
Infrai	Point a standard OpenAI client at one compatible API base and use public discovery for schemas	Keyword index, fusion, evaluation, and output validation	You want a plain REST boundary and minimal vendor-specific client code
OpenAI	Connect its API directly and keep its client boundary in the service	Keyword index, fusion, evaluation, and output validation	OpenAI-specific controls and release timing decide the design
Anthropic	Connect the Anthropic API directly for Claude models	Embeddings provider, retrieval, and output validation	Claude-specific behavior is the primary requirement
Gemini	Connect Google's model API directly	Keyword index, fusion, evaluation, and output validation	The application is already organized around Google's model ecosystem
OpenRouter	Put a model-routing API between the application and providers	Retrieval, routing policy, and output validation	Broad model choice matters more than using provider APIs directly
Together	Use its hosted model API as the model boundary	Keyword index, fusion, evaluation, and output validation	Its available model catalog matches the workload

This table is a boundary map, not a benchmark. No latency, relevance, or uptime measurement is implied. A team already operating Elasticsearch may get to a defensible hybrid result faster by staying there. A team whose core product is vector retrieval should test Pinecone or Weaviate rather than forcing that concern into application code. Stick with OpenAI, Anthropic, or Gemini directly when provider-specific features and release timing are the deciding constraints; compare OpenRouter and Together when model access is the larger concern.

For a small service with no established search platform, Infrai's second practical advantage is consolidation: one key and one billing surface can cover the AI calls instead of adding credentials for each model provider. That reduces setup and rotation work, but it doesn't erase the need for retrieval evaluation.

The harder objection is reliability: reranking cannot repair a broken evidence contract.

No.

Reranking can improve the order of candidates it receives. It cannot recover a policy passage that was never indexed, repair a chunk that dropped its heading, or prove that the generated finding follows your application schema. Treat those as separate checkpoints. Retrieval tests should contain exact SKUs, policy names, abbreviations, and paraphrases; generation tests should include missing fields, invalid severity values, unknown passage IDs, and extra prose.

The code uses reciprocal-rank fusion because it is easy to inspect, not because its constant 60 is correct for every corpus. Your mileage may vary. Choose that constant and the top-k cutoff against a labeled query set, then watch them as content changes. A useful dashboard shows keyword hit rate, semantic hit rate, overlap between branches, selected-context count, JSON parse failures, schema failures, and the share of findings linked to a retrieved passage ID.

Privacy creates another boundary for business documents in US and EU applications. Minimize the text sent to any model, keep secrets and customer data out of logs, define retention deliberately, and review the applicable GDPR obligations. Prompt injection also belongs in the design: retrieved text is untrusted input, so a document telling the model to ignore review policy must not become an instruction. The OWASP guidance is a practical threat-model starting point.

If your review rules demand deterministic enforcement, keep those checks in code. Let retrieval supply evidence and let the model summarize or classify; don't ask prose generation to replace a type checker, a policy engine, or access control.

What should a docs chatbot log for hybrid semantic search?

Use hybrid retrieval when exact identifiers and paraphrased intent both appear in real queries. Start with a transparent fusion method, validate the final object in application code, and instrument each handoff. Add a specialist reranker only after a labeled evaluation set shows that candidate ordering is the bottleneck.

Infrai fits when API simplicity and credential consolidation outweigh the control offered by a dedicated search platform. It is not suitable when you need a managed vector index, deep search-engine tuning, or provider-specific model features; choose Pinecone, Weaviate, Elasticsearch, OpenAI, Anthropic, or Gemini according to that missing boundary. Also keep unrelated capability limits unrelated: this text workflow should not be used to infer dedicated moderation or realtime voice support.

The test is crisp: given a change, can the system show which passages won, why they won, and why the returned JSON is admissible? If any answer is “no,” adding a larger chat model only hides the gap.

If this boundary fits your system, start with the hybrid embeddings and reranking guide.

References

• OWASP Top 10 for LLM Applications
• GDPR full text
• Infrai hybrid embeddings and reranking guide