DEV Community

RhysFalconer159
RhysFalconer159

Posted on

Private Knowledge Retrieval: Relevance Tuning Under a Game Bot Latency Budget Explained

The constraint that changes this design is not the embedding model. It is the answer deadline. A game support bot that takes 1.8 seconds feels broken during a live incident, even if its top result is perfect. For a privacy-focused personal knowledge manager, I would start with a local, hybrid retrieval pipeline, enforce a 300 ms retrieval budget, and only spend extra latency on reranking when the first pass is uncertain.

Short answer: keep raw notes on-device, retrieve a small candidate set with lexical plus vector signals, rerank only ambiguous queries, and measure relevance and p95 latency together.

What does a privacy-focused personal knowledge manager need from retrieval?

Privacy is an architecture property, not a checkbox in a settings page. The index, query text, access logs, and model prompts all carry more information than the final answer. Encrypt the note store and its index at rest, keep tenant keys separate, and make deletion propagate to chunks, vectors, caches, and evaluation traces. A “private” UI backed by a shared remote index is still a data leak waiting for a configuration mistake.

The corpus in this example is a game studio's internal playbook: patch notes, incident timelines, anti-cheat procedures, and build-release runbooks. Documents have owners, visibility labels, and an updated-at timestamp. Those fields belong in retrieval filters, not in a prompt after retrieval. A player-facing FAQ should never become a source for an employee-only rollback command.

I once treated every note as a flat paragraph. The first relevance check looked fine, then a query for “rollback after hotfix 1.4.2” returned a design discussion that merely mentioned the phrase. The actual runbook was split across three tiny chunks. Chunk boundaries were the bug in my design, not a mysterious model failure.

Build log: a small hybrid retriever

The first pass uses BM25-style lexical matching and cosine similarity from a local vector index. Reciprocal rank fusion makes their scores comparable without pretending either score is a probability. Keep the candidate pool small: 40 documents is enough to inspect, and it keeps reranking predictable.

Here is the complete shape of the query path. The interfaces are deliberately boring so the storage engine can change later.

type Hit = {
  id: string;
  text: string;
  lexicalRank?: number;
  vectorRank?: number;
  acl: string[];
  updatedAt: string;
};

function rrf(rank: number, k = 60): number {
  return 1 / (k + rank);
}

function fuse(lexical: Hit[], vector: Hit[]): Hit[] {
  const byId = new Map<string, Hit>();
  lexical.forEach((hit, index) => {
    byId.set(hit.id, { ...hit, lexicalRank: index + 1 });
  });
  vector.forEach((hit, index) => {
    const old = byId.get(hit.id);
    byId.set(hit.id, { ...(old ?? hit), vectorRank: index + 1 });
  });

  return [...byId.values()]
    .map((hit) => ({
      ...hit,
      score: rrf(hit.lexicalRank ?? 999) + rrf(hit.vectorRank ?? 999),
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, 40);
}
Enter fullscreen mode Exit fullscreen mode

The filter runs before fusion. It checks the requesting user's ACL and the document's retention state, then records only a pseudonymous query id. Do not log the full query by default; debugging a retrieval miss should require an explicit, time-limited capture mode.

How should relevance tuning balance latency for a game knowledge bot?

Start with an evaluation set, not vibes. I use 50 to 100 questions sampled from real support tickets, each with a judged relevant passage and an allowed freshness window. Track recall@10, MRR, citation coverage, and p50/p95 retrieval latency. A change is useful only when the relevance gain survives the latency budget.

The tuning loop is concrete:

  1. Sweep chunk sizes from 200 to 800 tokens while preserving section titles and table rows.
  2. Compare lexical-only, vector-only, and fused retrieval on the same frozen questions.
  3. Add a reranker for the top 20 only when the score margin between rank one and rank two is small.
  4. Reject stale passages with a freshness filter before generation.

My practical threshold is a 300 ms p95 retrieval target and a 1.2x latency ceiling for reranking. Your mileage may vary; networked model calls, disk type, and language mix can move those numbers. The important part is publishing the threshold beside the relevance score so a fast but wrong build cannot slip through.

A useful failure signal is disagreement. If lexical and vector rankers share fewer than five of their top ten ids, flag the query for review. That often means an acronym, version number, or permission filter was mishandled. It is a better trigger than blindly reranking every request.

Here is the kind of test that changed my defaults. I built a tiny fixture from 32 release notes and 18 incident pages, then wrote queries around exact build ids, slang used by QA, and one deliberately vague question: “Why did matchmaking stall after the weekend patch?” The vector index found the vague incident quickly but missed the exact build id; BM25 did the opposite. Fusion raised recall@10 from the weaker single signal, yet p95 jumped when I sent all 40 candidates to a reranker. Limiting reranking to cases where the top two fused scores were within 0.03 kept the tail under the budget while preserving the useful correction on ambiguous questions. That 0.03 is a local policy, not a universal constant. Recompute it whenever the corpus, embedding model, or hardware changes.

What changes at scale, and what does not fit?

At larger volume, separate ingestion from serving. Parse and chunk documents asynchronously, version the embedding model, and dual-write a new index before switching an alias. Keep a per-document checksum so unchanged notes are not re-embedded. For incident response, pin a snapshot id in the answer metadata; otherwise a reindex can make yesterday's citation impossible to reproduce.

The catch is that hybrid retrieval is extra machinery. It is not suitable when the corpus is tiny, access rules are trivial, and a single local full-text index already meets the deadline. Stick with lexical search in that case. Conversely, a vector-only design is a poor fit for exact build ids, error codes, and command names; those are where lexical matching earns its keep.

I am not sure a cross-encoder reranker belongs on every laptop. Memory pressure and battery cost can outweigh a few points of recall. Test it on the actual hardware, and make the reranker an optional policy rather than a hidden dependency.

Observability needs the same restraint as storage. Export aggregate latency histograms, hit counts, filter rejects, and judge labels. Redact document text from traces. When a user deletes a note, assert that its id disappears from the index and from the next evaluation run. That deletion test is more meaningful than a green health check.

Choose the simplest pipeline that passes both gates: the judged relevance target and the p95 latency target. For this game bot, that means local encrypted storage, ACL-aware hybrid retrieval, selective reranking, and citations tied to immutable document versions. The architecture earns its complexity only when an observed failure requires it.

References

Top comments (0)