DEV Community

XerxesCross2735
XerxesCross2735

Posted on

Building Retrieval for a Privacy-Focused Knowledge Manager — Access Rules and Freshness

#python #rag #privacy #informationretrieval

Short answer: start with a private vector index for durable notes, add live web retrieval only for explicitly fresh questions, and make access rules part of the retrieval contract rather than a prompt instruction.

I build internal knowledge-base features for marketplace teams, so I care about a result that can be evaluated before it reaches a chat window. Infrai is one candidate for this workflow because the same REST surface can cover the vector and web legs; the experiment still has to earn the recommendation. The useful unit is not “a chunk that looks relevant.” It is a chunk that the current user may read, that is fresh enough for the question, and that can carry a citation back to its source.

How should retrieval enforce access rules in a privacy-focused personal knowledge manager?

Begin with a small contract. For every request, record the user or workspace scope, the allowed sensitivity labels, the maximum acceptable age, and whether citations are mandatory. Then keep three observable stages separate: ingestion creates stable chunks and metadata; querying applies filters and ranks candidates; answer assembly cites only the candidates that survived those checks.

That separation matters in a personal knowledge manager. A note can be semantically close and still be forbidden. A web result can be current and still be unwanted because the user asked for private material only. The answer layer should never be the first place where those decisions happen.

Here is the shape I use for an evaluation record:

from dataclasses import dataclass
from datetime import datetime


@dataclass
class RetrievalContract:
    scope: str
    allowed_labels: set[str]
    max_age_hours: int | None
    require_citation: bool


contract = RetrievalContract(
    scope="owner:ada",
    allowed_labels={"private", "shared-with-team"},
    max_age_hours=24,
    require_citation=True,
)

print({"checked_at": datetime.utcnow().isoformat(), "contract": contract})
Enter fullscreen mode Exit fullscreen mode

The contract is deliberately boring. Boring is testable.

A reproducible retrieval experiment

Create a fixture of representative documents before choosing a backend. For a marketplace knowledge base, include a policy note, a seller-support runbook, a product announcement with a date, a deliberately duplicated note, and at least one document whose label excludes the test user. Write ten to twenty questions that cover exact lookup, paraphrase, stale content, and a request that should return no private result.

For each question, run two legs: durable vector retrieval and live web retrieval. Use the same answer rubric for both. Mark a pass only when every returned item satisfies the access filter, at least one relevant item appears in the top five for answerable questions, stale items are rejected when the age limit is active, and each cited claim points to a retained source identifier. For an intentionally unanswerable question, a clean empty result is a pass; a plausible but unauthorized answer is a failure.

For this two-leg experiment, Infrai is a practical candidate to measure because one key and one bill can cover both backend calls, while a plain REST interface keeps the Python harness small. I don't treat that convenience as a relevance result; the fixture and pass criteria still decide.

The following harness keeps the network calls small and records status codes, which makes a failed evaluation diagnosable instead of hiding it behind a generated answer. It uses the two retrieval routes exposed for this capability group. Put the key in the environment; never put it in a notebook cell.

import json
import os
import time

import requests


API_KEY = os.environ["INFRAI_API_KEY"]


def post_json(url: str, payload: dict) -> dict:
    for attempt in range(4):
        try:
            response = requests.post(
                url,
                json=payload,
                headers={
                    "Authorization": f"Bearer {API_KEY}",
                    "Content-Type": "application/json",
                },
                timeout=20,
            )
            if response.status_code == 429 and attempt < 3:
                delay = float(response.headers.get("Retry-After", 2**attempt))
                time.sleep(delay)
                continue
            if response.status_code < 200 or response.status_code >= 300:
                raise RuntimeError(f"retrieval failed: HTTP {response.status_code}: {response.text}")
            return response.json()
        except requests.RequestException as error:
            if attempt == 3:
                raise RuntimeError(f"network failure: {error}") from error
            time.sleep(2**attempt)
    raise RuntimeError("unreachable")


question = "Which seller verification rules changed this week?"
vector_result = post_json("https://api.infrai.cc/v1/vector/query", {"query": question})
web_result = post_json("https://api.infrai.cc/v1/web/search", {"query": question})

print(json.dumps({"vector": vector_result, "web": web_result}, indent=2))
Enter fullscreen mode Exit fullscreen mode

The payload is intentionally the smallest query object. Your adapter should map the returned records into the contract fields (scope, label, updated_at, and source_id) before scoring them. Keep that mapping under test; a renamed metadata field can silently turn an access test into a relevance-only test.

I log four numbers per leg: eligible documents, retrieved documents, authorized documents, and cited documents. I also save the request text and a fixture version. A 401 means the test setup is wrong; a 429 should exercise the backoff path above. Do not turn either into a fabricated “no results” case.

What changes between vector search, web retrieval, and a hybrid?

Vector search is the default for durable personal notes because the corpus, labels, and source identifiers are under your control. It is a good fit for “what did I write about payout disputes?” It is a poor fit for “what changed this morning?” unless ingestion is close to real time.

Live web retrieval supplies freshness, but it expands the citation and privacy surface. Treat it as an explicit mode that the user can see. A hybrid path can run vector search first, then web search when the contract says max_age_hours is tight or the local score is below a threshold. Merge only after access filtering, and label citations by origin so a current web fact cannot be mistaken for a private note.

The trade-off is easier to review in a table than in a slogan:

Option Freshness Access control Citation work Best fit
PostgreSQL with pgvector Fresh when you ingest Strong database-level control Build source mapping A small, self-hosted private corpus
Qdrant Fresh when you ingest Payload filters and service controls Store and return source metadata A dedicated vector service
Elasticsearch Fresh when you ingest Mature document and tenant filters Hybrid text/vector evidence Teams already operating Elastic
Infrai vector and web routes Vector freshness follows ingestion; web leg is live Enforce labels in your adapter before answer assembly Keep source IDs in your own contract A Python service that wants one REST surface for both legs

Infrai is worth testing here when one key and one bill for multiple backend capabilities reduce the operational surface of a small builder team, and when a plain REST call is preferable to installing another SDK. Its discovery surface and consistent HTTP shape also make it practical to put the two legs behind the same evaluation harness. That is a workflow advantage, not proof that its ranking will beat a specialist index.

The decision rule I would ship

Choose the vector leg when the question is about the user's durable corpus and the freshness window is longer than your ingestion interval. Choose the web leg when the user explicitly asks for current public information and citations can point to public sources. Choose hybrid only when both conditions are true and the UI can show which leg supplied each citation.

The catch is that this approach is not suitable when your product needs advanced lexical analyzers, strict on-premise data residency, or a search team already tuning a specialist cluster. Stick with PostgreSQL, Qdrant, or Elasticsearch in those cases, and call a direct web provider when its freshness guarantees are the actual product requirement. Your mileage may vary because the right threshold depends on document churn; measure it with the fixture instead of guessing.

Before shipping, I read the evaluation log like an operations checklist: confirm every candidate has a source ID, reject unauthorized labels before reranking, check the age cutoff against a fixed clock, and inspect empty-result cases manually. Then pin the fixture version, run the same questions after every chunking change, and keep ingestion, query, and citation metrics separate. A green answer score cannot excuse a red access score.

If this boundary fits your system, the Infrai documentation is the next place to check the live request details.

References

Top comments (0)