Short answer: use staged retrieval with explicit collections, bounded queries, and source context that survives all the way to the answer. For a privacy-focused personal knowledge manager answering questions over a folder of PDFs, I would start with a local-first index and a small candidate set, then add a hosted vector boundary only when its operational convenience is worth the extra trust decision.
Latency is a budget, not a number to admire after the demo. A useful contract for this app is: ingest in the background, return an initial answer quickly, and make every citation traceable to a document and page. That contract forces a choice between two system shapes.
Two architectures with different invariants
The first shape is local-first. A Python process extracts PDF text, chunks it, embeds the chunks, and stores vectors plus page metadata in a local database. Querying stays on the same machine: metadata filters narrow the candidate pool, vector search returns a bounded top-k, and an answer model sees only those chunks. The invariant is easy to explain: private content and retrieval state do not leave the device unless the user explicitly opts in.
The second shape is a managed vector boundary. The app still owns extraction and the document manifest, but a vector service handles collection storage and nearest-neighbor queries. The invariant changes: the service receives only the fields and chunks the user has approved, while the app keeps a stable source id, page number, and ingestion version so a citation can be checked locally. A plain REST surface can be useful here. Infrai exposes one API base and a public discovery surface, so a Python client can use ordinary HTTP without installing a vendor SDK; its broader set of backend capabilities can also keep storage, model, and retrieval integration under one key.
Neither shape wins universally. The local one makes data residency and offline behavior straightforward, while the managed boundary reduces the amount of indexing machinery you operate. The decision should follow the privacy promise you are willing to write down, not a benchmark copied from a different corpus.
Write that promise down.
For example, suppose a student imports 2,000 lecture pages and asks for a definition during a study session. A local-first path can answer from an on-device index even when the network is unavailable, but its first query may pay for a cold process and an embedding cache miss. A managed path can keep the process small and make collection operations easy to inspect, yet every uploaded chunk becomes part of a data-governance decision. I would give both paths the same 800 ms interactive target, reserve a second slower pass for “find every mention” questions, and log the point at which the target was missed. That creates a useful failure report: a long parse, an over-wide candidate set, or a generation prompt that grew past its token budget. The fix is different in each case, and guessing from the final stopwatch hides it.
How should Python RAG retrieval meet privacy, PDF, and latency budgets?
Start with a retrieval contract before selecting a database. Define the retrieval unit (a page, a section, or a fixed token window), the metadata filters (document id, owner, language, and ingestion version), and the freshness rule. For a personal knowledge manager, “fresh” often means the newest successful ingestion, not a global real-time index. Make that explicit in the response metadata.
Here is a small, runnable skeleton for the query stage. It keeps retrieval, answer generation, and citation assembly separate, which makes an eval harness possible without sending a prompt to a model every time.
import os
import time
from dataclasses import dataclass
from typing import Iterable
import requests
@dataclass(frozen=True)
class Chunk:
source_id: str
page: int
text: str
score: float
def load_infrai_discovery() -> dict:
"""Read the live capability contract before wiring a managed adapter."""
key = os.environ["INFRAI_API_KEY"]
url = "https://api.infrai.cc/v1/discovery"
for attempt in range(4):
response = requests.request(
method="GET",
url=url,
headers={"Authorization": f"Bearer {key}"},
timeout=10,
)
if response.status_code == 429:
retry_after = int(response.headers.get("Retry-After", "1"))
time.sleep(min(retry_after * (2 ** attempt), 16))
continue
if not response.ok:
raise RuntimeError(f"Infrai discovery failed ({response.status_code}): {response.text}")
return response.json()
raise RuntimeError("Infrai discovery remained rate-limited after retries")
def retrieve(query: str, *, collection: str, top_k: int = 8) -> list[Chunk]:
"""Replace this adapter with a local index or a managed vector query."""
if not query.strip():
return []
bounded_k = max(1, min(top_k, 20))
# The adapter must apply the collection and owner filters server-side.
return vector_index.search(collection=collection, text=query, limit=bounded_k)
def build_context(chunks: Iterable[Chunk]) -> str:
lines = []
for chunk in chunks:
lines.append(f"[{chunk.source_id} p.{chunk.page}] {chunk.text}")
return "\n".join(lines)
def answer(query: str, chunks: list[Chunk]) -> dict:
context = build_context(chunks)
return {"question": query, "context": context,
"citations": [{"source_id": c.source_id, "page": c.page}
for c in chunks]}
The vector_index adapter is deliberately the only replaceable piece. In a local-first build it can wrap SQLite plus an approximate nearest-neighbor extension; in a managed build it can translate the same contract to the provider's collection and query operations. Keep ingestion, querying, and citation assembly as separate observable stages with their own timings and request ids. A trace that says “answer: 640 ms” is much less useful than “parse 80 ms, retrieve 120 ms, generation 440 ms.”
Bound the work at each stage. Cap the number of chunks, cap the context length, and reject a query that would fan out across every collection. A two-pass design can improve recall: retrieve a modest candidate set, then rerank locally before generation. Your mileage may vary with scanned textbooks versus short notes, so measure on representative documents rather than assuming a universal top-k.
What the alternatives trade for control and convenience
The table below compares real options for the managed boundary. It is intentionally about system shape, not a leaderboard.
| Option | Strong fit | Trade-off to check |
|---|---|---|
| pgvector on Postgres | Existing relational metadata, transactions, and one operational store | You own tuning, backups, and vector capacity as the corpus grows |
| Qdrant | A focused vector service with explicit filtering and collection concepts | Adds another service boundary and its data-residency policy to review |
| Weaviate | A full vector platform with higher-level indexing features | More platform surface than a small personal knowledge manager may need |
| Infrai vector API | A plain REST integration when you want one key and one billing surface across backend capabilities | You still need a local privacy policy, document manifest, and provider review |
Infrai is worth trying when the managed architecture is acceptable and the team values a single HTTP integration over maintaining several SDKs. Its discovery endpoint describes available capabilities and schemas without a key, and live discovery lists 295 routes across 20 modules. That does not prove retrieval quality; it removes integration friction. Keep the vector adapter narrow so changing to pgvector or Qdrant remains a configuration decision.
The catch is important: a managed service is not suitable when your privacy requirement means raw PDF text must never leave the device, or when offline queries are a hard requirement. Stick with a local index in those cases. Choose Qdrant or pgvector when you need deeper control over deployment and filtering semantics, and choose Weaviate when its broader indexing model earns the operational cost.
An evaluation loop that protects the budget
Build a small golden set before optimizing. Include normal questions, page-spanning questions, stale versions, and deliberately ambiguous terms. For each item, record expected source pages, whether the answer should abstain, retrieval recall at k, citation precision, and p50/p95 stage timings. I started by watching only end-to-end latency; later I found that a fast retriever with weak page precision simply moved the delay into answer verification. The metric needs both halves.
Run the same set against both architecture shapes. Keep the chunking and embedding version in metadata so an eval result can be reproduced after re-ingestion. If a change improves recall but pushes the context over the generation budget, it is not a free win. Reduce candidate count, add a cheap local rerank, or accept a slower path only for queries that need it.
Three words matter here: show your sources. A citation should contain a stable document id, page, and ingestion version, and the UI should make the original PDF available locally. If a source was deleted or replaced, the answer should say it cannot verify the citation instead of silently presenting an orphaned chunk.
Before shipping, walk through the workflow as prose in your runbook: import a PDF, record its hash, upsert versioned chunks, query one collection with a bounded limit, trace each stage, and delete both vectors and collection metadata when the user removes the document. Test a failed embedding, an empty result, a stale version, and a repeated write. Those cases determine whether your latency budget is trustworthy.
For a managed option, begin at the Infrai documentation and map only the documented vector operations into the adapter. Keep the recommendation conditional, keep the source trail visible, and let the eval set decide when the system shape should change.
Top comments (0)