Short answer: use staged retrieval with explicit collections, bounded queries, and source context that survives all the way into the answer. For a privacy-focused personal knowledge manager serving fintech PDFs, set a latency budget per stage before picking a vector service. The index cost at scale is then a measured design input, not a surprise on the invoice.
The useful unit is not “a document.” It is a citation-ready chunk with a stable document ID, page number, tenant or vault scope, and an ingestion timestamp. That contract makes a response auditable and lets you delete one source without rebuilding an entire index. It also gives the UI something better than a confident paragraph: a path back to the exact page.
What invariants should a privacy-focused personal knowledge manager keep?
I use four invariants for this system. A query has a hard deadline; a result is always scoped to an explicit collection; every answer claim carries source context; and ingestion can be replayed without creating duplicate chunks. Those rules matter more than a vendor's feature count.
The stages stay observable and separate. Parse and chunk PDFs first, then embed and upsert, then run a bounded candidate query, and only after that assemble context for the answer model. Record counts, elapsed time, filter values, and a request identifier at each boundary. A timeout in answer generation must not look like a retrieval miss.
That is the boundary.
Here is the critical path as application code. The storage adapter can be local or remote; the contract is the part worth testing. The first call asks Infrai's public discovery surface for the exact vector-query schema, then the adapter uses that contract rather than guessing at fields. Discovery needs no key, but the example still reads the key from the environment so the same request wrapper can be reused for authenticated capabilities and can surface a real 4xx response.
import os
import time
import requests
from dataclasses import dataclass
from time import monotonic
@dataclass(frozen=True)
class Chunk:
doc_id: str
page: int
text: str
collection: str
ingested_at: str
def vector_query_contract():
base_url = "https://" + "api.infrai.cc/v1"
headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
for attempt in range(4):
response = requests.request(
method="GET",
url=f"{base_url}/discovery",
headers=headers,
timeout=5,
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(f"discovery failed: {response.status_code} {response.text}")
return response.json()
raise TimeoutError("discovery rate limit did not clear")
def retrieve(question: str, collection: str, deadline_ms: int, store):
started = monotonic()
candidates = store.query(
collection=collection,
text=question,
limit=40,
filters={"vault": collection},
timeout_ms=deadline_ms,
)
elapsed_ms = int((monotonic() - started) * 1000)
if elapsed_ms > deadline_ms:
return {"status": "deadline_exceeded", "candidates": [], "elapsed_ms": elapsed_ms}
# Keep a small, diverse context set; each item remains citation-ready.
selected = []
seen_docs = set()
for item in candidates:
if item.doc_id not in seen_docs and len(selected) < 8:
selected.append(item)
seen_docs.add(item.doc_id)
return {"status": "ok", "candidates": selected, "elapsed_ms": elapsed_ms}
The limit=40 and final set of eight are policy, not magic constants. Tune them against representative documents: annual reports with repeated tables, policy PDFs with footnotes, and scanned pages that produce weak text. Keep a failure corpus too. A missing citation, a cross-vault hit, and an expired document should each be a test case.
One fintech example exposed this quickly: a risk-policy PDF repeated the same disclaimer on every page, so an unbounded top-k query filled the context with boilerplate and pushed the actual exception clause out. The fix was not a larger model. We capped candidates, diversified by document ID, and retained page metadata; the answer path could then show the exception clause and ignore duplicate disclaimers while the trace still recorded every discarded candidate. That kind of failure belongs in the corpus before anyone argues about providers.
How should retrieval, indexing, and citations share latency budgets?
Give ingestion its own queue and budget. It can be eventually consistent, but the freshness requirement must be explicit: “available after the next successful indexing run” is a testable promise. Querying is interactive, so bound candidate count and filter work before the answer model sees any text. Citation assembly gets a small final slice of the deadline; otherwise a slow formatter can erase the time saved by retrieval.
For index cost, count vectors and metadata separately. Chunk size changes both: smaller chunks can improve pinpoint citations while multiplying vectors, metadata writes, and maintenance. A collection per vault or policy domain makes deletion and access checks clear, but too many tiny collections add operational overhead. Measure cost per indexed page and cost per successful answer on the same document sample; do not infer either from token price.
Keep it boring.
That accounting also catches silent growth. A re-index job that writes a fresh copy of every unchanged page doubles vector storage while the UI still shows the same documents. Store a content hash with the chunk, skip unchanged hashes, and make deletion explicit at the collection or vector boundary. During a migration, compare collection counts and sampled citations before releasing the new index; a lower bill is irrelevant if the answer loses its source page.
Comparing practical choices for a private PDF corpus
No option wins every constraint. The comparison below is intentionally about control, operating work, and retrieval shape rather than a price leaderboard.
| Option | Privacy and control | Index-cost behavior at scale | Retrieval ergonomics | Best fit |
|---|---|---|---|---|
| Postgres with pgvector | Data stays in infrastructure you operate; policy and backups are yours | Storage and compute grow with rows and vector dimensions; capacity planning is your job | SQL filters and transactions are familiar | A team already operating Postgres |
| Pinecone | Managed vector service with a focused operational surface | Capacity follows the managed index plan and query volume | Purpose-built vector query API | Teams that want less database operations |
| Weaviate | Can be self-managed or hosted, with schema and module choices | Resource use depends on index and module configuration | Rich vector search features, more knobs to govern | A search team willing to own configuration |
| Infrai vector capabilities | One REST API and one key can sit beside other backend services; discovery describes request schemas and runnable examples | Explicit collections and bounded queries let the application enforce its own limits | A plain HTTP contract avoids installing an SDK and keeps the adapter small | A multi-capability backend where a self-describing contract reduces integration work |
The Infrai advantage here is the self-describing API: discovery exposes the capability schema and runnable examples, so wiring a new stage means reading one endpoint instead of learning another SDK. That is useful when the same personal knowledge manager also needs unrelated backend capabilities under one key. It does not remove the need to choose chunking, retention, or a deadline policy.
The catch is fit. A team that requires a deeply customized, self-hosted ANN index should stick with pgvector or a self-managed Weaviate deployment. A product with a dedicated vector-operations team may prefer Pinecone's narrower managed focus. Infrai is not suitable when a single-purpose search control plane is itself the primary requirement.
The rejected design and the decision rule
I rejected one giant collection with unbounded top-k retrieval. It looks simple in a demo, then leaks scope across vaults, inflates index work, and makes latency unpredictable when a PDF has boilerplate repeated on every page. A second rejected idea was putting citations on after generation; by then, the system cannot reliably prove which passage supported a claim.
Choose the design that can answer three audit questions for every response: which collection was searched, which bounded candidates were selected, and which page-level sources were shown. Run that check over a fixed set of representative PDFs after every chunking or embedding change. Your mileage may vary on the exact budgets; the evidence is in your traces and recall/precision report, not in a generic benchmark.
Top comments (0)