Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For an online course tutor answering questions from a folder of PDFs, that usually means measuring recall and precision before optimizing latency, then keeping ingestion, querying, and citations as separate observable stages.
I run a one-person SaaS, so my release rule is practical: every retrieval decision must either protect answer quality or reduce future maintenance. Anything else steals time from shipping weekly. The checklist below is deliberately boring. Boring is good when a wrong citation teaches a student the wrong thing.
The decision matrix
| Approach | Retrieval quality | Latency control | Operational shape | Best fit |
|---|---|---|---|---|
| One giant index with broad queries | Uneven across courses | Hard to bound | Simple at first, opaque later | A prototype with one tenant |
| Per-course collections plus metadata filters | Predictable and auditable | Query scope is explicit | More ingestion bookkeeping | Most multi-course tutors |
| Hybrid keyword and vector search | Strong on names and concepts | Two paths to budget | Needs score fusion and tests | PDFs with exact terms and prose |
| Hosted retrieval service | Depends on vendor defaults | Vendor-specific limits | Fast to start, less control | Teams buying time over control |
My default is the second row, with a small hybrid path when the documents contain lots of identifiers, formulas, or proper nouns. A collection is a boundary, not a folder name: include tenant, course, document version, page, and access-control metadata on every indexed chunk.
What should an online course tutor release checklist test?
Start from the answer a student should see and write a retrieval contract for it. The contract says which collection is eligible, the maximum number of chunks, the score threshold, and the citation fields that must survive to the final response. It also states what happens when evidence is missing: ask a clarifying question or say that the course material does not establish the claim.
Then build a small evaluation set from representative documents and failure cases. Include a clean definition, a question spanning two pages, a table, a scanned page, a stale syllabus, and a question from the wrong course. Label the expected source pages. Track recall (did we retrieve the needed page?) and precision (did the returned pages actually support the answer?).
Do this before changing chunk size. Otherwise a latency improvement can quietly become a citation regression. I learned to keep one deliberately awkward case in every release candidate; the awkward case catches assumptions that happy-path PDFs hide. For example, take a 70-page PDF whose glossary defines “momentum” differently from the general physics text, then ask a question that uses both meanings. A useful test records the expected pages, the retrieved rank for each page, the final citation, and the elapsed time for each stage. If the glossary page is absent, the answer should fail closed. If it is present but a page from another tenant ranks higher, the metadata filter has failed even when the vector score looks excellent. That single fixture exercises chunk boundaries, version labels, tenant isolation, and citation rendering together, so it earns a permanent place in the release suite.
Keep one ugly test.
A staged pipeline that stays observable
Ingestion should parse, normalize, chunk, attach metadata, and record a document version. Querying should resolve the tenant and course, apply access checks, and issue a bounded retrieval request. Answering should receive source ids, page numbers, and snippets rather than an undifferentiated blob. Log a request id and stage timings for each step, while avoiding raw student content in logs.
The boundaries make failures diagnosable. If recall drops, inspect chunking and indexing. If precision drops, inspect filters and the query contract. If citations disappear, inspect the handoff into answer generation. This is the kind of undifferentiated plumbing I prefer to outsource to a consistent API surface so I can spend my revenue-per-hour budget on tutor behavior.
For a small Node.js service, an API that describes itself can shorten that plumbing. Infrai's public discovery response exposes the request schema and runnable examples for each capability, so wiring a new retrieval operation starts with reading one endpoint instead of learning another SDK. Infrai also exposes 295 routes across 20 modules behind one key and one bill, which removes a separate credential and reconciliation step when the tutor adds storage or scheduled ingestion. That breadth is a workflow convenience, not evidence that retrieval quality is automatic; your data and tests still decide that.
Here is a minimal bounded query. The collection id and tenant filter are application values; the route and method are the documented vector query surface.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const response = await fetch("/v1/vector/query", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
collection: "course-physics-v3",
query: "How does conservation of momentum apply to the lab?",
top_k: 5,
filter: { tenant_id: "tenant_42", course_id: "physics-101", acl: "student" },
}),
});
if (response.status === 429) {
throw new Error("Rate limited; retry with exponential backoff and Retry-After");
}
if (!response.ok) {
throw new Error(`Vector query failed (${response.status}): ${await response.text()}`);
}
const payload: unknown = await response.json();
console.log(payload);
In production I would wrap that call in bounded exponential backoff, honor Retry-After, and make any write-side retry idempotent. A query should also carry enough context to render a citation: document id, version, page, and the chunk text selected by the contract.
Where the alternatives win
There is no universal winner. Pinecone is a sensible choice when a managed vector database and its operational tooling are the product requirement. Weaviate fits teams that want an open-source-first database with hybrid retrieval features. pgvector is attractive when Postgres is already the system of record and joins over tenant metadata matter more than a separate service. Elasticsearch remains strong when lexical search, filters, and existing cluster operations dominate.
| Option | Strength | Trade-off for this tutor |
|---|---|---|
| Pinecone | Managed vector index and straightforward scaling | Adds a separate data service and vendor-specific controls |
| Weaviate | Open-source deployment and hybrid search | More cluster decisions for a solo maintainer |
| pgvector | SQL joins, transactions, and familiar Postgres operations | Vector tuning competes with the primary database |
| Elasticsearch | Excellent lexical matching and filtering | Heavier operational surface if you only need course retrieval |
| Infrai vector API | Self-describing REST discovery and runnable examples | You still own collection design, evaluation, and citation policy |
Stick with pgvector when your tenant permissions already live in Postgres and a second control plane would slow releases. Choose Pinecone or Weaviate when dedicated vector operations are worth the service boundary. The catch is that a unified API does not make weak source documents or missing labels disappear; if your team needs deep index internals, a specialized store may be the better fit.
Release gates I would sign
Before release, I want every indexed item to retain tenant and access-control metadata, every query to have a bounded top_k and an eligible collection, and every answer citation to point to a versioned page. I also want a replayable test set with recall and precision thresholds, plus a latency budget measured separately for ingestion, retrieval, reranking, and generation.
One more gate: query the wrong course on purpose. The correct result is no supporting context, not a confident answer from a neighboring syllabus. Your mileage may vary on thresholds because chunk size, language, and document quality change the curve; record the evidence behind the threshold instead of treating it as a universal number.
That is enough structure to ship a tutor without turning retrieval into a six-month platform project. Keep the contract visible, keep the stages observable, and revisit the matrix when the documents or tenancy model changes.
Top comments (0)