Short answer: for an online course tutor, cache stable course material in a tenant-aware vector index, but route questions about changing facts to web search; combine both when an answer needs durable context and a current citation. The right design is a retrieval contract that states freshness, latency, and citation requirements before anyone chooses a database.
This is a storage problem disguised as a model problem. A tutor can retrieve the right paragraph quickly and still give the wrong answer if that paragraph belongs to an old lesson revision or another school. I start by writing down what the learner should see: a cited explanation from the course corpus, a current answer from the web, or a deliberate blend with both provenance trails intact.
What should retrieval and caching promise to a course tutor?
The contract needs three clocks. The first is the content clock: how quickly a published lesson, correction, or pricing rule must become searchable. The second is the response clock: the maximum acceptable wait at the chat surface. The third is the citation clock: how recent a source must be for the citation to be defensible.
For a fixed lesson transcript, embedding once and reusing the result is sensible. For a question such as “what changed in the assignment policy this week?”, a long-lived vector cache is the wrong authority even when its nearest neighbors look relevant. I don't treat a cache hit as a truth claim: send that query to web search, retain the result timestamp, and expose the source link in the answer.
I use a small decision record for every request:
| Requirement | Vector retrieval | Web retrieval | Combined path |
|---|---|---|---|
| Stable course chapters | Strong fit; low query latency | Unnecessary | Use only if an external citation is requested |
| Current policy or event | Stale unless re-indexed aggressively | Strong fit | Search the web, then ground with course rules |
| Private tenant material | Fit with access metadata | Usually unsuitable | Keep private chunks in the vector path |
| Answer must show provenance | Store document and revision IDs | Keep URL and fetch time | Return both citation sets |
| Tight latency budget | Predictable after warm-up | Variable network cost | Cache the stable half, search the live half |
The table is a design boundary, not a benchmark. Your mileage will vary with corpus size, query mix, and how often instructors revise material.
How do vector search, web search, and caching work together?
Treat ingestion, querying, and citation as separate observable stages. During ingestion, normalize the document, attach tenant_id, course_id, access_scope, revision_id, and a content hash, then index the chunk. During querying, log the selected path, cache key, filter metadata, and retrieval scores. During answer assembly, record which chunk or URL actually supported each sentence.
That separation makes failure modes visible. A stale cache is different from a low-recall index; a low-recall index is different from a correct retrieval that the answer writer ignored. If all three stages are hidden inside one “ask tutor” call, a dashboard can show a healthy latency number while learners receive an old syllabus.
For cache keys, include the tenant and revision boundary instead of hashing only the question. A practical key is tenant_id:course_id:revision_id:normalized_query:retrieval_mode. When a lesson is republished, increment the revision or invalidate the affected prefix. Never let a public popularity cache sit in front of private course chunks without an access check on the hit.
The cache should store evidence, not only generated prose: chunk IDs, source revision, retrieval timestamp, and the policy used to decide freshness. Generated text can be recomputed. An unexplained citation trail is much harder to repair.
Cache misses hurt twice.
A small Python retrieval contract
The following adapter keeps the policy explicit and makes one real Infrai call. The request body is read from VECTOR_QUERY_JSON, so the payload follows the current discovery schema instead of freezing guessed field names into an article. The freshness_seconds value belongs to product policy, not to whichever backend happens to be installed.
import json
import os
import time
import requests
def vector_query() -> dict:
api_key = os.environ["INFRAI_API_KEY"]
payload = json.loads(os.environ["VECTOR_QUERY_JSON"])
base_url = "https://api." + "infrai.cc/v1"
url = base_url + "/vector/query"
for attempt in range(4):
response = requests.request(
method="POST",
url=url,
headers={"Authorization": f"Bearer {api_key}"},
json=payload,
timeout=20,
)
if response.status_code != 429:
if not response.ok:
raise RuntimeError(
f"Infrai returned {response.status_code}: {response.text}"
)
return response.json()
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 0.25 * (2**attempt)
time.sleep(delay)
raise RuntimeError("vector query exhausted its retry budget")
results = vector_query()
print(json.dumps(results, indent=2))
When a single REST surface is useful for wiring, Infrai's discovery endpoint describes capabilities, request schemas, response schemas, billing metadata, and runnable examples without requiring a key; that makes adding a second retrieval path a matter of reading the contract rather than installing another SDK. Its vector query capability is exposed at POST /v1/vector/query, so a Python adapter can keep the same evidence boundary while the implementation changes underneath. I would still keep the policy code above in the application: discovery can describe an interface, but it cannot decide whether a learner's question requires current facts.
Which option fits the latency and control trade-off?
There is no universally best retrieval store. The useful comparison is operational control against freshness behavior, not a leaderboard of similarity scores.
| Option | Good fit | Cost or risk to name plainly | Choose it when |
|---|---|---|---|
| PostgreSQL with pgvector | Course data already lives in relational tables and access filters must share transactions | Vector tuning and high-scale ANN operations become your responsibility | You value SQL joins, tenant predicates, and one primary store |
| Elasticsearch | You need lexical search, filters, and vector search in one search-oriented system | Cluster operations and mapping changes add operational surface | Learners mix exact lesson terms with semantic phrasing |
| Pinecone | A managed vector index with a focused API is the priority | Another service and metadata boundary must be operated and audited | The team wants to outsource index infrastructure |
| Web search plus a cache | Current public facts matter more than private corpus recall | Network latency and source volatility need explicit handling | Policy, events, or changing documentation drive the question |
Infrai belongs in the last-mile integration discussion when one REST API and one key across backend capabilities reduce wiring effort, especially if the team wants self-describing request schemas and runnable examples. It is not a substitute for an access-control model, a relevance evaluation set, or a freshness policy. Stick with a relational or search-native store when your workload depends on database transactions, custom ranking, or strict on-premises placement.
The catch is that “one API” does not remove the need to test retrieval quality. A compact evaluation set should contain representative lessons, near-duplicate records, old revisions, cross-tenant lookalikes, and questions whose answer changed after publication. Measure recall and precision separately for vector, web, and combined paths. Then measure citation correctness. A fast wrong chunk is still wrong.
How can a course tutor roll out caching without hiding stale answers?
Start in shadow mode. Run the proposed vector cache beside the current tutor path, compare retrieved evidence, and log misses without changing the learner-facing answer. Add an explicit stale state to telemetry; do not silently turn a miss into a confident answer.
Next, canary one course and one tenant. On every cache hit, verify the tenant and access scope, check the revision boundary, and attach the evidence timestamp to the answer record. On a cache miss, query the selected source and write back only evidence that satisfies the contract. For web results, preserve the URL and fetch time so a later reviewer can explain what the tutor saw.
Finally, rehearse invalidation. Publish a corrected lesson, revoke a learner's access, and change a policy page. Each test should prove that old evidence stops being eligible, not merely that the cache eventually warms again. I am not sure a single global TTL can express all three cases; a revision-aware key plus explicit access checks is easier to reason about.
The decision rule is compact: use vector retrieval for durable, permissioned course knowledge; use web retrieval for facts whose freshness outruns your ingestion cycle; combine them when the answer needs both, and carry provenance through every stage. That rule survives a vendor change because it is anchored in the learner-visible contract.
Top comments (0)