DEV Community

XerxesCross2735
XerxesCross2735

Posted on

Retrieval for an Online Course Tutor — 3 Cache Layers for Cited Logistics Results

Short answer: build retrieval for an online course tutor as a two-speed system: cache stable course chunks in a vector index, fetch changing logistics facts with web search, and cache the final evidence only when its citation and access metadata still match. The cache is an evaluation boundary, not a shortcut around grounding.

I use a logistics course as the concrete case. A learner asks, “Why did this shipment miss its delivery window?” The tutor should retrieve the lesson’s definition of a delivery window, then find the current carrier rule or warehouse notice, and show which source supports each claim. A fast answer with a stale citation is a wrong answer.

What should a cached retrieval contract contain?

Start with the answer contract before choosing a cache. For every response, record the query, tenant, learner permissions, course version, retrieved item IDs, source URLs, source timestamps, and the citation spans sent to the model. That record lets an eval harness distinguish a cache hit from a grounded hit.

The stable path is ingestion into a vector index. Course transcripts, glossary entries, and worked examples change when the curriculum is published, so give each chunk a content hash and a course_version. The live path is web search for carrier advisories, customs rules, or weather restrictions. It has a shorter freshness window and a different citation policy.

Do not put both paths behind one anonymous key. A useful cache key includes the normalized question, retrieval mode, tenant, permission scope, course version, and a policy version. If any of those changes, a previous result is not reusable. This is especially important in a multi-tenant tutor: access-control metadata must travel with every indexed item, through retrieval and into the citation check. A practical failure is easy to miss: a learner opens lesson 12, the tutor caches an answer, an editor republishes lesson 12 with a corrected customs example, and a later request still gets the old snippet because the key contains only the question text. The answer can look perfectly coherent while its citation points at a retired paragraph. Include the content or course version in the key, and make the invalidation event visible in your trace so an evaluator can prove why that old result was rejected.

Cache misses are data.

How do you build retrieval and caching for an online course tutor?

The following Python sketch keeps the interfaces boring on purpose. The vector and web clients are interchangeable adapters; the policy decides when their evidence may be reused. In production I put the same fields into tracing attributes so a failed answer can be replayed from its retrieval record.

from dataclasses import dataclass
from hashlib import sha256
from typing import Protocol


@dataclass(frozen=True)
class Evidence:
    text: str
    source_url: str
    source_updated_at: str
    item_id: str
    tenant_id: str
    course_version: str


class VectorStore(Protocol):
    def query(self, text: str, *, tenant_id: str, course_version: str) -> list[Evidence]: ...


class WebSearch(Protocol):
    def search(self, text: str, *, tenant_id: str) -> list[Evidence]: ...


def cache_key(question: str, tenant_id: str, course_version: str, policy: str) -> str:
    raw = "|".join((question.strip().lower(), tenant_id, course_version, policy))
    return sha256(raw.encode("utf-8")).hexdigest()


def retrieve(question: str, tenant_id: str, course_version: str,
             vector: VectorStore, web: WebSearch, cache: dict[str, list[Evidence]]) -> list[Evidence]:
    key = cache_key(question, tenant_id, course_version, "v3")
    if key in cache:
        return cache[key]

    stable = vector.query(question, tenant_id=tenant_id, course_version=course_version)
    live = web.search(question, tenant_id=tenant_id)
    evidence = stable[:4] + live[:3]
    cache[key] = evidence
    return evidence
Enter fullscreen mode Exit fullscreen mode

The seven-item cap is a starting point, not a magic number. Keep enough context for a citation checker, then measure answer quality and prompt tokens. In my eval harness, a cache hit is only a pass when the same question, permission scope, and course version produce the same evidence IDs and the answer cites those IDs. A hit that bypasses the citation check is a miss wearing a fast costume.

Keep the policy visible.

There are three useful cache layers. An embedding cache avoids recomputing vectors for an unchanged chunk. A retrieval-result cache stores ranked IDs and scores for stable course material. An answer-evidence cache stores the selected snippets and their source metadata, but it must have an explicit time-to-live for web evidence. I keep those layers separate because invalidating a changed web page should not force a full re-embed of a course transcript.

When does vector search beat web search, and when does it not?

Vector search wins for semantic questions over material you own: “Which lesson explains cross-docking?” It gives repeatable IDs, predictable access checks, and a clean citation back to a lesson or transcript. Web search wins for volatile facts: “Is the port restriction still active today?” Its result set can change between two identical requests, so cache the source metadata and expiry, not just the generated answer.

The comparison is a trade-off table, not a leaderboard:

Need Vector index Web search
Stable course definitions Strong fit; version the chunks Weak fit; pages may disappear
Current carrier or customs notice Stale unless re-ingested Strong fit with short TTL
Citation repeatability High when item IDs are durable Variable; preserve URL and fetch time
Tenant isolation Metadata filter is mandatory Apply tenant policy before display
Cost and latency Amortized ingestion, fast reads Per-query network work

The catch is that a web cache is not suitable when the learner needs a legally current source with a guaranteed freshness SLA you cannot enforce. In that case, use an authoritative feed or a human-reviewed document store. Stick with a vector-only design when the tutor is strictly course-bound and live facts are out of scope; adding web search then adds failure modes without improving the answer.

How do you test cache freshness and citation grounding?

Build a small, stubborn test set. Include a normal lesson question, a near-duplicate phrasing, a changed course paragraph, an expired web result, a cross-tenant query, and a question whose answer is absent. For each case, assert retrieval recall, citation precision, permission filtering, and invalidation behavior. The RAG paper’s central idea is useful here: generation should be conditioned on retrieved external memory, so your evaluation must inspect both retrieval and generation rather than grading prose alone (Lewis et al., 2020).

I also test the negative path. If the cache contains evidence for tenant-a and the request is for tenant-b, the lookup must return nothing. If course_version changes, the old result must be ineligible even when the text looks similar. If a source timestamp passes its TTL, the tutor should say it needs a fresh lookup instead of quietly citing yesterday’s page.

Measure p95 retrieval latency, cache hit rate by layer, stale-hit rate, citation coverage, and token count. Token cost matters because a cache that returns seven verbose snippets can cost more than a fresh, well-filtered query. Your mileage may vary across embedding models and traffic shapes; the eval set, not a generic hit-rate target, should decide the limit.

An operational checklist that survives a release

Before shipping, pin a cache-key schema and log its version. Store tenant and authorization fields beside every chunk. Give course vectors a content hash and web evidence an expiry. Run the same eval set on cache hits and misses. Alert on stale citations, empty retrieval, and permission-filter drops. Sample the longest prompts for token growth. Finally, make the tutor expose source links and a short “I need a fresh source” response when the policy says cached evidence is no longer trustworthy.

That workflow keeps caching in its proper role: reducing repeated work while preserving grounding. The decision is not “vector or web” in the abstract. It is which evidence can be reused, for whom, for how long, and with what citation proof.

References

Top comments (0)