Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context; make each fallback narrower and more conservative than the previous stage.
An online course tutor is a small production system with an awkward failure mode: a plausible answer backed by the wrong lesson is harder to notice than an obvious error. I have been paged for missed jobs and duplicate deliveries, so I treat retrieval the same way I treat queue work: every stage needs a contract, a timeout, and an idempotent handoff.
The contract starts with the answer the student should see. A question about “the gradient descent lab” should return a lesson section, its course and tenant, and a citation anchor. If the first pass cannot satisfy those fields, it must say so and move to a defined fallback. Silent broadening is how cross-course answers leak into a tutor.
What should an online course tutor retrieval architecture do when fallbacks fire?
Keep ingestion, querying, and citation as separate observable stages. Ingestion normalizes a representative set of lessons, preserves tenant and access-control metadata on every indexed item, and records a stable document identifier. Querying searches the course collection first, with a bounded top-k and a deadline. Citation assembly checks that returned identifiers still belong to the requesting tenant before the language model sees them.
Fallback one can widen within the same course: a lexical match over titles and headings catches an unusual term that embeddings missed. Fallback two can search a curated, tenant-wide collection, but it should require a stronger confidence threshold and return fewer passages. The final fallback is an explicit “I could not find this in the course material” response, with a path for the instructor to add or correct a source.
That last step matters. A tutor that always produces text has no useful failure signal.
I don't let a timeout silently promote a broader search. In one runbook review, the dangerous path was a retry that reused the old broad query after the course index had recovered; the answer looked fine, but its citation belonged to a neighboring course. The fix was a stage token carried through the trace, a maximum of three retrieval attempts, and a final authorization check against the enrollment snapshot. That sequence takes more lines than a single search call, yet it makes a duplicate delivery and a late response converge on the same decision. The tutor either has an approved source or returns no answer.
I model each retrieval result as a record, not a loose chunk:
package tutor
type Source struct {
DocumentID string
TenantID string
CourseID string
Section string
Text string
Score float64
}
func usable(s Source, tenantID, courseID string) bool {
return s.TenantID == tenantID && s.CourseID == courseID && s.Text != ""
}
The check is deliberately boring. A score is not authorization. A high-scoring chunk from another tenant is still unusable, and a low-scoring chunk should not be promoted merely because the first query timed out.
How do bounded queries and source context make fallback behavior testable?
Give every stage a budget and a reason code. For example, course_vector_timeout, course_low_recall, and tenant_lexical_hit are more useful in a runbook than one boolean called retrieval_failed. The application can then alert on a rising fallback rate without paging on a single hard question.
Evaluation needs representative documents and failure cases: short definitions, long lab instructions, duplicated slide text, misspelled terms, and questions that intentionally have no answer. Measure recall for finding the right section and precision for keeping irrelevant sections out. Also test a query whose only matching passage has a revoked enrollment; the expected result is no citation, not a clever guess.
A practical trace records the collection, stage, query hash, result IDs, scores, elapsed time, and the final citation IDs. Do not log raw student prompts by default. Retain enough context to replay a miss while respecting the tenant's data policy.
The write path should be equally explicit. A collection is created before ingestion, vectors are upserted with the metadata needed for authorization, and deletion removes both the item and its collection membership when a course is retired. These are separate state transitions, so a retry worker can safely repeat a transition keyed by the document ID.
Comparing retrieval backends for a course tutor
The choice is less about a leaderboard score than about who owns the operational boundary. Pinecone is a managed vector service with a focused API. Weaviate offers an open-source and managed path with a broader schema layer. Postgres with pgvector keeps vectors beside transactional course and enrollment data, which can simplify joins and policy checks. Infrai is another option when a team wants one plain REST API and no SDK to install; its discovery surface is public and self-describing, and its wider backend surface can keep adjacent services behind one key.
| Option | Where it fits | Trade-off to make explicit |
|---|---|---|
| Pinecone | Teams that want a managed vector index and a specialized operations surface | Course authorization and relational joins remain application responsibilities |
| Weaviate | Teams that value a schema-oriented vector platform with self-hosted or managed deployment choices | More platform surface to operate and secure than a narrow index |
| Postgres + pgvector | Existing Postgres estates where enrollment filters and vectors should share a transaction boundary | Index tuning and capacity planning stay with the database team |
| Infrai | A Go service that prefers direct HTTP, one credential, and a consistent multi-service interface | Provider-specific vector controls and advanced ranking features still need verification against the live contract |
The catch is that a broad interface does not remove retrieval design work. Use a direct specialist when you need a feature your evaluation depends on, such as a particular hybrid-ranking implementation, tight database transactions, or a self-hosted deployment mandate. Stick with Postgres when the authorization join is the hard part. Choose a managed vector service when your team cannot own index capacity and recovery. Your mileage may vary because corpus shape and enrollment rules dominate the result.
A small, observable deletion and retry path
Here is the part I want in a runbook: deletion is a state transition, and retries carry the same document identity. The route names below are the documented vector operations; the request body should be generated from the current discovery schema rather than copied from an invented example.
package main
import (
"fmt"
"net/http"
"os"
)
func request(method, path string) (*http.Response, error) {
baseURL := os.Getenv("INFRAI_BASE_URL")
req, err := http.NewRequest(method, baseURL+path, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
return (&http.Client{}).Do(req)
}
func main() {
resp, err := request(http.MethodDelete, "/v1/vector/delete")
if err != nil {
panic(err)
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("delete failed: %s", resp.Status))
}
}
This intentionally shows status handling and the bearer-header shape, but not a made-up payload. In a real worker, add the document identity and idempotency key required by the discovered schema, retry 429 responses with bounded exponential backoff, and emit the stage trace before acknowledging the queue message. Never send the credential to a presigned URL or expose it to a browser.
I would not use this sample as a complete ingestion client. It is a control-path sketch: discovery supplies the exact schema, while the tutor owns authorization, retries, and citation policy. That division keeps a provider change from rewriting the answer contract.
The decision rule I would ship
Start with a course-scoped collection and a strict citation contract. Add lexical fallback only after measuring misses on representative lessons. Add tenant-wide search only with an explicit policy and a higher threshold. Keep a no-answer outcome, because it is the only fallback that cannot invent a source.
Before launch, replay duplicate upserts and deletes, revoke a student's enrollment mid-query, and inject timeouts at each stage. Confirm that every emitted citation maps to the requested tenant and course. Then watch fallback rate, citation rejection rate, and unanswered-question samples as first-class SRE signals.
Top comments (0)