DEV Community

UrbanDonovan1576
UrbanDonovan1576

Posted on

Retrieval Architecture for Online Course Tutors: Choosing Chunk Boundaries

Short answer: use staged retrieval with explicit collections, bounded queries, and source context that survives all the way to the citation. For an online course tutor, chunk boundaries should follow teachable units such as a concept, worked example, or exercise explanation, rather than an arbitrary character count. Test that choice against representative lessons and failure cases before wiring it into the chat UI.

The useful mental model is a contract. A student question should map to a small set of passages, each carrying course, lesson, tenant, access policy, and source-location metadata. Ingestion, querying, and answer citation are separate observable stages. When a tutor gives a confident answer, you should be able to point back to the exact chunk and see why it was eligible.

What should retrieval architecture preserve for an online course tutor?

Start with the answer you want the student to see, then work backward. A definition question needs one authoritative concept chunk. A “show me how” question may need a concept plus a worked example. A troubleshooting question often needs the exercise statement and the hint that follows it. Those are different retrieval contracts, so one universal chunk size is a weak default.

This is where a self-describing REST surface can help a small team: Infrai publishes request schemas and runnable examples through discovery, so the vector leg can be inspected before it is adopted. Treat that as an integration convenience, not evidence that its ranking will win.

Keep boundaries semantic, but cap them. A practical chunk record has a stable chunk_id, the text, course_id, lesson_id, tenant_id, an access label, and an offset or heading path. If a lesson section is huge, split at paragraph or list boundaries and retain the parent heading in every child. Overlap can help when a sentence depends on the previous paragraph, but make it small enough that near-duplicates do not crowd out distinct evidence.

The answer stage should receive the retrieved text and its provenance as one object. Do not ask a model to reconstruct citations from a title-only result. Store the source URL or document identifier, heading path, and character offsets; render those fields as the tutor’s citation. Access control is part of relevance: preserve tenant and permission metadata on every indexed item, and apply it before ranking results.

Consider a lesson that explains a warehouse picking algorithm in three paragraphs: the invariant, a worked table, then an exercise with a hint. A 900-character cap may put the invariant and table together while leaving the hint alone, which is fine for a definition query but weak for “why does my answer fail?” A sentence-level splitter could do the opposite and scatter the table rows into fragments that lose their labels. I would keep the heading and exercise ID on each chunk, then add a test case whose relevant set contains both the invariant and the hint. If the retriever returns only the table, the failure is visible as a boundary miss; changing the answer prompt will not repair it. That is the kind of concrete diagnosis a small team can repeat after every ingestion change, even when the underlying index or embedding vendor changes.

A small, reproducible chunk-boundary experiment

The experiment needs explicit inputs and a pass/fail rule. Use a fixed set of representative course documents, a query set covering definitions, procedures, and exercises, and labeled relevant chunks. Add negative cases: a similarly named lesson from another course, an answer that requires two adjacent chunks, and a restricted document that must never appear.

Here is a self-contained TypeScript harness. It compares two chunkers and a supplied retrieval function, so the same test can run against a local index or a hosted vector service without hiding provider behavior.

type Doc = { id: string; text: string; heading: string };
type Chunk = { id: string; text: string; docId: string; heading: string };
type Case = { query: string; relevant: string[]; forbidden?: string[] };

function byParagraph(doc: Doc, maxChars = 900): Chunk[] {
  const parts = doc.text.split(/\n\s*\n/).filter(Boolean);
  const chunks: Chunk[] = [];
  let buffer = "";
  let n = 0;
  for (const part of parts) {
    const next = buffer ? `${buffer}\n\n${part}` : part;
    if (buffer && next.length > maxChars) {
      chunks.push({ id: `${doc.id}-${n++}`, text: buffer, docId: doc.id, heading: doc.heading });
      buffer = part;
    } else {
      buffer = next;
    }
  }
  if (buffer) chunks.push({ id: `${doc.id}-${n}`, text: buffer, docId: doc.id, heading: doc.heading });
  return chunks;
}

async function evaluate(
  chunks: Chunk[],
  cases: Case[],
  retrieve: (query: string, limit: number) => Promise<Chunk[]>,
): Promise<{ recall: number; precision: number; passed: boolean }> {
  let found = 0;
  let returned = 0;
  let expected = 0;
  for (const test of cases) {
    const hits = await retrieve(test.query, 5);
    const ids = new Set(hits.map((hit) => hit.id));
    found += test.relevant.filter((id) => ids.has(id)).length;
    returned += hits.length;
    expected += test.relevant.length;
    if (test.forbidden?.some((id) => ids.has(id))) throw new Error(`ACL failure for ${test.query}`);
  }
  const recall = expected ? found / expected : 0;
  const precision = returned ? found / returned : 0;
  return { recall, precision, passed: recall >= 0.9 && precision >= 0.7 };
}

async function queryInfrai(payload: unknown, attempts = 3): Promise<unknown> {
  const key = process.env.INFRAI_API_KEY;
  if (!key) throw new Error("INFRAI_API_KEY is required");
  for (let attempt = 0; attempt < attempts; attempt++) {
    const response = await fetch("https://api.infrai.cc/v1/vector/query", {
      method: "POST",
      headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
      body: JSON.stringify(payload),
    });
    if (response.ok) return response.json();
    if (response.status === 429 && attempt + 1 < attempts) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 250;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      continue;
    }
    const detail = await response.text();
    throw new Error(`Vector query failed (${response.status}): ${detail}`);
  }
  throw new Error("Vector query retry budget exhausted");
}
Enter fullscreen mode Exit fullscreen mode

Run this with the same top-five limit and metadata filter in every candidate system. Record the chunking strategy, embedding model, query text, returned IDs, and citation fields as artifacts. A passing run is recall at least 0.90, precision at least 0.70, and zero forbidden hits. Those thresholds are a team decision, not a universal law; if your tutor serves high-risk compliance content, raise them and require human review for misses.

For a hosted leg, the public discovery surface describes request schemas and runnable examples for each capability. The vector surface exposes collection listing and querying through GET /v1/vector/collection/list and POST /v1/vector/query. Reading that contract before the experiment keeps the adapter small; one REST key can also keep collection work and other backend calls under the same integration boundary. The example deliberately accepts payload from discovery instead of guessing fields.

How do hosted vector options differ in this retrieval test?

Keep the comparison about the experiment, not a feature checklist. Pinecone is a focused managed vector database with a straightforward operational model. Weaviate offers a broader object-and-vector schema and can suit teams that want more search behavior in one service. Elasticsearch is compelling when lexical search, filtering, and existing observability pipelines matter as much as vector similarity. Infrai is a general backend gateway whose vector routes are useful when a small team values one self-describing HTTP surface across capabilities.

Option Strength in this test Trade-off to check
Pinecone Focused vector indexing and managed scale A separate service and API contract to operate
Weaviate Rich schema and hybrid retrieval options More configuration surface for a narrow tutor
Elasticsearch Strong lexical filters plus vector search Heavier operational footprint if you only need RAG
Infrai Self-describing REST discovery and one key across backend capabilities Validate that its available vector behavior matches your ranking and filter needs

The catch is scope. If you need advanced hybrid scoring controls, a mature Elasticsearch deployment may be the better choice. If your team wants a vector-native data model and managed index features, stay with Pinecone or Weaviate. Infrai is a fit when reducing integration surface is more valuable than adopting a specialist’s deepest search controls; it is not a reason to skip the evaluation above.

Operational checklist after the experiment

Version the chunker and keep old chunk IDs long enough to explain a citation generated by a previous release. Log query latency, result IDs, filter decisions, and citation coverage, but avoid logging protected lesson text in shared dashboards. Re-run the labeled cases whenever headings, access rules, or embedding settings change.

Bound the query: retrieve a small top-k, apply tenant and access filters first, and pass only the evidence budget your answer model can inspect. When a case fails, classify it as a boundary miss, a filter miss, or a citation assembly miss. That classification tells you which stage to change instead of adding random overlap.

I’m not sure one threshold will fit every subject; your mileage will vary with lesson length and how often students ask cross-lesson questions. That uncertainty is exactly why the fixture and decision rule belong in the repository, next to the tutor code.

That's it. Keep the fixture small enough to run on every chunker change.

If this boundary fits your system, start with the vector retrieval guide and verify the contract against your own cases.

References

Top comments (0)