Asked in a Gen AI engineer interview.
What they are really testing:
They want to see whether you pick chunking based on the documents, the query patterns and the retrieval metric, rather than applying a default chunk size. They also want to know how you would prove the choice works.
A strong answer:
- Start from the corpus structure. Policy manuals, API docs, code, and chat logs have different natural units (sections, functions, turns). Respect those boundaries first, then apply size limits, instead of splitting every document into fixed windows.
- Set size and overlap from the embedding model and the answer shape. Many teams start around 300 to 800 tokens with 10 to 20 percent overlap, but the real target is whether a chunk can answer a question on its own with enough surrounding context.
- Consider hierarchical or parent-child designs. Index small chunks for precise matching and return the parent section to the LLM for context. This often beats one flat chunk size for long technical documents.
- Attach metadata to every chunk: source, section heading, date, version, access controls. Filtering on metadata often matters more than tuning the splitter.
- Validate with an evaluation set. Build question-to-gold-passage pairs, measure recall@k and answer faithfulness across two or three chunking variants, and let the numbers decide.
- Revisit the choice when the corpus or query mix changes, and log retrieval failures to see whether they are chunk-boundary problems.
Common mistakes:
- Using a fixed 512-token splitter everywhere without checking whether answers get cut across chunks.
- Optimizing chunk size on a handful of demo questions instead of a labeled retrieval set.
- Ignoring tables, code blocks and headings, which lose meaning when split mid-structure.
How would you approach this? Share the signals you would use to decide between semantic, structural and fixed-size chunking.
Originally shared on LinkedIn.
An AI interview question every day, leadership for tech people and hands-on AI learning:
Top comments (0)