DEV Community

Manohar Papasani
Manohar Papasani

Posted on

Design a RAG system

#aiinterview #rag #genai #aiarchitect

Asked in an AI engineer interview.

What they are really testing: whether you clarify requirements before drawing components, and whether you know which parts a first version can skip.

A strong answer:

  • Start with requirements: corpus size and update rate, query types, latency and cost budget, access control (per-document permissions enforced at retrieval time, not left to the prompt), and how quality will be measured.
  • Ingestion: parse and clean documents, chunk along structural boundaries (often a few hundred tokens with modest overlap), attach metadata such as source, date, version and ACL, and make indexing idempotent so re-runs update only changed documents.
  • Retrieval: use hybrid search, combining BM25 for exact terms and identifiers with dense embeddings for semantic matches. Apply metadata and permission filters before ranking, pull a broad candidate set (roughly 50 to 100), then rerank with a cross-encoder down to a handful of passages.
  • Generation: pass the retrieved passages with stable source IDs, instruct the model to answer only from that context with citations, and make "not found in the sources" an explicit, acceptable output rather than a guess.
  • Evaluation and operations: keep a golden set of questions with expected sources, measure retrieval recall at k separately from answer faithfulness, trace every request, and track latency and cost per query.
  • Optional pieces: knowledge graphs, multi-agent loops and fine-tuning are not needed for a first version. Add them only when evaluation shows a specific failure they would fix.

Common mistakes:

  • Drawing the vector database and LLM boxes first without asking about data freshness, scale or permissions.
  • Assuming the LLM compensates for weak retrieval, and never measuring recall at k.
  • Applying access control after generation, or relying on the prompt to hide restricted content.

How would you handle a corpus where documents change hourly and users have different access levels? Share your approach below.

Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (0)