I got asked to "design a machine learning system" in one interview and "design a GenAI system" in another, about two weeks apart. I walked into the second one assuming it was basically the same interview with different vocabulary.
It was not. And figuring out why took me embarrassingly longer than it should have.
Same energy, different everything
Both interviews start the same way: someone describes a product, and you're expected to design the system behind it. But the actual engineering problems underneath are pretty different, and conflating them is an easy way to give an answer that sounds right but misses what the interviewer is actually probing for.
Classic ML system design: prediction is the product
Think recommendation engines, fraud detection, search ranking, ad click prediction. The core loop looks like this:
Collect and process training data (often at massive scale, often messy)
Train a model offline
Serve predictions online, usually with a strict latency budget
Monitor for model drift and retrain periodically
The hard problems here are things like: how do you keep training data and serving data consistent (the classic training/serving skew issue)? How do you A/B test a new model safely? How do you handle a feature store that needs to serve both batch and real-time features?
It's fundamentally a data pipeline and prediction-serving problem. The model itself is often treated as a black box you're building infrastructure around.
GenAI system design: retrieval, generation, and control are the product
Now think RAG-powered support bots, AI coding assistants, document Q&A systems. The shape of the problem is completely different:
Retrieval: how do you find the right context to feed the model (chunking strategy, embeddings, vector search)?
Generation: which model do you call, how do you prompt it, how do you handle streaming responses?
Control: how do you keep the model from hallucinating, leaking data it shouldn't, or going off-script?
You're usually not training a model from scratch-you're orchestrating a pipeline around a model someone else built. The hard problems are things like keeping a knowledge base fresh without full re-indexing, routing between cheap and expensive models based on query complexity, and designing guardrails that catch bad outputs before a user sees them.
Where people mix them up
The most common mistake I see (and made myself) is bringing classic ML instincts into a GenAI interview. If someone asks you to design a RAG system and your answer is mostly about training data pipelines and model evaluation metrics, you're solving the wrong problem-the LLM itself usually isn't something you're training, so that entire section of classic ML system design doesn't apply here.
The reverse mistake also happens: bringing RAG/prompt-engineering instincts into a classic ML interview, like suggesting you'd "prompt" a fraud detection model, which doesn't map onto how that kind of system actually works.
A rough way to tell which one you're in
Ask yourself: is the interviewer's problem centered on predicting a number or a label from structured data (churn probability, fraud score, ranking position)? That's classic ML system design.
Is it centered on generating or retrieving unstructured content in response to a query (an answer, a summary, a piece of code)? That's GenAI system design.
If you're not sure which one you're being asked, it's a completely fair clarifying question to ask out loud-"just to confirm, are we training a model here, or orchestrating around an existing LLM?" Interviewers generally respect that question, because it shows you know the two are different disciplines instead of assuming they're the same skill with new terminology slapped on.
Why this gap matters more in 2026
Two years ago, most system design interviews with an ML flavor were classic ML system design. That's shifting fast-GenAI-flavored questions (RAG pipelines, AI agent architecture, vector database design) are showing up regularly now, especially at companies building AI-native products, and a lot of interview prep material hasn't caught up. It's easy to prep the "old" version of the ML interview and get blindsided by the new one.
I ended up leaning on ScaleDojo's GenAI Systems Lab specifically to close this gap-it's a dedicated track for RAG pipelines, AI agent design, vector databases, and model routing, separate from classic HLD/LLD practice, which turned out to be exactly the distinction I was missing.
If your prep has only ever covered one of these two flavors, it's worth checking which one, before you find out the hard way in an interview which one you were actually asked.
Top comments (0)