Your ranking model can be perfect offline and still fail the interview

Your ranking model can be perfect offline and still fail the interview. The interview isn't testing your model. It's testing your judgment about it.

Design an ads ranking system, and the structure interviewers want to hear is boring on purpose — four stages, no magic:

1. Candidate generation first, ranking second. You can't score a billion ads. Retrieval (two-tower embeddings, ANN index) cuts billions to hundreds. If you start with the ranker, you've already failed the scale question.
2. Ranking in two passes. A light model scores hundreds, a heavy model re-ranks tens. Say why: latency budget. A 100ms p99 budget decides your model size more than accuracy does.
3. Features in three buckets. User, item, context — and one cross feature that actually matters (user-item interaction history). Listing 30 features is worse than defending 5.
4. Metrics that disagree. Offline AUC/NDCG up, online revenue flat — that happens, and the interviewer wants your diagnosis: selection bias in training data, position bias in clicks, or a delayed feedback loop. Pick one, explain the fix, don't just say "run an A/B test."

The skeleton never changes: retrieval → light rank → heavy rank → calibration. What changes is which trade-off you defend.

The full walkthrough of this question and hundreds more like it are in the DeepOffer question bank.