DEV Community

Cover image for 🎯 The AI Engineer 🤖 Interview Playbook 📖
Truong Phung
Truong Phung

Posted on

🎯 The AI Engineer 🤖 Interview Playbook 📖

#programming #tutorial #ai #webdev

Everything you need to prepare for — and pass — an AI engineer interview in 2026. Straightforward, organized, and built from what companies actually test.

Synthesized from data-driven field research and practitioner guides: Alexey Grigorev's AI Engineering Field Guide (4,894 job descriptions + 100+ candidate stories), Amit Shekhar's AI Engineering Interview Questions, Rohit Ghumare's AI Engineering from Scratch, IGotAnOffer's AI engineer guide (with Meta engineering leader Viral G), Brian Kihoon Lee's Interviewing for ML/AI Engineers (Modern Descartes), 365 Data Science, and the writings of Chip Huyen, Eugene Yan, Hamel Husain, and successful candidates (Mimansa Jaiswal, Yuan Meng, Janvi Kalra).

📋 Table of Contents

⚡ TL;DR

The AI engineer role is software engineering with AI systems on top — you orchestrate models (LLMs, RAG, agents) into reliable products, not train models from scratch. Interviews test six things: ML/LLM fundamentals, applied ML, LLM/RAG engineering, coding, AI system design, and behavioral.

If you remember one thing: companies are hiring AI system builders, not people who can call an LLM API. The fastest way to stand out — think like a product + system owner, be explicit about failure modes, and show evaluation rigor. Evaluation is the single biggest skill gap among candidates, so it's your biggest opportunity.

The rest is discipline: solid DSA + Python, 2–3 deployed end-to-end projects, and the ability to explain trade-offs (quality vs. latency vs. cost) out loud.

1. 🧭 What an AI engineer actually is

The role is new and definitions are still settling, so the first job is knowing what you're being hired for.

Core responsibility: integrate AI into a product. Work with LLM providers (OpenAI, Anthropic) through their APIs, partner with PMs to find real user problems AI can solve, and ship reliably. It starts from a real problem — not "AI is cool, let's use it."

🔀 AI engineer vs. ML engineer vs. data scientist

Focus Owns Day-to-day
AI engineer Building with models Prompts, pipelines, integration RAG, prompting, tools, agents, evals
ML engineer Optimizing models Model weights, training Training, features, metrics
Data scientist Creating models Datasets, experiments Requirements → ML, modeling

The lines are blurry and the industry treats them as a spectrum. In practice, most postings are "ML engineer" or "software engineer with an AI focus." The consistent message from hiring managers: "Companies are not hiring for titles — they want to know if you can build reliable AI systems." If you can only do modeling or only do systems, you're already behind.

📈 Progressive complexity (know where a problem sits)

  • Simple: user input → prompt + LLM API → response.
  • RAG (~5× harder): add data pipelines, a search engine (vector/text), retrieval, reliability.
  • Agents (~10× harder): add tool calls, multi-step loops, trace instrumentation, tool-rollout management.

🚫 What AI engineers usually don't do

Create models from scratch, build custom architectures, or do heavy feature engineering. What they do: engineering best practices for AI systems, prompt design + versioning, product integration, and evaluation + monitoring.

2. 🗺️ The interview process (what to expect)

Based on analysis of real job postings and candidate reports: the median process is 4 steps, most fall in the 3–5 range, and the whole thing runs 2–6 weeks.

Round Typical length What it tests
Recruiter / talent screen 15–30 min Fit, salary expectations
Technical / coding 45–60 min LeetCode-style, sometimes AI-flavored
AI/ML deep-dive 45–90 min LLMs, RAG, hallucinations, fine-tuning vs. prompting
Take-home / project 1–7 days Build a RAG or agent system
AI system design 60 min Scale LLM apps, cost/latency optimization
Behavioral 30–60 min STAR/SAIL, ownership in ambiguous work
Hiring manager / founder 15–60 min Deep dive, motivation, values

🏢 Real loops (from candidate reports)

  • Mistral AI (Applied AI Engineer): LLM theory → coding → project deep-dive → tech manager → ML system design → take-home → values talk.
  • Amazon (GenAI, L6): LeetCode + practical ML coding (cosine similarity in NumPy) → SDE bar → GenAI depth (LLM/ViT architectures, fine-tuning, ROI estimation) → Leadership Principles throughout.
  • Eightfold.ai (Agentic AI): AI-agent-conducted coding round → 3-day take-home to build an agent → DSA interview with EM.
  • LangChain (AI Engineer): take-home (build an agent) → solution discussion → applied system design.
  • PostHog: talent call → 60-min technical → co-founder call → paid full-day SuperDay (compensated real work).
  • Microsoft (Applied AI/ML intern): AI-assisted coding (use ChatGPT, then re-prompt on a modified problem) → raw coding, no AI tools → behavioral.

Two trends to know: (1) in-person rounds are back (up from ~24% in 2022 to ~38% in 2025) to counter cheating; frontier labs increasingly require onsites. (2) References matter more — most top companies now want 2–3 references from recent managers.

3. 🎯 The six question categories

Nearly every AI engineer loop draws from these six buckets. Prepare all six; weight by seniority and role.

  1. ML & deep learning fundamentals — bias/variance, overfitting, precision/recall, ROC, gradient descent, CNNs, transformers, BERT/GANs.
  2. Applied ML & infrastructure — pipelines, fine-tuning, transfer learning, FP32/FP16/BF16 trade-offs, sparse vs. dense, deployment.
  3. LLM engineering & RAG — tokenization, context limits, cost/latency, hallucination, embeddings, vector search, chunking, grounding, re-ranking.
  4. Coding / Python fundamentals — DSA (indexing/search/graph/tree/heap), Python internals (GIL, is vs ==, mutable/immutable, async), SQL.
  5. AI system design — end-to-end pipelines, caching, cost, reliability, failure modes.
  6. Behavioral — ambiguity, communication, influence, AI ethics, trade-off ownership.

📌 Focus by seniority

Level Emphasis
Junior / Intern Coding fundamentals, basic ML concepts, project enthusiasm, willingness to learn
Mid End-to-end system knowledge, RAG pipelines, embeddings, production awareness
Senior Trade-off fluency, system design at scale, failure-mode reasoning, cost optimization
Staff+ Technical leadership, cross-team influence, project presentations, org impact

At senior/staff levels, interviewers pick 3–5 topics and drill deep into failure modes and trade-offs rather than covering many topics superficially. Depth beats breadth.

4. 🧠 Core knowledge checklist

The must-know surface area, grouped so you can self-audit. You don't need every advanced item, but you must be fluent in the basics and have opinions backed by trade-offs.

🔤 LLM fundamentals

  • Transformers: self-attention, Q/K/V, multi-head attention, positional encoding (RoPE), encoder vs. decoder vs. encoder-decoder.
  • Tokenization: BPE, WordPiece/SentencePiece, why domain terms get split badly.
  • Generation controls: temperature, top-p/top-k sampling, logits, context window, why the first token is slow (prefill vs. decode).
  • Efficiency: KV cache, quantization (INT8/INT4, FP16/BF16), distillation, MoE, Flash Attention, GQA.
  • Alignment: RLHF, DPO, instruction tuning, reward hacking, the "alignment tax."

📚 RAG (table stakes — expect deep questions)

  • Architecture: chunk → embed → index → retrieve → re-rank → generate.
  • Chunking strategies: fixed, recursive, semantic, parent-child. How to pick chunk size.
  • Retrieval: dense vs. sparse embeddings, cosine/dot/Euclidean, ANN, hybrid search, re-ranking.
  • Failure modes: hallucination despite good context, "lost in the middle," multi-hop questions, conflicting sources, stale data.
  • Query transforms: HyDE, decomposition, step-back prompting. Citation/source attribution.
  • The key trade-off: RAG vs. fine-tuning vs. prompt engineering — and when you'd NOT use RAG.

🤖 Agents

  • ReAct, Plan-and-Execute, Reflection patterns; tool use / function calling; MCP.
  • Agent memory (short-term, long-term, episodic); the agent loop and stop conditions.
  • Failure handling: infinite loops, wrong tool selection, bad parameter extraction, token/budget blowups, guardrails against irreversible actions.
  • Single vs. multi-agent; orchestration; human-in-the-loop.

🎛️ Fine-tuning

  • Full vs. PEFT; LoRA / QLoRA; prefix/prompt tuning; adapters.
  • When to fine-tune (extreme specialization or latency) vs. default to prompt + RAG.
  • Catastrophic forgetting, dataset prep, key hyperparameters (LR, epochs, LoRA rank).

🚀 LLMOps / production

  • Serving (vLLM, continuous batching, speculative decoding, paged attention).
  • Prompt caching, semantic caching, streaming, structured output.
  • Observability: TTFT, inter-token latency, tokens/sec, per-user cost, tracing, drift.
  • Cost & reliability: model routing, fallbacks, rate limiting, graceful degradation, provider redundancy.

🛡️ Safety

  • Prompt injection (direct/indirect), jailbreaks, data leakage, PII handling.
  • Input/output guardrails, content filtering, red teaming, hallucination detection.

💡 Depth test: interviewers value "when would you NOT use RAG?" over "what is RAG?" Every concept should come with a trade-off and a failure mode.

5. 💻 The coding round

The role is still mostly software engineering, so DSA fundamentals are non-negotiable. Algorithm rounds appear at OpenAI, Anthropic (90-min CodeSignal requiring perfect correctness), xAI (LeetCode Hard), Eightfold, and more.

What to drill

  • DSA: NeetCode 150/250, focus on patterns (indexing/search/graph/tree/heap) — not memorization. Use spaced repetition.
  • Python depth: GIL, concurrency vs. parallelism, async patterns, race conditions, is vs ==, mutable vs. immutable, reproducible code.
  • SQL: for handling datasets.
  • Full-stack basics: many AI roles are "low-key full-stack" — expect JS event loop, database choices, message queues.

AI-flavored coding (common warm-ups)

  • Cosine similarity / dot product / Euclidean distance from scratch (NumPy).
  • A basic RAG pipeline; semantic search; chunking strategies.
  • A simple agent with tool use; a function-calling handler.
  • Retry with exponential backoff; token counting / context management; a semantic cache.
  • From-scratch ML (frontier labs): multi-head attention, a transformer layer, LoRA, KV cache from memory. Use shape suffixes (Noam Shazeer method) to track tensor dimensions. Note: these rounds are often 25–35 min, no debugging.

⚠️ Modern interviewers may run AI-assisted coding rounds (solve with ChatGPT, then re-prompt when they change the problem). They're testing how you prompt, verify, and direct the tool — not whether you can code unaided.

6. 🏗️ AI system design

This is where senior candidates win or lose. The bar isn't "name the tools" — it's end-to-end system thinking plus a clear grasp of how the system breaks.

🧱 The frame that works

Present every solution as a pipeline, then stress-test each stage:

Input → Retrieval → Generation → Verification → Feedback
Enter fullscreen mode Exit fullscreen mode

For each stage, answer: how does it fail, and how would you fix it? "If you can't explain how your system breaks and how you'd fix it, you're not ready."

6 habits that impress

  1. Lead with product & business metrics. Anchor on user value: task success, retention, latency, cost — before naming a model.
  2. Think in lifecycles, not static pipelines. Start simple, measure, find bottlenecks, iterate. "Only add complexity where it moves metrics."
  3. Be fluent in trade-offs. Quality vs. latency vs. cost; internal model vs. external API; retrieval depth vs. hallucination risk.
  4. Call out failure modes proactively — hallucination, bad retrieval, prompt brittleness — and your mitigation.
  5. Show evaluation rigor (see §7).
  6. Demonstrate pragmatic judgment: "I wouldn't use an LLM here — it's overkill," "we can get 80% with a cheaper model + rules," "gate expensive calls behind a confidence threshold."

💵 Cost reasoning separates production thinkers from prototypers

Be ready to estimate on the whiteboard. Example:

100K daily users × 10 interactions × ~2K tokens = 2B tokens/day ≈ $13K/day on a premium model.

Then talk mitigation: caching, batching, model routing, smaller models behind confidence gates.

⚖️ Trade-off cheat sheet

The decisions interviewers drill most. For each, know the default, the trigger that flips it, and the cost of getting it wrong.

Decision Lean A when… Lean B when… Default
RAG vs. fine-tuning Knowledge is large/fresh/factual You need fixed format, tone, or behavior RAG first; fine-tune for style, not facts
RAG vs. long context Corpus is big or changes often A few docs fit and it's one-off RAG for scale; long context for one-shot
Prompt vs. fine-tune Iterating fast, low volume Consistent behavior at high scale/low latency Prompt + few-shot first
Bigger vs. smaller model Hard reasoning, quality-critical Simple/high-volume tasks, cost/latency matters Route: small by default, escalate on difficulty
Dense vs. sparse retrieval Semantic/paraphrase matching Exact terms, codes, names Hybrid — you rarely pick just one
More vs. less context Answer needs broad grounding Precision matters, cost/latency tight Retrieve broad, re-rank down to the best few
Single vs. multi-agent One coherent task Genuinely separable, parallel subtasks Single — multi adds latency, cost, failure modes
Sync vs. streaming Structured output / tool result User-facing chat/long answers Stream anything a human waits on
Self-host vs. API Sensitive data, scale economics, control Speed to ship, no infra burden API first; self-host when cost/compliance demands
Build vs. buy Core differentiator Commodity (vector DB, eval tooling, gateways) Buy the undifferentiated, build the edge
Small vs. large chunks Precise fact lookup Answers need surrounding context Small chunks + parent-child for context
Sync vs. batch/async Interactive, low-latency need Bulk jobs, cost-sensitive throughput Batch offline, sync only when latency matters

💡 The meta-pattern: almost every answer starts "it depends" — then names the metric that decides. Quality vs. latency vs. cost is the triangle underneath most of these; say which corner the use case actually cares about.

Common prompts

Design a RAG "chat with your docs," a deep-research agent, a multi-agent support system, an LLM inference platform, a recommender, content moderation, or an AI email assistant. A good scenario starts from a real user need and leaves the solution open — practice extracting the problem and asking clarifying questions before designing.

🎣 If you get an outdated prompt (e.g., "design a fixed-context RAG chatbot" when an agentic search design fits better), it's a signal about the company — its engineers may not be current. Answer well, but read the signal.

7. 📊 Evaluation — your biggest differentiator

Evaluation is the biggest skill gap among AI engineer candidates, which makes it your biggest edge. "Unsuccessful LLM products almost always share a common root cause: a failure to create robust evaluation systems."

What to be able to discuss

  • Metrics beyond accuracy: faithfulness (is it grounded?), usefulness (does it solve the user's problem?), safety (does it resist harmful inputs?).
  • Classic metrics & when they apply: BLEU, ROUGE, BERTScore — and their limits.
  • LLM-as-a-judge / G-Eval — how it works and its limitations (bias, self-preference).
  • RAG eval: faithfulness, answer relevance, context precision/recall (Ragas, DeepEval).
  • Offline vs. online: eval sets + regression suites vs. A/B tests + human-in-the-loop.
  • Golden datasets & continuous evaluation for catching regressions when a provider ships a new model.

The "beyond just call the API" story

Professional AI engineering, even for a simple task, looks like this — and telling this story signals real production experience:

  1. Prompt testing with known inputs/expected outputs
  2. An evaluation dataset that produces a metric
  3. Iterate on the prompt → rerun evals → confirm no regression
  4. Roll out via A/B test to a small cohort
  5. Production monitoring (error rates, failure cases)
  6. Collect logs; inspect inputs/outputs for misalignment
  7. Human annotators sample prod data → add hard cases to the eval set
  8. New provider model? Rerun the eval set to check for regressions
  9. Version prompts (Git/MLflow)
  10. Collect explicit (👍/👎) and implicit (user corrections) feedback

🎯 Prepare one concrete evaluation story from your own work — how you measured quality and detected regressions. It's the single most impactful thing you can bring.

8. 📦 The take-home assignment

Take-homes are common (build a RAG app or an agent, typically 2–3 hours to 3 days). Treat them like a mini job, not a homework problem — this is where strong candidates pull ahead.

How to win it

  • Document your decisions and the trade-offs behind them (a short DECISIONS.md).
  • Test edge cases and include an eval harness — even a small one.
  • Show production readiness: Docker, a bit of CI, basic monitoring/logging — not just a notebook.
  • Record a short Loom walking through your solution and reasoning.
  • Make trade-offs explicit: why this chunking strategy, why this model, where it would break at scale.

📈 A real example: one engineer built a CLI tool for summarizing PDFs with configurable models and chunking strategies, documented it well, and had two competing offers within 72 hours.

Some companies gate even earlier — a GitHub portfolio, a "best project" write-up with metrics, or a short essay on where companies go wrong with AI. Have 2–3 polished projects ready before you apply.

9. 🗣️ Project deep-dive & behavioral

Project deep-dive

You'll present a real project (class project, research, portfolio, or work). Interviewers assess seniority, communication, and depth. Structure it as: motivation → problem statement → approach → difficulties → trade-offs → impact (with metrics).

  • Talk like a builder, not a researcher: "We tried fine-tuning but it hallucinated too often, so we switched to hybrid RAG."
  • Lead with impact and metrics, then dive into the technical how.
  • Choose a project where you can go genuinely deep on follow-ups.

Behavioral

AI engineers get AI-flavored behavioral questions on top of the standard ones: comfort with ambiguity, influence without authority, explaining complex AI to non-technical stakeholders, and AI ethics.

  • Use SAIL (Situation, Action, Impact, Learning) or STAR. Map stories explicitly to company values.
  • Prepare distinct examples per interview — repeating the same stories sounds mechanical.
  • Common prompts: an unexpected challenge you solved, a time you used data in a high-ambiguity setting, how you handled a model producing biased/harmful output, a quality-vs-latency decision, how you'd explain to a PM why a 15% edge-case hallucination rate is risky.

Read up on AI ethics beforehand: bias mitigation, PII/GDPR, guardrails, appeals/audit trails.

10. 🌟 What separates candidates who get offers

Patterns from 50+ AI engineer interviews at top startups and multiple successful candidates:

  • The first 5 minutes decide a lot — lead with impact, not model names.
  • Cost awareness is a superpower. One engineer showed a before/after breakdown proving a 70% cut in OpenAI spend → offer the next day.
  • Honesty beats bluffing. "I haven't used LangSmith, but if you use it for evals I'd love to understand your metrics setup" → turned into an offer. "I need a hint" outperforms bluffing.
  • You don't need to be a unicorn. Companies hire strong generalists with depth in 1–2 areas. "Why you, why not anyone else?" is the central question — domain depth and passion alignment correlate with success more than flawless execution everywhere.
  • One brilliant answer on a fundamental can carry a mediocre interview — and failing one fundamental can tank a strong one.
  • Tinkerer mindset. Strong, current opinions on tools; comfort with uncertainty.
  • Verbal fluency signals experience. Practice explaining trade-offs out loud without hesitation.

The 90/10 rule: ~90% of interview success comes from prior career decisions and built skills; only ~10% is application strategy, networking, and negotiation. Invest in the skills first.

11. ⚠️ Common mistakes to avoid

Mistake Fix
Jumping to fine-tuning too early Default to prompt + RAG; fine-tune only for extreme specialization/latency
Treating the LLM as a source of truth Ground with retrieval, tools, or citations
Skipping evaluation & monitoring Always explain how you measure quality and catch regressions
Name-dropping tools without trade-offs Explain why LangChain/Redis/etc. — and when it's the wrong choice
Ignoring failure modes Discuss what breaks, how it's detected, graceful degradation
Over-engineering from the start Get a working version first; optimize on follow-ups
Bluffing on gaps Ask for a hint; disclose limits honestly
Weak fundamentals Know tokenization, transformers, next-token prediction, the GIL, race conditions
Not asking clarifying questions Questions demonstrate communication and scope control
Only chasing compensation Have a real answer to "what problem do you want to solve?"
AI-polished generic applications Recruiters detect it; authentic materials + referrals win

12. 📅 An 8–12 week prep plan

A proven timeline from candidates who landed offers at top labs and startups.

Weeks Focus Actions
1–2 Coding fundamentals NeetCode 150/250, patterns over memorization
3–4 ML/LLM implementation Transformers, attention, LoRA, KV cache from scratch in NumPy/PyTorch (practice on Deep-ML)
5–6 System design RAG architecture, agentic patterns, model serving; read Chip Huyen's AI Engineering + target-company eng blogs
7–8 Portfolio Build/polish 1–2 projects with evaluation, deployment, docs
9–10 Mock interviews Verbal trade-off explanations, SAIL/STAR stories, system-design walkthroughs aloud
11–12 Company-specific Study the target's blog, products, values; refine your self-presentation blurb; record yourself

🛠️ Build 2–3 end-to-end projects

A RAG app, an autonomous agent, and something deployed (Docker + CI + monitoring, not a notebook). "Start the job before you have it" — building is how you get the specific knowledge courses can't give you. Hackathons and building in public beat passive courses when the field moves this fast.

📚 High-signal resources

  • Books: Chip Huyen — AI Engineering (2025); Simon Prince — Understanding Deep Learning; Designing Data-Intensive Applications (skim ch. 1–11); Alex Xu — System Design Interview.
  • Courses/videos: Andrej Karpathy — Neural Networks: Zero to Hero; Maven — AI Evals for Engineers & PMs (Hamel Husain, Shreya Shankar).
  • Articles: Eugene Yan — Patterns for Building LLM-based Systems; What We Learned from a Year of Building with LLMs; Chip Huyen — Building a GenAI Platform.
  • Coding practice: NeetCode 250 (spaced repetition), Deep-ML (from-scratch ML), Great Frontend (for full-stack roles).
  • Question banks: the three GitHub repos in Sources — study the categories and drill trade-offs, don't rote-memorize.

13. 💰 Offers & negotiation

  • Move fast. Top candidates accept within 2–3 weeks; cluster your onsites so offers land together for leverage.
  • A competing offer is your strongest lever. Direct it toward equity grant size — base bands per level are narrow.
  • Benchmark total comp, not base. Equity/bonuses/AI-experiment credits can add 20–40%. AI engineers earn ~10–20% more than general SWEs.
  • Vet startups like an investor: revenue + growth rate, market size, customer loyalty, competitive position. Refusing to share financials after an offer is a red flag.
  • Watch expiration pressure. Ask for extensions on 7-day windows; refusal can signal cultural issues.

(Compensation varies widely by company, level, and location; treat any number as a rough anchor, not a quote.)

14. ❓ 80 most common questions (with answers)

Rapid-fire prep across the essential topics. Answers are deliberately tight — say this much, then be ready to go one level deeper on trade-offs and failure modes if pushed.

🔤 LLM fundamentals

1. How does an LLM generate text?
Autoregressively — it predicts a probability distribution over the next token given all previous tokens, samples one, appends it, and repeats. Two phases: prefill (process the whole prompt in parallel) and decode (generate tokens one at a time, which is why output is slower than input).

2. What is the attention mechanism?
Each token builds a Query, Key, and Value vector. Attention scores every token against every other via Query·Key, softmaxes to weights, and produces a weighted sum of Values — letting each token pull in context from the whole sequence. Multi-head runs this in parallel subspaces to capture different relationships.

3. What's the difference between encoder, decoder, and encoder-decoder models?
Encoder-only (BERT) sees the full sequence bidirectionally → good for classification/embeddings. Decoder-only (GPT) is causal/left-to-right → good for generation. Encoder-decoder (T5) encodes an input then decodes an output → good for translation/summarization.

4. What is tokenization and why does it matter?
Splitting text into subword units (BPE/WordPiece). It matters because cost, context limits, and latency are all measured in tokens, and rare/domain terms get split into many tokens — hurting quality and price.

5. What do temperature and top-p do?
Both control randomness. Temperature scales the logits before softmax (higher = flatter distribution = more random). Top-p (nucleus) samples only from the smallest set of tokens whose cumulative probability ≥ p. Use low temp for deterministic tasks, higher for creative ones.

6. What is the context window and why is it a constraint?
The max tokens (prompt + output) a model can attend to at once. Cost and latency grow with it, and quality degrades in the middle of long contexts ("lost in the middle"), so more context isn't always better.

7. What is a KV cache?
During decode, the Keys and Values of prior tokens are cached so each new token doesn't recompute attention over the whole history. It's the main reason generation is fast — at the cost of GPU memory that grows with sequence length.

8. What is quantization?
Storing the model's numbers (weights/activations) at lower precision (FP16, INT8, INT4) instead of full 32-bit floats — like rounding 3.14159 to 3.14. This cuts memory and speeds up inference for a small accuracy loss, so a model that needed an A100 might run on a laptop GPU.

9. What is RLHF?
Reinforcement Learning from Human Feedback. Humans rank model outputs best-to-worst; those rankings train a small reward model that scores answers; then the LLM is fine-tuned to maximize that score (or you skip the reward model and optimize preferences directly with DPO). It's what turns a raw next-token predictor into a helpful, aligned assistant.

10. Why do LLMs hallucinate?
They're trained to produce plausible continuations, not true ones — there's no built-in fact-checker. They confidently fill gaps when knowledge is missing, outdated, or the prompt is ambiguous. Mitigate with grounding (RAG), tools, and asking for citations.

11. What is positional encoding, and what is RoPE?
Attention itself is order-blind, so the model must be told each token's position. Classic transformers add fixed sinusoidal encodings to the embeddings; modern LLMs use RoPE (rotary position embedding), which rotates the Query/Key vectors by an angle based on position. RoPE encodes relative distance and extrapolates better to longer contexts.

12. Greedy vs. sampling vs. beam search?
Greedy always takes the single most likely next token — deterministic but often dull or repetitive. Sampling (with temperature/top-p) draws randomly from the distribution — diverse and creative. Beam search keeps several candidate sequences and picks the best overall — strong for translation/summarization, rarely used for open-ended chat.

📚 RAG

13. What is RAG and when would you use it?
Retrieval-Augmented Generation: fetch relevant documents at query time and inject them into the prompt so the model answers from your data. Use it for private/fresh/large knowledge bases and to reduce hallucination — without retraining the model.

14. Walk me through a RAG pipeline.
Ingest → chunk → embed → store in a vector index. At query time: embed the query → retrieve top-k (often hybrid dense + keyword) → optionally re-rank → build a grounded prompt → generate with citations → evaluate/monitor.

15. How do you choose a chunking strategy?
Match chunks to retrieval units: too small loses context, too large dilutes relevance and wastes tokens. Start with recursive/semantic chunking (~200–500 tokens with overlap); use parent-child when you retrieve small but need broad context for generation.

16. Dense vs. sparse retrieval — and what is hybrid search?
Dense (embeddings) captures meaning — it matches "car" with "automobile." Sparse (BM25/keywords) captures exact terms — product codes, names, error strings. Hybrid runs both and merges the rankings (e.g., reciprocal rank fusion, which blends the two ranked lists into one), so you get semantic recall without missing literal matches. It usually beats either alone.

17. What is re-ranking?
A second-stage model (cross-encoder) that re-scores the top-k retrieved chunks by joint query-document relevance. It's slower per item but much more accurate, so you retrieve broadly then re-rank down to the best few.

18. Your RAG returns good documents but still hallucinates. What's wrong?
The generation step, not retrieval. Check the prompt (is it instructed to answer only from context?), conflicting/duplicate chunks, "lost in the middle" ordering, or too much context. Fix with tighter prompting, citations, fewer/better chunks, and faithfulness evals.

19. RAG vs. fine-tuning vs. long context — how do you choose?
RAG for changing/large/factual knowledge. Fine-tuning for behavior, format, or style the model should internalize (not for facts). Long context for one-off documents that fit. They combine — fine-tune for tone, RAG for facts.

20. How do you evaluate a RAG system?
Separate retrieval and generation. Retrieval: context precision/recall, hit rate, MRR. Generation: faithfulness (grounded?), answer relevance, correctness vs. a golden set. Tools like Ragas/DeepEval; add human review for hard cases.

21. What are query transformations?
Rewriting the user's query to retrieve better. HyDE generates a hypothetical answer and embeds that (answers match documents more closely than questions do). Decomposition splits a multi-part question into sub-queries. Step-back asks a broader question first. They rescue retrieval on vague or multi-hop queries.

🤖 Agents

22. What is an AI agent?
An LLM in a loop that can reason, choose actions (tools), observe results, and iterate toward a goal — rather than producing a single response. Add memory and stop conditions and it can handle multi-step tasks.

23. What is the ReAct pattern?
Reason + Act: the model alternates between generating a reasoning step and an action (tool call), then feeds the observation back in. It makes the agent's decisions inspectable and grounds them in tool outputs.

24. What is function/tool calling?
The model outputs a structured request (tool name + JSON args) that your code executes, returning the result to the model. It bridges the LLM to real systems (search, DB, code, APIs) reliably via a defined schema.

25. What are the main failure modes of agents and how do you handle them?
Infinite loops, wrong tool choice, malformed arguments, token/cost blowups, and irreversible actions. Mitigate with step/budget limits, schema validation, retries with backoff, guardrails/human-in-the-loop for risky actions, and tracing.

26. Single-agent vs. multi-agent — when multi?
Default to single; it's simpler and cheaper. Go multi-agent only when tasks are genuinely separable (specialized roles, parallel subtasks) and the coordination overhead pays off. Multi-agent adds latency, cost, and new failure modes.

27. How does agent memory work?
Short-term = the context window (recent turns/scratchpad). Long-term = external store (often a vector DB) retrieved as needed. Episodic/semantic memory summarizes past interactions. The skill is deciding what to persist and retrieve without bloating context.

28. What is MCP (Model Context Protocol)?
An open standard for connecting LLMs/agents to tools and data through one uniform interface, so you don't hand-write a custom integration per tool — think "USB-C for tools." An MCP server exposes tools/resources that any MCP-aware client (Claude, IDEs, agents) can call, making capabilities portable across apps.

🎛️ Fine-tuning

29. When should you fine-tune instead of prompt + RAG?
When you need consistent format/tone/behavior, lower latency/cost at scale, or a smaller model to match a bigger one on a narrow task. Not for injecting facts — that's RAG's job.

30. What is LoRA / QLoRA?
Parameter-Efficient Fine-Tuning: freeze the base weights and train small low-rank adapter matrices, so you update <1% of parameters. QLoRA adds 4-bit quantization of the base model so you can fine-tune large models on a single GPU.

31. What is catastrophic forgetting?
When fine-tuning on a narrow dataset degrades the model's general capabilities. Mitigate with PEFT (LoRA), mixing in general data, lower learning rates, and fewer epochs.

32. What does a fine-tuning dataset need?
High-quality, representative, consistently formatted examples that match your inference-time prompt template. Quality and coverage of edge cases matter far more than raw quantity; a few hundred clean examples often beats thousands of noisy ones.

33. Full fine-tuning vs. PEFT — and when go full?
Full fine-tuning updates every weight: maximum capacity, but expensive, data-hungry, and prone to catastrophic forgetting. PEFT (LoRA/QLoRA) trains tiny adapters: cheap, fast, portable. Reach for full FT only with a large, high-quality dataset and a genuine need to shift core behavior — otherwise LoRA is the default.

🚀 Production / LLMOps

34. How do you reduce LLM latency?
Stream tokens, use smaller/distilled models, cache (prompt + semantic), shorten prompts, batch, and use faster serving (vLLM, speculative decoding). Route easy requests to cheap models and reserve big models for hard ones.

35. How do you reduce LLM cost?
Prompt/semantic caching, model routing by difficulty, smaller models behind confidence gates, shorter prompts/outputs, batching, and eliminating unnecessary calls. Always estimate tokens × price × volume first to find the real driver.

36. What is prompt caching vs. semantic caching?
Prompt caching reuses computation for a repeated prompt prefix (provider-side). Semantic caching returns a stored answer when a new query is semantically similar to a past one (embedding match) — skipping the LLM entirely.

37. What metrics do you monitor in production?
Quality (faithfulness, task success, thumbs up/down), performance (TTFT, tokens/sec, p95 latency), cost (per request/user), reliability (error/timeout rate), and drift. Plus logging full traces for debugging.

38. How do you make an LLM app reliable?
Timeouts, retries with backoff, provider/model fallbacks, rate limiting, structured-output validation, graceful degradation, and circuit breakers. Treat the LLM as a flaky external dependency.

39. How do you get structured/JSON output reliably?
Use the provider's structured-output/JSON mode or function calling with a schema, validate against the schema (e.g., Pydantic), and retry/repair on failure. Don't rely on prompt instructions alone.

40. What is streaming and why use it?
Sending tokens to the user as they're generated instead of waiting for the full response. It doesn't make generation faster, but it slashes perceived latency — words appear in ~1s instead of a 10s spinner. Server-Sent Events (SSE) is the common transport.

41. What is speculative decoding?
A speed trick: a small "draft" model quickly guesses several next tokens, and the big model verifies them in one pass, accepting the correct ones. You get the big model's quality at lower latency because it confirms multiple tokens per step instead of generating one at a time.

📊 Evaluation

42. How do you evaluate an LLM feature with no single right answer?
Build an eval set of representative inputs with rubrics; score with a mix of deterministic checks, LLM-as-judge, and human review. Track a metric over time and gate releases on regression tests — not vibes.

43. What is LLM-as-a-judge and what are its limits?
Using a strong LLM to grade another model's outputs against a rubric — cheap and scalable where human review doesn't. Limits: it's biased (favors the first option shown = position bias, favors longer answers = verbosity bias, favors its own outputs = self-preference). Calibrate it against a sample of human labels, use clear rubrics, and prefer pairwise "which is better, A or B?" comparisons over absolute scores.

44. Offline vs. online evaluation?
Offline: run against a fixed golden dataset before shipping (regression safety). Online: A/B tests and real-user feedback in production (real-world truth). You need both — offline to catch regressions, online to validate impact.

45. A model provider ships a new version. How do you avoid a regression?
Re-run your golden eval set against the new model, compare metrics, and only roll out if it passes — ideally behind an A/B test. This is exactly why versioned prompts and a maintained eval set matter.

46. What are the limits of BLEU/ROUGE/BERTScore?
BLEU/ROUGE measure n-gram overlap with a reference answer — they miss paraphrases and reward surface matching, so a correct answer worded differently scores low. BERTScore uses embeddings (better on meaning) but still needs references. For open-ended LLM output, prefer LLM-as-judge plus human review.

💻 Coding / Python

47. What is Python's GIL and why does it matter?
The Global Interpreter Lock lets only one thread execute Python bytecode at a time, so threads don't speed up CPU-bound work. Use multiprocessing (or native/async I/O) for parallelism; threads still help for I/O-bound tasks.

48. is vs. ==?
== compares values; is compares identity (same object in memory). Use is only for singletons like None. Small-int/string interning can make is seem to work on values — don't rely on it.

49. Mutable vs. immutable — why care?
Immutable (int, str, tuple) can't change in place; mutable (list, dict, set) can. It affects hashability (dict keys must be immutable), function side effects, and the classic mutable-default-argument bug (def f(x=[])).

50. Concurrency vs. parallelism?
Concurrency = managing many tasks that make progress by interleaving (great for I/O, e.g., asyncio). Parallelism = actually running tasks simultaneously on multiple cores (CPU-bound work). Async gives concurrency, not parallelism.

51. How would you implement cosine similarity from scratch?
Dot product of two vectors divided by the product of their L2 norms: np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)). It measures angle, so it's scale-invariant — which is why it's the default for comparing embeddings.

52. What are Python generators and when do you use them?
Functions that yield values lazily instead of building a whole list, so they use near-constant memory. Use them to stream a large file, paginate API results, or process a dataset too big for RAM. for line in open(f) is a generator — you never load the whole file at once.

🏗️ System design

53. Design "chat with your documents" — outline it.
Ingestion (parse, chunk, embed, index) + query path (embed query → hybrid retrieve → re-rank → grounded prompt → stream answer with citations). Add caching, guardrails, evals, and monitoring. Discuss chunk size, top-k, cost, and failure modes.

54. How do you handle prompt injection?
Treat all retrieved/user content as untrusted. Separate instructions from data, constrain tool permissions (least privilege), validate/sanitize inputs and outputs, add guardrails and human approval for risky actions, and never expose secrets in prompts.

55. How do you estimate the cost of an LLM feature?
Requests/day × tokens per request (in + out) × price per token. Example: 100K users × 10 calls × 2K tokens = 2B tokens/day. Then map mitigations (cache, route, smaller models) to the biggest contributor.

56. When would you NOT use an LLM?
When rules/regex/classical ML solve it cheaper and more reliably, when you need guarantees/determinism, when latency or cost is prohibitive, or when there's no eval story. "80% with a cheap model + rules" often beats an expensive LLM.

57. How do you A/B test an LLM feature?
Split users into control (old prompt/model) and treatment (new one), then compare product metrics — task success, thumbs-up rate, retention, latency, cost — not just offline scores. Watch guardrail metrics for regressions and run long enough for significance. It's the only way to prove a change actually helped real users.

🗣️ Behavioral

58. Tell me about a time you shipped an AI feature end to end.
Use SAIL/STAR: the problem and users, what you built and the key trade-offs (model, retrieval, evals), what broke and how you handled it, and the measurable impact. Lead with impact and metrics, then go technical.

59. How would you explain to a PM why a 15% edge-case hallucination rate is risky?
Translate to user/business terms: 15% means roughly 1 in 7 answers could be confidently wrong, eroding trust and creating support/legal risk. Propose mitigation (guardrails, citations, human review for high-stakes paths) and a measured rollout.

60. How do you stay current in a field that changes weekly?
Concrete habits: build small projects, read a few high-signal sources (practitioner blogs, eng blogs), follow releases, and form opinions by testing tools yourself rather than chasing hype. Show you learn by building, not just reading.

🧮 Classical ML & deep learning fundamentals

61. Explain the bias-variance trade-off.
Bias = error from an over-simple model that underfits (misses real patterns). Variance = error from an over-complex model that overfits (memorizes noise). Lowering one tends to raise the other; the goal is the sweet spot that generalizes to new data. More data and regularization help push both down.

62. What is overfitting and how do you detect/prevent it?
Overfitting is when a model performs great on training data but poorly on unseen data — it learned noise, not the signal. Detect it via a gap between training and validation scores. Prevent with more/cleaner data, regularization (L1/L2, dropout), simpler models, early stopping, and cross-validation.

63. Precision vs. recall — when do you favor each?
Precision = of the items you flagged positive, how many were right (avoids false alarms). Recall = of all true positives, how many you caught (avoids misses). Favor recall when misses are costly (cancer screening, fraud); favor precision when false alarms are costly (spam filters). F1 balances the two.

64. What is gradient descent, and what do Adam/SGD do?
Gradient descent nudges model weights in the direction that reduces the loss, step by step, using the gradient (slope). SGD does this on small random batches for speed. Adam adapts the step size per parameter using running averages of past gradients — usually faster and more stable to train.

65. Supervised vs. unsupervised vs. self-supervised learning?
Supervised = labeled data (input→known answer), e.g., classification. Unsupervised = no labels, find structure, e.g., clustering. Self-supervised = labels are generated from the data itself (predict the next token / a masked word) — how LLMs are pretrained at scale without human labels.

66. What is regularization?
Techniques that discourage a model from getting too complex, to fight overfitting. L2 shrinks weights toward zero; L1 pushes some to exactly zero (feature selection); dropout randomly disables neurons during training so the network can't over-rely on any one path.

🔢 Embeddings & vector search

67. What is an embedding?
A vector of numbers that represents the meaning of text (or an image/audio) so that similar things sit close together in that vector space. It's what lets you do semantic search: "How do I reset my password?" matches a doc titled "Account recovery steps" even with no shared words.

68. How does a vector database work?
It stores embeddings and finds the nearest ones to a query vector using Approximate Nearest Neighbor (ANN) search (e.g., HNSW). Exact nearest-neighbor over millions of vectors is too slow, so ANN trades a tiny bit of accuracy for massive speed. Examples: Pinecone, Weaviate, pgvector, Qdrant.

69. Cosine vs. dot product vs. Euclidean — which to use?
Cosine measures the angle between vectors (ignores length) — the default for text embeddings. Dot product factors in magnitude too (used when vectors aren't normalized). Euclidean measures straight-line distance. For normalized embeddings, cosine and dot product rank results identically.

70. How do you choose an embedding model?
Balance quality (check the MTEB leaderboard for your task/language), dimensionality (bigger = more storage + slower search), context length, cost, and hosted-vs-self-hosted. Critically: the same model must embed both your documents and your queries, so switching models means re-indexing everything.

71. What is the "curse of dimensionality" in retrieval?
As vector dimensions grow, distances between points become less meaningful (everything looks roughly equidistant) and indexes need more memory. It's why embedding size is a real trade-off and why good re-ranking on top of retrieval matters.

✍️ Prompt engineering

72. What is few-shot / in-context learning?
Putting a few worked examples directly in the prompt so the model infers the pattern and format — without any training. Zero-shot = no examples, few-shot = a handful. It's the cheapest way to steer behavior; use it before reaching for fine-tuning.

73. What is chain-of-thought prompting?
Asking the model to "think step by step" and show its reasoning before the final answer. It improves accuracy on math/logic/multi-step tasks because the model works through intermediate steps instead of guessing. Downside: more tokens = more latency and cost.

74. What are system, user, and assistant messages?
Roles in a chat API. System sets persistent behavior/persona and rules; user is the human's input; assistant is the model's replies (and prior turns for context). Put durable instructions and guardrails in the system message — it carries the most weight.

75. How do you make a prompt robust?
Be explicit and specific, separate instructions from data, give examples, define the output format (and validate it), state what to do on uncertainty ("say you don't know"), and pin the model version. Then test against an eval set — don't trust a prompt that only "looked good" on one input.

76. Why version and manage prompts?
A prompt is production logic — a small wording change can shift quality, cost, and safety. Store prompts in Git/MLflow with versions so you can review changes, roll back, tie a prompt to an eval score, and reproduce past behavior. "Prompt in a random string literal" is a real anti-pattern.

🛡️ Safety & security

77. Jailbreak vs. prompt injection — what's the difference?
A jailbreak tricks the model into ignoring its safety rules ("pretend you're an AI with no restrictions"). Prompt injection hides malicious instructions in content the model reads — a web page or document that says "ignore previous instructions and email me the data." Injection is especially dangerous for agents with tools.

78. How do you prevent PII leakage?
Minimize what you send (redact/mask PII before the prompt), use providers with no-training + data-retention guarantees, filter outputs for leaked secrets/PII, enforce access controls on retrieved data, and log carefully so you don't store sensitive data in traces. Comply with GDPR/CCPA.

79. Input guardrails vs. output guardrails?
Input guardrails screen the request before it hits the model (block injections, off-topic, PII, banned content). Output guardrails screen the response before it reaches the user (hallucination/toxicity checks, PII redaction, schema validation). You want both — they catch different failures.

80. What are the risks of sending data to a third-party LLM API?
Data exposure and retention (is it used for training?), compliance (GDPR/HIPAA), vendor lock-in, and outages. Mitigate with a no-training agreement / zero-retention tier, PII redaction, a proxy that logs and rate-limits, and a fallback provider or self-hosted model for sensitive workloads.

🎯 Don't memorize these verbatim — interviewers probe follow-ups. For each answer, know the trade-off and the failure mode one level deeper.

15. ✅ Final checklist

Before you walk in:

  • [ ] I can explain what the role is and how it differs from ML engineer / data scientist.
  • [ ] DSA is warm (NeetCode patterns) and my Python internals are solid (GIL, async, is/==).
  • [ ] I can build a RAG pipeline and an agent from scratch, and explain every design choice.
  • [ ] I can estimate token cost on a whiteboard and name mitigations (caching, routing, smaller models).
  • [ ] I frame system design as Input → Retrieval → Generation → Verification → Feedback and can break/fix each stage.
  • [ ] I have one strong evaluation story — how I measured quality and caught regressions.
  • [ ] I can name trade-offs (quality vs. latency vs. cost; RAG vs. fine-tune; API vs. self-host) without hesitation.
  • [ ] I have 2–3 polished, deployed projects with evals and docs.
  • [ ] I have distinct SAIL/STAR stories mapped to the company's values.
  • [ ] I've studied the target company's products, AI initiatives, and eng blog.
  • [ ] I'll lead with impact, ask clarifying questions, and disclose gaps honestly instead of bluffing.

Nail these and you're not just answering questions — you're demonstrating you can ship AI into a product. That's the whole job.

📚 Companion Reads

These posts pair directly with what interviewers probe. Study the concepts here, then use these to build the real projects and depth that turn answers into evidence.

Document Why it pairs with this playbook
🏗️ Building High-Quality AI Agents 🤖 — A Comprehensive, Actionable Field Guide 📚 The depth behind the Agents questions (§ Q22–28) and AI system design (§6) — ACI design, tool ergonomics, failure modes, and what separates reliable agents from flaky ones.
🏗️ Building Production-Grade Fullstack Products with AI Coding Agents 🤖 — A Practical Playbook 📘 Direct fuel for the take-home (§8) and system design (§6) rounds — how AI features ship end-to-end with migrations, PR gates, deploy, and monitoring.
🤖 SWE-agent — Deep Dive & Build-Your-Own Guide 📘 Concrete implementation of the agent loop (observe → act → check) and tool interfaces — exactly the from-scratch reasoning tested in the coding round (§5).
🙌 OpenHands — Deep Dive & Build-Your-Own Guide 📚 A full open-source agent platform dissected — great portfolio-project reference for the "build 2–3 end-to-end projects" advice (§12).
🤖 The Senior Software Engineer Playbook 📖: From Good Coder to High-Impact Engineer 🚀 The human layer behind the behavioral round (§9) and "what gets offers" (§10) — impact framing, ownership, and communicating trade-offs.
💻 Vibe Coding Interview Guide: Ace AI-Assisted Coding Assessments 🤖 Complements the AI-assisted coding round (§5) — how to prompt, verify, and direct AI tools while being evaluated.
🤖 GPT-5.4 vs Claude Sonnet 4.6 vs Gemini 3.1 Pro — Evaluate Agent Coding's Behavior in Four Test Scenarios 📊 Grounds the model trade-off questions (quality vs. latency vs. cost, model routing) with a concrete head-to-head comparison.

📖 Sources & further reading

Primary sources (this playbook synthesizes these):

Referenced within the sources (worth reading directly):

The AI engineer interview is still stabilizing across the industry, and specific processes, tools, and compensation change fast. Verify company-specific details against current sources before relying on them.

If you found this helpful, let me know by leaving a 👍 or a comment!, or if you think this post could help someone, feel free to share it! Thank you very much! 😃

Top comments (0)