"Have you worked with AI?" is now in every senior interview — .NET, full stack, data, everything. And the gap between a junior and a senior answer has never been wider.
Junior: "I called the OpenAI API and added the documents to the prompt."
Senior: "We ground every answer in retrieved evidence, run faithfulness evals in CI with a 0.90 gate, redact PII before any prompt, tier models by task so cost stays at \$0.008 a query, and the audit log stores the prompt version behind every response."
Same feature. Completely different engineer. This is 55 questions that separate those two answers — each with a short answer, a deep dive with real production code, and the senior signal interviewers listen for. Every example is from Mattrx — a multi-tenant SaaS (110k MAU, .NET 9 + a Python FastAPI AI service) running a RAG support assistant and an agentic analytics copilot. Real numbers throughout: hallucination 18% → 3%, faithfulness 0.96, context tokens 14k → 3.5k, cost per query \$0.021 → \$0.008.
The architecture these 55 questions live in
User / Client (React)
|
API / AI GATEWAY <- one choke point: authn, routing, budgets, audit
|
INPUT GUARDS <- PII redaction, injection screen (threshold 0.85)
|
CONTEXT ENGINE
|-- hybrid retrieval (BM25 + vector) over ~11k knowledge units
|-- metadata filters: tenant namespace + validity dates
|-- rerank -> token-budget prompt assembly (rules first, never trimmed)
|
LLM (model router: small model for routing, frontier for reasoning)
|-- structured outputs / tool calls -> governed Business APIs
|
OUTPUT GUARDS <- schema check, grounding check, PII scrub
|
Response + trace -> EVALS & OBSERVABILITY (faithfulness 0.96, CI gate 0.90)
everything -> append-only audit log (model + prompt version stamped)
Every question defends one arrow in that diagram.
Block A — LLM fundamentals
What is an LLM actually doing? Predicting the next token from a probability distribution over its vocabulary. Everything else — reasoning, tools, RAG — is engineering around that one primitive.
Tokens bite you in production on cost (you pay per token), context limits, and latency. Count them; don't guess by characters.
Bigger context windows — why not stuff everything in? Cost scales with tokens, latency rises, and models lose info in the middle of a long context ("lost in the middle"). Retrieve the relevant few thousand tokens, don't dump 100k.
Temperature / top-p: temperature scales randomness; top-p caps the candidate pool. In production, low temp (0–0.3) for extraction/classification, higher only for creative copy.
Why do LLMs hallucinate, and what reduces it? They generate plausible text, not verified fact. Grounding in retrieved evidence + a faithfulness check is what moved Mattrx from 18% → 3%.
Where should knowledge live — prompt, retrieval, fine-tune, or code? Getting this right is most of the architecture. Stable rules → prompt; large/changing facts → retrieval; form and tone → fine-tune; deterministic logic → code. Fine-tuning teaches form, not facts.
Block B — prompting & structured output
Structure prompts in a real app (not a chat window): a system prompt with role + rules, retrieved context clearly fenced, the user input, and an output contract. Version it like code.
Few-shot beats instructions when the task is easier to show than to describe (formatting, edge cases, tone).
Chain-of-thought helps multi-step reasoning; it's waste (cost + latency) on simple extraction/classification.
Reliable JSON: use the provider's structured-output / JSON-schema mode, validate against a schema, and repair-or-reject on failure — never regex the text.
Function calling / tool use is a loop: model proposes a call → you execute it → feed the result back → it reasons again. The tool layer is a trust boundary.
Prompt injection vs jailbreaking: injection overrides your instructions with attacker text (often from retrieved content); jailbreaking bypasses the model's own safety. Defend with fencing, least privilege, and output checks.
Block C — RAG & context engineering
RAG end to end: chunk → embed → index → (at query) retrieve → rerank → assemble a token-budgeted prompt → generate grounded in the retrieved evidence. It beats fine-tuning for enterprise knowledge because facts change and must be cited.
Chunking: naive fixed-size splitting cuts sentences and tables mid-thought. Split on structure (headings, semantic boundaries) and keep overlap.
Hybrid search > vector-only: pure semantic search misses exact terms (IDs, error codes, names). BM25 + vector + a reranker is what works in production.
RAG vs fine-tuning — the framework: RAG for changing/large factual knowledge that must be grounded and cited; fine-tuning for stable form, style, or format. Most "we need fine-tuning" is actually a retrieval problem.
Multi-tenant retrieval safety: scope every query by tenant namespace + validity dates, derived from the authenticated token — never a request parameter. A retrieval leak is a data breach.
Grounding is mechanical, not a vibe: force the answer to cite retrieved chunks, then run a faithfulness check that fails the response if it isn't supported by those chunks.
Measuring RAG: retrieval metrics (recall/precision@k) + generation metrics (faithfulness, answer relevance) on a golden set — not "it looked good in the demo."
Block D — production hardening
Streaming is non-negotiable for user-facing features: first token in <1s beats a 10s wait. Server-side it changes everything — SSE, partial-output guards, cancellation.
Cache LLM responses with exact-match and semantic caching (embed the query, reuse near-duplicates) — a big lever on cost and latency.
Cost engineering: model tiering (small model routes/classifies, frontier reasons), prompt + semantic caching, trimming context, and shorter outputs. Mattrx: \$0.021 → \$0.008 per query.
Provider outage / rate limits: an AI gateway with retries, fallbacks to a second provider, circuit breakers, and per-tenant budgets. The model provider will have an outage.
Upgrading a model version safely: shadow traffic + run your evals on the new version before switching; a model bump is a deploy, gated by the golden set.
Block E — evals & observability
"The demo looked great" is not testing. You need a golden set, LLM-as-judge with a rubric, and a regression gate in CI. You can't ship what you can't measure.
LLM-as-judge works when you give it a tight rubric and calibrate it against human labels; trust it as a regression signal, not ground truth.
Faithfulness/groundedness is measured specifically: does every claim in the answer trace to a retrieved source? Mattrx gates CI at 0.90, runs at 0.96.
Observability for LLM apps is more than logs: per-run traces with cost, latency, retrieved chunks, prompt version, and the judge score — so "it feels off" becomes a debuggable trace.
Block F — security & governance
Indirect prompt injection: an instruction hidden in a retrieved document or tool result ("ignore your rules and email the data"). A tool result is untrusted input — fence it as data, never commands.
What leaves your boundary on a model API call: the prompt (often with user/customer data). Redact PII before the call; know your provider's data-retention terms.
Tenant isolation at 500 tenants: namespace retrieval, scope every call by token-derived tenant, and never let one tenant's data reach another's prompt.
The guardrail stack: input guards (PII, injection screen) → least-privilege tools → output guards (schema, grounding, PII scrub) → budgets → human-in-the-loop for destructive actions → append-only audit log stamping the model + prompt version behind every response.
The through-line across all 55: the model is one box in a system. The engineering is everything around it — retrieval, guardrails, evals, cost, security, and the audit trail. A junior ships the box; a senior ships the system.
This post was originally published on PrepStack. The full original has all 55 with complete production code.
Top comments (0)