If you are running interviews for AI roles and your process is a LeetCode round plus a chat about transformers, you are measuring almost nothing that predicts performance. Attention mechanics are memorisable and rarely relevant to application work. Graph traversal has never told anyone whether a candidate can keep an inference bill under control.
What actually predicts success is judgement under production constraints. Here are the five questions we use, what a strong answer sounds like, and the take-home format we have found genuinely discriminating.
1. Describe an AI feature you shipped that did not work
You are listening for a failure signature, not an apology.
Strong answers are specific about how the failure showed up: a class of inputs that silently degraded, a retrieval step that returned plausible but wrong documents, latency that only broke under concurrency, a tool call that fired on untrusted user input. They usually include what the fix cost and what they would do differently.
Weak answers are either "we iterated on the prompt" or a story about something that was really a product disagreement. Neither tells you the candidate has watched a probabilistic component misbehave at scale.
2. How did you know it was getting better between releases?
This is the highest-signal question in the set, and most candidates have no real answer.
What you want to hear, unprompted: a golden dataset of representative cases including deliberately hard ones, an automated scoring method appropriate to the task, a regression run wired into CI, and traces in production that link a user complaint back to exact inputs and model version.
Follow up on the details, because that is where experience shows. How many cases? Who labelled them? What did you do when two labellers disagreed? What happened when the automated score and human judgement diverged? Candidates who have really built this have opinions and war stories. Candidates who have not will describe an observability vendor.
3. Walk a request through your system, including the timeout
A straightforward architecture question with a specific trap in it. Most people describe the happy path fluently and stop.
Push on failure. What happens when the model call times out — retry, fallback model, degraded response, queue? Is the retry idempotent? If the request had already triggered a tool call with a side effect, what undoes it? What does the user see meanwhile, and how does streaming interact with a mid-stream failure?
This is ordinary distributed systems thinking applied to a component that fails in unusual ways, which is why strong backend engineers often outperform AI specialists here.
4. What did it cost per thousand requests, and what did you do about it?
The economics question, and close to binary in its discriminating power.
In conventional software the marginal cost of a request rounds to zero, so engineers are trained not to think about it. In AI products it is real money, varies by an order of magnitude with design, and scales with success. Routing everything to a frontier model is the default that quietly destroys margins.
Good answers reach for cheap intent classification, caching, small models for the easy majority, escalation only where it matters, and trimming what goes into the context window. They usually come with a number and a story about the optimisation that was rejected as premature and later turned out not to be.
5. Where did you decide not to use a model?
The maturity question, and my personal favourite.
Experienced engineers have a ready list: a rules engine that was cheaper and testable, a database query that was exact, a well-designed form that removed the ambiguity entirely, a search index that beat a retrieval pipeline on both latency and accuracy. They explain it without defensiveness.
Candidates who answer every problem with an LLM, including ones a SQL query would solve, will build you an expensive and fragile architecture. This question surfaces that in about ninety seconds.
The take-home that actually works
Do not ask for a chatbot. Everyone can build a chatbot and you will learn nothing.
Give them thirty real examples of a classification or extraction task from your domain, including several genuinely ambiguous ones and — this is the important part — a few that are mislabelled. Ask for a working solution, an evaluation harness, and a short written analysis of where it fails. Cap it at four hours and say so explicitly.
The accuracy number barely matters. What you are grading is the analysis. Did they notice the bad labels and say so? Did they characterise the failure modes rather than just reporting a score? Did they explain what they would need to push it further?
Assume AI coding assistants will be used, because that is how the job is done now. Screening for their absence selects for the wrong thing. What you are testing is whether the candidate can critically review generated output — whether the harness measures the right thing, whether edge cases are covered, whether they caught the plausible-looking helper that quietly drops nulls.
Red flags worth encoding into your process
- A portfolio of demos with nothing operated behind any of them
- No answer at all to how quality was measured
- Framework fluency substituting for systems reasoning
- Reflexive model-first thinking on problems that do not need a model
- Vagueness about cost, which usually means nobody ever asked
- No instinct around prompt injection, cross-tenant leakage, or tool calls triggered by untrusted input
The full guide — including rate benchmarks by market, the four AI developer archetypes, and a framework for choosing between in-house hiring and a build partner — is here: Hire AI Developer in 2026. If you would rather not run the search at all, this is roughly how we structure LLM integration work for teams in that position.
Frequently Asked Questions
Should I still run an algorithms round for AI roles?
A short one is fine as a general coding filter, but do not weight it heavily. It measures nothing specific to AI engineering. The take-home plus the five questions above will give you far more signal per hour of interviewer time.
How do I test for agentic systems experience?
Ask what happens on step seven of a multi-step workflow when a tool returns something unexpected. Look for timeouts, idempotency, compensating actions and human handoff. It is distributed systems failure design, so engineers from that background pick it up fast.
Is prompt engineering still a real skill to hire for?
Not as a standalone one. Modern models are robust to phrasing and structured output modes removed most of the parsing tricks. It has collapsed into ordinary craft, the way writing good SQL did.
What if a candidate has no production AI experience at all?
Frequently fine, sometimes preferable. A strong senior engineer with cost discipline and a measuring habit ramps up quickly, and their fundamentals do not decay the way model-specific knowledge does. Screen the fundamentals hard and be explicit that ramp-up time is budgeted.
How long should the take-home be?
Four hours, capped and stated. Longer exercises select for people with free time rather than people with skill, and they materially reduce your acceptance rate among strong senior candidates who are already employed.
What is the most common hiring mistake you see?
Writing a job description that lists all four AI archetypes at once — product engineer, ML engineer, platform engineer, agentic systems engineer. It attracts generalists who are shallow in each. Pick one, and hire a team or a partner if you genuinely need coverage across several.
Top comments (0)