A generic coding test tells you almost nothing about an AI engineer. A strong backend developer can still get evaluation design wrong. A strong data scientist can still struggle to serve a model under load. The mistakes differ by role, so the interview has to differ too.
These are the design scenarios we use, one per role, with the evidence a candidate has to show to pass. Steal them for your own loop.
Machine learning engineer
Scenario: serve a ranking or forecasting model behind an API with a fixed latency budget, then plan how it gets retrained.
A pass shows:
- an offline metric tied to the business outcome, not just accuracy;
- a test set with no leakage from training;
- a serving plan with its latency and cost stated.
Generative AI engineer
Scenario: question answering over a large set of internal documents, with a ceiling on cost per request.
A pass shows:
- an evaluation set built before any prompt tuning;
- retrieval measured on its own, separate from generation;
- a cost per request they can estimate and defend.
AI agent developer
Scenario: an agent that opens and updates tickets through a ticketing API, where one tool times out halfway through a task.
A pass shows:
- timeouts and retries for each tool;
- actions that are safe to run twice;
- a log of every step;
- human approval where a mistake costs money.
The timeout is the whole point of this one. Most demo agents have never met a tool that fails.
Computer vision engineer
Scenario: a defect detector that has to keep up with a production line camera on an edge device.
A pass shows: a labeling guide with a check on label disagreement, metrics for each class, and latency measured on the device itself, not on a laptop GPU.
NLP engineer
Scenario: pull fields from scanned contracts with inconsistent layouts and OCR errors.
A pass shows: error analysis field by field, a simple baseline before any large model, and a plan for documents the model can't read.
MLOps engineer
Scenario: release a new model version through a canary, with an automatic rollback trigger.
A pass shows: versioned models and data, a rollout that stops itself on a bad metric, drift alerts with an owner, and a rollback they have actually run.
Data engineer
Scenario: design the tables a training pipeline reads from, with late-arriving events and an upstream schema change.
A pass shows: a data model that keeps history, quality checks that stop a bad batch before training, and backfills that can rerun without duplicates.
Two rules that make the scenarios work
- Change a constraint halfway through. Halve the budget or tighten the latency target and watch how the design holds up. Candidates who memorised an architecture stall; candidates who understand it adjust.
- Score each stage on its own. Someone who aces the coding screen and struggles in the design review shouldn't average out into a pass. Write down the evidence behind every score, so the hiring manager can read it later.
We run these as part of a four-stage evaluation at Bromena, and the client still runs its own interviews afterwards. If you use a different scenario for one of these roles, we'd like to hear what it catches that ours misses.
— Bromena team
Top comments (1)
The AI agent developer scenario is the one that matches what I see. A tool timing out halfway through is where most demo agents fall apart. I build Aladdin, a job agent. The rule I use is the same as your human approval line. A fresh worker reads the actual page and form itself. If it cannot read it, or an answer is unknown, it asks the user instead of guessing. Scoring each stage on its own is the part I would keep. Averaging a strong stage over a weak one hides the weak one.