We stopped sending take-home coding tests. Coding tools turned them into a measure of who has the better subscription. In their place we run a sixty-minute live debugging session in a small, realistic codebase, with any AI tool the candidate wants, and we watch how they read errors and how they decide what to trust. It predicts how someone performs in month three far better than a polished take-home ever did.
Why the take-home stopped working
The submissions got perfect and the conversations got worse. Clean code, full test coverage, a tidy README, sensible commit messages, and then a follow-up call where the candidate couldn't explain why a function existed. Meanwhile strong candidates were declining to spend six unpaid hours on a test, so we were filtering for free time rather than skill.
There's a bigger reason too. The way engineers work has shifted; coding agents are the fastest-growing thing on GitHub, and the day job now involves supervising generated code as much as writing it. A take-home tests the writing. We needed something that tests the supervising.
| What we wanted to know | Take-home test | Live debugging session |
|---|---|---|
| Can they produce clean code | Yes, or their tool can | Not measured |
| Can they explain their own work | Often not, on the follow-up call | Watched live, out loud |
| Do they verify a wrong suggestion | Invisible to us | The timezone bug tests it directly |
| Cost to the candidate | Around six unpaid hours | Sixty minutes, scheduled |
What the session looks like
A small service, roughly a thousand lines: an HTTP API, a database, a background job on a queue, and a tiny admin page. It runs locally in a container in under a minute, and we seed four bugs of different kinds into it before the call starts.
- A timezone bug that only shows up for users west of UTC.
- A race between the job and the API on the same row.
- An off-by-one in cursor pagination that drops the last record of every page.
- An error message that lies (it says "not found" when the real problem is a permission check).
The candidate gets the repo, a terminal, a browser, and a running database, and can use whatever assistant they normally use. We ask them to think out loud, and we say up front that fixing all four is not the goal. Most people fix two. Some fix one, and it's still a strong session.
What the rubric scores
The rubric has five rows, each scored one to three by two interviewers independently, and none of them counts bugs fixed. It scores how the candidate reads an error, whether they reproduce before fixing, whether they verify what the assistant suggests, how they pick between a quick patch and the right fix, and when they stop to ask.
- Reads the error and the surrounding code before prompting or searching.
- Reproduces the bug before trying to fix it.
- Verifies what the assistant suggests instead of pasting it. We specifically watch what happens when the tool proposes a plausible wrong fix for the timezone bug, which it usually does.
- Explains the trade-off of the fix and picks deliberately.
- Knows when to stop and ask a question instead of thrashing.
Two interviewers scoring independently, then comparing, cut most of the "I just liked them" bias we used to have.
What changed in who we hired
We hired fewer framework-trivia specialists and more people who can operate a system they didn't write. Two surprises came out of it: some senior candidates struggled because they hadn't personally debugged anything in years, and some juniors who learned to code by prompting froze when the assistant's fix didn't work.
Both were useful things to know before an offer, and neither would have surfaced in a take-home. Hiring steadily for a team of 80-plus engineers at Shanti Infosoft means we run this session a lot, and it keeps earning its slot. It costs us an hour of two engineers' time per candidate. It saves us the far larger cost of finding out in month two. We now run the same format when staffing an IT consulting engagement where the client interviews our people too, and it survives that scrutiny better than a portfolio does.
When did you last watch a candidate debug something live, with the tools they'd actually use on the job?
Sagar Jain is the technical co-founder of Shanti Infosoft, a CMMI Level 5 firm, and sits in on more of these sessions than any other kind of interview.
Top comments (0)