The standard engineering hiring screen, LeetCode-style algorithm problems, timed under pressure, on a platform with an unfamiliar IDE, measures one thing with reasonable reliability: the candidate's ability to solve algorithm problems under pressure on an unfamiliar platform. This correlates with job performance at approximately the same level as a coin flip for most engineering roles, where the actual work involves understanding existing systems, navigating ambiguous requirements, and communicating with non-technical stakeholders.
"The technical interview industry has optimized for filtering confidence on whiteboard problems rather than predicting job performance. The irony is that the engineers who are best at LeetCode interviews are often not the engineers who are best at the ambiguous, collaborative, context-dependent work that constitutes 90 percent of real engineering jobs."
— Gergely Orosz, Author of The Pragmatic Engineer newsletter and former engineering manager at Uber (2023)## What work sample tests actually measure
Work sample tests present candidates with a task that resembles the actual work they would do in the role. For a backend engineering role, this might be: here is an existing service with a bug, find and fix it. Here is a feature request, implement it. Here is a failing test, explain why it fails and fix either the test or the code.
These tasks measure code reading comprehension, familiarity with existing codebases, debugging methodology, and the ability to navigate realistic ambiguity, all of which are core to day-to-day engineering work and none of which are measured by algorithm problems. The validity concern: work samples require more preparation and take more time to evaluate. The tradeoff is worth it if you are making multiple hires per quarter.
Async take-home design
Take-home assignments address the artificial constraint of the timed synchronous interview. Candidates do their best work in their own environment, on their own schedule, without the cognitive overhead of being observed. The signal quality from a well-evaluated take-home assignment is often higher than from a 45-minute live coding session.
The design principles for a good take-home: scope it to under three hours of genuine work, make the problem representative of real work, evaluate the submission on criteria that were stated upfront, and have the same engineer evaluate all take-homes for a given role to ensure consistency. Consider paying candidates for the take-home, even a small amount, to acknowledge that their time has value.
System design interviews done right
System design interviews are often run badly: the interviewer has a specific solution in mind and evaluates the candidate on whether they converge to it. A better system design interview evaluates how the candidate handles ambiguity in requirements, how they reason about tradeoffs rather than which tradeoffs they make, how they communicate their reasoning, and whether they identify the constraints that change the design most significantly.
Reference check reframing
Reference checks are almost universally done badly. Asking "was this person good at their job?" produces a useless answer. Instead ask: "What is the most challenging problem they solved at your company?" "What kind of environment does this person thrive in, and what environment would frustrate them?" "If you were hiring for a role that required this specific skill, would you hire this person for it?" These questions produce specific, useful signal about fit and performance context.
📊By the numbers
MetricFindingSourceCorrelation of LeetCode performance with 1-year job ratingsr = 0.12 (weak)Google internal hiring study cited by Lazlo Bock, 2023Engineers who report current hiring screens are poor predictors62%Stack Overflow Developer Survey, 2024Companies using work-sample tests as primary screen31%LinkedIn Talent Trends Report, 2024
Originally published on Tech at Crescevo — subscribe free for more.
Top comments (0)