tl;dr Programming interview questions can feel unnecessarily difficult. Sometimes they actually are. And this isn't just because they make interviews excessively stressful. Our data shows that harder programming questions actually do a worse job of predicting final outcomes than easier ones.
Programming under time pressure is difficult. This is especially true during interviews. A coding exercise that would seem simple under normal circumstances somehow becomes a formidable challenge under the bright lights of an interview room. Stress hormones cloud your thinking during interviews (even though, sadly, neither fight nor flight is an effective response to a menacing programming problem). And it can almost feel like the questions are designed to be perversely difficult. I actually think this is more than just a feeling.
Interview questions are designed to be hard. Because the cost of hiring a bad engineer is so much higher than the cost of rejecting a good engineer, companies are incentivized to set a high bar. And for most companies that means asking hard questions. Intuitively this makes sense because harder questions seem like they should result in a more rigorous screening process. But intuition turns out to be a poor guide here. Our data shows that harder questions are actually less predictive than relatively easy ones.
Interview Questions Are Too Hard
Hard questions do filter out bad engineers, but they also filter out good engineers (that is, they have a high false-negative rate). Easy questions, in contrast, produce fewer false-negatives but more false-positives (since more engineers get them right, including some bad ones). Balancing these two signals is the core problem you face when selecting the optimal difficulty level for interview questions. Companies—seeking to avoid false positives at all cost—tend to skew towards the harder end of this spectrum.
However, whether or not a candidate answers a question correctly is not the only source of signal during an interview. You can also evaluate their process by, for example, observing how long it takes them to finish, how clean their code is, and how much they struggle while finding a solution. Our analysis shows that this second source of signal (process) is almost as predictive as the first (correctness).
But there is an additional trade-off here. The questions that carry the most process signal are significantly easier than the questions that carry the most correctness signal. The reason for this becomes clear when you distill process down to how much a candidate struggles while finding a solution (the aspect of process most directly related to question difficulty). If a question is hard enough to carry a strong correctness signal, then all candidates will struggle with it (even those who eventually answer it correctly). So struggle will carry no signal.
Conversely, questions that carry a strong process signal will be easy enough for most candidates to answer them correctly, thus having little to offer in terms of correctness signal. An optimally difficult question is one that balances process and correctness to extract the maximum signal from the combination of these two factors (which may not be the peak signal for either).
We interviewed thousands of engineers and scored their answers along multiple dimensions, including process and correctness, and compared these scores with later performance. And, after regression analysis (looking at both process and correctness signal) our data showed that the most predictive questions were actually much easier than we expected (and easier than the questions that many companies ask).
Harder questions ultimately filter out too many qualified candidates to be optimal. So, if you want to make your hiring process more accurate, you should probably ask easier questions.
I want to state clearly, however, that this doesn't mean you should lower the bar and pass more people. Asking easier questions doesn't necessarily mean making your interviews easier. The difficulty level of the questions you ask and where you set your decision-threshold are independent decisions. You can still implement an extremely rigorous hiring process by asking relatively easy questions and evaluating them demandingly. Our finding is simply that easier questions provide more signal. But what you do with that signal is up to you.
Easier interview questions are also less stressful, which is an important advantage. Stress causes candidates to under-perform. But, on the other hand, when candidates are more comfortable, they perform their true best, which actually makes interviews more predictive. I think that interviewers tend to under-estimate the effects of stress on candidates while over-estimating their own abilities. When you're the one asking the questions, it's easy to forget how hard it is to get much real programming done in 30-60 minutes. To counter this bias, we've adopted a rule at Triplebyte stating that interviewers have to give candidates 3X as much time to answer a question as they think it would take to solve it themselves. Usually this works out to be the right amount of time.
Interview Questions Are Too Short
Easier questions are beneficial for another important reason. They allow you to fit more content into your interviews. This means you can use longer, multi-part problems which have compounding benefits in terms of prediction. You can ask questions that ramp up in difficulty over time, and these longer, real-world questions are more predictive than their shorter and harder cousins.[1]
This is partly because longer questions serve as better proxies for real-life programming. Production programming involves working with a relatively large codebase over a long period of time, and longer questions just do a better job of approximating this reality.
Additionally, longer questions allow you to offer hints when a candidate gets stuck. I think this is vital because even strong engineers can get tripped up at some point during a coding problem. Asking longer questions gives candidates the chance to recover from errors and show off their skills later on during an exercise. A single misstep shouldn't ruin an entire interview. Finally, being able to offer help makes interviews less stressful which, again, results in more accurate outcomes.
Conclusion
Companies can get stuck in a vicious cycle when it comes to question difficulty. They start out asking questions that are already too hard and too short, which leads them to make sub-optimal hiring decisions based off noisy signals. When they experience poor outcomes, they may attempt to correct by making their process even more rigorous (which generally means asking even harder questions). But this only makes their interviews less accurate. And so on. Both companies and candidates are hurt by this cycle. I see companies missing out on talent and candidates missing out on jobs—and everyone getting more stressed than they need to be.
I hope interviewers will embrace this finding. Not only can it make their work more accurate, but easier as well. It's much less time intensive to come up with straightforward, multi-step problems than perversely difficult short challenges.
So, here's our advice, if you actually want to make your interviews more accurate, you probably need to start asking easier programming questions. This doesn't mean lowering the bar. It just means getting a better signal so you can hire the right people.
If you decide to ask easier questions during programming interviews, or find this post helpful in some other way, I'd love to hear from you! Email me at charlie@triplebyte.com
[1]“The best predictor of how someone will perform in a job is a work sample test (29 percent). This entails giving candidates a sample piece of work, similar to that which they would do in the job, and assessing their performance at it.” - (Wired Magazine) ↩
Top comments (2)
so, since this article was a little light on examples of easier questions or how steep the ramp-up is, i tried your guys' coding quiz to get a sense of what you're getting at here. and the quiz was pretty familiar to the kind of technical screenings i've experienced applying for positions at large companies with big standard banks of questions.
first couple of questions were pretty soft: which sql keyword gets data from a table or which http verb should NOT make changes to server/data. then they ramped up a little: which line of code is missing from this curried/partial-application function or follow this deeply nested functions-returning-functions chain to find the output. then they got a little esoteric: given a function, and given you call it three times, but the middle time you call it by putting it inside a setTimeout(func, 0), what order do the three calls execute... and why?
i don't know how much of your secret sauce you guys are trying to publish here. but i'd be curious to hear how this approach applies to phone or in-person interviews. it seems like it'd be effective, but would have to be modified from the quiz format pretty drastically.
hey thanks - our quiz is adaptive so it gets more difficult the better you do.
yeah - this definitely applies to our technical screen as well. we try to use longer multi-step questions (e.g. implement a tic tac toe game) - and then watch that process. we do ask knowledge-base questions, but that's because we're trying to get a really broad view of candidates so we can match them with the right companies. we definitely don't expect engineers to be great at everything we test, we just want to find out where strengths lie. most companies don't need to test as broadly as we do since they only have specific roles to fill.
not sure that's a helpful answer - feel free to email me and we can talk more too. thanks!