There seems to be a growing consensus that the interview process often adopted at tech companies for hiring technical staff, in particular developers, is broken. Almost a year ago, I featured it as a major problem in a post discussing the issues facing tech today and, suffice it to say, it hasn't been fixed in the time since.
One need not even look far to find it being discussed on Twitter or Hacker News of across a variety of blogs. The issues seem to boil down to three things:
- Coding tests are arbitrary, needlessly difficult and disconnected from the skills actually required for the job.
- The number of rounds and the time demands of interviewing are difficult to manage.
- Hiring decisions often seem arbitrary and communication about why someone failed a stage are often poorly communicated, when they are communicated at all.
Let's take a closer look at each of these.
Coding Tests and Whiteboards
Let's start with coding tests and whiteboarding since it is probably the most common frustration that developers cite.
The primary frustration developers express around coding tests have to do with them being needlessly difficult and frequently arbitrary. Many people describe being asked about complex computer science concepts and algorithms that they may have studied in college (assuming they were a CS major) but haven't had the need to reference since (or if they did, they just look it up).
I shouldn’t have to rely on an algorithm question lottery to demonstrate my skills and abilities during an interview. Luck and chance should not be part of the tech interview process... It’s like testing your dentist’s skills based on their ability to balance a redox equation using the oxidation number method from their undergraduate chemistry studies.
- Sahat Yalkabov, F*** You, I Quit — Hiring Is Broken
Part of the problem with the code test or whiteboarding is that it doesn't seem to be relevant to the job it is designed to screen for and often seems included as a way of gatekeeping.
And just how have these folks determined this definitive fact that I am not technically strong enough? Care to guess? If you said, "a live coding test with a random brain-teaser type problem you'd find on codewars.com," you'd be right. Now don't get me wrong, I have also passed interviews with these sorts of tests. Hmm.. Isn't that odd?
- Ambrose Little Technical Interviewing is Broken, But We Can Fix It
To me, there are two major issues with coding tests. The first is the disconnect between these coding tests and the supposed goal for which they were designed, which is to determine the skill and knowledge of the interviewee. It seems to me that a body of work - whether it be their education, work at a company, contributions to open source or side projects - is more illustrative of someone's technical skills than a random code challenge brought completely out of context. There is nothing especially unique about the job of being a developer that requires answering random technical trivia, but it seems to be a practice that is unique to this role.
More importantly, the second problem is that the practice prioritizes random technical knowledge (something that can relatively easily be learned) over other aspects like personality fit, ability to work with others, work ethic, eagerness to learn (things that cannot be easily learned). You can be a brilliant coder and a crappy person but I can tell you which part of that equation will have more impact your success in a developer job.
Extensive Time Demands
Tech interviews are often marathons. Many roles require extensive rounds of interviews in addition to time-consuming "homework" assignments.
This doesn't seem to be exclusive to the large, household-name tech companies. Across the board you hear stories of tech companies treating the process as if it is designed to weed out candidates who simply cannot make it to the end of a grueling race.
More and more employers are adopting pointless, expensive and time-consuming extra processes to make their recruiting pipelines even slower and more off-putting to candidates.
This is compounded by a slow or completely broken communication process.
I've gone through half-day to full-day interviews called "running the gauntlet" only to wait a month for them to say no or nothing at all. It's a part of the process and we all get over it. (shrug) Still, it's frustrating when you have to use precious vacation time to go on interviews that lead nowhere.
- Milecia McG, The Crazy Job Search Process
Often the communication breakdown seems to break down for a candidate who isn't continuing in the process, which is certainly unfair to the candidate left hanging. However, not infrequently, the communication is poor or nonexistent even for candidates who are moving to the next stage but are often left to wait weeks before that is communicated.
This seems entirely self-defeating for companies who are hiring in at least two ways. The first is that it pointlessly limits your potential candidates to those whose finances and/or job allow them to survive the countless missed hours and days required. Second, it misses out on candidates who drop out of the process because they either get frustrated with the lack of communication and length of the process and the other are, of course, people who simply find a different job during a process that need not be this protracted.
Arbitrary Hiring Decisions
The biggest issue seems to flow from the prior two, which is that rejections can seem completely arbitrary - so much so that there is a site dedicated to sharing rejection stories filled with recognizable names in the developer community.
Even many people involved in the hiring process seem to recognize that it is flawed.
Part of the problem stems from the screening process (something the stories on rejected.us frequently validate), which, according to the study cited below looking at Y Combinator hiring, accounted for nearly half of applicant rejections.
Most companies reject a high percentage of applicants during a recruiter call (or resume screen). Across the 25 companies we interviewed, an average of 47% of applicants were rejected in this way (the rate at individual companies went as high as 80%, and as low as 0%). The recruiters doing this rejecting are non technical. All they can do is reject candidates who don’t match the profile they’ve been taught to look for.
- Ammon Bartram, Who Y Combinator Companies Want
As the study notes, in many cases, this is because the screener generally isn't technical and therefore must rely on profile information they have been supplied. That's not to say that the screener should be technical, I believe there is a ton of merit in including people trained in and focused on recruiting, but it means we need to do a better job of making sure they understand the profile and are supplied with the information they need to properly evaluate folks.
The other problem is back to the coding tests. Their seeming trivia-like nature seems to lead to unpredictable results. Basically, good candidates fail a good percentage of the time, as noted in the following research.
Roughly 25% of interviewees are consistent in their performance, and the rest are all over the place...
To me, looking at this data and then pretending that I had to make a hiring decision based on one interview outcome felt a lot like peering into some beautiful, lavishly appointed parlor through a keyhole. Sometimes you see a piece of art on the wall, sometimes you see the liquor selection, and sometimes you just see the back of the couch.
The combination of screening, testing and time can mean companies hiring processes are not necessarily optimized for finding the best candidates.
This Isn't an Easy Problem to Solve
Look, I get it. It takes time and effort to interview someone, and most of you just want to get back to building stuff. Coming up with a standard question lets you get away with doing more with less effort, and gives you a modicum of an ability for comparison across different candidates.
But really take a long look at whether this selects the right candidates.
- Zach Holman, Startup Interviewing is F****d
As Zach notes, this is a difficult problem to solve, which may explain why the system seems to be currently supported by a combination of "that's the way we've always done it" and "that's the way everyone else is doing it." This reasoning gives companies the excuses they need to avoid taking a hard look at their processes to see if they are actually achieving the goals they are supposedly designed to achieve. Most companies seem to believe that, at worst, this system gives them a ton of false negatives but not false positives.
It seems that companies would benefit from asking themselves:
- Does our code test or whiteboarding exercise really improve our ability to identify good candidates? Or are there more relevant exercises (a reasonable homework assignment related to the day-to-day job requirements) or criteria (project experience or open source work that can be judged en lieu of a quiz or homework) that might be more useful?
- Do we really need full day interview marathons or 5+ stages of the interview process? Are there stages we can cut out and still be equally effective? Are we losing good candidates just because they cannot afford to complete our process?
- Are our screeners given the information they need to properly screen people or are we potentially losing highly qualified candidates? Are our criteria too specific (ex. looking specifically for Angular when JavaScript or JavaScript framework experience would suffice)? Does our process seem to result in a high degree of false negatives?
These aren't easy questions to answer, for sure, and, obviously, lots of good candidates still manage to make it through. But I think of it this way: As developers, we've developed system of unit testing to help prevent us from deploying buggy code. These tests don't always check for issues that would cause the whole application to break down. Some might cause the software to work improperly some significant percent of the time or for some significant percentage of users depending on specific criteria. However, we wouldn't allow our developers to deploy code that failed these tests just because the application might work for some users some percentage of the time. Our interview and hiring system is currently failing a ton of unit tests. Let's fix it.
Top comments (13)
I like the turn of phrase "interview/hiring system is failing a ton of unit tests."
Anyway, here's my controversial take of the day: maybe the job interview itself isn't an effective way to scale.
I actually, in writing a book, did a bunch of reading up on the history of the job interview, and it's surprisingly unchanged since emerging randomly as a management fad 100 years ago. Google did some internal research and found that the most effective interview-style, simulating the actual work in the role, was only something like 25% predictive of future performance. (IOW, any given single interview style would be the worst game in any casino for achieving good results).
The job interview is so iconic that when I bring this up, people think the idea abandoning is akin to a suggestion that they stop drinking water or something. But I can't help but wonder if, someday, the job interview won't look a lot in the rearview mirror like waterfall software development: "why did it take us so long to stop trying to predict the future and start adapting to the fact that we're not very good at predicting the future?"
FWIW, we've been building a business with an informal fundamental principle that we don't do job interviews. And it's worked quite well, so far, bringing us to 4 salaried staff and something like 100 engaged contractors. So, it might be possible to scale talent without conducting interviews at all.
/end food for thought
That's super interesting. I will say though that I have had a number of good interviews and interview processes. Often those focused on my working style, how I interact with other, my critical thinking skills, etc. rather than trying to decipher specific coding skills. Point is, I think there are better alternatives that focus on getting to know the person and how they fit into your team and culture.
I had one of those all day marathon interviews where they flew me 3000 miles for a single day and they asked me during lunch what I thought of their interview process. I had the guts to tell them that all these coding puzzles shows them very little of what I am capable of and doesn't even translate the the work they want me to do. I didn't do great on two of the puzzles in the afternoon and I didn't get the job. I only really thought of it because my interview for MongoDB (my previous employer) was the exact opposite. One of their managers even wrote a great piece on how to step up your own interview game. I was particularly excited by his idea of putting all your current employees through the interview you plan to give new people. You should expect them all to pass with flying colors.
That is a really interesting idea. It's an interesting way to validate that you will get the results you expect.
I am glad to see that I am not the only one who abhors the way tech interviews are conducted. If this were somehow producing better engineers I could understand, but the data does not suggest that it does.
viglobal.com/2018/06/13/tech-indus...
This is more than just a matter of preference to me.
Having been born with learning and attention differences I struggled daily against a system that seemed to place the highest value on memorization. Memory is not an indication of intelligence. I’d rather have 64GB of RAM and a 2.5GHZ quad core processor than a 16/32bit processor and 2TB of storage.
To me this is an issue of accessibility and a lack of inclusiveness.
PS. Been an engineer almost 10 years. I have not as of yet, needed to implement a bubble sort
I kind of think it comes down to those who gave you technical interviews. The bulk of them is from HR.
With little to no technical background to give you those tests and when you combine it with negotiating your salary & compensations with them and fulfilling their extensive checklist is kind of hard.
Don't get me started with a recruiters/head hunter who always sends me a cookiecutter message and thinks that I don't know it was a mass spam message to multiple developers on Linkedin.
I think this is definitely true for the screener process, though, in my own experience, the code quiz and technical interview beyond that aren't with HR.
This is something our team at Live Share has been deliberating for a while. Being able to find good candidates is often not a failure of talent but failure of the process. We actually just introduced Live Share interviewing- aka.ms/vsls-interviews so that you can recreate the real developer environment as much as possible.
Great article! One comment about these 'tests' and such. Since I'm quoted above I think folks know how I feel about it. However, I've had a few jobs ask me to do the following and I've been ok with it:
1) Write an article about our products
2) Give a presentation about one of our products
In both cases I was ok with it because a) I was given a LOT of time and b) I asked (demanded) to be able to share the stuff on my blog later.
So for me, while I was losing a bit of time even if I didn't get the offer (or it wasn't an offer I wanted), I at least learned something and got some content for my site.
I have had some experiences like that as well. One company asked me to write a blog post about their product and, should I not get an offer, even said they'd pay for the time. I think that a blog post or presentation is perfect for a DevRel role, but if it were used for a standard developer role, I'd think you'd have to ensure it isn't graded on writing or presenting skills that are likely not part of the job requirement.
I am definitely not opposed to relatively short, relevant homework even for developers. I don't equate these with coding quizzes and whiteboarding. However, I do think companies need to be aware of the time demands of their homework and make sure they are reasonable. I also don't think companies should use this as an opportunity for some free labor by making the developer do actual job work (like fix an issue on a repo) as homework without compensation.
Let me share some insight, at least from Europe. I've done interviews in Greece (where I'm from), Belgium, The Netherlands, Luxembourg and most recently UK.Just to be clear, I've not interviewed in all countries for developer roles as I've been doing Dev management and architect roles for the last decade.
I've only once had a code test, which I did from home with a time restriction. Then they checked and used some parts of it as discussion points. I got actually hired there. The test was ok and yes didn't reflect the job but showed if the candidate had some degree of attention to detail and the ability to google. Later on, they did the same interview to a former colleague of mine who failed on the subtleties and got rejected although this person is one of this commando people that will carry you out of shit situations. Recently, I was asked to prepare a presentation.
If you ask me, code tests could have some merit if you find something simple and you don't take it too seriously. Just use it as discussion points. Most important aspect is the discussion but to be able to proper evaluate candidates you need to be relevant and this is where the problem lies.
I actually think that biggest mistake done is this perception of hiring the best fit for an spot. Like it or not, most jobs don't differ from each other. A CMS is a CMS and yes we've all re-invented the wheel but only in our dreams is our code and product so unique. Hire someone who fits the profile, has potential and give it a try instead of spending so much time trying to find the best person. At least in Belgium this is the biggest mistake. To borrow an expression, they are kind of looking someone to be wedded with the job and this is wrong. If the talent is good then he/she might stay only if properly incentified. So I say, get people to get started, don't waste time and if doesn't work try again but don't wait to find the person while everyone else has moved on.
Interviews should be mostly about figuring out whether there is a good culture fit for the candidate within the team. If everyone on the team gets to work at noon, hacks around until 11pm, and goes out to paintball, a more-staid person probably won't work out.
The problem with so many hiring hurdles is that they are gameable. Want someone who knows data structures inside-out? Cool, but they may fall on their face when asked to write testable code that fits a business need. I recall years ago regaling an interviewer about puzzles (give me a word in the English language that changes pronunciation when capitalized/uncapitalized? Polish/polish).
If you don't try different approaches, and do follow-up a year or so later, how can you say your interview process works? I don't know any company that does such A/B testing, do you?
To be honest, I think much of the problem boils down to two major issues.
In the effort to appear more open to a more financially diverse group, college requirements have been dropped, but replaced by code issues not really encountered outside of college. This means self-learners like me are still effectively shut out, while the company can use the PR to pat themselves on the back for being more inclusive.
In the effort to show that they can't possibly hire citizens from their own country and absolutely must hire lower paid remote developers from another country, unicorn-level requirements are added in the interview process. This allows the company to weed out as many candidates as possible, claim there are no qualified workers in their home country, and hire only people who are willing to work for half the pay. This isn't the fault of the foreign hires, but of the company. This has led to some companies listing job requirements like -
Must be able to make the perfect cup of espresso, have full skills for making realistic CGI videos, be able to code in Fortran, Python, C++, Java, JavaScript, Kotlin, React, Angular, Vue, and Scalar, be able to use Excel and TextEdit, be a subject expert at Machine Learning, and be able to shimmy up a coconut tree and pick coconuts in under 2 minutes, 30 seconds. Benefits include a foosball table and a photobooth in the breakroom.
Shell games for the job market suck.