My regular readers (both of them) know that hiring and developer evaluation are some of my favorite topics. Or perhaps more accurately, they are two of my biggest pet peeves. In my (angry, annoyed, agitated) opinion, soooooo much of how we evaluate talent is fundamentally and critically broken.
Due to this agitation, I've already written tomes on this site about everything that I think is wrong. But some thoughtful commenters on past articles have flipped the equation back on me:
If these are all the things that you feel are "broken", what exactly would you contend is the "right" way to go about evaluating potential hires??
It's a fair challenge. After all, it's easy to sit on the sidelines and gripe about everything you don't like. It's quite another to present a meaningful, actionable plan on how to make it better. So here you have it: The patented Adam Nathaniel Davis Programmer Hiring Plan™.
[HINT: It's not nearly as difficult or as convoluted as so many companies-and-hiring-managers make it out to be.]
I'm not going to get into how you find programming candidates. I'm not a recruiter. From my perspective, quality candidates are dang-near everywhere. But then again, my perspective from within the career field is undoubtedly a bit... biased. I'll just assume that you can find a pile of candidates. But how do you know which one deserves an offer??
The Non-Technical Phone/Video Screen
I feel like this has become a fairly-standard first-step in most hiring processes. And quite frankly, I don't have any real problem with it. So if your organization wants to follow this model, I get it. Some non-techie type calls Candidate to do some really basic, really high-level scoping-out. It should be painless, simple, and brief - 30 minutes max.
It seems to me that the primary goal in these calls is usually to understand the Candidate's salary expectations. And many candidates are dumb enough to spit this out in the first call. (NOTE: This is definitely the subject of a future article.)
My only critique of the initial screening call is that, beyond the desire to discover salary expectations, I'm really not sure what else the Candidate can do on these calls to get eliminated. When I'm on one of these calls myself, it often seems that I'll be forwarded to a second interview even if I tell the screener that I have to cut the call short because I'm flaying someone alive in the basement and their screams are making it hard for me to hear the questions.
I do think there can be value on these calls - for both parties - if there are any aspects of the role that are, in any way, unique or non-standard. For example, if you're calling someone in Poland, but the job would require relocation to Bolivia, then it makes sense to start with the non-technical screening. No need to waste everyone's time if the "opportunity" is a complete non-starter for the Candidate.
This is also a good place to address any obvious high-level mismatches between the opportunity and the Candidate's CV. For example, maybe he has a mix of BA roles and coding roles on his resume, but you're hiring for a pure coding position. Is the Candidate open to this condition?
Although this screening is typically simple and difficult to "screw up" (for either party), there are a few habits I've seen from non-tech screeners that I find annoying AF.
For example, I don't know how many times I've had a scenario like this: A recruiter pings me and says, "Here are the details about a role with XYZ Corp. Can I submit your resume?" I agree and a technical screening will soon be scheduled, but the remainder of the conversation in the initial call goes something like this:
Screener: So how did you hear about XYZ Corp?
Me: A recruiter pinged me on LinkedIn. [The mere fact that I have to state this always strikes me as borderline-silly. I mean, nearly every opportunity that I care to pursue starts with... a recruiter pinging me. On LinkedIn.]
Screener: I see. Well, what do you know about XYZ Corp?
Me: I know that your recruiter pinged me on LinkedIn, which led to this call.
Screener: [facetious laughter] Oh, sure. OK. But... what makes you want to work for XYZ Corp?
Me: I have no idea if I want to work for XYZ Corp. You called me.
Screener: [more facetious laughter]
Me: Look, I gotta go. I'm flaying someone alive in the basement, and the screams are making it difficult to hear your questions.
The Technical Phone/Video Screen
This is also a fairly-standard aspect of many companies' hiring processes. And I don't necessarily have a problem with it. In fact, in some instances it can provide real value. But there are some significant "gotchas" here that I really need you to understand:
This should never be done in-person. I don't care if you're hiring for a strictly onsite role. There's still nothing in the technical screen that can't be handled effectively and efficiently in a phone/video call. Save your own time - and respect the Candidate's time - by making this a purely-remote aspect of the process.
Like the non-technical screen, this call should never exceed 30 minutes. The point of the call should not be to have the Candidate give you a 30-minute soliloquy about his entire CV. Nor should it be to delve deep into every corner of his technical knowledge. It should just be to ensure that the Candidate can speak "programmer-ese". If you can't complete this call in a half hour, you're doing it wrong.
This call should be to test the Candidate's knowledge of general concepts. Yes, those concepts will be "technical" - but they should still be high-level. Focus more on languages (e.g., JavaScript) rather than frameworks (e.g., React). Better yet, focus on even broader concepts (e.g., principles of web development). Do not get cute by trying to quiz someone on, say, the default parameters of some language's particular built-in function. Do not ask "gotcha" questions (e.g., "A
nullevaluates to what type in JavaScript?"). The goal here should not be to stump someone on esoteric concepts. The goal should be to verify that the Candidate is, indeed, a programmer with at least a basic grasp of key concepts. If you're quizzing the Candidate about the difference between theFunctionconstructor versus a function declaration, you're doing it wrong.Agree on the questions to be used in this screening, and give the same questions to every Candidate. This is one of my pet peeves. Joe performs some of your tech screenings, and he asks softball questions. Mary performs some of your other screenings, and she asks hardball "gotcha" questions. This is poor form and it will skew your evaluations.
Give serious consideration to making this screening conditional - based upon the Candidate's experience. In other words, if you're talking to a 30-year programmer, it's kinda ridiculous to put them through this screening. And yeah... I can already hear some of you screaming that you don't know if a candidate's experience/CV is legit, so that's why you put everyone through the same screening. To this I'd reply: Get over yourself. If someone has truly fabricated their resume, there will still be ample opportunities to expose them. (And NEWSFLASH: such blatant fabrication is pretty rare.) When I'm put through these screenings, it doesn't bother me at all. But it's really a waste of time - for me as much as for the employer. I know this'll sound cocky as hell, but I don't care. I'm gonna say it anyway: If I'm applying for a JavaScript position and I "fail" your 30-minute phone tech screening, it says far more about your tech screening than it does about my skills.
The Coding Assessment
OK... So now we come to the real "meat" of the hiring process: The dreaded, frequently-botched, and rightfully-maligned coding "assessment". Someone has passed your initial non-technical screening, and your technical screening, but now you wanna know if they can really code. So here's what you should do:
[NOTE: With modern tools, this should be every bit as manageable for remote employees/interviews as it is for in-person meetings.
Screensharing and collaborative code tools (e.g., StackBlitz) now make it incredibly simple to watch someone coding in a live setting.]
Sit the candidate down at a real, live coding environment. If your interview is in-person, have a workstation already set up. If the interview is done remotely, just have them share their screen with you as they work. If you have multiple evaluators, ensure that they are all present and able to watch the proceedings.
Ask the candidate to begin coding a single, basic app. It can be almost anything. It could be a to-do app, or a tic-tac-toe app, or a battleship app, or... anything. If you need to see the candidate creating a deeply-complicated esoteric feature that will implement live video streaming for your universe of authenticated users, with content controls and dynamic tracking of ad placements, you're doing it wrong.
Tell the candidate that they'll only be expected to code for an hour. That's right. One. Single. HOUR. No more.
Also tell them that they are not expected to complete the app in this time. If your idea of a coding assessment is to give candidates a task that will take hours to complete, you're being a jerk and you're self-disqualifying your company from considering many highly-skilled, vastly-experienced candidates who simply can't be bothered to burn an entire afternoon working on your demo app.
Tell the candidate that they needn't concentrate on minute details of UX. Unless the position is almost-entirely focused on UX, you shouldn't be getting hung up on whether someone has deployed a clean, fancy, responsive, and intuitive
<Grid>control. You should be looking for coders.Encourage them to talk through their solution as they're writing it. Any clarification they can add is useful and will make the subsequent question-and-answer session far smoother.
Tell them that, while they needn't complete the app, if they want to "stub out" basic functions / components / features that they would normally implement, that's fine. Like, if they would normally implement a wrapper for asynchronous calls, it's perfectly OK if they just "stub out" that functionality without necessarily building it from scratch.
Let the candidate code the app in whatever language/framework they prefer. That's right. ANY language/framework. If this sounds ridiculous to you, then you're absolutely not screening for the right "skills". You're looking for people who just happened to have mastered your chosen tech stack. You're not looking for hardcore coders. And you should ABSOLUTELY be looking for hardcode coders.
Once the hour is done, have the candidate stop coding. They can keep the solution on-screen. But you should absolutely be respectful of their time and you should not be expecting them to extend their time so they can dot all the i's and cross all the t's.
The coding session should be followed by a question-and-answer session with the evaluators. That session should also have a firm time control. Thirty minutes is usually sufficient. But an hour is not egregious. If you really need to talk to them so much longer than an hour, you probably weren't in love with their solution anyway, and it's probably unlikely that you'll consider extending them an offer.
Annnnd... that's it. No, really. That's IT. And right about now, you're having an apoplectic fit, right???
Objections
If you're feeling a little triggered right now, that's okay. Really. I get it. This doesn't match your hiring process and every step in your sanctimonious hiring process is sacrosanct, right? So allow me to cover some of the most frequent whinings I encounter when proposing such a process.
We can't take all that TIME to watch all the candidates code for an hour! And then spend time talking to them afterward!!
So lemme get this straight. You're thinking about offering this person a salary that, in many regions, may be well north of $100k. You're gonna invest countless hours getting this person acclimated to your environment. But you can't put in a few hours, upfront with your team, to actually watch this person code??? Think about that for a minute...
I recently did a "coding challenge" for a potential employer. After coding the solution, all by my lonesome, for several hours, they then had me meet with THREE of their most-senior people to talk through the solution for more than an hour. Then they scheduled a second call, with the same people, to talk through some of the finer details of what I'd written! (FWIW, I was offered the job - but I didn't accept it.)
But this is more time than you realize! We have a LOT of candidates to assess!
Then your pre-screening process is broken. If you wanna see coding assessments from a dozen (or more) candidates, then you either have a ridiculous bounty of unimaginably-qualified candidates, or your pre-screening process needs to be "sharpened" a bit.
Let's just imagine for a moment that you do have a dozen (or more) candidates who all seem to be really strong. I still find it hard to believe that some of those candidates aren't sitting closer to the top of your hierarchy? And if there are some candidates who seem to be more qualified than others, the answer is pretty simple: Interview the strongest candidates first. Then, if for some reason those "top" candidates don't pan out, simply move a little bit further down your list.
This isn't enough time for someone to demonstrate all the skills we need to see in a potential programmer!
Then you need to earnestly question the qualifications of your evaluators. Seriously. Every time I've sat down to code with someone else (or to simply watch them code) for the first time, I've been able to get a pretty accurate read on their skill level. And I usually have that "read" within MINUTES. If you've watched me code for a full hour, and you're still not sure if I'm knowledgeable enough to do the job, then I guarantee that I'm not.
But by forcing candidates to do a coding assessment on their own, we save a lot of time because we can eliminate many candidates before we even need to bother pulling anyone into a live interview/assessment!
Ahh, yes. So you're admitting that you're one of those jerks who thinks it's cool to ask an army of questionably-qualified candidates to do your "take-home assessment" (meaning that you're asking them to do FREE WORK). And when the submission doesn't pass your sniff test, you'll just summarily terminate their candidacy? Without even providing any feedback as to how, exactly, their submission "failed"??
Yeah. Gotcha. I know exactly what kind of employer you are. I have no desire to work for you. And guess what?? There's an ever-growing community out there of extremely-qualified candidates who have already come to the same conclusion.
We can't let people code in "any language/framework that they want" because we need a true apples-to-apples comparison!
So what you're saying is that, if your environment is entirely dependent upon Angular & Node, you don't think there's any way that my kick-butt solution, written in React (oh no!) and C#(egads!), may be at all transferable to your tech stack? If I truly wrote a solid solution in React/C#, do you really think it's gonna take me some ungodly amount of time to "translate" those skills into Angular/Node?? C'mon, mannn...
Trashing the Assessment??
If my previous suggestions have you tearing your clothes and gnashing your teeth, then you're really gonna love this suggestion: Sometimes, you should absolutely consider bypassing the coding assessment altogether!
No, I wouldn't normally suggest trashing your coding assessments altogether. But you should really think about which tasks you're asking of which candidates.
I have a fairly-robust GitHub profile now. It contains the code for live, publicly-accessible apps. It contains the code for all of my NPM modules. It contains the code for numerous other coding assessments. If you have any serious doubts about what I've written (or whether I've written it), that makes total sense. We can just do a live review session of the reams of code that I've already written. I can talk you through any design choices that I made. I can even make small changes for you, in real-time, to demonstrate that I own the code, I thoroughly understand it, and I know how to safely update it.
This probably sounds hella-arrogant. (And I couldn't care less.) But the simple truth is that, if you're asking me to write some new little demo app when I already have tons of live, high-quality code that may even be directly-applicable to your environment, then your "tests" are often rather, well... annoying.
And I can already hear the hiring-manager apologists who are saying things like, "Well, if you can't be bothered to do our coding assessment (which is really just another permutation of stuff that you've publicly-posted many times before), then we don't want to hire you." And that's... great. Seriously. It is. But keep in mind that your hiring process is almost-certainly causing a lot of other extremely-experienced candidates (candidates who already have jobs and don't necessarily need your amazing opportunity) to simply "bow out" of the process altogether.
Top comments (29)
Been there, said that. :-)
Nice to see you again on these pages!
What did they say? I am curious..
First of all I would not suggest anyone to be so cheeky, especially early in their career.
Second, I do my homework before interviews and always research the position and the company. It is unprofessional and dumb not to do so.
Third, I always said that with a charming smile. Just before showing them that I know why I applied.
But the point is, and this is the reason I say that, is that, really, they called me, I am happy where I am, and their recruiter teased me, so the question is dumb and inappropriate and they should adjust their interviews accordingly.
Same goes for coding challenges. If I applied you can ask me a 2h take home challenge, if you chase me because I look great for you... Well.. No freaking way.
But I am spoilt, arrogant and kinda settled. Don't do like me :-)
This is a great response. And yes, in case it's not clear to anyone reading, my "approach" is def influenced by the fact that I'm typically lucky enough (at this point in my career) to have companies/recruiters pursuing me. Obviously, if you feel that you really need the job, or if you are in the very-early stages of your career, you're prob stuck jumping through every single hoop they put in front of you. However, most employers aren't looking for "those" people. They're looking for the extremely-experienced, supremely-skilled coders. And when they're looking for that kinda coder, they're doing themselves a disservice by thinking that they can put up a huge set of obstacles for all of the candidates to jump through. Cuz the ones with options won't be bothered.
LOL, yeah... It's been a minute. Thanks!
I am wondering: do you have experience hiring people?
We do have a take home assessments, one for more UX oriented developers and one for more function (or technical) developers. These are extremely simple assessments that allow the developer to spend more time on it if they wish to do so. We allow it to be written using any programming language.
Our primary goal is to filter out people that don't actually know how to program. Our secondary goal is to see the developers 'handwriting' and style of programming. This allows us to determine how much time we will likely spend on educating the developer in the style of coding for our company.
We use this same assessment for freelancers. Here the secondary goal for new hires becomes the primary goal. This allows us to determine what role in which type of project they should have.
I've hired, or been directly involved in the hiring process, for more than a 100 people (easily).
I'm sure you believe so. And I might agree if I saw the assessment. But I have heard statements like these dozens of times - and they almost always end up being not-as-simple as the employer claims. For example, I was recently asked to do a demo app based on an existing repo - and the Webpack config was kinda jacked up. So I'd either have to burn time troubleshooting the Webpack config, or I'd have to deal with the repeated delays that were caused while doing the task because Webpack wasn't properly recompiling the app.
Maybe your take-home assessment is far simpler than that? No Webpack config. No frameworks. Maybe it's just, "Write a function that accepts X and returns Y." But if that's the case, there's no reason that you can't see them do that live.
And there's the rub. We swear that this is just a one-or-two hour task, knowing damn well that some people will take 8 hours to complete it (for whatever reason). And you won't even KNOW that it took them that long to do the task. They may have submitted an incredibly slick solution and they swear that it only took them an hour to do it. But how do you know if they're lying?? After all, you didn't watch them coding. So it could've taken them any amount of time. If the assessment is really so simple, and it took them a load of extra time, don't you think that would be value information??
This also puts the candidate in an untenable position. Maybe you claim that it should only take 2 hours. But at the end of 2 hours, they don't feel confident that they've truly completed the task, or that it's "clean" enough. Of course, they could just "put down their pencil" and submit what they've finished up to that point, but they also know that this will probably eliminate them. So instead, they sit up all night working (for free) on this assessment that you've sworn should be extremely simple.
I couldn't agree more. And that's why you should actually watch them code! If you've watched someone code for an hour, and you're still not sure if they know how to program, and you're still unsure of their 'handwriting' and style of programming, then something's really wrong with your evaluators. When you watch them code, it provides an assessment that's nearly impossible to "spoof" and it respects the candidates' time.
Thank you for taking the time to write this extensive reply! You certainly made me think, I will consider switching to this (or a similar) approach and discuss it with the other members of management.
The trickiest part I think about this approach is that some people might experience disproportionate stress, so making sure they feel at ease will be important. Especially ensuring that solving the problem is not the main goal of the exercise.
Thanks again.
I def agree that it's imperative to do everything possible to make the candidates comfortable. Much of that can be handled with the demeanor you employ when setting them up. But keep in mind that, even in the course of their "normal" job duties (if they were to be hired), it's not uncommon to have two coders working on something side by side (this can include "pair programming", but it might be as simple as someone helping you to troubleshoot a nasty bug). So it's not unreasonable to think that the candidate needs to be at least somewhat comfortable with having others watch them code.
@bytebodger Just letting you know that we are trying the live coding session next Friday with a potential hire.
Very cool!
Please lemme know how it goes. I'd be happy to hear any feedback you have on it.
@bytebodger
It went quite well. Our assignment however is not suitable for live coding. A big part of the assignment is to see how people find and apply documentation and information.
During live coding people feel some form of pressure (unless you have been working with them for quite some time) that prevents them from taking the time to think and read. So at some point I excused myself (it was a digital meeting) to allow the candidate to be on his own. While this was only 3 - 5 minutes, it helped the candidate to focus and read.
During coding there were a lot of excuses and even the phrase: "normally people only see the code once I have cleaned it up".
All in all it was a good experience. I will try it again next time, but I have to carefully redesign the assignment for this experience. I will most likely build in a break (even if it's not digital) at a certain point of the assignment. I might even go as far as breaking the assignment in different stages.
We did it for 1 hour which I think is too short to cover the different aspects of programming:
This is GREAT feedback! And yes, I totally agree that the nature of the assignment itself may need to be tweaked to accommodate this approach. Thank you for following up!
I also wanna make it clear that I don't think the one-hour time limit is sacrosanct. Yes, I'd prefer it be kept to one hour. But I realize that, in some scenarios, you may need more time. What's much more important (IMHO), is to have a set time limit. I firmly believe it's patently unfair to give someone an open-ended assignment, knowing that some people, (especially, desperate people), will end up burning eight-plus hours on the assignment you genuinely believe should take much less time.
I forgot to mention. For junior candidates we often do the assessment up front, before the first interview. For medior and senior we tend to do the assessment between the 2 interviews with more focus on the secondary goal.
And when someone "fails" your assessment (especially junior candidates), what kind of feedback do you give them on why their assessment failed??? Since you're giving all the junior candidates this assessment right upfront, I'm absolutely certain that you'd never think about silently eliminating them without at least giving them some kind of substantive feedback based on what they've submitted, RIGHT???
Haha, a lot of question marks.
But yes, most certainly. I (or my right hand) will call them personally to explain why they have failed the test. This only happend a few times, most likely because they were not completely honest in listing their experience or education.
I feel it is important to give them something tangible to work on. In a few occasions they even re-applied some time later.
My favorite coding assessment to administer is live coding of measurements and conversions. It is simple enough to be done in an hour, universal, and you can't be a good programmer unless you have an innate understanding of the topic.
Can you provide an example?
@bytebodger , Adam, this is a very interesting series. I'm working on a project to solve some of the issues you mentioned. Would love to dig a bit deeper into your ideas, if you can spare a few minutes for a chat. OK to send you an email?
Certainly
I couldn't disagree with this more, I firmly believe that Deliveroo have missed a trick by not requiring riders to be able to recite from memory the steps required to disassemble a derailleur gear, explain the differences in pad wear between disc and rim braking and be able to clearly articulate the practical reasons for frame colour choice. Clearly the interviewing team should review these responses and reserve the right to call the candidate back multiple times to describe the nuances of their answers.
Oh no.
That makes no sense right?
Next time I'm hiring I am going to do this ride along thing, before we've always asked people (who are definitely being interviewed) to talk through a project that they've built outside of test conditions but to my specs. I prefer your suggestion for sure.
I liked your article on how to hire programmers. I think you gave some good advice and tips on how to find and evaluate the best candidates. I’m curious about how you manage and motivate your programmers and what are the challenges you face. Do you have any resources or recommendations for finding programmers for hire? I would love to hear more from you. 😊
Loved this part...."If I truly wrote a solid solution in React/C#, do you really think it's gonna take me some ungodly amount of time to "translate" those skills into Angular/Node?? C'mon, mannn..."
I don't understand what's the obsession with the mighty TECH STACK, especially, in a web platform based products.....
Great post! What about meeting the team? Could be the best candidate but if it doesn't fit, it's a no go if you ask me
Agreed. And IMHO, this fits quite well with the post-assessment Q&A session.
I am sharing this as widely as I possbiy can. 10/10. Thank you * 1e12
My stupid grammar nitpick: " But you should really think about which tasks your asking of which candidates." your -> you're
Thank you! And I've fixed the grammatical error. I genuinely appreciate having that stuff pointed out, cuz I do strive to be clear in my writing. (Even if my chosen style is, quite admittedly, rather "colloquial".)