Are Technical Interviews a good measure of software engineering ability?

Amanda Sopkin on May 01, 2018

Sarah Mei @sarahmei Why do we keep doing whiteboard interviews, even though they tell us little about how good the applicant is at actual sof... [Read Full]
markdown guide
 

This is an awesome and insightful breakdown, Amanda. I really enjoyed reading it. My experience has been that candidates tend to perform better in the following interview settings:

  1. Discussing side projects they've worked on in the past at a technical level.
  2. Discussing open-source projects they've contributed to.
  3. Participating in a pair programming session with a senior dev on the team.
  4. Completing a take-home programming assignment (with compensation for their time) followed by a code review with a senior dev.

Recently launched Whiteboardfree, a job board to help devs seek out companies that do not include whiteboarding/riddles/games in their interview process. Very curious to hear your thoughts about this idea and if it's a useful way to incentivize companies to adopt more mutually beneficial interview practices.

 

I’d say it heavily depends on what the candidate can show before the interview started. 30 projects on github and golden badge in the language of the choice on stack overflow would definitely make the whiteboard ...hhhm redundant.

On the other hand, if the candidate has nothing to show upfront, I would definitely do the whiteboard. After all, they check your driver license before hiring you as a taxi-driver.

 

I would still prefer to hire someone with communication and social skills, than someone that is a "ninja" and only knows to output code. We need humans, not machines.

 

People without rock solid communication and social skills are indeed still human beings, not machines, in the first place. If abusing them for the lack of communication skills is not a harassment, I don’t know what is.

Also, I am a software engineer, not a cheerleaders manager. We run a business that requires strong technical skills and (surprisingly enough) does not require strong social skills. So I test for what is really required, not for what is written in modern era motivational bullshit books. And you know what? The social environment is better when developers can share their interests, not just smile one at another because of “great social and communication skills.”

Developers sharing their interests is a communication skill. All the people I met with golden badges on stackoverflow do it during business hours while being paid to deliver other type of business value. Also, I've worked in strong technical environments and is not as funny as working on a mixed one where you can cheerlead juniors and so forth.

do it during business hours while being paid to deliver other type of business value

You understand basically nothing about what people are being paid for.

Also, people with golden badges spend the same or less amount of time on SO compared to people living there to get answers to anything.

And, the last but not least, to blame authoritative SO answerers for whatever one should stop using their answers upfront in the first place, otherwise it looks too unfair.

It seems you understand/know basically everything, that would be a first step to not work with someone like you. That's why social is so much important, even maybe more than technical.

Feel free to share your ideas on what is important with your boss.

You are both right, first you mustn't discriminate on a persons lack of social skills, but also you shouldn't ignore persons inability to emotionaly undestand and work thogether in a team as part of a team!

 

When I do interviews I get the impression that we aren't looking for great candidates, we're really just trying to route out the bad ones. So many of the people I interview simply aren't ready for a programming position. They lack either the skills, the mind-set, or something else.

Against this flood of under-qualified candidates, anybody who looks even acceptable suddenly flourishes in any process we set-up. I'm positive I could setup a test that only the top qualified candidates would pass, however, it has these problems:

  • We'd have to deal with passing near 0% of applicants, perhaps just 1-2% would make it to an on-site interview.
  • A candidate worthy of passing would rightfully expect some form of compensation for their time, which wouldn't be an issue, except you couldn't exclude the other 98% from this easily.
  • A rigid process can have the effect of scaring away good candidates that couldn't be bothered to deal with nonsense. It's a communication problem to indicate you have a strong process, and not just needless hurdles.
  • The company behind the process must be willing to pay a good compensation to those people that make it through the process. Part of the reason I suspect some companies don't attract the best is because they just aren't offering a good enough return.
 

For me, Whiteboard Interviews are the equivalent of asking a musician to show him how to play "Piano Man" without a piano.

 
 

Whiteboard is unideal and very narrow engineering culture test and it very prone to interviewer - interviewee personality mismatch. As well as it does not cover all worth for measurement dimensions of an autonomous and productive software engineer.

 

Great post! Love the various resources you brought together. This is definitely a tricky problem. Just thought I'd share some thoughts that came to me as I read it.

I was fortunate that when I started getting into web development, there was no whiteboard or test. Places were so desperate that showing any real ability was enough. All I brought was an app I'd built on my spare time. I had no formal schooling or formal experience. The .com boom days were unique and I was extremely fortunate. Had I been starting out today, my experience would have been very different and would certainly have required much more of me just to break into the industry.

I think you touch on some important points in that there is often too much focus on specific skillsets rather than overall fit. There seems to be too much emphasis on "hit the ground running" and very little effort to train and develop people. When I've had the opportunity to conduct interviews, I was often less interested in what specific language you knew or testing you on how deep your language-specific or computer science knowledge was than on finding out about your willingness and eagerness to learn. Language, framework and other specific skills can be taught, but you need someone who wants to learn. Companies need to learn to be patient and develop talent - I believe it pays off in the long run.

I'd also say that I have worked with all types. For example, some people were not most brilliant coders, but they were incredibly productive and didn't mind doing the mundane jobs that needed to be done, but that the "top 5%" coder would frown on doing because it felt beneath them. They played a critical role in the team regardless. My point here is, I also wonder if we often focus so intently on the pieces of the center pieces of the puzzle and forget the pieces around the edge that make the picture complete.

 

The whole whiteboard thing sucks for people, like me, who have insane amounts of anxiety. The whole time you feel like you are doing terribly, and when you don't get that call or you do get the email saying you have been rejected, it makes you feel like you're terrible. I get to use an ide or vim with a ton of plugins and google for my job, why am I not allotted the same resources in an interview?

 

The interview process for technical roles is broken on every level. Consider that if your interview is a test, and not a replication of a real world situation, then the candidates that will do well are students (or recent grads). Why? Because they have been taking tests for the last 4 years.

If you assign a project as the employment test the candidates will submit projects produced in their coding environment. You won't be judging them on the right parameters which would be the same stack the company is using. Not their stack of choice.

I appreciate all the research that went into your post; but did you know that highly intelligent people generally do not test well? Most tend to overthink info/situations and develop anxiety in situations where the outcomes are focused on quantifying performance by standardized answers (rather than giving answers that reflect what they think, or would do, to solve the problem).

The truth is that this kind of testing was born out of the fact that most recruiters/employers don't understand programming/engineering. These ridiculous tests are a crutch for the interviewer at best. At worst the testing process has become an industry. A way of monetizing the recruitment process ("cracking the coding interview", Leetcode, etc).

You want to know if an engineer can do the job? Check the work they have done for the companies they have worked for. Ask their references about them. Have her/him meet the team and hang out. See what kind of repore they develop with everyone on the team. This interviewing process is called: THE OLD FASHIONED WAY. And one more important step: Take it offline. Make it face to face. There is NO online way to get a gut feeling about a candidate.

 

Thank you Amanda, this is so important. If companies get it right, or at least better than the status quo, it could reshape the environment. More emotional intelligence and diversity and less rock stars :-D

ps. manholes are often square here so I would have definitely failed that question :D

 
 

Yeah, this hits home. I interview a lot of technical candidates and it's pretty hard to determine who is going to be a good contributor based on technical interviews. The other problem, which technical interviews don't address, is getting someone who's good at churning out code, but steps on everyone else in the process...I've definitely hired a few of those in the past, and it is a huge detriment to the team they end up on. Thanks for this great article, it helps. I still don't know what the right solution is (currently doing a take-home mini project with a team interview that's more focused on fit and some technical questions) but I'm pretty sure it doesn't involve a whiteboard.

 

Amanda, what a great topic! Thank you for bringing this up!

I'm a software engineer with over 35 years of experience. I have owned a hardware/devops shop. I have worked as Principal Software Architect, Principal Software Engineer, Tech Lead, and Line Programmer over the course of my career.

I've lived through the mindset of a lot of the responses that I've read in these great follow up posts, and I relate.

There are some steps that I'd like to bring up that I didn't see mentioned (or missed) in the follow up posts.

So, the first step in this process is someone who has the say-so and need defines a personnel requirement for a resource to help with a project that is just getting started, or has grown to the point where additional resources are needed. The requirements for this new resource need to be clearly defined. Once this is done, a personnel search for candidates must be exercised through whatever resources the company has at it's disposal to do so. The result of this effort will be a resume list of possible candidates that satisfy the requirement. The senior technical resource at the company might whittle this down to a "short list" of resumes. These resumes should contain relevant experience that addresses the requirements. This will make that technical screener's job easier.

Picking up where this wonderful topic kicks in, finding the best from this short list...

Let's define some goals for the person(s) doing the initial technical screening.
1) First, and most important, end with the candidate having a "positive experience". You are representing your company, and the company's owner(s). Remember that.
2) Go over the candidate's resume, and discuss the individual projects that they have participated in, and especially their contribution to the project. Lead them into discussing in detail the contribution that they made, the problems that they faced, how they addressed those problems, and what they learned during the project. As the selected subject matter expert to do the technical screen, you should be able to make assumptions about the details in their description of their contribution to the project, and glean the depth of their technical experience during this conversation.
3) Lead the candidate though a "soup to nuts" discussion about software engineering in general. Their experiences. The processes that they've used. The projects that they've worked on that were disasters, and get their 20/20 hindsight on lessons learned. The projects that they've worked on that were golden, on-time, on-budget deliveries. The lessons that they learned.

4) Discuss the non-relevant technical experience on their resume. Any side projects that they've done, including self-funded development efforts (like making an iOS game, etc.).

By the time this conversation ends, you should have established common ground with the candidate, and they should have a good feeling about you, the company that you represent, and a good feeling about their accomplishments. You will also have a clear understanding of the depth of the technical knowledge, and be able to describe that in detail to the other members of the search team.

If the first step that I described has been done effectively, you should wind up with over an 80% success rate on your technical screens.

If you aren't, then it is time to look inward.

I haven't mentioned the HR processes associated with this that makes the candidate work from an HR perspective. Happy to follow up if anyone is interested...

Again, really enjoyed this... -Jeff

 

Great post, @amandasopkin ! I really enjoyed reading this.

I wrote about some of my own conclusions after diving into similar research a couple of years ago here, "Hiring is Hard", and here, "Update on Hiring: It's Still Hard"

I'm still experimenting, trying to improve my own process and pipeline. That said, with several years more experience working on this, I have even more confidence in the list of five items from my first post:

  1. Resumes, on their own, are a poor method of sourcing candidates.
  2. The best indicator of skill & talent is the candidate doing what they would do on-the-job.
  3. Whiteboard interviews likely cause you to miss out on great hires.
  4. Pedigree (e.g., school, previous employer) is not an indicator of talent.
  5. Cultural fit and technical communication skills are at least as important as technical talent.
 

Agree with the article except the title.

Software Engineering is a vast domain and most of it relates to project management, process management, quality conformance and certification. So from this perspective, yes, interviews are the best way to test someone in SE.

Here's a definition of SE: whatis.techtarget.com/definition/s...

If you want to recruit for other roles (developer, architect, tester) then the points you have raised are all valid.

 

Those two tweets being referenced were from 3 years ago. Nothing has changed much in the past 3, or even 23 years. Every company claims to only hire "A Players". Yet we all have ample evidence to the contrary.

I think you hit the nail on the head with the notion of "random elements". Those claims of good engineers being scarce are nonsense. That is just HR legal cover to allow companies to discriminate on every basis except ability. There is no magic secret to finding good people. Advertise according to the work being done, not a "purple squirrel" set of requirements that no one could ever meet. Identify candidates who have the education and/or experience to do the job. Hire at random to achieve a diverse workforce.

 

Great article!

But I still think that recruiting processes adopted by big companies are not necessarily ideal for smaller ones. I mean, what if a startup with a team of 5 decides to hire a new member? I'm not sure whether there will be enough resources to do a group interview.

Or should you ask candidates about advanced computer science topics when you are looking for a person who will be working with parsing and producing JSONs most of their time?

Basically, companies should be looking for people who can manage to get the job done, and not "A-players", "ninjas", "gurus", whatever.

 

Was this a follow-up from 2015? (Given date of tweets)

 

I've been seeing a lot of Contract-to-hire positions. I have a feeling it's just like Helpful, to see if the candidate is a good fit, just a longer time frame, normally they're a 6 month contract.

code of conduct - report abuse