DEV Community: The Bar Raiser

The Bar Raiser Interview — Amazon LP - Deliver Results

The Bar Raiser — Wed, 12 Aug 2020 09:10:54 +0000

DISCLAIMER: All interview questions used here are already available publicly from sites like Glassdoor and other blogs. I will not be providing any "special" or "confidential" questions from Amazon, so if that's why you are here, sorry, but you can close this right away. All opinions, aLpHaBeTs, punctuation; and grammar here are my own and do not represent my employer.

Delivering these will take no time!

Deliver Results: Leaders focus on the key inputs for their business and deliver them with the right quality and in a timely fashion. Despite setbacks, they rise to the occasion and never settle.

Interestingly, the Amazon Jobs website lists Deliver Results as the final competency on the list. So obviously the first Leadership Principle that I wanted to start talking about is Deliver Results ;)

Why start with this? Because I just find so many people misunderstand this principle and think it means something it most likely does not.

Let's start with a typical Deliver Results question like so:

Q: Tell me about a time where you worked against tight deadlines. What did you do to make sure you delivered on time?

Now, let's look at a typical Candidate response when faced with that question:

A: I made sure that I worked on evenings and weekends to make sure the timelines were met. It was a difficult 3 weeks, but the feature was super critical to the customer and so I did my best to deliver this.

The Candidate worked extra hard to deliver the right result for the customer. Pretty good, right?

Well, not quite, we have no data on raising the bar here. Why?

Let's see what the missing bits are in this situation:

The Candidate worked evenings and weekends. In other words, worked more than 3 weeks to get it delivered in 3 weeks.
It's not clear if the Candidate ever questioned the timelines.

So let's go ahead and ask the Candidate about timelines again:

Q: Why were the timelines fixed? Did you talk to stakeholders about moving the timelines?
A: No, I didn't question the timelines since it was given to me by my manager.

Ooof. Sorry to tell you my dear reader, but that does not raise the bar. It directly violates the first part of the principle: Leaders focus on the key inputs for their business. If you are not questioning the inputs and validating that we are doing the right thing, what is the point in even doing it?

Delivering results isn't about working hard. It's about working smart and getting things done.

But BR, what if it really was fixed timelines?

Ok, let's try that:

Q: Why were the timelines fixed? Did you talk to stakeholders about moving the timelines?
A: Yes, but since this was a regulatory requirement, there wasn't much we could do to change the timeline.

Ok, better, that definitely counts as a setback as mentioned in the principle. Still not bar-raising though, so let's ask more questions and see if we can get a good response:

Q: Ok, so the date couldn't be moved. What did you do to make sure you hit the deadline?
A: I asked my manager for extra resources to make sure we hit the deadline.

Sweet! So we tried to work smarter to hit the goal by asking for more resources. Surely that must be bar-raising?

Well, not quite. Depending on what level you were interviewing for, it could still be just meeting the bar.

I hear you thinking:

But BR! You can't keep moving the goalpost like that. Tell me how does it matter what level I am interviewing for?

Ok, here is my best attempt to make this clear without divulging any internal specific details:

L3/L4 - "The noob" - These are considered entry-level jobs for relatively inexperienced folks. These could be people who are just out of school, are switching job families, or are coming back to work after a gap.
So for these levels, if you raised a concern with deadlines or asked for additional resources to make sure a project gets completed, there is a 7/10 chance that it not only meets the bar but might even raise it.
L5 - "The Intermediate aka Limbo stage" - This is the "I am no longer a new hire" but "I am not yet a senior" position. People in these roles are typically expected to not need direction to work on defined solutions to problems, but still need guidance on coming up with solutions to problems. So the response above fits in with "I asked for clarity and help, but I didn't do anything beyond that". That translates to a "Meets" in my book about 5/10 of the time. There are times where this might not be a "meets", e.g. Candidate needed prodding from manager to ask these, or someone else was leading the project and asked for extra resources, etc.
L6 - "The Senior" - This is considered a senior position for most job families, and is the first level that has the word "Senior" in the title. Woo! This is also considered a level where people can work autonomously without any direction needed to solve a problem. They might still need guidance on handling or influencing cross-team initiatives but are more often than not very capable of handling things on their own. If you stopped with the above response for an L6 level, there's a high chance (8/10) that you might not meet the bar, let alone raise.
L7+ - If you were interviewing with Amazon for L7, there's a pretty high chance you aren't reading this. If you still need data around this, hit me up and I will be more than happy to update this section :D

So, getting back to the example, what is an example of something that truly raised the bar, regardless of level?

Q: Why were the timelines fixed? Did you talk to stakeholders about moving the timelines?
A: Yes, but since this was a regulatory requirement, there wasn't much we could do to change the timeline.
Q: Ok, so the date couldn't be moved. What did you do to make sure you hit the deadline?
A: I asked my manager for extra resources to make sure we hit the deadline, but we didn't have anyone else to pick this up.
Q: Ok, so what other strategies did you use?
A: So the first thing I did was to figure out which parts of the deliverables could be cut down. I noticed that we could get an MVP version out in 4 weeks. Once I had the subset validated by stakeholders, I then worked on optimizing the other aspects of launching a feature.

I realized that since this was a feature on an existing system, we would be ok with testing only the new feature and its integration with the system, and depend on the regression tests to catch any additional issues instead of spending 2 weeks on manual E2E testing. So that cut down a week of effort, putting us squarely at 3 weeks. I further made sure that all stakeholders including my manager and product teams were aware of what we committed to and where we were with delivery.

Once we got closer to the delivery date, we had a go/no-go meeting every week to see where we stood and what things we needed to re-prioritize to get the product out the door. Before the launch, I helped my team write documentation and SOPs (Standard Operating Procedures) to handle any failures we might face.

The week of launch, I made sure that our metrics were set up to track any failures. Since we had compromised on testing, it was important to track metrics to make sure nothing was off. This would allow us to launch with a high amount of confidence that things would work as intended. Once we launched successfully, I ensured that my team went back and wrote those additional tests.

Whoa, Whoa, WHOA. Slow down a bit BR, what is all this stuff that I have never heard of before?

Ok, let's step back and see what all this Candidate did:

They raised a flag that timelines are hard to hit.
They asked for resources.
They tried to reduce the scope and cut down on work. Further, they tried to optimize existing processes and leverage those to deliver faster.
They kept everyone informed.
They constantly re-prioritized according to the changes in the situation.
They had a mitigation plan in case everything failed and they had a way to track it.
They went back and fixed their shortcuts so that they didn't always have to depend on alternative ways to catch failures

Let's see what all checkboxes they ticked:
✅ Leaders focus on the key inputs for their business - Yup, they checked this for sure.
✅ Deliver them with the right quality - Notice that they delivered the right quality, sure it wasn't the perfect quality you expect, but they made this trade-off consciously and went back and fixed it.
✅ in a timely fashion - Yup, hit the dates, sure they dropped a couple of features, but everyone, including the customer, was happy.
✅ Despite setbacks, they rise to the occasion and never settle. - Yup, they definitely rose to the challenge and made sure they resolved the key risks and deliver on time.

They ticked all the checkboxes and then some. That, my dear reader, is truly bar-raising for this question.

Delivery always matters. Because customers always want things yesterday. Yesterday!

So how do I prepare for the Leadership Principle - Deliver Results?

Ok, so we saw what a bar-raising example looks like for that particular question. But what about other questions? Surely I cannot write a blog post for every (20+) question that Amazon asks a Candidate for Deliver Results?

You are right. And that is not even the purpose of this post.

The purpose of this blog series is not to prepare you for an interview, it is to get you prepared to be a top-notch employee at Amazon who can be successful there long-term.

So what can we do to get there? First, let's focus on the interview part. A good first step is writing down the things that stopped or slowed you from delivering a project. Examples of these might be:

Issues with timelines - Unrealistic timelines, sudden changes in timelines, etc.
Resourcing shortages - Your boss went on vacay, your co-worker got married, your other co-worker went scuba diving for 5 weeks, etc.
Scope creep - customers suddenly need a hundred other things
Miscommunication during projects
Customers not being accommodative
Missing tools or systems to execute your tasks - a missing saw, broken computer, cord system not accurate, etc.

Then start thinking about what you did to solve each of those. Similar to our bar-raising Candidate, think of:

Raising flags when something is not going to get done
Re-prioritizing goals and resources.
Reducing the scope and optimizing existing processes.
Leveraging existing things - standing on the shoulders of giants.
Communicating. Constantly.
Coming up with a mitigation plan for the risks.
Going back and improving processes so that the next time the thing that stopped/slowed you down does not happen again.

If you cannot think of situations where these things happened or where you handled them well, it might be a good idea to start using these to guide you during your next project so that you see how these help in improving the quality of your projects. This is normal for folks who are interviewing for L3/L4/L5 since they might not have much industry experience. Start applying these principles at your current workplace and you will notice that it drastically increases your value to the company.

So, that's it folks. Deliver Results is easy enough to showcase and raise the bar if you have done the right things at your current job. Even if not, start practicing some of what you learned today so that you can apply these at your current job and get even better at it.

Good luck with your Amazon Interview!

Sincerely,
The Bar Raiser.

But.. how do I raise the bar for other LPs or functional competencies?

That’s what this entire series of articles is about. In each article in this series, we focus on a specific LP or functional competency where I will dive deep into what each of them means and what it takes to raise the bar for a given Leadership Principle or competency.

I also fully intend to write about Bar raising for technical roles, including, but not limited to, SDEs, FEEs, WDEs, Data Engineers, Data Scientists, Applied Scientists, Database Engineers, Performance Engineers, and BI Engineers. So stay tuned!

P.S: If you want another take about interviewing with Amazon and learn more about the Leadership Principles, Dave Anderson has an excellent article describing those in-depth.

P.S.2: The hoodie image was found on Amazon.com and is in no way affiliated with me or Amazon afaik. I just happen to randomly own one ;)

The Bar Raiser Interview — Getting a job at Amazon

The Bar Raiser — Wed, 12 Aug 2020 06:54:14 +0000

Dispelling the myths and misconceptions about Bar Raisers at Amazon and what it means for you as an interviewee.

So, you have an upcoming interview with Amazon. Since you're a smart and resourceful person, you have already asked your recruiter what the interviews look like, and since they are a nice person, they already told you to prepare for a bunch of things, including specifically, Leadership principles. You google around a little bit and keep running across this thing called a "Bar Raiser", and there are horror stories of how Bar Raiser rounds are "difficult", "hard-to-clear", "a big barrier", and even borderline evil.

The internet doing its thing — exaggerating things as usual.

Just reading the feedback on the internet, you would think these rounds are purely designed to keep people out of Amazon. The truth, as usual, is a whole lot different from that.

The bar is an imaginary line that represents the 50th percentile of all Amazon employees currently working at that role. A Bar Raiser's role is to make sure that every hiring committee focuses on maintaining and improving this bar.

The whole purpose of this article is to help you understand what Bar Baisers (BRs) do at Amazon, clear up some myths and misconceptions, and show you how to apply this new-found knowledge to better prepare for the Amazon interview. So here goes…

Myth #1: The Bar Raiser is a Gatekeeper

One of the biggest myths around is that the Bar Raiser acts as a gatekeeper and decides whether a hire decision goes through or not.

The goal of the Bar Raiser is to act as a "Risk Manager". During the debrief, we typically summarize the risks we see with a Candidate and make an evaluation of whether that level of risk is acceptable or not. Once we evaluate this, we make a recommendation to the Hiring Manager, who then gets to decide whether or not the candidate gets hired.

The Bar Raiser typically evaluates your risk profile holistically, taking into account your performance on the Interview, the feedback from interviewers for both functional competencies and leadership principles, your prior feedback from phone screens, online assessments, and any red/yellow flags on your resume.

How to make use of this: Try to make sure that you do not say/do anything that will cause the Bar Raiser to think that hiring you is a huge risk. For example, blaming an old teammate or a manager for a project that went sideways is usually a red flag for "Earn Trust", and is something that is almost instantly flagged by any interviewer (not just the BR) and discussed during the debrief.

Myth #2: Bar Raiser interview is the toughest round

Folks on GlassDoor and other review websites often try to guess which round is the Bar Raiser round, and a commonly pushed idea appears to be that the toughest round is the Bar Raiser round and that for technical roles, they grill the Candidates on their technical competencies.

Often, this is not just inaccurate, but straight out wrong.

Why? Because the BR doesn't have to be from the same role that you are getting hired for. They also don't have to be from the same organization and are most likely the only people on the loop who will not grill you on your technical strengths. For instance, I have seen Economist BR's handle SDE loops and vice-versa.

BRs typically tend to focus more on the leadership side of things (especially if they are from outside the org/role), and while it might be harder to come up with examples for these for people who are relatively new to the industry, any experienced hire should have no difficulty in coming up with nice bar-raising examples.

How to make use of this: It doesn't matter which round the Bar raiser round is. Guessing it does not help you in any way. If a round is particularly tough, just move on from it and focus on what's ahead of you. If you want to do well on the Bar Raiser round, just make sure you write down examples for different Leadership Principles and if the BR is also asking a technical question, answer it the same way you would have if the question had been from a non-BR round.

Myth #3: Bar Raiser round is the only round that matters

FALSE. FALSE. FALSE.

This is the one myth that I absolutely detest. Not only is this just plain wrong, but it also propagates the other myths and an unhealthy obsession with finding who the BR is, rather than focusing on excelling holistically across the interviews.

Every interviewer on the loop checks for Bar raising attributes from the Candidate, not just the Bar Raiser.

I repeat: Every interviewer on the loop checks for Bar raising attributes from the Candidate, and if you lower the bar in any of them, it is a concern. If you raise the bar in any of them, it is a strength. The Bar Raiser will collect all the data from everyone on the loop and perform the risk assessment to figure out if the concerns are outweighed by the strengths or vice-versa.

How to make use of this: This goes without saying, but don't mess up any interview by saying or doing things that you wouldn't do in real life. This includes lying about things that happened in your prior work experiences and trying to fluff-answer a question by just mumbling buzzwords. Just avoid it. As long as you don't do terribly on any single interview, and you did really well on at least one interview, you will still be considered a good hire after the Bar Raiser makes the risk-reward assessment.

A follow up to this myth is the next one:

Myth #4: You need to raise the bar in every interview

Unlike the last one, at least this one tries to push you to do better on every interview, so I guess it's not all that bad. However, just to set things straight:

To get hired at Amazon, you don't need to raise the bar at every competency, you just need to raise the bar in some competency, and not lower the bar anywhere else.

The Bar Raiser will evaluate your application holistically and will ask every interviewer whether your responses were in line with raising/lowering/meeting the bar: an imaginary line that represents the 50th percentile of all Amazon employees currently working at that role. As long as you get at least one bar-raising data point, ideally in a competency deemed critical for that role, you will be considered for the role.

So what constitutes a "Critical competency" for a given role? Well, it depends on the role, but generally, this is the competency that the Candidate is expected to excel at for that role. Examples include Coding or System design for SDE's, FEE's, etc. or Business Acumen for Sales Managers, or "Developing the best" for people manager roles.

How to make use of this: Prepare well for your critical competencies. If you are applying for an engineering role, don't mess up the functional competencies section. If you are applying for a leadership role, show good leadership skills, and get examples that align strongly with Amazon's Leadership principles. Remember, you only need to raise the bar in some aspects, not every aspect.

Myth #5: Bar Raisers decide how much compensation you end up with

Bar Raisers almost never have any input into what your compensation is/was/will be. The only time this happens is if the BR feels that a Candidate is an absolute must-hire in which case the BR can make a slight recommendation to the recruiting team that we should treat this person as a "Strong hire". That's it.

How to make use of this: The only time you need to think of negotiation is when you talk to your recruiter. Don't worry about your current compensation or talk about it to any other interviewer or the Bar Raiser. There simply is no benefit to doing it, and it's downright illegal for BRs/Interviewers to know that in several states of the USA. Instead, focus more on negotiating with the recruiters, and do yourself a favor and read this amazing article written by Patrick McKenzie on negotiation.

So how do I prepare for this Bar Raiser interview thingy?

Writing down the things you have worked on is pretty much a “must do”, not just for the Bar Raiser interview, but for all Amazon interviews. Try to answer the following questions by yourself about your projects:

What went right with those projects?
What went wrong with those projects?
What were the biggest risks? and what did you do to mitigate them?
What were the biggest decisions? what data did you have to consult to make those decisions?
Who did you have to talk to?
What were the major points of disagreement? How did you handle it?
Who did you have to convince to make changes to something?
What existing processes did you improve while working on this project?
What could you have done differently?
Knowing what you know now, what would you change?

If you cannot think of the answers to these questions at all, it might be a good idea to start using these questions to guide you during your next project so that you see how these help in improving the quality of your projects.

Once you have most of these down, we have a great starting point for raising the bar and getting hired at Amazon. A general piece of advice is to be prepared enough that you are comfortable discussing these, but not memorize them so that the answers appear mechanical. Bar Raisers are notorious for flipping into different paths while probing a question, so you want to be open to having a genuine conversation. A few things that help:

Don’t fluff your answers: If you did not do something or have not been in a situation, let the Interviewer know. Getting caught on a lie and not being able to answer further severely diminishes your chance of getting hired.
Quantify where possible. Don’t editorialize. Try to get numbers to back you up. e.g. Instead of “I was the tech lead and we delivered the project very quickly and got praised by management”, try “I led a team of X people to successfully complete the ABC project in Y months where we had initially planned for Z months with a budget of $XXX — a reduction of Tk% from the planned budget of $ZZZ.”
Don’t feel like the Interviewer is questioning your credibility: Sometimes Bar Raisers have to go down and peel the onion multiple layers before they get useful data. During this, there might be questions about why you did things a certain way, or why you didn't. Don’t get offended or become defensive when this happens. Talk about the data you used to make those decisions and let the data justify your position.
Be vocally self-critical: If you did something wrong on a previous project, or said something incorrect during the interview, just own up to it. You lose nothing by owning it, and might actually gain some “Earn Trust” and “Ownership” data points while you are at it ;)
Don’t be afraid to question the Bar Raiser’s questions: Feel free to tell them you don’t understand the question or ask them what they are looking for in a certain question. Reframe the question to them and ask if that is what they were looking for, sometimes this makes the question a whole lot easier to answer for you as well.

But.. how do I raise the bar for specific LPs or functional competencies?

That’s what this entire series of articles is going to be about. In each subsequent article after this, I will dive deep into what each Leadership Principle at Amazon means, and what it means to raise the bar for a given Leadership Principle. I wanted to start with the Leadership Principles since these apply to every job family at Amazon.