Last year I wrote a couple articles about my "Amazon journey". You can read them here https://dev.to/bytebodger/how-i-got-hired-at-amazon-3ajl and here https://dev.to/bytebodger/what-its-like-to-code-for-amazon-4nke. My Amazon "career" lasted all of 10 months. On January 18th, Amazon released 18,000 workers. And I was one of them.
I've hemmed-and-hawed over writing this article for a few months. The first few times I tried to write this, it was wayyyy too bitter, and it would've come across merely as "sour grapes". So I scrapped those original drafts. But now, after having a few months' separation from the Amazon machine... I think it's time.
I'm gonna be frank here. This article is basically my own self-administered therapy. If it doesn't strike you as being "professional" or if you feel like hopping into the comments and deriding me for my opinions, well... I really couldn't care less. You can write your own blog post about how I'm an idiot, or about how everyone jettisoned from Amazon deserved what they got, or about how this is all just "the way of the world". But this is my post. And I'm gonna explain things exactly as I experienced them.
Laying My Cards On The Table
Before I get into details, I wanna be dead-honest about a few key points. First, there's nothing about my experience that can be taken as a blanket statement on Amazon as a whole. The company is huge. Their workforce - even after the layoffs - is huge. There are still more than a million Amazon employees. And even if we filter out all the non-coders, it's still a massive workforce. So if you worked at Amazon in the past, or if you work there now, your experience may be completely different from my own.
Second, I can honestly tell you that I was "all-in". I sincerely hoped that Amazon would be my last employer. I really wanted to make it work. So it's not like I came onboard at Amazon trying to "make waves".
Third, it would be really easy in this situation to point to the company-wide layoffs and insist that my removal had nothing to do with me. Afterall, they cut 18,000 people (and they just announced another 9,000 impending layoffs). So I could easily fall back on the comforting bromide that I was a victim of broader market forces. But that would be... wrong.
No one else on my team was released. I was singled out. And it caused me an extreme level of mental anguish from November of 2022 through January of 2023.
So what exactly went wrong? Well, let's dive into that...
Living On An Island
In a quarter-century of software engineering, I've never worked at any company where I felt quite like I did at Amazon. Like I was working on an island. I was hired into a team of backend devs who were accustomed to working only in Java. And to be clear, that's fine. I'm a Java dev myself. But I was specifically hired to work on the frontend app.
No one on my team could even tell me how to run our app locally. No one. You may be thinking, "Well... what's so hard about npm start?" But you could not run our frontend app with npm start or yarn or any other tools that are customary for frontend devs.
When I came onboard, there were a series of instructions in the repo to tell you how to get the app running. They did not work. At all. It took me a few weeks to even figure out how to get the app running in a test environment. And I had to figure it out by myself, because when I reached out (repeatedly) to other devs on the team to figure out how to make it all run... they were clueless.
Even when I did get it running, the "default" configuration was to run everything on an EC2 instance (internally referred to as your Cloud Desktop). And that may have been fine. Afterall, I've worked on EC2 instances before. But the only way to make it run on EC2 was to do a complete compile-and-push of the entire app - meaning, frontend and backend code - that took 13 MINUTES every time I wanted to test my changes.
Everyone else on the team seemed to think that this was just "business as usual". They couldn't seem to understand why I was exasperated to learn that every single code change would take 13 MINUTES to verify.
Granted, I did eventually figure out how to sync my local environment in real-time to the EC2 instance. But it took me a long time to work that out. Once I had the sync running, I could see my changes on the EC2 instance in about 10 seconds. But I had to work that all out on my own, because none of the existing documentation gave any clue as to how I could make that work.
Once I figured this out, I documented all of the steps so that future devs would never need to go through the headache I'd experienced. But I could already tell that my manager, Hardik, was growing exasperated by the delay before I could write new code efficiently.
Even with no meaningful guidance, my path to efficient development would've been much smoother if I wasn't working on a Windows machine. When I was hired, several weeks before my start date, I received an email asking me if I wanted a Mac or a Windows machine. Although I've worked on Macs, I've always done much more work on Windows (yes, I know, this is unusual), so I simply replied that I'd like a Windows machine. That was a big mistake.
All of the internal Amazon documentation starts from the assumption that you're on a Mac. Even after I "translated" many of the commands from bash to PowerShell, I still encountered many roadblocks that were maddening to solve. There were many command line utilities they expected you to use that simply would not run on a Windows maching. Although I eventually did solve everything, after I'd been there for a couple months, I finally said, "Can you guys just send me a Mac?"
To be fair, they did in fact send me a Mac. But by that point, I was already seen by Hardik as someone who was lagging behind.
Farmed Out
Once I finally reached a state where I could efficiently code in the environment, I was immediately farmed out to another team on a short-term, high-priority project. Of course, that project came with a whole other docket of onerous setup requirements.
I was the only frontend dev working on this new initiative. When they introduced me to the project, they pointed me to the repo - but it wouldn't even compile. In fact, I'd get on a conference call with the team lead, Shawn, and it would take him hours to get the app running locally.
When we went through these sessions, Shawn repeatedly apologized to me about the state of the project. He also told me, on numerous occasions, that they probably weren't even ready to have a frontend dev working on the project yet. But I was still expected to crank out frontend code.
They were building a brand new app. But the pipeline was configured to only accept ES5-compliant JavaScript. When I brought this up to them as a major red flag, they literally couldn't understand why I had any problem writing brand new code, in 2022, with vars - and all of the other ancient standards of ES5.
And those compilation problems didn't go away after the first time that I finally got the app running. I'd start working the very next day, after changing nothing in the code, and I'd find that, once again, the app wouldn't compile. It didn't compile because the backend devs were constantly changing key parameters of the app. But those parameters were not all checked into a cogent changeset. So even if the app worked perfectly fine yesterday, I'd get latest when I started working the next morning and... nothing worked.
When this happened, I'd get back on a conference call with Shawn, and even for him, it would still take him hours to get the latest build to a point where it would simply compile. He'd still apologize to me about it the whole time. And he kept telling me that they really weren't even ready to have a frontend dev working on the code yet. But I was still working on the project. And nearly every day I ran into the same kinds of issues.
Even when I could get the project running, it was still maddening to figure out exactly what I was supposed to code. They kept telling me that I had to code for certain variables that would be coming from the backend. But those backend services weren't anywhere near being completed. And they couldn't even give me a mockup of what the data was supposed to look like. I was literally coding blind.
I'd conference with my regular manager, Hardik, every coupla days. And every time I told him that this project was a nightmare and that I really shouldn't be on the project at all until they'd figured out their backend issues. And Hardik agreed with me.
I also repeatedly told the team lead, Shawn, that my time would be best spent if I came back to the project a few months later. And he agreed with me. But it took more than a month before I was officially able to beg myself off the project.
Despite all these issues, that were acknowledged by my regular manager, Hardik, and by the team lead of the secondary project, Shawn, I learned a few months later that I was reported to senior management as being "difficult to work with". It was an absolute betrayal. But I tried not to let it bother me, because by the time that I received this information I was already moved back to my original team.
Back On The Reservation
At this point, I'd successfully navigated the insanely-obtuse process of working on my main team's frontend application. And I'd weathered the nightmare of the temporary project on which I was farmed out. But it was clear that my manager, Hardik, thought that I was somehow lagging behind.
He and I had a meeting where he suggested that I should just concentrate on cranking out code for our main app (it's called "Sonar"). He specifically recommended that I focus on trying to get 10 pull requests submitted over the next month.
I totally agreed with him and I dove into our "regular" work with aplomb. In fact, I had more than 10 pull requests submitted in the next week. But from that point forward, he continually talked about me as though I wasn't doing much work.
This situation continued on for several months. It actually got to the point where I showed him, with git commits, and pull request history, and completed tickets, that I'd completed more than four times as much work as anyone else on the team. But it just didn't seem to register with him. He'd always talk about me - to my face, and in group meetings - as though I was somehow failing to deliver.
I wasn't the only one who noticed Hardik's disconnect with reality. We had a senior architect, Andy, embedded within our team. He would routinely talk to me on Slack about how frustrated he was with Hardik. He told me that he'd reported these frustrations to our director. And he encouraged me to submit a report of my own experiences. However, I held back from doing so because, at that point, I'd still only been there for about five months and I really didn't think it would be a "good look" for the New Guy to be griping to the director about his manager.
Captain Rollback
I soon learned that Hardik was also Captain Rollback. In our codebase, we had a handful of ancient unit tests that were borderline useless. (Also, fun fact: No one knew how to run them and they weren't hooked up to the deployment pipeline in any way.) Even if you could figure out how to run them (I did), they covered such a small part of the app as to be useless.
There were also significant parts of our app that no one on our team even knew how to use. So between the worthless test "coverage" and the complete lack of institutional knowledge, the only real way to test your changes was to push them to your EC2 environment and check them manually. But without sufficient end-to-end knowledge of the app (which no one had), manual testing was hit-or-miss.
Perhaps because of these factors, Hardik was Captain Rollback. Whenever any bug - no matter how trivial - was discovered in a LLE, his first-and-immediate instinct was always to rollback the code. Of course, when you've just deployed to production and you found a bug, I can totally understand this impulse. Or if you have a production deployment pending in the very near future, I also understand this impulse.
But if there was any issue with anything I'd written - no matter how trivial the issue - he never asked what it would take to simply fix the issue. He immediately wanted to rollback.
On several occasions, after a bug was found in production, he wanted to rollback my latest changes in the LLEs, even though my changes had not yet been deployed to production. On other occasions, he wanted to rollback my code, even though the change I'd made had no conceivable connection whatsoever to the observed bug.
And I wasn't the only one telling him this. Other devs, with much more experience on the team, would tell him the same thing. But it didn't matter. Any observed bug led to an immediate knee-jerk directive from Hardik to rollback my latest changes.
The Project From Hell
Although I had some ongoing "issues" with Hardik, I was still intent on delivering solid work. And that "solid work" took the form of a big new project that had to be implemented in Sonar.
To be clear, this "big new project" didn't really seem like any kinda major project upfront. We went through sprint planning and I pointed my tasks as presumably taking several weeks. I was giving my estimates based on everything I knew about the app (and the newly-required features) at the time. But there was much that I couldn't have possibly understood about the requirements in the early stages...
We worked off a continuous deployment model. And even though my piece of the project seemed small, it was clear that the entire effort would take a few months. So when I started planning how to do my pieces of the project, it was clear that I couldn't just commit them upfront - because if I did, they'd get deployed to production and they would break because the backend pieces hadn't been completed.
I talked with our architect, Andy, about how to handle this. After talking through a few options, we both agreed that this should be deployed under a feature flag, so that my new pieces wouldn't break anything before the full project was released.
Then I asked him, "Can you show me in the codebase where we're already using feature flags, so I can follow the same methodology?" He replied that there was no existing feature-flag functionality in the codebase. And that was fine. I set about building a new feature-flag system. But of course, that set back my original estimates.
Granted, I copiously communicated this to the team every step of the way. I created new tickets to track the feature flag development. I repeatedly advised the team about the revised timelines during nearly every standup. I also built the feature-flag functionality to be "universal", so that it was highly configurable and could be used for any-and-all conditional features that may be needed in Sonar in the future. Nevertheless, Hardik was not pleased.
Shifting Data Models
Once the feature-flag functionality was in-place, I set about coding the original requirements for the project. The frontend functionality seemed, on the surface, to be pretty simple. Basically, we needed to account for a few new values coming from the API. Originally, I was led to believe that someone else would be altering the API. But I eventually learned that I would need to do it myself. Which was... fine. But that was never in my original estimates.
So I created new tickets to track the updating of our APIs. And I communicated this all to the team, repeatedly, during standups. Then I coded the Java updates to deliver the values that were needed.
Only after I'd done this, Andy informed me that my data model wouldn't "work". It's hard to explain here, but suffice it to say that, in their odd data model, I instead needed to infer the values based on a very convoluted set of other variables that would be returned from the API. Of course, I had no way of knowing that when I made my original estimates.
After I'd already coded a new Java API to handle what I presumed would be the new variables, Andy informed me that he would be spinning up his own set of APIs that I'd consume. Which meant that my previous Java API work was wasted. It also meant that the frontend code I'd written, which assumed it would be receiving the values I'd coded into the API, had to be completely rewritten. He was also unable to give me any kinda putative mock data that I could use to write new frontend code. In other words, I was stuck waiting on him to show me what the new data would look like.
I communicated all of this to the team, repeatedly, during standups, and I did what little frontend coding I could in lieu of any knowledge of the data model.
During this same time, Andy continued to tell me how frustrated he was with Hardik's crappy management. In fact, one day he scheduled a one-on-one meeting with me where he told me that he was transferring to another team. During this meeting, he also told me that, "If you'd ever like to move to another team, just let me know. I'd be happy to work with you again."
Deadlines Uber Alles
We were supposed to deliver this new functionality in the middle of October. We'd previously planned to have the functionality in our LLEs right around October 1st so we could begin testing. I finally received a mockup of the data model against which I needed to code my features on Friday, November 30th.
By Monday, October 3rd, Hardik was in a tizzy. He was trying to ping me on an hourly basis for updates about when my piece would be completed. I was cranking out code in a rapid manner. And I had done a good bit of "pre-work". But there was only so much I could do to finish the features that relied on a data model I'd only been given one business day prior.
By Tuesday night, Andy was pinging me on Slack and telling me that Hardik was gonna throw a whole bunch of other devs into the mix. Presumably, to implement the features that I hadn't been able to complete. (Again, these were the features that I could only really finish coding since the previous Friday.)
For the next week, I wasted a massive amount of effort. We had four different devs coding multiple features that resided in the same handful of code files. On numerous occasions, I went to submit a PR, only to realize that someone else had already submitted their own version of the feature. All of us were constantly dealing with merge conflicts.
Despite all of this anarchy, we got the features implemented and promoted to the LLEs. That's when I went to Seattle for a team onsite meeting in the third week of October.
Stay tuned for Part 2...
Top comments (30)
I worked for amazon 4 years ( I left 1 year ago) as frontend mainly. First it is really wired they laid off only 1 member of a team, usually they go full org. Might have been the case for making up a number they choosed you as the newest hire.
You described some half reality about the tooling system, yes most of them are old, but the good part of amazon is that no one can stop you from changing things, indeed that is a plus part, you can do all the changes that improve the developer pain as long as you do not delay projects.
I am not sure how can an amazon pipeline be bind to ES5 as you describe, most probably what you wanted to say is that the node build version of the pipeline was old, again nothing you could not upgrade.
One thing that you miss, is that these are big companies very big, build over years, so you will find legacy code and tools, which are hard to change and migrate without the proper business justification. You have to deal with it, happens in all big Enterprises. I can tell you IBM servers run still on Cobol...
Engineering wise, in my opinion, amazon is one of the strongest company I have ever worked.
The EC2 cloud desktop was not that bad as you describe it, and you have the option to chose Mac or Windows laptops. I have developed in linux,windows,mac, and honestly the best experience is on a Mac, for many reason I will not outline here.
Documentation is not that bad, there is an exhaustive internal wiki, maybe organization of it is not great.
Pipeline, tools, monitors, automatic rollbacks, are really really at an outstanding level. I repeat engineering wise amazon, imho, is on top of many other tech companies ( I thought sometime I could not find worst places that this but I was wrong).
Said that, the real issue I saw in that company, was toxicity, pressure, deadlines, politics, promotions, cross team collaboration, adding more scope to projects, changing last minute, keep same deadlines ect.... this is I think the biggest issue. The promotion process does not depend on how hard you work or what cool stuff you did, it is purely a show off, pissing context, and how good you play with management...
Feedback are used as weapons, leadership principles are used to make a wrong point (even it is not the right way)...
And I changed, moved to another big, which I cannot say the name , and guess what I found the same problems, at a worse level, ( I was thinking cannot be worse than where I was before but ... it can actually, even much more).
but that is a story for another time
First, THANK YOU for this thorough and thoughtful reply. I'll go through it point-by-point, not as a rebuttal, but more to clarify and/or agree with your points:
Yes, it is weird. That's why I openly admitted, near the top of the article, that I was singled out. Given that my new manager loved me and I was receiving glowing feedback from inside and outside my group, I don't think that I would've been released if there were no company-wide layoffs taking place. What really happened was a combination of factors:
Even if everyone on my current team was pleased with my work (and I have every reason to believe they were), there were several people from my previous team (including my previous manager) who weren't happy with me.
Although they had no intention to strategically axe my whole team, I'm certain that, at the director level, there was pressure to identify X number of people who could be released. So when he (the director) is trying to meet his quota by identifying X number of people, he looks into my team and he sees a guy who's A. still very new, B. under a Performance Improvement Plan (PIP), and C. has garnered an HR complaint (for the non-infraction of smoking a hookah). BTW, the PIP and the HR complaint are covered in Part 2.
I agree with this... to a point. I absolutely saw that the devs are tremendously "empowered" at Amazon. And yes, they can make a lot of the changes you mention. But on my team I found that, if I even spent an extra half day working on something to improve the developer experience, and even if that still allowed me to complete all my work according to the original timeline, my manager was still... annoyed.
That being said, I did do numerous things to improve the developer experience. Specifically, I ensured that none of the new devs who came on after me (there were several) would ever have to go through the same headaches.
First, you are correct about Node versions. On my "main" project, me and another senior dev worked to upgrade the Node version so that modern standards could be used.
Second, the ESLint configuration files were set to block ES6+ standards. Yes, you can change the ESLint files. But on the project I was farmed out on, the other team specifically resisted any suggestion that we would update those configurations. To be more blunt about it, they liked the fact that the code was all restricted to ES5.
Oh, sure. I totally understand that. Big/old companies, by definition, will have a bunch of big/old legacy apps. That's just part of the job. But one thing I didn't cover in the article is that, when I was hired, I was specifically given a mandate to update/upgrade the legacy app. Of course, you can't just jump in and do all those updates at once. You have to pick-and-choose what you can update and when. But I was supposed to be actively looking for ways to update the system - and that's exactly what I was doing, while still keeping current deadlines in-mind.
There's nothing inherently wrong with developing on an EC2 machine. That wasn't the problem. The problem was that, when I got there, they had no way to deploy the frontend app independent of the backend Java it was tied to. So every time I wanted to deploy/verify my frontend changes, I had to compile-and-deploy the entire frontend and backend app. That compile/deploy process took 13 minutes. Every time. I timed it.
To be clear, I eventually figured out how to reconfigure the tooling so that I could deploy only my frontend changes. That's when I was able to drop the compile/deploy process down from 13 minutes, to about 13 seconds. And I documented everything about how that was done so no one else after me would have to figure it out on their own.
You're absolutely correct that I had the option. The "issue" was that, before I was ever officially onboard, and before I ever saw our environment or had a chance to talk to any of the other devs on the team, I received an email simply asking me if I wanted to Mac or Windows laptop. Because I've been most accustomed to working on Windows, I requested a Windows machine.
Ideally, someone on my team (presumably, my manager) should've reached out to me first and said, "Look. You can work on either kinda machine. But everyone else is using a Mac, our toolset is largely tilted towards Macs, and you'll probably have a much easier time if you request a Mac." If that had happened, I wouldn't have stubbornly stuck with my request for a Windows machine. I would've simply requested a Mac. And many of my initial headaches would've been avoided.
I would never argue that Amazon's documentation across the board is lacking. Quite the opposite. They have sooooo much documentation that, at times, it's kinda mind-blowing. But with regard to the app that I was working on, there was no such documentation.
BINGO! Yes. Sooo much YES. To all of this. I couldn't agree with this paragraph more.
Yes. And I wanna be clear here. I've been in this career now for a quarter century. I know that no place is ever gonna be perfect. Every job/company/team/codebase is gonna have unique challenges when you come onboard. I totally get that.
My "problem" wasn't with the fact that everything wasn't easy for me. My problem was the blatant backstabbing and the complete disconnect from my original manager. All he cared about was covering his backside - even if he had to throw his own team members under the bus to do it.
You have been chosen surely to match numbers, PIP and HR do not come into play for new hires at all in the first 2 years. 1 year at amazon is still considered onboarding.
ohh backstabbing was a common thing there, that's how some people use to get credit, pointing fingers and playing bad..
it's quite a complex situation, after these experiences, now I just keep in mind, that we are only a number, a product, disposable (as the recent lay off showed) despite you can try to be honorable, loyal to the company and team, trying to invent stuff, put passion in the work, do best code, ect...
Most of these companies had huge profits, they still ran the lay offs... think about.
So now I am focus on leverage, I am not focus anymore on doing the best thing, but the one that gives me the most output... Edmond Lau has written a good book "The effective engineer".
That idea of "leverage" is a very good (and interesting) one. I may indeed check out that book. At the very least, it will give me some important food-for-thought as I think about how to grow in my next role.
Thank you!
It sounds like you had a lot of challenges, but you also learned a lot. I'm glad you were able to find a mentor who was able to help you. I think the advice you gave about being honest with your manager about your struggles is really helpful. I'm also interested in what you said about the lack of documentation. I've heard that this is a common problem at Amazon. It's frustrating to have to learn things by trial and error, but it's also a great opportunity to learn and grow.
On my team, there was actually a massive focus on documentation. But it wasn't the documentation that most devs think of - meaning, documentation on how our own software worked. Instead, their idea of "documentation" was to write up a sizeable document to accompany almost any of our daily tasks. Like, documenting... what you did. Or documenting... what you were planning to do. But that documentation had nothing to do with traditional software documentation. Our main, legacy app was almost completely undocumented in any way.
I can relate to your experience with documentation on your team. I've been on teams where there was a lot of emphasis on process documentation, but not necessarily on traditional software documentation. Thanks for sharing your perspective on this.
This is a great honest look at being a backend Java dev in an increasingly front end world, and I can honestly say I've had a lot of the same pain points. Keep writing and I hope parts 2-7 see you off to a better job.
Haha, I wrapped it up at 3 parts. Of course, I'll continue to blog about many more topics around software development. But there's only so much navel gazing I can do about my time at Amazon.
Looks like you're, in fact, hard to work with! :) Comparing your git commits with the ones from your colleagues is a big nono iny my opinion. Also you don't have to build the most universal feature flag implementation. It's about getting shit done and if you take weeks to even setup the project and aren't proactive regarding improving the dx, I would have kicked you out as well
Oh, cool! Someone else who can't read!
You conveniently omit the fact that I compared git commits, and PRs, and completed tickets. And you also omit the fact that I only did this because my manager was repeatedly inferring that I was somehow doing less work than my colleagues. But I guess your answer is that I should just sit there and nod along when he says these things - even though the data showed the exact opposite.
I was pretty clear in the article that I reached out, repeatedly, to others. No one else on the team was able to help in any way. I was also clear in the article that I documented the whole process so that no one after me would have to go through the same headaches.
But I'm sure you know all about it. You were there. And you obviously have all the details.
I have no details. I'm just giving you my opinion. I also wouldn't expect you to agree with me obviously :)
Are you Hardik with a different username? 🤔🤔🤔
First things first,
I like the format of your experience report. For sure ist is about complaints. Everyone complains in the SD area. That's normal. It is about constant improvement and the will to go forward. Not everyone likes it and management does not understand, mostly.
I found myself in your post in few areas. I also had troubles in the past and got kicked out of a fancy startup because I was sick of the bullshit from the management and was criticizing alot. My learning path was to watch my mouth and keep out emotions from work. Also I needed to be more patient in general. I can not expect that others have the same view as I do have. So the key was communication. Since two years I am team lead with ten colleagues in a huge project. The management is also really nice. Slow but nice and agile transformation ist just a word tbh. But I find my way in it and I care for my team's concerns a lot.
Hope you find satisfaction with your next project.
Don't mind the negative comments. There are a lot of jerks in this world.
This reminds me of an old job. Rank incompetence, self-serving managers, backstabbing culture, newbies thrown to the wolves. Only way to stay on board is to have an incredible tolerance for pain and a large helping of luck.
Sorry you had this experience. Hope your next roles are happier.
Ohh, I know that you felt with run project. In perfect world I wanna run only one command and expect that project runs
Amazon is on my top list of companies I would never consider working for - this was your first mistake, entering this toxic vortex. It clearly escalated from there. I am sorry to hear about your experience - looking forward to part 3. Godspeed brother.
I hear ya. And I can't say at all that I disagree with you. But I'll also be dead honest and admit that, when they came calling, the $$$ was nearly impossible to turn down. It was a clear deal with the devil. And I I'm not gonna act like I was dragged along against my will.
Somehow this blog entry was suggested to me by Google dashboard.
And I gotta say, I've been through a very similar experience and if you wanna talk about it hit me up
Some comments may only be visible to logged-in visitors. Sign in to view all comments. Some comments have been hidden by the post's author - find out more