DEV Community: David Zhang

System Design is More Important Than You Think

David Zhang — Wed, 07 Feb 2024 00:56:39 +0000

This past month, I've been going all in on system design. After face-planting badly in one of my interviews (Temper your imposter syndrome by reading my previous blog post about my hilariously bad performance), I took the failure personally and decided to dedicate some serious time towards improving my system design knowledge and skills. If there's one thing I learned on this journey, it's that system design isn't just an interview prep thing, or something that only senior engineers should worry about. It's a vital skill that every software engineer should try to regularly practice and develop.

Story time: I launched System Design Daily, an interactive study tool for myself mainly, but also other engineers who want a more engaging way to learn these concepts. I've recently added some gamification features that I'm pretty excited about, but that's a story for another article.

Hard-stuck bronze in both Valorant and system design interviews

After launching the site, I got a whopping 3 users: me, my test account, and my partner who I definitely did not force into using it.

Me checking my user numbers at launch

It was time for some #digitalmarketing and #buildinginpublic. The only problem was I had about 14 followers on Twitter and about 4 posts in the span of 9 years. I'm not a big social media guy.

At first I went the Herbalife route and hit up all my software engineer friends and family, begging for some feedback on the site. This kind of worked, though one issue was that all my close friends and family were job-havers, and weren't actively interviewing. So this tool wasn't really of much use to them. (Hopefully by the end of this article I'll change their minds).

I decided to change up my strategy and talk to strangers on the internet instead. Instead of asking them to go use my site, I asked for stories and experiences. I began by hitting up my LinkedIn network, specifically looking for #opentowork folks like myself. I then expanded to Reddit and random Discord communities. And in speaking to people and reading people's comments and replies, I began to realize a few things about system design.

The Job Market 💩🔥

Speaking to fellow unemployed brothers and sisters yielded two key realizations about the state of tech hiring in 2024:

The job market is super competitive and terrible
If you're a new grad - point 1, but like put it in big red text with flames and a skeleton for extra emphasis.

It's freaking brutal out there. Engineers are getting laid off left and right. Tons of people with lots of experience are flooding the market. Many of them are absolute Leetcode gods - I've seen some profiles with more Leetcode problems solved in a single day than I do in like a whole year.

What does that mean? It means that everybody is good at the coding interviews now. There are half a billion Youtube channels explaining Three Sum and Fast and Slow Pointers. Literally every question ever asked or ever will be asked is on Leetcode. As a result, the interview landscape is starting to change now that companies are starting to get wise to Leetcode grinders memorizing solutions, and, in some cases, system design and behavioral interviews are supplanting Leetcode-style interviews entirely. The bottom line is: if you are an engineer trying to compete, Leetcode is no longer enough.

For industry hires especially, you can't really afford not to be good at system design. At your level, system design and behavioral interviews are going to be the determining factor for whether or not you even get the job. System design in particular will be used to determine your seniority level. So it also serves you well to know this stuff if you want to make the big money and impress blind.com.

The Average System Design Studier, circa 2024

For you new grads out there, you might be thinking - well I'm off the hook right? Wrong! (kind of). While it's true that you won't normally be asked to showcase system design knowledge in your interviews, it's an extremely important long term skill to develop. That's because knowing system design will accelerate your career. I'll get into this in more detail, but long story short, you'll build up a design "sense" and systems knowledge that'll empower you to reach for more responsibility in your job. The end result is a fast track towards those senior / principal engineering roles with the giant paychecks.

And another thing for the new grads out there - While the system design reaper won't be knocking at your door for now, it's evil little brother low level object-oriented design most likely will. Sure, it might not operate at the same granularity, but it exercises the same overall skillset: you'll need to gather requirements, organize and describe your classes, and explain how they interact with each other.

Career Growth

So you might be thinking to yourself, "well, I have a job. Let those unemployed fellas worry about system design"

First of all, I admire your confidence in believing you'll have permanent job security - especially with the recent mass layoffs in tech and the rise of AI.

But secondly, I believe that system design can be useful even for you employed folks out there.

There's a reason why system design is used to hire for senior engineers. System design is a gauge of experience. The people who are best at system design have seen a lot of things and have built a lot of systems. They know what can go wrong because they've been there, and can design accordingly to mitigate potential problems.

But just how do you get to work on those systems? Sure, maybe your manager will trust you enough to just hand you large scale projects unprompted. But 99% of the time, you'll need to ask for more responsibility and ownership. And in order to do so with confidence, you'll need to prove that you understand things like architectural patterns, data storage technologies, and scaling strategies. In short, system design will help you go from junior to senior.

Ok, so what if you're already a senior or principal engineer? How does this help you? First of all, teach me pls. But second of all, technology is constantly changing, and it's easy to lose these skills if you don't keep them sharp.

When I worked at Amazon, one of the biggest things I was cautioned about was how easy it was to get isolated from the rest of the wider software world. When you always use the same tools and processes, you start to rely on their ability to do the right thing, and eventually forget why they're built the way they are in the first place. You also shut yourself off from new ways of approaching problems that you might not have thought of. Reviewing system design topics will also help you stay on top of the latest paradigms and technologies and will make you more adaptable.

System Design Ain't Leetcode!

That brings me to my next point. System design is not like Leetcode. What do I mean by that? Well, there's a popular mentality among software engineers that "interview prep stuff" (which usually consists solely of Leetcode), is a thing that only sees the light of day when it's time to find a new job. Under normal circumstances, "House Robber II", "Merge K Sorted Lists", and the rest of the Blind 75 gang get buried under a foot of cement in the basement of your memory.

Engineers re-starting their Leetcode premium subscription be like

To be honest, this is a perfectly fine way to approach Leetcode and DSA/coding interview prep in general. That's because most of the problems you'll be solving aren't things you'll encounter in your actual job. And what's more, the "whiteboard interview environment" is a completely unnatural setting contrived by large companies trying to filter out applicants. In no real world context will you ever have to produce an efficient implementation of an often obscure algorithm entirely from memory in under 40 minutes. So, like a pull-based CDN, you pretty much only retrieve this knowledge and skillset from object storage when needed.

This isn't a new idea at all. "Leetcode interviews" have been getting (rightfully) roasted by tech influencers for some time now. But I'll argue that while this "cold storage" mentality makes sense for Leetcode, it does not and should not apply to system design.

As I've mentioned before, system design actually has real world applications. You won't always have a ton of opportunities to design high level system architectures in your job. Your day to day will inevitably get crammed with non-design related work, so you need to exercise your design muscles with a healthy regimen of study and practice.

Furthermore, system components and design patterns are constantly evolving alongside the tech landscape as a whole. That means you need to keep up with them if you want to stay on top. Understanding system design gives you a framework to do that effectively.

The Takeaway: System Design Is Dope

At the end of the day, knowing system design will make you a better engineer, plain and simple. Though it might just sound like I'm just saying that because I'm working on a system design study tool, I really believe that understanding how scalable distributed software systems work will make you better at building scalable distributed software systems.

I've also just found it fun and cool. Of course, the prescribed methods and resources for system design can feel like the opposite. But if you can get past the often dry and bland presentation format of the material, you might, like me, become captivated by how companies with insane scale deal with the unique, often unexpected problems that come with it. For all the gamers out there, it's not unlike playing one of those optimization or management simulator games (think Factorio, Two Point Hospital, etc.).

Anyways, I just think System Design is great and underrated and wanted to talk about this. That's about it. See ya.

The Problem with System Design Interview Prep

David Zhang — Fri, 26 Jan 2024 21:02:28 +0000

The Situation

Back in December, I got the unfortunate news that I had just been laid off from my job. I’ll never forget that moment — I was on vacation at Disney World’s Animal Kingdom, sitting on the bleachers with my AirPods in, eating cold $12 Dinoland USA chicken strips. As I watched Goofy and Pocahontas wave to the audience from their party boat sailing across the lake, I grimly listened to our CEO talk about COBRA and equipment return policies.

Anyways, being freshly unemployed meant it was time to get back on the job-hunt saddle and start shotgunning my resume out into the ether. Since I’ve been lucky enough to have had steady employment for the past few years, my interview skills were pretty rusty to say the least.

But I had a plan. I would embark on an exhaustive, fine-tuned training regimen that would get me back to my glorious post-college interviewing prime. I daresay it’s the most comprehensive, sophisticated prep methodology ever devised by man or beast. I was unshakably confident that it was 100% scientifically guaranteed to land you any job at any company.

It went like this:

Go to leetcode.com
Do problem

Much to my surprise, this fool-proof process didn’t work as well as I thought. There was one key difference between me and the me of 5 years ago — 5 years. The jobs I was eligible for were no longer “New Grad” or “Entry-Level”. They were now “Mid-Level”, “Industry Hire”, or even “Senior”. And that meant Leetcode problems were no longer the main course. They were the appetizer.

Enter two new villains in the tech interview canon: System Design and Behavioral.

Behavioral interviews aren’t usually that stressful for me, since I’m generally a pretty passionate guy who loves to talk about things. Just ask my partner how many times I’ve kept her up until 3AM ranting about basketball. That being said, I’m not some master of behavioral interviews and there are plenty of ways I’ve f-ed these up (which I might cover in a different post)

System design, on the other hand, was a different beast entirely. It was an interview form I’d never seen before. My very first encounter with system design will forever be etched into my memory. In a cruel twist of fate and irony, I was asked to design a Leetcode-style coding interview platform. My answer went a little something like this:

ME: “Architecture? Sure, uhhh… I guess I’d have a client that talks to a server…And then I’d have that server talk to a database…”

Interviewer: “OK.”

ME: “…”

Interviewer: “…”

ME: “I am finished.”

Interviewer (thinking to himself): “You sure are, buddy”

And don’t even get me started on the incredible answers I’d give when they asked me questions about design details:

Interviewer: “So what happens if your database fails?”

ME: “….Two databases????”

Interviewer: “What’s the difference between SQL and NoSQL?”

ME: “One has SQL and the other one has no SQL.”

My interviewer, watching me draw my entire architecture diagram consisting of a box labelled “Client” that points to another box labelled “Server”

The Task

The task ahead of me was pretty simple. Boot up the ol’ terminal and git gud at system design. Like with any other software engineering problem, I figured Google Search was my best friend here. And so, after wiping my tears and deleting my rejection email from my inbox, I type “System Design interview guide” into the box and hit enter.

Immediately I am greeted with a flood of information. Atomicity! Consistency! Isolation! Durability! Consistency Again, but not the same as the previous Consistency! Availability! Partition-Tolerance! I find myself watching a 2 hour Harvard lecture from 2012 about load balancers and PHP. I watch countless Youtubers explain and re-explain how Twitter caches news feeds. I get recommended Designing Data-Intensive Applications (or “The Piggy Book”, as I like to call it), so I purchase a physical copy on Amazon and excitedly crack it open, expecting to gorge myself on the delicious knowledge within.

The "Piggy Book” about Computers and Stuff

I promptly fall asleep after reading 5 pages of the preface. Don’t get me wrong, it’s got lots of great info. I'd highly recommend those who like getting really in depth with scalable software systems go pick it up. But as a non-book reading doofus with a TikTok attention span, I really struggled to keep myself focused.

Me reading “Chapter 1: Reliable, Scalable, and Maintainable Applications” for the seventeenth time.

By the end of an 8 hour System Design bender, I find myself filled to the brim with knowledge, like Spongebob with Fine Dining and Breathing. That night, instead of ranting to my partner at 3AM about Steve Kerr’s rotations on the Warriors, I rant about LFU distributed cache eviction policies. (Just use LRU dammit!).

The next morning I wake up and it’s all gone.

The Action

Okay, it’s not all gone — that was, admittedly, for dramatic effect. But I did find myself suffering from chronic you-use-it-or-you-lose-it syndrome. Since I wasn’t actively using the system design information, I found the knowledge gradually slip away as I started doing other things. It was almost as if the information was being evicted from my mind-cache, replaced by more recently used keys like Baldur’s Gate 3 builds or Steve Kerr’s underutilization of Jonathan Kuminga on the Warriors.

Unlike with Leetcode, system design resources don’t give you a nice way to engage with the material. Which is a big problem for me, since I like to get hands on with stuff. Really get in there and get dirty with it.

My Github, where I remake the same Todo-list app every time a new JS framework is released

System design prep resources are entirely passive. It’s all reading articles and watching videos. You’re just soaking in information, like my dad at the local 24-Hour Fitness jacuzzi.

And furthermore, there’s no tangible feeling of progression, no feeling of pride and accomplishment after studying system design. (Believe me, I worked at EA so I know all about Pride and Accomplishment). For example, one of the best feelings you can ever experience is when you finally solve a Leetcode problem all by yourself. No Neetcode video explanation. No defeated scrolling through the Discuss tab. You excitedly run your code, pass all the test cases, smaaash that green submit button, wait for that little spinner and boom! It beats 0.2% of all solutions. It's terrible, but it's progress.

So being the do-er I am, I get an idea. Why don’t I just build it?

The Result

January 2nd, 2024 — I’m at my parent’s house watching some 34 minute Youtube video about the Punic Wars. I absentmindedly go to Github, thinking about better times when I programmed stuff and got paid for it.

I pull up an old hackathon project from sophomore year of college: https://github.com/david8zhang/nodepad, an interactive study tool that automatically converts your class lecture notes into daily quizzes. I marvel at how big a piece of crap it is and how bad I used to be at writing code. I also think about the major roadblock I hit during the hackathon. What I was trying to do required a powerful natural language understanding model that was far beyond the capabilities of a 2nd year CS student fueled only by free pizza and dreams.

I suddenly realize that a certain powerful natural language understanding model has just recently exploded into the public zeitgeist. Moreover, it’s extremely accessible — used by 2nd year middle school students, and it now threatened to devour software, art, and human interaction altogether.

Existential fear aside, I decided to dig nodepad out from the catacombs of my Github and reawaken it from its near-decade long slumber. Though the code was completely dead and unusable, its spirit could live again. The automatic quiz-generation concept could finally realize its true potential now that integrating sophisticated NLU is as easy 1–2-GPT-Three (ok, that sounded better in my head). This time, I set my sights on quiz-ifying the mountain of system design material that I had tried to cram into my tiny little noodle.

Fast forward a couple weeks and I’ve put together https://systemdesigndaily.com/, an interactive system design study guide that uses AI-generated daily quizzes to help keep those concepts “in working memory”, to extend the caching metaphor.

For now, it’s effectively just a glorified outline of all the notes I took during my weeklong system design binge. But being the gamer that I am, I’m planning to extend it with all kinds of fun mechanics, e.g. a rank progression system akin to competitive online games, daily rewards and badges, and maybe even a global leaderboard. I also have more features on the horizon, such as free-response tests and a “back-of-the-napkin scale estimation” practice tool.

I would love feedback for https://systemdesigndaily.com/, so if you have any, please feel free to DM me on Twitter / X: https://twitter.com/de_zhango. Please roast me if the topic article content is incorrect. After all, it’s entirely based on my own research and understanding of the concepts, and, if you haven’t figured it out already, I am a bozo.