The word system design got pretty popular with software engineers ever since the big guns like Facebook, Google, Palantir added an essential system design round in their interview process. I mean that’s how I came across the term.
Now the obvious question that would pop up in our mind is why the knowledge of large-scale system design so important to these companies? I mean, if a candidate is a pro-coder, has the superpower of writing a O(1) complexity solution to any problem. Is it really necessary for him to have a good knowledge of system design too? Okay, O(1) was a bit of an exaggeration but I think you got the point. We rarely get to design large scale services from the bare bones in our software dev career. Isn’t this too much to expect?
Let’s talk.
Why You Should Master Designing Large Scale Systems?
First off, we will discuss why is system design important for software developers in general. Not for the interviews, not for getting hired. But for your knowledge and career growth or maybe you want to start your own thing, you have a cool idea and want to design a system that would scale to millions of users. How would you do it?
Would you hire an architect or would you design it yourself? Of course, design it yourself. And to design a scalable system wouldn’t the knowledge of web application/software architecture, picking the right technology for your use case come in handy?
Having the knowledge of large-scale system design, in other words, the knowledge of web architecture, software architecture, distributed systems, gives you an edge as a software developer.
In your job when implementing new features or when designing a new module from the bare bones or when trying to scale the platform to deal with the traffic influx, you can make better decisions by applying your system design knowledge & really stand out amongst your peers.
Let me walk you through the process of how software is typically developed from the bare bones & why application design knowledge is super important to the developers.
Significance Of System Design – The Importance Of Getting The Application Architecture Right
The architecture is the base of an application & has to be carefully thought, to avoid any major design changes & code refactoring at a later point in time.
Speaking with experience, you don’t want to delve into re-designing stuff. It eats up your time like a black hole. It has the potential to push your shipping date further down the calendar by months, if not longer. And I won’t even bring up the wastage of engineering & financial resources which is caused due to this.
It also depends on what stage of the development process we hit an impasse due to the hasty decisions taken during the initial design phases. So, before we even touch the code & get our hands dirty, we have to make the underlying architecture right.
A look at the architecture of our app should bring a smile to everyone’s face.
Though software development is an iterative and evolutionary process, we don’t always get things perfect at the first go. Still, this can’t be an excuse for not doing our homework.
An Overview Of The Software Development Process
In the industry, architects, developers and product owners spend a lot of time studying & discussing business requirements. In software engineering jargon, this is known as the Requirement Gathering & Analysis.
Once we are done with the business requirements, we sit down & brainstorm the use cases which we have to implement. This involves figuring out the corner cases as early as possible & fitting the Lego blocks together.
If you’re a fan of documentation, you might also want to write a high-level design document.
Now, we have an understanding of the business requirements, use cases, corner cases and all. It’s time to start the research on picking the right technology stack to implement the use cases.
Proof Of Concept
After we pick the fitting tech stack, we start writing a POC (Proof of Concept)
Why a POC?
A POC helps us get a closer, more hands-on view of the technology & the basic use case implementation. We get an insight into the pros and cons of the tech, performance or other technical limitations if any.
It helps with the learning curve if we’re working with completely new tech, also the non-technical people like the product owners, stakeholders have something concrete to play with & to base their further decisions on.
Now, this is only for an industry scale product. If you are a solo indie developer or a small group, you can always skip the POC part and start with the main code.
So, we showcase the POC to the stakeholders & if everyone is satisfied, we finally get down to creating the main repo & our first dev branch on GitHub, or any other similar code hosting service which the business prefers.
Phew!!
So, by now you would have realized how important it is to get the architecture right for the first time & the knowledge of web architecture to developers.
It shouldn’t be that only if you work for a really long while in the industry you should look into software architecture rather it should be the other way around. As you learn to code, you should keep acquiring the knowledge of architecture.
So, this was about having the knowledge of system design to write your own thing. Now let’s find out what companies are looking for in a candidate during their system design rounds.
System Design Interview Process
What Companies Are Looking For In A Candidate During A System Design Interview Round?
Companies don’t expect us to be an expert in system design. The design discussions are open-ended, there is no perfect answer to a solution, no silver bullet, there always trade-offs involved.
They want to gauge our thought process, they want to understand how we approach a design problem or do we immediately go into panic mode when given a problem that we’ve never seen before.
The problem that is given to us is intentionally underspecified. They want us to trigger a conversation and ask clarification questions on the problem like if there are any constraints, what kind of input the system has to handle, how much traffic should the system expect, traffic patterns and so on.
The interviewer as opposed to looking for a specific solution wants to see how we approach the problem as I already said. And as we proceed with designing a system, we end up having a pretty detailed discussion with the interviewer on many different topics like the scalability strategies, fitting technology, data handling strategies, design choices and so on. During the conversation, the interviewer tries to gauge the breadth and depth we have on the topic.
We need to take the ownership of the system design process, and the onus is on us to explain it to the interviewer the design choices we make, the database type we pick, if there is really a need for the message queue or any other component in the architecture and so on.
Companies want to take engineers on board who they can trust to come up with a good solution for a problem without having a lot of supervision. They want engineers who understand the use case and the requirements well and come up with a fitting solution as opposed to being just driven by the buzzwords & the hype in the industry.
Let’s hear what Google has to say about the system design interviews
Prepare For Your Google Interview – Systems Design
The best way to perform in these interviews is to relax and think of the whole round as a session where you are designing the system for your own startup. Bounce your thoughts off the interviewer. Lead the design process, having a discussion with him.
The knowledge of designing large scale distributed systems is one of the most important skills a software developer should have. I’ve developed numerous applications from the bare bones and the knowledge of the fundamentals has always helped me save time and a serious amount of money.
So, till this point, we talked about why you should master system design. Time to talk about how to master system design?
The Right Way To Educate Yourself On System Design
Most of the resources on the web, that I’ve come across, will tell you Oh!! You have an interview at X Company it will take you a couple of weeks to prepare for the system design interview. No, it won’t. I strongly disagree with that notion.
Designing large scale distributed systems is an art. You can’t learn this in a couple of weeks. Trying to learn everything in a couple of weeks is a sure shot way to fail your interview round. When the interviewer starts to dig deeper into specific components and design strategies, you’ll be standing there with a blank face. If you have a few years of experience in the industry, just the surface level info won’t help you nail that system design interview round.
Okay!! So, what do I do?
Learning software architecture takes time, you have to continually educate yourself over time or you’ll end up overwhelming yourself.
Here are a few good resources that I recommend:
InfoQ is an amazing website to follow high-level architectural stuff.
This Software Architecture subreddit is a good place to discuss software architecture.
You can read the engineering blogs of large-scale internet services like Reddit, Facebook and so on. Just Google “engineering blog of X service”
I write about real-life architectures on my blog. Here are a couple of popular real-life architecture posts written by me.
YouTube Database – How Does It Store So Many Videos Without Running Out Of Storage Space?
How Does Instagram Store & Search Billions of Images
You can join my newsletter to stay notified of the new content on the blog.
Here are two system design courses that I recommend.
Best System Design Courses
Speaking of two top-notch system design courses that’ll put you upto speed are:
Web Application & Software Architecture 101
This course is written by me, launched on Educative.io on December 2019 & within a very short span of time, it has become one of the best-selling courses on the platform. It’s a stepping stone to your system design interview preparation.
So, What Is The Course All About?
If you need a right from the basics, insight into web architecture. That would be a step by step walkthrough through different components and concepts involved when designing the architecture of a web application, right from the user interface, to the backend, including the message queues, databases, streaming media, picking the right technology stack & much more. You should take this course.
In this course, we’ll go through the techniques of picking the right architecture and the technology stack to implement our use case. I’ll walk you through different use cases which will help you gain an insight into what technology & architecture fits best for a certain use case when writing a web application. You’ll come to understand the technology trade-offs involved.
If you are a beginner just starting your career in software development, this course will help you a lot. Designing software is like fitting the lego blocks together. With this course, you’ll develop an insight into how to fit them together and build cool stuff. It will also help you with the software engineering interviews, especially for the full stack developer positions.
By the end of the course, you’ll have a comprehensive insight into the web application architecture. You’ll be all set on the path to designing applications from the bare bones. If you have a startup idea & you are asking yourself, how do I implement my app? What technologies do I use? Where do I start? This course will help you kickstart your entrepreneurial journey.
Click here to buy the course on Educative.io There is a discount of 40% on the course, right now, only for so long.
If You Are Wondering Who I AM, A Bit About Me:
I am Shivang. I’ve been writing code from the past 8 years professionally & 14 years personally. In my career, I’ve gotten the opportunity to work on large scale internet services for some of the industry giants on several different domains such as E-commerce, Fintech, Telecom and others.
I’ve written applications from the bare bones, right from the idea to production. I’ve maintained code, as well as worked in the production support for systems receiving millions of hits every single day.
My last job was at Hewlett Packard Enterprise as a Full-Stack developer in their Technical Solutions – R&D team.
Via this course, I’ve tried my best to share the knowledge, insights and the experience gained in my years of software development, with all of you guys!!.
You can read more about me here. Here is my LinkedIn profile.
There is also a detailed post on the course on my blog here. Just in case if you want to read it.
Next up…
Grokking The System Design Interview
Grokking the system design interview course deals with designing large scale services the way companies expect you to design large scale services like Twitter, Uber, Facebook. Instagram, Dropbox and a lot more. Collectively there are 15 case studies discussed in the course.
I highly recommend this course after you are done with the fundamentals or if you are a seasoned developer and are well aware of the fundamentals. You can directly take this course.
The course is developed by hiring managers who’ve been working at Google, Facebook, Microsoft and Amazon. The case studies that the course contains have been repeatedly asked at the top companies.
Click here to buy the course on Educative.io
Disclaimer: This is an affiliate link. When you buy the course I get a small cut without you paying anything extra.
Both the courses are available on Educative.io platform. Educative is a platform that offers interactive programming courses, with embedded coding environments, for software developers, to make learning quick and efficient. No setup required, developer environments are pre-configured in the cloud. You can learn at your own pace with in-browser exercises.
So, Fellas!! 🙂 we’ve reached the end of this long post. I’ll keep updating it as I find more interesting helpful resources on system design. You can bookmark this page & can also join my newsletter to stay notified of the new content published by me.
See you around…
Cheers!!
Top comments (1)
Thanks for your article, I found out lots of helpful information for me to thinking about system design with scalability.