DEV Community: Nilesh Mahajan

How to create a VPC, subnet, and networking setup in AWS using terraform?

Nilesh Mahajan — Wed, 26 May 2021 11:51:40 +0000

Overview

AWS is a very big and complex platform. Creating a SAAS on AWS can be a daunting task, especially if you are not familiar with AWS. There are so many configurations about VPC, subnets, security groups, etc to be created before you even begin provisioning your application/database servers. The UI and documentation can be very confusing.

So I prefer a script/code-based approach to creating infrastructure. Terraform is a popular choice to create your AWS infrastructure programmatically. This how-to guide helps you create a basic networking stack using terraform that you can use to create your SAAS.

What you will create?

If you are starting a new SAAS project and planning to run your infrastructure on AWS, then you are going to need

A VPC1 public/private subnet per availability zoneNetworking set up like a public internet gateway and a private NAT gatewaySecurity group to control ingress/egress traffic

The overall setup will look like below -

What you will need?

AWS Account with programmatic access Terraform (>0.12) installed on your operating system and binary is included in your system's PATH.

If you are using mac, you can get terraform using homebrew easily.

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/terraform

# To verify everything is right, check version 
$ terraform version
Terraform v0.15.1
on darwin_amd64

Continue reading this ...

The remaining steps to build this are available in a nicely written walkthrough here -

https://walkthrough.so/pblc/pHjklthzWFvI/how-to-create-a-vpc-subnet-and-networking-setup-in-aws-using-terraform

Write your own codelab style tutorials with Walkthrough.so

Nilesh Mahajan — Wed, 30 Dec 2020 09:50:55 +0000

In my previous work experience as a platform developer at Uber, I had a multi-faceted role. As an engineer, I was responsible for building and maintaining our technology platform responsible for Uber's trip flow. But this platform was extremely complex with 10s of microservices, mobile apps, data models, and 100s of APIs. So we also had to act like platform advocates/consultants and continuously engage with teams across the organization to help them be successful on our platform for their features and products.

We were a small team. Automating everything was not feasible. So to make customer adoption easier and faster, we wrote various forms of "how-to" guides explaining platform setup, migration, features, and capabilities. We had used all the different types of tools that were out there - Confluence/wiki, Phabricator, Google Docs, Google Slides, Markdowns, Pydocs, etc. But none of these tools worked as well as we expected.

A lot of our valuable engineering time was lost in this.

Challenges with tutorials

Tutorials created using the above tools are often ineffective from a reader's perspective because -

Procedural content like tutorials/guides with screenshots, code snippets tend to grow longer. For content longer than 2-3 pages, readers find it difficult to follow instructions and end up making mistakes.
Longer content increases perceived complexity. Readers would abandon the tutorials. (or worst, internal customers will ask for a meeting to understand better 🙄).
Tutorials can get buried deep into your documentation.
Wikis/docs/slides can often look very dull and have poor engagement. Readers lose interest in them quickly.

And from the writer's perspective,

To create a good tutorial, with annotated screenshots, code snippets, syntax highlighting, etc can take a lot of time. If readers drop out of the tutorials, it is a big wastage of the writer's efforts.
Content usually needs periodic updates and poor tooling creates resistance for the writer to keep content up to date.

This problem got stuck in my head for a while. While working on a hack project, I got introduced to - Codelabs. In Google's own definition -

Codelabs are interactive instructional tutorials

But compared to all previous solutions, Codelabs felt a much better solution.

Why codelab is a great format for tutorials?

Codelab is almost like a hybrid between a document and a presentation. Codelabs are a very effective format for writing anything procedural content - Tutorials, API Tours, Migration Guides, Onboarding, etc.

Google has been adding 10s of codelabs every day and experimenting with this format. Some interesting aspects of codelabs are -

It is a proven format of tutorials that have been effective whether it's a frontend or backend technology. Googlers have written codelabs on almost all of its technologies including Flutter, Android, Kotlin, TensorFlow, Google Cloud, etc.
Codelabs bring all of your content - documentation, code snippets, videos, etc in one place. This helps readers to connect the dots between all pieces of information. It breaks down the content into logically ordered steps. So it is better for comprehension and it builds muscle memory.
Codelabs give a time estimate for each step. This is a small but quite important thing. Readers often do not finish tutorials in one sitting. Having an estimate helps them plan their tutorials better.
Codelabs push the focus on interactivity and engagement in the forms of questions/forms and embedding different types of content like podcasts, videos, code editors, etc.
The UX is focused on distraction-free learning.

A lot of technical writers have shown interest in writing content in codelabs format. But writing codelabs has not been very straight forward. Googlers have open-sourced their tools used to generate codelabs. However, the tooling requires self-hosting, familiarity with Go and Javascript/CSS, and writing the content in markdown or Google Doc.

Introducing Walkthrough.so

To ease that pain and put this experience on steroids, I developed walkthrough.so

With Walkthrough, you get,

WYSIWYG Editor Using a nice minimalistic block style editor, it becomes very easy to create the commonly required content blocks like - alerts, images, code snippets, iframes, attachments, etc. This makes writing/modifying extremely easy as compared to any document or markdown.

Sharing, Hosting, and Discoverability
Walkthrough makes it easy to share and host such tutorials for your target audience - privately to internal customers or publically to anyone. Walkthrough takes care of putting your images in cloud storage, CDN in the front, and SEO.
Styling And Customizations
Everyone wants to showcase their brand and style of their content. We make it easy to make codelabs look styling and customize the looks.
Own Your Content
A walkthrough can export your tutorial into your Github repository in Pure JSON format for backup and versioning purposes.

Pricing

Walkthrough is entirely free for individuals to use. You have no reason to not give it a try :)

Examples

This is how a walkthrough I wrote recently,

Check out what walkthrough offers in detail - The walkthrough of the Walkthrough to learn what you can do with it.
More examples are available here - Walkthrogh Learning Hub

Why I left my cushy job to work for myself In the middle of a pandemic?

Nilesh Mahajan — Fri, 09 Oct 2020 18:29:00 +0000

In Aug 2020, I left my cushy job as a senior engineer at Uber and work for myself in the middle of a pandemic and a looming recession.

To be honest, this was not sudden. Becoming an entrepreneur was always a dream since 2011. I think this is because of how my beliefs got shaped up over the years. It probably had to do with my father being a serial entrepreneur + some of my side projects got moderate traction + me spending years in the silicon valley + me believing entrepreneurship is one of the best ways to learn about the world and contribute. But my finances were not good enough to give me ample time to start something of my own. In 2014, on the very first day to work for Uber, while walking from the SF Caltrain station 1455 Market St, I made a promise to myself - "This would be my last job before I do something of my own".

Fast forward to the next 5 years during which my professional and personal life made tremendous strides. I got married, traveled a lot with my wife, and relocated to India. On the professional side, I had a couple of promotions, a great rise in pay, and opportunities to build a new team. It was a quite satisfying job in most ways. The dream was still back of my mind but a great job and a peaceful personal life had dampened my enthusiasm. Almost 99% of colleagues and friends I was surrounded with had no such interests. The Tech sector is in such a boom that many top tech companies are literally showering engineers with stocks and money. Most of them were content with that. Breaking away from the herd felt scary. So I started having second thoughts on why should I put myself through such hardship when things are just so great. While I was going through this confusion, Corona was declared a pandemic, and the world turned upside down.

Ultimately, in Aug-2020, I quit my job. Here is what led me to my decision -

1. A Builder's Mind

Answers to many confusing questions often come from within. I started introspecting my life and tried remembering what made me happy and proud in the last 10 years. While I remembered my graduation, salary hikes, promotions, etc, I realized most of my sense of accomplishment came from my passion project in all these years. I have a builder's mind. All my side projects were for solving a real-world problem using technology. I thoroughly enjoy learning and building stuff. I did so many side projects on a personal as well as the professional side until the last few years. In fact, much of professional success also came when my side projects eventually became mainstream at work. On the personal side, my proudest moments had come from reading the reviews of 5000+ random strangers who loved my transliteration app. And from a small invoicing/inventory management app based on a tablet that has been helping my family operate our manufacturing business post my father's death in 2012. That impact made, no matter how small it was, felt extremely rewarding.

But as I grew senior in a company, my roles and responsibilities changed. I had to focus more on consultation, cross-time zone collaboration, processes, goal setting, consensus building, and decision making, etc. I always saw these as necessary evils against something you do out of passion. I was spending more time on these and less time building new stuff. Working on side projects had gotten harder lately because of a demanding job and the necessity to maintain a good work-health-family balance. With the addition of a newborn in early 2020, choices became clearer. I cannot spend my personal time after office work to work on things I am passionate about to keep myself satisfied. I should quit my day job and work on passion projects and find ways to get paid for them. But is it too ambitious to do?

2. Rational Risk Assessment

One of the interesting points, I learned from Naval's podcast is

Starting a company has asymmetric risk. i.e. there are huge upsides and limited downsides.

If you make sure that you maintain a healthy life balance and not squander money, there are literally no other risks. Building a SAAS company has become easier than ever. Capital requirements are quite low. Access to capital is becoming easier. Hardware costs are constantly going down. Software building just requires a few laptops and time. You can sell your products globally. You may not always succeed in the first shot, but if you give enough shots, the probability of finding gold is good. And even if you don't, there is so much demand for good engineers that you can always fall back. In fact, if you have learned some new skills and met similar minded people along the way, you may land even a better opportunity than before.

But is it too risky to do this with a pandemic and a looming recession? These are actually good times to try out. The world is going through a massive transformation and is scrambling to find solutions for the new problems that arose after the pandemic. A mindset shift has kicked in. New habits are getting formed. The world will not go back completely to where it was before the pandemic. If you can put together something, you may get an early movers' advantage.

3. Compounding Effects

Knowledge and relationships have compounding effects. I had earned more money in the last 1 year than the total of the first 5 years of my job. This is because I had grown an expert and built good relationships in what I was doing. So just like they say in investing, time in the market is more important than timing the market. If you aim to be X in life, then starting as early in that direction as possible is more important than waiting for a perfect time to make a move. There will never be a more perfect time than now. If you think, someday you will have a mindblowing idea that you love to build with and would pay great, and you will have a great friend or partner who will also be equally passionate to do it with you, it's almost like expecting all the stars in the universe to align for you.

It is not necessary to have a good idea, a solid plan, and partners even before you begin. Just make sure you have enough financial cover for a reasonable time, say a year. In that 1st year, all you have to do is figure out something pays for monthly expenses and buys you another year of doing this. I know this sounds simpler than it is. But the internet has tremendous power and we have many such examples out there. One thing that I related to most from the famous Steve Job Stanford speech is -

“You have to trust in something -- your gut, destiny, life, karma, whatever. Because believing that the dots will connect down the road will give you the confidence to follow your heart even when it leads you off the well-worn path, and that will make all the difference.”

With these thoughts, in July 2020, I quit my job. As I embark on this journey, I wish to meet new like-minded people, build new stuff with them, learn new skills, and enjoy new experiences. I am optimistic that I will find something I love to do and get paid for.

Writing these down, was important for me as a reminder every now and then of why I chose this path and share my journey with similar minded people. My heartfelt thanks to my family, friends, and colleagues who have been vehemently supportive of my decision.

To learn more about how this is going, follow me on twitter
https://twitter.com/nilesh_mahajan

6 Tips to Master Your Software Architecture Interviews

Nilesh Mahajan — Sat, 22 Aug 2020 03:43:54 +0000

In my last 10 years at Uber + eBay, I have taken on 100+ interviews with Candidates coming from top companies like FAANG to no-name startups. I myself have interviewed in 10s of great companies. I have interviewed in the Bay area as well as Bangalore. During all this time, I made tons of mistakes in my interviews and lost many interesting opportunities. Like everyone, I blamed myself, but not for the right reasons. After being on both sides of the table for years, I learned how this actually works. I wish I had known these things earlier. But it is never too late to share. So starting with this one, I plan to talk a lot about how tech interviewing works or doesn't ;)

Among all different types of interviews, the Architecture round is one of the most crucial rounds for Senior Software Engineers. Many engineers who are really well at coding and problem solving, make a few mistakes in the Architecture round and are then either rejected or down-leveled. In fact, in strong tech companies, even Senior Managers are expected to do well in this interview. While the expected level of depth may vary, the fundamentals expectations remain the same.

That is why, I intend to demystify the architecture round in this blog. If you are preparing for an interview at some of the top companies like Amazon, Apple, Google, Facebook, Uber, hottest startups or equivalent tech companies, read on.

Let's first try to understand this interview better. The fundamental question is - *What is the interviewer seeking by asking you to design your own twitter or URL shortener or Video Streaming Service? * Generally, many good interviewers tell their expectations upfront. You should really pay attention to that. But if I have to write it in a one-liner, the purpose of this round is -

To understand how the candidates break down a high-level system into its sub-components, identify the most appropriate technologies to build those components, keeping principles like separation of concern, efficiency, reliability, security, and scalability in mind.

But in order to give the interviewer answers to what he/she is seeking, you need to approach the Architecture problem in the right manner.
I have broken down this into 6 different areas to explain this.

1. Collaborate with Interviewer

Interviews are very much like role-play scenarios to mimic a real work environment. The interviewer just initiates the conversation by putting forth a problem statement. After that. you are expected to drive the discussion forward. But like a real work situation, the interviewer wants to collaborate with you to reach the final solution. As you go on discussing the problem, you should take the interviewer's response into consideration, propose choices, and make decisions that are correct and agreeable to both. This process is more like a conversation than QnA.

2. Ask Right Questions

Just like a real work environment, the problems are never crystal clear. After giving a high-level goal like say "design Netflix like system", the interviewer expects you to refine this problem. The refinements are in multiple dimensions. The candidate is expected to ask questions, state assumptions and bring clarity into -

Business goals - i.e. Define the primary entities and use cases of the system.
Engineering goals - i.e. Number of users, frequency of engagement, security, scale, performance, reliability, developer experience and responsiveness of the system

Spend a good 5-7 minutes to discuss these and understand what kind of system you are about to design. Instead of following this like a checklist, ask the questions that make the most sense for your problem statement. e.g. If you are designing a banking system, security may take the highest importance. So right from early design, keep the focus on these aspects.

3. Divide and Conquer

Now that you understand the problem better, the interviewer is expecting you to define sub-parts of your system. Remember the principles of good designs e.g. separate of concern, synchronous vs asynchronous flows, customer obsession, etc.

But in a real work environment, systems are designed iteratively. So, don't go crazy calling everything as a separate component. In most likely cases, either interviewer is not interested in all of them or not going to have enough time to go into all of them. So rather, define a few high-level components, define their responsibilities and then break them down as conversation starts going deeper into each of them.

The same goes for your data modeling/architecture. Define key attributes about each entity to store and relationship between them etc.

4. Choose the right technologies to build components

The next step after defining components is implementation choices. In order to implement any component right, one must understand how use cases drive the usage of each component. e.g. Is your system read-heavy or write or both? Is data freshness important in this search functionality? How many jobs are expected to be run on this scheduler? What is the criticality of this component? How many developers are going to write code into this framework? etc. So again, collaborate with the interviewer to be clear on goals and then choose the right technology to fill the gap.

Often, there are more than 1 technologies that can solve the same problem. So you should propose all options and discuss the trade-offs before you commit it as a solution. This gives the interviewer insight into the depth and breadth of understanding of the technologies you are proposing. Also, the interviewer expects you to know much deeper about the technologies you have used even in your past experiences. So go prepared for that :)

5. Think of points of failures

For all the core flows of each component, you should talk about how failures and hotspots will be handled. Being proactive to call these out matters a lot. Also. be specific about things that should be done instead of just throwing buzzwords e.g. X system is at risk if this Y service goes down and hence we should have a circuit breaker here. Or this image cache should have an LRU eviction policy.

Most outages in companies happen because of human mistakes. Great senior engineers not only talk about the technical failures but also talk about what best engineering practices to be followed to make the system more resilient. e.g. keeping a free form JSON data in a NoSQL database can lead to abuse and mistakes. So there should be schema management with it. Best practices could be about configs management, testing, environment management, monitoring, and logging. You should proactively talk about these.

6. Manage time efficiently

This interviewer is going to be your representative in the final debrief. Do not beat around the bushes and waste time into trivial conversations e.g defining 10s of attributes for a User entity. Defining the key ones like user_id, email, password, and a few secondary ones are enough. While covering all the aspects of the system design in a given time can be challenging. But you want to make sure that a moderate level of depth and breadth of the system has to be discussed for the interviewer to feel confident.

That's it! If you have made it this far reading into the blog, you can see how Architecture interviews go both - broad and deep. Mastering this interview is hard, but make sure you do enough practice with your peers, senior colleagues. Nail the fundamentals and all the best!

Thanks for reading. If you find want to talk more on this subject, you can reach out to me on Twitter

Hello World!

Nilesh Mahajan — Mon, 29 Jun 2020 17:22:55 +0000

Hoping to use this instead of Medium!