DEV Community: Rahul

Never assume that patch updates are always non-breaking!

Rahul — Wed, 14 Apr 2021 13:45:48 +0000

We learnt it the hard way. 👇

Last Friday afternoon, we bumped the Rails version from 5.2.3 to 5.2.5. We had to do this because one of the dependent gems of rails had been yanked very recently and rails quickly released a patch update to address this issue.

We meticulously went through the changelog because we didn't want this to break our system especially since it was on a Friday afternoon. (We have a history of Friday deployments causing outages :D )

Changelog said, good to go.

We went ahead and prepared for the release.

Things looked alright on our staging(test) environment and we deployed it to prod. Our production deployment pipeline failed. (Eyebrows were raised at this moment)

Because until that point, deployments were going through without any hassle as Docker had been using the cached version of the yanked gem. Since there's a failure now, the cache is gone and we can no longer push any further deployments without fixing this. 🤦‍♂️

So how did it pass the sanity on our staging?

Funnily, the release also had a log to one of our background processes and I just checked if the latest code was there on the new pod. But what I didn't notice was that the application pods were crashing. Sanity was done on the previous release 🤨

I tried making some change to the buggy release and pushed it to the stage and we found the issue on staging now.

Application pods weren't getting up because the health checks were failing. (We did not have any health checks for our background job pods)

So what's the big deal?

No further deployments can be pushed to production
Our staging was down.

Shoot! Almost everyone was blocked in one way or the other.

At first, I thought this was an infra issue. But soon I realised, the health check request wasn't even going through. The API was broken. 🤯

We were also using a gem called grape for APIs and health checks were going through that API.
Yes, Grape broke!

Wait, a patch update of rails that had almost nothing in the changelog broke grape? YES!!!

So who's the culprit?

Rack - it was bumped from 2.0.7 -> 2.2.3 (We missed this as there were a lot of dependent gems that got updated)

Rack is the middleware that forwards the requests to either grape or rails API. The response that it sends over had been changed (god knows why) and grape wasn't yet ready for this. The cascading effect was that all the grape APIs were failing including the health checks and our system was down.

We now had no other option but to update grape to the latest version and hope that it fixes this issue!
Thankfully, it fixed the issue.

What other option did we have?

Had it not fixed the issue, we would have been forced to move all the APIs out of grape to rails API.
Just the thought of this made me claustrophobic because that would not only ruin my Friday night but also would have consumed my weekend!

Lucky escape indeed!

Though it was pretty scary when it happened, I would take this learning on any day.

Lessons learnt ✅ fortunately without any major outage. 🤞

PS: This post was originally tweeted as a thread here.

Understanding Database Connection Pool

Rahul — Wed, 31 Mar 2021 13:31:03 +0000

Recently we faced an issue on our production where the CPU of our Postgres instance was hitting 99%. It was struggling to serve requests and as a result, our APIs were failing.

To start with, the issue happened in one of our Golang based microservice. This is a very thin legacy microservice that serves two lightweight APIs, so this service was never touched often and nobody cared.

1674 active connections were ridiculously high for this service.

Looking at this image, it was obvious that the number of active database connections was constantly increasing and a quick look at the code told us, we were not making use of the connection pool.

Database connection is costly

When the application needs to query something, it creates a new connection to the remote database and uses it to query and then discards the connection.

Creating a new connection is costly because it involves the following: -
the application has to read the database connection string, send the TCP/IP call to the DB instance, the database now has to authenticate/authorize the request and then let the application establish the new connection. The application can now make the actual query with the new connection.

Consider this happening for every time the application needs a DB connection, there's a waste of time and resources and this is why creating a new connection is termed costly.

And this is the problem that the connection pool solves.

What is a connection pool?

As the name indicates, the connection pool creates a set of connections and caches them. These cached connections are then given to the application on demand and can be used to perform database operations. Once the application gets its job done with the connection, it returns the connection to the pool and the connections can be reused thereby eliminating the creation of new connections. Ultimately it improves the performance of your application.

There are three connection pool variables that the golang SQL driver supports

MaxOpenConns

This specifies the maximum number of open connections that can be made to the database. In our case, since it was not defined, by default it allows an unlimited number of connections. This is the reason why the DB connections were ever-increasing.

Consider that the MaxOpenConns variable is set at 5, now if all the connections are being used and the application needs another connection, it has to wait till one of those 5 connections is returned to the connection pool by the application.

You have to estimate two things before setting this value, one - the number of connections your DB can handle, two - the number of parallel workers you expect your application to work with.

We have set the `MaxOpenConns` at 30 for our instances.

MaxIdleConns

This specifies the number of connections that can be idle in the connection pool (not used by the application). By default, the number is 2.

Consider that the MaxOpenConns variable is set at 30 and MaxIdleConns is set at 10. If the application is currently using only 12 of those connections, then the number of idle connections will be 18. Since the allowed MaxIdleConns is 10, the rest 8 connections will be returned to the database.

If your service can get a sudden increase in traffic every now and then, you should consider having a higher value for MaxIdleConns so that when there's a spike in traffic, not a lot of new connections are created. So based on your load pattern, you can set the number of idle connections.

We have set the `MaxIdleConns` at 20 for our instances.

ConnMaxLifetime

This specifies the validity of the connections in the pool. Once expired, the connections have to be reestablished. By default, the validity is forever. Unless you have your database behind a load balancer and swap databases, this value can be set quite high.

We have set the `ConnMaxLifetime` at 30 minutes for our instances.

After setting these variables, the number of connections came down to 180 (we have 6 pods) but the CPU was still at 99%. Turned out that the issue was because of some queries consuming more time than they should. Had to index a column as the data grew enormously over time which resulted in queries taking forever to complete.

The Lesson: Always write code that works at scale!

Taking baby steps to protect privacy!

Rahul — Fri, 27 Nov 2020 12:46:07 +0000

You would have already read a number of articles related to privacy and said this to yourself, "I don't have anything to hide. Why should I be concerned?". This argument is totally wrong. Allow me to explain.

Look at these for instance.

You google search to explore something like, "Which one should you buy, MacBook Air vs Pro?", the next day you would have received an email digest from Quora,

Still curious about MacBook Air vs MacBook Pro

You want to check out a product that your friend has bought, you explore it.

You start getting ads for that product from different e-commerce platforms for a lesser price. (No, I don't want to buy that thing.)

There are a lot of instances like these where a third-party slyly sends you something based on your usage and history. This is because of cross-site trackers that track and monitor you all the time.
You could still feel this is really helpful to you and there's no harm in someone capturing all the data.

Let me explain in a way that makes you feel it's wrong.

You talk with your friend and someone is listening to all your talk. Will you be okay with this?
You have someone observing and taking notes on you all the time from morning till you hit the bed on things like, what you like, what you don't like, whom do you talk to often, which app do you use frequently, what are your wants and many more.

Now, you would definitely want to run away from this person even though he assures you that he is only going to help you with these data. Right? This is what is happening currently when you do anything online. Every activity of yours gets recorded.

Data is Power

You have given them (read Google) the unsupervised access to your everyday activities, usage pattern, your preferences, your photos, your locations, and everything and anything that can be tracked online. The amount of data that they possess on you is huge. Even you wouldn't be sure of all the insights that they get from your data.

If you're still not convinced about protecting your privacy, read these posts, this and this. This will help you get started.

Okay, But why now?

Recently Netflix released a docudrama Social Dilemma that explains a lot of essential points like the impact of social media on mental health, how social media tries to exploit the users and manipulates them, how it segments each profile and sends addictive content one after the other to keep the user engaged for a long time, how it fine-tunes the notifications to bring the user back to the app and a lot more.

The one thing that made me sad after watching it is that the number of actionable items is very less. After explaining all the things, the show just tells you to Turn off Notifications and nothing else.

I decided to explore more on this and take baby steps in protecting my privacy.
I wanted to take one step at a time.

Chrome - the browser

Moving away from Chrome has forever been on my todo list. Having been used to several extensions, I didn't want to leave the ecosystem even with a lot of complaints on Chrome's battery consumption and higher RAM consumption.

What are the alternatives?

There are a lot of alternatives but I chose Brave because then I wouldn't miss the ecosystem. Brave like Chrome is also a Chromium-based web browser, so all the extensions would work here as well. Apparently, it has also received a lot of recommendations because it blocks ads and website trackers by default.

The migration was pretty smooth. Loving it so far.

Google as a search engine

Hands down, Google is the undisputed winner here. It is the best search engine we have ever had. But the fear with Google as the search engine is due to the amount of data it keeps collecting. If you haven't noticed it yet, spend some time here. It shows your entire online history. Scary, ain’t it?

What are the alternatives?

Duckduckgo and Qwant were my options as they don't profile users and respect user's privacy with the downside of not getting personalised results.

I tried them both but came back to Google instantly. Google has made giant leaps in terms of user experience, showing better results, showing the results in the search page itself for a lot of queries.

For instance, if you search for EPL table it would show the table in the search page itself. Being used to this, it is unthinkable to switch to other search engines that do not provide this luxury.

So, let's accept the fact that moving away from Google as the search engine is difficult and not everyone would do it. If you prefer to continue with Google, this is the least that you can do to protect your privacy.

Delete your old web activity.
Turn off Web and App Activity.

What did I do?

Having said these, I would still want to get the best results when I work, I didn't want to end up spending more time just because I moved to a different search engine. So I chose to continue with Google as my search engine but with a small change.

As of today, I have two combinations of browser and search engine.

Brave + Duckduckgo:

This takes care of all personal usage. Logged in with my personal mail and all personal stuff goes here. For eg: I am writing this post on Brave + Duckduckgo.

Chrome + Google Search Engine:

This is for all dev tasks and professional usage. I would also do the single page results here. I don't care even if I get tracked here because it would never affect my personal usage. I have used my office mail id to log in here but you can choose to even use a secondary/dummy mail for this. I have also installed the Privacy Badger extension that blocks the trackers.

Both these browsers are active always and I choose one of them based on the usage type. It was hard to do these switches at first, but having used this for over a month now, the muscle memory has started to kick in and the switch happens spontaneously.

If you are a strong consumer of youtube as me, you might feel regretful to turn off your activity as it affects the personalisation. The workaround I have done is to create a secondary google account for this. IMHO, it just took a week or two to get a similar kind of feed in this new account too.

I don't really know how effective the steps that I have taken are gonna be. But, I am happy that I have done something to protect my privacy.

* This is my first step. *

Check out these actionables in case you're interested. There is so much that we can do to take back our lost privacy!

Thoughts on Google Photos discontinuing free unlimited storage

Rahul — Thu, 12 Nov 2020 18:59:52 +0000

If you have missed the news, here's a quick overview

Currently, Google Photos allows storage of unlimited high-quality photos and videos on your cloud.
Only original quality media counts toward the 15GB free storage.
Beginning June 2021, high-quality photos and videos too will start counting towards your storage quota.
Media uploaded till June 2021 will remain available for free.
Pixel users will not be affected by this.

We knew this was coming, primarily because of the volume of photos and videos that get uploaded daily. As Google claims, 28 billion photos and videos are uploaded every week. Google had to decide on this, the question was just when.

Google Photos wasn't a charity, it had its purpose

IMO, the purpose of google photos in being free was to train its AI and computer vision based on the real-world photos and videos and Google Photos is their most dependable source for these datasets. If you have keenly observed, Google Photos as a product has had significant improvements over the years. It's been 5 years, even if the models aren't perfect yet, they already have ample data to train them better.

This reminds me of a famous quote,

If you're not paying for it, you're the product

Now that we aren't the product anymore, the focus shifts to the actual product Google one

Promoting Google One

Google one - the subscription service was waiting for this move. Over these years, a substantial number of people would have been so used to the ecosystem that they would rather subscribe for the service instead of finding an alternative and migrate everything. This is a potential business model. Providing 100GB for less than 2$ is a pretty neat deal too especially with google's family sharing features. Most of the users would not even have a second thought over subscribing it right away.

Yet another typical product

In hindsight, Google could have easily done this years ago but as usual, it went with that typical product philosophy, "First, get the user to the ecosystem, iterate and provide quality service, keep them happy, make it difficult for the user to leave the ecosystem, now charge for the service". I feel Google Photos has passed the test with flying colours. Now is the time for Google Photos to reap the rewards for all the hard work.

Btw, ICYMI Amazon Photos provides unlimited storage for its prime users. Heard that somewhere before?😉

I'd be expecting your views on this too.

Understanding Offset vs Cursor based pagination

Rahul — Sat, 31 Oct 2020 18:27:01 +0000

What is Pagination?

Pagination comes into the picture when you have a large dataset to be sent to the user and you choose to send it in chunks instead of sending all of it in a single response. It is pretty common that you would find its implementation in most of the sites that you visit on a daily basis.

There are two types of paginations that gets implemented.

Offset based pagination

Let's take dev.to for our 1st type example.

When you open dev.to, in the feed, you see a list of posts sorted by some filter. This is the request that fetches the posts,



https://dev.to/search/feed_content?per_page=15&page=1&sort_by=hotness_score&sort_direction=desc&approved=&class_name=Article

There aren't page numbers on dev.to similar to the ones you find on Amazon product listing page or a google search page.

The reason is that dev.to has implemented an infinite scrolling wherein the next set of items are fetched in the background periodically to give the user a seamless experience.

When you scroll the feed continuously, these are the requests that are sent in the background.

Only two parameters change for every request in this type of pagination

per_page - number of items required per page
page - current page number

In the backend, the query would look something similar to this, (made it simple to understand the query better)



 select * from posts order by rank limit 15 offset 0;

This is for the first page.
For any page, the variable that changes is the offset.

offset = current_page_number * number of items per page

The query for different pages would look like this,



 select * from posts order by rank limit 15 offset 750; #page50
 select * from posts order by rank limit 15 offset 775; #page51
 select * from posts order by rank limit 15 offset 1500; #page100

There are a few important caveats of using this approach,

Performance Hit:

Before the explanation, hit these APIs on the browser directly and observe the time taken for each.

Request for page 1
https://dev.to/search/feed_content?per_page=15&page=1&sort_by=hotness_score&sort_direction=desc&approved=&class_name=Article

Request for page 650
https://dev.to/search/feed_content?per_page=15&page=650&sort_by=hotness_score&sort_direction=desc&approved=&class_name=Article

You should feel a slight delay in the second response.
Now you would also get an idea why some of your pagination APIs are taking more time as the page number increases.

Explanation:

If you look at the offset for 650th page, it is 9750.
What this essentially means is that to pick 15 items for the 650th page, MySQL has to pick 9750 records from the table and discard the rest of 9735 records one by one.

Imagine a use case where the offset is even higher, let's say 100,000. In this case, MySQL has to discard 99975 records to pick just 15 records and this happens on each request.

There has to be an efficient way to pick these 15 records right?

Blind Pagination

Since it relies on the limit and offset, this has a chance of showing duplicate data and skipping some data.

Skipping some data

Let's say the user has seen the response of first two pages(30 posts), now consider I delete my post that was earlier on the first page.

Now when the posts are fetched for the third page, the user wouldn't notice any change but behind the scenes,



select * from posts order by rank limit 15 offset 30;

The post which was at 31st position would have now gone to the 30th position in the table since I deleted my post which was at 7th position.

As a result, the user would never be able to see this post unless he refreshes the page.

Duplicate data

The same applies when a post jumps from 31st position to 30th position and now the 30th position post which the user has seen already will be shown again at the 31st position.

Having said these, offset based pagination is easy to implement and gives the flexibility to the user to jump to any specific page.

If you're using Ruby, there are a lot of gems that help you do offset based pagination within a few minutes. Here a few kaminari, will_paginate,pagy.

Cursor based pagination

Let's take Twitter feed for this example.



https://twitter.com/i/api/2/timeline/home.json?&cursor=HBbM%2Fv%2FnttbR2iQAAA%3D%3D&lca=true&ext=mediaStats%2ChighlightedLabel&pc=0

The cursor is the key parameter in this approach. The client receives a variable called cursor with the response. The cursor is a pointer to a specific item which is to be sent with the following request. The server uses the cursor to fetch the next set of items.

Here's an example of the response.



{
    "posts": [...],
    "next_cursor": "123456",  # the post id of the next item
}

For example, this cursor could be the id of the first element in the next dataset. The simplified query,



select * from posts where id >= 123456 limit 15;

Furthermore, if the visited page is the last page, then the next_cursor will be empty.



{
    "posts": [...],
    "next_cursor": ""
}

The advantage of this approach is that MySQL picks only the required records (15 records) from the database.

Performance-wise, this approach gives the result in a constant time O(limit) irrespective of the page number.

Consider for the 10 millionth request

offset based approach - O(limit+offset) = O(15+10M) MySQL picks 10M records from the database and discards nearly everything.
cursor based approach - O(limit) = O(15) MySQL picks only 15 records

Look at the performance uplift. Cursor based approach is mighty impressive but there are a few things that stop developers from using it without a second thought.

It is not possible to jump to a random page in this approach as the server relies on the cursor to fetch the records.

This approach is not straightforward like offset and is trickier to implement sometimes.

The cursor must be based on a unique or sequential column in the table and pagination goes for a toss if the edge cases aren't tested.

There aren't a lot of gems/libraries that support the cursor-based approach compared to the offset based approach. So development time increases as well.

pagy-cursor, paging-cursor support cursor based pagination.

Cursor based pagination is an outright better option over offset based pagination in terms of performance but the decision to choose the approach depends on the use case and the kind of impact the pagination is going to have on the product itself. For much simpler paginations with comparatively less dataset, one might still prefer offset based pagination.

What is more important is to, not choose a particular pagination blindly. Instead, understand the trade-offs between the two and choose the one that suits the most.

Don't be proud of pulling off an all-nighter

Rahul — Sat, 24 Oct 2020 18:22:38 +0000

When I decided to start writing, I consciously told myself to write on other topics too, apart from code. Like things that I wish someone told me when I was a fresher.

Sleep is one of them.

I have done so many all-nighters, during my college days and during the early years at my job. I used to feel so proud for pulling off those for some unknown reasons. Maybe I felt like being a * techie * requires you to do all-nighters. Maybe to show off your friends that you could do all-nighters. Interestingly, I did not even have comp-offs for those all-nighters yet I impulsively did those.

In hindsight, I regret the decision, pulling off all-nighters isn't good for you after all. Depriving oneself of sleep is the worst thing that mankind could do to itself. Today I would prioritise a better sleep over an all-nighter any day.

Here are some of my observations after quite a few all-nighters.

Decision Making/ Quality of Code:

Decision making is poor during all-nighters. Many a time, I had to redesign database design/code structures that I did during the all-nighter. I have found myself unanswerable to a few decision making questions that I have the following day or a few days later.

To overcome this, I decided that all the decision making has to be completed beforehand so that I could just code during the all-nighter. But the quality of the code that you write during an all-nighter is never the best code that you could write. More often than not, the bugs that I find in my code would have been written during an all-nighter.

How productive are you the day after you pull off an all-nighter?

For me, the next day has always been a zero productivity day. I have looked back at those days, thinking I could have completed the same job without doing an all-nighter.

Apart from these productivity issues, sleep deprivation is bad for your health which is the primary reason to not do an all-nighter ever.

Sleep Deprivation is injurious to health:

There are a lot of bad effects of sleep deprivation. It drains your mental abilities and puts your physical health at risk. Scientifically sleep deprivation has been the cause of many problems, from weight gain to a weak immune system, from heart attacks to memory issues and lowered thinking ability.

You will never know the effects when you're young because you're full of energy and your thought process goes something like this,

To compensate for the lost sleep during the weekday,
I will sleep for more hours during the weekend.

This is wrong on so many levels.

Sleep once lost, is lost forever. It is not like a bank where you put more sleep to compensate for the loss. It doesn't work like that. Moreover, oversleeping will only make you unproductive that day as well.

Regularity is the King:

Always remember this and make sure you have a good sleep every single day. The number of hours required for good sleep is subjective. It varies from 6-9 hours depending on the individual. So find what's comfortable for you and hit that number of hours every single day.

The regularity of sleep is the secret behind good mental health and being productive during the day. Try to regularize your sleep time by sleeping at the same time. For instance, from 10:30 pm to 6:30 am every day. Over time, your mind respects sleep time and falls asleep effortlessly.

Sleep is vital for the human body to function properly, akin to how we eat, how we breathe. Never take it for granted.

The hard part isn't getting your body in shape.
The hard part is getting your mind in shape.

Sleep has been the underrated reason behind people not in a good mental space.

Respect sleep time:

The reason for us staying up late is mostly because of the fact that you're not able to get up early. But the focus should not be to get up early, it has to be * to sleep on time *. Once you regularise your sleep time and sleep on time, you will see yourself get up on time as well.

As a practice, you should always have alarms, not to wake you up early but to * remind you to sleep on time *. If you ask all those people who start their day so early, the reason for them being consistent is because they sleep on time consistently no matter what.

Consistency means sleeping on time every single day, not just on weekdays and being a night owl during the weekends. Make it a habit, keep an alarm 5-10 minutes before your bedtime and turn off everything and hit the bed at that time. This has to be your P0 task.

To sleep faster/better, avoid using mobile/laptop at least 15-30 minutes before your sleep time.

Instead, try to

read books,
spend time with your family,
walk around,
meditate,
retrospect your day,
jot down tasks for next day,
play musical instruments

or anything that calms your mind. A healthy mind is one that has a healthy sleep.

** Again, never lose your sleep for anything. **

A silly bug that sneaked into production

Rahul — Fri, 16 Oct 2020 11:15:55 +0000

So, this is what happened.

I was in the middle of some task and suddenly had to make a context switch to a different task that I had pushed earlier. I forgot to handle a failure case and the code went to production already. But the change was pretty simple.

I had to add a condition to some existing code that looked similar to this,

Consider this is the condition that I had to add.

With this done, I asked myself if I should test this out before pushing and the genius in me replied that this was a simple change and I could go ahead without any testing. I also had this pre-commit hook to save me from any syntax errors. So, I felt I was pretty much covered and good to go.

I pushed it, the pre-commit hook passed, shipped to prod as well 🚀 and I went back to my other task.

Within a few minutes, our error monitoring tool started showing even more errors, this time 100% of the requests were failing compared to the negligible 1% requests that were failing earlier.

Guessed what could have gone wrong already? Kudos to you.👏

If not, here's the error that will help you find it.

TypeError (class or module required)

If you think that MyObject is not a class or module, you're wrong. Because that piece of code was already in production and is syntactical fine.

So what could have gone wrong? Give it a moment to look into it again.

Okay, here it goes, I didn't tell ruby explicitly about the execution precedence and as a result, it considered MyObject && (self.some_validation? || self.other_validation?) as the argument to the method self.is_a?.

So the result of MyObject && (self.some_validation? || self.other_validation?)? is either one of these.

MyObject && true #true
MyObject && false #false

As a result, the argument to self.is_a? is now a boolean,

self.is_a? true

which throws the error, * TypeError (class or module required) *

The fix is pretty simple though, just add a bracket to let ruby know the precedence of the execution.

Days like these are bound to happen and it just makes you more humble and makes you learn the nuances of the language.

Automating my zoom stand-up meeting

Rahul — Sat, 10 Oct 2020 08:32:19 +0000

WFH has become the new normal for all of us and so are the daily stand-up meetings.

As standups happen at the same time on all working days, the need to find the zoom link every day and open it became a boring task for me. More than that, I also had to keep an eye on the clock so that I don't make my team wait.

Being a lazy guy, it didn't like doing this daily. I just wanted it to be automated once for all. 🤷🏻‍♂️

Cron Jobs 😎

Cron jobs are specifically designed to run a job at a particular time. For example, you can configure a job to run every hour or every 10th minute or every Tuesday and more.

Now that you know what cron jobs are, there's a software utility crontab available in Unix operating systems. It is the daemon that monitors the list of scheduled jobs and runs it at the specified time. Jobs can be configured by firing,

crontab -e

The config is a one-liner with two values,

The time to run
An executable script that has to be run at that time.

Each line in this file represents a job and there can be n number of jobs.

The job that I had to do was straightforward. A simple script that can open the zoom app and join my standup meeting.
Here's the code for doing that.

#file_name: join_zoom_standup.sh
open 'zoommtg://us04web.zoom.us/join?action=join&confno=<conf_id>&pwd=<pwd>'

Make sure the script is executable.

chmod +x join_zoom_standup.sh

The only thing left is to configure the cron job to run from Monday to Friday at 10.30 am. (If you're not comfortable with cron expressions, head over here, a neat editor that helps you generate the expressions.)

Finally, this is the configuration for the same.

30 10 * * 1-5 ./custom_jobs/join_zoom_standup.sh

All set.

From now, you can just do your work and all of a sudden, the zoom app pops up, you hear people talking and realise it's time for the standup call.😉
This is worth spending 5 minutes. Isn't it!

You can view the list of scheduled jobs using crontab -l

Apart from this, there are a lot of tasks that you can force yourself to do just by automating it. I have done a few of them.

30 22 * * Wed ./custom_jobs/reading_list.sh
15 15 * * * ./custom_jobs/open_duo.sh
30 15 * * * ./custom_jobs/da_chore.sh
45 10 * * Fri ./custom_jobs/jira_board.sh
35 10 * * Mon ./custom_jobs/dev_scripts.sh

Quick info on what these are,

Open my reading list from multiple places. Eg: twitter bookmarks page, onetab page, dev.to reading list page and more.
Making sure that I have updated my JIRA board every Friday.
Booting all applications that I work with, running all the dev scripts on Monday mornings. Eg: Starting dev servers, Starting Docker containers and more.

Being developers, we should always try automating the tasks that can be automated and focus on tasks that demand our focus.💪

** Happy Automating! **

Why do microservices need an API Gateway?

Rahul — Fri, 02 Oct 2020 13:22:39 +0000

API Gateways are becoming increasingly popular with the microservice architecture. Recently, Google announced its own api-gateway. The time is ripe to take a look at why microservice architecture needs them and how they currently look without the api gateway in place.

Let's look at a microservice architecture without an api gateway.

Each microservice apart from its core functionality, traditionally have been doing these as well,

Authentication of requests based on OAuth or JWT token or a simple key-based authentication. (This authentication is to verify if other services or users have access to this service, not the typical user authentication that happens based on the application data.)
Allow CORS requests from other microservices
Allow/Deny requests based on their IPS.
Rate Limiting - Allow only a certain number of requests. Requests over the specified limit will be responded with a status code 429 (Too many requests). These rules also protect the microservice from DDOS attacks that happen at the application layer.
Monitoring - Collecting metrics from the requests/response to gain valuable insights. Eg: number of requests per minute, number of requests per API, number of requests a particular user has hit above the rate limit.
Alerting - This is a subset of monitoring, where alerts are generated for specific events. Eg: Generating an alert when the response time is over 500ms for 1000 requests. Using an observability tool like Prometheus helps in both monitoring and alerting.
Logging - Logging all the requests that are made to the server.
Request Termination - Temporarily disable the request for some APIs or disable the service during downtime.

Alright, these are a few of them. There might be even more to it as well.

Major Disadvantages of this approach:

When a new microservice comes up, all these functionalities need to be replicated.
Any change to one functionality should be repeated across all services. Eg: Moving the logging of requests from Loggly to StatsD.
Logically looking, all these functionalities are not specific to the underlying application. These can be decoupled from the application itself.

API Gateway:

API Gateway doesn't need any introduction now. An API gateway can be considered as yet another microservice in your architecture that does all the aforementioned functionalities.

It is the entry point for your microservices and acts as a gatekeeper doing all the basic functionalities before passing the request to the respective microservice.
All the functionalities now reside at a centralised place, making it easy to maintain and analyse them.
When a new microservice comes up, all it has to do is to process the requests and send the response back to the gateway. API gateway takes care of the rest.
With API gateway in place, functionalities like Request/Response Transformation, rolling out canary deployments become possible as well.

I have jotted down some of the problems that an API gateway solves. Having said these, it's really up to the architecture to decide if an API gateway is a must to have or good to have. They are a must to have especially when there are a lot of microservices in the architecture.

Irrespective of whether there's an absolute need for an API gateway or not, just by looking closely at the design before the existence of API gateway, it's evident that it was violating Single Responsibility Principle, Don't repeat yourself and High Cohesion and low coupling.

CORS & Preflight Request!

Rahul — Fri, 25 Sep 2020 11:28:44 +0000

What is CORS?

CORS - Cross-Origin Resource Sharing
In simple terms, when you want to allow requests from a different domain (read origin) to your server, CORS comes into the picture. CORS is a policy that is enforced by the browser.

But what is Cross-Origin? 🤔

Let's say you're reading this post on Dev.to. Dev.to is the origin here and it's allowed to request for resources (make https calls) that are present in its origin only. If it's making calls to any other origin, even to its sub-domain, the request will be termed cross-origin request.

Show me a use case!

In the world of microservices, even within your architecture, you might have different services talking to multiple servers. CORS is a mechanism to let only the trusted origins make the Cross-Origin HTTP request to your server.

Consider this naive example where there's an application running at rahul.dev.to and there's a functionality to edit my posts. Once the post is edited, I have to update the post across all my blogging sites - dev.to, medium.com, blogger.com

For this hypothetical case to work, I would need to hit this patch API on dev.to.

PATCH https://dev.to/rahul_ramfort/post/20
Request-Headers: User-Agent, api_key

(for brevity, ignoring medium and blogger API calls)

I am trying to post the data from my server (rahul.dev.to) to another server (dev.to) and I might or might not be allowed to actually make this request on dev.to.

This is the problem at hand. Browsers do not know if it's safe to make this request.

Enter Preflight Requests! ✈️

To solve this, Browsers for security reasons, do not directly allow this cross-origin requests to go through. Before firing the actual patch request, it instead fires an OPTIONS request to the cross-origin (dev.to) with all the details of the CORS request.

The details include:

Origin of the requested server - rahul.dev.to
Method rahul.dev.to is trying to hit - PATCH
Headers rahul.dev.to is trying to send - User-Agent, api_key

Dev.to, the cross-origin receives the OPTIONS request and can deny or allow the origin (rahul.dev.to) to make requests.

In this case, dev.to would have configured a list of trusted origins that can make the CORS requests at its application layer.

#sample configuration at the nginx layer (dev.to)
'Access-Control-Allow-Origin' "https://api.dev.to, https://rahul.dev.to"
'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, DELETE, PUT, PATCH'
'Access-Control-Allow-Headers' 'User-Agent,Content-Type,access-key,api-key'
'Access-Control-Max-Age' 86400

If rahul.dev.to is listed as one of the trusted origins, the browser receives a successful 204. Now the browser understands that it is safe to allow the CORS request and fires the actual PATCH request.

If rahul.dev.to is not listed in the allow-origin, the server denies the OPTIONS request.

The browser considering this as a potential threat, will not fire the actual PATCH request throwing an error,

Preflight response is not successful

Understanding the CORS response headers:

These are the headers received for the preflight request.

Access-Control-Allow-Origin - specifies the requested origin if it has access.
Access-Control-Allow-Methods - specifies which methods are allowed for CORS.
Access-Control-Allow-Headers - specifies which headers can be used with the actual CORS request.
Access-Control-Max-Age - specifies how much time (in seconds) the response of the preflight request can be cached. The browser will skip further preflight requests and directly hit the actual request during that time period.

Note:

Access-Control-Allow-Origin: '*'

It is pretty common to see people configuring like this as a workaround to allow CORS requests.

What this essentially means is that your server is allowing all the origins to hit CORS requests. No, do not do this. 🙅🏻‍♂️ Allow only trusted origins here and using '*' should totally be avoided.

Further, if you want to reduce the frequency of preflight requests for your trusted origins, you can set the Access-Control-Max-Age header to a higher value.

To read more: 📖

Recent Favorite Rails Tip!

Rahul — Fri, 18 Sep 2020 04:09:34 +0000

Rails is a powerful framework known for boosting productivity and being developer-friendly. There are many traits that help you in being productive. Let's talk about the one that I found very recently and how difficult was it to get the job done without it.

Rails migrations. If you haven't heard this term, in short, migrations help in changing the schema of your database. Read more here in case you're not familiar with it.

Rails provides scaffolding to create migration files. You need not create a migration file from scratch nor copy from the existing files. This is pretty standard.

rails generate migration add_email_to_users email:string

generates the following file

class AddEmailToUsers < ActiveRecord::Migration[5.0]
 def change
   add_column :users, :email, :string
 end
end

After that, you just run

rake db:migrate

Making a change to the database hardly takes a minute!

But consider this scenario where you created a table and started working on it.

Now for some reasons, you want to rename the column names and also add another column to it.

Since the migration already ran, these are the steps you're required to do to run the migration again.

Get the migration version from the file name 20200827072540
Remove the entry from the schema_migrations table
Drop the table transaction_dumps
Run the migration rake db:migrate again

This set of things gets really frustrating if you want to play around in your development or you're not really sure of the final schema.

A better way to do this is,

rake db:rollback #rolls back the latest migration change
rake db:migrate #runs all the pending migrations again

Neat right.

Hang on. This post ain't about this.

If you want to rollback multiple migration files and run them again, the headache is back. You have to get the migration versions and do,

rake db:migrate:down VERSION=20200827072540
rake db:migrate:down VERSION=20200825121103
rake db:migrate

There is a Rails way to do this and this is my recent favourite go-to option when I am playing with the migrations.

rake db:migrate:redo

This rolls back the latest migration and runs the migration again. Here's the elegant part, if you want to redo the last n migration files, you could just feed that as a variable STEP.

rake db:migrate:redo STEP=2

Easy and painless!

Always remember this, if you're doing something that takes too much time and makes you feel that Rails isn't fun to work with, you might be doing it the wrong way 😉

Thanks @Prathamesh for this tip.

What is a DDoS attack anyway?

Rahul — Sat, 12 Sep 2020 16:32:45 +0000

Recently in my organization, we decided to have an API gateway and we were wondering if we could also remove Cloudflare WAF layer which had very recently averted a few DDoS attacks on our website.

I previously had read a nginx article and I was of the thought that DDoS attack was all about the application layer attacks and adding a rate-limiting at the Nginx layer could prevent the DDoS attacks.

I was totally wrong!

Here's a brief of types and a variety of DDoS attacks.

Before jumping in, DoS stands for Denial of Service attack that aims at crashing the target server or disrupting its normal functioning by creating congestion on its network.
DDoS is a distributed DoS attack where multiple systems attack the target server.

There are three categories of DDoS attacks:

Protocol Attacks
Application Layer Attacks
Volumetric Attacks

Protocol Attacks:

Protocol attacks happen in layer 3 (Network Layer) or layer 4 (Transport Layer) of the OSI Model. These attacks aim at creating congestion or exhaust the server resources by attacking the intermediaries between the server and the website like firewall and load balancers.

Before going into individual attacks, let's revisit how the TCP handshake works.

SYN - the client requests for a new connection by sending a SYN with its random value - RANDOM_A
SYN-ACK - the server keeps a port ready and replies with a SYN-ACK, acknowledging the client's SYN and sending back its random value RANDOM_B
ACK - the client acknowledges the server's SYN and sends back an ACK and the connection is successfully made.

Technically, an ACK/SYN packet is a TCP packet with the "ACK" or "SYN" flag set in the header.

SYN Flood

This attack uses the vulnerability of TCP handshake by sending
the server a large number of SYN packet. These SYN packets are normally sent using spoofed IP addresses. This results in the server sending a SYN-ACK to the spoofed IP address and waiting for the final ACK forever. The purpose of the attack is to exhaust the server resources by opening a useless connection every time and prevent it from creating new requests for the actual users.

ACK Flood

This attack sends an enormous number of ACK packet to the server. The server has to process each packet to find if the packet is a legitimate one or not. The idea is to use up the resources of the server by sending junk ACK packets that are of no use to the server.

SYN ACK Flood

This attack is slightly different as the clients never send SYN-ACK to the server during the three-way handshake. The objective remains the same, it is to waste resources of the server.

QUIC Flood

Quic is a new transport protocol and it's a faster and secure way to send data. It does not use TCP, hence there's no three-way handshake here. It instead uses UDP. Since UDP is less reliable it sends multiple streams of data to make up for any loss of packets. Using TCP, the data has to be sent over HTTPS to be encrypted, whereas, in QUIC, the data is encrypted by default using TLS encryption.

This is briefly how the QUIC protocol works. QUIC flood aims at sending overwhelming data over QUIC. Since all the QUIC data is encrypted, the server has to spend even more resource to actually authenticate the legitimacy of the request.

Unlike other floods where the server sends comparatively less data(ACK or SYN), here the server has to send back its TLS certificate in the first request itself. Spoofing the IPs and flooding with QUIC, results in the server sending a larger unwanted data to the spoofed IPs.

UDP Flood

When a server receives a UDP packet, it has to check if there are any processes listening at the specified port and if it doesn't find one, the server pings the client that the destination was unreachable.
As this process has to be followed for each packet, UDP flood is effective in depleting the server resources.

DNS Flood

The functionality of the Domain name system is to translate the website names to its IP addresses. This attack uses devices that have high bandwidth connections and hits a huge number of requests so as to prevents legitimate users from accessing it.

Ping Flood

Normally PING(ICMP) is used to test the health status and connectivity of the server. This attack sends out overwhelming ping requests to use the server resources. This is the least effective mode of attack as the attacker has to send the maximum number of requests to bring the server down. This is because the cost of sending the ping reply is pretty low.

Volumetric Attacks:

These are the most common types of DDoS attacks. These attacks are reflection-based in nature where the attacker spoofs the IP and makes the server respond to the spoofed IP.

Amplification Technique:

Volumetric attacks use amplification technique where the attacker requires less cost to make the target server send large amounts of data. As it amplifies the effect on the server by using fewer resources, it is called Amplification attack.

As Cloudflare neatly puts it,

Amplification can be thought of in the context of a malicious teenager calling a restaurant and saying “I’ll have one of everything, please call me back and tell me my whole order.” When the restaurant asks for a callback number, the number given is the targeted victim’s phone number. The target then receives a call from the restaurant with a lot of information that they didn’t request.

To create a large amount of traffic, the attacker structures the request in a way that generates a large response from the server as possible.

DNS Amplification

This attack differs from DNS flood as it primarily uses amplification to make many small requests for very large DNS records data. As it's reflective in nature, the server sends the response to the spoofed IPs.

NTP Amplification

Network Time Protocol servers have a functionality Get Monlist that sends out the history of last 600 IP addresses that hit the NTP server.

This is a massive increase in the request-to-response size ratio. The server will be sending 200 times larger data than the request size. This effectively means with just 1 GB of bandwidth, the attacker can create 200 GB attack on the server.

Application layer Attacks:

These are the attacks that happen at layer 7 (Application Layer) which involves sending the HTTP requests (GET, POST) and receiving the data.

The idea is the same here, by consuming the server CPU resources, the server will not be able to process the genuine requests.

Two frequently used techniques here are

Bombarding the server with a very huge number of request that the server will not be able to handle and goes down in the process. Considering the computing power of any normal application, it will be too difficult for it to handle these attacks.
Slow attacks - that keeps the connection open by sending partial headers to the server or delaying the data to be sent for a post request and making the server wait for longer periods. These kinds of attacks are generally difficult to identify and the server might drop genuine slow requests as well if it restricts slow requests altogether.

These are some of the common DDoS attacks and the server should be equipped to protect itself from any of these attacks.

For instance,

Layer 7 attacks can be protected by adding a rate-limiting that prevents IPs from hitting more than the specified number of requests.
NTP Amplification on Get Monlist can be averted by disabling the functionality itself.
Having a Web Application Firewall(WAF) like CloudFare WAF, AWS Shield protects us from a variety of these attacks.