DEV Community: Kartikey Srivastava

Optimizing an API Response

Kartikey Srivastava — Thu, 17 Oct 2024 03:30:00 +0000

As backend engineers, we frequently deal with API latency issues. It’s easy to create an endpoint and invoke a method when the endpoint is hit, but the real question is: Is that API efficient?

What is the response time? If it’s not a background task, anything over 1500 ms can feel excessive. Now, imagine you’re on a platform’s checkout page, ready to make a payment. You click “Pay,” and a message appears: “Do not refresh or press the back button.” But the screen gets stuck. Your money has been debited, yet you still don’t see a successful transaction. How frustrating would that be?

These situations often make the end users feel like switching to some other platform and use their services. The business can get affected and what not.

While keeping that scenario in mind, let’s shift our focus to the solution. It’s time to optimize our APIs.

Keep Payload size in check:

Ever tried uploading an image to some platform and it says, the image should be within “x” kb/mb?
Performance largely depends on how quickly the server is able to process the request.
As you increase the payload size, it becomes more and more complex for the server to process the same and it might compromise on the latency while giving the response.
Applications dealing with large image files often define limits to prevent the end users from uploading images beyond a certain size.
This helps server to function smoothly.
So, the next time a site asks you to reduce image size, don’t complain,,, just remember it’s for better performance! :)

Compress your API response:

Let’s understand this with an example of Git ZIP file analogy.
When we talk about compressing api responses, think of it like downloading a Git respository as a ZIP file.
Here’s how: Imagine you are cloning a git repository. You have 2 options:
Clone the entire repo with Git which pulls entire file and history as it is.
Download it as a ZIP file that compresses the file into a smaller package making the download faster.
In the same way, when your browser requests data from a server, instead of sending that huge data(like cloning an entire repo), it can compress the data/response (similar to zip file).
This makes it faster to transfer the data across the internet.
The only difference that lies here is that you manually extract the files whereas the browser does it automatically.
The idea is to compress the size of the data to make the process more efficient.

Pagination:

Imagine you are trying to buy a smartphone from Amazon in it’s Great Indian festival sale.
If for a smartphone search request, Amazon loads all the smartphone at once on a single page it would lead to slower page load or it might even crash.
To prevent this failure, you see somewhere around 20–30 smartphones per page and in the end you see a “Next” page button.
This helps in smooth functioning of the backend servers as with less data(30 items) to handle, it can send back the response faster improving overall speed.
Pagination basically means breaking down a large data set into smaller chunks that are easy to manage.
Instead of showing all the data at once, you only display a small chunk of it.

Remove unnecessary processing on the server:

This is one of the most avoided mistakes developers do in daily life.
It might be overuse of printing logs in the console after every line or it might be making db calls every now and then.
If there is some frequently accessed data, or maybe some calculation done on the same data repeatedly for different users, the application can cache the result and avoid hitting the db.
This can result in significant load reduction on the db as well as it will help the server to retrieve results more quickly, improving overall performance.

Optimizing API responses is crucial for creating a good user experience. By keeping the above points in check, one can enhance his/her API’s performance.

Can you think of some other methods that might help us in improving our backend more in terms of api latency?

If you liked this post, do consider liking and you can also share and follow for more such content :).

How to choose the right database?

Kartikey Srivastava — Wed, 16 Oct 2024 03:30:00 +0000

Being a software developer isn’t just about making a system and getting it running. Imagine you’ve developed an application without knowing how big its user base could grow. You tried to make its performance exceed expectations by using an in-memory database like Redis. You are very happy with the response and money you’re making because of your hard work. Now imagine the user base starts to explode and your current database setup isn’t ready for that. You don’t want to lose your current users but you’re not even able to handle them. Yes I am talking about a social media giant named Instagram.

Instagram in its initial stage used Redis as its database. Redis is a fast in-memory key-value store. It worked really well when the Instagram’s user base was small. But as the Instagram’s popularity exploded, it saw limitations. We like, comment, post media on Instagram every second. That basically means we are talking of not less than a million writes on the database per second. It wasn’t easy to manage this volume of writes using Redis which is a pure in-memory database(meaning your data handling would always be limited according to your RAM).

Eventually, Instagram made a significant switch to Cassandra, a distributed NoSql database. End of story.

I want you to understand how important it is to choose the right tools in your software development journey. Ignoring it today will make you suffer tomorrow.

Before we move on to “why this, why not that?”, let me walk you through the introduction of “this” and “that”.

Relational Database:

We all know these databases organize data in table format where each table is a collection of rows and each row is a collection of columns. These databases in their early stages were used to track sales or process bank transactions. A developer interacted with the data through SQL. To give you an example, PostgreSQL is one of the most widely used relational database.

NoSQL:

A twitter hashtag that was supposed to be catchy ended up giving name to another type of database. This system gained a lot of popularity because of various reasons some of which were:

Scales better than relational databases.
Higher write throughput
Queries on unstructured data was relatively easier as compared to relational database.

Different systems have different needs, and no single database can cover them all. Some need high availability(just like Instagram) whereas some need high consistency(just like Banks). As a result, organizations today use multiple databases that include both relational and non-relational databases. This approach is termed as Polyglot Persistence.

Next time you listen a system using multiple DBs, consider it a polyglot persistent system.

The above picture resembles a resume structure. As you can see in order to store the profile of the person mentioned we have multiple tables. Lets say you want to fetch the “end_year” of the above person’s education.

Your query would somewhat look like:

SELECT e.end_year FROM education_table e JOIN user_table u ON e.user_id = u.user_id;
Now imagine, if there were more tables inside this and you had to fetch something very granular you would have ended up writing an essay of joins.

{ "user_id": 251, "first_name": "Bill", "last_name": "Gates", "summary": "Co-chair of the Bill & Melinda Gates... Active blogger.", "region_id": "us:91", "industry_id": 131, "photo_url": "/p/7/000/253/05b/308dd6e.jpg", "positions": [ { "job_title": "Co-chair", "organization": "Bill & Melinda Gates Foundation" }, { "job_title": "Co-founder, Chairman", "organization": "Microsoft" } ], "education": [ { "school_name": "Harvard University", "start": 1973, "end": 1975 }, { "school_name": "Lakeside School, Seattle", "start": null, "end": null } ], "contact_info": { "blog": "[http://thegatesnotes.com](http://thegatesnotes.com/)", "twitter": "http://twitter.com/BillGates" } }
Above is a json representation of the same resume. Now, if you want to fetch the end year from the education your input would look something like:

db.collection_name.find( { "user_id": 251 }, { "education.end": 1, "_id": 0 } )

Above NoSQL query has been taken from chat-gpt as I haven’t worked quite a lot on NoSQL.

If we consider the above case, we can see that the JSON representation has better readability than the traditional schema. If you want to fetch a profile in the realtional example, you need to perform multiple queries or do a “JOIN” dance between the user table and its subordinate tables.

Below is the json representation of the above One-to-Many relation(user and his data).

There are a lot of reasons of using one database over other but there is no way one can say NoSQL is always better or SQL always sucks. It totally depends on the use cases of both of them and hence most organizations use them together.

That’s it from my side. If you want to read more about it I’d recommend you a book : Designing Data Intensive Applications by Martin Kleppmann . It’s a gem of a book.

If you liked this post, please consider liking it and you can also share and follow me for more such content :).

Caching — An overview

Kartikey Srivastava — Tue, 15 Oct 2024 03:30:00 +0000

In simple terms, caching means storing the frequently accessed data in a storage where one can quickly retrieve it without the querying the source of truth i.e. the database. This storage is generally a temporary storage. Caching helps in improving the performance of an application by loading the data somewhere near to the application so the time taken in querying the db is reduced and also the number of times a data source is accessed is also reduced.

Applications can respond to the user requests more quickly improving the overall user experience. It also helps in reducing the load on the database since the data is now being accessed from the cache and not the db. One thing to note here is, Caching is suitable for frequently accessed data.

We all know CDN servers, DNS servers, Redis cache and Apache Ignite are some of the most commonly used cache in today’s world. I have already talked about CDN and DNS in my previous posts in case you want to read.

Let’s get an overview again:

CDN(Content Delivery Network) caches the static content and serves them to the users located in the nearby region.

DNS(Domain Name System) can cache the IPs of the servers and avoid the long path from the client’s IP to the authoritative server.

Redis and Apache Ignite are the caches mostly used as databases. Redis follows a Master-slave architecture whereas Ignite follows a distributed data structure.

There are many types of caching some of which are:

Client caching
Distributed caching
Database caching
Application caching

Let’s understand them one by one with the help of examples:

DISTRIBUTED CACHING

A type of caching that involves storing data across multiple servers/nodes in a network.
A perfect example for this is Apache Ignite.
It has 2 caching modes: PARTITIONED and REPLICATION.
In case of partitioned mode, the data is stored in chunks across multiple servers so if you have 100 units of data and 3 servers…30:30:40 can be the ratio of their storage.
In case of replication mode, the data is stores as a whole across all the servers so if you have 100 units of data and 3 servers… each one of them would store all 100 units.
This type of caching is useful when the application requires high availability and scalability. You can always add more nodes in order to scale and also in case a node goes down, the remaining nodes can handle the user requests.
However, distributed caching trades of consistency in order to achieve high availability.
Read about CAP Theorem and you’ll get a better clarity of why distributed caching follows an eventual consistency pattern.

DATABASE CACHING

Database caching involves storing frequently accessed data in-memory rather than fetching it from the primary database every time, which reduces database load and improves overall performance.
By routing the requests to the cache, you can achieve a much lower latency and also reduce the load on the database which includes querying it every time a request comes in.
Instead of querying the database for every request, your application can first check the cache. If the data is found(cache hit), the request is served from the cache. If not found(cache miss), the request is sent to the database. Now this request can be stored in cache for future queries.
For example, a user requests his/her information for the first time on a server. This request is served directly from the database. With caching, now the requested data is stored closer to the client after the first request, now for every similar request it would be served from the cache instead of your primary db.
Caching solutions like Redis comes into play here.

CLIENT CACHING

This type of caching involves storing data directly on the client device be it a web browser or a mobile phone.
It becomes useful for applications that frequently access client’s locally stored information.
Obviously, the application’s performance improves since now the number of requests to the server has been reduced that also leads to less data being travelled over the network.
A perfect example for the same is Password storage in Mobile applications. Applications that ask you to enter a password to log into them, they might offer you to save the password on your device with a message like: “Don’t worry, this will be saved locally on your device.”
Once you give your consent, the password gets stored locally on your device in some encrypted format and the next time you log in, the password field gets automatically populated.

APPLICATION CACHING

Here the caching is done within the scope of application.
API responses, User serssions can be cached and served to the user requests.
Do not confuse it with database caching. They tend to be similar but are different when it comes to what is being cached.
Application caching can cache api responses whereas database caching can cache query results.
Anything at the application level can be cached using application caching be it an api response, a token or some computed value.
Application cache lies closer to the application whereas database cache lies closer to the database.
For example, let’s say you have an e-commerce platform. When a user logs in, his/her user session can be stored in the application’s in memory cache and now this session dadta can be served quickly without the need to go to the database.

These are some of the caching techniques. I have kept this post short and brief as I wasn’t feeling quite well but had to keep the streak of posting on weekends alive. Will come with better and detailed posts in future.

If you liked this post do consider liking it and you can also share and follow me for more such content :).

Bloom Filters

Kartikey Srivastava — Mon, 14 Oct 2024 03:30:00 +0000

Imagine walking into a mobile shop that only sells Samsung and OnePlus phones. You ask the shopkeeper for an iphone. Without wasting any time, he says, “No, we don’t sell iphones.” Now, you change your mind to buy a One Plus model(a rare one). Instead of denying straightaway, the shopkeeper responds, “It might be in stock, but I can’t be 100% sure until I look.”

This is similar to how a Bloom Filter works, it can definitely tell you if something doesn’t exist(like an iphone in an android shop), but if it says something might exist, there is a certain uncertainty of the data’s presence.

To keep it more simple, Bloom Filter can give you a false positive(i.e. return true for something that is false). But it can never give you a false negative(i.e. return false for something that is true).

We all have encountered those warning messages while signing up on some platform that says, “Username already taken/exists”. That’s Bloom Filters in picture(not always as there are some more methods like querying the database or cache but that becomes inefficient when the user traffic explodes).

Let’s understand this Data Structure bit by bit:

A Bloom Filter is a probabilistic data structure. Now what does that mean?

Probabilistic in context of data structures means that this data structure provides answers with a certain possibility of being correct. This filter can definitely say when an element is not present inside it, but it might be wrong in some cases when it says, “Yes, I have this data”.

A bloom filter uses a bit array to store the data. Now when I say it stores the data, it doesn’t store the actual raw data but applies some hash functions to those data sets and accordingly marks the corresponding positions in the bit array. For example, if you want to check if “Kartik” exists inside a bloom filter it will apply multiple hash functions to this input and then mark their corresponding indexes as 1.

Let’s break this down:

Input — ‘Kartik’ to be added inside a bloom filter.
Hash functions applied: Since a bloom filter uses multiple hash functions, we will consider it has 3 for better understanding.
Positions to be marked: Since this filter uses a bit array(a relatively large size bit arrray) where all indexes are set to 0(switch off) by default, based on the hash function’s result the positions will be marked as 1(switch on).
That’s it. The data is stored now. Go ahead to check if it is present or not.
Below is a visual representation of hash functions being applied to an input:

Now, the outputs of the above hash functions(3, 5, 9) are the positions that need to be set. Below is its visual representation:

As you can see the positions 3,5 and 9 are set to 1.

Checking if the data is present in the filter or not:

You give the input. Let’s say the input is again “Kartik”.
The same hash functions are applied to “Kartik.”
On applying the hash functions we would get the same result for obvious reasons which would fetch us the same positions inside the bit array i.e. 3,5 and 9.
The filter checks if these positions in the bit array are set to 1.
Oh, these indexes are already marked as 1(they are on). This means the data might be present.(Username already exists, please try using a different username).
However, if any of the hash function results point to a position that is still 0, the filter can confidently say, “No, the data is not here.” This is because the same input, when hashed, will always give the same result, so if those positions aren’t set, the data was never added.

A Bloom filter can give false positives (saying something might exist when it doesn’t), but it can never give false negatives (saying something doesn’t exist when it actually does). So, when you check for “Kartik” and all bits are 1, it means “Kartik” might exist. If even one bit is 0, the filter is sure it doesn’t exist.

To minimize the number of false positives you carefully need to decide the length of the bit array to be used(within your system’s capabilities) and also the number of hash functions.

Increasing the number of hash functions exponentially would result in slowing down the process but decreasing them won’t increase the speed but would result in an increased number of false positives.

Read more about how to calculate the above specifications here

That’s all that I know about Bloom Filters as of now. I have used it in my personal projects but not on big scale ones so the choice of using this data structure should be entirely yours. I’m attaching a link where you can experiment on this data structure and find out how many times you get a false positive?

Here you go:

https://llimllib.github.io/bloomfilter-tutorial/

If you read it till here, I’d like you to please share your story in case you used this data structure in your experience. If not used, do consider giving it a try and if you liked this post, please like, comment and share to increase the reach. You can also follow me for more such content. I post on weekends. Thanks :)

Encryption Symmetric

Kartikey Srivastava — Sun, 13 Oct 2024 12:25:36 +0000

In simple words, converting a readable information(plain text) into something unreadable(also known as cipher text) to protect it from anyone who isn’t supposed to see it is what is called as Encryption. Encryption involves scrambling of a plain text to produce a cipher text with the help of a key.

Imagine you(Person A) want to send some physical item to a friend(Person B) sitting abroad. This thing is very personal and if it gets exposed both of you guys might get into some problem. So you pack the thing inside a box and lock it with a key. Both you and your friend have same copy of this key. You locked the box with the thing inside and send it to your friend. Your friend received the box and unlocked it with the same key he had and access the thing.

In this scenario, ‘A’ did encryption(boxing) whereas ‘B’ did decryption(unboxing). Since both of the parties use the same key to both encrypt and decrypt hence this process is known as ‘Symmetric Encryption’.

Below is the representation of how symmetric encryption works:

The key thing to note here is that the algorithm used here is just a mathematical formula which is designed to scramble the input whereas the key is used as a part of this formula. This algorithm is generic but the key, this is what ensures the uniqueness of the scrambled data.

Let’s understand one of the simplest encryption algorithm, called the Caesar Cipher.

This algorithm is a very basic one which simply replaces each alphabet with its subsequent character. Simply speaking, A becomes B, B becomes C and so on.

With this algorithm in play, “Hello” can become “Ifmmp” which isn’t readable hence known as the Cipher text.

This is a very poor algorithm and is rarely used in the industries as we all know a simple brute force can help us determine the actual input. Who would want their credit card information to be leaked this easily?

Modern encryption algorithms like AES-256 ensure proper uniqueness and hence are very secure from threats. I’ve myself used this algorithm in one of our company’s project. Considering the current computing capabilities, it could take almost a trillion year to decrypt this information.

Symmetric Encryption uses the same key for encryption and decryption. Hence, its very important that the key should be kept secure. Sharing this key could lead to security issues and your data can be easily exposed.

An effective strategy to use encryption algorithm is to generate the key at runtime.

This means, a new key is generated each time the data needs to be encrypted or decrypted. By new key, I mean that for a single transaction the key would remain same but for every new transaction it should be unique. After the session ends, you can either drop the key(if stored in the database) or you can expire it.

Symmetric Encryption is usually used for encrypting data at rest such as for files stored on a disk or a database. Databases that store sensitive information (like user credentials or payment details) often use symmetric encryption to secure that data when it is not actively being used.

This is it about Symmetric encryption. In future posts, we’ll dive deeper into other encryption techniques, including asymmetric encryption as well.

If you read it till here, I’d like you to please share your story in case you used this encryption algorithm in your experience. If not used, do consider giving it a try and if you liked this post, please like, comment and share to increase the reach. You can also follow me for more such content. I post on weekends. Thanks :)