DEV Community: Nixon Islam

Little GRPC History

Nixon Islam — Fri, 19 Mar 2021 06:41:28 +0000

This post in based on another youtube video. I am writing my summery of the video's content.

Learning:

Here the author explains why gPRC is being created. What type problem it going to solve, what are the problems with multiple client server communication protocol

Problem:

There are so many client server protocol exits. Like, http/1, http/2, tcp/ip etc. These are the basic protocols which are used to communicate between multiple servers to server, server to client, client to client. But they are not languages agnostic. Their implementation of each programming language is very different. Plus there are so many libraries to support. If one protocol is being changed the whole supported library needs to adapt that feature and update the library. And the consumer also needs to adapt the libraries too.

Solution:

So the gRPC solves the loads of client libraries for different protocols, different types of languages.

One client side library for all languages. No need to change all libraries for new changes like before. It has a generator which will generate client side libraries for all types of major languages.
It’s using http/2 under the hood. So no need to add new codes to adapt the changes for converting http/1 to http/2. Or in future if the protocol is being changed to http/3, it will be the same too for the client side.
The messaging format is using protobuf or protocol buffers. So any client with supported grpc client libraries can communicate with each other

The original Link can be found here, This is why gRPC was invented

Redis Based Lightweight Microservices

Nixon Islam — Fri, 19 Mar 2021 05:41:10 +0000

This is another blog post is based on a youtube video I have watched in youtube. This whole post is the summarised version of the video and my understanding of the use-cases. :)

Summery:

The main discussion is about how to use a single redis instance to build a system of microservices.

Learning:

To build microservices we gonna need these things
- Service discovery
- Messaging
- Load balancing
- Health
- Presence
- Logging queuing
How to share key space with all of the services. To do that we need to maintain a specific type of pattern while saving/retrieving keys in redis. Like in the video the author used the pattern like this:

Prefix:service_name:instance_unique_id:type

Example:
- hydra:service:order-svc:289347928374982:health:log
- hydra:service:order-svc:service
- hydra:service:order-svc:2893479223474982:presence
- hydra:service:order-svc:routes
This way in a shared redis all of the information can be stored and all other services can access/filter these without any hassle. Like service 1 can create routes of order service, service 2 can get the routes of order service by accessing the order-svc routes keys without using extra api calls or other stuff.
Check if a service is present This is needed for load balancing, service discovery, routing etc.
- So the idea is when a service is being upped it will create a presence key with it’s instance id in the redis key store with a ttl. After the ttl the key will expire. It will be the service's responsibility to update the ttl or regenerate the presence key if needed.
- So using this other services can find a service instance just by filtering proper keys. Like filtering hydra:service:order-svc will provide the list of services
Service health check: A shared key similar to presence can be used to store a service's health after each 5 sec. This way a key can be used in any place to check service wise health checks, or other information.
Service discovery: for service discovery using some customized query can help find you service instances, instance presences, routes.
Service routes: again this can be easily achieved by querying proper way. We can find all service’s routes by querying generic routes type query, ger service specific routes by querying service routes query. We can easily store a list of routes in each service keys.
Load balancing: This can be done by service discovery, presence, routing features. Redis keys, list, string etc feature can be used for this
Messaging: We can use redis pubshub to use messaging between microservices. Like each service will listen to two channels one is a channel name of its own instance id, service name and another is service name. Whenever a normal message needs to be processed, services should go through the message to the global service channel, if the message needs to be sent to an individual service that message will be sent to an instance id specific channel.
Queues: using redis message queue can be a very easy solution for us. We can use redis lpush and rpoplpush along with service sacrifice queues. Publishers can push data/messages in service specific queues by lpush then listeners can consume those by pop methods.
Logging: can be used as a distributed logging system. Same as the queuing process.
Config management: can be used as shared config management across the whole system. Can store config using service keys prefix.

The original Link can be found here, Building Lightweight Microservices Using Redis

5 Redis Use Cases - Redis Lab

Nixon Islam — Sat, 27 Feb 2021 12:02:56 +0000

So I was seeing a video in youtube about redis use case for a specific scenarios or a system design. So I am writing down my understanding of those use-cases

Learning:

The key redis use-cases are being used in the video

Caching
Queuing
Locking
Throttling
pubsub’ing

Summary:

They were explaining how they integrated redis in their system in multiple layer as explaining their's system workflow.

They cached the db queries in redis.
Then when a user requests a query it queues in redis and a consumer consumed the request and start processing for the query.
Before starting the query they locked the user request so that his further requests are not being processed by other consumers. If another consumer fails to lock a query(means it is already being locked) they re-queue the query again with fibonacci series seconds.
After they get a response from the db they generate the cache for the query and send back the request to redis by publishing(pubsub) it and on the other side the web server was listening(pubsub) for the response in the redis(it started listening right after it made the request to that service). When it gets back the result it responds back to the client.

The original Link can be found here, 5 Redis Use Cases with Gur Dotan

Clean Sentry On-Premise Database

Nixon Islam — Mon, 22 Feb 2021 07:50:41 +0000

So we track our error logs via Sentry on-premises which we hosted on our own. Recently it started to give error, the disk space is running out!!!

WHY??

Apparently we needed to clear the database after certain time. Otherwise it will stack up all of the error logs.

Solution??

We log into our server. Sentry was running in docker. We went to the docker folder ran this
docker-compose exec worker bash
After that from the worker bash we ran
sentry cleanup --days 30
basically this will clean up all the events data before 30 days.

After this we go inside of the database by running these

docker-compose exec postgres bash
psql -U postgres
VACUUM FULL; Point to be noted, VACUUM FULL; will lock your db tables unless the full vacuum is being done.

Voila! Database and hard driver storage cleaned up! :)

Best Practices Working with Billion-row Tables in Databases

Nixon Islam — Sun, 21 Feb 2021 13:13:48 +0000

This whole post is about a video from Hussein Nasser I saw from youtube. This is just the summarise version of the video along with my key takeaways from the video

Learning:

The discussion is on

How to handle data in billion row based table
What kind of approach can be taken
How to redesign the table to handle 2 billion rows based table

Summary:

Here the discussion is being started with how a twitter follower table has been designed. A simple approach, make a table with which person follows whom. A 2-3 columns based rows. But it will be a huge table in the long term in sense if we use this for twitter. So what can be done

Do normal query without the concept of indexing. Just brute forcing the data without the concept of anything. Do multi threading, multi processing and find the data from the table using lots of machines (map reduce)
Use indexing on the table and find the data from the table using the indexed data.
Now you have billions of data so indexing is huge, to search that use database partitioning in the same disk. Use pair partitioning
Now to optimize more use sharding in the system(multiple host). But it adds more complexity in the system. The client needs to be aware of the shard info before querying, then needs to find the proper partition for the query, then make an actual query. Which makes another layer of logics along with business logics.
Another way is to redesign the system like in the profile table add 2 more rows to hold db columns like follower count, followers(in json). In that way a profile holds all information about followers of a profile. The problem might generate how to write/edit this data. But that is another kind of system design like querying, CQRS, event based solution. It solves current issues.

Circuit Breaker Pattern in Nutshell

Nixon Islam — Sun, 21 Feb 2021 13:00:12 +0000

Recently I was reading a blog on Circuit Breaker by Martin Fowler. I decided to summarise the content based on the blog and my experiences on this topic.

Learning and Summery:

In distributed systems a system often needs to call remote service calls to get proper data. But it can generate critical resource outages, cascading failures across multiple systems if the remote services went down and lots of service calls are being queued up till timeout happens. These situations can be avoided easily by implementing circuit breaker patterns.

Circuit breaker pattern is simply like this, if there is too much error(based on desired error threshold) simply terminate the call to remote service immediately after a remote call is being received. It will start making the calls after it restores the communication with the remote service. It should check the service health after certain period to check the service availability after enabling the circuit breaker.

Basic Example:

Let's say we implement a circuit breaker on a remote service call. Whenever a request needs to be executed it sends to a queue. Then a consumer reads the queue later and executes the service call one by one or multiple threads and response back.

So in this situation we set rules like if 20 service calls in the last 60 seconds are being timed out or get network/gateway errors like(5xx errors, 422 etc) we will enable circuit breaker for 60 sec.

Means for next 60 sec no service calls will be executed or the consumer will not consume any service request from the queue.

In the background there will be a worker who will start checking the health status of the remote service in every 5 sec right after the circuit breaker is being enabled. Once it gets the healthy service response it will monitor till 20 more sec. After that it will reset the circuit breaker or increase the circuit breaker timeout for more 60 sec.

Benefit: Assume we had million service requests in the queue, and the consumer has 50 threads to process the queue. Without the circuit breaker it would try to execute all the requests even if the remote service died or made errors. Right after we got errors(till the threshold) for the service calls, we started holding the requests till the service came back online and had a smooth response. This makes a smooth transaction between distributed services and less error handling.

Sample Package to Check: I checked one of the python packages for circuit breakers. Thought i did not use that personally, but seems it has some functionality out of the bat. Will definitely try later.