DEV Community: Nirmal kumar

Navigating the gRPC Galaxy: A Different view into Efficient 'api to api' Communication

Nirmal kumar — Fri, 07 Jun 2024 14:44:33 +0000

Overview:

RPC means Remote Procedure Call. It's a client-server communication mechanism, the client calls locally a procedure which is actually in remote server.
gRPC is Google's version of RPC open source simple service definition framework.
When we work with micro services architecture, in few scenarios we need to make multiple api calls which will introduce high latency. gRPC service or in other words contract-first api can help with this problem by leveraging http/2.0 protocol and binary serialization technique called protocol buffers.
Protocol buffers (protobuf) are language & platform agnostic for serializing data. It serializes the data into a compact format optimized for network transmission over HTTP2.0

gRPC vs REST:

REST is text based and uses HTTP1.1 whereas gRPC is protobuf based and uses HTTP2.0
REST leverage json/xml vs gRPC uses protocol buffers for data serialization.
REST is request/response pattern whereas gRPC is encode/decode and streaming pattern.
REST is resource oriented design as we need to communicate using HTTP verbs; gRPC is service oriented design where client knows all types and procedures locally before making a call.

When to choose gRPC:

If we have multiple teams who are specialized in different programming languages and target to build lot number of inter connected services based on domain driven design, we can design micro service architecture by leveraging gRPC framework where needed. NOTE: REST is more established, popular and simple than gRPC and it is still preferred predominantly everywhere.
If we need to stream or transfer huge data between client and server.
As gRPC is strongly typed, we need to choose for a use-case where client and server are tightly coupled, maintained or enhanced together by the same team.
Choose gRPC where performance is critical and non-negotiable.

Templates to build gRPC applications:

gRPC has wide number of language support. C++, C#, Dart, Go, Java, Kotlin, Node.js, Objective-C, PHP, Python, Ruby, and Swift. Sample

PROS:

Better performance, low latency, high thorough-put.
Ideal for low bandwidth networks.
Efficient for internal service to service communication.
Strongly typed with built in support to validate every request.
Native protobuf compiler to generate code for any supported language based on '.proto' file.
Built-in better load balancing routine and selective message compression capability. Note: I am yet to explore these advanced options.

CONS:

Not evolved fully yet with all browser types. Thus, we can't fully leverage from browser to server communication.
Not adopted widely yet.
Binary serialization makes it tedious to debug during transport.

How gRPC works:

gRPC Client:

Client calls a local procedure which is a gRPC client stub created based on '.proto' file.
gRPC runtime will binary encode and transport the request to server over http/2.0

gRPC Server:

gRPC runtime in server once it receives the request will binary decode and extract the request as server stub (created based on '.proto' file).
gRPC service will process the request and return the response back.

Note: When the response goes back to client, encode/decode will happen vice-versa between client and server.

Referene

official gRPC link

What is your cloud data backup strategy ?

Nirmal kumar — Fri, 24 May 2024 14:57:14 +0000

Did any of you think about how will we respond to a situation if our entire business data got wiped out from cloud data center?

What's our Plan B in such cases ? Are we really prepared and ready to face it ?

Everyone has to revisit cloud backup strategy for worst case scenarios. Do you know that recently an incident happened in Google Cloud that it mistakenly wiped off an entire client's private workspace including the client's entire data backup because of a misconfiguration. Client's business data worth is ~ 125billion.

Article : reference

How did they recover ? Luckily this client had backup's backup in a different cloud vendor / data center using which they came back online. It took them a week.

If we don't have a clear data backup strategy defined, then it's crystal clear that our business is simply sitting on a bomb waiting to explode.

Below are my strategic thought regarding data backup. Feel free to enter yours for safe business in the comment below,

Rigorous data backup can be done based on below strategies.

Clear data classification of every application and its data. Ex: public, internal, classified, restricted.
Clear communication with users and documentation about backup strategies and direction from leadership team.
Optimize backup strategy based on
1. data retention policy,
2. how frequent we might access,
3. how frequent we update or read it back,
4. Backup data center geographic location,
5. Data backup storage disk type (hardware),
6. Backup storage size
Data-at-rest encryption plays a vital role with data security.
Optimize data retrieval plan based on data classification and carefully evaluate business impact when needed.
Real time monitoring, reports and alerts on data backup jobs. * How often are we doing the backup and how often are we deleting the backup based on retention policy ? This is a deep question.
Thorough review and sign off on the deleted data reports by corresponding application owners.
Data backup cost comparison between cloud vendor vs on-prem data center will give practical ideas and ways to optimize data backup plan.
Regular planned switch over to Disaster recovery environment data center and allow the application to run on it for few months will give practical experience to entire engineers to have a safe environment to practice what should be done during actual production failure major scenarios.
Data backup access control plan and clear segregation of duties will safeguard from many manual errors.

Ease your automated api testing using PostMan

Nirmal kumar — Thu, 23 May 2024 08:48:17 +0000

With fast emerging technology and architecture, building & testing apis are an inevitable task for every developer.

If you are not yet aware about PostMan, please do check. It will help your api development life easier.

Frequent issues I face as an api developer are,

Api consumer will shout that the deployed api is not up and running.
If it's running, api consumer will report that it is producing irrelevant response.

If the number of api is less in numbers, it should be okay if we manually debug and fix the root cause. But in cloud native Microservices architecture, we will end up managing 100s of apis that belong to our domain. Imagine the number of apis to manage, if we have multiple environments such as dev, qa, uat, training, staging, performance, production, disaster recovery, etc.,

As an api producer/developer, I want the api consumer to be happy. If any issues, I would want to resolve it super quick.

This is where PostMan comes into picture where they provide an easy option to write unit test cases directly in their GUI for each of my api, using which I can run automated tests in seconds and identify which one is not up & running and why.

This helps me to resolve issues quicker and focus on actual business problem.

Example:

Below are simple step by step approach on how to write test cases for an api in postman,

Once we have postman installed, up and running in our machine, add a new collection & enter details about our api endpoint and enter request parameters.

Here I am using default postman HTTP GET REST endpoint for demo purpose, it doesn't have any input parameters.

Click 'Send' button in postman, it will make a HTTP GET request to the mentioned endpoint and we can see the response in PostMan bottom section 'body' tab.

As an Api producer, after we deploy our api to an environment we either use Open Api documentation portal ex: Swagger UI if we use Asp.Net Core WebApi or PostMan to verify whether our deployed api is working smoothly for every scenario. It's a manual process.

We can go to 'Scripts' tab in PostMan to write our test cases in 'Post-Response' sub section which will be automatically executed when we trigger the endpoint manually by clicking 'Send' button or we run the entire collection of apis.

Now, let's go deeper into test case.

pm.test("Response status code is 200", function () {
    pm.expect(pm.response.code).to.equal(200);
});

PostMan uses chai.js syntax to write BDD.
PostMan uses pm object to access request/response objects.
'test' is a javascript test function.
"Response status code is 200" is test name which will be printed in test results window.
'pm.expect' is assertion to verify the test.
'pm.response.code' access the Http Status Code of received response back from api.
'.to.equal' follows Chai.js BDD syntax
'200' is expected result.

Below are few more test cases for the same endpoint,

pm.test("Response time is within an acceptable range", function () {
  pm.expect(pm.response.responseTime).to.be.below(500);
});


pm.test("Response has the required fields - args, headers, and url", function () {
    const responseData = pm.response.json();

    pm.expect(responseData).to.be.an('object');
    pm.expect(responseData).to.have.property('args');
    pm.expect(responseData).to.have.property('headers');
    pm.expect(responseData).to.have.property('url');
});


pm.test("Headers contain all the expected fields", function () {
    const responseData = pm.response.json();

    pm.expect(responseData.headers).to.be.an('object');
    pm.expect(responseData.headers).to.include.all.keys(
        'x-forwarded-proto',
        'x-forwarded-port',
        'host',
        'x-amzn-trace-id',
        'user-agent',
        'accept',
        'cache-control',
        'postman-token',
        'accept-encoding'
    );
});


pm.test("Validate the schema of the response JSON", function () {
    const responseData = pm.response.json();

    pm.expect(responseData).to.have.property('args').that.is.an('object');
    pm.expect(responseData).to.have.property('headers').that.is.an('object');
    pm.expect(responseData).to.have.property('url').that.is.a('string');
});

We can verify the test results in bottom section,

With this test setup in postman, I can verify my any number of deployed api in any environment within a minute and identify which one is not working and why.

Below is a sample of how I run an entire collection of api with it's unit test cases,

Note: We can schedule these tests, run it from cli or even integrate with our CI devops pipeline.

Entire collection test results:
Note: It ran in 2seconds. All I need is 2-5 seconds to identify which of my api is making trouble.

PostMan is so powerful. It has many features like api performance testing, api monitoring, api observability, etc., I will come back with an another blog later.

Thank you for reading! :-)

C# : Avoid loops, start using equality comparer

Nirmal kumar — Wed, 22 May 2024 17:07:23 +0000

Are you writing multiple loops to compare two lists of same type in C# because of complex comparison criteria ?

There is an another simple way to achieve it.

Use Equality Comparer and write your own comparison logic with complex objects.

Remember the 'GetHashCode' logic has to return same value for duplicate instances.

Simple Example:
EmployeeEqualityComparer implements IEqualityComparer which defines criteria on how to compare two lists of same type through methods 'Equals' & 'GetHashCode'.


public class EmployeeEqualityComparer : IEqualityComparer<Employee>
{
    public bool Equals(Employee? x, Employee? y)
    {
        if(x is null || y is null) return false;

        if(ReferenceEquals(x,y))return true;

        return x.Id == y.Id && x.DepartmentId == y.DepartmentId;
    }

    public int GetHashCode([DisallowNull] Employee obj)
    {
        return obj.Id + obj.DepartmentId;
    }
}

Employee model class:

public class Employee
{
    public string Name{get;set;}
    public int Id {get;set;}
    public string Department{get;set;}
    public int DepartmentId{get;set;}
    public string City{get;set;}
    public string Zip{get;set;}
}

EmployeeEqualityComparerTestClass to compare two lists of same type and test equality comparer:

    public class EmployeeEqualityComparerTestClass
    {
        public void EqualityCheck()
        {
            //prepare test data

            var employeesA = new List<Employee>()
            {
                 new Employee()
                    {
                        Name = "Nirmal kumar",
                        Id = 1,
                        Department = "Engineering",
                        DepartmentId = 1,
                        City = "Raleigh",
                        Zip = "27616"
                    },
                     new Employee()
                    {
                        Name = "John",
                        Id = 2,
                        Department = "Engineering",
                        DepartmentId = 1,
                        City = "Raleigh",
                        Zip = "27616"
                    },
                    new Employee()
                    {
                        Name = "Nirmal kumar",
                        Id = 1,
                        Department = "Engineering",
                        DepartmentId = 1,
                        City = "Raleigh",
                        Zip = "27616"
                    },
            };

            var employeesB = new List<Employee>()
            {
                 new Employee()
                    {
                        Name = "Nirmal kumar",
                        Id = 1,
                        Department = "Engineering",
                        DepartmentId = 1,
                        City = "Raleigh",
                        Zip = "27616"
                    },
                     new Employee()
                    {
                        Name = "Tesla",
                        Id = 3,
                        Department = "Engineering",
                        DepartmentId = 1,
                        City = "Raleigh",
                        Zip = "27616"
                    }
            };

            //prepare equality comparer
            var equalityComparer = new EmployeeEqualityComparer();

            //check distinct using equality comparer
            var distinctEmployees = employeesA.Distinct(equalityComparer);

            Console.WriteLine($"Total distinct employees : {distinctEmployees.Count()}");
            Console.WriteLine("Distinct employee names in employeeA list:");
            foreach(var employee in distinctEmployees)
            {
                Console.WriteLine(employee.Name);
            }

            Console.WriteLine();
            Console.WriteLine("Let's union two different lists of same type with duplicates");

            //compare two lists of same type using equality comparer
            var employees = employeesA.Union(employeesB, equalityComparer);

            Console.WriteLine($"Total distinct employees from 2 lists : {employees.Count()}");
            Console.WriteLine("Distinct employee names:");
            foreach(var employee in employees)
            {
                Console.WriteLine(employee.Name);
            }

            Console.ReadKey();
        }
    }

Output:
Total distinct employees : 2
Distinct employee names in employeeA list:
Nirmal kumar
John

Let's union two different lists of same type with duplicates
Total distinct employees from 2 lists : 3
Distinct employee names:
Nirmal kumar
John
Tesla

C# Indexers. When do we use it ?

Nirmal kumar — Mon, 20 May 2024 10:58:11 +0000

C# Indexers:

C# indexers are like properties. It allows to access instances like an array with an index.
It always represents collection objects but exposes arguments using which we can use to identify a particular element in the collection at a time.
We can access an indexer by indexing the class instance with the appropriate index parameter.
We can overload indexers with different data types and create multiple indexers for a class to expose & access different collections using the same class instance.
It acts as a member of a class but is used to access a data item of a collection from outside the class.

When do we use it?

Let's say we have to load a collection object in memory and we don't know upfront exactly how much memory it would take. Indexers help to set up an approach to load a small chunk of the entire data load, process it, and then move on to the next chunk.
It helps to abstract collection type in the model from the usage class as it always allows to access one data item at a time. This helps code easier to understand and maintain.
Indexer should be used if providing an index to an object makes sense.

Simple Example:


public class Employee
{
    public string Name{get;set;}
    public int Id {get;set;}
    public string Department{get;set;}
    public int DepartmentId{get;set;}
    public string City{get;set;}
    public string Zip{get;set;}
}

public enum DepartmentEnum{
    None = 0,
    Engineering,
    Accounting,
    Economics, 
    Politics
}

internal class Program
{
    private static void Main(string[] args)
    {
        Console.WriteLine("Hello, World!");

        var processor = new EmployeeProcessor(DepartmentEnum.Engineering);

        processor.ProcessPayroll();
        processor.ProcessHealthInsurance();

        Console.WriteLine($"Total engineering employees - {processor[DepartmentEnum.Engineering].Count()}"); 

        Console.ReadKey();
    }
}

public class EmployeeProcessor
{
    private List<Employee> employees = new List<Employee>();

    public EmployeeProcessor(DepartmentEnum department)
    {
        LoadEmployeesByDepartment(department);
    }

    public List<Employee> this[DepartmentEnum department]
    {
        get
        {
            return employees.Where(x=> x.DepartmentId == (int)department).ToList();
        }
    }

    private void LoadEmployeesByDepartment(DepartmentEnum department)
    {
        //load it from the data source. I am Hard coding for a blog 

        if(department == DepartmentEnum.Engineering)
        {
            employees.Add(
                new Employee()
                    {
                        Name = "Nirmal kumar",
                        Id = 1,
                        Department = "Engineering",
                        DepartmentId = 1,
                        City = "Raleigh",
                        Zip = "27616"
                    });
        }
        else if(department == DepartmentEnum.Accounting)
        {
            employees.Add(new Employee()
            {
                Name = "Preethi",
                Id = 1,
                Department = "Accounting",
                DepartmentId = 2,
                City = "Atlanta",
                Zip = "12345"
            });
        }
        else if(department == DepartmentEnum.Economics)
        {
            employees.Add(new Employee()
            {
                Name = "Adithi",
                Id = 1,
                Department = "Economics",
                DepartmentId = 3,
                City = "New york",
                Zip = "23456"
            });
            employees.Add(new Employee()
            {
                Name = "Pranavi",
                Id = 2,
                Department = "Economics",
                DepartmentId = 3,
                City = "New york",
                Zip = "23456"
            });
        }
    }

    public void ProcessPayroll(){}

    public void ProcessHealthInsurance(){}
}

Amazing Azure's data redundancy strategies

Nirmal kumar — Sun, 09 Aug 2020 02:33:30 +0000

I felt that there are no simple depiction of how Azure's data redundancy strategies work which made me to write this.

Data center is a place where Microsoft is having a bulk of servers with it's own dedicated electricity, network, cooling system, security, etc., in a single physical location like how we see in Mission Impossible movies where everyone will try to insert a chip to hack the servers. But in reality, it is almost impossible because of dedicated high level security measures to protect our data by none other than Microsoft.

Microsoft groups it's data center using these terms
i) Region
ii) Availability Zone
iii) Geography

One or more data centers groups together and called Azure Region.
3 data centers in a region form an Azure availability zone.
Two or more Azure region groups together and called Azure Geography.

Now we are good with basic terms. Let's see about data redundancy. What is it ? what is the need ?
Well! Did you ever think of what are you gonna do if your mobile fell into water ? How are you going to backup or restore the data in that crashed mobile ? What if you have an another place where your mobile syncs all your data everytime whatever you make changes in your mobile. That is data redundancy. Whatever you save to Azure to a storage account based on the redundancy option you selected it copies or make similar copy of same data in other place as backup so that in case of a disaster our data is always safe in Azure and can be easily recovered.

There are 6 redundancy options available in Azure.

a. Local redundant storage (LRS)
b. Zone redundant storage (ZRS)
c. Geo redundant storage (GRS)
d. Geo Zone redundant storage (GZRS)
e. Read Access Geo redundant storage (RA-GRS)
f. Read access Geo zone redundant storage (RA-GZRS)

Local redundant storage:

Zone redundant storage:

Geo redundant storage (GRS):

Geo Zone redundant storage (GZRS):

Read Access Geo redundant storage (RA-GRS):

Read access Geo zone redundant storage (RA-GZRS):