Let's Do Tech

Posted on Feb 8 • Originally published at letsdote.ch on Feb 21

Intro to gRPC and Protocol Buffers using Go

#systemdesign #architecture #communication #distributed

Inter-service communication is perhaps one of the fundamental aspects of distributed computing. Almost everything relies on it. Distributed architectures consist of multiple microservices with multiple running instances each. The workloads run long running tasks on virtual or physical servers, containers, or Kubernetes clusters, while simpler tasks are run as serverless functions.

gRPC caters to the scenarios where there is a need for distributed workloads to be tightly coupled – those services which rely on communicating data, and where speed matters. The usual JSON based REST APIs don’t fall short, but if teams are seeking “even more” performance improvements in their distributed and tightly coupled architectures should consider using gRPC instead.

In this blog post, I will not go through the theoretical details of gRPC, and rather focus on the practical example to introduce it to you, especially when you are short on time. The gRPC documentation is a great resource for understanding this technology in detail, otherwise. Topics covered in this post are listed below.

Introduce the server and client example
Define interfaces in .proto file
Generating gRPC code for Go
Server implementation
Making a gRPC call in client
Testing and conclusion

This was originally published on Let's Do Tech. Subscribe to Let's Do Tech News for timely notifications!

Server And Client Example

Before we proceed to discuss gRPC, let us establish a baseline requirement using client server architecture. In a hypothetical scenario, let us assume that there are two services – a calculator server and client which consumes the calculator logic. The calculator server implements the logic to perform addition operation. A client application hosted on a different host calls the addition function to fetch the processed result.

To keep things simple, let us implement the client and server logic in the same repo as shown in the file structure below.

The server code below is currently a basic Go code that implements addition logic with hardcoded values. Whether you are starting from scratch, or already have a server implementation and now looking forward to implementing gRPC in existing code – this blog post would serve both purposes.

package main

import (
  "log"
  "net"
)

// Calculator server implementation
func main() {
  sum := Add(1, 2)
  log.Printf("Sum: %d", sum)
}
func Add(num1, num2 int) int {
  return num1 + num2
}</textarea>
package main

import (
  "log"
  "net"
)

// Calculator server implementation
func main() {
  sum := Add(1, 2)
  log.Printf("Sum: %d", sum)
}
func Add(num1, num2 int) int {
  return num1 + num2
}

Let us assume the client code implements the client logic that depends on the functionality exposed by the calculator server, as shown below.

package main

func main() {
  // Client application logic
}</textarea>
package main

func main() {
  // Client application logic
}

Initialize the Go module by giving it a suitable name. This step will differ depending on how the client and server code is currently organized in your environment. For this example, we will create all components in the same Go module. I have used “ldtgrpc01” as the module name.

go mod init ldtgrpc01  
go mod tidy

Define interfaces in .proto file

gRPC implements Protocol Buffers (Protobuf) to serialize and deserialize the data. Protobuf offers a language and platform neutral way to serialize structured data making it smaller and thus reducing latency. Using Protobuf files, we can define services and messages to be implemented for the sake of communication between server and client.

In this example, the calculator server implements the addition function, which is called by the client. To make it compatible with gRPC protocol, first define these specifications in a .proto file. Create a 3rd directory to manage the Protobuf files, and create a calc.proto file as shown below.

syntax = "proto3";

package calc;
option go_package = "ldtgrpc01/proto";

// Define the service
service Calculator {
    rpc Add(AddRequest) returns (AddResponse) {}
}

// Define the messages
message AddRequest {
    int32 num1 = 1;
    int32 num2 = 2;
}

message AddResponse {
    int32 result = 1;
}</textarea>
syntax = "proto3";

package calc;
option go_package = "ldtgrpc01/proto";

// Define the service
service Calculator {
    rpc Add(AddRequest) returns (AddResponse) Array
}

// Define the messages
message AddRequest {
    int32 num1 = 1;
    int32 num2 = 2;
}

message AddResponse {
    int32 result = 1;
}

Explanation of the code below.

After specifying the syntax version as “proto3” (details), we define the package name. When we compile this file into native Go code, it will be created in the package name we specified here. Note that if you are specifying go_package, then package calc; is not required as the package will be named as per go_package. I have included it as best practice.
Then we define the Calculator service, and specify an Add method, which takes AddRequest message as parameter, and returns AddResponse message. Note that this method is defined as rpc.
Further, we define both AddRequest and AddResponse messages with appropriate parameters.

You can think of this as language-neutral interface specification, which only defines the interface, and doesn’t implement the same. The implementation will follow later in the server code.

Generating gRPC code in Go

Using the calc.proto file, you can generate native application code in multiple languages. Since we are dealing with Go, we will use the command below to generate Go code which we would use in client-server communication.

protoc –go\_out=. –go\_opt=paths=source\_relative –go-grpc\_out=. –go-grpc\_opt=paths=source\_relative proto/calc.proto

Protoc is a Protobuf compiler used to compile .proto files into native code. Refer to this document for more information on the parameters used in the above command. The compilation results in creation of 2 Golang code files within the proto directory, as represented below.

And after this operation, the resulting directory structure and files there should look like below.

The calc.pb.go and calc_grpc.pb.go files are automatically generated files from your Protocol Buffer file (calc.proto). They serve different but complementary purposes as described below.

calc.pb.go

Contains the Go struct definitions for your messages (AddRequest and AddResponse)
Includes serialization/deserialization code for these messages
Handles the basic Protocol Buffer encoding/decoding logic
Generated from the message definitions in your proto file

calc_grpc.pb.go

Contains the service definitions and interfaces for your gRPC service
Includes the client and server code for your Calculator service
Provides the RPC communication layer implementation
Generated from the service definitions in your proto file

Thus, we have used protoc compiler to compile the Protobuf definitions into native Go code, which can be integrated into client and server applications. Note that, you should never modify these files. If you want to do the changes to the interface, modify and recompile the calc.proto file.

Tip: Feel free to go through this code to understand more details about how Go implements gRPC protocol.

Server implementation using gRPC

To update the calculator server code to implement the interfaces generated using gRPC, you need to import it as a package in the application code. The diagram below shows how the compiled proto package is used by server code to expose its functionality.

Now that we have the native code in place which defines the gRPC interface, we need to implement the server logic. Below is the updated code for the calculator server which exposes the Add() function using gRPC. The explanation follows.

package main

import (
  "context"
  pb "ldtgrpc01/proto" // replace with your module name
  "log"
  "net"

  "google.golang.org/grpc"
)

type server struct {
  pb.UnimplementedCalculatorServer
}

func main() {
  lis, err := net.Listen("tcp", ":50051")
  if err != nil {
      log.Fatalf("failed to listen: %v", err)
  }

  s := grpc.NewServer()
  pb.RegisterCalculatorServer(s, &amp;server{})
  log.Printf("Server listening at %v", lis.Addr())

  if err := s.Serve(lis); err != nil {
      log.Fatalf("failed to serve: %v", err)
  }
}

// // Add method implementation
func (s *server) Add(ctx context.Context, req *pb.AddRequest) (*pb.AddResponse, error) {
  result := req.Num1 + req.Num2
  return &amp;pb.AddResponse{Result: result}, nil
}</textarea>
package main

import (
  "context"
  pb "ldtgrpc01/proto" // replace with your module name
  "log"
  "net"

  "google.golang.org/grpc"
)

type server struct {
  pb.UnimplementedCalculatorServer
}

func main() {
  lis, err := net.Listen("tcp", ":50051")
  if err != nil {
      log.Fatalf("failed to listen: %v", err)
  }

  s := grpc.NewServer()
  pb.RegisterCalculatorServer(s, &serverArray)
  log.Printf("Server listening at %v", lis.Addr())

  if err := s.Serve(lis); err != nil {
      log.Fatalf("failed to serve: %v", err)
  }
}

// // Add method implementation
func (s *server) Add(ctx context.Context, req *pb.AddRequest) (*pb.AddResponse, error) {
  result := req.Num1 + req.Num2
  return &pb.AddResponse{Result: result}, nil
}

To make use of the protobuf interface code we built in the previous step, first, import it in the server code.
Define a server struct type by simply calling the UnimplementedCalculatorServer method. I will not cover what is meant by this method name generated by protoc in this blog post. For now, just know that it is this easy to implement the interface in a Go code for a gRPC server.
Update and add the Add() method to the server. Note that we are using pb.AddRequest as input params, which is implemented by the protobuf Go code. Accordingly, we are using the Num1 and Num2 to calculate the sum, as per our definition in the calc.proto file.
Finally, in the main function, we create a new instance of gRPC server using Google’s grpc package, and register the interface using RegisterCalculatorServer function (from proto package) to the same. This exposes the calculator functions like Add() to be used by clients.

Making a gRPC call in Client

Similar to how we used gRPC package to implement the server side, we import the proto package in Client code, and use it as shown below.

Assuming the clients are able to access the calculator server – current example runs on localhost – we use the same grpc library from Google, and Dial into the server to establish a connection. Using this connection, we create a client instance that represents all the methods exposed by the server to the client as seen in the code below.

package main

import (
  "context"
  "log"
  "time"

  pb "ldtgrpc01/proto" // replace with your module name

  "google.golang.org/grpc"
  "google.golang.org/grpc/credentials/insecure"
)

func main() {
  conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(insecure.NewCredentials()))
  if err != nil {
      log.Fatalf("did not connect: %v", err)
  }
  defer conn.Close()

  c := pb.NewCalculatorClient(conn)

  ctx, cancel := context.WithTimeout(context.Background(), time.Second)
  defer cancel()

  // Make the gRPC call
  r, err := c.Add(ctx, &amp;pb.AddRequest{Num1: 5, Num2: 3})
  if err != nil {
      log.Fatalf("could not calculate: %v", err)
  }
  log.Printf("Result: %d", r.GetResult())
}</textarea>
package main

import (
  "context"
  "log"
  "time"

  pb "ldtgrpc01/proto" // replace with your module name

  "google.golang.org/grpc"
  "google.golang.org/grpc/credentials/insecure"
)

func main() {
  conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(insecure.NewCredentials()))
  if err != nil {
      log.Fatalf("did not connect: %v", err)
  }
  defer conn.Close()

  c := pb.NewCalculatorClient(conn)

  ctx, cancel := context.WithTimeout(context.Background(), time.Second)
  defer cancel()

  // Make the gRPC call
  r, err := c.Add(ctx, &pb.AddRequest{Num1: 5, Num2: 3})
  if err != nil {
      log.Fatalf("could not calculate: %v", err)
  }
  log.Printf("Result: %d", r.GetResult())
}

Once the client (‘c’ above) is created using NewCalculatorClient, it can be thought of as a local object instance of the calculator server and its methods (functions) are called as we would normally do. In the code above, observe the line where Add() function is being called. In this gRPC version, passing the number parameters is a bit different. Here we use the AddRequest Protobuf message (check the calc.proto file), to pass the same.

Once the Add() is called on the client code, this happens:

The client serializes the Protocol Buffer message
The message is sent over the network to the server
The RPC (Remote Procedure Call) includes metadata and the context
Server executes the “addition” logic and generates the return value/response
Server creates the response message
Response is serialized
Sent back over the network to the client
Client deserializes the response, continues it processing

Testing and Conclusion

To test this code, first run the calculator server code so that it listens on the port 50051, and then run the client code. The client simply calls the Add() function with hardcoded values 5 and 3. The calculator server processes this gRPC request and responds with the addition, as seen in the image below.