DEV Community: Homayoon Alimohammadi

gRPC Name Resolution & Load Balancing on Kubernetes: Everything you need to know (and probably a bit more)

Homayoon Alimohammadi — Tue, 06 Feb 2024 10:04:05 +0000

Lately, we’ve faced a peculiar problem in our company. To give you a bit of context, we maintain a fleet of hundreds of micro-services in various languages, mostly Python and Go, mainly communicating via gRPC on top of a couple of self hosted Kubernetes clusters.

The problem became apparent while we were trying to do zero downtime rolling updates. With a pretty basic Kubernetes deployment configuration like the one below (and to be honest, a little more care! Read here for more info) one might expect their services to stay roughly 100% available while performing the rolling update:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
     # ...
    spec:
     replicas: 5
       # ...
     template:
       # ...
     strategy:
       type: RollingUpdate
       rollingUpdate:
         maxUnavailable: 2
         maxSurge: 3

But we noticed that we’re facing temporary downtimes (less than a minute) almost every time that the rolling update was happening. Likewise, killing a subset of pods resulted in the same thing. It became apparent after a thorough investigation that the problem was how gRPC resolved service names. Let’s dive deep into it, shall we?

DISCLAIMER: I make lots of mistakes on a daily basis! If you notice any, please let me know.
This blog is a digest of many amazing blogs by amazing people. I’ll try to put a link to each one wherever possible. Also you can find links to all the references at the end of the blog as well as a Github Repo for a sample project regarding this subject.

Kubernetes Load Balancing

Kubernetes default load balancing happens at the connection level (per TCP connection, L3/L4). HTTP/1.1 introduced the concept of “Persistent Connections” which allows multiple request/response pairs to be sent over a single connection (one after the other). While this allows saving some resources, it appears troublesome while we consider how Kubernetes is going to load balance the calls from a client to a backend server.

Most likely in a Kubernetes environment a service is going to have multiple replicas in order to prevent having a single point of failure. Imagine our client and server communicate via HTTP/1.1. On the first call, a TCP connection is created between our client and **one of the many backend services **and request/response pairs travel back and forth on the same connection. This means that the rest of the backend replicas are left untouched, i.e. no load balancing at all!

In order to increase throughput, multiple TCP connections are created between client and backend replicas in order to reduce the waiting time of each request for other requests to complete. This means that if we assume a backend service replica (endpoint) is selected (roughly) at random for each TCP connection, and the load distribution is (roughly) uniform amongst different TCP connections, then the TCP connection based load balancing that Kubernetes offer might be “good enough” for our HTTP/1.1 client/server architecture.

Also, as a bonus point, configuring MaxConnectionAge for persistent connections can contribute to the load balancing as it can ensure connections are destroyed and recreated in smaller intervals (hence super low / super high load connections won’t stick around for too long).

gRPC Load Balancing on Kubernetes

What about gRPC? Since it’s on top of HTTP/2 and utilizes multiplexing and *streams *(i.e. sending requests and responses concurrently), it can perform amazingly well using only a single TCP connection without having to create multiple connections like HTTP/1.1 does. But what happens to do connection based load balancing in Kubernetes? Well, you guessed it! It’s non-existant!

In fact, some kind of load balancing does occur while creating this single TCP connection, but as soon as it’s created, since no more connections are going to be created, nothing is load balanced any further. To slowly work our way through the problem, let’s analyze what we are working with step by step.

Kubernetes Services

In Kubernetes, services that get assigned an IP address and port (e.g. ClusterIP) are just simply iptables rules. No process is listening on their IP and port. Kubernetes uses kube-proxy and iptables to direct traffic to pods without an actual load balancing process. iptables can be configured in a smart way to somehow act as a random pick load balancer like below:

Here are the rules:

Select pod 1 as the destination with a probability of 33%. Otherwise, move to the next rule
Select pod 2 as the destination with a probability of 50%. Otherwise, move to the next rule
Select pod 3 as the destination

Each pod has a 33% probability (1/3 to be exact) to get picked. All that said, Kubernetes load balancing is not round-robin or anything, it’s a random selection.

So when we use an IP assigned to a, let’s say, ClusterIP service, we are essentially looking up the iptables to rewrite the destination IP with another one belonging to a pod that the service abstracts.

In case of a gRPC communication, the iptables are invoked upon the first RPC since we need to establish a TCP connection. Although for further RPCs, no connection needs to be created, and so the iptables will not be invoked hence no load balancing occurs.

Load Balancing Options

It seems like that the Kubernetes default load balancing mechanism is not going to cut it for us, so let’s consider and evaluate other options:

Proxy (Server Side) Load Balancing

Clients send requests to a load balancer proxy, which then distributes the call to backend servers
Suitable for user-facing services with untrusted clients from the open internet
Backends report load to the LB for fair distribution

Client Side Load Balancing

Clients are aware of multiple backends and choose one for each RPC, using server load reports to inform decisions
Simpler configuration may not consider server load, relying on round-robin selection

this type of load balancing mostly has two variations:

Thick Client: Integrates load balancing logic, mantaining server availability, workload, and selection algorithms, often in conjuction with service discovery and other infrastructure libraries.

Lookaside Load Balancing: Offloads balancing to a special LB server, with clients querying for the best servers to use, allowing sophisticated algorithm implementtaion on the LB side while enabling simple client-side logic.

Now that we know options, let’s first choose to go with the the “Thick Client” one and see what we’re gonna need.

Implementing Client Side Load Balancing

Our implementation has two main parts. A name resolver and a load balancer. Let’s start with the name resolver.

Name Resolver

Name resolvers are responsible for resolving available endpoints for a given backend service. One of the most important things while implementing a resolver is to make sure endpoints are maximally up-to-date at any given point in time. This means that old endpoints (which their pods are deleted) should not be available in address pool while the new endpoint (which their pods are just created) should.

Let’s consider our options for the name resolver with regards to grpc-go:

Passthrough (default): Just returns the target provided by the ClientConn without any specific logic.
DNS: Periodically resolves a name to a list of endpoints and updates the ClientConn on each resolution.
Custom (more on this later)

How can we choose which name resolver we want to use? The name (and type) of a resolver is tightly coupled with what we provide in the target string:

    grpc.Dial("myservice.namespace.cluster.local") // target

The string above is called the target. The general format for a target is like this:

    [scheme]://[authority]/endpoint

The first example we provided does not have a Scheme nor an Authority. So the default values for both of them are going to be used. Let’s consider another examples which define schemes:

    grpc.Dial("dns:///myservice.namespace.cluster.local")
    // or 
    grpc.Dial("myScheme:///myservice.namespace.cluster.local")

In the first example dns is the scheme specified in the target while myScheme is specified in the second example.

All the name resolvers, whether they are implemented in grpc-go by default (e.g. passthrough or dns) use some kind of a slug as their unique identifier. As you might have guessed, the slug for the passthrough implementation is passthrough and the for dns, it’s dns.

Upon calling grpc.Dial, if we’ve specified a scheme in target, it will be used to find its corresponding resolver and that resolver is going to do the name resolution for the rest of our client lifecycle.

If the scheme is not specified in the target though, the defaultScheme is going to be used which is just a variable created somewhere in grpc-go.

    var defaultScheme = "passthrough"

    // default scheme can be changed and it should 
    // happen before grpc.Dial in order to have effect
    func SetDefaultScheme(s string) {
      defaultScheme = s
    }

How is this slug used anyway? Well this slug maps us to a certain Builder for that name resolver. So having the resolver slug leads us to the resolver Builder which will “build” the resolver for us (i.e. starts name resolution with that specific resolver).

    var m = make(map[string]Builder)

    // this function is used to search for the builder
    // which corresponds to our specified scheme in the target
    func Get(scheme string) Builder {
     if b, ok := m[scheme]; ok {
      return b
     }
     return nil
    }

    // if no scheme was provided or the scheme was unknown 
    // (the function above returned nil) we fall back to the default scheme
    func GetDefaultScheme() string {
     return defaultScheme
    }

So that’s how the name resolver is selected. Let’s see what’s the main duty of a every name resolver.

The resolver is missioned to update the state of a client connection like below:

    func (r *resolver) resolve() {
      NewState := gatherNewState()
      r.clientConn.UpdateState(NewState)
    }

The Updated state contains both the new (updated) endpoints as well as a ServiceConfig. ServiceConfig is something that will come handy in the next step, where we need to choose and implement a load balancing policy. Here is how the State struct looks:

    type State struct {
     // Addresses is the latest set of resolved addresses for the target.
     //
     // If a resolver sets Addresses but does not set Endpoints, one Endpoint
     // will be created for each Address before the State is passed to the LB
     // policy.  The BalancerAttributes of each entry in Addresses will be set
     // in Endpoints.Attributes, and be cleared in the Endpoint's Address's
     // BalancerAttributes.
     //
     // Soon, Addresses will be deprecated and replaced fully by Endpoints.
     Addresses []Address

     // Endpoints is the latest set of resolved endpoints for the target.
     //
     // If a resolver produces a State containing Endpoints but not Addresses,
     // it must take care to ensure the LB policies it selects will support
     // Endpoints.
     Endpoints []Endpoint

     // ServiceConfig contains the result from parsing the latest service
     // config.  If it is nil, it indicates no service config is present or the
     // resolver does not provide service configs.
     ServiceConfig *serviceconfig.ParseResult

     // Attributes contains arbitrary data about the resolver intended for
     // consumption by the load balancing policy.
     Attributes *attributes.Attributes
    }

In order to have the updated endpoints for our destination we need to utilized **Headless **services in Kubernetes. Since upon resolution, they will return the endpoints of their underlying pods instead of their own IP which will get replaced by one of the pod’s IP by the iptables.

It seems like we’re going to do just fine if we utilize a headless service with the dns resolver as our name resolver, right? Well it’s a bit tricky!

Remember that we said we need to have the most up-to-date version of endpoints in any given time? grpc-go built-in dns resolver has a tricky part that might counter act what we’re trying to achieve. Even after a successful DNS resolution, the resolver for wait for at least 30 seconds before attempting a re-resolution:

    var (
     // MinResolutionRate is the minimum rate at which re-resolutions are
     // allowed. This helps to prevent excessive re-resolution.
     MinResolutionRate = 30 * time.Second
    )

We’re hoping to make this rate at least optional via this pull request but as the time of writing this blog, it’s nothing we can do to circle our way around it.

Implementing a custom resolver is our last and probably most viable option to reach our goal of up-to-date endpoints. for this custom resolver, we can utilize Kubernetes endpoints controller in order to get notified about every endpoint change event and act upon them, as well as doing periodic DNS resolutions just to make sure.

To wrap up name resolvers, let’s consider them as “always running processes in the background” that are responsible for providing their underlying ClientConn with the latest “State” including updated endpoints as well as another thing called ServiceConfig (more on this one in the following section).

Load Balancing

The State which was returned by the name resolver in the previous steps contains both a list of endpoints and a configuration called ServiceConfig. One of the main use cases of the ServiceConfig is to decide which load balancing policy to go for (e.g. round_robin). ServiceConfig is based on a proto message but the name resolver returns it in JSON form. Let’s look at an example ServiceConfig:

The loadBalancingConfig is what we use in order to decide which policy to go for (round_robin in this case). This JSON representation is based on a protobuf message, then why does the name resolver returns it in the JSON format? The main reason is that loadBalancingConfig is a oneof field inside the proto message and so it can not contain values unknown to the gRPC if used in the proto format. The JSON representation does not have this requirement so we can use a custom loadBalancingConfig .

gRPC client can be configured to consider default service config upon dial as well:

    grpc.Dial(
      target,
      grpc.WithDefaultServiceConfig(`"loadBalancingConfig": [{ "round_robin" }]`),
    )

ServiceConfig can also be encoded in a DNS TXT record and get resolved by the resolver as well:

    grpc_config.myserver  3600  TXT "grpc_config=[{\"serviceConfig\":{\"loadBalancingPolicy\":\"round_robin\",\"methodConfig\":[{\"name\":[{\"service\":\"MyService\",\"method\":\"Foo\"}],\"waitForReady\":true}]}}]"

After selecting the load balancing policy via the ServiceConfig, we need to know the basic duties and requirements of a load balancer.

Load Balancing policies are responsible for receiving server addresses and configurations, managing sub-channels, setting channel connectivity state (more on this later) and determining a sub-channel for each RPC.

Two of the built-in load balancers in grpc-go are pick_first andround_robin .

Connectivity State

gRPC channels are an abstraction that describe the overall state of a connection from origin to destination (e.g. service A to service B). Each channel contains a number of sub-channels each one having a so called “Connectivity State” which can be described by a state machine and has the following states:

CONNECTING: Channel is attempting to establish a connection, including name resolution, TCP, and TLS handshakes
READY: A successful connection has been established, and the channel can facilitate communication
TRANSIENT_FAILURE: Temporary failures have occurred, prompting the channel to retry connection establishment with exponential backoff
IDLE: The channel is not actively attempting to establish a connection due to lack of RPC activity. New RPCs can transition the channel out of this state
SHUTDOWN: The channel is shutting down, either by user request or due to a non-recoverable error, and no new RPCs will be initiated

Depending on the implementation of the load balancer, the channel will evaluate its overall state by aggregating states from all its sub-channels.

So let’s wrap up what we’ve learned about load balancers. Via ServiceConfig we can decide which policy to use. The implementation is tasked with obtaining updated endpoints from the name resolver, creating and managing TCP connections with each one of them and route a given RPC upon arrival to one of those connections based on a specific logic.

gRPC Client Initialization Flow

The pictures above depict how the gRPC client initialization happens under the hood.

Availability in rolling updates

Another thing to keep in mind, which to be honest does not relate directly to the gRPC load balancing is how a rolling updates occur and how they can affect availability.

You might notice temporary downtimes while performing rolling updates even if you’ve configured everything as we pictured just fine. That’s because the signal to change the iptables and exiting the process happens in parallel (we might get unlucky)

Try it yourself

As pictured in the sections above, proper gRPC load balancing can be achieved via different ways (e.g. Thick Client or Proxy LB). In this GitHub repo there is a sample project which you can play around with and see for yourself how all these pieces of puzzle come together.

It requires you to have a Kubernetes cluster up and running for which I recommend using Kind.

There’s a couple of HTTP/1.1 services, one acting as the client and the other one acting as the server. You can observe the logs to see that HTTP/1.1 is actually load balanced well enough on Kubernetes.

On the other hand, a pair of gRPC services are available that act exactly the same as their HTTP/1.1 counter parts, except the fact that they communicate via gRPC. Ramp up the replicas of the server and witness that client calls are not balanced at all (all land on the same pod). You can monitor what’s happening under the hood thanks to channelz as well (it’s implemented in the gRPC services)

From here, you can install and inject LinkerD to your gRPC deployments (as described in the README instructions) and after that it becomes apparent that LinkerD is doing an amazing job load balancing the requests to all the available pods of the server.

Conclusion

Load balancing gRPC requests on Kubernetes can be challenging. In this blog we tried to deep dive into the inner-workings of each part and find how everything work and why they sometimes don’t.

If you’ve liked this blog, a share and a clap would really mean a lot to me. Also you can find links to all the references used below.

Feel free to contact me via LinkedIn and Twitter. Also you can find more about me on my personal blog.

References

Protoc Plugins in Go: gRPC-REST Gateway From Scratch

Homayoon Alimohammadi — Sun, 07 Jan 2024 11:38:45 +0000

gRPC-REST gateway is one of the most popular projects in the gRPC-Ecosystem. Since I really like to understand how things work under the hood and how they are implemented, I decided to try and build a minimal gRPC-REST Gateway from scratch. If you are interested as well, make sure to read through the article as I’ve learned a bunch while trying to code it up.

I tend to make lots of mistakes on a daily basis, so if you’re one of the many that encountered one, I’d be grateful if you would correct me.

Introduction

Choosing a gRPC-powered backend is not an uncommon thing nowadays, considering all the benefits it might provide if used in the proper environment, especially in a micro-service architecture. But what if you’re planning to migrate your currently REST backend services to gRPC? You’re probably going to face some issues. Since gRPC relies on HTTP trailers to operate, it’s a bit tricky to call gRPC backend services from a JS frontend client. Feel free to read further about the curious, yet questionable use of HTTP Trailers in the design of gRPC in Carl’s Blog, as well as Akshayshah’s. You probably don’t have many options. The best one might be the gRPC-Web project which enables you to proxy an HTTP/1.1 request (made by gRPC-Web) to an HTTP/2-gRPC backend through an Envoy proxy (more on this in the gRPC-Web official documentation). But for various reasons (e.g. migration, development and maintenance costs) you might not have the luxury to do so. What are you options then? Introducing gRPC-REST Gateway.

gRPC-REST Gateway

What if you could deploy an HTTP/1.1 server, accepting REST requests from the frontend, converting them to gRPC requests, calling the gRPC services and returning their responses in a format that the REST clients will expect? It looks pretty similar to the gRPC-Web request flow, but now your frontend is completely oblivious about anything gRPC-related happening behind the curtains. Removing the need for an Envoy proxy or any proxy for that matter, it will arguably reduce the related costs as well.

In the incoming sections, we’re going to implement a minimal version of the well-known gRPC-REST Gateway. We will be using a top-down approach as I personally understand details and reasonings easier and better when I know the high-level concepts and requirements.

Implementation

The first thing that we might ask ourselves, is what behaviour do we expect from our mini gRPC-REST gateway? Let’s say there is a gRPC backend running on port 50051 (this is easily generalizable to a service running in a Kubernetes cluster). Here’s a .proto file as well as the code needed for a simple gRPC server:

    syntax = "proto3";

    package post;

    option go_package = "github.com/HomayoonAlimohammadi/mini-grpc-gateway/pb/post";

    message Empty {}

    message GetPostResponse {
        string title = 1;
        string description = 2;
    }

    service PostService {
      rpc GetPost(Empty) returns (GetPostResponse);
    }

First of all, yes, I know there is a google.protobuf.Empty message and I don’t need a custom Empty{} message, but it will result in an easier implementation later on. Our service description has nothing speciall. Just a GetPost RPC without any input (which is not realistic at all, but again, let’s keep it simple) and a simple response. Let’s briefly look at the Go implementation of this service as well:

    package main

    import (
     "context"
     "fmt"
     "log"
     "net"

     "google.golang.org/grpc"
     "google.golang.org/grpc/reflection"

     "github.com/HomayoonAlimohammadi/mini-grpc-gateway/pb/post"
    )

    type serverImpl struct {
     post.UnimplementedPostServiceServer
    }

    func main() {
     lis, err := net.Listen("tcp", ":50051")
     if err != nil {
      log.Fatalf("failed to listen: %v", err)
     }

     grpcServer := grpc.NewServer()
     post.RegisterPostServiceServer(grpcServer, newServer())
     reflection.Register(grpcServer)

     fmt.Println("running server on :50051")
     log.Fatal(grpcServer.Serve(lis))
    }

    func newServer() *serverImpl {
     return &serverImpl{}
    }

    func (s *serverImpl) GetPost(ctx context.Context, _ *post.Empty) (*post.GetPostResponse, error) {
     return &post.GetPostResponse{Title: "some title", Description: "some description"}, nil
    }

Again, nothing special here, just a repetitive response for any given request.

What we might expect is being able to make an http/1.1 call to an endpoint and receive a response from this gRPC service:

    curl localhost:8000/api/post

    >>> {"title": "some title", "description": "some description"}

But what is powering this behaviour? Let’s have a sneak peek on the final implementation of our Protoc plugin:

    func main() {
     r := mux.NewRouter()

     PostServiceHandler, err := GetPostServiceHandler()
     if err != nil {
      log.Fatalf("failed to get PostService handler: %v", err)
     }

     r.HandleFunc("/api/post", PostServiceHandler.GetPostHandler)

     srv := &http.Server{
      Handler:      r,
      Addr:         ":8000",
      WriteTimeout: 15 * time.Second,
      ReadTimeout:  15 * time.Second,
     }

     log.Print("serving on :8000")
     log.Fatal(srv.ListenAndServe())
    }

Let’s see what’s going on here:

A new router is created (or we could just use the http.HandleFunc and skip this step).
A handler struct is initialized for a given gRPC service (since we’re better off having the same underlying gRPC client for each service, more on this later).
A new route with a matcher for the URL path is registered.
Server is configured and started.

All the magic is happening in the GetPostHandler:

    func (sh *postService) GetPostHandler(w http.ResponseWriter, r *http.Request) {
     ctx := r.Context()
     in := makeGetPostRequest()

     resp, err := sh.client.GetPost(ctx, in)
     if err != nil {
      writeError(
       w, fmt.Errorf("failed to call GetPost: %w", err),
       http.StatusInternalServerError,
      )
      return
     }

     b, err := protojson.Marshal(resp)
     if err != nil {
      writeError(
       w, fmt.Errorf("failed to convert response to json: %w", err),
       http.StatusInternalServerError,
      )
      return
     }

     w.Write(b)
    }

    func makeGetPostRequest() *post.Empty {
     return &post.Empty{}
    }

    func writeError(w http.ResponseWriter, err error, status int) {
     w.WriteHeader(status)
     w.Write([]byte(err.Error()))
    }

Let’s review what happens in the code snippet above:

gRPC Request Proto message and context is prepared and the request is made.
The Response is marshaled into JSON and sent back to the REST client.

It’s probably a good idea to have a single gRPC client for any given gRPC backend service (or a client pool if faced with an immense load or complex load balancing). So let’s initialize our service handler like below:

    type postService struct {
     client post.PostServiceClient
    }

    func GetPostServiceHandler() (*postService, error) {
     conn, err := grpc.Dial(":50051", grpc.WithTransportCredentials(insecure.NewCredentials()))
     if err != nil {
      return nil, fmt.Errorf("failed to dial 50051: %w", err)
     }

     return &postService{
      client: post.NewPostServiceClient(conn),
     }, nil
    }

It seems like we have everything we’ll ever need for a proper gRPC-REST gateway. There is just one huge problem: Are we going to write and maintain it all manually? What if a service is removed? What if one is added? What happens if an endpoint changes? Or when a service port is changed?

The solution is obvious: code generation. But how? Well, a Protoc Plugin might the best fit since it is both self-documenting and easy to maintain. Again, with a top-down approach, let’s see what our desired .proto file might look like:

    import "google/protobuf/descriptor.proto";

    service PostService {
      rpc GetPost(Empty) returns (GetPostResponse) {
        option (mini_grpc_gateway_options) = {
            url: "/api/post"
        };
      }
    }

    message MiniGRPCGatewayOptions {
        string url = 1;
    }

    extend google.protobuf.MethodOptions {
        MiniGRPCGatewayOptions mini_grpc_gateway_options = 1234562;
    }

Notice that there is only a minor difference in the service definition here. A custom mini_grpc_gateway_options is added to the only RPC, denoting a url that is likely the endpoint that we’re going to use to fetch a GetPost response. In order to create this custom option, we need to extend the google.protobuf.MethodOptions message and add our custom option as a new field. The very large field tag is there to ensure our custom option does not overwrite any other fields in the original message.

Next, we need to parse and extract the needed information from the .proto file in order to generate the output. Achieving it through Protoc is pretty straight forward. First we need to build the executable binary for our plugin and then put it somewhere in our $PATH.

    func main() {
     var flags flag.FlagSet

     protogen.Options{
      ParamFunc: flags.Set,
     }.Run(func(gen *protogen.Plugin) error {
      conf, err := config.ExtractServiceConfig(gen, config.ServicesConfigJSON)
      if err != nil {
       panic(err)
      }

      export.Export(gen, conf)

      return nil
     })
    }

    go build -o $(GOPATH)/bin/protoc-gen-mini-grpc-gateway main.go

Now we can use our custom plugin just like any other Protoc plugin (e.g. protoc-gen-go):

    protoc --go_out=./pb/post --go_opt=paths=source_relative \
     --go-grpc_out=./pb/post --go-grpc_opt=paths=source_relative \
     --mini-grpc-gateway_out=./pb/post service.proto

I’m not going to walk through the rest of the generation process as I’ve covered the intricacies of writing a custom Protoc plugin in one of my previous blogs: Protoc Plugins in Go. Feel free to check it out as well. The source code also contains the complete plugin as well.

The main idea is to extract what ever we need from the .proto files as well as a service configuration source (e.g. a JSON file or maybe a centralized config, controlled and maintained via a control plane in a Kubernetes cluster) for our gRPC clients to be configured correctly. Given that information, we should be able to generate handlers and clients for every method of every service annotated with our custom MiniGRPCGatewayOptions method option.

Note that for the sake of simplicity, I skipped arguably pretty important features. Feel free to contribute to the project with any of the following or your own idea:

Treating URL params as input for our backend gRPC call
Using parametric path in order to enrich backend gRPC call input (e.g. /api/post/{token})
Limiting the JSON response to only specific gRPC response fields.

Conclusion

gRPC-REST Gateway is an interesting, yet pretty intuitive piece of software. I really enjoyed implementing a simple version of it. I wish you’ve enjoyed and hopefully learned something as well. Feel free to contact me in any means (the comments, my LinkedIn and Twitter).

References

Source Code (Github Repository): https://github.com/HomayoonAlimohammadi/mini-grpc-gateway
Carl Mastrangelo — Why does gRPC insist on Trailers: https://carlmastrangelo.com/blog/why-does-grpc-insist-on-trailers
Akshayshah — gRPC Doesn’t Need Trailers: gRPC Doesn’t Need Trailers: https://akshayshah.org/grpc-doesnt-need-trailers
gRPC-Web: https://github.com/grpc/grpc-web

Golang: to Point or not to Point!

Homayoon Alimohammadi — Tue, 29 Aug 2023 16:54:28 +0000

Pointers have numerous benefits and to imagine coding without them, let’s say it’s not so pleasant. But they do have cons, one of them (at least for me) might be that they can cause confusion and may even break the code if not used with caution. I think we’ve all experienced when things just did not work as intended, and after a debugging session found out that a pointer caused the unexpected behavior. But after all making a decision about whether or not to pass (or in general, use) a pointer is not so easy for me. So, I decided to study and learn from people much smarter and experienced than me. Hopefully sharing these findings might come interesting to you as well.

This article is heavily inspired by **Hossein Nazari* in their invaluable article titled “Whether or not to use pointers while passing structs”, published on Gocast (FYI: it’s in Persian).*

One of the main deductions that the original author made was:

Set your default to not using pointers, unless you know for a fact that its fields are going to change.

Passing by “value” vs “reference”

Go passes everything by value. Even pointers get copied when passing them to a function, yet the copy still represents the same underlying memory and so we are able to change the underlying value.

First, let’s take a look at some details revolving around pointers. An interesting article emphasizes on the fact that go is pass-by-value but there are certain times when that doesn’t sound like the case. There are primitive types, e.g. int, string, byte, rune, bool that are considered “value” types whereas pointer types are “reference” ones. But there are certain types that have somewhat of a different nature, as the article puts:

However map, slice and channel are all special types that *are *or *contain *references.

But the interesting thing about these types is that they all point to some other underlying data and even if you pass them (seemingly) by value in an arbitrary function, changing their content in the function will change their original value. To give an example:

    package main

    import "fmt"

    func changeKey(m map[string]string) {
        m["key"] = "new_value"
    }

    func main() {
        m := map[string]string{"key": "value"}
        fmt.Println(m) // map[key: value]
        changeKey(m)
        fmt.Println(m) // map[key: new_value]
    }

So it seems like the maps are references, though in fact they’re not. Take a look at this example:

    package main 

    import "fmt"

    func new(m map[string]string) {
        m = make(map[string]string)
    }

    func main() {
        var m map[string]string
        new(m)
        fmt.Println(m == nil) // true
    }

If the map m was a C++ style reference variable, the m declared in main and the m declared in newwould occupy the same storage location in memory. But, because the assignment to m inside newhas no effect on the value of m in main, we can see that maps are not reference variables.

A map reference is a pointer to a map header struct (of type hmap) in memory which contains various fields, including the number of cells and some other things.

Copying a map or passing it as a value doesn’t actually copy the underlying hashmap — it merely copies the pointer to the header struct. Therefore, copying a map reference is just giving you multiple references to the same underlying map in memory. It is the map reference that is being passed-by-value, not the contents of the map

The final result is depicted here as well:

First, Go technically has only pass-by-value. When passing a pointer to an object, you’re passing a pointer by value, not passing an object by reference. The difference is subtle but occasionally relevant. For example, you can overwrite the pointer value which has no impact on the caller, as opposed to dereferencing it and overwriting the memory it points to.

Performance

One of the major concerns while using pointers is performance. And it seems that using a pointer as a way to improve the overall performance and memory efficiency of a piece of code is not always correct.

Backed up with proper benchmarking, Mario Macías suggests an interesting point about returning structs vs pointers:

Despite copying a struct with several fields is slower than copying a pointer to the same struct, returning a struct value may be faster than returning a pointer if we consider escape analysis particularities.
Returning structs allows the compiler to detect that the created data does not escape the function scope and decides to allocate the memory in the stack, where the allocation and deallocation is very cheap if compared to managing memory in the heap.

Also this article points out that in order to justify sharing a pointer to a struct it should be particularly big (how big you might ask? I’m not sure, maybe a few hundred bytes?) and its life-cycle should be reasonably large. And there is another benchmark suggesting that you rarely need to share a struct by its pointer.

It turns out that only extremely large structs can cause slower performance compared to pointers.
If I choose to use a pointer, the variable itself should hold a valuable purpose for that entire duration.

With heap allocations the garbage collector causes brief pauses form time to time (for a couple of milliseconds on average) to clean that up. Since we’re talking about stack and heap, it’s worth noting that accessing the stack is particularly faster, as describe here:

The stack is faster because the access pattern makes it trivial to allocate and deallocate memory from it (a pointer/integer is simply incremented or decremented), while the heap has much more complex bookkeeping involved in an allocation or deallocation. Also, each byte in the stack tends to be reused very frequently which means it tends to be mapped to the processor’s cache, making it very fast. Another performance hit for the heap is that the heap, being mostly a global resource, typically has to be multi-threading safe, i.e. each allocation and deallocation needs to be — typically — synchronized with “all” other heap accesses in the program.

So how does this concern us? Well, it turns out that what we do with return values or input parameters of a function (i.e. deciding whether or not to use a pointer) may result in allocations. This process is thoroughly and beautifully describe by James Kirk in their article:

A general rule we can infer from this is that sharing pointers up the stack results in allocations, whereas sharing points down the stack doesn’t.

We’re going to see more about stack and heap in the following section.

At least for me, one of the major things that I’ve seen many times is that if a team is wishing to evolve and improve their many services, projects and overall code base overtime, it’s almost crucial to prioritize the readability of the code. Writing something that is idiomatic and can be reasoned about fairly easily seems to be a more successful approach rather than trying to achieve even the tiniest optimizations at the cost of the code becoming an unreadable mess. In Golang, generally writing idiomatic and readable code seems to be of great importance and value. William Kennedy described this idea of pointers and probable performance trade-offs in this article:

I am asked quite a bit about when and when not to use pointers in Go. The problem most people have, is that they try to make this decision based on what they think the performance trade-off will be. Hence the problem, don’t make coding decisions based on unfounded thoughts you may have about performance. Make coding decisions based on the code being idiomatic, simple, readable and reasonable.

He later adds that in general if we are to group implementation as either a “primitive data value” or a “to be changed” it’s wise to share the changing one with a pointer:

If you review more code from the standard library, you will see how struct types are either implemented as a primitive data value like the built-in types or implemented as a value that needs to be shared with a pointer and never copied. The factory functions for a given struct type will give you a great clue as to how the type is implemented.
In general, share struct type values with a pointer unless the struct type has been implemented to behave like a primitive data value.

And also:

If you are still not sure, this is another way to think about. Think of every struct as having a nature. If the nature of the struct is something that should not be changed, like a time, a color or a coordinate, then implement the struct as a primitive data value. If the nature of the struct is something that can be changed, even if it never is in your program, it is not a primitive data value and should be implemented to be shared with a pointer. Don’t create structs that have a duality of nature.

By the way I’m not really sure if I can fully understand the very last sentence. “duality of nature” doesn’t seem like a well defined thing to me. To be honest most of the structs with a “changing nature” that I’ve seen so far kind of had their own consistent and unchanging inner state as well. To get you an example of that, consider a struct explaining a vehicle on its way from an origin to a destination (my apologies if this example turned out to be kind of half-baked and not so true, it’s just an example):

    package main

    type Point struct {
        Lat float64
        Long float64
    }

    type Vehicle struct {
        id string
        Origin Point
        Destination Point
        GasolineLeft float64
    }

I’m not saying that he’s wrong, not at all, all I’m saying is that this duality of nature is something that might be observed here and there, and that’s fine (you can argue that the design above can be altered in a way that maybe origin and destination are stored somewhere along with the id or something, but, you get the point). My own take on that article is to use a pointer for the structs that are going to change even slightly.

And the reference types (e.g. map, slice, chan, interface and function values) happen to be shared with a pointer only in rare scenarios, and if you’re not sure, just don’t use a pointer.

Reference types are slices, maps, channels, interface and function values. These are values that contain a header value that references an underlying data structure via a pointer and other meta-data. We rarely share reference type values with a pointer because the header value is designed to be copied. The header value already contains a pointer which is sharing the underlying data structure for us by default.
If you review more code from the standard library, you will see how values from reference types in most cases are not shared with a pointer. Since the reference type contains a header value whose purpose is to share an underlying data structure, sharing these values with a pointer is unnecessary. There is already a pointer in use.
In general, don’t share reference type values with a pointer unless you are implementing an unmarshal type of functionality.

Stack vs Heap

We’ve seen that in general sharing the pointers up the stack results in heap memory allocations. Let’s dive deeper into that.

This section is inspired by Jacob Walker, in their video “The Stack and the Heap — GopherCon SG 2019” publish on YouTube.

There are two kinds of memories, Stack and Heap, and with go, we have multiple stacks (i.e. a stack for each goroutine), and a heap which basically houses everything else that is not in the stacks.

While we can’t really tell by ourselves if a variable is going to be on the stack or on the heap, the main question is “Does it really matter?”.

For the correctness of a program, NO. But it does effect the performance of a program, because everything on the heap is managed by the garbage collector, which causes some latency for the whole program.

First of all, let’s make sure that we are not excessively spending time on over engineering and optimizing our program while we see no real need nor a benchmark proving a poor performance. In other words, if the program is “fast enough”, just let it be, there are probably other more important things you can work on.

Don’t optimize in the dark.

If you finally decided on optimizing your program, optimize for correctness first, not performance. All that being said, let’s study a couple of different scenarios to figure out how it’s decided whether to put a variable on the stack or on the heap.

— — —
*Disclaimer:

GOVERSION: go1.20.1
GOOS: linux
GOARCH: amd64
Inlining optimizations are disabled — — —*

Consider the example below where there is no pointers involved:

    package main

    func main() {
        n := 4
        n2 := square(n)
        println(n2)
    }

    func square(x int) int {
      return x * x
    }

Why did I use println instead of fmt.Println? Just keep reading!

When starting the program, a stack “frame” is created for the func main with the values n = 4 and n2 = 0 . When the func square gets called, another frame is created for that function which houses x = 4 inside of it. After func square returns, the result n2 = 16 is going to be set in the first stack frame. Go does not clean up after itself, the second stack frame is still available in the stack and what go does is it keeps track of what is valid and what is invalid (the black line you can see in the image below). Finally when println is called a new frame is created for it with a = 16 and this new stack is placed in the valid section. The line moves up and down in order to define valid vs invalid area. This is why stacks are considered “self cleaning”, any variable on the stack is cleaned up as that space is reused (i.e. the addresses of x and a can be the same, and in fact they are).

Now, let’s add some pointers:

    package main

    func main() {
        n := 4
        inc(&n)
        println(n)
    }

    func inc(x *int) {
        *x++
    }

Again, walking through this code in memory and skipping to the interesting part, when func inc is called, a new stack frame is created for it containing x = 0xc000044780 which actually points to the n = 4 in the first stack. When the pointer to n is dereferenced and incremented, the value of n in the first stack is updated and the second stack just get pushed into the invalid section. When reaching println, this function reclaims the space formerly owned by func inc and the process is continued as before. We are using pointers here but it was able to stay in the stack.

Sharing down typically stays on the stack.

Returning pointers?

    package main

    func main() {
        n := answer()
        println(*n/2)
    }

    func answer() *int {
        x := 42
        return &x
    }

When we first call main it sets n = nil to begin with (in its original stack frame), as the func answer is called, the compiler already knows that it is not safe to put the variable x (created by func answer) on the stack frame, so instead x is declared somewhere on the heap. “Why?” you might ask, if the variable was going to be declared on the stack frame, then n would have to point to a value in the invalid section (that value being x = 42 and n would be something like 0xc000044770 and that would cause problems. We say that “x escaped to the heap”, but it doesn’t actually get moved at runtime, this happens at compile time. Compiler initially knows that this variable is going to be constructed on the heap.

Sharing up typically escapes to the heap.

The word “typically” both here and in the quote above, means that actually, Only the compiler knows.

According to the Golang FAQ:

When possible, the Go compilers will allocate variables that are local to a function in that function’s stack frame. However, if the compiler cannot prove that the variable is not referenced after the function returns, then the compiler must allocate the variable on the garbage-collected heap to avoid dangling pointer errors.

What we’re talking here is called “escape analysis”. It’s what compiler does, looking at our code to see if any of these variables need to be on the heap.

In the current compilers, if a variable has its address taken, that variable is a candidate for allocation on the heap. However, a basic escape analysis recognizes some cases when such variables will not live past the return from the function and can reside on the stack.

So if only the compiler knows, let’s ask it!

    go help build
          usage: go build [-o output] [build flags] [packages]
          ...
          -gcflags '[pattern=]arg list'
                 arguments to pass on each go tool compile invocation.

We use go build to build our programs, but the compiler actually is the go tool compile.

    go tool compile -h
            usage: compile [options] file.go...
            ...
            -l disable inlining
            ...
            -m print optimization decisions


    # Example 1
    go build -gcflags "-m -l" example1.go
    # no output, since inlining is disabled

    # Example 2
    go build -gcflags "-m -l" example2.go
    ./example2.go:9:10: x does not escape

    # Example 3
    go build -gcflags "-m -l" example3.go
    ./example3.go:9:2: moved to heap: x

Here we can see that the compiler decides whether or not to put a variable on the heap at compile time.

But there’s a catch! You may have noticed that instead of fmt.Println which is arguably the common way to print out a variable, I considered using println. “Why is that?” you might ask, well, while playing out with fmt.Println I noticed something weird (to replicate these results, just replace println with fmt.Println in the examples above):

    # Example 1
    go build -gcflags "-m -l" example1.go
    ./example1.go:8:13: ... argument does not escape
    ./example1.go:8:14: n2 escapes to heap

    # Example 2
    go build -gcflags "-m -l" example2.go
    ./example2.go:11:10: x does not escape
    ./example2.go:8:13: ... argument does not escape
    ./example2.go:8:14: n escapes to heap

    # Example 3
    go build -gcflags "-m -l" example3.go
    ./example3.go:11:2: moved to heap: x
    ./example3.go:7:13: ... argument does not escape
    ./example3.go:7:17: *n / 2 escapes to heap

It seems to be an open issue on Golang GitHub:

So let’s wrap up. When are values constructed on the heap?

When a value could possibly be referenced after the function that constructed the value returns.
When the compiler determines a value is too large to fit on the stack.
When the compiler doesn’t know the size of a value at compile time (i.e. in the case of a slice).

Some commonly allocated values (not exhaustive):

Values shared with pointer.
Variables stored in interface variables.
func literal variables (or anonymous functions).
Backing data for map, channel, slices and string (strings are effectively a special slice of bytes).

Which one to choose?

After all the discussions above, it might still be unclear, let’s say, whether to pass a copy of a struct or the pointer to it, or to use a pointer receiver or not. Hence, let’s review some opinions from various sources.

An overall conclusion is perfectly depicted here:

Methods using receiver pointers are common; the rule of thumb for receivers is, “If in doubt, use a pointer.”
Slices, maps, channels, strings, function values, and interface values are implemented with pointers internally, and a pointer to them is often redundant.
Elsewhere, use pointers for **big structs **or structs you’ll have to change, and otherwise pass values, because getting things changed by surprise via a pointer is confusing

Another informative discussion can be found here:

Is the struct reasonably large? Prematurely optimizing is rarely good, but if you have a struct with more than a handful of fields and/or fields containing large strings or byte arrays (e.g. a markdown body) the benefits to returning a pointer instead of a copy becomes more apparent.
What’s the risk of the returning function mutating the struct (or object) after it returns it? (i.e. some long-running task or job)
I nearly always return a pointer from a constructor as the constructor should be run once.

Also in the same discussion William Kennedy pointed out:

I have a bit of a different philosophy. Mine is based on making a decision about the type you are defining. Ask yourself a single question. If someone needs to add or remove something from a value of this type, should I create a new value or mutate the existing value. This idea comes from studying the standard library.

There are certain situations in which we (mistakenly) assume that a poor performance is mainly due to not utilizing pointers well, but with a little bit of more evaluation, we figure out a new way of doing things (e.g. a better design) in order to improve the performance of the software in a readable and idiomatic way, without adding to the complexity and confusion of the code by introducing unnecessary pointers. Also as the Zen of Go suggests:

If you think it’s slow, first prove it with a benchmark. In practice, I have never had to care about the level of performance where the difference between a pointer and a non-pointer mattered. I believe that the instances where that might matter are very rare.

Summary

In this article (actually my first one as well!) I tried to explore the mysterious area of pointers and to answer the question of whether or not to use a pointer.

My own deduction with regards to the references above is to try not using a pointer. If faced with poor performance, first just prove it with a benchmark and don’t make unfounded assumptions about performance. If you’ve made sure that you actually *do *have performance issues, reevaluate the code in the hope of coming up with a better design. If failed to do so, or for whatever reason that design change didn’t seem reasonable, then I guess you have no choice but to use a pointer.

Regardless, there are certain situation in which it seems reasonable to use a pointer. Some of the more common ones being:

You’re dealing with a changing struct.
You’re dealing with a mutex, which is somehow the same idea as a changing struct (i.e. it’s constantly getting locked and unlocked).
Your struct is particularly big (maybe in case of a config struct or something, with an imaginary threshold of like a few hundred bytes).

We’ve seen that despite the “memory efficient” nature of pointers (since they prevent a whole struct getting copied, instead only the pointers themselves are getting copied), if not used properly, they tend to generate a lot of overhead and cause performance degradation as well as introducing unnecessary complexity and confusion.

Finally, I wanted to thank you for reading this blog, and hopefully that wasn’t a total waste of your time! I’d love to learn more technical stuff from you guys and would love to hear your suggestions and opinions about my writings. You can reach me via LinkedIn and Gmail.

Circuit breakers (in Go)

Homayoon Alimohammadi — Wed, 23 Aug 2023 19:55:50 +0000

Circuit breakers are a fundamental and basic, yet vital part of a system, especially if you are dealing with a micro-service architecture. Recently one of my colleagues was wondering what these ‘proxy like’ pieces of code do, and while explaining to him, I figured although I kind of know what a circuit breaker is at a high level, I don’t really know what is going on under the hood. So I decided to dig deeper and share what I’ve found. In this article we will both learn about circuit breakers at a high level, as well walking through a basic implementation of the circuit breaker pattern in Go.

Introduction

Imagine “Service A” is trying to contact “Service B” (as the server) via an RPC. From the stand point of Service A, it kinda looks like this:

    package service_A

    func callServiceB(req Request) {
      resp, err := serviceB.SomeMethod(context.Background(), req)
      if err != nil {
        log.Printf("got error: %s", err.Error())
        return
      }

      return log.Printf("got response: %s", resp)
    }

Well, this is probably the most basic way to make a call to another service, but there are a number things that can (an will) go wrong. Let’s say we did not receive any response from Service B, what are the possible reasons for that? One of them (which is not necessarily our concern right now) is that since we’re mostly dealing with unreliable network connections, it’s quite possible that the request is never processed by Service B. In this case we won’t solely rely on TCP retransmission, in fact most RPC frameworks provide their own retry mechanisms and it will be mostly handled in the application layer. But the fact that there was no response from the server does not always mean that there was a network issue, maybe there is something wrong with Service B.

In general, services (even the most reliable ones) tend to get slow or unavailable from time to time. This might result in an error in response to some (or all) of the incoming calls. If a service is really busy processing lots of requests, we might want to limit our request rate to that service, or even stop sending any requests for a while, letting it to take a breath and do whatever it was doing in peace.

In another case, let’s say the service was unavailable for a while and our previously failed requests (let’s say 100 requests) are being retried one after another. If we don’t limit the number of our future requests to that service, and make an additional 100 requests, when the service eventually recovers it will be faced with 200 requests at the same time which may even cause the service to immediately crash once again.

Although in the real world most RPCs are made with a specific timeout and won’t be retried afterwards, it is still a good practice not to flood a service with requests while it has problems responding.

Another positive point of this limiting mechanism for us is that we don’t bother making any request to that service (for a while), hence no time consuming I/O. This is definitively faster than just waiting for a while and then receive the failed result (like a 5xx status code).

Circuit Breakers

In simple terms, an RPC between two services is like drawing 2 straight lines between the two. One line sends the request from service A to service B, and the other returns the response from service B to service A. But in order to implement that “limiting” policy, we need to have some kind of a middle-man which decides whether to direct a request to the destination or not.

This middle-man, proxy (not a network proxy) or wrapper is either going to let the circuit (or connection) between two services “Closed”, or stop one from calling the other, hence “Opening” the circuit.

The main idea behind a circuit breaker is as follow:

By default the circuit is in the Closed mode and you are allowed to make calls freely to the destination. after a certain amount of failed responses from the destination (a threshold, let’s say 5), it is going to stop you from making any further requests for a while (a backoff time, like 30 seconds) in which the circuit is considered Open. **After that interval is over, It goes into a **Half-Open state. If the next request is going to determine whether we’re going to end up in the closed staet or get back in the open state. If successful, the circuit is going to be closed, but if the request fails we will be back in the open state and forced to wait for another back-off interval.

Let’s see how a simple circuit breaker is implemented in Go.

Implementation

From the stand point of the service which is using a circuit breaker in order to make requests to another service, making the RPC is a bit different:

All the code you see below can be found in this github repo as well.

    package service_A

    func callServiceB(cb CircuitBreaker, req Request) {
      r, err := cb.Execute(func() (interface{}, error) {
        return serviceB.SomeMethod(req)
      })
      if err != nil {
        log.Printf("got error: %s", err.Error())
        return
      } else {
        resp, ok := r.(serviceB.Response)
        if !ok {
          log.Println("type assertion failed")
          return
        }

        log.Printf("got response: %s", resp)
      }
    }

Let’s take a look at what the cb.Execute does:

    package circuitbreaker

    func (cb *circuitbreaker) Execute(req func() (interface{}, error)) (interface{}, error) {
     err := cb.doPreRequest()
     if err != nil {
      return nil, err
     }

     res, err := req()

     err = cb.doPostRequest(err)
     if err != nil {
      return nil, err
     }

     return res, nil
    }

The actual request is going to happen on line 8, res, err := req(), before which there should be a beforeRequest functionality:

    func (cb *circuitbreaker) doPreRequest() error {
     if cb.state == open {
      return ErrRefuse
     }

     return nil
    }

This checks to see whether the current state is open or not. If the circuit is open, simple return an error indicating that the connection is refused. Let’s look at the structure of our circuit breaker so far:

    type State string

    const (
     open     State = "open"
     closed   State = "closed"
     halfOpen State = "half-open"
    )

    type circuitbreaker struct {
     state State
    }

Fine, what about the afterRequest ?

    func (cb *circuitbreaker) doPostRequest(err error) error {
     cb.mutex.Lock()
     defer cb.mutex.Unlock()

     if err == nil {
      if cb.policy == MaxConsecutiveFails {
       cb.fails = 0
      }
      cb.state = closed
      return nil
     }

     if cb.state == halfOpen {
      cb.state = open
      cb.openChannel <- struct{}{}
      return err
     }

     cb.fails++
     if cb.failsExcceededThreshod() {
      cb.state = open
      cb.openChannel <- struct{}{}
     }

     return err
    }

This functions introduces a bunch of new fields and methods, let’s describe the logic behind this function and then expand our circuit breaker structure.

mutex is used in order to make sure Read-Write-Modify cycles are done in a safe way, in case we have concurrent attempts to modify the circuit breaker state
err is the actual error that *destination *returned. If nil, basically reset the state and go on. fails are the number of the failed requests in the current closed state. MaxConsecutiveFails is a policy which means that the circuit breaker must open the circuit after experiencing n number of consecutive fails.
If there was an error and we were in a half-open state, open the circuit and go on. The cb.openChannel <- struct{}{} triggers the waiting interval
If there was an error and we were not in the half-open state, simply increment the number of failed attemps in the current state.
Check to see if the number of failed attempts excceeded the threshold regarding your policy. If so, open the circuit and trigger the waiting interval

Let’s take a look at the complete structure of our circuit breaker:

    type Policy int
    type State string

    const (
     // MaxFails specifies the maximum non-consecutive fails which are allowed
     // in the "Closed" state before the state is changd to "Open".
     MaxFails Policy = iota

     // MaxConsecutiveFails specifies the maximum consecutive fails which are allowed
     // in the "Closed" state before the state is changed to "Open".
     MaxConsecutiveFails
    )

    const (
     open     State = "open"
     closed   State = "closed"
     halfOpen State = "half-open"
    )

    type circuitbreaker struct {
     policy              Policy
     maxFails            uint64
     maxConsecutiveFails uint64
     openInterval        time.Duration

     // fails is the number of failed requets for the current "Closed" state,
     // resets after a successful transition from half-open to closed.
     fails uint64

     // current state of the circuit
     state State

     // openChannel handles the event transfer mechanism for the open state
     openChannel chan struct{}

     mutex sync.Mutex
    }

And the helper functions we’ve seen so far:

    func (cb *circuitbreaker) failsExcceededThreshod() bool {
     switch cb.policy {
     case MaxConsecutiveFails:
      return cb.fails >= cb.maxConsecutiveFails
     case MaxFails:
      return cb.fails >= cb.maxFails
     default:
      return false
     }
    }

    func (cb *circuitbreaker) openWatcher() {
     for range cb.openChannel {
      time.Sleep(cb.openInterval)
      cb.mutex.Lock()
      cb.state = halfOpen
      cb.fails = 0
      cb.mutex.Unlock()
     }
    }

The openWatcher simply listens on the openChannel and upon receive, sleeps the goroutine for the openInterval and then resets the number offails, then changes the state to half-open. The cycle then repeats. But when is the openWatcher called? Just when we are initializing our circuit breaker:

    type ExtraOptions struct {
     // Policy determines how the fails should be incremented
     Policy Policy

     // MaxFails specifies the maximum non-consecutive fails which are allowed
     // in the "Closed" state before the state is changd to "Open".
     MaxFails *uint64

     // MaxConsecutiveFails specifies the maximum consecutive fails which are allowed
     // in the "Closed" state before the state is changed to "Open".
     MaxConsecutiveFails *uint64

     OpenInterval *time.Duration
    }

    func New(opts ...ExtraOptions) Circuitbreaker {
     var opt ExtraOptions
     if len(opts) > 0 {
      opt = opts[0]
     }

     if opt.MaxFails == nil {
      opt.MaxFails = literal.ToPointer(uint64(5))
     }

     if opt.MaxConsecutiveFails == nil {
      opt.MaxConsecutiveFails = literal.ToPointer(uint64(5))
     }

     if opt.OpenInterval == nil {
      opt.OpenInterval = literal.ToPointer(5 * time.Second)
     }

     cb := &circuitbreaker{
      policy:              opt.Policy,
      maxFails:            *opt.MaxFails,
      maxConsecutiveFails: *opt.MaxConsecutiveFails,
      openInterval:        *opt.OpenInterval,
      openChannel:         make(chan struct{}),
     }

     go cb.openWatcher()

     return cb
    }

Let’s try what we’ve written with the most simplistic way possible:

    package main

    func main() {
     cbOpts := circuitbreaker.ExtraOptions{
      Policy:              circuitbreaker.MaxFails,
      MaxFails:            literal.ToPointer(uint64(5)),
      MaxConsecutiveFails: literal.ToPointer(uint64(5)),
      OpenInterval:        literal.ToPointer(160 * time.Millisecond),
     }
     cb := circuitbreaker.New(cbOpts)
     wg := &sync.WaitGroup{}
     for i := 1; i < 100; i += 1 {
      wg.Add(1)
      go makeServiceCall(i, cb, wg)
      time.Sleep(10 * time.Millisecond)
     }

     log.Println("sent all the requests")
     wg.Wait()
     log.Println("got all the responses, exiting.")
    }

    func serviceMethod(id int) (string, error) {
     if val := rand.Float64(); val <= 0.3 {
      return "", errors.New("failed")
     }
     return fmt.Sprintf("[id: %d] done.", id), nil
    }

    func makeServiceCall(id int, cb circuitbreaker.Circuitbreaker, wg *sync.WaitGroup) {
     defer wg.Done()

     resp, err := cb.Execute(func() (interface{}, error) {
      return serviceMethod(id)
     })
     if err != nil {
      log.Printf("[id %d] got err: %s", id, err.Error())
     } else {
      log.Printf("[id %d] success: %s", id, resp)
     }
    }

Feel free to experiment with different values for policy, MaxFails, MaxConsecutiveFails and openInterval.

Conclusion

So far we’ve seen how circuit breakers play a vital role in reliability and performance. Without circuit breakers the client might waste lots of time trying to connect to an unavailable service and waiting for the whole timeout period each time. The target service on the other hand, might be bombarded with dozens of requests right after recovering from a down time or disaster, leading to another unfortunate outage. We’ve also walked through a simple implementation of the circuit breaker pattern in Go. Needless to say that I’d love to know your opinion about this subject as well as any other comments or suggestions that you might have. I learn a lot reading through your comments as well as corrections or additional information that your provide in your comments.

Protoc Plugins with Go

Homayoon Alimohammadi — Fri, 18 Aug 2023 17:31:40 +0000

In this article we take a look at Protocol Buffers (protobufs) and explore the intricate workings of protoc plugins in the context of Go programming. With a focus on demystifying protoc plugins and their role in code generation, we walkthrough a humble overview of the inner workings of plugins with practical snippets. Throughout the article, we will introduce and take a look at key concepts such as the Protobuf Compiler Plugin API, Go’s plugin package, and the integration of custom plugins into the protoc compilation pipeline. So, if you’ve ever found yourself grappling with the complex situations while working with protocol buffers (as I did) or are simply interested in harnessing the power of custom protoc plugins in Go, join me on this journey as we try to scratch the surface of Protocol Buffers together.

Introduction

Protocol Buffers (protobuf) have become a popular choice for data serialization and communication between services in the world of modern software development. With their language-agnostic nature, efficiency, and ease of use, protobufs have emerged as a go-to solution for structuring data interchange formats.

While protobufs offer an extensive set of features to define and handle messages, there are times when developers encounter complex scenarios that require additional customization. One such scenario that I personally faced was when I tried to deal with logical and domian specific constraints on the data. Things that can not be forced or even mentioned in a proto file outside of raw comments or loose conventions.

By developing a custom protoc plugin, we can extend the capabilities of the protobuf compiler (protoc) to automatically generate code that makes working with proto messages more intuitive and efficient. We will discuss the various components involved in the creation of such a plugin, including the Protobuf Compiler Plugin API, Go’s plugin package, and the necessary steps to integrate the plugin into the protobuf compilation process.

DISCLAIMER: I tend to make lots of mistakes on a daily basis, so if you’re one of the many that encountered one, I’d be grateful if you would correct me in the comments.

Picturing the problem

In my humble opinion, type-safety, while sometimes forces you to respect certain boundaries, can be of great value especially if you’re trying to build scalable, maintainable, understandable and easy-to-develop software. Of course this can be the topic for another detailed (and maybe controversial) discussion some time later, but for now, let’s just temporarily imagine that this fact holds true and we are on the same page.

Consider the message below:

    syntax = "proto3";

    import "google/protobuf/struct.proto";

    message Game {
      google.protobuf.Struct data = 1;
    }

The data field above is obviously not type-safe and might contain anything from a simple integer to multiple complex nested structures. Serialized into json, it might turn into something like this:

    {
      "data": {
        "title": "Elden Ring",
        "description": "Third-person, action, RPG with a focus on combat and exploration",
        "developer": {
          "name": "FromSoftware Inc.",
        }
        "category": "role-playing",
        "number_of_bosses": 18,
      }
    }

Let’s say we examined and analyzed every saved Game object in our database and gathered enough information to organize this data structure to something like this:

    message Game {
      string Title = 1;
      string Description = 2;
      message Developer {
        string Name = 1;
      }
      Developer developer = 3;
      enum Category {
        UNKNOWN = 0;
        ROLE_PLAYING = 1;
        SHOOTING = 2;
      }
      Category category = 4;

      message RolePlayingExtra {
        int32 number_of_bosses = 1;
      }
      message ShootingExtra {imagine
        int32 number_of_guns = 2;
      }

      oneof extra {
        RolePlayingExtra role_playing_extra = 5;
        ShootingExtra shooting_extra = 6;
      }
    }

And as a domain specific logic, let’s say that we need to make sure for every category (e.g. ROLE_PLAYING ), only its respective extra should be filled (e.g. RolePlayingExtra). Notice that the extra message will not be of a concrete type (at least in Go) since it can contain fields of different types (e.g. RolePlayingExtra and ShootingExtra). Let’s take a look at it’s translation in Go code:

    type Game struct {
     ...
     // Types that are assignable to Extra:
     //
     // *Game_RolePlayingExtra_
     // *Game_ShootingExtra_
     Extra isGame_Extra `protobuf_oneof:"extra"`
    }

    type isGame_Extra interface {
     isGame_Extra()
    }

    type Game_RolePlayingExtra_ struct {
     RolePlayingExtra *Game_RolePlayingExtra `protobuf:"bytes,5,opt,name=role_playing_extra,json=rolePlayingExtra,proto3,oneof"`
    }

    type Game_ShootingExtra_ struct {
     ShootingExtra *Game_ShootingExtra `protobuf:"bytes,6,opt,name=shooting_extra,json=shootingExtra,proto3,oneof"`
    }

    func (*Game_RolePlayingExtra_) isGame_Extra() {}

    func (*Game_ShootingExtra_) isGame_Extra() {}

The Extra field is of type isGame_Extra which is an interface implemented by Game_RolePlyaingExtra_ and Game_ShootingExtra_ structs and should be casted to the desired concrete type upon request. protoc-gen-go actually does a great job of handling this type cast:

    func (x *Game) GetRolePlayingExtra() *Game_RolePlayingExtra {
     if x, ok := x.GetExtra().(*Game_RolePlayingExtra_); ok {
      return x.RolePlayingExtra
     }
     return nil
    }

    func (x *Game) GetShootingExtra() *Game_ShootingExtra {
     if x, ok := x.GetExtra().(*Game_ShootingExtra_); ok {
      return x.ShootingExtra
     }
     return nil
    }

Yet, we have two problems here:

For any given Game struct, in order to set one of the extra fields (e.g. NumberOfGuns) we have to take care of the type casting ourselves which is not a big deal but can become cumbersome and will increase the clutter in our code.
Protobuf on its own will not at all guarantee that Category and Extra fields are valued accordingly and with their relation in mind. This means that we can technically have a game with the Category being SHOOTING and the Extra being RolePlayingExtra. If not handled properly, this can cause inconsistencies and incorrectness in our data.

Considering the problems above, let’s see how a custom protoc plugin can help us overcome them.

Protoc Plugins

Let’s take a look at how protoc-gen-go generates a .pb.go file. First step is to use the plugin in as an argument to the protoc (protobuf compiler):

    protoc --go_out=. --go_opt=paths=source_relative game.proto

So the --go_out=. argument will tell the protoc that the protoc-gen-go plugin should be invoked. Notice that --go_opt=... just passes some optional arguments to the protoc-gen-go which might affect the generation behavior. In order to have any kind of protoc plugins (both official plugins such as protoc-gen-go or any private or custom plugin) it needs to meet 2 requirements:

The executable file should be placed somewhere in your $PATH.
It should be named by the pattern protoc-gen-.
So it can be invokes like so:

    protoc --<MY_PLUGIN>_out=. --<MY_PLUGIN>_opt=... some.proto

Now let’s take a look at the source code of the protoc-gen-go plugin:

    package main 

    import (
      "github.com/golang/protobuf/internal/gengogrpc"
      "google.golang.org/protobuf/compiler/protogen"
    )

    func main() {
     ...
     protogen.Options{...}.Run(func(gen *protogen.Plugin) error {
      ...
      for _, f := range gen.Files {
       ...
       g := gengo.GenerateFile(gen, f)
       ...
      }
      ...
      return nil
     })
    }

Lines populated with the ellipsis indicate additional information that are not really important for us at the moment. Let’s take a look at what we have:

protogen.Options{}: a struct that provides configuration options for the protoc plugin. It allows you to customize the behavior of the plugin and specify various settings related to code generation. Here is a sample snippet if you want to use flags as your options:

    var (
      flags flag.FlagSet
      myOpt = flags.String("myopt", "", "my random option")
     )
     protogen.Options{ParamFunc: flags.Set}.Run(func(gen *protogen.Plugin) error {...})

These flags can be passed into our plugin like so:

    protoc --<MY_PLUGIN>_out=. --<MY_PLUGIN>_opt=myopt=something some.proto

Run(func(gen *protogen.Plugin) error): executes a function as a protoc plugin. It reads a CodeGeneratorRequest message from os.Stdin, invokes the plugin function, and writes a CodeGeneratorResponse message to os.Stdout. If a failure occurs while reading or writing, Run prints an error to os.Stderr and calls os.Exit(1).
Lastly we iterate over the files specified in the command line and pass them to gogen.GenerateFile which we will take care of the rest of the process (i.e. creating the actual output file and putting the autogenerated content into it).

Let’s take a look at the gogen.GenerateFile function:

    // GenerateFile generates the contents of a .pb.go file.
    func GenerateFile(gen *protogen.Plugin, file *protogen.File) *protogen.GeneratedFile {
     filename := file.GeneratedFilenamePrefix + ".pb.go"
     g := gen.NewGeneratedFile(filename, file.GoImportPath)

     ...

     g.P(packageDoc, "package ", f.GoPackageName)
     g.P()

     ...

     genImport(gen, g, f, imps.Get(i))
     genEnum(g, f, enum)
     genMessage(g, f, message)
     genExtensions(g, f)
     genReflectFileDescriptor(gen, g, f)

     ...

     return g
    }

We can create as many new files as we want using the command gen.NewGeneratedFile by passing our desired fileName and the GoImportPath defined in the proto file. This will provide us with an instance of *protogen.GeneratedFile with its super handy function P(). Let’s take a closer look at it:

    // P prints a line to the generated output. It converts each parameter to a
    // string following the same rules as fmt.Print. It never inserts spaces
    // between parameters.
    func (g *GeneratedFile) P(v ...interface{}) {
     for _, x := range v {
      switch x := x.(type) {
      case GoIdent:
       fmt.Fprint(&g.buf, g.QualifiedGoIdent(x))
      default:
       fmt.Fprint(&g.buf, x)
      }
     }
     fmt.Fprintln(&g.buf)
    }

To put it simply, P() iterates over the inputs and tries to export them into the generated file with regards to proper Go-specific indentation, so we can easily write any legit go statement or declaration that we want, no matter the indentation or leading tabs/spaces.

Other lines presented in the GenerateFile() function are just taking care of different components of our proto file (e.g. messages, enums, etc…)

Now that we have a fundamental understanding of how protoc plugins work, it’s time to see how we can leverage these tools to write our own custom plugin.

Protoc Custom Plugin

You can find the complete code in my github repo: https://github.com/HomayoonAlimohammadi/protoc-gen-gamedata

Let’s make it as simple as possible, here’s a main.go to start with:

    package main

    import (
     "github.com/HomayoonAlimohammadi/protoc-gen-gamedata/gamedata"
     "google.golang.org/protobuf/compiler/protogen"
    )

    func main() {
     protogen.Options{}.Run(func(p *protogen.Plugin) error {
      return gamedata.Generate(p)
     })
    }

It invokes the Generate function from gamedata package, which, as you can see, is really simple and minimal:

    func Generate(p *protogen.Plugin) error {
     g := p.NewGeneratedFile("autogen/gamedata.autogen.go", protogen.GoImportPath("gamedata"))
     g.P("// Code generated by gamedata. DO NOT EDIT")
     g.P("package gamedata")

     game, err := extract(p, g)
     if err != nil {
      return err
     }

     genHelpers(g, game)

     return nil
    }

After creating a new file in the autogen relative directory, we just put a single comment, indicating that it was automatically generated along with the package name.

There is this extract function in the middle which we will discuss in a moment, but before that, the genHelpers is actually going to generate Go code in our gamtedata.autogen.go file. It is responsible for generating the functions that help us overcome our initial challenges (setting extra fields according to the category field with the auto type casting in mind).

As you might have inferred from the code above, the extract function tries to summarize and return the information about our Game message, like below:

    type Game struct {
     Fields     []Field
     CatToExtra map[string]Extra
    }

    type Extra struct {
     Name string
     Fields []Field
    }

    type Field struct {
     Name string
     Type string
    }

    func extract(p *protogen.Plugin, g *protogen.GeneratedFile) (*Game, error) {
     for _, f := range p.Files {
      for _, m := range f.Messages {
       if m.Desc.Name() == "Game" {
        return extractGameData(m, g)
       }
      }
     }

     return nil, errors.New("failed to find `Game` message")
    }

After finding the Game message from all the available files (p.Files, here only one) and messages (f.Messages,again only one) it translates the components into a Game struct. Note that you can access FieldDescriptor, MessageDescriptor, EnumDescriptor etc… with the Desc() method on the respective *protogen.Field, *protogen.Message, etc...:

    func extractGameData(m *protogen.Message, g *protogen.GeneratedFile) (*Game, error) {
     game := &Game{CatToExtra: make(map[string]Extra)}

     prefillCategories(m, game)

     for _, f := range m.Fields {
      if f.Desc.ContainingOneof() != nil && f.Desc.ContainingOneof().Name() == "extra" {
       err := fillExtras(g, f, game)
       if err != nil {
        return nil, fmt.Errorf("failed to fill extra: %w", err)
       }
      }

      game.Fields = append(
       game.Fields,
       Field{
        Name: string(f.Desc.Name()),
        Type: fieldGoType(g, f),
       },
      )
     }

     return game, nil
    }

Let’s break down this function so we can better understands how it works:

First we prefill the categories, this will come handy later.

    func prefillCategories(m *protogen.Message, game *Game) {
     for _, e := range m.Enums {
      if string(e.Desc.Name()) == "Category" {
       for _, v := range e.Values {
        if string(v.Desc.Name()) == "UNKNOWN" {
         continue
        }

        game.CatToExtra[string(v.Desc.Name())] = Extra{}
       }
      }
     }
    }

We iterate over Fields of our Game message
If we encounter a field that is contained in the extra oneof, we try to find its respective category and fill in the CatToExtra:

    func fillExtras(g *protogen.GeneratedFile, f *protogen.Field, game *Game) error {
     if !strings.HasSuffix(strings.ToLower(string(f.Desc.Name())), "_extra") {
      return fmt.Errorf("extra message %s does not end with _extra", f.Desc.Name())
     }

     name := strings.TrimSuffix(strings.ToLower(string(f.Desc.Name())), "_extra")

     var fields []Field
     for _, innerf := range f.Message.Fields {
      fields = append(fields, Field{Name: string(innerf.Desc.Name()), Type: fieldGoType(g, innerf)})
     }

     _, ok := game.CatToExtra[strings.ToUpper(name)]
     if !ok {
      return fmt.Errorf("no category available for `%s`", strings.ToUpper(name))
     }

     game.CatToExtra[strings.ToUpper(name)] = Extra{
      Name:   string(f.Desc.Name()),
      Fields: fields,
     }

     return nil
    }

As required by our domain logic, we also make sure that all extra fields do end with the _extra suffix and that there are no dangling extra messages available as well (dangling as if there is no Category related to that extra).

The only thing left is to implement a logic to create the helper functions we needed to maintain consistency and correctness in our data. Since it won’t really help with understanding protobuf any more, I’m not going to write the code for that here, but I’m sure you can easily figure out how it’s done.

HINT: It’s basically a bunch of iterations, if/else statements and g.P()

In order to see what the code above gives us, let’s quickly serialize it to json and look at the results:

    b, _ := json.MarshalIndent(game, "", "\t")
    fmt.Println(string(b))

    {
     "Fields": [
      {
       "Name": "Title",
       "Type": "string"
      },
      {
       "Name": "Description",
       "Type": "string"
      },
      {
       "Name": "developer",
       "Type": "*game.Game_Developer"
      },
      {
       "Name": "category",
       "Type": "game.Game_Category"
      },
      {
       "Name": "role_playing_extra",
       "Type": "*game.Game_RolePlayingExtra"
      },
      {
       "Name": "shooting_extra",
       "Type": "*game.Game_ShootingExtra"
      }
     ],
     "CatToExtra": {
      "ROLE_PLAYING": {
       "Name": "role_playing_extra",
       "Fields": [
        {
         "Name": "number_of_bosses",
         "Type": "int32"
        }
       ]
      },
      "SHOOTING": {
       "Name": "shooting_extra",
       "Fields": [
        {
         "Name": "number_of_guns",
         "Type": "int32"
        }
       ]
      }
     }
    }

Finally, we have to build our plugin and place it somewhere in our $PATH so that it can be invoked via protoc:

    go build -o $(GOPATH)/bin/protoc-gen-gamedata main.go
    protoc --gamedata_out=. game.proto

Wrap it up

In conclusion, we have explored the fascinating world of protoc plugins in Go and demonstrated how to write a custom plugin to enhance code generation in Protocol Buffers. By unraveling the intricacies of protoc plugins and providing practical examples, hopefully you might have a good idea how these interesting plugins work and how you can write your own. Needless to say that your comments, suggestions and ideas as well as your overall impression is of great importance to me.

Handling JSON in Go

Homayoon Alimohammadi — Thu, 17 Aug 2023 18:59:52 +0000

In Go, the encoding/json package offers built-in JSON marshaling and unmarshaling capabilities, allowing us to effortlessly convert Go data structures to JSON and vice versa. However, there are scenarios where the default behavior falls short, such as handling complex data structures, managing custom field names, or dealing with special data types. This article aims to provide an introduction on how JSON can be handled in Go, and as always, with a bit of customization.

Disclaimer: I tend to make lots of mistakes on a daily basis. If you’ve noticed one, I’d be more than grateful if you let me know.

All the code snippets can be found in my Github repo.

Understanding JSON Marshaling and Unmarshaling

JSON (JavaScript Object Notation) is a lightweight data interchange format widely used for communication between web services and applications. It represents data as key-value pairs, arrays, and nested structures, providing a simple and human-readable format for data exchange.

In Go, the encoding/json package is a powerful tool for working with JSON data. It offers the Marshal function to convert Go data structures into JSON, and the Unmarshal function to deserialize JSON into Go data structures. These functions handle most common cases automatically, mapping Go struct fields to JSON object properties and the other way around. However, when dealing with more complex scenarios or requiring fine-grained control over the serialization and deserialization process, JSON tags as well as custom marshaling and unmarshaling functions shine.

JSON Tags

The way that Go handles JSON revolves around structs and field tags. Structs represent the data structure to be serialized or deserialized, with each field of the struct corresponding to a property in the JSON representation. Field tags, defined using the json struct tag, provide metadata that controls the serialization and deserialization behavior. By leveraging field tags, we can specify custom names for JSON properties, handle null values, control field omission, and more. Let’s see an example of these tags:

    type Person struct {
        Name      string    `json:"full_name"` 
        Age       int       `json:"-"` 
        Email     string    `json:"email,omitempty"` 
    }

Name is mapped to the JSON property full_name
Age will be fully ignored (won’t show up no matter what)
Email will be mapped to email and the omitempty is a reserved flag which ensures that if the email field is empty (e.g. contains the zero value of its type) it’s not going to show up after being serialized into JSON

Let’s see how these tags work in action:

    pp := []Person{
        {
           Name:  "first person",
           Age:   20,
           Email: "first@person.com",
        },
        {
           Name:  "second person",
           Age:   25,
           Email: "",
        },
    }

    b, _ := json.MarshalIndent(pp, "", "\t")
    fmt.Println(string(b))

    [
        {
            "full_name": "first person",
            "email": "first@person.com"
        },
        {
            "full_name": "second person"
        }
    ]

Note that omitempty if used solely as a JSON tag will be considered the field name rather than being treated as a flag.

    type Person struct {
        Name      string    `json:"omitempty"`
    }

    pp := []Person{
        {
           Name:  "first person",
        },
        {
           Name:  "second person",
        },
    }

    b, _ := json.MarshalIndent(pp, "", "\t")
    fmt.Println(string(b))

    [
        {
            "omitempty": "first person",
        },
        {
            "omitempty": "second person"
        }
    ]

NOTE: **Duplicate names will be failed to get serialized silently (e.g. no errors, yet **none of the duplicate fields appear in the output), although you’ll probably be warned by a static code analysis tool or a linter.

JSON tags (or let’s say any custom tag) can be obtained at runtime via reflection like below:

    func getFieldTags(t reflect.Type) map[string][]string {
       if t.Kind() != reflect.Struct {
          panic(fmt.Sprintf("t should be struct but is %s", t))
       }

       tags := make(map[string][]string)
       for i := 0; i < t.NumField(); i++ {
          f := t.Field(i)
          jTags := strings.SplitN(f.Tag.Get("json"), ",", -1)
          tags[f.Name] = jTags
       }

       return tags
    }

    func main() {
       tags := getFieldTags(reflect.TypeOf(Person{})) 
       fmt.Printf("%+v\n", tags) // map[Age:[-] Email:[email omitempty] Name:[full_name]]
    }

Custom JSON Marshaling/Unmarshaling

In order to implement a custom JSON marshal/unmarshaler you have to implement the json.Marshaler or json.Unmarshaler interfaces accordingly. Let’s have a look at these interfaces:

    // Marshaler is the interface implemented by types that
    // can marshal themselves into valid JSON.
    type Marshaler interface {
       MarshalJSON() ([]byte, error)
    }

    // Unmarshaler is the interface implemented by types
    // that can unmarshal a JSON description of themselves.
    // The input can be assumed to be a valid encoding of
    // a JSON value. UnmarshalJSON must copy the JSON data
    // if it wishes to retain the data after returning.
    //
    // By convention, to approximate the behavior of Unmarshal itself,
    // Unmarshalers implement UnmarshalJSON([]byte("null")) as a no-op.
    type Unmarshaler interface {
       UnmarshalJSON([]byte) error
    }

Let’s say that we have to make a call to some remote API in order to obtain information about certain products, and for the sake of the example let’s assume that the API response is as follows:

    {
        "products": [
            "prod A",
            "prod B",
            "prod C"
        ],
        "prices": [
            99.99,
            199.99,
            299.99
        ]
    }

We know (and expect) that the same indexes contain information about a certain product, e.g. prices[0] belongs to products[0]. in order to better handle this data we want to deserialize (unmarshal) the API response into the following structures:

    type Product struct {
       Name  string
       Price float64
    }

    type Cargo struct {
       Products []Product
    }

In order for that to work, we need to declare an UnmarshalJSON method for our destination (here Cargo) struct:

    func main() {
       // our imaginary API
       d := GetProductPrices()

       cargo := &Cargo{}
       err := json.Unmarshal(d, cargo)
       if err != nil {
         log.Fatal("failed to unmarshal to cargo:", err)
       }

       fmt.Printf("%+v\n", cargo)
    }

    func (c *Cargo) UnmarshalJSON(d []byte) error {
       ...
    }

In order to have fine grained control over how our data is deserialized we can temporarily unmarshal it into any desired intermediary structure:

    func (c *Cargo) UnmarshalJSON(d []byte) error {

       // desired intermediary structure
       type temp struct {
        Products []string  `json:"products"`
        Prices   []float64 `json:"prices"`
       }

       tmp := &temp{}
       err := json.Unmarshal(d, tmp)
       if err != nil {
        return fmt.Errorf("failed to marshal JSON in temp: %w", err)
       }
       ...
    }

Or we can even use fully extensible data structure like map[string]any :

    func (c *Cargo) UnmarshalJSON(d []byte) error {

       // desired intermediary structure
       tmp := make(map[string]any)
       err := json.Unmarshal(d, &tmp)
       if err != nil {
        return fmt.Errorf("failed to marshal JSON in temp: %w", err)
       }
       ...
    }

Having that the deserialized data is ready for further processing, the next step is to transform it into whatever structure we want:

    func (c *Cargo) UnmarshalJSON(d []byte) error {
       type temp struct {
          Products []string  `json:"products"`
          Prices   []float64 `json:"prices"`
       }

       tmp := &temp{}
       err := json.Unmarshal(d, tmp)
       if err != nil {
          return fmt.Errorf("failed to marshal JSON in temp: %w", err)
       }

       if len(tmp.Prices) != len(tmp.Products) {
          return fmt.Errorf("length of products (%d) and prices (%d) does not match", len(tmp.Products), len(tmp.Prices))
       }

       for i := 0; i < len(tmp.Products); i++ {
          c.Products = append(c.Products, Product{Name: tmp.Products[i], Price: tmp.Prices[i]})
       }

       return nil
    }

Finally we can expect to have a nicely organized data like below:

    {
     "Products": [
      {
       "Name": "prod A",
       "Price": 100
      },
      {
       "Name": "prod B",
       "Price": 200
      },
      {
       "Name": "prod C",
       "Price": 300
      }
     ]
    }

While I was experimenting with these stuff I came up with a somewhat interesting question, something that might be bothering you right now as well:

In the case that our structs do not implement json.Marshaler or json.Unmarshaler interfaces, Go still finds a way to serialize from and deserialize to our structures. How is that possible?

Well, under the hood, what Go really does is that it looks to see if the given struct implements either of those interfaces. “How?” you might ask. Well, again, all thanks to reflection:

    // json/decode.go 
    // check if a certain type implements "Unmarshaler" (v is reflect.Value)
    if v.Type().NumMethod() > 0 && v.CanInterface() {
        if u, ok := v.Interface().(Unmarshaler); ok {
            return u, nil, reflect.Value{}
    }

    // json/encode.go
    // check if a certain type implements "Marshaler" (t is reflect.Type)
    marshalerType = reflect.TypeOf((*Marshaler)(nil)).Elem()
    if t.Kind() != reflect.Pointer && allowAddr && reflect.PointerTo(t).Implements(marshalerType) {
        return newCondAddrEncoder(addrMarshalerEncoder, newTypeEncoder(t, false))
    }

In case the type fails to pass the test (e.g. does not implement neither json.Marshaler nor json.Unmarshaler ) Go falls back on something like a plan B.

To be honest I mostly use custom mashaling/unmarshaling capabilities to permenantly change JSON files embedded in the program, by first reading the JSON file, deserializing it into whatever structure I want and then serialize it back to a new .json file. Something like this:

    type CustomStruct struct {
      // whatever field I want at the end
    }

    func (c *CustomStruct) UnmarshalJSON(d []byte) error {
      // custom JSON unmarshalling
    }

    func main() {
      // read, change, write
      d, err := os.ReadFile("old.json")
      if err != nil {
        log.Fatal("failed to read file:", err)
      }

      st := &CustomStruct{}
      err = json.Unmarshal(d, st)
      if err != nil {
        log.Fatal("failed to unmarshal:", err)
      }

      b, err := json.MarshalIndent(st, "", "\t")
      if err != nil {
        log.Fatal("failed to marshal struct:", err)
      }

      err = os.WriteFile("new.json", b, 0644)
      if err != nil {
        log.Fatal("failed to write new file:", err)
      }
    }

Note that there are two major things that I’ve not considered here, one is reading the file in a streaming manner (so I don’t flood my memory by reading the file as a whole at once) and the other is partial unmarshaling in JSON. We might take a look at these subjects later.

Conclusion

Custom JSON marshaling and unmarshaling in Go allows us to overcome the limitations of the default JSON handling capabilities provided by the standard library. By leveraging custom marshalers and unmarshalers, we can finely tune the serialization and deserialization processes, making our code more expressive, efficient, and tailored to specific use cases. We have explored various aspects of custom JSON handling in Go, from implementation techniques to real-world examples. I hope you’ve enjoyed our journey. Thanks for your support in advance!

Enhancing gRPC Error Handling in a Microservice Architecture

Homayoon Alimohammadi — Mon, 14 Aug 2023 18:54:35 +0000

Error handling and the way it is done is a crucial part of software engineering. Poorly returned errors and non-informative ones can cause unimaginable headaches. In this article I’m going to demonstrate the problems that I’ve faced with regards to gRPC error handling and what we might be able do to in order to improve upon our originally not-so-useful gRPC errors.

I make lots of mistakes on a daily basis and I try to fix them and learn from them. If you’ve noticed one, I’d be more than grateful if you would correct me however you like.

Introduction

First of all, let’s start with explaining the actual problem that got me here. Inter-service communication (and the way it’s done) is probably one of the more important concerns in a microservice architecture. While here are certain concerns when using RPCs as a whole, (which can be mitigated as beautifully described by Netflix Tech Blog) it can’t be denied that the self-documenting nature of request/response messages and RPCs (via protobuf), coupled with server reflection has proved extremely comfortable and handy.

One of the most overlooked aspects of gRPC might be the way the errors are returned from one service to another. The most simplistic way to return an error in an RPC might be as follows:

func (s *serverImpl) UnaryRPC(ctx context.Context, req *pb.Request) (*pb.Response, error) {
    // do something
    if somethingFailed {
        return nil, errors.New("custom error")
    }

    return resp, nil
}

The problem with the example above is that it provides nearly zero information to the caller about what went wrong and why. From the caller perspective the error might be handled like this:

    reps, err := client.UnaryRPC(ctx, req)
    if err != nil {
      st, ok := status.FromError(err)
      if ok {
        if st.Message() == "custom error" {
          // handle custom error
        } else {
          // handle other errors
        }
      }
    }

Hopefully this is already bothering you a lot simply because we are tightly coupled to an error string from another service which might get changed at any moment which renders our glorious error handling useless. Let’s take a look at how status.FromError works to see if we have any other useful information available:

    func FromError(err error) (s *Status, ok bool) {
      if err == nil {
        return nil, true
      }

      // doing some stuff with interface{ GRPCStatus() *Status } type
      // our simple error does not implement any custom interface
      // ...

      return New(codes.Unknown, err.Error()), false
    }

This signals to us that other attributes of the status.Status (e.g. Code and Details ) are simply meaningless. Code is automatically set to Unknown and Details will be zero-valued.

You can safely use st := status.Convert(err) which just skips the ok handling part since the Go implementation of gRPC guarantees that all RPCs return status type errors. We can also confirm this by heading to the http2_client.go operateHeaders method:

    func (t *http2Client) operateHeaders(frame *http2.MetaHeadersFrame) {
      // ...
      var (
        // ...
        grpcMessage string
        statusGen *status.Status
        rawStatusCode = codes.Unknown
        // ...
      )
      // ...
      for _, hf := range frame.Fields {
        switch hf.Name {
        // ...
        case "grpc-message":
          grpcMessage = decodeGrpcMessage(hf.Value)
        case "grpc-status":
          code, err := strconv.ParseInt(hf.Value, 10, 32)
          handle(err)
          rawStatusCode = codes.Code(uint32(code))
        case "grpc-status-details-bin":
          statusGen, err = decodeGRPCStatusDetails(hf.Value)
          handle(error)
        // ...
        }
        // ...
      } 
      // ...
      if statusGen == nil {
        statusGen = status.New(rawStatusCode, grpcMessage)
      }
      // ...
      t.closeStream(s, io.EOF, rst, http2.ErrCodeNo, statusGen, mdata, true)
    }

Let’s be polite and provide more information to our caller:

    func (s *serverImpl) UnaryRPC(ctx context.Context, req *pb.Request) (*pb.Response, error) {
      // do something
      if somethingFailed {
        return nil, status.Error(codes.Internal, "internal error")
      }

      return resp, nil
    }

Now, the client can also make important and useful assumptions about the returned code:

    resp, err := client.UnaryRPC(ctx, req)
    if err != nil {
      st := status.Convert(err)
      switch st.Code() {
      case codes.Internal:
        // handle internal error
      default:
        // handle other errors
      }
    }

As informative and descriptive as it might look, this method falls short of explaining and reasons behind its (status) code and error.

Status Details

Returning status codes was the farthest we’ve come with regards to returning descriptive errors from our microservices. But there are certain scenarios in which we simply can’t afford to solely rely on the status codes. Why, you might ask? Let me explain.

Assume that we have two services, the first one which interacts with our end users is called Submit and the other one,Storage, is considered an internal service, talking only to other services and not to external clients/customers or end users.

Consider the flow as follows: the end user tries to submit his/her data through our Submit service, which in turn tries to persist the users data by saving it in the Storage service (let’s say that Storage simply wraps some arbitrary DB).

In a peculiar scenario, due to some business logic,Storage denies to save the data provided by Submit and returns the reason like below:

    func (s *storageService) Save(ctx context.Context, req *pb.Req) (*pb.Resp, error) {
      // the actual logic
      if cantSave {
        return nil, status.Error(codes.FailedPrecondition, "the fully described reason here")
      }  
      // ...
    }

Assuming that the described reason is specific to Storage service domain, meaning that Submit won’t be able to reproduce the exact descriptive reason in order to show the client, let’s explore different ways that Submit can handle this scenario and return some error to the end user.

Return the error as is

    func (s *submitService) Submit(ctx context.Context, req *pb.Req) (*pb.Resp, error) {
      resp, err := storageClient.Save(ctx, req)
      if err != nil {
        st := status.Convert(err)
        switch st.Code() {
        case codes.FailedPrecondition:
          return nil, st.Err()
        default:
          return nil, status.Error(codes.Internal, "oops!")
        }
      }
    }

While this method might simply solve our problem (the end user gets to see the fully descriptive error as is) a major headache is introduced as well: Debugging this error will be an absolute hassle. Submit is actually returning this error, but good luck finding the text in this repo. Imagine doing the same thing with a couple of other services apart from Storage. Soon you’ll find yourself searching all your client services for some error string that seeminglySubmit returns, scratching your head and regretting every decision you’ve made in your life. (just think about what happens if Storage returns codes.FailedPrecondition in several scenarios, some with error strings containing private information and details that end users should not know about)

Drop the original error and replace with our own

    func (s *submitService) Submit(ctx context.Context, req *pb.Req) (*pb.Resp, error) {
      resp, err := storageClient.Save(ctx, req)
      if err != nil {
        st := status.Convert(err)
        switch st.Code() {
        case codes.FailedPrecondition:
          return nil, errors.New("dear end user, you can not do that")
        }
      }
    }

This way we can at least make sure that Submit won’t return unwanted error messages or ones that are not available in the Submit repo, yet the returned error string is not exactly what we wanted our end user to see. It’s not descriptive enough. Also, we still have the same problem with multiple codes.FailedPrecondition errors as above.

Storage service provides further error details

In order to enable effective communication between these two services, it might be a good idea that the error details are structured as a proto message. The first (and probably) more obvious approach is to extend the Storage response message and add the extra Error message inside it:

    message SaveError {
      int32 first_field = 1;
      bool second_field = 2;
    }

    message SaveResp {
      string first = 1;
      string second = 2;  
      SaveError error = 3;  
    }

    service Storage {
      rpc (SaveRequest) returns (SaveResponse);  
    }

This way the SaveError provides enough information for the Submit service to reproduce the descriptive error string and finally show it to the end user.

     func (s *submitService) Submit(ctx context.Context, req *pb.Req) (*pb.Resp, error) {
      resp, err := storageClient.Save(ctx, req)
      if err != nil {
        // some other error
      }
      if resp.Error != nil {
        return nil, createCustomError(resp.Error)  
      }
    }

But there’s something off with this kind of error handling. The fact that we should handle two sources of error from a single RPC doesn’t sound like a good idea and might introduce some unnecessary complexity and confusion (it suggests that our client call might still be failed even though the RPC itself was successful!).

What if we want the Storage service to provide the much needed error details but not exactly in the body of the response? We can add Details to our status in form of proto messages:

    func (s *storageService) Save(ctx context.Context, req *pb.Req) (*pb.Resp, error) {
      // some logic
      if cantSave {
        st := status.New(codes.FailedPrecondition, "any grpc message")
        st, err := st.WithDetails(&pb.SaveError{...})
        handle(err)  
        return nil, st.Err()  
      }
    }

The st.WithDetails() should be given an instance of proto.Message. Feel free to pass your own custom proto messages, or use the errdetails package. From the caller perspective, the detail can be simply retrieved as well:

    func (s *submitService) Submit(ctx context.Context, req *pb.Req) (*pb.Resp, error) {
      resp, err := storageClient.Save(ctx, req)
      if err != nil {
        st := status.Convert(err)
        for _, d := range st.Details() {
          switch detail := d.(type) {
          case *pb.SaveError:  
            return nil, createCustomError(detail)  
          default:
            // other details
          }
        }
      }
    }

Now we have the luxury of not only conditioning on the status code, but to rely on a documented and agreed upon detail without being overly dependent on any other service. Yay!

More on status details in this wonderful article by Johan Brandhorst.

Status under the hood

We’ve seen how the client transport provides status in any gRPC error, so we might have a rough idea of how the situation is handled from the server’s point of view, e.g. what happens under the hood when we return a status.Status with Code, Message and Details. Basically if we take a look at the http2_server.go WriteStatus method we can see how these grpc-specific headers are written in action:

    func (t *http2Server) WriteStatus(s *Stream, st *status.Status) error {
      // ...
      headerFields := make([]hpack.HeaderField, 0, 2)
      // ...
      headerFields = append(headerFields, hpack.HeaderField{
        Name: "grpc-status", 
        Value: strconv.Itoa(int(st.Code())),
      })
      headerFields = append(headerFields, hpack.HeaderField{
        Name: "grpc-message",
        Value: encodeGrpcMessage(st.Message()),
      })

      if p := st.Proto(); p != nil && len(p.Details) > 0 {
        stBytes, err := proto.Marshal(p)
        handle(err)
        headerFields = append(headerFields, hpack.HeaderField{
          Name: "grpc-status-details-bin",
          Value: encodeBinHeader(stBytes),
        })
      }  
      // use the headerFields ...
    }

gRPC uses “Trailers” to convey its final status as well as any messages and/or details in case of an error. Trailers are just a subset of headers which are sent after the response body (in contrast to the actual response headers that are sent before the response body in the HTTP protocol). More about these trailers is available in Carl Mastrangelo’s “Why does gRPC insist on Trailers?” and SoByte’s “Talking about gRPC’s Trailers Design”, both of which I really encourage you to take a look at, they’re extraordinary. Here is a couple of quotes from the latter:

So the question is, why does gRPC rely on Trailers? The core reason is to support a streaming interface. Because it is a streaming interface, it is not possible to determine the length of the data in advance, and it is not possible to use the HTTP Content-Length header. The corresponding HTTP request looks like this.

    GET /data HTTP/1.1
    Host: example.com

    HTTP/1.1 200 OK
    Server: example.com

    abc123

What? Uncertain length? Carl points out that using chunked is ambiguous. He gives the following example.

    GET /data HTTP/1.1
    Host: example.com

    HTTP/1.1 200 OK
    Server: example.com
    Transfer-Encoding: chunked

    6\r\n
    abc123\r\n
    0\r\n

Suppose there is a proxy before the client and the server. The proxy receives the response and starts forwarding the data to the client. The first thing is to send the Header part to the client, so the caller determines that this time the status code is 200, which is successful. Then the data part is forwarded paragraph by paragraph. If the server is down after the proxy forwards the first abc123, what signal does the proxy need to send to the client?
Because the status code has been sent, there is no way to change 200 to 5xx. You can’t send 0\r\n directly to end the chunked transfer, so that the client learns that the server has quit abnormally. The only thing you can do is to close the corresponding underlying connection directly, but this will consume additional resources as the client creates a new connection. So we needed to find a way to notify the client of the server error while reusing the underlying connection as much as possible. The gRPC team finally decided to use Trailers for transport.

Trailers in action

Imagine that we have a server-side streaming RPC like below. It basically sends 3 valid response messages and then an error:

    func (s *serverImpl) ServerStream(req *pb.GreetRequest, stream pb.Greeter_ServerStreamServer) error {
      ctx := stream.Context()
      for i := 0; i < 5; i++ {
        select {
        case <-ctx.Done():
          return status.Error(codes.DeadlineExceeded, "deadline exceeded")
        default:
          if i == 3 {
            st := status.New(codes.Internal, "something went wrong")
            st, _ = st.WithDetails(&errdetails.ErrorInfo{
              Reason: "some random reason",
              Domain: "some.random.domain",
              Metadata: map[string]string{
                "first": "something",
                "second": "another thing",
              },
            })
            return st.Err()
          }

          err := stream.Send(&pb.GreetResponse{Greet: fmt.Sprintf("hello %s!", req.Name)})
          if err != nil {
            log.Print("failed to send: ", err)
          }
        }
      }

      return nil
    }

After calling this method with a client (I chose just a simple gRPC client written in go) we can take a look at what’s being transmitted in something like Wireshark (I’ve omitted certain lines to reduce unwanted clutter):

    POST /test.Greeter/ServerStream

    HyperText Transfer Protocol 2
        Stream: DATA, Stream ID: 1, Length 15
            Flags: 0x01
                .... ...1 = End Stream: True

    Protocol Buffers: /test.Greeter/ServerStream,request
        Message: test.GreetRequest
            [Message Name: test.GreetRequest]
            Field(1): name = homayoon (string)

In the response stream, first we’ll receive the following packet:

    HyperText Transfer Protocol 2
        Stream: HEADERS, Stream ID: 1, Length 14, 200 OK
            Flags: 0x04
                .... ...0 = End Stream: False
                .... .1.. = End Headers: True
            Header: :status: 200 OK
            Header: content-type: application/grpc
        Stream: DATA, Stream ID: 1, Length 22
            Flags: 0x00
                .... ...0 = End Stream: False
            0... .... .... .... .... .... .... .... = Reserved: 0x0
            .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
            [Pad Length: 0]
            DATA payload (22 bytes)

    Protocol Buffers: /test.Greeter/ServerStream,response
        Message: test.GreetResponse
            [Message Name: test.GreetResponse]
            Field(1): greet = hello homayoon! (string)

followed by two other packets without the :status 200 OK header (since it’s already sent in the first one):

    HyperText Transfer Protocol 2
        Stream: DATA, Stream ID: 1, Length 22
            Flags: 0x00
                .... ...0 = End Stream: False

    Protocol Buffers: /test.Greeter/ServerStream,response
        Message: test.GreetResponse
            [Message Name: test.GreetResponse]
            Field(1): greet = hello homayoon! (string)

Finally we’ll have the final packet which contains the gRPC trailers (notice the End Stream: True:

    HyperText Transfer Protocol 2
        Stream: HEADERS, Stream ID: 1, Length 226
            Flags: 0x05
                .... ...1 = End Stream: True
                .... .1.. = End Headers: True
            Header: grpc-status: 13
            Header: grpc-message: something went wrong
            Header: grpc-status-details-bin: CA0SFHNvbWV0aGluZyB3ZW50IHdyb25nGoEBCih0eXBlLmdvb2dsZWFwaXMuY29tL2dvb2dsZS5ycGMuRXJyb3JJbmZvElUKEnNvbWUgcmFuZG9tIHJlYXNvbhISc29tZS5yYW5kb20uZG9tYWluGhIKBWZpcnN0Eglzb21ldGhpbmcaFwoGc2Vjb25kEg1hbm90aGVyIHRoaW

Needless to say that we’ll easily be able to retrieve the original details in the grpc-status-details-bin, let’s do it the hard way and just copy the base64 encoded string above (rather than retrieving it in the gRPC client):

    import (
       spb "google.golang.org/genproto/googleapis/rpc/status"
    )

    d := "CA0SFHNvbWV0aGluZyB3ZW50IHdyb25nGoEBCih0eXBlLmdvb2dsZWFwaXMuY29tL2dvb2dsZS5ycGMuRXJyb3JJbmZvElUKEnNvbWUgcmFuZG9tIHJlYXNvbhISc29tZS5yYW5kb20uZG9tYWluGhIKBWZpcnN0Eglzb21ldGhpbmcaFwoGc2Vjb25kEg1hbm90aGVyIHRoaW5n"
    b, _ := base64.StdEncoding.DecodeString(d)
    st := &spb.Status{}
    proto.Unmarshal(b, st)
    for _, dt := range st.Details {
      errD := &errdetails.ErrorInfo{}
      dt.UnmarshalTo(errD)
      b, _ := json.MarshalIndent(errD, "", "  ")
      fmt.Println(string(b))
    }

Viola!

    {
      "reason": "some random reason",
      "domain": "some.random.domain",
      "metadata": {
        "first": "something",
        "second": "another thing"
      }
    }

Conclusion

In this article we explored how to provide descriptive and informative errors in a gRPC microservice architecture communicating via gRPC. We did also take a look at how different gRPC trailers are handled under the hood. Finally, we discussed the gRPC trailers and experimented with a mock server to better understand how and when these headers/trailers are passed. I hope you’ve found this article helpful. I’d be grateful if you would like and share this article as well.