Automatically Reporting Errors from Go gRPC Services

Introduction

When developing (micro)services, I always see the need to implement automatic reporting of internal errors, no matter the language or framework. I've done this for years with my Node.js HTTP and gRPC services and now that I'm writing gRPC services in Go I've had to learn how to do it within this tech stack. Therefore I'm writing up this tutorial to show how it's done.

As you have two fundamental types of error in Go, i.e. error instances and panics, to detect internal errors in a gRPC service written in Go, you have to both catch non-client errors being returned from request handlers and trap panics. By doing some research, I found out that the best way of accomplishing this was to use what Go-gRPC calls interceptors. These are essentially hooks to let you process incoming requests and call the handler yourself.

When it comes to the actual reporting of errors, I've chosen to use email in this tutorial and more specifically the Mandrill service for sending them. There's no particular reason for my choosing Mandrill except that I've used that service for many years myself and can easily use it. You may want to adjust the code to use another mailing service if you like (that's the beauty of open source after all).

This tutorial will discuss an example gRPC-Go microservice I've written that does nothing but provide methods for failing with an internal error and panicking, respectively. If you've configured the service correctly, it will report both types of internal error to your provided email address.

Adding the gRPC Interceptor

Since the request handlers are non-streaming, I'm using the
UnaryInterceptor option to grpc.NewServer and instantiating it with the interceptor function. The interceptor does two things: Detect internal errors and report them and trap and report panics. I will discuss both scenarios in the two following sections. The code for the interceptor is just below:

// Interceptor intercepts every gRPC request.
// If an error with either the code Unknown or Internal is returned from the handler,
// we report it by email.
func NewInterceptor(conf Config) grpc.UnaryServerInterceptor {
  return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler) (resp interface{}, err error) {
    logger := log.With().Str("component", "server.interceptor").Logger()
    logger.Debug().Msgf("Processing request")

    defer func() {
      if r := recover(); r != nil {
        err = status.Errorf(codes.Internal, "panic: %s", r)
        logger.Error().Msgf("Panic detected: %s", r)
        if e := SendBugMail(errors.Wrap(err, 0), conf, logger); e != nil {
          logger.Error().Err(e).Msgf("Reporting bug by email failed")
        }
      }
    }()

    resp, err = handler(ctx, req)
    errCode := status.Code(err)
    if errCode == codes.Unknown || errCode == codes.Internal {
      logger.Debug().Err(err).Msgf("Request handler returned an internal error - reporting it")
      if e := SendBugMail(errors.Wrap(err, 0), conf, logger); e != nil {
        logger.Error().Err(e).Msgf("Reporting bug by email failed")
      }

      return
    }

    logger.Debug().Msgf("Request finished successfully")
    return
  }
}

Reporting Internal Errors from Request Handlers

Detecting internal errors, i.e. errors not caused by the gRPC client, is quite simple. It's just a matter of checking whether the error returned by the request handler has code Unknown or Internal. If the returned error has either of those codes, a bug report is sent by email to the configured recipient. I've included the snippet of the source code responsible for detecting and handling internal errors from request handlers below:

resp, err = handler(ctx, req)
errCode := status.Code(err)
if errCode == codes.Unknown || errCode == codes.Internal {
  logger.Debug().Err(err).Msgf("Request handler returned an internal error - reporting it")
  if e := SendBugMail(errors.Wrap(err, 0), conf, logger); e != nil {
    logger.Error().Err(e).Msgf("Reporting bug by email failed")
  }

  return
}

logger.Debug().Msgf("Request finished successfully")
return

It's worth mentioning that I use a particular error wrapping library,
go-errors/errors, to get error stacktraces as these are not available with vanilla Go errors. The bug report includes the formatted stacktrace, which is really helpful in tracking down bugs.

Reporting Panics in Request Handlers

In order to trap panics in request handlers, one has to use the
recover function. What's more, it must be called within the interceptor as each request is handled in its own goroutine, and if recover were called in the main goroutine it wouldn't trap panics in request handlers.

As you can see from the code, recover gets called within a deferred function. This is necessary for recover to trap panics, and it also enables us to modify the error returned from the interceptor (by re-assigning to the named return variable err). I've chosen to return error code Internal for panics, instead of aborting the process as is the default behaviour. It's apparently common practice in gRPC middleware to do this, and there seems to be little benefit to crash the process due to a panicking request handler, so I've done the same.

defer func() {
  if r := recover(); r != nil {
    err = status.Errorf(codes.Internal, "panic: %s", r)
    logger.Error().Msgf("Panic detected: %s", r)
    if e := SendBugMail(errors.Wrap(err, 0), conf, logger); e != nil {
      logger.Error().Err(e).Msgf("Reporting bug by email failed")
    }
  }
}()

Sending Bug Reports

Let's discuss the sending of bug reports. As mentioned before, I've decided on using the Mandrill mailing service for this tutorial, simply because it's the one I'm the most familiar with. The sending of error reports is encapsulated in the function SendBugMail. It uses the Mandrill REST API and quite simply composes an HTML document with the bug report, including the error stacktrace, and posts it to the Mandrill endpoint as a JSON object with the required parameters.

func SendBugMail(caughtErr *errors.Error, conf Config, logger zerolog.Logger) error {
  client := &http.Client{
    Timeout: time.Minute,
  }

  msg := strings.ReplaceAll(caughtErr.ErrorStack(), "\n", "<br>")
  version := "(master)"
  dateTimeStr := time.Now().UTC().String()
  body := map[string]interface{}{
    "key": conf.MandrillSecret,
    "message": map[string]interface{}{
      "html": fmt.Sprintf(`<p>%s - an error was detected in Service
%s.</p>

<pre><code>%s</code></pre>
`, dateTimeStr, version, msg),
      "subject":    fmt.Sprintf("Error Detected in Service %s", version),
      "from_email": conf.SenderEmail,
      "from_name":  conf.SenderName,
      "to": []map[string]string{
        map[string]string{
          "email": conf.RecipientEmail,
          "name":  conf.RecipientName,
          "type":  "to",
        },
      },
      "headers": map[string]string{
        "Reply-To": conf.SenderEmail,
      },
    },
  }

  var bodyJson []byte
  bodyJson, err := json.Marshal(body)
  if err != nil {
    return err
  }
  resp, err := client.Post("https://mandrillapp.com/api/1.0/messages/send.json", "application/json",
    bytes.NewReader(bodyJson))
  if err != nil {
    return err
  }
  defer resp.Body.Close()

  b, err := ioutil.ReadAll(resp.Body)
  if err != nil {
    return err
  }
  var respBody map[string]interface{}
  if err := json.Unmarshal(b, &respBody); err == nil && respBody["status"] == "error" {
    return errors.Errorf("Mailing failed: %s", respBody["message"])
  }

  logger.Debug().Msgf("Finished sending bug report by mail")

  return nil
}

Testing

In order to test the service, and its error reporting, you'll first of all need a Mandrill account. Then get an API token for said account, and create a file config.toml in the service Git repository, with contents as follows (make sure to replace values enclosed in angle brackets):

mandrillSecret = <mandrill-token>
senderEmail = <email-address-for-sender>
senderName = <name-of-sender>
recipientEmail = <email-address-for-recipient>
recipientName = <name-of-recipient>.

After creating config.toml, enter the repository in a terminal and start the server:

$ ./build/service server
1:40PM INF listening on port 9000... component=app

After doing so, you can test both error and panic detection, in another terminal:

$ ./build/service fail
1:40PM INF Calling service
1:40PM INF Called service successfully
$ ./build/service panic
1:41PM INF Calling service
1:41PM INF Called service successfully

If you've configured everything correctly, you should get a couple of error reports mailed to you.

Conclusion

That's it! Hope you found this tutorial useful in learning how to best write gRPC microservices in Go!