Building Microservices in Go: Graceful Shutdown

When building any long-term running process, like a webserver or a program importing data, we should consider providing a way to gracefully shut it down, the idea behind this is to provide a way to exit the process cleanly, to clean up resources and to properly cancel that said running process.

The steps to support Graceful Shutdown in Go consist of two steps:

1. Listen for OS signals, and
2. Handle those signals.

In Go those signals are provided by the os/signal package. Starting with Go 1.16 the way I like to implement Graceful Shutdown is by using os/signal.NotifyContext, this function provides an idiomatic way to propagate cancellation when using goroutines, which is usually the case when dealing with long-term running processes.

Keep in mind that depending on how our main package is implemented you may need to refactor it, having the main function to reach the following objectives:

1. Call a Parse function, if needed, like flag.Parse(), and
2. Call a run-like function.

The run function is the one orchestrating all the different types, initializing everything, connecting all the dots and perhaps using explicit Dependency Injection, and more importantly it may run a few goroutines to implement the call to signal.NotifyContext that in the end is going to handle the logic for implementing Graceful Shutdown.

Let's look at some concrete examples.

Using signal.NotifyContext

Starting in Go 1.16 signal.NotifyContext is the way I like to recommend when handling signals, this replaces the previous way where a channel was required.

For example having the same main():

func main() {
    errC := run()
    if err := <-errC; err != nil {
        fmt.Println("error", err)
    }
    fmt.Println("exiting...")
}

When using signal.Notify:

func run() <-chan error {
    errC := make(chan error, 1)

    sc := make(chan os.Signal, 1)

    signal.Notify(sc,
        os.Interrupt,
        syscall.SIGTERM,
        syscall.SIGQUIT)

    go func() {
        defer close(errC)

        fmt.Println("waiting for signal...")

        <-sc

        fmt.Println("signal received")
    }()

    return errC
}

And when using signal.NotifyContext:

func run() <-chan error {
    errC := make(chan error, 1)

    ctx, stop := signal.NotifyContext(context.Background(),
        os.Interrupt,
        syscall.SIGTERM,
        syscall.SIGQUIT)

    go func() {
        defer func() {
            stop()
            close(errC)
        }()

        fmt.Println("waiting for signal...")

        <-ctx.Done()

        fmt.Println("signal received")
    }()

    return errC
}

In practice both of them work achieve the same goal because both of them are meant to listen to signals, however the biggest difference is that signal.NotifyContext provides a context ctx that could be used for creating more complex propagation rules (like timeouts for example) that we can use to cancel other goroutines, instead of doing more work manually.

Implementing Graceful Shutdown in HTTP Servers

The code used for this post is available on Github.

Included in the standard library, in net/http, Go includes its own HTTP Server in net/http.Server, this server defines a method called Shutdown meant to be called when the server is supposed to exit and it shutdowns the server gracefully.

If we use the snippet we defined above we can write our code to handle Graceful Shutdown for HTTP servers in the following way:

    // ... other code initializing things used by this HTTP server

    go func() {
        <-ctx.Done()

        fmt.Println("Shutdown signal received")

        ctxTimeout, cancel := context.WithTimeout(context.Background(), 5*time.Second)

        defer func() {
            stop()
            cancel()
            close(errC)
        }()

        srv.SetKeepAlivesEnabled(false)

        if err := srv.Shutdown(ctxTimeout); err != nil {
            errC <- err
        }

        fmt.Println("Shutdown completed")
    }()

    go func() {
        fmt.Println("Listening and serving")

        // "ListenAndServe always returns a non-nil error. After Shutdown or Close, the returned error is
        // ErrServerClosed."
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            errC <- err
        }
    }()

    return errC

Conclusion

The goal of implementing Graceful Shutdowns is to allow defining some clean-up steps when dealing with a long-running process, in cases where perhaps we need to commit some database transactions, remove some used files or maybe trigger an event to indicate some other process should take over the subsequent events.