Panic, Recover, and Relax

Art by Ashley McNamara

Panic at the Disc-Go

We have all encountered panics when writing Go programs:

Panic is a built-in function that stops the ordinary flow of control and begins panicking. When the function F calls panic, execution of F stops, any deferred functions in F are executed normally, and then F returns to its caller. To the caller, F then behaves like a call to panic. The process continues up the stack until all functions in the current goroutine have returned, at which point the program crashes. Panics can be initiated by invoking panic directly. They can also be caused by runtime errors, such as out-of-bounds array accesses

Panics are required when there is simply no alternate behaviour for the relevant stack to exhibit. However, crashing our entire application abruptly may be undesirable for our system and our users. For example, web server frameworks will recover from panics caused by user-defined handler stacks so that a concurrent panic only affects a single request. See Gin's default recovery middleware for example.

Panics can only be recovered from within the goroutine that the panic occurred. This is because the Go runtime causes the entire process to exit when the panicking goroutine's stack has been completely unrolled.

So if we import a third party module that runs its own goroutines, there is simply no way we can stop our application from crashing if those goroutines panic. However, most third party modules probably don’t control concurrency themselves. Arguably they shouldn’t, and don’t need to, control concurrency when being reused by other projects; there is always a level of abstraction that can allow for other users to import your code and control concurrency for themselves. Anywho, thats not important right now. But that stance on concurrency is perhaps enough for us to justify further exploration of how we can build Go applications that guarantee panic recovery. So please indulge me on this.

By the way, if you are interested in digging into how panic and recover are implemented in the Go runtime, I will publish a companion article shortly.

Convincing Go to Relax

A single panic doesn’t need to end an entire application abruptly. We can (if desired) recover and shutdown gracefully, instead. But ensuring that panics are handled in complex projects is no simple feat; we can make Go relax, it just takes some convincing.

Because panic recovery is limited to goroutine scope, each recoverable goroutine needs to defer a call to recover().

Recover is a built-in function that regains control of a panicking goroutine. Recover is only useful inside deferred functions. During normal execution, a call to recover will return nil and have no other effect. If the current goroutine is panicking, a call to recover will capture the value given to panic and resume normal execution

Every time we wanted to turn a panic into an error, we would have to repeat the following snippet for each corresponding goroutine we launch:

defer func() {
    if r := recover(); r != nil {
        // Closure assigns panic to error
        err = fmt.Errorf("recovered from: %v": r)
    }
}()

We need to use an anonymous func here because we require a closure if we are to assign the recovered content to an error declared in the surrounding scope. You can imagine that most developers would not bother with this boilerplate. But even if you do bother, there are some pitfalls to be aware of. If the func within which the defer call is made is to actually return the error assigned inside the deferred closure, that func must define named return variables:

… if the deferred function is a function literal and the surrounding function has named result parameters that are in scope within the literal, the deferred function may access and modify the result parameters before they are returned

These complexities add up, making panic recovery a burden that developers often wouldn’t bother with. But we can simplify things using the relax module, which abstracts away these complexities and makes it easy for us to write relaxed Go applications which always shutdown gracefully.

Instead of using the native go keyword, you can use relax.Go() which handles all the complexities of panic recovery for us:

routine := relax.Go(func() error {
        []int{}[0] = 1 // Panic for example
})

Once you have the relax.Routine which is returned from relax.Go(), you can either wait for it to return an error or release it with a callback that handles the returned error concurrently:

// Block on routine completion
if err := routine.Wait(); err != nil {
    // Handle the error...
}

// Handle error concurrently
routine.Release(func(err error) {
    // Handle the error...
})

The relax module also provides a wrapper around errgroup. Just like relax.Routine, relax.RoutineGroup will let you run goroutines without worrying about panic recovery:

// Instantiate the routine group
group, ctx := relax.NewGroup(relax.Context())

// Launch some goroutines via group.Go()
...

// Wait for routine group to return an error
if err := group.Wait(); err != nil {
    // Handle the error...
}

You may have noticed the relax.Context() above. The relax module also provides a way of shutting down your application gracefully when SIGINT and SIGTERM signals are received. You just have to derive all your application contexts from the one returned by relax.Context().

Recover, Relax, and Shutdown Gracefully

Being able to recover from panics doesn’t necessarily mean it is appropriate for your runtime to panic, recover, and then continue.

For example, if you have a runtime that executes some logic in an infinite loop, and that logic relies on side effects (such as state recorded in the relevant struct type), introducing a recover-and-continue pattern may just swap one error state for another, due to the effect of a panic on relevant side effects. In this case, you could try factoring your loop to reset state on recovery, but it is probably much simpler to shutdown gracefully after recovering from the panic.

Not all applications care about shutting down gracefully; it is perfectly fine to crash on panic if there are no negative consequences to doing so.

So what kind of applications care about shutting down gracefully? Here are a few concrete examples:

• API servers that may leave dangling resources in their corresponding databases or abruptly terminate a large number of parallel client connections due to a panic;

• CLIs whose stdout format is regularly relied upon by, for example, unix command pipelines;

• Applications that write state to a filesystem which may produce irrecoverable state if a series of dependent file writes is interrupted by a panic; and

• Tests, such as integration tests using Go's coverage capabilities, where a panic can cause you to lose your prized test results.

When implementing your next application, have a think about the consequences of an abrupt, panic-induced crash. Will it cause UX problems? Will it cause problems for some system upstream? Will it leave dangling resources in your data store? If so, consider taking measures to keep your application relaxed in the case of panics.

But remember not to overdo it - there are plenty of situations where its simpler and safer to let the panic unroll the stack and cause the application to exit.

Thanks!

Thank you for taking the time to read this article. It has been an interesting rabbit hole to explore and I hope you enjoyed the read.

If the relax module gets any contributions and usage, we can work towards a 1.0 release. Please feel free to submit contributions or create issues and discussions.

Comments

[Serge Radinovich]

If anyone is interested, I have published a companion article which digs into the implementation of Go's panic and recover functionality in the Go runtime!

dev.to/sergerad/panic-and-recover-...