Graceful shutdown of your legacy backend service

Introduction

If you are dealing with a multinode backend legacy application using stickysessions, consider implementing a graceful shutdown. Your service consumers will thank you when you are servicing nodes with zero downtime and session errors. There are many ways to do this but your loadbalancer needs to provide some supporting functionality - if you don't want all consuming application(s) to implement session restore capability that is.

Painpoint

If you recognize the below pattern you are already one step ahead. It means you have taken operations seriously and configured some sort of healthcheck or other monitoring for your problematic legacy web service. What you see is the periodic downtime or errorrate increasing each time a backend node is taken offline, e.g. for deployment (Thanks to Jose Arroyo Rodriguez for the inspiration).

Solution

There are two main ways to tackle this

1. Implement shared state for the nodes. This option could be too costly if you are dealing with a huge legacy application where you probably don't want to touch to much of it's code. You also need some additional loadbalancer capabilities anyways when your node "signs out" and it could have a negative performance impact.
2. Use a loadbalancer (LB) with configureable graceful shutdown of sessions. This happens when you node says "I'm out", e.g. by specific http statuscode or body in a healthcheck service operation. The LB then stops sending new sessions to the node but keeps existing ones alive. This allows for sessions to dry out naturaly, but may take a while. After draining the node it can then be shutdown gracefully.

Conclusion

Other than the above you can of course always convince your consumers to implement retry/session restore capabilities. This could even be more cost-efficient if you only have a single consumer. I find solution approach no. 2 the most elegant but it requires a powerful LB. As far as I know only HAProxy and F5 support it in a manner that does not involve changing configurations every time.

Do you know a better solution? Any feedback, recommendations are much appreciated ;)

A more comprehensive article on the matter can be read here in the context of NodeJS: Implementing NodeJS HTTP Graceful Shutdown.

Comments

[Mariano Martinez Peck]

Great tip! That sounds very useful for Seaside apps, right?

[Thomas Wegmann]

I'm not familiar with seaside in particular, but it could be usefull for any app that was not designed in a stateless manner. If it was, usually a simple round-robin LB is sufficient