Roll out new versions without losing any requests: health checks, overlap of containers, connection draining, and instantaneous rollbacks
Why the naive deploy drops requests
The naïve approach to deployment – stopping the old container and starting the new one – fails the user in two ways. During the window between stop and the new container being ready, every request will be served by an HTTP connection error; and at the stop point, requests currently being served will be abruptly cut off mid-stream. Once per day, this is a nuisance; continuously, it is a persistent stream of errors for users.
Rolling update process
Zero-downtime updates overcome the two failure windows by deploying new and old containers simultaneously via the reverse proxy:
Run the new container along with the old one in the proxy network
Ensure the health check passes for the new container; note that “running” does not mean “ready”; readiness implies availability of all dependencies and serving traffic
Update the route of the proxy to the new container atomically
Drain the old container; allow in-progress requests to complete (with a delay of 10-30 seconds)
Terminate and delete the old container but keep its image for rollback
Two things your app is responsible for
Orchestration is taken care of by the platform infrastructure itself, but it can only do so much based on what your application gives it. The first one is an honest health check endpoint: report 200 only when ready to handle requests (database up and caches available), as it gates the rollouts. The second one is clean shutdown after a SIGTERM.
// the pattern in any language
onSignal('SIGTERM', async () => {
server.stopAccepting();
await server.drainInflight({ timeout: '8s' });
process.exit(0);
});
The edge cases worth knowing
Database migrations: run before the switch, and keep each migration compatible with the previous release, both versions briefly run against the same schema
Long-lived connections (WebSockets, SSE): draining cannot wait forever; clients must reconnect gracefully, which well-built realtime clients already do
Background workers: rolling updates apply too, finish the current job on SIGTERM and let the queue redeliver; make jobs idempotent
Singleton constraints: if two instances must never overlap (a legacy cron-in-app), fix the design or accept a maintenance-window deploy for that service
Rollbacks and failed deploys
The same machinery gives you two safety properties for free. If the new container never passes its health check, the rollout aborts and traffic never left the working version, a bad build becomes a log entry instead of an outage. And because the previous image remains on the host, rollback is the identical sequence pointed at the old image: seconds, no rebuild, no drama. In Peon both behaviours are defaults, deploys gate on health, and every previous release is one click away.
Top comments (0)