A SaaS platform uses EC2 Auto Scaling (automatically launches or terminates EC2 instances based on demand) to run its API tier. Instances run the application server, hold in-memory session state for active requests, and drain gracefully on shutdown by finishing in-flight requests before terminating.
The platform deploys a new version via a rolling update. The Auto Scaling group terminates old instances and launches new ones. During each termination event, the ALB (Application Load Balancer, stops routing new requests to draining targets) deregisters the instance and marks it as draining — but active requests that were mid-flight at the moment of deregistration return 502 errors to clients.
The engineering team increases the ALB deregistration delay from 30 seconds to 120 seconds. The 502 errors continue.
What is the actual problem?
A) The ALB deregistration delay (tells ALB to wait N seconds before fully deregistering — allows in-flight requests to complete) is still too short — increase it to 300 seconds to give all in-flight requests time to complete; but the 502s continued after increasing from 30 to 120 seconds — if delay length were the only problem, the errors would have decreased, not continued unchanged
B) The EC2 Auto Scaling lifecycle hook (pauses instance termination until your code signals completion — coordinates ALB drain timing with actual EC2 termination) is not configured — without it, instances are terminated immediately after receiving the termination signal, before the ALB deregistration delay can drain connections
C) The application server is not handling SIGTERM (Linux signal sent to process on termination — process should finish in-flight work then exit cleanly) — it exits immediately on the termination signal instead of waiting for in-flight requests to finish
D) The ALB (Application Load Balancer) is sending new requests to the draining instance during the deregistration window — this is a known ALB bug at high connection counts; but ALBs do not route new requests to targets in draining state — this is not a bug
Answer in the comments.
Top comments (5)
The answer is B.
The ALB (Application Load Balancer) deregistration delay and the EC2 instance termination lifecycle are two separate, independent timers — and they are not coordinated by default.
When Auto Scaling (automatically launches or terminates EC2 instances) decides to terminate an instance, it sends a termination signal. Without a lifecycle hook, the instance is terminated almost immediately — in practice within seconds. The ALB deregistration delay (set to 120 seconds) tells the ALB to stop sending new requests and wait for in-flight requests to drain. But if the EC2 instance is already terminated before those 120 seconds elapse, there is nothing left to drain. The ALB's drain window is irrelevant because the instance is gone.
EC2 Auto Scaling lifecycle hooks (pause instance termination until your code signals completion) solve this. A
autoscaling:EC2_INSTANCE_TERMINATINGlifecycle hook puts the instance into aTerminating:Waitstate, pausing the actual termination. Your drain logic runs — waiting for in-flight requests to complete and for the ALB deregistration delay to elapse — then sendsCompleteLifecycleActionto allow termination to proceed.The correct configuration: lifecycle hook on termination → hook handler waits for drain to complete → sends CompleteLifecycleAction → Auto Scaling terminates the instance.
A — Increasing the deregistration delay to 300 seconds does not fix the problem because the instance is being terminated before the delay can do anything. The root cause is the missing lifecycle hook that coordinates timing between ALB drain and EC2 termination.
C — SIGTERM (Linux process termination signal) handling is worth verifying independently. But if SIGTERM were the only problem, increasing the deregistration delay would have reduced errors. The fact that 502s continue unchanged after the delay increase points to the instance being gone before the drain window even starts.
D — ALBs do not route new requests to targets in draining state. Deregistration correctly stops new request routing while allowing in-flight requests to complete — assuming the instance is still running.
Also, it would mean a lot to me if you could support my content and stay in touch 🙏
YouTube: youtube.com/@system-design-lab
LinkedIn: linkedin.com/in/joud-awad/
Medium Blog: joudwawad.medium.com/
Substack: joudawad.substack.com/