Friday afternoon. The code is merged, the pipeline is green and the release is ready. But instead of pressing the deploy button and heading out for the weekend, your team is haggling who needs to stick around "just in case" everything catches fire.
Let’s be real. Deploying new code to production is like placing a bet on the roulette table.
You start the app, watch the server logs like a hawk, and hope the customer service channel is silent. If an alert is triggered, there is panic. You hustle to determine what broke, patch it live, or execute a nasty manual rollback while users are down.
Does this sound familiar? This is a stressful routine and is very typical in software engineering. But it doesn’t have to be this way. Modern traffic-shifting tactics on Amazon Elastic Container Service (ECS) can turn stressful releases into repetitive, automated processes.
Now you can take control of your deployments with Blue-Green and Canary tactics.
The Vicious Cycle of Deployment Dread
Here is the point … when delivering software is hard, teams naturally want to do it less often.
If releasing code takes downtime, late night coordination and significant risk, you start batching your changes. You don't deploy a single minor feature on a Tuesday, you combine three weeks of updates into one enormous weekend release.
And that’s a huge problem. Larger batches of code have more variables, more possible conflicts and a much greater danger of breaking something vital. The pain when that huge discharge inevitably bombs strengthens your fear of deploying.
To break this loop you need to be able to test in a true production environment without impacting the users and you also need to be able to roll back changes immediately if something goes wrong.
The Two-House Strategy: Blue-Green Deployments
Imagine that you are moving into a new residence. You simply build an exact replica of your house next door instead of loading all your stuff into a truck, selling your old house and praying the new one has functional plumbing.
You bring your furniture in, check the water pressure, sleep on the bed for a night, and make sure it’s perfect. Then when you are happy you just move your mailing address to the new house. If the roof starts leaking the next day, you just change your address back and move next door to your previous, perfectly functional home.
This is called a Blue-Green deployment in the cloud world.
How the Dual Environment Works
You have the same infrastructural settings. We can call the existing live environment “Blue” and the freshly updated environment “Green”.
Step 1: The Shadow Launch. Your ECS pipeline distributes the new container image to the Green environment. At this stage 100% of your live client traffic is still going to the Blue environment.
Step 2: Private Validation. Your team does testing against the green environment. It is a totally accurate testing ground because it is connected to the same production databases and employs the same networking rules.
Step 3: The Swap. When the Green environment has passed all of the checks, you adjust your Application Load Balancer (ALB) to send traffic to the Green environment.
Step 4: The Safety Net. The old Blue environment is still running for a pre-determined cool down period.
You’re probably wondering… what if a little bug snuck through our testing? You just flip the load balancer back to Blue. The rollback is seconds, not hours.
Defining Blue-Green Infrastructure
To get this working within AWS using Terraform you need to create two target groups and let ECS manage the traffic moving.
# The target group for our current live traffic
resource "aws_lb_target_group" "primary_tg" {
name = "app-primary-tg"
port = 80
protocol = "HTTP"
vpc_id = aws_vpc.main_network.id
target_type = "ip"
health_check {
path = "/api/health"
interval = 15
healthy_threshold = 2
}
}
# The target group for our incoming new releases
resource "aws_lb_target_group" "secondary_tg" {
name = "app-secondary-tg"
port = 80
protocol = "HTTP"
vpc_id = aws_vpc.main_network.id
target_type = "ip"
health_check {
path = "/api/health"
interval = 15
healthy_threshold = 2
}
}
You set up your ECS service to use the CODE_DEPLOY controller (or native ECS deployment tools) to handle the transition between the two target groups.
resource "aws_ecs_service" "web_api" {
name = "core-web-api"
cluster = aws_ecs_cluster.production.id
task_definition = aws_ecs_task_definition.api_def.arn
desired_count = 4
deployment_controller {
type = "ECS"
}
# Instructing ECS to keep the old tasks around for a safety buffer
blue_green_deployment_config {
deployment_ready_wait_time_in_minutes = 10
terminate_blue_tasks_on_deployment_success {
enabled = true
termination_wait_time = 30
}
}
}
The Taste Test: Canary Releases
While Blue-Green provides a good safety net, it still requires all your users to move to the new version at the same time.
And here’s where it gets interesting...what if you only exposed a very small fraction of your consumers to the new code?
Think of it as making a big pot of soup for a banquet. You taste one spoonful, then serve it to 500 visitors. You put sugar instead of salt in the soup; by accident, you only spoiled one spoonful, not the whole dinner party.
Canary deployments slowly transfer traffic to the new version in small increments.
The Phased Rollout Process
Instead of a hard swap, ECS’s canary deployment uses the weighted routing features of an AWS Application Load Balancer.
Phase 1: The Initial Dip. You push the new version out, but only send 5% of your live traffic to it. The other 95% continue with the stable version.
Phase 2: Metric Monitoring. You keep a careful eye on your dashboards. Is that 5% seeing error rate spikes? Seeing database slowness growing up?
Phase 3: The Ramp-Up. If the system seems healthy after a period of time, you bump the weight up to 20%, then 50% and finally 100%.
If the new version faults out at any stage, you immediately set the ALB weights for the new version back to 0%. The bug’s blast radius is tightly controlled to a small subset of users.
Configuring Weighted Traffic
To do this, you create a listener rule on your AWS load balancer to route traffic based on the supplied weights.
resource "aws_lb_listener_rule" "canary_traffic_split" {
listener_arn = aws_lb_listener.https_listener.arn
priority = 50
action {
type = "forward"
forward {
target_group {
arn = aws_lb_target_group.stable_version.arn
weight = 90
}
target_group {
arn = aws_lb_target_group.new_release.arn
weight = 10
}
# Critical: Ensure users don't bounce between versions
stickiness {
enabled = true
duration = 3600
}
}
}
condition {
path_pattern {
values = ["/api/*"]
}
}
}
See the stickiness configuration in the codeblock. This ensures that a user who lands on the updated version of your app stays on that version for the length of their session. Randomly bouncing a user between two separate codebases every click will be a poor user experience.
Choosing Your Deployment Weapon
Fair enough... neither is always "better" than the other. They encounter quite different operating challenges.
Opt for Blue-Green when:
You are installing internal tools or APIs where a harsh cutover is tolerable.
You want to be able to conduct intensive integration tests on the exact production infrastructure before any real traffic actually hits the servers.
You want your deployment pipeline to be simple and speedy.
Opt for Canary when:
You are deploying really critical functionality (like payment gateway) where even 1 minute outage is fatal.
You need to validate business metrics (e.g. conversion rates or user engagement) and technical data before a full launch.
If the traffic volumes are high, then you will have enough data even with a small 2% slice of traffic to identify anomalies.
Rules for Safe Deployments
Whichever approach you pick, dynamically rerouting traffic adds layers of complication. You can't just lay down in these infrastructure models and expect miracles if you disregard the rest of the ecosystem.
1. The Database Schema Trap
Simply said, your database is not rollback-able, but your code is.
If you change a database column name in your new deployment and direct traffic to the new version, the previous version will immediately crash because it is expecting the old column name. If you ever have to roll back, your application is now broken forever.
If you want zero-downtime deployments then database modifications must be fully backward-compatible. You have to follow a pattern of “expand and contract”:
Deploy 1: Add the new database column (both code versions still function).
Deploy 2: Update the application code to read/write to the new column.
Deploy 3: Remove the old column days later, long after the rollback window has closed.
2. You Must Have Excellent Metrics
Driving blindfolded is like moving traffic without visibility.
How do you know a Canary release is working when you route 10% of your traffic to it? You can’t depend on users to file support tickets. You want automatic dashboards on HTTP 500s, response times, and CPU utilization.
3. Automate the Rollback
Humans panic. Systems do not.
Don’t have a stressed engineer manually modify load balancer weights when anything goes sideways. Use automatic lifecycle hooks. You can use AWS to trigger Lambda functions on ECS deploys. Your Lambda can automatically abort the deployment and roll back traffic if it finds that the new target group is failing health checks.
Key Takeaways
Blue-Green deployments enable an easy escape route by keeping two similar settings that offer a suitable testing ground before going live.
Canary releases minimize the damage of uncovered bugs by first exposing new code to a small subset of users and growing up gradually.
It must be able to work with databases backward. If your database updates break your previous code, you don't have your safety net of rollback anymore.
Observability is a must. If you don't have analytics to see how your application is functioning in real-time, you can't securely move traffic.
Conclusion
Software deployments should be dull, at the end of the day.
By using Blue-Green cutovers for a speedy escape route, or Canary rollouts to reduce the blast radius of issues, you safeguard your users from your mistakes. When engineers are no longer afraid to break the system, they code with more confidence. They merge pull more quickly. They provide little, digestible updates rather than terrifyingly large monoliths.
Long story short...investing time in your ECS deployment architecture is more than simply focusing on server health. It’s about maintaining the mental health of your engineering staff, keeping your weekends intact, and delivering uninterrupted value to your users.
About the Author
As an AWS Community Builder, I enjoy sharing the things I've learned through my own experiences and events, and I like to help others on their path. If you found this helpful or have any questions, don't hesitate to get in touch! 🚀
🔗 Connect with me on LinkedIn
Top comments (0)