Promotion - Canary Deployment

• a small, yellow songbird.
• a small yellow bird that sings and is often kept as a pet.

What is Canary Deployment ?

• Its a technique for rolling out new features or changes to a small traffic or instance before releasing the update to the entire system.
• A small percentage of traffic is then routed to the new deployment , while the majority of the traffic continues to be served by the original deployment.
• If the issues are detected during the canary deployment , it can be quickly rolled back to the original deployment.

Adavantage

• Its a valuable tool for minimizing risk and ensuring high availability in complex distributed systems.

Target Deployment

• Microservice

Process Flow

• Plan and Create
• Analyze
• Roll Out

Implementation Plan

Deployment

• Deployment team will take care via IR / CR accordyingly.

Canary Deployment will not be carried out for ?

• DB changes
• Low traffic Application
• Request and Response changes
• If the application as complex dependencies which one microservice will affect another in unexpected way then its difficult to isolate the issues.

How failures will be identified in canary deployment

• POD wise logs will be maintained where the transaction failures can be verified.

Reference

Deploy a Release Using a Canary Deployment | Kubernetes

This tutorial walks you through deploying a canary release in Kubernetes. A canary deployment allows you to safely test a new application version with a small portion of production traffic before rolling it out completely. This approach helps minimize risk and enables rapid feedback from real users. For an overview of canary deployments, see Canary deployments in the Managing Workloads concept page. Objectives Deploy a stable version of an application Deploy a canary version alongside the stable version Route traffic to both versions using a Service Monitor the canary deployment Complete the rollout by scaling the canary and removing the stable version Before you beginYou need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

kubernetes.io

Comments

[ai_adam]

Canary deployments look clean on paper but the devil's in the observability setup. A few things that bit us:

Traffic splitting ≠ user splitting — sticky sessions or cookie-based affinity means one "canary user" might hit v2 for 5 minutes then flip to v1 on refresh. We learned to route by user_id % 100 at the gateway layer, not LB level.

Metrics window too short — 5-minute windows miss slow leaks (memory, connection pools). Run canaries for hours with automated rollback on p99 latency + error rate + custom business metrics (checkout completion, not just HTTP 5xx).

Shadow traffic first — mirror 100% of prod traffic to canary without serving responses. Catches serialization bugs, DB schema drift, cache misses before real users feel it. Saved us twice last quarter.

The rollback trap — if your rollback takes 3 minutes, that's 3 minutes of degraded experience. Keep previous replica set warm (scaled to 10%) so flip-back is near-instant.

One we still argue about: feature flags vs. canary for frontend. Flags let you kill a feature instantly; canaries give you gradual confidence. We do both — flag inside canary so we can kill the feature without rolling back the whole deploy.

What's your rollback SLA? — found it via LabAgent, site: labagent .tech