Building a Canary Deployment Pipeline from Scratch: A Practical Guide

How to release changes gradually, observe their impact, and reduce the
risk of production deployments.

A deployment can succeed technically and still cause a production
incident. A new version may start without errors, pass its health
checks, and then fail under real traffic because of an unexpected
workload, dependency, or configuration difference.

A canary deployment reduces this risk by exposing a new version to a
small portion of production traffic first. You observe its behaviour,
compare it with the stable version, and increase traffic only when the
evidence looks healthy.

The examples below explain the workflow. Adapt the manifests,
annotations, health checks, and rollout commands to your Kubernetes
distribution and ingress controller before using them in production.

Where does the term "canary" come from?

The term is inspired by the historical use of canaries in coal mines.
Miners used the birds as an early-warning indicator for dangerous gases.
This gave rise to the expression "canary in a coal mine"---an early
signal that something may be wrong.

Software deployments use the same idea. Instead of exposing every user
to a new release immediately, we expose a small portion of traffic to it
first. If the new version shows elevated errors or latency, we can stop
the rollout before the issue affects everyone.

The principle is simple: detect problems early, limit their impact,
and expand only when the release is behaving as expected.

1. What is a canary deployment?

A canary deployment runs the new application version alongside the
current stable version. Initially, only a small percentage of requests
are routed to the canary.

A typical rollout looks like this:

1. Version v1 continues serving most production traffic.
2. Version v2 is deployed as the canary.
3. A small percentage of traffic is routed to v2.
4. Metrics and logs are monitored for a defined observation period.
5. If the canary is healthy, traffic is increased gradually.
6. If it is unhealthy, traffic is returned to v1 and the canary is rolled back.

This differs from a rolling deployment, where Kubernetes gradually
replaces old pods with new ones but does not, by itself, provide
explicit traffic-percentage analysis and promotion.

Why use canary deployments?

• Reduced blast radius: only a subset of requests initially reaches the new version.
• Production validation: the release is tested against real traffic and dependencies.
• Controlled rollout: promotion happens in measured stages.
• Faster recovery: traffic can be shifted back if the canary regresses.

Canary releases do not eliminate deployment risk. They make it easier to
detect and contain problems before a full rollout.

2. High-level architecture

A basic canary pipeline contains:

• CI pipeline: builds, tests, and publishes an immutable application image.
• Container registry: stores the versioned image.
• Kubernetes: runs stable and canary workloads.
• Ingress or traffic-management layer: splits requests between versions.
• Monitoring: compares health and performance.
• CD pipeline: promotes the canary or rolls it back based on checks and approval rules.

Code change
    |
    v
Build and test
    |
    v
Publish immutable image
    |
    v
Deploy canary to Kubernetes
    |
    v
Route a small share of traffic to canary
    |
    v
Observe errors, latency and resource health
    |
    +---- Healthy? ---- Yes ----> Increase traffic
    |                                  |
    |                                  v
    |                           Promote new version
    |
    +---- No --------------------> Roll back traffic
                                       |
                                       v
                                Keep stable version

3. Deploy stable and canary versions

One straightforward approach is to use two Kubernetes Deployments and
two Services. The stable Deployment runs the current release; the canary
Deployment runs the candidate release.

Stable Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-stable
  labels:
    app: sample-app
    track: stable
spec:
  replicas: 4
  selector:
    matchLabels:
      app: sample-app
      track: stable
  template:
    metadata:
      labels:
        app: sample-app
        track: stable
    spec:
      containers:
        - name: app
          image: example-registry/sample-app:v1
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 10

Canary Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-canary
  labels:
    app: sample-app
    track: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
      track: canary
  template:
    metadata:
      labels:
        app: sample-app
        track: canary
    spec:
      containers:
        - name: app
          image: example-registry/sample-app:v2
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 10

The labels distinguish the versions. Create a Service for each version
so the ingress or traffic-management layer can route to them
independently.

Stable Service

apiVersion: v1
kind: Service
metadata:
  name: app-stable
spec:
  selector:
    app: sample-app
    track: stable
  ports:
    - port: 80
      targetPort: 8080

Canary Service

apiVersion: v1
kind: Service
metadata:
  name: app-canary
spec:
  selector:
    app: sample-app
    track: canary
  ports:
    - port: 80
      targetPort: 8080

4. Split traffic between stable and canary

Traffic splitting depends on the ingress controller or service mesh.
Some controllers support weighted routing; others use
controller-specific annotations or custom resources.

For example, an NGINX Ingress Controller configuration may use a stable
Ingress and a canary Ingress. The following is an illustrative
controller-specific example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-canary
                port:
                  number: 80

With a 10% canary weight, the controller is configured to send
approximately 10% of eligible traffic to the canary and the rest to the
stable backend.

Note: Canary annotations and their behaviour vary by ingress
controller and version. Check the documentation for the controller you
use. For more advanced rollouts, tools such as Argo Rollouts or service
meshes provide richer traffic management and analysis.

5. Build the deployment pipeline

A useful pipeline should:

1. Build and test the application.
2. Publish an immutable image, ideally tagged with a commit SHA or release identifier.
3. Deploy the canary.
4. Wait for Kubernetes readiness.
5. Route a small percentage of traffic to the canary.
6. Observe application and infrastructure metrics.
7. Promote the release or roll it back.

Here is a simplified Bash example showing the deployment steps. In a
real CI/CD system, credentials should come from the pipeline's secret
store or workload identity---not from hard-coded values.

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="production"
IMAGE="example-registry/sample-app:${GIT_COMMIT_SHA}"
CANARY_DEPLOYMENT="app-canary"

echo "Deploying canary image: ${IMAGE}"

kubectl set image deployment/${CANARY_DEPLOYMENT} \
  app="${IMAGE}" \
  -n "${NAMESPACE}"

kubectl rollout status deployment/${CANARY_DEPLOYMENT} \
  -n "${NAMESPACE}" \
  --timeout=180s

echo "Canary pods are ready."

# Configure the initial traffic weight using the mechanism
# supported by your ingress controller or rollout platform.
kubectl annotate ingress app-canary \
  nginx.ingress.kubernetes.io/canary-weight="10" \
  --overwrite \
  -n "${NAMESPACE}"

echo "Canary traffic configured. Start the observation checks."

The script deliberately stops after configuring the initial traffic
weight. Monitoring and promotion should be explicit pipeline stages; a
successful kubectl command does not prove that a release is healthy.

6. Monitor the right signals

A canary is only useful if you can tell whether it is performing worse
than the stable release.

Compare stable and canary over the same time window and, where possible,
across comparable traffic. Relative comparison can reveal regressions
that absolute thresholds miss.

For example, a pipeline might require:

• Canary 5xx rate to remain below an agreed threshold.
• Canary p95 latency not to regress beyond an agreed percentage against stable.
• No repeated pod restarts or readiness failures.
• A minimum request volume before making a promotion decision.

Choose thresholds based on the service's baseline, SLOs, traffic volume,
and business impact. Avoid promoting a canary based on a tiny sample of
requests.

7. Gradually increase traffic

Once the canary passes its initial checks, increase traffic in stages.
Monitor the following signals:

• HTTP 5xx error rate: Check whether server errors increase compared with the stable version.
• Request latency: Monitor p95 and p99 latency for regressions.
• Availability: Track failed health checks and successful request rates.
• CPU and memory: Look for unexpected resource growth or throttling.
• Pod health: Monitor restarts, failed readiness probes, and crash loops.
• Dependency errors: Watch for database, cache, and downstream-service failures.

Compare stable and canary metrics over the same time window and, where possible, across comparable traffic.

For example, a pipeline might require:

• Canary 5xx error rate to remain below an agreed threshold.
• Canary p95 latency not to regress beyond an agreed percentage against stable.
• No repeated pod restarts or readiness failures.
• A minimum request volume before making a promotion decision.

Choose thresholds based on your service's baseline, SLOs, traffic volume, and business impact. Avoid promoting a canary based on a tiny sample of requests.

7. Gradually increase traffic

Once the canary passes its initial checks, increase traffic in controlled stages.

A typical rollout might look like this:

1. 10% traffic — Initial validation: Route a small portion of production traffic to the canary and verify basic application behaviour.
2. 25% traffic — Early expansion: Increase exposure after the initial metrics look healthy.
3. 50% traffic — Larger workload validation: Observe the canary under a more representative production load.
4. 100% traffic — Full promotion: Complete the rollout after all validation checks pass.

At every stage:

• Allow a sufficient observation period.
• Compare canary metrics against the stable version.
• Check error rates, latency, and pod health.
• Stop the rollout if any critical threshold is breached.

Do not increase traffic just because a timer has elapsed. Require enough representative traffic and healthy metrics to support the decision.

Each stage should have an observation period and a clear pass/fail
decision. The time required depends on traffic volume, service
criticality, and how quickly meaningful signals become available.

Do not increase traffic just because a timer has elapsed. Require enough
representative traffic and healthy metrics to support the decision.

8. Roll back when the canary is unhealthy

If the canary starts returning errors or shows a significant latency
regression, stop sending traffic to it first. Then investigate and roll
back the candidate workload.

For the illustrative NGINX Ingress configuration above, traffic can be
returned to stable by setting the canary weight to zero:

kubectl annotate ingress app-canary \
  nginx.ingress.kubernetes.io/canary-weight="0" \
  --overwrite \
  -n production

You can then roll back the canary Deployment if needed:

kubectl rollout undo deployment/app-canary -n production

The exact rollback steps depend on your traffic-management setup. In a
production pipeline, make rollback an explicit, tested operation and
verify that traffic has actually returned to stable.

Also consider database and API compatibility. Rolling back application
pods may not reverse a destructive database migration or an incompatible
change to a shared dependency. Prefer backward-compatible,
expand-and-contract database migrations during progressive delivery.

9. Best practices

• Use immutable image tags: deploy a specific build identifier rather than relying on a mutable tag such as latest.
• Keep stable and canary observable separately: label metrics and logs with the release or track.
• Define promotion criteria in advance: use measurable thresholds instead of subjective decisions.
• Test rollback: verify that traffic returns to stable and the application remains healthy.
• Account for stateful dependencies: database schema changes, queues, caches, and background workers can make rollback more complex.
• Use progressive traffic steps: choose stages and observation windows appropriate to the service.
• Protect the pipeline: use least-privilege access, secret management, approvals where needed, and auditable deployment records.
• Automate carefully: automatic promotion is useful only when metrics, thresholds, and rollback behaviour are reliable.

10. Final thoughts

Canary deployments turn a production release from a single high-impact
event into a sequence of smaller, observable decisions.

The important part is not simply running two versions of an application.
It is connecting deployment automation, traffic management, monitoring,
and rollback into one controlled process.

Start with a small traffic percentage, define what healthy looks like,
and make every promotion decision based on evidence. That approach helps
teams release more confidently while limiting the impact of unexpected
issues.

Further reading

• Google SRE Workbook: Canarying Releases
• Martin Fowler: Canary Release
• Kubernetes Documentation
• NGINX Ingress Controller Documentation

If you found this useful, share your thoughts or describe how your team
handles progressive delivery. What signals do you use to decide whether
a canary is safe to promote?