DEV Community: Anuj Tyagi

Canary Deployments with Flagger

Anuj Tyagi — Tue, 01 Jul 2025 03:59:04 +0000

Introduction

In the fast-paced world of software deployment, the ability to release new features safely and efficiently can make or break your application's reliability. Canary deployments have emerged as a critical strategy for minimizing risk while maintaining continuous delivery. In this comprehensive guide, we'll explore how to implement robust canary deployments using Flagger, a progressive delivery operator for Kubernetes.

What is Canary Deployment?

Canary deployment is a technique for rolling out new features or changes to a small subset of users before releasing the update to the entire system. Named after the "canary in a coal mine" practice, this approach allows you to detect issues early and rollback quickly if problems arise.

Instead of replacing your entire application at once, canary deployments gradually shift traffic from the stable version (primary) to the new version (canary), monitoring key metrics throughout the process. If the metrics indicate problems, the deployment automatically rolls back to the stable version.

Why Choose Flagger?

Flagger is a progressive delivery operator that automates the promotion or rollback of canary deployments based on metrics analysis. Here's why it stands out:

Automated Traffic Management: Gradually shifts traffic between versions
Metrics-Driven Decisions: Uses Prometheus metrics to determine deployment success
Multiple Ingress Support: Works with NGINX, Istio, Linkerd, and more
Webhook Integration: Supports custom testing and validation hooks
HPA Integration: Seamlessly works with Horizontal Pod Autoscaler

Prerequisites and Setup

As shared above, Flagger provides multiple integration options but I used Nginx ingress controller and Prometheus for metrics.

Required Components

NGINX Ingress Controller (v1.0.2 or newer)
Horizontal Pod Autoscaler (HPA) enabled
Prometheus for metrics collection and analysis
Flagger deployed in your cluster

Verification Commands

# Check NGINX ingress controller
kubectl get service --all-namespaces | grep nginx

# Verify HPA is enabled
kubectl get hpa --all-namespaces

# Confirm Flagger installation
kubectl get all -n flagger

Step 1: Installing Flagger

Flagger can be deployed using Helm or ArgoCD. Once installed, it creates several Custom Resource Definitions (CRDs):

kubectl get crds | grep flagger
# Expected output:
# alertproviders.flagger.app
# canaries.flagger.app  
# metrictemplates.flagger.app

Step 2: Understanding Flagger's Architecture

When you deploy a canary with Flagger, it automatically creates and manages several Kubernetes objects:

Original Objects (You Provide)

deployment.apps/your-app
horizontalpodautoscaler.autoscaling/your-app
ingresses.extensions/your-app
canary.flagger.app/your-app

Generated Objects (Flagger Creates)

deployment.apps/your-app-primary
horizontalpodautoscaler.autoscaling/your-app-primary
service/your-app
service/your-app-canary
service/your-app-primary
ingresses.extensions/your-app-canary

Step 3: Creating Your First Canary Configuration

Here's a comprehensive canary configuration example:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: my-app
  namespace: production
spec:
  provider: nginx

  # Reference to your deployment
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app

  # Reference to your ingress
  ingressRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: my-app

  # Optional HPA reference
  autoscalerRef:
    apiVersion: autoscaling/v2
    kind: HorizontalPodAutoscaler
    name: my-app

  # Maximum time for canary to make progress before rollback
  progressDeadlineSeconds: 600

  service:
    port: 80
    targetPort: 8080
    portDiscovery: true

  analysis:
    # Analysis runs every minute
    interval: 1m

    # Maximum failed checks before rollback
    threshold: 5

    # Maximum traffic percentage to canary
    maxWeight: 50

    # Traffic increment step
    stepWeight: 10

    # Metrics to monitor
    metrics:
    - name: "error-rate"
      templateRef:
        name: error-rate
      thresholdRange:
        max: 0.02  # 2% error rate threshold
      interval: 1m

    - name: "latency"
      templateRef: 
        name: latency
      thresholdRange:
        max: 500  # 500ms latency threshold
      interval: 1m

    # Optional webhooks for testing
    webhooks:
    - name: load-test
      url: http://flagger-loadtester.test/
      timeout: 15s
      metadata:
        cmd: "hey -z 1m -q 10 -c 2 http://my-app-canary:8080/"

Step 4: Setting Up Service Monitors

For Prometheus to collect metrics from both primary and canary services, you need to create separate ServiceMonitor resources:

# Canary ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-canary
spec:
  endpoints:
    - port: metrics
      path: /metrics
      interval: 5s
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app-canary

---
# Primary ServiceMonitor  
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-primary
spec:
  endpoints:
    - port: metrics
      path: /metrics
      interval: 5s
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app-primary

At this point, you may find metrics discovery in the Prometheus,

Step 5: Creating Custom Metric Templates

Flagger uses MetricTemplate resources to define how metrics are calculated. Here's an example for error rate comparison:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: error-rate
spec:
  provider:
    type: prometheus
    address: http://prometheus:9090
  query: |
    sum(
      rate(
        http_requests_total{
              service="my-app-canary",
              status=~"5.*"
          }[1m]
      ) or on() vector(0))/sum(rate(
          http_requests_total{
              service="my-app-canary"
          }[1m]
      ))
    - sum(
      rate(
        http_requests_total{
              service="my-app-primary",
              status=~"5.*"
          }[1m]
      ) or on() vector(0))/sum(rate(
          http_requests_total{
              service="my-app-primary"
          }[1m]
      ))

This query calculates the difference in error rates between canary and primary versions. The or on() vector(0) ensures the query returns 0 when no metrics are available instead of failing.

Understanding the Canary Analysis Process

The Promotion Flow

When Flagger detects a new deployment, it follows this process:

Initialization: Scale up canary deployment alongside primary
Pre-rollout Checks: Execute pre-rollout webhooks
Traffic Shifting: Gradually increase traffic to canary (10% → 20% → 30% → 40% → 50%)
Metrics Analysis: Check error rates, latency, and custom metrics at each step
Promotion Decision: If all checks pass, promote canary to primary
Cleanup: Scale down old primary, update primary with canary spec

Rollback Scenarios

Flagger automatically rolls back when:

Error rate exceeds threshold
Latency exceeds threshold
Custom metric checks fail
Webhook tests fail
Failed checks counter reaches threshold

Monitoring Canary Progress

# Watch all canaries in real-time
watch kubectl get canaries --all-namespaces

# Get detailed canary status
kubectl describe canary/my-app -n production

# View Flagger logs
kubectl logs -f deployment/flagger -n flagger-system

Advanced Features

Webhooks for Enhanced Testing

Flagger supports multiple webhook types for comprehensive testing:

webhooks:
  # Manual approval before rollout
  - name: "confirm-rollout"
    type: confirm-rollout
    url: http://approval-service/gate/approve

  # Pre-deployment testing
  - name: "integration-test"
    type: pre-rollout
    url: http://test-service/
    timeout: 5m
    metadata:
      type: bash
      cmd: "run-integration-tests.sh"

  # Load testing during rollout
  - name: "load-test"
    type: rollout
    url: http://loadtester/
    metadata:
      cmd: "hey -z 2m -q 10 -c 5 http://my-app-canary/"

  # Manual promotion approval
  - name: "confirm-promotion"
    type: confirm-promotion
    url: http://approval-service/gate/approve

  # Post-deployment notifications
  - name: "slack-notification"
    type: post-rollout
    url: http://notification-service/slack

HPA Integration

When using HPA with canary deployments, Flagger pauses traffic increases while scaling operations are in progress:

autoscalerRef:
  apiVersion: autoscaling/v2
  kind: HorizontalPodAutoscaler
  name: my-app-primary
  primaryScalerReplicas:
    minReplicas: 2
    maxReplicas: 10

Alerting and Notifications

Configure alerts to be notified of canary deployment status:

analysis:
  alerts:
    - name: "canary-status"
      severity: info
      providerRef:
        name: slack-alert
        namespace: flagger-system

Production Considerations

Traffic Requirements

For effective canary analysis, you need sufficient traffic to generate meaningful metrics. If your production traffic is low:

Consider using load testing webhooks
Implement synthetic traffic generation
Adjust analysis intervals and thresholds accordingly

Metrics Selection

Choose metrics that accurately reflect your application's health:

Error Rate: Monitor 5xx responses
Latency: Track P95 or P99 response times
Custom Business Metrics: Application-specific indicators

Deployment Timing

Calculate your deployment duration:

Minimum time = interval × (maxWeight / stepWeight)
Rollback time = interval × threshold

For example, with interval=1m, maxWeight=50%, stepWeight=10%, threshold=5:

Minimum deployment time: 1m × (50/10) = 5 minutes
Rollback time: 1m × 5 = 5 minutes

Troubleshooting Common Issues

Missing Metrics

Problem: Canary fails due to missing metrics
Solution: Verify ServiceMonitor selectors match service labels

Webhook Failures

Problem: Load testing webhooks time out
Solution: Increase webhook timeout and verify load tester accessibility

HPA Conflicts

Problem: Scaling issues during canary deployment

Solution: Ensure HPA references are correctly configured for both primary and canary

Network Policies

Problem: Traffic routing issues
Solution: Verify network policies allow communication between services

Best Practices

Start Small: Begin with low traffic percentages and gradual increases
Monitor Actively: Set up comprehensive alerting for canary deployments
Test Thoroughly: Use webhooks for automated testing at each stage
Plan for Rollback: Ensure your rollback process is well-tested
Document Everything: Maintain clear documentation of your canary processes

Conclusion

Flagger provides a robust, automated solution for implementing canary deployments in Kubernetes environments. By gradually shifting traffic while monitoring key metrics, it enables safe deployments with automatic rollback capabilities.

The combination of metrics-driven analysis, webhook integration, and seamless traffic management makes Flagger an excellent choice for teams looking to implement progressive delivery practices. Start with simple configurations and gradually add more sophisticated monitoring and testing as your confidence grows.

Remember that successful canary deployments depend not just on the tooling, but also on having appropriate metrics, sufficient traffic, and well-defined success criteria. With proper implementation, Flagger can significantly reduce deployment risks while maintaining the agility your development teams need.

Additional Resources

[Boost]

Anuj Tyagi — Sun, 22 Jun 2025 19:53:14 +0000

Anuj Tyagi

Jun 20 '25

KEDA Upgrade Debugging: When Empty Triggers Break Your Scaling

#keda #eventdriven #kubernetes #debugging

Comments

4 min read

Collect AWS Lambda@Edge metrics with Prometheus

Anuj Tyagi — Fri, 20 Jun 2025 05:25:29 +0000

This post is about the problem I worked 2 years ago but should be still valid. Why? As I solved the problem internally back in past but forgot to create PR in the official public YACE github repo. If you don't undestand what I am talking about. I will expand this blog in future.

Let me explain from the beginning.

I was working on implementing monitoring for a enterprise infrastructure. I was using Prometheus with YACE (yet another cloudwatch exporter) to collect metrics.

What is YACE exporter?
It's like a plugin used with Prometheus to collect metrics from AWS. We have another option, CloudWatch exporter for the same use case but I am going ahead with YACE exporter.

From the examples, collecting metrics was straightforward but then I was stuck when I had to collect metrics from Lambda edge but unlike other examples, YACE was not supporting metrics discovery through AWS Lambda edge .

So, I created a Github Issue in YACE repo: https://github.com/prometheus-community/yet-another-cloudwatch-exporter/issues/876

I received response, Lambda@edge don't support tags so it's metrics can't be collected via service discovery. This was blocking my project so I have to somehow solve this problem.

How I solved this problem?

I figure out another approach to collect metrics by using static configuration if you know which regions are you using to collect metrics via Lambda@edge.

How to collect metrics via Static approach?

 apiVersion: v1alpha1
  static:
    - name: us-east-1.<edge_lambda_function_name>
      namespace: AWS/Lambda
      regions:
        - eu-central-1
        - us-east-1
        - us-west-2
        - ap-southeast-1
      period: 600
      length: 600
      metrics:
        - name: Invocations
          statistics: [Sum]
        - name: Errors
          statistics: [Sum]
        - name: Throttles
          statistics: [Sum]
        - name: Duration
          statistics: [Average, Maximum, Minimum, p90]

As you can see, I added all regions my Lambda@edge is using. I also created PR for this in YACE repo.

Hope this helps someone.

KEDA Upgrade Debugging: When Empty Triggers Break Your Scaling

Anuj Tyagi — Fri, 20 Jun 2025 04:20:39 +0000

This is one of the past use case to troubleshooting KEDA, Kubernetes based event driven autoscaler during upgrade in a non production environment.

So, I was working to upgrade KEDA from v2.10 to v2.15 for a infra unfamiliar to me. It was my first hands on experience with KEDA. I quickly understood purpose of KEDA, I worked more with HPA before that.
If you're not aware of the difference between all pod scaling options, you can read my last post
on Kubernetes pod scaling patterns

My goal was to upgrade KEDA from v2.10 to v2.15 and ensure all existing ScaledObjects continued to function properly. The environment had been running with KEDA v2.10 for months, and all configurations appeared to be working correctly.

Initial Error Analysis

After the upgrade, the KEDA operator logs showed concerning errors:

2024/11/04 17:57:49 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
{"level":"info","ts":"2024-11-04T17:57:49.765Z","logger":"setup","msg":"KEDA Version: 2.15.1"}
{"level":"info","ts":"2024-11-04T17:57:49.765Z","logger":"setup","msg":"Git Commit: 123543fnerfin4fcw3d23d23b"}
I1104 17:57:49.866460    1 leaderelection.go:250] attempting to acquire leader lease keda/operator.keda.sh...

The key concern was that if the last line shows only attempting to acquire leader lease without the follow-up successfully acquired lease, it means the leader is locked and can't work as leader. But this is fine. It can also mean, another pod is working as a leader.

I went ahead to understand leader election process.
Understanding KEDA's leader election process was crucial. A healthy startup sequence looks like:

I1106 21:42:09.498384       1 leaderelection.go:254] attempting to acquire leader lease keda/operator.keda.sh...
I1106 21:42:55.066863       1 leaderelection.go:268] successfully acquired lease keda/operator.keda.sh
2024-11-06T21:42:55Z    INFO    Starting EventSource    {"controller": "scaledobject"}
2024-11-06T21:42:55Z    INFO    Starting Controller {"controller": "scaledobject"}

The sequence should include:

Attempting to acquire lease
Successfully acquiring lease
Multiple controller initialization messages

Configuration Investigation

Examining the failing ScaledObject revealed the root cause:

kubectl get scaledobject app -n test-app -o yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: webapp
  namespace: test-app
  creationTimestamp: "2024-05-10T13:16:22Z"  # Created months ago
spec:
  scaleTargetRef:
    name: webapp
  minReplicaCount: 1
  maxReplicaCount: 1
  triggers: []  
status:
  conditions:
  - message: ScaledObject doesn't have correct triggers specification
    reason: ScaledObjectCheckFailed
    status: "False"
    type: Ready

The Real Issue Discovery

When I checked another KEDA operator pod, I found the root cause:

error":"no triggers defined in the ScaledObject/ScaledJob"

I spend more time searching, why KEDA was screaming for empty trigger now in v2.15 but not in v2.10. So, any release after v2.10 added this as exception and log message.

KEDA v2.10 behavior: Silently accepted empty triggers (triggers: []) and created a default HPA with 80% CPU utilization
KEDA v2.15 behavior: Validates triggers and throws errors for empty arrays
Timeline: This ScaledObject had been running incorrectly from past 6 months, but v2.10 hid the problem.

The Fix Implementation

I found specific Github issue and PR:

The empty triggers validation was introduced in:

GitHub Issue: #5520 - "KEDA doesn't validate empty array of triggers"
Pull Request: #5524 - "fix: Validate empty array value of triggers in ScaledObject/ScaledJob creation"
KEDA Version: Introduced in v2.14, refined in v2.15
Merge Date: February 2024

Configuration Fix

The solution was to add proper triggers to the ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: webapp
  namespace: test-app
spec:
  scaleTargetRef:
    name: webapp
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: http_requests_per_second
      threshold: "100"
      query: sum(rate(http_requests_total[1m]))

Validation Commands

To identify similar issues across the cluster:

# Find ScaledObjects with empty triggers
kubectl get scaledobjects -A -o jsonpath='{range .items[?(@.spec.triggers[0] == null)]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}'

# Check ScaledObject status
kubectl get scaledobjects -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[?(@.type=='Ready')].status"

Key Takeaways

1. Silent Failures Are Dangerous

KEDA v2.10's behavior of silently creating default HPAs masked configuration errors for months. The application had been using basic CPU scaling instead of the intended event-driven scaling.

2. Validation Improvements

The upgrade didn't break anything - it revealed existing problems. KEDA v2.15's strict validation prevents:

Misleading functionality (thinking event-driven scaling is active when it's not)
Resource waste from inappropriate scaling decisions
Configuration drift

3. Understanding Version Changes

Breaking changes often fix underlying issues. The validation was introduced because:

Empty triggers create meaningless ScaledObjects
Default CPU-based scaling defeats KEDA's event-driven purpose
Silent failures violate "fail fast, fail loud" principles

4. Debugging Best Practices

When investigating KEDA issues:

Check leader election sequence completion
Examine ScaledObject status conditions
Validate trigger configurations before upgrades
Test in non-production environments first

5. Prevention Strategies

Implement CI/CD validation for empty triggers
Monitor ScaledObject health status
Set up alerts for configuration failures
Review configurations before major upgrades

Conclusion

What initially appeared to be a breaking change in KEDA v2.15 was actually a long-overdue fix for silent configuration failures. The ScaledObject had been misconfigured since May 2024, but v2.10 had been hiding the problem by falling back to default CPU-based scaling.

This experience reinforces that sometimes "breaking" changes reveal existing problems rather than creating new ones. The improved validation in KEDA v2.15 ensures that event-driven autoscaling works as intended, making the system more reliable and preventing future silent failures.

Understanding the difference between a tool breaking and a tool revealing existing breakage is crucial for effective debugging and system maintenance.

Scaling patterns in Kubernetes: VPA, HPA and KEDA

Anuj Tyagi — Fri, 20 Jun 2025 02:26:52 +0000

I've been working with a mostly HPA as a scaling options in past but last year I started working with KEDA. So, I thought to write post to explain possible options in pod autoscaling. On the other side, manually adjusting parameters is not only slow but also inefficient. If you decide to allocate too little resource and you'll deliver subpar user experience or can experience application outages. If you over-provision resources "just in case" and you'll waste money and resources. That's where Kubernetes autoscaling comes to the rescue and deliver the right resources when required.

Understanding Kubernetes Pod Autoscaling Fundamentals

Autoscaling in Kubernetes means dynamically allocating cluster resources like CPU and memory to your applications based on real-time demand. This ensures applications have the right amount of resources to handle varying levels of load, directly improving application performance and availability.

Key Benefits of Autoscaling:

Cost Efficiency: Pay only for the resources you need instead of over-provisioning

Environmental Impact: Reduced power consumption and carbon emissions through better resource alignment

Time Savings: Automates manual resource adjustment tasks, freeing up valuable DevOps time
Performance Optimization: Ensures applications maintain optimal performance under varying loads

The Three Pillars of Kubernetes Autoscaling

Kubernetes offers three primary autoscaling mechanisms, each serving different purposes:

Vertical Pod Autoscaler (VPA) - Adjusts resource requests and limits within individual pods
Horizontal Pod Autoscaler (HPA) - Scales the number of pod replicas up or down
Kubernetes Event-Driven Autoscaler(KEDA) - Scales based on external events and custom metrics

Let's explore each of them one by one.

Vertical Pod Autoscaler (VPA): Right-sizing Your Pods

What is VPA?

The Vertical Pod Autoscaler automatically adjusts the CPU and memory requests and limits of individual containers within pods based on historical usage patterns. Instead of scaling the number of pods, VPA makes your existing pods "beefier" or "leaner" based on their actual resource needs.

How VPA Works

VPA operates through three core components:

Recommender: Calculates optimal resource values based on historical metrics from the Kubernetes Metrics Server, analyzing up to 8 days of data to generate recommendations.
Updater: Monitors recommendation changes and evicts pods when resource adjustments are needed, forcing replacement with updated allocations.
Admission Webhook: Intercepts new pod deployments and injects updated resource values based on VPA recommendations.

When to Use VPA

VPA is ideal for:

Stateful applications that can't be easily scaled horizontally
Resource optimization scenarios where you need to fine-tune individual pod resources
Applications with unpredictable resource patterns that traditional static allocation can't handle
Cost optimization efforts to eliminate resource waste.

VPA configuration example

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: "Deployment"
    name: "my-app"
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      maxAllowed:
        cpu: 1
        memory: 500Mi
      minAllowed:
        cpu: 100m
        memory: 50Mi

Challenges with VPA

Despite its benefits, VPA has several limitations:

Incompatibility with HPA: Cannot run both tools together for CPU/memory-based scaling

Limited historical data: Only stores 8 days of metrics, losing data on pod restarts

Service disruption: Pod evictions cause momentary service interruptions

No time-based controls: Pod evictions can happen at any time, including peak hours

Cluster-wide configuration: Limited per-workload customization options

Horizontal Pod Autoscaler (HPA): Scaling Out Your Application

What is HPA?

HPA automatically scales the number of pod replicas in a Deployment, ReplicaSet, or StatefulSet based on observed metrics like CPU utilization, memory usage, or custom metrics. It's the most fundamental and widely-used autoscaling pattern in Kubernetes.

How HPA Overcomes VPA Challenges

While VPA adjusts resources within pods, HPA takes a different approach:

No service disruption: Scaling replicas doesn't require pod eviction
Works with stateless applications: Perfect for horizontally scalable workloads
Predictable scaling: Based on well-understood metrics like CPU and memory
Mature and stable: Built-in Kubernetes feature with extensive community support

When to Use HPA

HPA is perfect for:

Stateless applications where pods are interchangeable
Predictable workloads with clear load patterns
Web applications that experience traffic variations
Microservices that can benefit from horizontal scaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

HPA Limitations

While HPA is powerful, it has constraints:

Limited to resource metrics: Basic HPA only works with CPU/memory metrics
Not suitable for event-driven workloads: Can't scale based on queue lengths or custom events
Reactive scaling: Only responds after metrics breach thresholds
No scale-to-zero: Cannot scale down to zero replicas

KEDA: Event-Driven Autoscaling for Modern Applications

What is KEDA?

Kubernetes Event-Driven Autoscaling (KEDA) extends Kubernetes' native autoscaling capabilities to allow applications to scale based on events from various sources like message queues, databases, or custom metrics. KEDA graduated as a CNCF project, highlighting its importance in the cloud-native ecosystem.

How KEDA Overcomes HPA Limitations

KEDA addresses several HPA shortcomings:

Event-driven scaling: Scales based on queue lengths, database records, HTTP requests, and more
Scale-to-zero capability: Can scale applications down to zero when no events are present
Rich ecosystem: Supports 50+ event sources including Kafka, RabbitMQ, Azure Service Bus, AWS SQS
Custom metrics: Works with any metric source through external scalers

When to Use KEDA

KEDA excels in:

Event-driven architectures with message queues and event buses
Serverless-style workloads that benefit from scale-to-zero
Batch processing jobs triggered by data availability
IoT applications processing sensor data streams Machine learning pipelines processing inference requests

KEDA configuration example

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: rabbitmq-scaler
spec:
  scaleTargetRef:
    name: message-processor
  triggers:
  - type: rabbitmq
    metadata:
      protocol: amqp
      queueName: work-queue
      mode: QueueLength
      value: "5"

KEDA vs HPA: Key Differences

Choosing the Right Autoscaling Strategy

Use VPA When:

You have stateful applications that can't scale horizontally
Resource optimization is your primary concern You need to fine-tune individual pod resources Applications have unpredictable resource usage patterns

Use HPA When:

You have stateless, horizontally scalable applications
Traditional web applications with predictable load patterns
Simple microservices that scale based on CPU/memory
You need a proven, stable autoscaling solution

Use KEDA When:

Building event-driven or serverless-style applications
Processing messages from queues or streams
Need to scale based on custom or external metrics
Cost optimization through scale-to-zero is important

Real-World Implementation Scenarios

Scenario 1: E-commerce Platform

Frontend services: HPA for web servers based on CPU utilization
Order processing: KEDA for scaling based on order queue length
Database connections: VPA for optimizing connection pool resources

Scenario 2: IoT Data Pipeline

Data ingestion: KEDA scaling based on message queue depth
Stream processing: HPA for consistent throughput requirements
Analytics services: VPA for memory-intensive data processing

Scenario 3: Machine Learning Platform

Model serving: HPA for inference API endpoints
Training jobs: KEDA triggered by training request queues
Feature processing: VPA for compute-intensive transformations

Best Practices and Recommendations

Start Simple: Begin with HPA for basic scaling needs, then add KEDA for event-driven requirements
Monitor and Adjust: Continuously monitor scaling behavior and adjust thresholds
Combine Strategies: Use different autoscalers for different components of your application
Set Resource Limits: Always define appropriate resource limits to prevent runaway scaling
Test Thoroughly: Validate autoscaling behavior under various load conditions

Conclusion
Kubernetes autoscaling is not a one-size-fits-all solution. The choice between VPA, HPA, and KEDA depends on your specific application requirements, architecture patterns, and operational needs. VPA optimizes resource utilization within pods, HPA provides reliable horizontal scaling for traditional workloads, and KEDA enables sophisticated event-driven scaling for modern cloud-native applications.
By understanding the strengths and limitations of each approach, you can design a comprehensive autoscaling strategy that optimizes both performance and cost while maintaining the reliability your applications demand.

If you are looking for a more deep dive course and hands on labs on Kubernetes Autoscaling and KEDA, you can checkout official LinuxFoundation course for no cost.

[Boost]

Anuj Tyagi — Mon, 14 Apr 2025 05:02:22 +0000

Anuj Tyagi for AWS Community Builders

Apr 14 '25

Collect Aurora audit logs in Firehose

#aws #firehose #cloudwatch #logging

Comments

3 min read

Collect Aurora audit logs in Firehose

Anuj Tyagi — Mon, 14 Apr 2025 04:55:47 +0000

In our last post, we enabled audit logs using parameter groups in Aurora Postgres.

Now, we are collecting our required Aurora logs in CloudWatch but we need to filter our those logs and send to S3 to archive for analysis and long term storage.

Why this is useful?
We can set retention on CloudWatch logs and keep our audit logs in S3. This will help to save cost. In a different use case, we can send to another external destination also for audit or analysis.

At this point, I am assuming you already have your application logs in CloudWatch. For our use case, I am collecting Aurora logs in CloudWatch as explained in part of this series. Although below use case should work for any logs in CloudWatch

In order to send logs to S3 from CloudWatch, we will create subscription filter which can help to stream log data in near realtime to destinations.

What is Subscription Filter in CloudWatch?
CloudWatch subscription filter provide filter patterns and options to deliver logs events to AWS services. It can provide log delivery of events to multiple destinations.

CloudWatch provide multiple service options to create subscription filter.

OpenSearch
Kinesis
Data Firehose
Lambda

We will go with Firehose considering log volume and cost, and deployment for Firehose is comparatively easier for our goal to steam logs to S3.
Firehose can transform records or convert format before delivery to the S3

To begin with, we need to follow these steps.

Create S3 bucket
Create Firehose Stream
Create IAM role for Firehose
Create CloudWatch subscription filter
Validation

The reason we need to follow this approach as we need S3 bucket when creating Firehose stream. For CloudWatch subscription, we need to have Firehose stream first.

Step1: Create S3 bucket
Creating S3 bucket is straightforward. You need to search S3 service and create S3 bucket with default settings.

Step2: Create Firehose Stream

You can keep the option to create IAM roles by itself.

It can take a few minutes for Firehose Stream to get created and will show active status.

Note: We can't change destination for Firehose after creating stream.

Step3: Create IAM role to allow CloudWatch logs -> Firehose

Create IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPutToFirehose",
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": "arn:aws:firehose:<region>:<account-id>:deliverystream/<your-firehose-name>"
    }
  ]
}

Create IAM role LogsToFirehose

Update Trust Policy as

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.<region>.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step4: Create CloudWatch Subscription Filter
Now, switch back to our log group in CloudWatch.

Click on Create Amazon Data Firehose subscription filter

Now, after adding filter name, we need to select Firehose stream in current account that we created in Step2.

We can also add pattern if we want to filter our logs further before sending to Firehose and add prefix.

Also, assign IAM role to grant permission to receive logs by Firehose from CloudWatch. We created this IAM role in Step3. Now, Create Subscription button.

We should see subscription filter for our logs added like this.

Step5. Validate logs
After creating subscription filter, we need to check Firehose Stream monitoring metrics, if it shows data getting collected

From our metrics, we can say, we are collecting logs.

Now, we need to go to S3 our final destination to confirm if we are getting those in bucket.

We should see logs organized in bucket with year, month and day.

That concludes our goal.

Enable Aurora logs for security audits

Anuj Tyagi — Sun, 06 Apr 2025 03:32:31 +0000

AWS Aurora provides serverless database capability with enhanced features that you can find here. By default, AWS Aurora enables error logs but audit logs are disabled. When running a database in production, one can collect logs from application and other aws services. For Aurora monitoring, we can check metrics from CloudWatch and more granular metrics by enabling performance insight. However, under certain requirements, we may need to analyze database transactions for which we want to enable audit logs. As an SRE/DevOps or DB admin, one use case I came across is when the analytics team wants to analyze those audit logs. In another situation, in order to follow PCI, PII, SOC2 and more security compliance we need to enable audit logs.

I am also adding steps of creating a cluster, assuming you want to test this in a lab or test account.

Step1: Creating the Aurora cluster

Search for Aurora service from the search bar.

Note: As shared in the above screenshot, I am using the postgreSQL engine from available options.

Here, I have selected dev/test in the template option. For Master username, you can change it, I am using default. After creating the cluster, automatically credentials will be saved in the secret manager service. You can use it to get the password to connect with the database.

Before selecting create cluster at the bottom, you may see Log exports with options

Instance log and PostgreSQL logs. In case you are thinking, if any of these options will enable audit logs, the answer is No. Let’s know about these two options first.

Note: To send PostgresSQL logs to CloudWatch, you must enable PostgreSQL log option otherwise you won't even see CloudWatch log group.

By default, Aurora enables error logs. Error store these logs to log/postgresql.log file. This will capture all errors like query failures, server errors, login failures.

If you select the PostgreSQL log option, the cluster will keep error logs but in addition enable PostgreSQL general logs like log_statement, log_duration etc and export to CloudWatch.

If you enable the instance log option, there will be a separate file created for it with name instance.log. You will it in logs and events tab.

Now, we understand the basics on log options available when creating the Aurora cluster.

Now, click on the Create Cluster from the bottom right button.

We will see cluster creation in progress and the state will be changed to available in a short time.

Step2: Enable access logs

As we have a cluster running, we need to open the cluster page.

Click on the parameter groups as highlighted in the screenshot on left sidebar.

When you click on the parameter groups link, you will see options custom and Default.

Default option will show groups that were created by default during cluster creation. We don't have permission to edit parameters in Default parameter groups.

So, we will switch back to the custom option and click on the Create Parameter group button on the right.

Now, we open our custom parameter and Click Edit on the right.

In my case, I only had to enable log_connections and log_disconnections. So, I can updated them with available binary value 1 to enable them. This means to log all connection and disconnection to track record of users login and logout. This is useful for security audits.

LOG:  connection authorized: user=app_user database=mydb application=psql host=10.0.1.5

and

LOG:  disconnection: session time: 0:00:05.233 user=app_user database=mydb host=10.0.1.5

To improve the log_line structure and add log_line_prefix and change default value from

%t:%r:%u@%d:[%p]:

%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h

This will improve log_line output structure and provide logs

2025-04-02 15:35:12 [19456]: [3-1] user=app_user,db=finance_db,app=psql,client=10.1.2.3 AUDIT: SESSION,1,1,READ,SELECT,,,SELECT * FROM accounts;

Now, we save these changes and switch back to our cluster page.

Click on the Modify option.

Click the instance and go to the configuration tab.
Search for the parameter option and update it with our custom parameter from default parameter.

It will take sometime for the cluster changes to take effect.
Now, check in the cloudwatch, you will find log-group for Aurora postgres.

That concludes our article. We will take it further in next blog.

For more granular logging, you can use pg_audit extension for which there is a detailed guide available in aws documentation.

Learn how to use oidc token instead of access+secret keys.

Anuj Tyagi — Wed, 19 Mar 2025 17:29:08 +0000

Secure continuous Integration with Dockerfile, Github Actions and AWS ECR

Anuj Tyagi ・ Mar 3 '25

#docker #awslambda #githubactions #awsecr

[Boost]

Anuj Tyagi — Wed, 19 Mar 2025 17:27:14 +0000

Docker CMD vs ENTRYPOINT: Understanding the Differences

Anuj Tyagi ・ Mar 3 '25

#docker #dockerfile #cmd #container

Docker CMD vs ENTRYPOINT: Understanding the Differences

Anuj Tyagi — Mon, 03 Mar 2025 02:06:48 +0000

Docker is a platform used to manage applications within containers. To run an application in a container, a Docker image needs to be created first which can be done using a Dockerfile. A Dockerfile is a text document that has one or more commands that are executed by Docker to build the image. Two important instructions among these are CMD and ENTRYPOINT for defining how a container operates.

In this blog post, we will explore CMD and ENTRYPOINT, their differences, and when to use each.

Prerequisites

You will need a code editor and Docker Desktop application on your device to follow through with the practical examples demonstrated in this guide.

What Is CMD in Dockerfile?

CMD any instructions within the Dockerfile that comes into effect by default. It is executed when no command is executed while using the container.

Example of CMD in a Dockerfile

Make a directory called docker-demo, and in there, a new file called Dockerfile with this content:

FROM alpine
CMD ["echo", "Welcome to AItechNav!"]

In this Dockerfile:

FROM alpine sets the base image.
CMD ["echo", "Welcome to AItechNav!"] defines the default command to run inside the container.

Build Image and Run the container

To create a Docker image, execute the following command inside the docker-demo directory:

docker build . -t custom-image:v1

Once the image is built, verify it using:

docker image list

Now, run a container using the image:

docker container run custom-image:v1

Output:

Welcome to AItechNav!

Overriding CMD at Runtime

CMD allows users to override the default command:

docker container run custom-image:v1 echo "Empowering AI Enthusiasts!"

Output:

Empowering AI Enthusiasts!

This shows that CMD acts as a default but can be replaced when running a container.

What Is ENTRYPOINT in Dockerfile?

ENTRYPOINT, like CMD, specifies the command to execute when running a container. However, unlike CMD, the ENTRYPOINT directive cannot be overridden at runtime; instead, additional arguments are appended to it.

Example of ENTRYPOINT in a Dockerfile

Modify the Dockerfile to use ENTRYPOINT instead of CMD:

FROM alpine
ENTRYPOINT ["echo", "Welcome to AItechNav!"]

Building and Running the Docker Image

Build the new image:

docker build . -t custom-image:v2

Verify the image and run a container:

docker container run custom-image:v2

Output:

Welcome to AItechNav!

Attempting to Override ENTRYPOINT at Runtime

Unlike CMD, if we attempt to override ENTRYPOINT:

docker container run custom-image:v2 echo "AI and Cloud Learning Hub!"

Output:

Welcome to AItechNav! AI and Cloud Learning Hub!

The argument is appended instead of replacing the default command.

Using CMD & ENTRYPOINT Together

You can combine CMD and ENTRYPOINT to define a fixed command while allowing users to provide different arguments.

Modify the Dockerfile:

FROM alpine
ENTRYPOINT ["echo", "Welcome to"]
CMD ["AItechNav!"]

Building and Running the Image

Build the image:

docker build . -t custom-image:v3

Run the container:

docker container run custom-image:v3

Output:

Welcome to AItechNav!

Overriding CMD but Not ENTRYPOINT

Run the container with a custom argument:

docker container run custom-image:v3 "the future of AI!"

Output:

Welcome to the future of AI!

In this setup:

ENTRYPOINT (echo Welcome to) remains unchanged.
CMD (AItechNav!) is overridden by the future of AI! at runtime.

Difference Between CMD & ENTRYPOINT

The table below highlights their differences:

Feature	CMD	ENTRYPOINT
Provides a default command	Yes	Yes
Allows command override at runtime	Yes	No (arguments are appended)

When to Use CMD vs. ENTRYPOINT

Use CMD when you want to provide a default command but allow users to override it at runtime.
Use ENTRYPOINT when you want to enforce a specific command while allowing additional parameters.
Use both CMD and ENTRYPOINT together when you want a fixed command but allow users to modify arguments.

Conclusion

CMD and ENTRYPOINT are essential Dockerfile instructions that define how a container runs. CMD provides flexibility by allowing users to override the default command, whereas ENTRYPOINT enforces a fixed command and appends additional arguments. By understanding their differences and best use cases, you can structure your Dockerfiles efficiently for various application needs.

Happy containerizing!

Understanding Dockerfile: A Guide to Building Efficient Docker Images

Anuj Tyagi — Mon, 03 Mar 2025 01:57:30 +0000

At the core of Docker's containerization process lies the Dockerfile, a powerful tool that automates the creation of Docker images. In this blog post, we will explore what a Dockerfile is, how it works, and best practices to optimize your builds. Let's dive in!

What is a Dockerfile?

A Dockerfile is a script-like text file containing instructions that define how a Docker image should be built. Each line represents a specific command followed by arguments, forming a sequential process that constructs the final image. By convention, commands are written in uppercase to improve readability.

Example Dockerfile for a Python Application

FROM python:3.10
WORKDIR /app
COPY requirements.txt /app
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app
CMD ["python", "app.py"]

How the Build Process Works

When you build an image from this Dockerfile, the following steps occur:

Base Image Selection: Docker searches for the specified base image (python:3.10). If it's not available locally, it fetches it from Docker Hub.
Setting Up the Working Directory: The WORKDIR /app command creates a directory inside the container where subsequent commands will execute.
Copying Dependencies File: The COPY requirements.txt /app instruction transfers the dependencies file to the container.
Installing Dependencies: The RUN pip install --no-cache-dir -r requirements.txt command installs all required Python packages.
Copying Application Files: The COPY . /app command copies all remaining application files into the container.
Defining the Default Command: The CMD instruction specifies the default command to run inside the container, starting the application with python app.py.

Common Dockerfile Instructions

Here are some key Dockerfile commands and their purposes:

FROM: Specifies the base image for the build process.
ADD / COPY: Transfers files from the host to the container. ADD can handle remote URLs and extract compressed files, but COPY is recommended for local file transfers.
WORKDIR: Defines the working directory for subsequent commands.
RUN: Executes commands during the image build process, such as installing software packages.
CMD / ENTRYPOINT: Determines the default command executed when the container starts. ENTRYPOINT is immutable, while CMD can be overridden.

Understanding Dockerfile Layers

Each command in a Dockerfile creates a new layer in the final image. These layers are stacked, and Docker efficiently caches them to speed up future builds. You can inspect the image layers using:

docker history <IMAGE_NAME>

or check the number of layers with:

docker inspect --format '{{json .RootFS.Layers}}' <IMAGE_NAME>

Leveraging Docker's Build Cache

Docker optimizes image builds using a caching mechanism. When a layer remains unchanged, Docker reuses the cached version instead of rebuilding it. However, if an instruction is modified, all subsequent layers are rebuilt. This behavior impacts how Dockerfiles should be structured to minimize unnecessary rebuilds.

For example, consider a build process where the initial build takes 1244.2 seconds, but subsequent builds (without modifications) reduce the time to 6.9 seconds due to caching.

Best Practices for Writing Dockerfiles

To enhance efficiency, follow these best practices:

1. Use a `.dockerignore` File

Similar to .gitignore, a .dockerignore file helps exclude unnecessary files from the build context, reducing image size and improving performance.

2. Minimize Image Layers

Fewer layers result in faster builds. Consolidating multiple RUN commands into a single command reduces the number of layers. Instead of:

RUN apt-get update
RUN apt-get install -y nginx
RUN apt-get clean

Use:

RUN apt-get update && \
    apt-get install -y nginx && \
    apt-get clean

This approach maintains readability while optimizing build efficiency.

3. Optimize Layer Order for Caching

Since Docker rebuilds layers sequentially, placing frequently changing instructions at the end improves cache utilization. Consider this inefficient order:

FROM python:3.10
WORKDIR /app
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt
CMD ["python", "app.py"]

Here, any code change invalidates the cache, leading to unnecessary reinstallation of dependencies. Instead, use:

FROM python:3.10
WORKDIR /app
COPY requirements.txt /app
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app
CMD ["python", "app.py"]

By copying requirements.txt first, dependencies are installed before the entire codebase is copied. This ensures that dependency installations are only re-run when requirements.txt changes.

Conclusion

In this guide, we explored Dockerfiles, their core commands, how layers affect builds, and best practices for optimizing Docker images. By structuring Dockerfiles efficiently, you can improve build speed, reduce image size, and streamline the containerization process. Happy coding!