DEV Community

Cover image for Kubernetes Autoscaling with KEDA: Event-Driven Scaling for Queues and Microservices - Part 1
Suave Bajaj
Suave Bajaj

Posted on

Kubernetes Autoscaling with KEDA: Event-Driven Scaling for Queues and Microservices - Part 1

#kubernetes #cloud #devops #keda

KEDA & Event-Driven Autoscaling: Solving Queue Backlogs in Kubernetes

If you've ever run Kafka or RabbitMQ consumers in Kubernetes, you know the pain: your consumers are either underutilized or overloaded, and CPU-based HPA just doesn’t cut it.

Enter KEDA — Kubernetes-based Event-Driven Autoscaling.
KEDA Github

What is KEDA?

KEDA is a lightweight autoscaler that scales your Kubernetes workloads based on external events or metrics, such as:

  • Queue length in Kafka or RabbitMQ
  • Prometheus metrics
  • Cloud events (Azure, AWS, GCP)

Unlike HPA which scales on CPU/memory, KEDA lets your pods scale dynamically based on actual workload.

Important: KEDA itself runs as a Deployment in your Kubernetes cluster.

  • It’s not a StatefulSet or a special Deployment of your app.
  • It consumes CPU/memory like any other Deployment depending on the number of ScaledObjects and triggers you define.
  • This means if you define many queues, KEDA’s own Deployment will grow in resource usage.

How KEDA Works

KEDA introduces a new Kubernetes object called a ScaledObject:

  • ScaledObject → Deployment mapping: tells KEDA which workload to scale.
  • Each ScaledObject can have one or more triggers.
  • Scaling logic:
    • OR logic across triggers (if any trigger exceeds threshold → scale).
    • Cannot independently scale subsets of pods within the same Deployment.

Example: RabbitMQ ScaledObject 

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-consumer-scaler
spec:
  scaleTargetRef:
    name: orders-consumer
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: rabbitmq
    metadata:
      queueName: orders-queue
      host: amqp://guest:guest@rabbitmq:5672/
      queueLength: "100"
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • orders-consumer Deployment will scale 1→10 pods depending on queue length.
  • Scaling happens if the queue length > 100.

KEDA & Event-Driven Autoscaling: Solving Queue Backlogs in Kubernetes

If you've ever run Kafka or RabbitMQ consumers in Kubernetes, you know the pain: your consumers are either underutilized or overloaded, and CPU-based HPA just doesn’t cut it.

Enter KEDA — Kubernetes-based Event-Driven Autoscaling.

How KEDA Determines the Number of Pods

KEDA computes desired replicas based on the queue metric and thresholds:

  1. Threshold value (queueLength)

    • The metric KEDA monitors (queue depth).

  2. Min and Max Replicas

minReplicaCount: 1
maxReplicaCount: 10
Enter fullscreen mode Exit fullscreen mode
  1. **Desired replicas calculation (simplified): 
desiredReplicas = ceil(currentQueueLength / queueLengthThreshold)
Enter fullscreen mode Exit fullscreen mode
  1. **Scaling applied via HPA

KEDA updates the Deployment replicas between minReplicaCount and maxReplicaCount

Visual Example

Imagine orders-queue has a threshold of 100 messages:

Queue Depth → Desired Replicas → Deployment Pods
------------------------------------------------
50          → 1                 → 1 pod (minReplicaCount)
120         → 2                 → 2 pods
250         → 3                 → 3 pods
1050        → 11                → 10 pods (maxReplicaCount)
Enter fullscreen mode Exit fullscreen mode

Diagram: Queue Depth → Desired Replicas → Deployment Scaling

[Queue Depth: 250]
        │
        ▼
[Threshold: 100 per pod]
        │
        ▼
[Desired Replicas: ceil(250/100) = 3]
        │
        ▼
[orders-consumer Deployment scales to 3 pods]
Enter fullscreen mode Exit fullscreen mode

Limitations of KEDA

While KEDA is powerful, it has some critical limitations:

N queues → N consumers is not supported natively

You cannot define multiple queues with different consumer Deployments in a single ScaledObject.

Scaling logic is OR across triggers; all pods scale together.

Not ideal for microservices with multiple queues and workflows

Example: if you have 10 queues with different processing logic/endpoints, KEDA alone cannot scale them independently.

You’d need multiple Deployments + multiple ScaledObjects, which becomes cumbersome.

Resource consumption

KEDA itself is a Deployment; each ScaledObject adds CPU/memory usage.

Scaling many queues can increase the resource footprint on the cluster.

Kubectl Example

After creating multiple ScaledObjects:

kubectl get scaledobjects
Enter fullscreen mode Exit fullscreen mode

Example output:

NAME                       SCALETARGETKIND      SCALETARGETNAME     MIN   MAX   TRIGGERS
orders-consumer-scaler     apps/v1.Deployment   orders-consumer      1    10    rabbitmq
payments-consumer-scaler   apps/v1.Deployment   payments-consumer    1     5    rabbitmq
kafka-consumer-scaler      apps/v1.Deployment   kafka-consumer       1    15    kafka
Enter fullscreen mode Exit fullscreen mode

Each ScaledObject targets a single Deployment.

Multiple triggers for the same Deployment scale all pods together (OR logic).

Takeaways

  • KEDA is great for simple event-driven scaling.
  • Each ScaledObject → one Deployment.
  • OR logic for multiple triggers → not suitable for microservices with multiple queues and workflows.
  • KEDA computes replicas automatically using the metric and threshold, scaling Deployment pods between min and max.
  • Resource footprint increases with more ScaledObjects and triggers.

Part 2 will show how to build a custom autoscaler that can handle multiple queues, multiple consumers, and independent scaling logic for complex microservices.

Pro Tip: Start with KEDA for simple queue scaling. For complex, multi-queue microservices, a custom HPA/KEDA solution is usually required.

Top comments (0)