K8s POD Auto scaling with KEDA and Azure Service Bus

In one of my projects we had a requirement to scale the pods based on the queue depth of the Topic in Azure Service Bus and the search ended in KEDA - Kubernetes Event Driven Autoscaling which is a cloud native foundation, incubation project. Now let's look how we can setup Keda in our cluster.

KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication.

Keda got two key roles within the cluster, keda-operator which scales from minimum to maximum pod counts set in the ScaledObject manifest file via Kubernetes Horizontal Pod Autoscaler and keda-operator-metrics-apiserver which gets the data for the scaling decision.

Keda Installation
Keda operator and metric api-server can be installed in its own namespace and we will deploy the actual scaledobject component in the same namespaces where the pods to be scaled are.

Keda can be installed in different ways, I chose using Helm, kubectl scripts are also available in the Keda website

Install Keda using Helm
Add Helm repo
helm repo add kedacore https://kedacore.github.io/charts

Update Helm repo
helm repo update

Install Keda Helm chart
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda

Configuring the Azure Service Bus SAS policy
For configuring KEDA to operate on the ServiceBus, it would require management permission.

You can select the connection strings from this SAS key for updating in the KEDA yaml manifest

YAML Manifests
Scaling is made by a ScaledObject, which in turn uses the TriggerAuthentication object for the Azure Service Bus authentication.

apiVersion: v1
kind: Secret
metadata:
  namespace: poc 
  name: keda-secrets
  labels:
    app: exchange-jobs
data:
  management-connectionstring: <***Your base 64 encoded connection string from Azure SAS Key section with manage permissions****>
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  namespace: poc
  name: exchange-jobs-keda-scaler
spec:
  scaleTargetRef:
    name: exchange-jobs
  pollingInterval: 1   # Optional. Default: 30 seconds - checks for the changes
  cooldownPeriod: 20   # Optional. Default: 300 seconds - shutdown - need to find a sweet spot , Its better to keep default so that we don't interrupt any running processes
  minReplicaCount: 0
  maxReplicaCount: 50  # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      topicName: mytopic
      subscriptionName: S1
      namespace: test-sb-keda-scaler   
      messageCount: "5" # need to find a sweet spot
    authenticationRef:
      name: keda-sbus-auth

---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  namespace: poc
  name: keda-sbus-auth
spec:
  secretTargetRef:
  - parameter: connection
    name: keda-secrets
    key: management-connectionstring

The above yaml can be applied to the namespace where the pods are present and it should work as expected.

Things to note:

• The cool down lesser than 300 seconds (which is the default) will only work if you set the minReplicaCount to 0. If you don't specify a minReplicaCount Keda assumes as 0.

• SB Connection string should be given as base 64 encoded.

• TriggerAuthentication object parameter is connection not connectionString

If you need a SBUS topic sender and receiver for testing console app please find it here

Some useful references.
◉ Kubernetes Event-driven Autoscaling – https://aka.ms/azfr/662/01
◉ KEDA on GitHub – https://aka.ms/azfr/662/02
◉ Azure Functions on Kubernetes with KEDA – https://aka.ms/azfr/662/03
◉ Azure Friday - Azure Serverless on Kubernetes with KEDA – https://aka.ms/azfr/662/04