DEV Community: Hamdi (KHELIL) LION

🤖 Kubernetes as Your AI Control Plane: Running Claude and Ollama Agents with kagent 🧠

Hamdi (KHELIL) LION — Fri, 03 Jul 2026 06:12:43 +0000

👋 Hey there

If you run a homelab, you already have a Kubernetes cluster sitting there humming away. So why send every AI workload off to a SaaS dashboard when you can run agents right next to your pods, with the same kubectl and GitOps flow you already trust?

That is exactly what kagent gives us. It is a CNCF sandbox project (started at Solo.io, built by folks from the Istio world) that turns AI agents into first class Kubernetes workloads. Agents, models, and tools all become custom resources you apply with kubectl.

👉 The big idea: kagent puts one clean abstraction between your agents and your models, so you can point the same agent at Claude in the cloud or at Ollama running in your own cluster, and switch by changing a single field.

In this guide we will build a small but real inference stack in a homelab cluster, wired to both.

🧱 What we will cover

✅ What the kagent inference stack actually looks like
✅ Installing kagent with Helm
✅ Wiring up Claude for cloud inference (the ModelConfig resource)
✅ Deploying Ollama inside the cluster for fully local, on prem inference
✅ Building agents that use each model with real Kubernetes tools
✅ Swapping models with a one line change
✅ Adding a human approval gate so your agent never nukes the wrong namespace

🗺️ The inference stack, layer by layer

Before we touch YAML, here is the mental model. In kagent your "inference stack" is just a few layers, each a Kubernetes resource or component:

✅ Model provider: where the tokens come from. Claude via the Anthropic API, or Ollama serving a local model.
✅ ModelConfig (CRD): tells kagent which provider, which model, and which secret holds the key.
✅ Agent (CRD): a system prompt plus a set of tools plus a reference to a ModelConfig.
✅ Tools over MCP: kagent ships a tool server that exposes Kubernetes actions as tools your agent can call.
✅ Controller and runtime: the Go controller watches the CRDs, and the runtime (Agent Development Kit) runs the agent loop.
✅ Storage: a bundled PostgreSQL instance for agent state (swap it for your own in production).

The nice part is that the Agent resource does not care where the model lives. It just references a ModelConfig by name. Cloud or local, the agent spec looks the same.

🧰 Prerequisites

Nothing exotic, just homelab basics:

✅ A running Kubernetes cluster (k3s, kind, minikube, or your bare metal cluster all work)
✅ kubectl and helm installed and pointing at that cluster
✅ An Anthropic API key for the Claude part (grab one from the Anthropic console)
✅ Enough CPU and RAM on a node for a local model (an 8B model like llama3.1 is happy with a couple of cores and around 8Gi of memory)

Heads up on versions: everything below was verified against kagent v0.9 (CRD apiVersion kagent.dev/v1alpha2), current Claude model ids claude-haiku-4-5 and claude-sonnet-4-5, and the Ollama llama3.1 model, all checked in July 2026. kagent moves fast, so if a field looks different, check the release notes.

📦 Step 1: Install kagent

kagent installs as two Helm charts: one for the CRDs, one for the controller and friends. Since v0.7, the kmcp tool subproject comes bundled, so you get the built in Kubernetes tools out of the box.

First, install the CRDs:

helm install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
  --namespace kagent \
  --create-namespace

Now install kagent itself. We will set Anthropic as the default provider so we get a working Claude backed model config the moment the pods come up:

export ANTHROPIC_API_KEY="your-anthropic-key-here"

helm install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --namespace kagent \
  --set providers.default=anthropic \
  --set providers.anthropic.apiKey=$ANTHROPIC_API_KEY

Give it a minute, then check the pods:

kubectl get pods -n kagent

You should see the controller, the UI, the kagent tool server, and a bundled PostgreSQL pod.

Want the dashboard? Port forward the UI:

kubectl port-forward -n kagent svc/kagent-ui 8080:8080

Then open http://localhost:8080.

Homelab tip: the default install ships a bundled postgres:18 pod so you can play right away. For anything you actually care about, point kagent at your own PostgreSQL instead.

☁️ Step 2: Wire up Claude for cloud inference

Because we passed the Anthropic provider at install time, kagent already created a default ModelConfig for us. In v0.9 that default uses claude-haiku-4-5, which is fast and cheap and perfect for quick cluster questions.

Let us look at what landed:

kubectl get modelconfigs -n kagent

Now let us be explicit and add our own ModelConfig that pins a stronger model, claude-sonnet-4-5, for the heavier reasoning tasks. First store the key in a Secret (kagent reads the model key from a Secret you control):

export ANTHROPIC_API_KEY="your-anthropic-key-here"

kubectl create secret generic kagent-anthropic -n kagent \
  --from-literal ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY

Then create the ModelConfig that references it:

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: claude-sonnet
  namespace: kagent
spec:
  provider: Anthropic
  model: claude-sonnet-4-5
  apiKeySecret: kagent-anthropic
  apiKeySecretKey: ANTHROPIC_API_KEY
  anthropic: {}

Apply it:

kubectl apply -f claude-modelconfig.yaml

That is your cloud inference layer done. Two model configs now live in the cluster: a fast Haiku default and a Sonnet config for the deeper work.

Note on model ids: claude-haiku-4-5 and claude-sonnet-4-5 are convenience aliases that resolve to the latest dated snapshot. If you want a fully pinned version for reproducibility, use the dated form such as claude-haiku-4-5-20251001.

🏠 Step 3: Deploy Ollama for on prem inference

Now the fun part for homelabbers: running the model yourself, no tokens leaving your network.

We will run Ollama as a normal Deployment, give it a PersistentVolumeClaim so the downloaded model survives restarts (models are big, you do not want to re download on every reschedule), and expose it with a Service.

Create the namespace:

kubectl create ns ollama

Apply the Ollama stack:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-models
  namespace: ollama
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      name: ollama
  template:
    metadata:
      labels:
        name: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
            - name: http
              containerPort: 11434
              protocol: TCP
          resources:
            requests:
              cpu: "2"
              memory: 8Gi
            limits:
              memory: 12Gi
          volumeMounts:
            - name: models
              mountPath: /root/.ollama
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ollama
spec:
  type: ClusterIP
  selector:
    name: ollama
  ports:
    - name: http
      port: 80
      targetPort: http
      protocol: TCP

Wait for the pod to be ready:

kubectl get pods -n ollama -w

Once it is running, pull a model. This matters: kagent drives agents through tool calling, so you must use a model that supports function calling. Plain llama3 does not do this reliably, but llama3.1 and newer (or qwen2.5 and above) do. We will use llama3.1:

kubectl -n ollama exec deploy/ollama -- ollama pull llama3.1

Confirm it landed:

kubectl -n ollama exec deploy/ollama -- ollama list

Your local model is now reachable inside the cluster at:

http://ollama.ollama.svc.cluster.local

(The Service listens on port 80 and forwards to the container on 11434, so no port number is needed in the URL.)

🔌 Step 4: Tell kagent about your local model

Same ModelConfig pattern as Claude, just a different provider and a host that points at the in cluster Ollama Service.

One small quirk: Ollama does not need an API key, but the ModelConfig field still expects a secret reference. So we create a throwaway placeholder secret (Ollama ignores the value):

kubectl create secret generic kagent-ollama -n kagent \
  --from-literal OLLAMA_API_KEY=ollama-placeholder

Now the ModelConfig:

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: ollama-llama31
  namespace: kagent
spec:
  provider: Ollama
  model: llama3.1
  apiKeySecret: kagent-ollama
  apiKeySecretKey: OLLAMA_API_KEY
  ollama:
    host: http://ollama.ollama.svc.cluster.local

Apply it:

kubectl apply -f ollama-modelconfig.yaml

Check that both cloud and local configs are now registered:

kubectl get modelconfigs -n kagent

You now have a hybrid inference stack: Claude for the heavy lifting, Ollama for private, offline friendly work, both described the same declarative way.

🧠 Step 5: Build an agent that uses your models

Here is where the abstraction pays off. An Agent is just a system prompt, a set of tools, and a modelConfig reference. kagent ships a built in tool server called kagent-tool-server that exposes Kubernetes actions as tools, so our agent can actually inspect the cluster.

Let us create a Claude backed cluster helper:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: cluster-helper-claude
  namespace: kagent
spec:
  description: A friendly Kubernetes helper backed by Claude.
  type: Declarative
  declarative:
    modelConfig: claude-sonnet
    systemMessage: |
      You are a friendly and careful Kubernetes helper.

      # Instructions
      - Use the available tools to answer questions about the cluster.
      - If a question is unclear, ask for clarification before running any tool.
      - Never invent an answer. If you are unsure, say so.

      # Response format
      - Always format your response as Markdown.
      - Summarize the actions you took and explain the result.
    tools:
      - type: McpServer
        mcpServer:
          name: kagent-tool-server
          kind: RemoteMCPServer
          apiGroup: kagent.dev
          toolNames:
            - k8s_get_resources
            - k8s_get_available_api_resources
            - k8s_describe_resource

Apply it:

kubectl apply -f agent-claude.yaml

👉 Now the magic trick. To run the exact same agent on your local model instead of Claude, you change one field: modelConfig. Here is the on prem twin:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: cluster-helper-local
  namespace: kagent
spec:
  description: The same helper, running fully local on Ollama.
  type: Declarative
  declarative:
    modelConfig: ollama-llama31
    systemMessage: |
      You are a friendly and careful Kubernetes helper.

      # Instructions
      - Use the available tools to answer questions about the cluster.
      - If a question is unclear, ask for clarification before running any tool.
      - Never invent an answer. If you are unsure, say so.

      # Response format
      - Always format your response as Markdown.
      - Summarize the actions you took and explain the result.
    tools:
      - type: McpServer
        mcpServer:
          name: kagent-tool-server
          kind: RemoteMCPServer
          apiGroup: kagent.dev
          toolNames:
            - k8s_get_resources
            - k8s_get_available_api_resources
            - k8s_describe_resource

kubectl apply -f agent-local.yaml

Same prompt, same tools, different brain. That is the whole point of putting a ModelConfig in front of your agents.

💬 Step 6: Talk to your agents

You can chat through the dashboard, but the CLI is quicker for a test. Grab the kagent CLI:

brew install kagent

curl https://raw.githubusercontent.com/kagent-dev/kagent/refs/heads/main/scripts/get-kagent | bash

List your agents:

kagent get agent

Ask the Claude agent something:

kagent invoke -t "What API resources are available in my cluster?" --agent cluster-helper-claude

Then ask the local one the same thing and compare:

kagent invoke -t "List the pods in the kagent namespace and tell me if any are unhealthy." --agent cluster-helper-local

The local agent keeps every token inside your homelab, which is a lovely thing when you are poking at private workloads.

🛡️ Step 7: Add a safety gate (highly recommended)

Giving an AI agent access to your cluster is great until it decides to be too helpful. kagent has a built in Human in the Loop feature: mark any tool as needing approval, and the agent pauses for your yes or no before running it.

Here is an agent that can read freely but must ask before it changes anything:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: cluster-operator
  namespace: kagent
spec:
  description: A Kubernetes operator agent with approval gates on destructive actions.
  type: Declarative
  declarative:
    modelConfig: claude-sonnet
    systemMessage: |
      You are a Kubernetes operator assistant.
      Before making any change, explain what you plan to do and why.
    tools:
      - type: McpServer
        mcpServer:
          name: kagent-tool-server
          kind: RemoteMCPServer
          apiGroup: kagent.dev
          toolNames:
            - k8s_get_resources
            - k8s_describe_resource
            - k8s_apply_manifest
            - k8s_delete_resource
          requireApproval:
            - k8s_apply_manifest
            - k8s_delete_resource

kubectl apply -f agent-operator.yaml

Now the read tools run freely, but the moment the agent tries to apply or delete something, the UI shows Approve and Reject buttons. If you reject, your reason is fed back to the model as context. Two lines of YAML, a lot of peace of mind.

Good to know: every tool you list in requireApproval must also appear in toolNames, and kagent validates this when the resource is created.

🔁 Why this pattern is nice for a homelab

✅ One workflow: agents, models, and tools are all just kubectl apply, so GitOps and PR reviews work exactly like the rest of your cluster.
✅ No lock in: swap Claude for Ollama, or route different agents to different models, without rewriting anything.
✅ Cost control: send routine questions to a local model, save the cloud model for the hard stuff.
✅ Privacy by default: keep sensitive workloads on the on prem model, no data leaves the house.
✅ Guardrails built in: approval gates and clear observability instead of a black box.

🚀 What is next

A few directions to keep exploring from here:

✅ Point kagent at an external PostgreSQL and enable memory so your agents remember context across sessions.
✅ Try the Go runtime for faster agent cold starts by setting runtime: go in the declarative spec.
✅ Add your own MCP tools with kmcp so agents can talk to your internal APIs.
✅ Put a GPU node under Ollama and jump to a bigger local model for better tool use.

🦆 Bonus: drive it all from Goose

Once your stack is up, you do not have to live inside the kagent UI. Goose is a local, open source AI agent (CLI and desktop, now part of the Linux Foundation Agentic AI Foundation) that speaks MCP. That means it can plug straight into what you just built, in two complementary ways.

Think of it like this: Goose needs a brain (a model provider) and it can borrow tools (MCP extensions). Your cluster can supply both.

🔑 First, give Goose a brain (your deployed models)

Install Goose from the official install page (https://goose-docs.ai/docs/getting-started/installation), then run the config wizard:

goose configure

Point it at Claude, using the same Anthropic key you gave kagent. Goose will ask for your API key and store it in your system keyring:

┌   goose-configure
│
◇  What would you like to configure?
│  Configure Providers
│
◇  Which model provider should we use?
│  Anthropic
│
◇  Enter a model from that provider:
│  claude-sonnet-4-5
└  Configuration saved successfully

Prefer to stay fully local and reuse the Ollama you deployed in Step 3? Goose runs on your workstation, so first port forward the in cluster Ollama Service (it listens on port 80 and targets the container on 11434):

kubectl -n ollama port-forward svc/ollama 11434:80

Then configure Goose to use Ollama. If you do not set a host, Goose defaults to localhost:11434, which is exactly where the port forward lands:

┌   goose-configure
│
◇  What would you like to configure?
│  Configure Providers
│
◇  Which model provider should we use?
│  Ollama
│
◇  Enter a model from that provider:
│  llama3.1
└  Configuration saved successfully

👉 One thing to keep in mind: Goose leans heavily on tool calling, so pick a model that is good at it. llama3.1 and up or qwen2.5 handle light tasks, but for serious agent loops Claude will be a lot more reliable.

🔗 Then, hand Goose your kagent agents as tools

kagent exposes every running agent through an MCP server built into the control plane, over the Streamable HTTP transport. So Goose can discover and call your kagent agents as if they were tools.

Port forward the kagent control plane:

kubectl port-forward -n kagent svc/kagent-controller 8083:8083

Add it to Goose as a remote extension:

goose configure

┌   goose-configure
│
◇  What would you like to configure?
│  Add Extension
│
◇  What type of extension would you like to add?
│  Remote Extension (Streamable HTTP)
│
◇  What would you like to call this extension?
│  kagent-agents
│
◇  What is the Streamable HTTP endpoint URI?
│  http://localhost:8083/mcp
│
◇  Please set the timeout for this tool (in secs):
│  300
└  Added kagent-agents extension

That writes an entry to ~/.config/goose/config.yaml that looks like this:

extensions:
  kagent-agents:
    enabled: true
    type: streamable_http
    name: kagent-agents
    uri: http://localhost:8083/mcp
    timeout: 300

The kagent MCP server hands Goose two tools: list_agents to discover what is available, and invoke_agent to run a specific agent by name (with session support for follow ups). Start a session:

goose session

and ask something like:

List my kagent agents, then use the cluster-helper-local agent to check for unhealthy pods in the kagent namespace.

Goose will call list_agents, pick cluster-helper-local, and delegate the cluster work to it through kagent. You end up with Goose as the friendly front door, your kagent agents as specialized workers, and Claude or Ollama as the brain, all wired together through open protocols.

Note: this kagent endpoint currently supports Streamable HTTP, not SSE. If you put kagent behind a gateway you can skip the port forward and point Goose at the public URL instead.

🎁 Wrapping Up

We went from an empty homelab cluster to a working, hybrid AI inference stack: kagent as the control plane, Claude for cloud inference, and Ollama running fully local, all described as plain Kubernetes resources. The best part is how boring it feels in the best way. Agents are just workloads, models are just config, and swapping between cloud and on prem is a one line change.

If you have been waiting for a reason to bring agentic AI into your cluster without handing everything to a SaaS, this is a really pleasant place to start.

Happy clustering and stay safe! 🧑‍🚀

🚦 Meet Kueue: Smart Job Queueing for Kubernetes 🧠⚙️

Hamdi (KHELIL) LION — Tue, 30 Jun 2026 06:32:30 +0000

Hey everyone 👋

If you run batch jobs, data pipelines, or any kind of AI and ML training on Kubernetes, you have probably hit this wall. Kubernetes is fantastic at deciding WHERE a pod should run, but it is surprisingly clueless about WHEN a job should start. 😅

You submit ten jobs, the cluster fills up, and the rest just sit there as Pending. No real queue, no priority, no fairness between teams. One noisy team can eat all your expensive nodes while everyone else waits. 🥲

That is exactly the gap Kueue fills, and today I want to walk you through it with a pile of hands on examples you can run on any cluster, even your homelab. 🏡

👉 Key takeaway up front: Kueue is a job level manager that holds your jobs in a real queue and only admits them when there is enough quota to actually run them.

🧪 Everything in this guide was tested against Kueue v0.18.1 using the v1beta2 API. I pinned every command and manifest to that version so you do not get surprised by API drift.

📋 What we will cover

✅ Why Kubernetes needs a queue
✅ The building blocks in plain language
✅ Installing Kueue
✅ Setting up quota with a ResourceFlavor, a ClusterQueue, and a LocalQueue
✅ Submitting a Job and watching it get queued and admitted
✅ Priority based admission
✅ Partial admission and elastic jobs
✅ Multiple resource flavors for x86 and arm
✅ Fair sharing between teams with cohorts
✅ Dedicated quota with a shared fallback
✅ Queueing a plain Pod
✅ Why this matters a lot for GPUs and your cloud bill

🤔 Why Kubernetes needs a queue

Native Kubernetes scheduling is pod centric. The scheduler looks at one pod at a time and tries to place it. That works great for long running services.

Batch workloads are different. They have a beginning and an end, they often need a fixed chunk of capacity, and they compete with other teams for the same nodes.

Without a queueing layer you get:

✅ Jobs that fail or stay Pending when resources are tight
✅ No quota governance, so one team can starve the others
✅ No admission priority, so a quick experiment can block production training

🧠 What is Kueue

Kueue is a Kubernetes native job queueing system, maintained as a kubernetes-sigs project. It does not replace the scheduler. It sits in front of it. 🛂

Here is the simple mental model. Think of the Kubernetes scheduler as the runway, and Kueue as the control tower deciding which flight is cleared for takeoff and when. ✈️

When a job arrives, Kueue suspends it, creates a matching Workload object, checks if there is enough quota, and only then lets the pods be created. If there is no room, the job waits politely in the queue instead of failing.

🧩 The building blocks

There are four pieces you need to know, plus one bonus piece for teams.

✅ ResourceFlavor 🍦
Describes a type of resource, usually tied to node labels. For example x86 nodes versus arm nodes, or GPU nodes versus CPU nodes. If you do not need to distinguish node types, you use one empty flavor.

✅ ClusterQueue 🏦
A cluster scoped object that holds the actual quota. This is where you say how much cpu, memory, or how many GPUs are available. Users do not submit to it directly.

✅ LocalQueue 📥
A namespaced object that points to a ClusterQueue. This is what users actually target with their jobs.

✅ Workload 📦
The internal object Kueue creates for each job to track its admission state. You usually just observe it.

✅ Cohort 👥 (bonus)
A group of ClusterQueues that can borrow each other unused quota. This is the magic behind fair sharing between teams.

🛠️ Step 1: Install Kueue

The simplest method is to apply the released manifests with server side apply.

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.1/manifests.yaml

The controller runs in the kueue-system namespace. Give it a few seconds and check it is healthy.

kubectl get deploy -n kueue-system

You should see the controller manager become ready.

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
kueue-controller-manager   1/1     1            1           30s

Prefer Helm? Kueue publishes an OCI chart for each release. Just make sure the chart version matches the release you want.

helm install kueue oci://registry.k8s.io/kueue/charts/kueue \
  --version=0.18.1 \
  --namespace kueue-system \
  --create-namespace \
  --wait --timeout 300s

🍦 Step 2: Create a ResourceFlavor

Since we are not distinguishing node types in this first demo, an empty flavor is all we need.

# default-flavor.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: "default-flavor"

Apply it.

kubectl apply -f default-flavor.yaml

🏦 Step 3: Create a ClusterQueue

Now we define the quota for the whole cluster. Here we allow 9 cpu and 36Gi of memory, all served by our single flavor.

# cluster-queue.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {} # match all namespaces
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 9
      - name: "memory"
        nominalQuota: 36Gi

Apply it.

kubectl apply -f cluster-queue.yaml

One important detail. The flavor name under spec.resourceGroups must match the ResourceFlavor name from step 2. If they do not match, the ClusterQueue will not become ready. 🔗

📥 Step 4: Create a LocalQueue

Users cannot send work to a ClusterQueue directly. They need a LocalQueue in their namespace that points to it. We will put ours in the default namespace.

# default-user-queue.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "default"
  name: "user-queue"
spec:
  clusterQueue: "cluster-queue"

Apply it.

kubectl apply -f default-user-queue.yaml

Quick tip: you can apply all three of the above at once using the example bundle from the project.

kubectl apply -f https://kueue.sigs.k8s.io/examples/admin/single-clusterqueue-setup.yaml

🚀 Step 5: Submit your first Job

This is the only change your users need to make to an existing Job. Add the kueue.x-k8s.io/queue-name label pointing to the LocalQueue, and make sure each pod declares resource requests.

# sample-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  generateName: sample-job-
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  parallelism: 3
  completions: 3
  template:
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        command: [ "/bin/sh" ]
        args: [ "-c", "sleep 60" ]
        resources:
          requests:
            cpu: "1"
            memory: "200Mi"
      restartPolicy: Never

Notice that you do not need to set the job to suspended yourself. Kueue manages suspension for you through a webhook and decides the best moment to start it. 🪄

Create the job.

kubectl create -f sample-job.yaml

🔭 Step 6: Watch the queue work

List your local queues. The alias queues also works.

kubectl -n default get localqueues

NAME         CLUSTERQUEUE    PENDING WORKLOADS
user-queue   cluster-queue   0

Kueue creates a Workload object for your job. Have a look.

kubectl -n default get workloads.kueue.x-k8s.io

NAME               QUEUE         RESERVED IN     ADMITTED   AGE
sample-job-xxxxx   user-queue    cluster-queue   True       3s

Want the full story? Describe the workload. When there is not enough quota, you will see it sit unadmitted with a clear message.

kubectl -n default describe workload sample-job-xxxxx

Status:
  Conditions:
    Message:  workload didn't fit
    Reason:   Pending
    Status:   False
    Type:     Admitted

The moment quota frees up, Kueue admits it automatically. If you describe the Job itself, the event timeline tells the whole story.

Events:
  Type    Reason            From                  Message
  ----    ------            ----                  -------
  Normal  Suspended         job-controller        Job suspended
  Normal  CreatedWorkload   kueue-job-controller  Created Workload: default/sample-job-xxxxx
  Normal  Started           kueue-job-controller  Admitted by clusterQueue cluster-queue
  Normal  Resumed           job-controller        Job resumed
  Normal  Completed         job-controller        Job completed

No babysitting required. 🎉

🥇 Example: priority based admission

Inside a queue, not all jobs are equal. With a WorkloadPriorityClass you can control admission and preemption priority independently from pod priority. Production training jumps the line ahead of throwaway experiments. 🏎️

First create the priority class.

# sample-priority.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: sample-priority
value: 10000
description: "Sample priority"

Then point a Job at it with the kueue.x-k8s.io/priority-class label.

# priority-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: sample-job
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    kueue.x-k8s.io/priority-class: sample-priority
spec:
  parallelism: 3
  completions: 3
  suspend: true
  template:
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:latest
        args: ["pause"]
      restartPolicy: Never

Higher value means higher priority for queuing and preemption. The neat part is this priority does not touch the pod priority, so it does not interfere with your normal Kubernetes scheduling. 👌

✂️ Example: partial admission

Sometimes a big job can still make progress with fewer pods. With the kueue.x-k8s.io/job-min-parallelism annotation, Kueue can admit the job at a reduced parallelism instead of leaving it Pending.

# partial-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: sample-job-partial-admission
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
  annotations:
    kueue.x-k8s.io/job-min-parallelism: "5"
spec:
  parallelism: 20
  completions: 20
  template:
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        args: ["entrypoint-tester", "hello", "world"]
        resources:
          requests:
            cpu: 1
            memory: "200Mi"
      restartPolicy: Never

If only 9 cpu is free, this job is admitted with parallelism 9 instead of waiting for all 20. The completions count stays the same. 🙌

📈 Example: elastic jobs

Elastic jobs let you change a running Job parallelism without recreating, restarting, or suspending it. This is an alpha feature, so you must enable the ElasticJobsViaWorkloadSlices feature gate and annotate the Job.

# elastic-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: sample-elastic-job
  namespace: default
  annotations:
    kueue.x-k8s.io/elastic-job: "true"
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  parallelism: 3
  completions: 100
  template:
    spec:
      containers:
      - name: dummy-job
        image: registry.k8s.io/e2e-test-images/agnhost:2.53
        command: [ "/bin/sh" ]
        args: [ "-c", "sleep 60" ]
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
      restartPolicy: Never

When you bump parallelism up, Kueue creates a new admitted Workload for the new pod count and marks the old one as Finished. When you scale down, the extra pods terminate and no new Workload is created. Smooth. 🧘

🧱 Example: multiple resource flavors

Real clusters often mix node types. Say you have x86 and arm nodes labelled with cpu-arch. You can create one flavor per architecture.

# flavor-x86.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: "x86"
spec:
  nodeLabels:
    cpu-arch: x86

# flavor-arm.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: "arm"
spec:
  nodeLabels:
    cpu-arch: arm

Then reference both in a single ClusterQueue. Here cpu is split across the two architectures, while memory uses the simple default flavor because we do not care which architecture provides it.

# cluster-queue-multi.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {} # match all
  resourceGroups:
  - coveredResources: ["cpu"]
    flavors:
    - name: "x86"
      resources:
      - name: "cpu"
        nominalQuota: 9
    - name: "arm"
      resources:
      - name: "cpu"
        nominalQuota: 12
  - coveredResources: ["memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "memory"
        nominalQuota: 84Gi

The labels in the ResourceFlavor must match the labels on your nodes. If you use the cluster autoscaler, make sure it adds those labels to new nodes too. 🏷️

👥 Example: fair sharing between teams

This is where Kueue really shines. Put two ClusterQueues in the same cohort and they can borrow each other unused quota.

# team-a-cq.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "team-a-cq"
spec:
  namespaceSelector: {}
  cohortName: "team-ab"
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 9
        borrowingLimit: 6
      - name: "memory"
        nominalQuota: 36Gi
        borrowingLimit: 24Gi

# team-b-cq.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "team-b-cq"
spec:
  namespaceSelector: {}
  cohortName: "team-ab"
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 12
      - name: "memory"
        nominalQuota: 48Gi

Both queues belong to the cohort team-ab. Team A has its own guaranteed quota, but it can also borrow idle capacity from Team B, up to the borrowingLimit of 6 cpu and 24Gi. When Team B needs its capacity back, Kueue handles it. ⚖️

🎯 Example: dedicated quota with a shared fallback

A ClusterQueue can borrow from the cohort even when it has zero nominal quota for a flavor. This lets you give each team dedicated capacity on one flavor, plus a shared pool to fall back on.

# team-a-cq.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "team-a-cq"
spec:
  namespaceSelector: {} # match all
  cohortName: "team-ab"
  resourceGroups:
  - coveredResources: ["cpu"]
    flavors:
    - name: "arm"
      resources:
      - name: "cpu"
        nominalQuota: 9
        borrowingLimit: 0
    - name: "x86"
      resources:
      - name: "cpu"
        nominalQuota: 0
  - coveredResources: ["memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "memory"
        nominalQuota: 36Gi

# shared-cq.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "shared-cq"
spec:
  namespaceSelector: {} # match all
  cohortName: "team-ab"
  resourceGroups:
  - coveredResources: ["cpu"]
    flavors:
    - name: "x86"
      resources:
      - name: "cpu"
        nominalQuota: 6
  - coveredResources: ["memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "memory"
        nominalQuota: 24Gi

Read it like this:

✅ team-a-cq has a borrowingLimit of 0 on the arm flavor, so its arm capacity is truly dedicated and cannot be borrowed away.
✅ team-a-cq has a nominalQuota of 0 on the x86 flavor, so it has no x86 of its own and can only borrow x86 from shared-cq.

This pattern is great for giving each team a guaranteed slice while still pooling the expensive shared hardware. 🤝

🫙 Example: queue a plain Pod

You are not limited to Jobs. Kueue can manage plain Pods too. Just add the queue-name label and resource requests.

# kueue-sleep-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  generateName: kueue-sleep-
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  containers:
  - name: sleep
    image: busybox
    command:
    - sleep
    args:
    - 3s
    resources:
      requests:
        cpu: 3
  restartPolicy: OnFailure

Kueue injects a kueue.x-k8s.io/managed=true label to mark the pods it manages. The same label driven approach works for Deployments, StatefulSets, RayJobs, JobSets, and Kubeflow jobs as well. 🧰

🧭 Bonus: the kueue kubectl plugin

Kueue ships a kubectl plugin so you can manage queues without writing kubectl get workloads.kueue.x-k8s.io every time. Once installed, you get handy commands.

# List workloads in a namespace
kubectl kueue list workload

# Stop a workload (it stays in the queue but will not be admitted)
kubectl kueue stop workload sample-job-xxxxx

# Resume it later
kubectl kueue resume workload sample-job-xxxxx

It also covers create, delete, describe, edit, get, and patch for clusterqueues, localqueues, resourceflavors, and workloads. A nice quality of life upgrade for operators. 🧑‍🔧

💰 Why this is a big deal for GPUs and FinOps

Here is the part that makes finance happy. 🤑

GPU and accelerator nodes are expensive, and they are often the scarcest resource in the cluster. The worst outcome is a job that partially grabs a few GPUs, then waits forever for the rest while those GPUs sit idle and billed.

With Kueue you get:

✅ Quota governance so no single team hoards the accelerators
✅ Admission only when the capacity a job needs is available
✅ Priority so production training is admitted before throwaway experiments
✅ Borrowing so idle quota is actually used instead of wasted

That combination is exactly why Kueue is becoming a key building block for running AI and ML workloads on Kubernetes at scale. 🚀

⚠️ A few gotchas to save you time

✅ Always set resource requests on your pods. If you only set limits, Kueue treats the limits as requests. If you set neither, quota accounting cannot work.
✅ The queue-name label must point to a LocalQueue that exists in the same namespace as the job.
✅ The flavor names in the ClusterQueue must match your ResourceFlavor names exactly.
✅ Elastic jobs are alpha, so remember to enable the ElasticJobsViaWorkloadSlices feature gate.
✅ Stick to one API version. This guide uses v1beta2, which is the current served version in v0.18.1.

🏁 Wrapping Up

Kueue takes Kubernetes from I will place pods wherever and whenever to I will admit jobs in a fair, prioritized, quota aware order. For batch, data, and AI workloads that is a huge upgrade, and it costs you almost nothing to adopt since your jobs only need one extra label. 🙌

To recap the flow:

✅ Install Kueue
✅ Create a ResourceFlavor
✅ Create a ClusterQueue with your quota
✅ Create a LocalQueue per namespace
✅ Add the queue-name label to your jobs
✅ Layer on priority, partial admission, elastic scaling, and cohorts as you grow

Give it a spin on a small cluster first, watch the Workload objects, and you will quickly get a feel for how admission works.

What is next? I am going to bring an AI agent into my own homelab cluster and show the full setup, so stay tuned for that one. 🤖🏡

Happy queueing and stay safe! 👋

🧹 Keeping Your Kubernetes Cluster Clean with the Descheduler

Hamdi (KHELIL) LION — Fri, 26 Jun 2026 08:51:59 +0000

Your scheduler did a great job placing pods this morning. But your cluster never stops moving, and by this afternoon those decisions are already a little bit wrong. 😅

The kube-scheduler only decides things once, at pod creation time. After that it walks away. The descheduler is the friend that comes back later, looks at the mess, and gently tidies up.

Let me walk you through it. 🚀

🗺️ what we will cover

🤔 why a perfectly scheduled cluster slowly drifts
🧹 what the descheduler does (and what it does not do)
🧩 the building blocks: profiles, plugins and extension points
🔌 the strategy plugins you will actually reach for
🛡️ the Default Evictor, your safety net
🚀 installing it with Helm (full values file)
⏱️ CronJob vs Deployment mode
🧪 testing safely with dry run
🧯 a production safety checklist
🧠 the gotcha that bites everyone

🤔 why your cluster drifts

The scheduler makes a one time decision based on the cluster as it looked the moment a pod appeared. Real clusters keep changing under that decision:

🆕 New nodes join (autoscaler or manual) and sit half empty while old nodes stay packed.
🔧 You drain and uncordon a node for maintenance, and nothing moves back onto it.
🏷️ Labels and taints change, so pods that matched the old rules now violate them.
💥 Nodes fail, pods pile up elsewhere, and the spread never recovers on its own.

The result is drift: hotspots, lopsided utilization, and affinity rules that are quietly broken.

👉 The descheduler exists to correct this drift, on a schedule, without you babysitting it.

🧹 what it actually does

Here is the part people get wrong, so let me be blunt about it.

The descheduler does not schedule pods. It only evicts them. 🙅

It finds pods that are poorly placed, evicts them through the standard Kubernetes Eviction API, and then trusts the normal kube-scheduler to recreate them in a better spot.

That has two big consequences:

✅ It plays nicely with the scheduler you already have, no replacement needed.
✅ It respects PodDisruptionBudgets, because it uses the same eviction path everything else does. If an eviction would break a PDB, the request is rejected and that pod is skipped.

So the mental model is simple: the descheduler makes room, the scheduler fills it. 🔁

🧩 the building blocks

Modern descheduler config uses the descheduler/v1alpha2 API. Three concepts matter.

1. Profiles 📋
A profile is a named bundle of plugins and their config. You can run more than one.

2. Plugins 🔌
Each strategy is a plugin (for example LowNodeUtilization). You configure it under pluginConfig and then switch it on under plugins.

3. Extension points 🪝
This is where a plugin runs. The two you care about for strategies are:

deschedule: looks at pods one by one and evicts the bad ones.
balance: looks across nodes and evicts to even things out.

There are also filter and preEvictionFilter points, which the Default Evictor uses to decide what is safe to touch.

Here is the shape of a policy so the pieces click:

apiVersion: "descheduler/v1alpha2"
kind: "DeschedulerPolicy"
profiles:
  - name: default
    pluginConfig:
      - name: "LowNodeUtilization"
        args:
          thresholds:
            cpu: 20
            memory: 20
            pods: 20
          targetThresholds:
            cpu: 50
            memory: 50
            pods: 50
    plugins:
      balance:
        enabled:
          - "LowNodeUtilization"

🔌 the strategies you will actually use

There are many plugins, but a handful do most of the work. Let me group them by extension point.

⚖️ balance plugins

These rebalance placement across the cluster.

🔥 LowNodeUtilization: finds underused nodes and evicts pods from overused ones, hoping they reschedule onto the quiet nodes. Note: utilization here is based on pod requests vs allocatable, not live metrics, unless you wire up a metrics provider.
🌶️ HighNodeUtilization: the opposite idea, pack pods off underused nodes so they can be scaled down. Great with a cluster autoscaler that scales in.
👯 RemoveDuplicates: stops multiple pods of the same workload from piling onto one node, which is exactly what you want for high availability.
🗺️ RemovePodsViolatingTopologySpreadConstraint: re-balances pods so they respect your topology spread constraints again.

🧨 deschedule plugins

These walk pods and evict the ones that no longer belong.

🚫 RemovePodsViolatingNodeAffinity: evicts pods stuck on nodes that no longer match their node affinity.
🧪 RemovePodsViolatingNodeTaints: evicts pods that no longer tolerate a node's taints.
🧲 RemovePodsViolatingInterPodAntiAffinity: cleans up pods that now break their anti affinity rules.
♻️ RemovePodsHavingTooManyRestarts: evicts crash looping pods past a restart threshold so they can try a healthier node.
⏳ PodLifeTime: evicts pods older than a max lifetime, handy for forcing periodic recycling.
🧟 RemoveFailedPods: clears out pods stuck in a failed state.

👉 Start with RemoveDuplicates and LowNodeUtilization. They give the most value with the least surprise.

🛡️ the Default Evictor, your safety net

Before any strategy evicts a pod, the Default Evictor decides whether that pod is even allowed to be touched. This is your most important safety layer, so do not skip it.

The args you will reach for:

🧷 nodeFit: true: only evict a pod if there is actually another node it could land on. This single setting prevents most pointless eviction loops.
🔢 minReplicas: never evict if fewer than this many replicas exist, so you do not knock out a lonely pod.
👑 evictSystemCriticalPods: false: leave system critical priority pods alone (this is the default).
💾 evictLocalStoragePods: false: do not evict pods using local storage, unless you really mean to.
🐝 evictDaemonSetPods: false: leave DaemonSet pods in place (this is the default).

apiVersion: "descheduler/v1alpha2"
kind: "DeschedulerPolicy"
profiles:
  - name: default
    pluginConfig:
      - name: "DefaultEvictor"
        args:
          nodeFit: true
          minReplicas: 2
          evictSystemCriticalPods: false
          evictLocalStoragePods: false
          evictDaemonSetPods: false

Newer releases also offer a podProtections block (with defaultDisabled and extraEnabled lists, plus extras like PodsWithPVC and PodsWithoutPDB). The classic args above still work and are easier to read when you are getting started.

🚀 installing it with Helm

The official Helm chart is the cleanest path. These commands are for the chart published by the project.

helm repo add descheduler https://kubernetes-sigs.github.io/descheduler/
helm repo update
helm install descheduler descheduler/descheduler --namespace kube-system

Now here is a complete, opinionated values.yaml you can actually start from. It runs as a CronJob, enables the safe high value strategies, and keeps the Default Evictor strict.

# values.yaml
kind: CronJob
schedule: "*/15 * * * *"   # run every 15 minutes, tune to taste

# pin the image to a known version
image:
  repository: registry.k8s.io/descheduler/descheduler
  tag: v0.36.0

deschedulerPolicyAPIVersion: "descheduler/v1alpha2"

deschedulerPolicy:
  # cluster wide eviction guard rails
  maxNoOfPodsToEvictPerNode: 5
  maxNoOfPodsToEvictPerNamespace: 10
  maxNoOfPodsToEvictTotal: 50

  profiles:
    - name: default
      pluginConfig:
        - name: "DefaultEvictor"
          args:
            nodeFit: true
            minReplicas: 2
            evictSystemCriticalPods: false
            evictLocalStoragePods: false
            evictDaemonSetPods: false

        - name: "RemoveDuplicates"
          args:
            excludeOwnerKinds:
              - "DaemonSet"

        - name: "LowNodeUtilization"
          args:
            thresholds:
              cpu: 20
              memory: 20
              pods: 20
            targetThresholds:
              cpu: 50
              memory: 50
              pods: 50

        - name: "RemovePodsHavingTooManyRestarts"
          args:
            podRestartThreshold: 100
            includingInitContainers: true

      plugins:
        balance:
          enabled:
            - "RemoveDuplicates"
            - "LowNodeUtilization"
        deschedule:
          enabled:
            - "RemovePodsHavingTooManyRestarts"

Apply it with:

helm upgrade --install descheduler descheduler/descheduler \
  --namespace kube-system \
  --values values.yaml

Prefer Kustomize? The project ships base manifests too. Swap the ref for the release branch that matches your version (for example release-1.36 for v0.36.0):
kustomize build 'github.com/kubernetes-sigs/descheduler/kubernetes/cronjob?ref=release-1.34' | kubectl apply -f -

⏱️ CronJob vs Deployment mode

The descheduler can run in three shapes. Two matter day to day.

🕒 CronJob: runs on a schedule (schedule: "*/15 * * * *") and exits. Simple, cheap, easy to reason about. Great default.
♾️ Deployment: runs continuously and re-evaluates every deschedulingInterval (for example 5m). Pair it with leader election if you run more than one replica.

# Deployment mode snippet
kind: Deployment
replicas: 1
deschedulingInterval: 5m
leaderElection:
  enabled: true

👉 If you are not sure, start with CronJob. You only move to Deployment when you want tighter, continuous reaction.

🧪 test safely with dry run first

Please do not point this at production blind. Run it in dry run mode and read the logs first. 🙏

With Helm:

helm upgrade --install descheduler descheduler/descheduler \
  --namespace kube-system \
  --values values.yaml \
  --set cmdOptions.dry-run=true

Then watch what it would have done:

kubectl -n kube-system logs -l app.kubernetes.io/name=descheduler -f

You will see lines naming the pods it would evict and the strategy that picked them. Tune your thresholds until the list looks sane, then turn dry run off.

🧯 production safety checklist

Before you let it evict for real, walk this list. ✅

✅ PodDisruptionBudgets everywhere that matters. The descheduler honors them, so a good PDB is your strongest guard against an outage.
✅ nodeFit: true so it never evicts a pod that has nowhere else to go.
✅ minReplicas set so single replica workloads are left alone.
✅ Eviction caps via maxNoOfPodsToEvictPerNode, maxNoOfPodsToEvictPerNamespace and maxNoOfPodsToEvictTotal so a bad run cannot churn the whole cluster.
✅ System namespaces protected, and system critical priority pods left untouched.
✅ Sensible schedule. Every few minutes is rarely needed. Every 15 to 30 minutes is plenty for most clusters.

📊 keep an eye on it

The descheduler exposes Prometheus metrics, served on https://localhost:10258/metrics by default. You can change the address with the --binding-address and --secure-port flags.

Scrape it, then watch how many pods it evicts per run. A healthy cluster should settle into a small, steady number. A number that never drops is a signal that something keeps fighting it. ⚠️

🧠 the gotcha that bites everyone

Here is the classic trap: an eviction and reschedule loop.

It usually looks like this:

You enable a strategy (say node affinity based) to nudge pods toward preferred nodes.
Those preferred nodes are not available yet.
The descheduler evicts the pod, the scheduler puts it right back where it was, and the cycle repeats forever. 🔄

This shows up a lot with spot or autoscaled node pools, where the target nodes are still being provisioned when the eviction happens.

How to avoid the loop:

🧷 Keep nodeFit: true so it will not evict when there is no valid destination.
🐢 Use a calmer schedule so the cluster has time to settle between runs.
🎯 Be careful with preferred affinity rules as eviction triggers, since "preferred" is never fully satisfied and can churn endlessly.
🚦 Cap evictions so even a misconfiguration stays contained.

👉 If you see the same pods evicted over and over in dry run, fix that before going live. Dry run is exactly how you catch this.

🎁 wrapping up

The descheduler is one of those tools that quietly earns its keep. It does not replace your scheduler, it just keeps cleaning up after entropy:

🧹 It evicts poorly placed pods and lets the scheduler re-place them.
🛡️ It respects PDBs and the Default Evictor so it stays polite.
🚀 It installs in minutes with Helm and tunes with a single values file.
🧪 It is safe to trial thanks to dry run.

Start small. Turn on RemoveDuplicates and LowNodeUtilization, run in dry run, read the logs, then let it loose. Your nodes will thank you. 😄

Happy clustering and stay safe! 🧹🚀

🧠 Demystifying Metrics in Kubernetes

Hamdi (KHELIL) LION — Mon, 09 Feb 2026 07:48:25 +0000

Kubernetes does not magically know when to scale your workloads 🤖
It relies on metrics exposed through dedicated APIs to make scaling decisions.

There are three types of metrics you need to understand:

⚙️ Resource Metrics
📊 Custom Metrics
🌍 External Metrics

Each one represents a different kind of pressure on your system.

🤔 Why Metrics Matter for Autoscaling

Autoscaling in Kubernetes is mainly handled by the Horizontal Pod Autoscaler (HPA).

The HPA keeps asking:

“Hey… are my Pods struggling?” 😅

The answer comes from metrics. Without them, Kubernetes is basically guessing.

⚙️ 1 Resource Metrics = Pod Health Signals

These are the native Kubernetes metrics.

They come from the Metrics Server and only cover:

🧮 CPU usage
🧠 Memory usage

They are exposed via:

metrics.k8s.io

🧩 What they represent

They describe resource consumption, not business traffic.

Your app might be slow because of a database… but CPU could still be chill 🧊

📄 Example HPA based on CPU

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

If average CPU across Pods goes above 70 percent → more replicas 🔥

✅ Pros

Super simple
Works out of the box

❌ Limits

CPU is not always equal to user traffic
Memory often reacts too late

📊 2 Custom Metrics = Your App Talking to Kubernetes

Custom metrics come from applications inside your cluster.

They describe business or application load 💼

They are exposed through:

custom.metrics.k8s.io

🔁 Typical data flow

App exposes /metrics
Prometheus scrapes
Prometheus Adapter maps metrics → Kubernetes API

Now Kubernetes can ask:

“How busy is this Deployment really?” 👀

🌐 Example 1 HTTP requests per second

Metric in Prometheus:

http_requests_per_second

🧠 What it means
Real traffic handled by each Pod

📦 In Kubernetes
A metric attached to Pods

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: 200

If each Pod handles more than 200 requests per second → scale out 🚀

⏱ Example 2 Request duration

Metric:

request_duration_seconds

🧠 What it means
Application performance and saturation

Used as an Object metric:

- type: Object
  object:
    metric:
      name: avg_request_duration
    describedObject:
      apiVersion: apps/v1
      kind: Deployment
      name: api
    target:
      type: Value
      value: 0.5

If average latency goes above 500ms → time to add Pods 🏃‍♂️

🧵 Example 3 Active background jobs

Metric:

active_background_jobs

🧠 What it means
Internal workload of a worker

Each Pod reports its own load, and HPA scales when workers are overloaded 📈

✅ Pros

Scaling reflects real app behavior
Way smarter than CPU only

❌ Limits

Requires Prometheus + Prometheus Adapter
More components to maintain 😬

🌍 3 External Metrics = Work Waiting Outside the Cluster

External metrics come from systems outside Kubernetes.

They are exposed via:

external.metrics.k8s.io

These metrics describe work your Pods must process, even if it lives elsewhere 🌎

📬 Example 1 SQS queue length

Metric:

ApproximateNumberOfMessagesVisible

🧠 What it means
Number of messages waiting in the queue

In Kubernetes
A global metric not tied to specific Pods

If the queue grows → spawn more workers 💪

🐘 Example 2 Kafka consumer lag

Metric:

kafka_consumer_lag

🧠 What it means
Delay between producers and consumers

More lag = your consumers are falling behind 😱
Scale them up!

📦 Example 3 Redis job queue size

Metric:

redis_list_length

🧠 What it means
Number of jobs waiting in Redis queues

Perfect for worker autoscaling 🔄

⏰ Example 4 Time based scaling

Scale more Pods during office hours
Scale down at night 🌙

This is also treated as an external signal, because it’s not tied to Pod resource usage.

🧮 How HPA Uses These Metrics

HPA periodically queries:

Metric Type	API	What it measures
Resource	metrics.k8s.io	CPU and memory
Custom	custom.metrics.k8s.io	App level load
External	external.metrics.k8s.io	Event or system load

Then it calculates how many replicas you need 📐

⚡ KEDA vs Prometheus Adapter

Here comes the game changer 🎮

KEDA (Kubernetes Event Driven Autoscaling) focuses on event driven autoscaling and makes custom and external metrics way easier to use.

🧩 With Prometheus Adapter

You must:

Run Prometheus
Install and configure the Adapter
Write mapping rules
Manage RBAC and certs

It works, but it’s heavy 🏋️

⚡ With KEDA

You define a ScaledObject and KEDA does the magic ✨

Example SQS scaler:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-scaler
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.eu-west-1.amazonaws.com/123/my-queue
      queueLength: "10"
      awsRegion: eu-west-1

KEDA fetches the metric
Exposes it to Kubernetes
Creates and manages the HPA
Handles provider auth 🔐

All without Prometheus Adapter 🤯

🧭 When to Use What

If you want to scale on…	Use…
CPU or memory	Resource metrics
HTTP traffic or app load	Custom metrics
Queues, streams, SaaS	External metrics + KEDA
Events or scale to zero	KEDA

🎯 Final Takeaway

Kubernetes becomes truly powerful when scaling is driven by real workload signals, not just CPU.

⚙️ Resource metrics are the starting point
📊 Custom metrics bring application awareness
🌍 External metrics unlock event driven architectures
⚡ KEDA makes advanced autoscaling simple and production friendly

Once you understand these three metric types, autoscaling stops being magic and becomes a design tool you control 💡🚀

Happy clustering :)

Building a Kubernetes homelab the hard but right way 🧱☸️

Hamdi (KHELIL) LION — Tue, 20 Jan 2026 10:32:46 +0000

If you have been following my previous articles, you already know the drill 😄
I like setups that are:

boring
repeatable
close to real production

This article is not about spinning up a quick cluster.
It is about building a platform.

A platform you can:

rebuild from scratch
scale without fear
explain to a client or a teammate
reuse in real consulting missions

And yes, it runs at home 👀

The mindset first 🧠

Before touching any tool, I decided on one rule:

👉 no manual action that I cannot reproduce with code

That single rule drives everything else.

So the stack becomes very natural:

Proxmox for virtualization
Terraform to create machines
Cloud Init to bootstrap the OS
Kubespray to install Kubernetes
Helmfile for everything after day 1

Each layer has one job only.
No overlap. No shortcuts.

The big picture 🗺️

Here is what actually happens when I type terraform apply 👇

Proxmox clones VMs from a cloud init template
Each VM boots with the right user, SSH keys and network
Nodes are reachable immediately
Kubespray turns them into a real HA Kubernetes cluster
Helmfile deploys platform components on top

Once you understand that flow, debugging becomes easy and scaling becomes boring.
And boring is good 😌

Step 1 Proxmox as a real compute layer ⚙️

I treat Proxmox like a private cloud, not like a lab UI.

No clicking. No guessing.

Terraform talks directly to the Proxmox API and creates Kubernetes nodes exactly the same way every time.

What I am doing here

cloning from a golden cloud init template
injecting user data via snippets
assigning static IPs
tagging nodes for clarity
keeping compute concerns separate from Kubernetes

At this stage, Kubernetes does not exist yet.
And that is exactly what I want 👍

Step 2 The cluster is just data 📐

This is one of my favorite parts.

The cluster topology lives in a tfvars file.
No logic. No magic. Just data.

This is extremely important because it means:

the same code can create one cluster or ten
topology changes do not require refactoring
environments stay consistent over time

Adding a node or a whole new cluster is just editing data 🔁

Step 3 Reusable Terraform modules 🧩

Everything goes through a compute module.

Terraform is responsible for:

machines
networking
bootstrapping access

And nothing else.

Once the VMs exist, Terraform is basically done.

This clear boundary avoids a lot of confusion later.

Step 4 Kubespray does the heavy lifting ☸️

Kubespray is where Kubernetes actually comes to life.

It handles:

HA control plane
stacked etcd
container runtime
CNI and kubelet configuration
sane defaults and hardening

This is not a quick kubeadm script.
This is a production grade installer that I fully trust.

Step 5 Bootstrap only the essentials 🚦

At cluster creation time, I only enable what is strictly required:

metrics server
ingress nginx
metallb
gateway api

Nothing fancy. Nothing opinionated.

The goal here is to have a usable Kubernetes cluster, not a fully loaded platform.

Everything else can wait.

Step 6 Day 2 is simple and explicit with Helmfile 📦

This is important.

There is no fancy GitOps setup here.

No Argo CD bootstrap.
No Flux managing Flux.
No recursive GitOps inception 😄

Instead:

Helmfile is run explicitly
changes are intentional
failures are visible
debugging stays simple

For a single cluster or a homelab, this is often the best tradeoff.

When GitOps really makes sense 🚀

Now, things change when you have multiple clusters to manage.

At that point, this setup becomes extremely powerful.

Because:

Terraform already creates the machines
Kubespray already installs Kubernetes
Helmfile already describes the platform state

You can easily plug a GitOps layer on top to:

create a new cluster from scratch
apply a standard baseline
end up with a ready to use Kubernetes platform
with almost zero manual action

Think of it as:
👉 one command to pop a full Kubernetes cluster, ready for workloads

This is where Argo CD or Flux start to shine, but only when the scale justifies it.

Until then, keeping things simple is often the most mature choice 🧘

Why I like this approach so much ❤️

Because it grows with you.

simple when you have one cluster
scalable when you have many
understandable at every step
close to how real platforms are built

No magic.
No hidden automation.
Just solid engineering.

If you can run this at home, you can run it anywhere 🌍

What is next 🔮

In upcoming articles I will dive into:

storage choices and csi
dns and external-dns
security with kyverno
observability and metrics
multi cluster patterns

Stay tuned 👋 and happy clustering :)

Building a Kubernetes HomeLab - The Hard Way - DNS first, or everything else lies to you 🔥🌐

Hamdi (KHELIL) LION — Mon, 22 Dec 2025 08:08:54 +0000

Let’s be very clear from the start 👇
If DNS is shaky, everything above it becomes unreliable, misleading, and borderline hostile.

You will blame Kubernetes.
You will blame cert-manager.
You will blame yourself.

And it will still be DNS 😅

This article builds a real DNS layer, outside the cluster, the way it is done in production environments.
Not fancy. Not clever. Just solid.

DNS is not a feature, it is the foundation 🧱

Most homelabs treat DNS like a checkbox:

router default
maybe 1.1.1.1
maybe 8.8.8.8
call it a day

This works until you introduce:

multiple Proxmox nodes
Kubernetes bootstrap
internal TLS
ExternalDNS
service to service communication At that point, DNS stops being optional. It becomes infrastructure.

If this layer lies, everything else lies with it.

what I assume before we start breaking things 🔧

This article assumes:

you are comfortable with Linux and SSH
you control your LAN
you can assign static IPs
you are fine rebuilding things if needed

Network baseline used in examples:

LAN CIDR: 192.168.1.0/24
router: 192.168.1.1
DNS node: 192.168.1.169
internal domain: home.arpa

Why home.arpa?

defined by RFC 8375
designed for private networks
no collision with public DNS
predictable behavior across operating systems

.local is convenient.
.local is also a trap ☠️

how DNS flows in this homelab, no magic involved 🧭

The DNS request path is intentionally boring:

client
→ Pi-hole
→ Unbound
→ root servers
→ TLD servers
→ authoritative servers

Why this design?

Pi-hole sits in front:

single DNS endpoint for the entire LAN
full visibility into queries
local DNS records for infrastructure
API compatible with ExternalDNS later

Unbound sits behind:

full recursive resolver
DNSSEC validation
zero dependency on Google or Cloudflare
behaves like enterprise DNS stacks

One DNS server for the LAN:

deterministic behavior
easier troubleshooting
required for Kubernetes stability

building the DNS node, properly and calmly 🧱 A Raspberry Pi that does exactly one thing 🥧

Reserve a static IP from your router:

hostname: dns-01
ip: 192.168.1.169

Update the system:

sudo apt update
sudo apt upgrade -y
sudo apt install curl gnupg lsb-release -y

This node is sacred:

DNS only
no Docker
no Kubernetes
no experiments

If DNS breaks, you want zero doubt about the cause.

installing Pi-hole, the DNS entry point 🚦

Pi-hole will be the only DNS server exposed to the LAN.

Install it:

curl -sSL https://install.pi-hole.net | bash

During the installer:

interface: eth0
upstream DNS: custom
do not select any public DNS provider
enable the web admin interface

Once installed:

http://192.168.1.169/admin

Immediately change the admin password:

pihole -a -p

At this stage:

Pi-hole is answering DNS queries
but it should not resolve anything yet

installing Unbound, the actual DNS resolver 🔁

Unbound will do the real work: recursive resolution and DNSSEC validation.

Install Unbound:

sudo apt install unbound -y

Create a dedicated configuration for Pi-hole:

sudo tee /etc/unbound/unbound.conf.d/pi-hole.conf > /dev/null <<EOF
server:
  interface: 127.0.0.1
  port: 5335
  do-ip4: yes
  do-ip6: no
  do-udp: yes
  do-tcp: yes
  root-hints: "/var/lib/unbound/root.hints"
  harden-glue: yes
  harden-dnssec-stripped: yes
  use-caps-for-id: yes
  edns-buffer-size: 1232
  prefetch: yes
  verbosity: 1

forward-zone:
  name: "."
  forward-addr: 0.0.0.0
EOF

Fetch root hints:

sudo curl -o /var/lib/unbound/root.hints https://www.internic.net/domain/named.root

Enable and start Unbound:

sudo systemctl restart unbound
sudo systemctl enable unbound

At this point:

Unbound listens on 127.0.0.1:5335
performs full recursive resolution
validates DNSSEC

connecting Pi-hole to Unbound, the clean way 🔗

In the Pi-hole admin UI:

settings
dns
upstream DNS servers
custom server: 127.0.0.1#5335
disable all other upstream resolvers

Now the chain is complete:

clients talk to Pi-hole
Pi-hole forwards to Unbound
Unbound talks to the internet

No leaks.
No shortcuts.

naming things like an adult 🏷️

internal DNS records

Define local DNS records in Pi-hole:

proxmox-01.home.arpa   192.168.1.41
proxmox-02.home.arpa   192.168.1.42
proxmox-03.home.arpa   192.168.1.43
nas-01.home.arpa       192.168.1.50

ip plan

component   | ip
router          | 192.168.1.1
dns-01          | 192.168.1.169
proxmox-01  | 192.168.1.41
proxmox-02  | 192.168.1.42
proxmox-03  | 192.168.1.43
synology    | 192.168.1.50

Predictable naming beats clever naming.
Every single time.

proof before trust, always 🔍

Test recursive resolution:

dig google.com @192.168.1.169

Test DNSSEC:

dig dnssec-failed.org @192.168.1.169

Expected result:

SERVFAIL means DNSSEC is working ✅ Test internal names:

dig proxmox-01.home.arpa

Watch live DNS traffic:

pihole -t

If something breaks later, this is where you start.

how people accidentally sabotage their own DNS ☠️

Adding a public DNS as secondary:

internal names stop resolving
ExternalDNS becomes unreliable
cert-manager fails silently

Using .local:

mDNS conflicts
inconsistent resolution
debugging hell

Running DNS inside Kubernetes first:

bootstrap deadlock
cluster depends on itself
guaranteed pain

why this DNS layer unlocks everything else 🚀

With this setup in place:

Proxmox nodes resolve consistently
Kubernetes nodes bootstrap cleanly
ExternalDNS can manage records safely
cert-manager can issue internal certificates
ingress becomes predictable

This is the point where the homelab stops feeling fragile.

Next article:
Teaching your router who’s boss 🧠📡

lessons learned the hard way, so you do not have to 😅

DNS should be boring.
Silent.
Predictable.

If DNS is exciting, something is wrong.

This setup is:

reproducible
observable
production-inspired
intentionally boring

Hard now.
Peace later.

Happy Clustering :)

🏗️ Setting Up a Proxmox Cluster for My Homelab Kubernetes Environment

Hamdi (KHELIL) LION — Wed, 05 Nov 2025 06:27:25 +0000

Hey everyone 👋

Now that you know the hardware powering my homelab
it is time to show how I turned those machines into a Proxmox cluster ready to host a production style Kubernetes environment 🧑‍💻⚙️

This article covers the entire setup
including fixes for a few annoying real world problems 😅

Let’s get into it 🚀

✅ Step 1. Update BIOS on Lenovo Tiny Nodes

Before installing Proxmox
I strongly recommend updating the BIOS of each node

Why it matters
✅ fixes stability bugs
✅ improves CPU performance and thermal handling
✅ avoids random shutdowns or throttling
✅ unlocks better hardware support

Old BIOS = unpredictable issues later
Flash now and enjoy peace later ✅

✅ Step 2. Install Proxmox VE

I installed Proxmox VE on each of my three Lenovo M920q nodes
using this simple naming and addressing plan:

Node Hostname IP

Node 1 pve1 192.168.1.21
Node 2 pve2 192.168.1.22
Node 3 pve3 192.168.1.23

DNS points to my Raspberry Pi ✅

Installation takes a few minutes
and Proxmox is ready through a browser 🎉

✅ Step 3. Fix NIC Offloading Stability Issues

This one hit me hard 😬
I kept encountering:

❌ random node connectivity loss
❌ nodes freezing during network traffic
❌ SSH dropping without reason

Cause
👉 NIC offloading on Intel network adapters

Fix using this excellent community script
https://community-scripts.github.io/ProxmoxVE/scripts?id=nic-offloading-fix

After applying this
the cluster became rock solid ✅

If you run Lenovo Tiny
this fix is almost mandatory 👀

✅ Step 4. Create the Proxmox Cluster

On node pve1:

pvecm create homelab-cluster

On other nodes:

pvecm add 192.168.1.21

Now
I can manage all nodes from a single interface ✅
VMs can migrate easily between hosts ✅
Monitoring is unified ✅

Production vibes at home 😎

✅ Step 5. Flat Network Setup First

Managed Switch Comes Later

For now
the network is simple:

one Gigabit flat network

vmbr0 bridging the physical NIC

DHCP/static addressing for VMs

Benefits
✅ simple
✅ stable
✅ easy to troubleshoot

Future upgrade:
🔹 Managed PoE switch
🔹 VLAN segmentation for Kubernetes traffic
🔹 Insights and better monitoring

Great now
but room to grow ✅

✅ Step 6. Main Storage: Synology NAS (NFS + iSCSI)

Persistent workloads require persistent storage
so my Synology NAS handles all Kubernetes storage

✅ NFS for most apps
✅ iSCSI for database style block volumes
✅ RAID and snapshots for safety
✅ Backup target for Proxmox VMs

The NAS is the data backbone of the homelab
exactly how storage is handled in production 🚀

✅ Step 7. Automated Ubuntu 25.x Templates With Packer

To avoid manual VM setup
I automated template building using Packer

Operating system
🧩 Ubuntu Server 25.x
Modern kernel and great container support

My Packer template creates VMs that are:
✔ cloud init ready
✔ qemu guest agent installed
✔ ssh enabled
✔ updated and hardened
✔ swap disabled for Kubernetes

This allows me to deploy
a brand new Kubernetes node VM in under 30 seconds 🤯

Fast
repeatable
version controlled

Infrastructure as code for real 😁

⚠️ Important Real World Gotchas

Here are issues I personally hit
and how I fixed them ✅

Problem Cause Solution

Node freezes or network drops NIC offloading Apply NIC fix ✅
Unexpected reboots or throttling Old BIOS Flash BIOS ✅
High CPU temps under load Small chassis Tune cooling profile ✅
VMs or pods OOM Overcommit Reserve RAM for hypervisor ✅

Homelabs teach you technology
by breaking and fixing it
the best learning path 🔥

✅ What We Have Achieved So Far

Feature Status

Three node Proxmox cluster ✅
Network stability ✅
NAS integrated for storage ✅
Custom VM templates ✅
Automation friendly ✅
Kubernetes ready foundation ✅

A real mini datacenter is now running in my home
quietly
efficiently
and fully under my control 🏡✨

✅ Coming Next

Deploying Kubernetes with Kubespray

In the next article you will see:

🔥 3 high availability control plane nodes
🔥 6 workers ready for workloads
🔥 DNS fully internal
🔥 main storage already integrated
🔥 GitOps and monitoring soon after

This is when the real fun begins 😎🔥

Stay tuned

🧰 The Hardware That Powers My Homelab Kubernetes Cluster

Hamdi (KHELIL) LION — Tue, 28 Oct 2025 09:44:55 +0000

Hey everyone 👋

Before diving deeper into the software stack of my homelab
I want to fully introduce the physical components that make this whole project come alive 🧑‍🔧

This article focuses entirely on hardware choices and why each element matters

Let’s explore the building blocks of my mini data center 🏡⚡

🖥️ The Compute Layer

Three Lenovo ThinkCentre M920q Tiny

These three machines are the heart of the homelab
They provide the CPU and memory resources needed to run Kubernetes workloads inside virtual machines

Why I chose them
✅ compact and silent
✅ excellent performance per watt
✅ reliable and low power for 24x7
✅ scalable memory support

Specs per node

Intel CPU with multiple cores
32 GB RAM
NVMe SSD for fast VM storage

Together they give me:
🧠 96 GB RAM total
⚡ 36 vCPUs total

This allows me to:

simulate real production workloads
run 3 HA control plane nodes
run 6 worker nodes
add utility VMs when needed

Small machines
big power 😁

🌐 The Networking Layer

Gigabit Switch Today

Managed PoE Switch Tomorrow

Right now my network is built on a simple Gigabit Ethernet switch
Flat network
no VLANs
no advanced routing

This keeps things:
✅ simple to maintain
✅ easy to debug
✅ perfectly functional for a first iteration

But I am already thinking ahead 👀

A managed switch with PoE is on my radar because it would allow:
🧩 VLAN segmentation for test environments
📡 powering future access points or cameras directly from the switch
🛡️ network isolation for storage, monitoring or nodes
📈 better performance analysis through advanced metrics

For now
simplicity wins
but I want room to grow

🍓 The Brain of the LAN

Raspberry Pi as DNS Server

DNS is a critical piece in any infrastructure
especially Kubernetes

The Raspberry Pi handles:
✅ local DNS resolution
✅ service discovery across the home network
✅ potential Pi hole extension later

It ensures zero reliance on public DNS for internal services
Fast
lightweight
and rock solid 🧩

📦 The Data Layer

Synology NAS for Storage and Backups

Applications need persistent data
databases
monitoring stack
GitOps metadata

The Synology NAS provides:
✅ NFS shares for general storage
✅ iSCSI volumes for testing block level workloads
✅ snapshots and RAID protection
✅ backup space for Proxmox VMs

It is the data backbone of the homelab
keeping stateful workloads safe 🧡

🔌 Hardware Architecture Diagram

Here is a more complete view of the hardware layout

                Internet
                    │
                 Router
               with VPN
                    │
             ┌──────┴──────┐
             │             │
       Gigabit Switch      │
    (Flat network today    │
     but future Managed PoE)
             │
 ┌───────────┼───────────────┬───────────┐
 │           │               │           │
[Node 1]  [Node 2]        [Node 3]   Synology NAS
Proxmox   Proxmox         Proxmox         
 │         │               │              
 └─────────┴───────────┬───┘              
                       │                  
                Raspberry Pi (DNS)

Everything works together as a single coherent platform
Compute + Networking + DNS + Storage
self hosted and fully controlled ✅

🎯 Why this hardware matters

Every piece of this hardware plays a key role

Hardware	Role
Lenovo Tiny Nodes	Compute layer for Kubernetes
Gigabit Switch	Fast and reliable local communications
Managed PoE Switch future	Network control and segmentation
Raspberry Pi	DNS authority for all internal traffic
Synology NAS	Persistent data and VM backup storage

This creates a production style environment
ready for real workloads and real failures
without cloud overhead or noise pollution

A smart and scalable foundation 🤝

✅ Up Next

Next article in the series
➡️ Turning this hardware into a Proxmox cluster
➡️ VM management
➡️ Template strategy
➡️ Solid virtualization layer for Kubernetes

Stay tuned
The fun is only beginning 😎

🏡 Why I Built My Own Homelab to Run Kubernetes

Hamdi (KHELIL) LION — Sun, 26 Oct 2025 20:10:21 +0000

Hey everyone 👋

If you have been following my posts, you might have noticed that I have been pretty quiet lately 😅
There is a good reason for that

I have been spending a lot of time building my own homelab a place where I can safely test, break, and rebuild complex infrastructures without worrying about cloud costs or production risks 🧑‍💻

My goal is simple
Create a production grade Kubernetes environment at home using Proxmox for virtualization and Kubespray for cluster provisioning 💡
This environment allows me to test real world scenarios and also experiment with features for my Kubermates project 🚀

Let me walk you through the why and the how

⚙️ The Motivation

As a DevOps and SRE engineer, Kubernetes is part of my daily life.
But testing new tools, scaling strategies, GitOps workflows, or security rules often requires:

✅ complete freedom
✅ zero fear of downtime
✅ repeatability
✅ controlled costs

A homelab gives me full control of the stack including: network, storage, virtualization, routing, DNS, automation and resilience strategies 💪

It rapidly became much more than a hobby. It is now my mini data center

🧩 Proxmox as the Foundation

To host everything, I chose Proxmox VE

Reasons behind this choice:
🖥️ clean and powerful web UI and CLI
⚙️ excellent virtualization and storage integration
🌐 flexible networking and clustering
🔁 snapshots and backup system made for tests and disasters

Each hypervisor node runs multiple virtual machines that will become control planes, workers or utility nodes like DNS and monitoring services

Proxmox gives me the freedom to rebuild everything as many times as needed

☁️ Kubernetes Deployment With Kubespray

Once the virtual machines were ready, I needed a repeatable approach to deploy Kubernetes.
I selected Kubespray because:

🔥 it is production proven
🧩 uses Ansible which makes automation easy
🔧 allows deep customization
🔄 supports highly available clusters

This lets me rebuild the entire cluster in minutes while keeping the configuration under version control

Perfect for experimentation

💪 Homelab Specs

Right now, my homelab is running with:

🧠 96 GB of RAM
⚡ 36 vCPUs

This gives enough headroom to simulate:

Autoscaling and performance constraints

Monitoring with Prometheus, Loki, Thanos etc

GitOps flows with Argo CD and Helmfile

Network isolation and multi tenancy

Production security patterns

Computing constraints actually make the optimization challenges even more interesting

🔬 A Playground for Kubermates

If you know me, you know I am working on Kubermates
This homelab is now my real life testing ground for the platform

Here I can:
✅ test onboarding flows and automation patterns
✅ simulate multi tenant setups
✅ validate best practices for deploy and operate
✅ try both cluster and application level policies

Everything is reproducible and controlled
Exactly what I need to test responsibly

🌐 Secure Remote Access With WireGuard VPN

Although everything runs at home, I still want to access the environment securely when I travel or work away from my desk

For this, I use WireGuard VPN running on a small gateway VM

Why WireGuard

🔒 strong encryption

⚡ ultra fast performance

🧩 minimal configuration

📱 clients for Linux, Mac, Windows, iOS and Android

My access architecture:

Only VPN traffic can reach the Proxmox cluster and Kubernetes nodes

No ports are exposed directly to the internet

Internal DNS and Ingress work exactly as inside the local network

So I get full access to Grafana, Argo CD, dashboards, kubectl and SSH
as if I was physically at home 🏠

This keeps everything private and secure
Perfect for a personal lab

🧠 Key Lessons Learned So Far

Building Kubernetes at home gives you a deeper understanding of what is behind the cloud curtain

Main takeaways

Networking is always trickier than expected 😅

Local DNS matters more than anyone admits

Good VM templates save hours of setup time

Resource constraints teach real FinOps awareness

Breaking things makes you learn faster

And the best part
It is incredibly fun

🚀 What Comes Next

This is just the beginning

Upcoming articles in the series
1️⃣ Proxmox hardware and virtualization best practices
2️⃣ Kubespray automation and cluster bootstrapping
3️⃣ DNS and networking architecture for homelabs
4️⃣ GitOps, certificates and observability stack deployment
5️⃣ How to simulate production workloads at home

The goal
Help anyone build a professional grade Kubernetes experience locally
without cloud billing surprises

Thanks for reading and stay tuned for more 🔥

🧩 GitHub Actions Composite vs Reusable Workflows

Hamdi (KHELIL) LION — Fri, 18 Jul 2025 08:41:53 +0000

How to standardize and supercharge your CI/CD pipelines across projects

When your teams manage multiple projects with similar deployment patterns, repeating the same GitHub Actions steps over and over can become tedious, error-prone, and hard to maintain

Thankfully, GitHub Actions offers two powerful solutions to help standardize, reuse, and scale your CI/CD pipelines: Composite Actions and Reusable Workflows. When used together, they form a clean, modular, and DRY (don’t repeat yourself) CI/CD strategy

🔄 Composite Actions vs Reusable Workflows

🧱 Composite Actions

Composite Actions allow you to group steps (like docker build, terraform plan, or trivy scan) into a reusable component. Think of it like a function.

Best for:

Reusing logic (e.g., build and push images)
Hiding complex logic from the main workflow
Keeping workflows minimal and maintainable

Use it with:

uses: ./.github/actions/your-action

🔁 Reusable Workflows

Reusable Workflows are full workflows that can be invoked with inputs and secrets.

Best for:

High-level CI/CD orchestration (build, test, scan, deploy)
Enforcing common practices across repositories
Abstracting full pipelines

Use it with:

uses: org/repo/.github/workflows/your-workflow.yml@ref

🐳 Example: Docker Build and Scan with Trivy

Here is an example of a composite action to build and push a Docker image to Azure Container Registry (ACR) and scan it using Trivy.

`.github/actions/build-and-scan/action.yml`

name: Docker Build and Push

inputs:
  dockerfile:
    default: Dockerfile
  context:
    default: .
  image_name:
    required: true
  tag:
    required: true
  build_args:
    default: ""
  report_name:
    default: trivy-report

runs:
  using: "composite"
  steps:
    - uses: azure/login@v2
      with:
        creds: ${{ env.AZURE_CREDENTIALS }}

    - uses: azure/docker-login@v2
      with:
        login-server: ${{ env.CONTAINER_REGISTRY }}
        username: ${{ env.ACR_USERNAME }}
        password: ${{ env.ACR_PASSWORD }}

    - name: Build and Push Image
      shell: bash
      run: |
        docker build ${{ inputs.build_args }} \
          -t ${{ env.CONTAINER_REGISTRY }}/${{ inputs.image_name }}:${{ inputs.tag }} \
          -f "${{ inputs.dockerfile }}" "${{ inputs.context }}"
        docker push ${{ env.CONTAINER_REGISTRY }}/${{ inputs.image_name }}:${{ inputs.tag }}

    - name: Scan with Trivy
      uses: aquasecurity/trivy-action@0.32.0
      with:
        image-ref: '${{ env.CONTAINER_REGISTRY }}/${{ inputs.image_name }}:${{ inputs.tag }}'
        format: 'sarif'
        output: 'trivy-results.sarif'
        exit-code: 0

    - name: Upload SARIF to GitHub Security tab
      uses: github/codeql-action/upload-sarif@v3
      with:
        sarif_file: 'trivy-results.sarif'
        category: trivy

Usage in Workflow

- name: Build and Scan
  uses: ./.github/actions/build-and-scan
  with:
    image_name: my-app
    tag: ${{ github.sha }}

🌍 Example: Terraform/Terragrunt Validation with Checkov

Now let’s see a reusable workflow to validate Terraform code and scan for misconfigurations using Checkov.

`.github/workflows/infra-check.yml`

name: Infra Validation Workflow

on:
  workflow_call:
    inputs:
      working_dir:
        required: true
        type: string

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Terragrunt and OpenTofu
        run: |
          curl -s https://raw.githubusercontent.com/opentofu/opentofu/main/scripts/install.sh | bash
          curl -L https://github.com/gruntwork-io/terragrunt/releases/latest/download/terragrunt_linux_amd64 -o terragrunt
          chmod +x terragrunt
          sudo mv terragrunt /usr/local/bin/

      - name: Validate HCL Syntax
        run: |
          terragrunt hclfmt --terragrunt-check --terragrunt-diff --recursive
        working-directory: ${{ inputs.working_dir }}

      - name: Check Misconfigurations with Checkov
        uses: bridgecrewio/checkov-action@v12
        with:
          directory: ${{ inputs.working_dir }}
          quiet: true

Usage in Project Workflow

jobs:
  call-validation:
    uses: your-org/your-repo/.github/workflows/infra-check.yml@main
    with:
      working_dir: infrastructure/staging/eu-west-1

⚙️ Best Practices

✅ Keep composite actions in .github/actions
✅ Keep reusable workflows in .github/workflows
✅ Abstract frequently repeated patterns
✅ Use inputs, outputs, and env vars to increase flexibility
✅ Use workflow_call and workflow_dispatch to structure triggers
✅ Defer secrets to the calling workflow

💡 Conclusion

Using composite actions and reusable workflows together can transform your CI/CD processes into clean, repeatable, and secure pipelines. Whether you're building containers, provisioning infrastructure, or scanning for vulnerabilities, this modular approach will scale with your teams and your systems.

Enjoy building better pipelines !

🚀🌐 Elevating Infrastructure: From Terraform/Terragrunt Foundations to Platform Engineering 😊

Hamdi (KHELIL) LION — Fri, 18 Apr 2025 10:24:24 +0000

Hey there, cloud adventurers! 🚀 Let’s chat about why keeping Terraform (or OpenTofu) and Terragrunt in their own lanes is absolutely essential—and how using Terraform JSON tfvars makes life easier when you’re building nifty tools on top. Ready? Let’s dive in! 😄

Why Separation Is a Must, Not an Option 🙅‍♂️🙅‍♀️

It might be tempting to mix Terraform and Terragrunt into one big file—after all, they work together, right? But trust me, keeping them decoupled is a game‑changer:

🔒 Clear boundaries: Terraform focuses purely on resource definitions, while Terragrunt handles orchestration, remote state, and DRY patterns.
🔄 Independent versioning: You can upgrade your Terraform (or OpenTofu) modules (semantic versioning FTW!) without touching your Terragrunt configs—and vice versa!
📦 Reusability everywhere: Modules stay generic and shareable across projects when they don’t carry Terragrunt baggage.

By giving each tool its own space, you simplify CI/CD, make upgrades safer, and keep responsibilities crystal‑clear. Win‑win! 🏆

Module Versioning with Terraform 🎯

Hosting Terraform (or OpenTofu) modules in a versioned repo (GitHub, GitLab, private registry—you choose!) lets you tag releases like v1.2.3. Then, Terragrunt can lock to that exact version:

// modules/storage-account/main.tf
resource "azurerm_storage_account" "sa" {
  name                     = var.account_name
  resource_group_name      = var.resource_group_name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

variable "account_name" {
  type = string
}

variable "resource_group_name" {
  type = string
}

variable "location" {
  type = string
}

// modules/storage-account/versions.tf
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.0"
    }
  }
  required_version = ">= 1.5"
}

Publishing this as, say,

git::ssh://git@github.com/myorg/terraform-modules-repo.git//storage-account?ref=v1.0.0

means everyone downstream knows exactly what they’re getting—no surprises! 🎉

Decoupling Terraform (or OpenTofu) & Terragrunt 🛠️

For true decoupling, host your Terraform/OpenTofu modules in a separate Git repository, each with its own lifecycle, version tags, and release process. Then reference those modules from Terragrunt using Git sources. This keeps module development and environment orchestration fully independent.

Example Terragrunt config:

# live/dev/terragrunt.hcl
terraform {
  source = "git::ssh://git@github.com/myorg/terraform-modules-repo.git//storage-account?ref=v1.0.0"
}

inputs = jsondecode(file("${get_terragrunt_dir()}/variables.tfvars.json"))

The modules repository evolves on its own schedule.
Terragrunt configs in live/ reference specific module versions via Git source.

This pattern prevents accidental cross‑pollination of concerns, making upgrades, reviews, and rollbacks a breeze. ✨

Why Terraform JSON `tfvars` Rocks 📑✨

Sure, HCL is human‑friendly, but JSON tfvars shine when you need to build tools:

🤖 Machine‑readable: Every language can parse JSON out of the box—no extra libraries needed!
✔️ Schema validation: Hook up a JSON Schema to catch mistakes before you even hit “apply.”
💡 Dynamic generation: Your self‑service portal or CLI can spin out a JSON file in milliseconds.

Example variables.tfvars.json:

{
  "account_name": "myappstorage-${env}",
  "resource_group_name": "rg-${env}",
  "location": "eastus",
  "tags": {
    "environment": "dev",
    "owner": "platform-team"
  }
}

And Terragrunt gobbles it up with:

# live/dev/terragrunt.hcl
terraform {
  source = "git::ssh://git@github.com/myorg/terraform-modules-repo.git//storage-account?ref=v1.0.0"
}

inputs = jsondecode(file("${get_terragrunt_dir()}/variables.tfvars.json"))

See how clean that is? 🥳

Building Your Own Abstraction Tools 🧰

Looking to give your team a self‑service experience? Here are two solid approaches:

1. CLI with Go & Cobra ⚙️

Fast scaffolding: Cobra generates a project structure with commands, subcommands, and flag parsing out of the box.
Type safety & performance: Go’s static typing ensures reliable flag handling, and compiled binaries run blazingly fast.
Plugin architecture: Easily hook in custom modules—for example, a command like infra gen that generates Terraform JSON tfvars files for your Terragrunt inputs.
Example snippet:

  package main

  import (
    "encoding/json"
    "io/ioutil"
    "github.com/spf13/cobra"
  )

  var (
    accountName       string
    resourceGroupName string
    location          string
    env               string
    owner             string
    varsFile          string
  )

  var rootCmd = &cobra.Command{
    Use:   "infra",
    Short: "Self-service infra CLI",
  }

  var genCmd = &cobra.Command{
    Use:   "gen",
    Short: "Generate Terraform tfvars JSON file",
    RunE: func(cmd *cobra.Command, args []string) error {
      config := map[string]interface{}{
        "account_name":        accountName,
        "resource_group_name": resourceGroupName,
        "location":            location,
        "tags": map[string]string{
          "environment": env,
          "owner":       owner,
        },
      }
      data, err := json.MarshalIndent(config, "", "  ")
      if err != nil {
        return err
      }
      return ioutil.WriteFile(varsFile, data, 0644)
    },
  }

  func init() {
    genCmd.Flags().StringVar(&accountName, "account-name", "", "Azure storage account name")
    genCmd.Flags().StringVar(&resourceGroupName, "resource-group", "", "Resource group name")
    genCmd.Flags().StringVar(&location, "location", "eastus", "Azure location")
    genCmd.Flags().StringVar(&env, "env", "dev", "Deployment environment")
    genCmd.Flags().StringVar(&owner, "owner", "platform-team", "Resource owner")
    genCmd.Flags().StringVar(&varsFile, "out", "variables.tfvars.json", "Output JSON tfvars file")
    rootCmd.AddCommand(genCmd)
  }

2. Self‑service portal with Backstage 🎭

Unified developer portal: Backstage by Spotify lets you surface docs, tooling, and pipelines in one UI.
Infrastructure as Code plugin: Create a custom Backstage plugin to list available Terraform/OpenTofu modules and trigger Terragrunt runs with JSON inputs.
Policy & approval workflows: Integrate with existing Identity/Access tools and GitOps pipelines for reviews before provisioning.
Rich catalog: Use Backstage’s software catalog to track module versions, show metadata (owner, tags, docs), and link to Git repos.

Combine these approaches with strict Terraform/Terragrunt separation and JSON tfvars, and you’ll empower your team with both CLI efficiency and a polished web portal.

Wrapping Up 🎁

Separating Terraform (or OpenTofu) modules from Terragrunt configs isn’t just a “nice to have”—it’s essential for maintainable, scalable infrastructure. Pair that with JSON tfvars, and you’re set to build amazing tooling that your whole team will love. Happy provisioning! 🌟

🔐 Secure Secret Management with SOPS in Helm 🚀

Hamdi (KHELIL) LION — Thu, 27 Feb 2025 08:15:10 +0000

When managing applications deployed on Kubernetes, keeping secrets safe while still making them accessible to Helm charts is a challenge. Storing secrets in plaintext is a security risk 🚨 — and that’s where SOPS (Secrets OPerationS) and the Helm Secrets plugin come in!

In this guide, we’ll cover:

✅ How to use SOPS with age and GPG
✅ How to configure SOPS with sops.yaml for better management
✅ How to use Helm Secrets Plugin to manage encrypted secrets directly in your Helm charts
✅ A GitHub Actions workflow to securely deploy Helm charts using encrypted secrets

📌 Why Use SOPS with Helm?

SOPS is an open-source tool from Mozilla that lets you encrypt and decrypt secrets with ease. When combined with the Helm Secrets plugin, you can safely store your sensitive data in Git repositories and automatically decrypt them during Helm deployments. Here’s why it’s awesome:

✅ Keeps secrets encrypted in your repos
✅ Works with YAML, JSON, and ENV files
✅ Integrates seamlessly with Helm via the Helm Secrets plugin
✅ Fits perfectly into CI/CD pipelines like GitHub Actions for secure deployments

🔑 Using SOPS with `age`

Age is a modern, simple, and secure encryption tool. If you’re new to encryption, age is a great alternative to GPG.

✨ Step 1: Install `age` and `sops`

Install age and sops:

sudo apt install age    # Ubuntu/Debian

✨ Step 2: Generate an `age` Key

Run:

age-keygen -o ~/.config/sops/age/keys.txt

This will generate a key similar to:

# public key: age1xxxxxxx
AGE-SECRET-KEY-1XXXXXXYYYYYYYYZZZZZZ

Copy the public key (age1xxxxxxx)—this will be used for encryption.

✨ Step 3: Encrypt a YAML File with SOPS

Create a file called secrets.yaml:

db_user: "admin"
db_password: "supersecret"

Now, encrypt it using SOPS:

sops --encrypt --age age1xxxxxxx -i secrets.yaml

When you open secrets.yaml, you’ll see it’s fully encrypted! 🛡️

To decrypt:

sops --decrypt secrets.yaml

🔧 Configuring `sops.yaml` for Better Management

Instead of specifying the encryption method manually every time, SOPS supports a configuration file (.sops.yaml). This makes it easier to manage secrets across your team.

Create .sops.yaml in your repository:

creation_rules:
  - path_regex: secrets/.*\.yaml$
    age:
      - age1xxxxxxx  # Replace with your public key
  - path_regex: secrets/.*\.json$
    pgp:
      - ABC12345  # Replace with your GPG key ID

Now, when encrypting secrets inside the secrets/ folder, SOPS will automatically use the right encryption method! 🎉

Encrypt a new secret:

sops --encrypt -i secrets/app.yaml

🛠️ Using Helm with the Helm Secrets Plugin

The Helm Secrets plugin allows you to work with encrypted secrets directly in your Helm charts—no need to expose sensitive data!

✨ Step 1: Install the Helm Secrets Plugin

Install the plugin with:

helm plugin install https://github.com/jkroepke/helm-secrets

This plugin leverages SOPS to decrypt your secret files during Helm chart deployments.

✨ Step 2: Encrypt Your Secrets File

Create a file named secrets.yaml (if you haven’t already):

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: YWRtaW4=          # base64 encoded
  password: c3VwZXJzZWNyZXQ=   # base64 encoded

Encrypt it using SOPS:

sops --encrypt -i secrets.yaml

✨ Step 3: Deploy with Helm Using Encrypted Secrets

Deploy your Helm chart using the encrypted secrets file:

helm secrets upgrade --install my-release ./my-chart

The Helm Secrets plugin will automatically decrypt secrets.yaml during the deployment process. 🚀

🤖 Using SOPS and Helm in GitHub Actions

Integrate your secure secrets management into your CI/CD pipeline with GitHub Actions. Here’s an example workflow that deploys your Helm chart with encrypted secrets:

✨ Step 1: Store the `age` Private Key in GitHub Secrets

In your GitHub repository, navigate to Settings → Secrets and variables → Actions, and add:

SOPS_AGE_KEY: The private key from ~/.config/sops/age/keys.txt

✨ Step 2: Create the GitHub Actions Workflow

Create .github/workflows/deploy.yml:

name: Deploy with Helm & SOPS

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y sops age
          curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
          helm plugin install https://github.com/jkroepke/helm-secrets

      - name: Set up SOPS
        run: |
          mkdir -p ~/.config/sops/age/
          echo "${{ secrets.SOPS_AGE_KEY }}" > ~/.config/sops/age/keys.txt
          chmod 600 ~/.config/sops/age/keys.txt

      - name: Deploy with Helm
        run: |
          helm secrets upgrade --install my-release ./my-chart

🔥 What Happens in This Workflow?

Checks out the code ✅
Installs SOPS, age, Helm, and the Helm Secrets plugin ✅
Loads the age private key from GitHub Secrets ✅
Deploys the Helm chart with decrypted secrets on the fly ✅

Security Tip:

Make sure that any decrypted files are never committed to your repository! Always keep them out of version control. 🔒

🎯 Wrapping Up

SOPS and the Helm Secrets plugin offer a powerful and secure way to manage secrets in your Kubernetes deployments. With age encryption, a handy .sops.yaml configuration, and seamless integration via Helm, managing secrets has never been easier! 💪

By integrating these tools into your workflow, you get:

✅ Encrypted secrets safely stored in Git repositories
✅ Automatic decryption during Helm deployments
✅ Secure usage of secrets in CI/CD pipelines

Want to take it a step further? Try exploring AWS KMS, GCP KMS, or Azure Key Vault for even tighter security! 🔐🚀

Have questions or suggestions? Drop them in the comments! 💬

Happy clustering and stay safe! 🔐😊