DEV Community: cloud-sky-ops

Post 6/10 — Helm Fundamentals Done Right: Chart Architecture, Values Schema, and Reuse

cloud-sky-ops — Sun, 18 Jan 2026 20:22:48 +0000

If you're reading this blog I'd guess you're familiar with Helm and want to utilize if effectively. In case you aren't, I got you. Helm is a powerful packaging tool that help scale the Kubernetes manifest in a structured manner. I have highlighted the importance of Helm in some of my previous blogs and I'm covering some extra bits in this one. Just want my DevOps Engineer to not miss out on this simple yet effective tool. Do check my previous releases on Helm on my profile.

1) Executive Summary

Helm charts can be a blessing or a mess — depending on how disciplined your structure is. In this post, I’ll show how to design charts that scale: clean folder layout, validated values, reusable helpers, and consistent environment overlays.
Think of this as the point where “copy-pasted YAML” evolves into a templatized, validated, and reusable release artifact.

2) Prereqs

You should already be comfortable with:

kubectl basics and manifests (Deployment, Service, Ingress).
helm install, upgrade, and uninstall.
YAML indentation and Go template syntax ({{ .Values.image.repository }}).

If not, go back to Post 5 where we replaced Ingress with Gateway API — that’s the level of confidence you need.

3) Concepts

a) Chart layout & sane defaults

A Helm chart is just structured YAML + templates + metadata.

Default layout:

mychart/
├── Chart.yaml
├── values.yaml
├── values.schema.json
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── _helpers.tpl
└── charts/

Why it matters:

Chart.yaml defines metadata and dependencies.
values.yaml holds configuration knobs (default values).
templates/ defines the actual Kubernetes manifests.
_helpers.tpl hosts shared snippets like names and labels.

Decision cue:
If you can’t read your chart structure at a glance — it’s probably too magical. Keep it explicit.

b) `values.schema.json` validation

This file enforces structure and type-checking for your values.yaml.
Without it, Helm silently ignores typos — your app may deploy with wrong configs.

Before → After

Before:

# values.yaml
image:
  repo: nginx
  tag: latest  # typo: should be 'repository'

After:

// values.schema.json
{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "properties": {
    "image": {
      "type": "object",
      "properties": {
        "repository": { "type": "string" },
        "tag": { "type": "string" }
      },
      "required": ["repository"]
    }
  }
}

Now, helm lint will fail fast if someone mis-keys a field.

Decision cue:
If multiple people edit values.yaml, a schema is mandatory.

c) Helpers/partials & library charts

Avoid repetition. Use helpers (_helpers.tpl) for common names, labels, and annotations.

# _helpers.tpl
{{- define "app.fullname" -}}
{{ printf "%s-%s" .Release.Name .Chart.Name }}
{{- end -}}

Then reuse it:

metadata:
  name: {{ include "app.fullname" . }}

When multiple services share common patterns (e.g., serviceaccount.yaml, probes, labels), extract them into a library chart.
Library charts have no manifests — just helpers and templates that other charts can import.

Decision cue:
When you copy the same YAML block in more than two charts — extract it into a helper or library.

d) Environment overlays strategy

Instead of maintaining values-dev.yaml, values-stg.yaml, and values-prod.yaml in chaos, organize by overlays:

values/
  base.yaml
  dev.yaml
  stg.yaml
  prod.yaml

Each extends the base:

helm install myapp . -f values/base.yaml -f values/prod.yaml

Best practice:

Keep common knobs in base.yaml.
Only override what truly differs across envs.
Avoid env-specific conditionals in templates (if .Values.env == "prod" → smells).

4) Mini-Lab — From Raw Manifests to Chart

Start with raw YAML:

   kubectl create deploy web --image=nginx --dry-run=client -o yaml > deployment.yaml
   kubectl expose deploy web --port=80 --dry-run=client -o yaml > service.yaml

Create a chart:

   helm create webapp

Move manifests: replace defaults with your generated files under templates/.
Add schema validation: Create values.schema.json to validate image & replica fields.
Extract helpers: Move naming logic to _helpers.tpl.
Test locally:

   helm template webapp .
   helm lint webapp
   helm install webapp . --dry-run

Result: a reusable, validated chart with clear structure.

5) Cheatsheet

Command	Purpose
`helm show values <chart>`	Display default values
`helm lint .`	Validate chart + schema
`helm template .`	Render YAML locally
`helm upgrade --install <release> . -f values/prod.yaml`	Idempotent deploy
Schema tip:	Run `jq . values.schema.json` to quickly validate structure

6) Pitfalls

Pitfall	Why It Hurts	Fix
Values drift	Each team edits `values.yaml` differently.	Enforce schema + overlays.
Over-templating	Logic explodes inside YAML.	Keep logic minimal; pre-compute in CI/CD.
No library reuse	Duplication everywhere.	Move shared blocks to `_helpers.tpl` or library charts.
Missing schema	Silent deployment errors.	Add `values.schema.json` early.

7) Wrap-up — Next: Shipping Charts with Confidence: Lint, Tests, Diff, OCI & Provenance (Post 7)

We now have clean, validated, reusable Helm charts that can be tested in isolation.
Next, in Post 7, we’ll cover how to ship charts with confidence: Lint, Tests, Diff, OCI & Provenance.

Helm is powerful when it’s predictable.
Do it right once, and every future deployment becomes safer, faster, and auditable. Thanks for sticking around till the end, drop your comments and feedback below.

Post 5/10 — From Ingress to Gateway API: Safer, Smarter Traffic Control

cloud-sky-ops — Sun, 07 Dec 2025 10:52:32 +0000

The question of why traffic should be controlled using additional means when Kubernetes offers in-built traffic management capabilities. Many Cloud Service Providers also have offerings around the same but having a set of defined policies to execute traffic management is necessary. This helps the application distribute the load in the system strategically. It also ensures lower latency in fulfilling request. Moreover, these policies make application secure my allowing limited access of resources both in and out of the Kubernetes cluster.

1) Executive Summary

When I first learned Kubernetes networking, Ingress felt like the magic front-door. But as clusters, tenants, and compliance needs grew, that door started to creak. Enter Gateway API — the modern, role-aware evolution that finally decouples platform and app teams safely.

This post walks through what changes (and why), shows a Before → After migration, and ends with a mini-lab you can run on any cluster.

2) Prereqs

A working cluster (minikube/kind/EKS/GKE).
kubectl ≥ 1.28 + gateway-api CRDs installed (kubectl apply k https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml).
Basic understanding of Ingress objects, Services, and TLS secrets.

3) Concepts

Ingress vs Gateway mental model

Aspect	Ingress	Gateway API
Scope	Cluster-wide front door	Namespaced, multi-tenant gateways
Roles	One YAML = shared config	Split into Gateway (ops) + HTTPRoute (dev)
Extensibility	Annotations galore	Typed fields + policies
Status	Controller-specific	Standardized conditions

Think of Ingress as a flat router file. Gateway API adds layers:

GatewayClass – defines the implementation (e.g., Istio, NGINX).
Gateway – deployed by platform teams (where traffic enters).
HTTPRoute – attached by app teams (what traffic does).

Gateway + HTTPRoute basics

Gateway: specifies listeners, ports, and TLS termination.
HTTPRoute: defines matches (hosts, paths, headers) and forwards traffic to services.

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: web-gw
  namespace: platform
spec:
  gatewayClassName: nginx
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: my-cert
    allowedRoutes:
      namespaces:
        from: All

# route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop-route
  namespace: shop
spec:
  parentRefs:
  - name: web-gw
    namespace: platform
  hostnames: ["shop.example.com"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: shop-svc
      port: 80

TLS, Retries, Timeouts, Header Matching

Gateway API formalizes what used to be annotation chaos:

  rules:
  - matches:
    - headers:
      - name: x-region
        value: eu
    backendRefs:
    - name: shop-svc-eu
      port: 80
  filters:
  - type: RequestTimeout
    requestTimeout:
      duration: 5s
  - type: Retry
    retry:
      count: 3
      statusCodes: ["5xx"]

Canary / Mirroring / Multi-tenant Gateways

Weighted traffic splits and mirrored routes are first-class now:

  backendRefs:
  - name: shop-svc
    port: 80
    weight: 80
  - name: shop-svc-canary
    port: 80
    weight: 20

Multiple HTTPRoutes can attach to one Gateway, safely namespaced.
Ops maintain TLS + exposure policies; devs just ship routes.

4) Mini-Lab — Replace Ingress with Gateway/HTTPRoute + Canary + TLS

Before (legacy Ingress):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts: [shop.example.com]
    secretName: my-cert
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: shop-svc
            port:
              number: 80

After (Gateway API):
Apply gateway.yaml and route.yaml from above.

Verify:

kubectl get gateways -A
kubectl get httproutes -A
kubectl describe httproute shop-route

Add a canary service and observe:

kubectl get endpoints shop-svc-canary -o wide
curl -H "x-canary:true" https://shop.example.com

You’ll see controlled traffic flow, better observability, and cleaner ownership.

5) Cheatsheet

Task	Command
List GatewayClasses	`kubectl get gatewayclass`
Inspect listeners	`kubectl describe gateway <name>`
List routes per gateway	`kubectl get httproute -A --field-selector spec.parentRefs.name=<gw>`
Apply TLS secret	`kubectl create secret tls my-cert --cert cert.pem --key key.pem -n platform`
Test connectivity	`curl -k https://host`

Key fields:

listeners.protocol, tls.mode, allowedRoutes
rules.matches, filters, backendRefs.weight

6) Pitfalls

Policy attachment order: Gateway policies apply before Route policies → watch precedence.
Overlapping routes: Multiple HTTPRoutes with same host/path → controller decides priority.
Cross-namespace refs: If allowedRoutes.from=Same, dev teams can’t attach externally.
Controller support: Check status.conditions for accepted listeners.

7) Wrap-Up

With Gateway API, Kubernetes finally gives us policy-aware traffic control that scales across teams and environments — safer, cleaner, and built for automation.

Next up in the series: Helm Fundamentals (Post 6) — we’ll templatize these Gateway objects and parameterize routes the right way.

Post 4/10 — Smart Scaling & Cost Control: HPA/KEDA + Cluster Autoscaler vs Karpenter

cloud-sky-ops — Wed, 15 Oct 2025 03:43:00 +0000

All businesses need a means to minimize cash leaks in its infrastructure. A DevOps engineer with the understand of autoscaling can help with this significantly. While the aspect of writing lightweight applications is still crucial, there still might be cases where the business use case is expecting inconsistent spikes in traffic, example: on a ticketing platform when tickets are live for a popular music tour. This is where we need a reliable system that can manage this sudden spike gracefully. “Scaling is easy — until it isn’t. Then it’s your cloud bill that reminds you.”

1️⃣ Executive Summary

This post is about smart elasticity — how Kubernetes scales workloads and clusters while keeping cost in check.
I’ll walk through the difference between workload-level autoscaling (HPA/KEDA) and cluster-level autoscaling (CA vs Karpenter), with a hands-on demo and a practical comparison.

By the end, you’ll know:

How requests + limits shape node packing and spend.
When to use HPA vs KEDA.
Why Karpenter often replaces the traditional Cluster Autoscaler.
How to avoid the most common scaling pitfalls — thrash, cold starts, and QoS chaos.

2️⃣ Prereqs

A basic Kubernetes cluster (minikube/kind/EKS/GKE).
kubectl, helm, kubectl top.
Sample deployment (e.g., an Nginx or Go web app).
Optional: KEDA + Metrics Server installed.

3️⃣ Concepts

Requests / Limits & Rightsizing

Each container declares:

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 256Mi

Requests → scheduler guarantee.
Limits → runtime cap.
Under-request = OOM; over-request = waste.

Decision cue: profile your pods with kubectl top pod and rightsize before enabling autoscaling; otherwise, autoscalers just magnify inefficiency.

HPA (Horizontal Pod Autoscaler)

HPA scales replicas based on metrics — typically CPU or memory:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization

✅ Before: Fixed 3 replicas → wasted CPU at night.
✅ After: HPA 2-10 replicas → 40 % cost reduction, stable latency.

🔁 KEDA ( Kubernetes Event-Driven Autoscaler )

KEDA extends HPA with external triggers (Queue, Kafka, Prometheus, etc.):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: queue-scaledobject
spec:
  scaleTargetRef:
    name: worker
  triggers:
  - type: azure-queue
    metadata:
      queueName: orders
      connection: AzureWebJobsStorage
      queueLength: '5'

Decision cue: use KEDA when metrics are outside Kubernetes — SQS depth, Kafka lag, HTTP requests/sec, etc.

Cluster Autoscaler (CA) Overview

CA watches unschedulable pods and adds/removes nodes via your cloud provider’s ASG or NodePool.
It’s conservative — scales in minutes, not seconds — and tied to fixed instance groups.

Decision cue: CA is fine for homogeneous workloads with predictable demand.

Karpenter

Karpenter is the next-gen autoscaler for AWS (and now CNCF incubating):

Launches any instance type on demand, not limited by node groups.
Supports Spot + On-Demand mix.
Consolidates underutilized nodes automatically.

Example Provisioner:

apiVersion: karpenter.sh/v1beta1
kind: Provisioner
metadata:
  name: default
spec:
  limits:
    resources:
      cpu: 1000
  requirements:
  - key: "node.kubernetes.io/instance-type"
    operator: In
    values: ["m6i.large", "m6a.large"]
  consolidation:
    enabled: true

Decision cue: choose Karpenter when you need rapid scale-out (seconds), mixed instance types, or Spot savings.

4️⃣ Mini-Lab — HPA + CA vs Karpenter

Deploy a sample app

   kubectl create deploy web --image=nginx --replicas=2
   kubectl expose deploy web --port=80

Apply HPA

   kubectl autoscale deploy web --cpu-percent=50 --min=2 --max=10

Generate load

   kubectl run load --image=busybox -- /bin/sh -c \
     "while true; do wget -q -O- http://web; done"

Observe scale-up

   kubectl get hpa
   kubectl get pods -w

Compare cluster-level response

With Cluster Autoscaler, new nodes join slowly, via ASG.
With Karpenter, new instance spins up in < 60 s.

If you can’t run Karpenter (non-EKS), compare plan latency and node variety conceptually.

5️⃣ Cheatsheet

Task	Command
View metrics	`kubectl top pods`
Deploy HPA	`kubectl autoscale deploy <app> --cpu-percent=X --min=A --max=B`
Install KEDA	`helm install keda kedacore/keda`
Inspect ScaledObjects	`kubectl get scaledobject`
Check node scale	`kubectl get nodes -w`
View CA logs	`kubectl -n kube-system logs deploy/cluster-autoscaler`
Describe Karpenter provisioners	`kubectl get provisioner`

6️⃣ Pitfalls

Thrash: aggressive min/max or low cooldown = scale in/out loops.
Cold starts: node boot latency + image pull → avoid per-request scaling.
QoS classes: BestEffort pods may be evicted first; use Guaranteed for critical services.
Over-requests: rightsizing before autoscaling saves more than any tuning flag.

7️⃣ Wrap-Up

Kubernetes scaling is now less about can it scale and more about how smartly it scales.
HPA/KEDA handle micro elasticity.
Karpenter redefines macro elasticity and cloud efficiency.

In the next post — Modern Traffic Shaping (Post 5) — we’ll explore how to handle that scale with smart routing: Ingress, Service mesh, and load-balancing patterns.

Diagram 1: Demand vs Capacity Timeline

Diagram 2: Provisioning Decision Tree

Post 3/10 — Safe Cadence: Kubernetes Upgrades, Version Skew, and Feature Gates

cloud-sky-ops — Fri, 10 Oct 2025 02:55:00 +0000

Executive Summary

Upgrades are where clusters live—or die—by their discipline. A rushed kubectl upgrade can silently break workloads, APIs, or CRDs.
In this post, I’ll walk you through how to plan, test, and execute Kubernetes upgrades safely, understand version-skew guarantees, manage feature gates, and validate everything before production.

By the end, you’ll know why cadence matters more than version chasing.

Prereqs

Familiarity with kubectl, kubeadm, or managed control planes (EKS, GKE, AKS).
Working understanding of pods, CRDs, controllers, and API objects.
Access to a staging or pre-prod environment where you can test.

Concepts

1️⃣ Support Windows & Version Skew

Kubernetes maintains a 1-year support window with three active minor releases (e.g., 1.29 → 1.31).
Control plane and kubelet skew rules:

Component	Supported Skew
`kube-apiserver` vs others	±1 minor
`kube-controller-manager`, `scheduler`	same as API Server
`kubelet`	up to 1 minor older
`kubectl`	±1 minor

Decision cue: If your nodes or clients are >1 minor behind control plane → plan upgrade immediately.

Before → After

Before	After
1.24 control plane, 1.21 nodes	❌ Unsupported (skew = 3)
1.28 control plane, 1.27 nodes	✅ Supported (skew = 1)

2️⃣ Upgrade Order: Control Plane → Nodes

Always upgrade control plane first, then worker nodes, then addons.

Flow:
etcd → apiserver → controller-manager → scheduler → kubelet → kube-proxy → CNI → CSI → custom controllers

Decision cue:

Never let newer nodes talk to an older API Server.
The API Server must always be the newest component in the cluster.

3️⃣ Deprecation Checklist & API Migration

Each minor release deprecates APIs. To find the list, run:

kubectl krew install deprecations
kubectl deprecations --k8s-version v1.31
kubectl deprecations --k8s-version 1.31

Or if you don’t have that plugin:

kubectl get --raw /openapi/v2 | grep v1beta1

Then use:

kubectl convert -f old.yaml --output-version apps/v1 > new.yaml

Before → After Example

Before:

apiVersion: apps/v1beta2
kind: Deployment

After:

apiVersion: apps/v1
kind: Deployment

Decision cue:
→ Fix API versions before upgrading; otherwise manifests may fail silently during rollout.

4️⃣ Feature Gate Evaluation in Pre-Prod

Feature gates toggle experimental behavior.
Check current state:

kube-apiserver --help | grep feature-gates

Enable safely:

apiServer:
  extraArgs:
    feature-gates: "PodDisruptionPolicy=false,JobPodFailurePolicy=true"

Decision cue:

Always enable gates only in pre-prod first; validate e2e conformance before rollout.

🧪 Mini-Lab — Plan a Minor Upgrade

Goal: Upgrade 1.29 → 1.30 in staging safely.

# 1. View current versions
kubectl get nodes -o wide

# 2. Drain one node
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# 3. Upgrade control plane
sudo kubeadm upgrade apply v1.30.1

# 4. Upgrade kubelet/kubectl
sudo apt install kubelet=1.30.1-00 kubectl=1.30.1-00
sudo systemctl restart kubelet

# 5. Uncordon and validate
kubectl uncordon node-1
kubectl get nodes

Validate with conformance tests:

sonobuoy run --mode=certified-conformance
sonobuoy status
sonobuoy retrieve .

Toggle a feature gate safely:

kubectl patch deployment kube-apiserver \
  -n kube-system --type merge \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"kube-apiserver","args":["--feature-gates=DynamicResourceAllocation=true"]}]}}}}'

Rollback with kubectl diff → kubectl apply -f rollback.yaml --server-side --dry-run=server.

Cheatsheet

Task	Command
List deprecations	`kubectl deprecations --k8s-version X.Y`
Convert manifest	`kubectl convert -f old.yaml --output-version apps/v1`
Dry-run on server	`kubectl apply -f file.yaml --server-side --dry-run=server`
Diff check	`kubectl diff -f manifests/`
Upgrade order	`etcd → control plane → nodes → addons → CRDs`

Pitfalls

CRD / Version Drift
CRDs registered via Helm or Operators may pin old API versions (e.g., apiextensions.k8s.io/v1beta1).
→ Always kubectl get crd and check .spec.versions.

Admission Policy Surprises
Mutating / Validating webhooks compiled against old libraries can block Pods post-upgrade.
→ Audit webhooks with kubectl get validatingwebhookconfigurations and test dry-runs in staging.

Etcd Storage Version Drift
Even after API migrated, etcd may still store old serialized versions.
→ Use kube-storage-version-migrator to reconcile.

Diagram — Upgrade Workflow

✅ Wrap-Up

Upgrades are not a sprint—they’re a cadence.
Each minor version should move through your environments with predictable rhythm, pre-validation, and rollback clarity.

Next in the series (Post 4) → Smart Scaling & Cost Control: HPA/KEDA + Cluster Autoscaler vs Karpenter.

Post 2/10 — Reliability by Design: Probes, PodDisruptionBudgets, and Topology Spread Constraints

cloud-sky-ops — Sun, 05 Oct 2025 02:37:00 +0000

Now you’ve laid a baseline of namespace isolation, quotas, network policy, and PSA (if not, see Post 1 of the series), the next layer is reliability. In this post I walk you through the key Kubernetes primitives that help your workloads survive disruptions and evolve safely: probes, PDBs, topology constraints, and rollout strategies. This blog dives deeper into these nuanced offerings by Kubernetes, buckle up for some fun, hope you enjoy the ride.

Executive Summary

Use liveness, readiness, and startup probes to let Kubernetes detect and recover from unhealthy application states.
A PodDisruptionBudget (PDB) ensures voluntary disruptions (e.g. node drain, rolling upgrades) don’t violate your availability SLO.
TopologySpreadConstraints force pods to be balanced across failure domains (zones, nodes) to reduce blast radius.
Carefully configure rollout strategies (surge, maxUnavailable) in your Deployment to control downtime vs speed.
Together, these tools let you design reliability from the start—preventing cascading failures rather than firefighting.

Prereqs

You already have a Kubernetes cluster with kubectl access (as assumed in Post 1).
You have an existing Deployment (or create one) that you can modify.
You have at least two nodes (ideally in different zones or failure domains) to test spread constraints.
You can cordon/drain nodes (kubectl drain) to simulate disruption.

Concepts

A. Probes: Liveness / Readiness / Startup

Definition: Probes are periodic checks (HTTP, TCP, exec) that Kubernetes makes into containers to detect their health or readiness. Without them, a stuck process can stay “Running” indefinitely, and traffic may go to unhealthy pods.
Best practices:
- Always include readiness in your services so endpoints only include truly ready pods.
- Use startup probe for apps with long initialization (so liveness doesn’t kill them prematurely).
- Be conservative: gentle probe intervals and timeouts to avoid false negatives under GC / background load.
- Test locally to find thresholds under load.

Commands / YAML snippets:

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 2

livenessProbe:
  httpGet:
    path: /live
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
  timeoutSeconds: 2

startupProbe:
  httpGet:
    path: /ready
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Before → After:

Before: Pod is “Running” perpetually, even if app crashes internally; Service routes traffic to a dead process.
After: Kubernetes restarts the pod automatically (liveness), and the pod is only added to load via readiness when healthy.

When to use: Always for production; startup probes if your app has long boot phases.

B. PodDisruptionBudget (PDB)

Definition: A PDB is a policy that defines the minimum number (or fraction) of pods that must remain available during voluntary disruptions. Ensures your system doesn’t accidentally violate availability during upgrades, node drains, or autoscaling events.
Best practices:
- Use minAvailable when you want a floor on availability, or maxUnavailable for a cap on disruption.
- Don’t set them too tight (you might block your own updates). Leave wiggle room.
- Align PDBs with your rolling update strategy (surge / unavailable) to avoid deadlocks.
- Monitor PDB status (kubectl get pdb) to detect stuck updates.

Commands / YAML snippet:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

Before → After:

Before: During a node drain, all pods could be disrupted and cause downtime.
After: Only one pod can be evicted at a time, preserving minimal availability.

When to use: For any service with more than one replica; optional for batch jobs but still beneficial.

C. TopologySpreadConstraints

Definition: A declaration in pod spec that controls how Kubernetes spreads pods across failure domains (nodes, zones) to enforce balance. Avoid overconcentration: if one zone or node goes down, you don’t lose your entire workload.
Best practices:
- Use well-known node labels: e.g. topology.kubernetes.io/zone or kubernetes.io/hostname.
- maxSkew = 1 is a typical starting point (difference <=1 pod across domains).
- whenUnsatisfiable: use DoNotSchedule for strict spreading or ScheduleAnyway for softer enforcement.
- Use the same spread constraints on all revisions of your Deployment to maintain consistency.

Commands / YAML snippet:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app

Before → After:

Before: All pods end up in one zone or node (fastest node, better resources).
After: Pods evenly distributed across zones/nodes, reducing risk if one fails.

When to use: For replicated workloads in HA setups, especially multi-zone or multi-node clusters.

D. Rollout Strategies (Surge / maxUnavailable)

Definition: In a Deployment’s strategy.rollingUpdate, maxSurge is how many extra pods can be created during upgrade; maxUnavailable is how many pods are allowed to drop at once. They control the trade-off between speed and availability during upgrades.
Best practices:
- Use maxUnavailable: 0 and maxSurge: 1 (or more) for zero-downtime (if resources allow).
- For batch or low-priority workloads, allow some unavailability for faster rollout (e.g. 20–30%).
- Always test with your PDB + spread settings to ensure upgrade doesn’t stall.

Commands / YAML snippet:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

Before → After:

Before: Default 25% surge/unavailable may drop too many pods at once, violating SLO during high load.
After: You control how many can be updated and how many remain live.

When to use: Always set explicitly rather than relying on defaults; vary per workload type.

Mini-Lab

Let’s walk through building an app that has probes, PDB, and topology spread constraints. Then simulate node disruption and see your reliability in action.

Step 1: Namespace + sample deployment

kubectl create ns reliability-demo
kubectl config set-context --current --namespace=reliability-demo

Deploy a simple app (e.g. HTTP echo) with 3 replicas:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  labels: { app: echo }
spec:
  replicas: 3
  selector:
    matchLabels: { app: echo }
  template:
    metadata:
      labels: { app: echo }
    spec:
      containers:
      - name: app
        image: hashicorp/http-echo:0.2.3
        args:
        - "-text=hello"
        ports:
        - containerPort: 5678

kubectl apply -f echo.yaml
kubectl get pods -o wide

Step 2: Add probes, PDB, and topology constraints

Edit the above spec:

# inside spec.template.spec.containers[0]:
        readinessProbe:
          httpGet:
            path: /health
            port: 5678
          initialDelaySeconds: 3
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /live
            port: 5678
          initialDelaySeconds: 10
          periodSeconds: 10
# Also add at spec
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: { app: echo }
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: { app: echo }

Create a PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: echo-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: echo

Apply both:

kubectl apply -f modified-echo.yaml
kubectl apply -f pdb-echo.yaml

Check status:

kubectl get pods -o wide
kubectl get pdb
kubectl describe pdb echo-pdb

Step 3: Simulate node drain and verify behavior

Pick one node:

NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
kubectl cordon $NODE
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --disable-eviction

Watch pods:

kubectl get pods -o wide -w

Expectations:

One pod evicted (PDB ensures minAvailable 2 remain).
New pods rescheduled onto other nodes (respecting topology constraints).
No downtime for readiness endpoints if requests route to healthy pods.

Restore node:

kubectl uncordon $NODE

Bonus: Trigger a rollout:

kubectl set image deployment/echo app=hashicorp/http-echo:0.3.0
kubectl rollout status deployment/echo

Observe that upgrades are safe, respecting availability and spread.

Cheat Sheet Table

Action	Command / YAML	Purpose / Note
Add readiness probe	see snippet above	Ensures pod is only traffic-ready when healthy
Add liveness probe	see snippet above	Restarts stuck pods
Add startup probe	similar pattern as above	Prevents liveness kill during slow init
Create PDB	`kubectl apply -f pdb.yaml`	Enforce minimal availability
Check PDB	`kubectl get pdb` / `kubectl describe pdb`	Confirm eviction limits
Set rollout strategy	`strategy.rollingUpdate`	Control surge / downtime
Drain a node	`kubectl drain <node> --ignore-daemonsets`	Simulate voluntary disruption
Check rollout	`kubectl rollout status deployment/<name>`	Wait for safe update
View pod distribution	`kubectl get pods -o wide`	Inspect zone/node distribution

Pitfalls & Gotchas

Misconfigured probes kill healthy pods / false positives.
Probe timeouts too aggressive lead to unnecessary restarts. Tune after load testing.
PDB deadlocks your upgrades.
If your PDB demands minAvailable = replicas, plus you set maxUnavailable = 0 and maxSurge = 0, your rollout cannot make progress. Always allow some headroom (e.g. maxSurge: 1) or loosen PDB.
Skew violations during rolling updates.
Spread constraints sometimes misbehave during updates: the scheduler considers old and new pods together in balancing, so you may temporarily skew. Use ScheduleAnyway as fallback or trigger rescheduling.
Asymmetric zones or resource imbalance.
If one zone has less capacity, strict constraints may block scheduling. Use a softer whenUnsatisfiable: ScheduleAnyway or allow some skew.
Spread constraints don’t rebalance after scale-down.
When pods are removed, existing pods may end up unevenly distributed. Use a Descheduler or manual intervention to rebalance.
Startup probe too permissive or missing disables liveness protection.
Without startup probe, a slow boot may trigger liveness failure. With one, if too lenient, you delay detecting broken pods.

Wrap-up & Bridge to Post 3

With probes, PDBs, topology spread, and rollout control, you now have a robust reliability foundation. Your services will survive node drains, upgrades, and zone outages while satisfying availability SLOs.

In Post 3, we’ll build on this: version upgrades, canary/blue-green deployments, cluster upgrades, and rollback strategies, so you can evolve your system safely under load.

Diagram 1 — Rolling update timeline

Diagram 2 — Zone spread visualization

Drop a comment if you learned something new and share your thoughts. Thank You.

Mastering GitHub Actions: Insights and pitfalls of "if conditions"

cloud-sky-ops — Wed, 01 Oct 2025 21:28:00 +0000

As a DevOps engineer, I live inside CI/CD pipelines. GitHub Actions has become one of my go-to tools, and I’ve spent countless hours tweaking workflows, battling edge cases, and learning lessons the hard way. One of the powerful offering of Actions is the use of if conditions on jobs and steps.

In this post, I’ll walk you through the main if conditions available, share a real-world scenario where things went sideways for me, and explain how I fixed it. Hopefully, this saves you from a few hours of frustration the next time you’re wiring up your own pipelines.

The Essential `if` Conditions in GitHub Actions

Here are the most common conditional expressions you’ll run into:

if: success() – Runs the job only if all dependencies succeeded.
if: failure() – Runs the job only if at least one dependency failed.
if: cancelled() – Runs the job only if a dependency was cancelled.
if: always() – Runs the job regardless of the success/failure of dependencies. The catch with always is that it will also run the job is the workflow is cancelled by the user. So always means ALWAYS, no exceptions.

Due to this, it safer (and recommended by GitHub) to use the following option:

if: !cancelled() – Runs the job unless a dependency was cancelled (my personal favorite fallback for robust pipelines).

These look straightforward, but once you combine them with needs: and multi trigger workflows, like I did, a real quirks emerges.

When `if: ${{ !cancelled() }}` Bit Me Back

I was working on a CI/CD workflow that needed to support two types of pushes: to main and to release/* branches. The requirement was simple enough, there were 4 jobs:

run-test-cases: runs for all branches.
run-script-on-main: runs only when the branch is main.
docker-build-push: should run on both branches. On main, it must wait for run-script-on-main to complete. On release/*, it should still run regardless of the run-script-on-main job being skipped
trigger-deployment: runs after docker-build-push (and run-test-cases) for both main and release/*.

Here’s the workflow:

name: CI/CD Pipeline

on:
  push:
    branches:
      - main
      - release/*

jobs:
  run-test-cases:
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/run-tests.sh

  run-script-on-main:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/main-only.sh

  docker-build-push:
    needs: [run-test-cases, run-script-on-main]
    if: ${{ !cancelled() }}
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/docker-build-push.sh

  trigger-deployment:
    needs: [run-test-cases, docker-build-push]
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/trigger-deployment.sh

The Problem

What I expected:

On main: run-script-on-main runs, then docker-build-push, then trigger-deployment.
On release/*: skip run-script-on-main, run docker-build-push, then trigger-deployment.

What actually happened:

On main: ✅ everything worked fine.
On release/*: ❌ docker-build-push ran, but trigger-deployment was skipped.

This was confusing to me because trigger-deployment didn’t even depend on run-script-on-main (using needs section).

DAG Construction & Pruning (root cause)

In GitHub Actions, the entire workflow is modeled as a Directed Acyclic Graph (DAG):

Directed: The flow of execution moves in one direction, from dependencies to dependents.
Acyclic: There are no cycles; a job cannot depend on itself (directly or indirectly).
Graph: Jobs are nodes, and needs: defines the edges (dependencies) between them.

This DAG determines the exact order in which jobs are executed, skipped, or pruned. GitHub Actions builds the DAG at workflow parsing time before it executes any jobs.

Example

Imagine three jobs:

jobs:
  A:
    runs-on: ubuntu-latest
    steps:
      - run: echo "Job A"

  B:
    needs: A
    runs-on: ubuntu-latest
    steps:
      - run: echo "Job B"

  C:
    needs: [A, B]
    runs-on: ubuntu-latest
    steps:
      - run: echo "Job C"

The DAG looks like this:

A → B → C

A runs first.
Once A succeeds, B can run.
Once both A and B succeed, C can run.

If A fails, both B and C are automatically skipped because their prerequisites were never satisfied.

Practical Example in GitHub Workflows

Take a workflow where lint and unit-tests must run in parallel, and build depends on both.

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - run: npm run lint

  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - run: npm test

  build:
    needs: [lint, unit-tests]
    runs-on: ubuntu-latest
    steps:
      - run: npm run build

Here the DAG is:

[ lint && build ] → unit-tests

If both lint and unit-tests succeed, build executes.
If either lint or unit-tests fails (or gets skipped), build won’t run unless you explicitly allow it with if: always() or if: !cancelled().

DAG in Practice: Why It Matters

The problem I hit with if: ${{ !cancelled() }} happened because DAG pruning is strict.

When a job is skipped, it’s not just “ignored”—it still exists as a node in the DAG.
Any job that needs it inherits that skipped state unless conditions explicitly override it.
That’s why my trigger-deployment got skipped even though its direct needs (docker-build-push, run-test-cases) were satisfied: GitHub propagated the “skipped upstream” status through the graph.

👉 Understanding DAGs is the key to predicting workflow behavior in GitHub Actions. If you model jobs as a dependency graph instead of thinking in linear “step order,” it becomes much clearer why some jobs run, skip, or silently disappear.

The Fix

After banging my head on figuring out the why, I started looking into the workarounds, I realized the simplest and most robust solution was to avoid making docker-build-push depend on an optional job. Instead, I folded the branch-specific logic into a step:

docker-build-push:
  needs: [run-test-cases]
  runs-on: ubuntu-latest
  steps:
    - run: ./scripts/docker-build-push.sh
    - if: github.ref == 'refs/heads/main'
      run: ./scripts/main-only.sh

Now the dependency chain is clean:

No skipped optional jobs breaking the DAG.
docker-build-push always runs.
On main, the extra script runs as a conditional step.

This avoided the skipped-job propagation entirely and made the workflow behave exactly as intended.

Key Takeaways

Be cautious with needs: when optional jobs are involved.
if: ${{ !cancelled() }} is safer than if: always(), but it can still create tricky dependency issues.
When in doubt, prefer conditional steps inside a single job over branching jobs with if:. It keeps the DAG simpler and avoids hidden propagation quirks.

In the end, this was another lesson in the “everything is a DAG” world of GitHub Actions. The more predictable your graph, the fewer surprises you’ll get down the road. Thanks for making it till the end. Do share your thoughts and any alternate approaches on the solution. Have a beautiful day.

Post 1/10 — Multi-Tenancy & Security Baseline with Namespaces, Quotas, NetworkPolicies, and Pod Security Admission

cloud-sky-ops — Sat, 27 Sep 2025 22:26:21 +0000

Author: A senior DevOps engineer who’s broken (and fixed) too many clusters so you don’t have to.

Executive Summary

Carve tenants with Namespaces to isolate RBAC, policies, and quotas per team.
Enforce fair sharing with ResourceQuota + LimitRange so noisy neighbors can’t starve the cluster.
Lock down traffic with NetworkPolicy: start with default-deny, then allow the minimum (DNS, app→DB).
Harden workloads with Pod Security Admission (PSA) to block privileged/unsafe pod specs at the namespace boundary.
Ship a repeatable baseline: two team namespaces, quotas/limits, default-deny + specific allows, PSA restricted.

Prereqs

A Kubernetes cluster (≥ v1.28 recommended) and kubectl configured.
Cluster has CNI that supports NetworkPolicy (Calico, Cilium, Antrea, etc.).
Helm optional (not required here).
You have cluster-admin for initial setup.

kubectl version --short
kubectl get nodes -o wide
kubectl get pods -n kube-system

Concepts

1) Namespaces for isolation

Definition: Namespaces slice a cluster into logical tenants with independent RBAC, quotas, and policies. Prevents accidental cross-team impact, scopes access and makes policy application simple (kubectl label ns once).
Best practices:

One namespace per team/app-tier, not per environment object (use labels like env=prod).
Name predictably: team-a, team-b, platform, etc.
Attach PSA labels, default network policies, and quotas at namespace creation.

Commands:

kubectl create ns team-a
kubectl create ns team-b
kubectl label ns team-a env=dev --overwrite
kubectl get ns --show-labels

Before → After:

Before: All pods in default, broad access, hard to apply policies.
After: team-a/, team-b/ with their own quotas, PSA, and network baselines.

When to use: Always—namespaces are table stakes for multi-tenancy.

2) ResourceQuota & LimitRange

Definition: ResourceQuota caps aggregate usage per namespace; LimitRange sets per-pod/container defaults and maxima. Stops runaway resource grabs and ensures every pod has sensible requests/limits for scheduling and stability.

Best practices:

Pair them: RQ for team-level ceilings; LR for sane per-workload defaults.
Set CPU/memory requests+limits and object counts (pods, PVCs, services).
Include ephemeral-storage where supported; keep room for rollouts (e.g., 20–30% headroom).

Commands:

kubectl apply -n team-a -f quota-team-a.yaml
kubectl get resourcequota -n team-a
kubectl describe limitrange -n team-a

Before → After:

Before: Pods without limits; one job consumes all CPU → others starve.
After: Each container gets defaults; namespace can’t exceed its slice.

When to use: Whenever multiple teams share nodes or costs matter (i.e., always).

3) NetworkPolicy

Definition: NetworkPolicy declares which pods may talk to which, for ingress and egress. Prevents lateral movement, accidental chats between unrelated services, and data exfiltration.

Best practices:

Start with default-deny for both ingress and egress.
Then allow only what you need (DNS 53, app→DB 5432, etc.).
Use labels consistently; avoid relying on IPs. Add namespaceSelector + podSelector for system services like CoreDNS.

Commands:

kubectl apply -n team-a -f np-default-deny.yaml
kubectl apply -n team-a -f np-allow-dns.yaml
kubectl apply -n team-a -f np-allow-app-to-db.yaml

Before → After:

Before: Any pod can connect to any pod/Internet.
After: Only DNS + app→db allowed; everything else dropped.

When to use: In any regulated or multi-tenant cluster; also in prod by default.

4) Pod Security Admission (PSA)

Definition: PSA enforces Kubernetes security profiles (privileged, baseline, restricted) via namespace labels. Blocks dangerous specs (privileged, hostPID/IPC, hostPath, capability escalation) before pods land on nodes.

Best practices:

Default restricted enforce on app namespaces; use baseline temporarily while migrating.
Apply enforce, warn, and audit labels during rollout to see breaks before enforcing.
Keep images running as non-root; avoid broad capabilities and host volumes.

Commands:

kubectl label ns team-a \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version=latest \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/audit-version=latest --overwrite

Before → After:

Before: Developers could deploy privileged pods or mount /var/run/docker.sock.
After: Such pods are rejected at admission with a clear error.

When to use: Immediately after namespace creation; dev/prod alike (stricter in prod).

Diagram 1 — Namespace/Tenant Layout

Mini-Lab (≈25 min): Two tenants, quotas/limits, default-deny, app→db allow, PSA restricted

You can paste these as-is; tweak CPU/memory as needed.

Create namespaces + PSA labels

kubectl create ns team-a
kubectl create ns team-b

for ns in team-a team-b; do
  kubectl label ns $ns \
    pod-security.kubernetes.io/enforce=restricted \
    pod-security.kubernetes.io/enforce-version=latest \
    pod-security.kubernetes.io/warn=restricted \
    pod-security.kubernetes.io/warn-version=latest \
    pod-security.kubernetes.io/audit=restricted \
    pod-security.kubernetes.io/audit-version=latest --overwrite
done

Apply ResourceQuota + LimitRange

quota-team-a.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
spec:
  hard:
    pods: "30"
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    persistentvolumeclaims: "10"
    services: "10"
    services.loadbalancers: "2"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: team-a-defaults
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: "200m"
      memory: "256Mi"
    default:
      cpu: "500m"
      memory: "512Mi"
    max:
      cpu: "2"
      memory: "2Gi"

Apply to both namespaces:

kubectl apply -n team-a -f quota-team-a.yaml
kubectl apply -n team-b -f quota-team-a.yaml

Deploy a tiny app and a “db” in team-a

app-db.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
  labels: { app: db }
spec:
  replicas: 1
  selector: { matchLabels: { app: db } }
  template:
    metadata: { labels: { app: db } }
    spec:
      containers:
      - name: db
        image: postgres:16-alpine
        env:
        - { name: POSTGRES_PASSWORD, value: dev }
        ports:
        - { containerPort: 5432, name: pg }
---
apiVersion: v1
kind: Service
metadata:
  name: db
  labels: { app: db }
spec:
  selector: { app: db }
  ports:
  - { port: 5432, targetPort: pg, protocol: TCP }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels: { app: web }
spec:
  replicas: 1
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
      - name: web
        image: curlimages/curl:8.10.1
        command: ["sleep","infinity"]

kubectl apply -n team-a -f app-db.yaml
kubectl wait -n team-a --for=condition=available deploy/web deploy/db --timeout=90s

Smoke test (pre-policy, should connect):

POD=$(kubectl -n team-a get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
kubectl -n team-a exec -it "$POD" -- sh -lc 'curl -m2 db:5432 || true'
# Expect: connection established or at least a TCP handshake banner (not blocked)

Enforce default-deny (ingress+egress) in team-a

np-default-deny.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}        # select all pods
  policyTypes:
  - Ingress
  - Egress

kubectl apply -n team-a -f np-default-deny.yaml

Test (should now fail):

kubectl -n team-a exec -it "$POD" -- sh -lc 'curl -m2 db:5432 || echo BLOCKED'
# Expect: BLOCKED / timeout

4) Allow only DNS + app→db

np-allow-dns.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: { }      # all pods may resolve DNS
  policyTypes: [ Egress ]
  egress:
  - to:
    - namespaceSelector:
        matchLabels: { kubernetes.io/metadata.name: kube-system }
      podSelector:
        matchLabels: { k8s-app: kube-dns }
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }

np-db-ingress-from-web.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-allow-from-web
spec:
  podSelector:
    matchLabels: { app: db }
  policyTypes: [ Ingress ]
  ingress:
  - from:
    - podSelector:
        matchLabels: { app: web }
    ports:
    - { protocol: TCP, port: 5432 }

np-web-egress-to-db.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-allow-to-db
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [ Egress ]
  egress:
  - to:
    - podSelector:
        matchLabels: { app: db }
    ports:
    - { protocol: TCP, port: 5432 }

kubectl apply -n team-a -f np-allow-dns.yaml -f np-db-ingress-from-web.yaml -f np-web-egress-to-db.yaml
kubectl -n team-a exec -it "$POD" -- sh -lc 'curl -m2 db:5432 || true'
# Expect: succeeds (policy allows only web→db and DNS egress)

Repeat the same baseline (quotas/limits, default-deny, PSA labels) for team-b.

Diagram 2 — NetworkPolicy Traffic Matrix

YAML & Bash Reference

PSA labels (namespace):

kubectl label ns team-a pod-security.kubernetes.io/enforce=restricted --overwrite
kubectl label ns team-a pod-security.kubernetes.io/{enforce, warn, audit}-version=latest --overwrite

ResourceQuota & LimitRange:

apiVersion: v1
kind: ResourceQuota
metadata: { name: <ns>-quota }
spec:
  hard:
    pods: "30"
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
---
apiVersion: v1
kind: LimitRange
metadata: { name: <ns>-defaults }
spec:
  limits:
  - type: Container
    defaultRequest: { cpu: "200m", memory: "256Mi" }
    default:        { cpu: "500m", memory: "512Mi" }
    max:            { cpu: "2",    memory: "2Gi" }

Default-deny + allows (template): see np-default-deny.yaml, np-allow-dns.yaml, np-db-ingress-from-web.yaml, np-web-egress-to-db.yaml above.

Namespace-scoped context (handy):

kubectl config set-context --current --namespace=team-a
# or create a named context once:
kubectl config set-context team-a --cluster=$(kubectl config current-context) --user=$(kubectl config view -o jsonpath='{.contexts[?(@.name=="'$(kubectl config current-context)'")].context.user}') --namespace=team-a
kubectl config use-context team-a

Cheatsheet Table

Task	Command / File	Notes
Create namespace	`kubectl create ns <name>`	Add labels (`env=prod`, PSA) immediately.
Label PSA restricted	`kubectl label ns <name> pod-security.kubernetes.io/enforce=restricted --overwrite`	Add `warn`/`audit` to preview breaks.
Apply quotas/limits	`kubectl apply -n <ns> -f quota-<ns>.yaml`	Pair `ResourceQuota` with `LimitRange`.
Check quota usage	`kubectl describe resourcequota -n <ns>`	Watch `Used` vs `Hard`.
Default-deny policy	`kubectl apply -n <ns> -f np-default-deny.yaml`	Deny both Ingress and Egress.
Allow DNS	`kubectl apply -n <ns> -f np-allow-dns.yaml`	Needed for service discovery.
Allow app→db	`kubectl apply -n <ns> -f np-allow-app-to-db.yaml`	Pair with ingress on db and egress on app.
Switch namespace	`kubectl config set-context --current --namespace=<ns>`	Keeps commands short & safe.
Dry-run a spec	`kubectl apply -f x.yaml --dry-run=server`	Admission & schema checks without deploying.

Pitfalls & Recovery

“Policies don’t seem to apply” (order confusion).
Symptom: You created allows but traffic still blocked.
Why: Policies are additive; any default-deny remains in effect unless an allow matches both sides (ingress on target + egress on source if you deny both).
Fix: Ensure you have egress allow on source and ingress allow on destination.
DNS broke after default-deny egress.
Symptom: curl db or kubectl logs stalls; apps can’t resolve service names.
Fix: Add np-allow-dns.yaml allowing egress to kube-system/k8s-app=kube-dns on TCP/UDP 53.
Pods rejected after enabling PSA restricted.
Symptom: Error from server (Forbidden) with fields like runAsNonRoot.
Fix: Adjust workload security context (non-root, drop caps, no hostPath). Temporarily set enforce=baseline and warn=restricted to migrate, then flip back.
Quota exceeded during rollout.
Symptom: New ReplicaSet can’t scale (exceeded quota).
Fix: Increase pods/requests.cpu/memory quota, or lower replicas. Keep rollout headroom (20–30%).
No limits → eviction during pressure.
Symptom: Pods OOMKilled or preempted under load.
Fix: Set defaults via LimitRange; ensure requests approximate real usage (watch kubectl top pod).
East-west traffic across namespaces unexpectedly blocked.
Symptom: Cross-ns calls time out.
Fix: Add namespaceSelector + podSelector in allow rules, or route via ingresses with clear policies.

Wrap-Up (What this unlocks for reliability—Post 2 teaser)

With Namespaces, Quotas/LimitRanges, NetworkPolicies, and PSA in place, you’ve built a security and fairness floor: teams can’t trample each other, pods can’t talk without permission, and unsafe specs are stopped at the door.

In Post 2, we’ll layer observability SLOs, PodDisruptionBudgets, health/readiness gates, and autoscaling on top of this baseline to keep releases smooth and reliability measurable.

Appendix: “Before → After” Quick Contrasts

Networking:
Before: web can reach everything → lateral risk.
After: Only web → db + DNS; all else denied.
Resources:
Before: No limits; one job spikes → node thrash.
After: Default requests/limits; quotas cap team usage.
Security:
Before: Privileged pods slip through.
After: PSA restricted blocks them with actionable errors.

Got questions or want a tailored baseline for your stack? Drop them in, and I’ll fold them into a separate post.

Post 0/10 — Foundations: Zero to Base Setup

cloud-sky-ops — Tue, 23 Sep 2025 03:10:00 +0000

I’m a DevOps engineer and in this blog I’ll get you from zero to a working local cluster, deploy an app with raw YAML, then switch to Helm—plus the mental models and commands you’ll reuse daily.

Executive Summary

Stand up a local Kubernetes cluster (kind/minikube) and verify it with kubectl.
Cement a mental model of Pods → ReplicaSets → Deployments → Services → Controllers.
Apply clean YAML (Deployment + Service), then Helm-ify it with a minimal chart.
Learn the 12 commands I actually use day-to-day (kubectl + Helm).
Practice pitfall recovery (images won’t pull, pending pods, bad Services, context issues).

Prereqs

minikube
kind
Docker running (required by kind & often by minikube).
kubectl (v1.28+ recommended)
helm

Here’s the updated Install Hints section starting numbering from 1 instead of 0:

Install Hints (macOS, Linux, Windows)

1. Docker (required for kind & often for minikube)

macOS (with Homebrew):

  brew install --cask docker
  open /Applications/Docker.app

Tip: Ensure Docker Desktop is running before creating clusters.

Linux (Debian/Ubuntu):

  sudo apt-get update
  sudo apt-get install -y ca-certificates curl gnupg lsb-release
  sudo mkdir -p /etc/apt/keyrings
  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  sudo apt-get update
  sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

  # Add your user to docker group (log out/in after)
  sudo usermod -aG docker $USER

Windows (PowerShell as Admin):

  choco install docker-desktop

Tip: After install, restart and launch Docker Desktop.

2. kubectl

macOS:

  brew install kubectl

Linux (Debian/Ubuntu):

  sudo apt-get update && sudo apt-get install -y apt-transport-https gnupg
  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg
  echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
  sudo apt-get update
  sudo apt-get install -y kubectl

Windows (PowerShell as Admin):

  choco install kubernetes-cli

3. kind (Kubernetes in Docker)

macOS:

  brew install kind

Linux:

  curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64
  chmod +x ./kind
  sudo mv ./kind /usr/local/bin/kind

Windows (PowerShell as Admin):

  choco install kind

4. minikube

macOS:

  brew install minikube

Linux:

  curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
  sudo install minikube-linux-amd64 /usr/local/bin/minikube

Windows (PowerShell as Admin):

  choco install minikube

5. Helm

macOS:

  brew install helm

Linux:

  curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Windows (PowerShell as Admin):

  choco install kubernetes-helm

Verify Installations

kubectl version --client
kind version
minikube version
helm version
docker --version

Concepts & Skills

1) Kubernetes Mental Model

Definition: Kubernetes reconciles desired state (your specs) to actual state (running Pods) via controllers. It explains everything: you ask for N replicas, deployments manage ReplicaSets which manage Pods; Services route to healthy Pod endpoints.

Best practices:

Treat Pods as ephemeral; deploy via Deployments, not bare Pods.
Scale & roll out via Deployments; don’t hand-edit Pods.
Expose traffic via Services; keep labels/selectors consistent.
Let controllers do the work; think “declare, don’t script.”

Commands you’ll use:

kubectl get deploy,rs,pods,svc -A
kubectl describe deploy <name>
kubectl rollout status deploy/<name>
kubectl scale deploy/<name> --replicas=3

Before → After

# Before: single Pod (fragile)
apiVersion: v1
kind: Pod
metadata: { name: hello }
spec: { containers: [{ name: app, image: nginx:1.25 }] }

# After: Deployment + Service (managed, scalable)
apiVersion: apps/v1
kind: Deployment
metadata: { name: hello }
spec:
  replicas: 2
  selector: { matchLabels: { app: hello } }
  template:
    metadata: { labels: { app: hello } }
    spec:
      containers: [{ name: app, image: nginx:1.25 }]
---
apiVersion: v1
kind: Service
metadata: { name: hello }
spec:
  type: ClusterIP
  selector: { app: hello }
  ports: [{ port: 80, targetPort: 80 }]

Decision cues (when to use):

Deployment for stateless apps, rolling updates.
StatefulSet for ordered/identity-bound Pods (DBs).
DaemonSet for per-node agents.
Job/CronJob for finite/recurring work.

2) YAML Hygiene

Definition: Kubernetes resources are structured YAML: apiVersion, kind, metadata, spec. 90% of “Why won’t it work?” is indentation, wrong apiVersion, or misplaced fields.

Best practices:

Keep a stable skeleton: apiVersion/kind/metadata/spec.
Use 2 spaces; never tabs.
Verify with kubectl explain and kubectl apply --dry-run=client -f.
Prefer labels (e.g., app: hello) for selectors; avoid ad-hoc names.
Pin images (e.g., nginx:1.25) to avoid surprise upgrades.

Helpful commands:

kubectl explain deployment.spec --recursive | less
kubectl apply --dry-run=client -f hello.yaml

Before → After

# Before: wrong apiVersion, mixed tabs
apiVersion: apps/v2
kind: Deployment
metadata:
    name: hello   # <-- tab!
spec: {}

# After: correct and clean
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 1
  selector: { matchLabels: { app: hello } }
  template:
    metadata: { labels: { app: hello } }
    spec:
      containers: [{ name: app, image: nginx:1.25 }]

Decision cues:

Use kubectl explain if you’re guessing a field.
Use --dry-run for fast validation before real apply.

3) `kubectl` Basics

Definition: kubectl is the CLI to talk to the API server using your current context/namespace. Wrong context/namespace is the #1 cause of “it’s not found.”

Best practices:

Set a default namespace per context.
Use wide output and labels in get.
Rely on describe and events to debug.
Use -o yaml to see server-filled fields.
Use kubeconfig contexts; don’t point prod by accident.

Commands:

kubectl config get-contexts
kubectl config set-context --current --namespace=dev
kubectl get pods -o wide
kubectl describe pod <pod>
kubectl get events --sort-by=.lastTimestamp

Before → After

# Before: implicit default namespace (surprise!)
kubectl get pods

# After: explicit context & namespace
kubectl config set-context --current --namespace=dev
kubectl get pods -n dev

Decision cues:

If a resource “disappears,” check namespace.
If a command “hangs,” check context/cluster.

4) Local Cluster: kind or minikube

Definition: kind runs Kubernetes in Docker containers; minikube runs a local Kubernetes VM/container. Fast, reproducible clusters for development and demos.

Best practices:

Use kind for simple, Docker-backed clusters.
Use minikube for addons/ingress and driver flexibility.
Name clusters per project (kind create cluster --name demo).
Export kubeconfig only for the current shell (avoid prod collisions).

Commands:

# kind
kind create cluster --name k90
kubectl cluster-info
kubectl get nodes

# minikube
minikube start
minikube status
kubectl get nodes

Before → After

# Before: ad-hoc envs, “works on my machine”
docker run -p 8080:80 nginx

# After: real cluster parity locally
kind create cluster --name k90
kubectl apply -f k8s/hello.yaml

Decision cues:

kind if you already live in Docker land & want speed.
minikube if you need addons and drivers (HyperKit, Docker, etc.).

5) Helm Basics

Definition: Helm is Kubernetes’ package manager: it templatizes YAML, versioned as charts, and then installed as releases. Stop copy-pasting YAML; manage environments, values, upgrades, and rollbacks cleanly.

Best practices:

Keep charts minimal; prefer a small set of templates.
Parameterize only what changes across envs.
Validate with helm lint and render with helm template.
Track releases with helm list and rollback confidently.

Commands:

helm create hello
helm lint hello
helm template hello
helm install hello ./hello -n dev --create-namespace
helm upgrade hello ./hello -f values-dev.yaml
helm rollback hello 1

Before → After

# Before: multiple hand-maintained YAML files per env
kubectl apply -f dev/hello.yaml
kubectl apply -f prod/hello.yaml

# After: one chart, many values
helm install hello ./hello -f values-dev.yaml
helm upgrade hello ./hello -f values-prod.yaml

Decision cues:

Use raw YAML to learn and for tiny one-offs.
Use Helm once you need env variants, upgrades, teams, reuse.

Diagrams

Architecture (request → Service → Pods)

Control Plane Path (sequence)

Hands-on Mini-Lab (20–30 min)

Goal: Create a cluster, deploy “hello” with raw YAML, then with Helm; compare manifests.

1) Create a local cluster

kind create cluster --name k90
kubectl cluster-info
kubectl get nodes
kubectl config set-context --current --namespace=dev

(macOS: if bash errors, run in zsh or install newer bash with Homebrew.)

2) Raw YAML: Deployment + Service

Create k8s/hello.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels: { app: hello }
spec:
  replicas: 2
  selector: { matchLabels: { app: hello } }
  template:
    metadata: { labels: { app: hello } }
    spec:
      containers:
        - name: app
          image: nginx:1.25
          ports: [{ containerPort: 80 }]
---
apiVersion: v1
kind: Service
metadata:
  name: hello
  labels: { app: hello }
spec:
  type: ClusterIP
  selector: { app: hello }
  ports:
    - name: http
      port: 80
      targetPort: 80

Apply and verify:

kubectl apply -f k8s/hello.yaml
kubectl rollout status deploy/hello
kubectl get svc hello -o wide
kubectl get pods -l app=hello -o wide

Port-forward to test:

kubectl port-forward svc/hello 8080:80
# open http://localhost:8080

3) Helm-ify it

Create a chart and trim it down:

helm create hello-chart
# Keep only templates/deployment.yaml and templates/service.yaml; delete extras like hpa, serviceaccount, tests.

hello-chart/values.yaml (minimal):

replicaCount: 2
image:
  repository: nginx
  tag: "1.25"
  pullPolicy: IfNotPresent
service:
  type: ClusterIP
  port: 80
labels:
  app: hello

hello-chart/templates/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "hello-chart.fullname" . }}
  labels:
    {{- include "hello-chart.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Values.labels.app }}
  template:
    metadata:
      labels:
        app: {{ .Values.labels.app }}
        {{- include "hello-chart.labels" . | nindent 8 }}
    spec:
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: 80

hello-chart/templates/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: {{ include "hello-chart.fullname" . }}
  labels:
    {{- include "hello-chart.labels" . | nindent 4 }}
spec:
  type: {{ .Values.service.type }}
  selector:
    app: {{ .Values.labels.app }}
  ports:
    - name: http
      port: {{ .Values.service.port }}
      targetPort: 80

Render & compare with your raw YAML:

helm template hello ./hello-chart > rendered.yaml
diff -u k8s/hello.yaml rendered.yaml || true

Install and test:

helm install hello ./hello-chart -n dev --create-namespace
kubectl get all -l app=hello
kubectl port-forward svc/hello 8080:80

Upgrade and rollback:

# Bump replicas via values
helm upgrade hello ./hello-chart --set replicaCount=3
helm history hello
helm rollback hello 1   # back to first revision

Cheatsheet Table (Top 12)

Command	What it does
`kubectl config get-contexts`	List kubeconfig contexts; know where you’re pointed.
`kubectl config set-context --current --namespace=dev`	Set default namespace for current context.
`kubectl get pods -o wide`	Show Pods with node/IP; quick health snapshot.
`kubectl describe pod/<name>`	Deep dive into events, containers, reasons for failures.
`kubectl get events --sort-by=.lastTimestamp`	Recent cluster events for fast debugging.
`kubectl apply -f file.yaml`	Declaratively create/update resources.
`kubectl rollout status deploy/<name>`	Watch rollout until success/fail.
`kubectl logs deploy/<name> -f`	Stream app logs from all Pods in the Deployment.
`kubectl port-forward svc/<name> 8080:80`	Access ClusterIP services from localhost.
`helm template <rel> <chart>`	Render manifests locally (no cluster changes).
`helm install <rel> <chart> -f values.yaml`	Install a chart as a named release.
`helm upgrade --install <rel> <chart>`	Idempotent deploy; create or upgrade in one.

Pitfalls & Recovery

ImagePullBackOff / ErrImagePull: Repository or tag wrong, or no registry creds.
Fix: kubectl describe pod, verify image:; try docker pull locally; add imagePullSecret if private.
Pods stuck Pending: No schedulable nodes, resource requests too high, or PVC issues.
Fix: kubectl describe pod; check kubectl get nodes; reduce resources.requests; with minikube, minikube addons enable storage.
Service not routing: selector doesn’t match Pod labels or targetPort mismatch.
Fix: Compare spec.selector to pod.metadata.labels; align targetPort with containerPort.
Wrong context/namespace: Resources “missing.”
Fix: kubectl config current-context; kubectl get ns; set proper namespace.
Helm upgrade fails (immutable fields): Some fields (e.g., spec.clusterIP) can’t change in-place.
Fix: helm diff to see changes; for Services, preserve clusterIP; otherwise helm uninstall and re-install.
RBAC forbidden: In restricted clusters, applies fail.
Fix: Ask for right Role/RoleBinding; test with kubectl auth can-i.
Tabs/indentation in YAML: Parsing errors or ignored fields.
Fix: Convert tabs to spaces; validate with kubectl apply --dry-run=client -f.

Quick Bash (≥4) Scriptlets — Point‑wise Explanation

Below is a line‑by‑line breakdown of the helper script that creates or deletes a local kind cluster and sets a default namespace.

#!/usr/bin/env bash
set -euo pipefail

CLUSTER=${1:-k90}

case "${2:-up}" in
  up)
    kind create cluster --name "$CLUSTER"
    kubectl config set-context --current --namespace=dev
    ;;
  down)
    kind delete cluster --name "$CLUSTER"
    ;;
esac

What each line does

#!/usr/bin/env bash
Shebang. Asks the OS to execute this file with the first bash found in your PATH (portable across systems).
set -euo pipefail
Enables strict mode:

-e: exit immediately if any command exits with non‑zero status.
-u: error if using an unset variable (catch typos/assumptions).
-o pipefail: a pipeline fails if any command fails (not just the last).

CLUSTER=${1:-k90}
Positional argument \$1 is the cluster name; if omitted, default to k90.
Examples: ./cluster.sh ⇒ name k90; ./cluster.sh demo ⇒ name demo.
case "${2:-up}" in
Dispatches on the action provided in \$2; defaults to up if not given.
Usage examples:

./cluster.sh → up (default)
./cluster.sh demo → up on cluster demo
./cluster.sh demo down → down on cluster demo

up) block

kind create cluster --name "$CLUSTER" creates a Docker‑backed Kubernetes cluster named CLUSTER.
kubectl config set-context --current --namespace=dev sets the default namespace for the current kubeconfig context to dev, so you don’t need -n dev for every command.

down) block

kind delete cluster --name "$CLUSTER" removes the cluster and its Docker containers cleanly.

;; and esac ;; ends each case arm; esac closes the case statement.

Why this script is useful

Idempotent lifecycle: One command to bring the cluster up or tear it down.
Safer defaults: Strict mode prevents partial/hidden failures.
Less typing: Sets your working namespace so kubectl get pods “just works.”
Portable: env shebang finds the right bash; easily adapted for zsh.

Common tweaks you might add

Wait for node readiness:

  kubectl wait --for=condition=Ready nodes --all --timeout=120s

Install an ingress addon (minikube):

  minikube addons enable ingress

Switch context safely (guard rails):

  kubectl config current-context | grep -q kind- || {
    echo "Refusing to run outside a kind context" >&2; exit 1;
  }

Parameterize namespace:

  NS=${NS:-dev}
  kubectl config set-context --current --namespace="$NS"

macOS / Linux / Windows notes

macOS: If /bin/bash is 3.2, run with zsh (#!/bin/zsh) or brew install bash and use /usr/local/bin/bash (or /opt/homebrew/bin/bash on Apple Silicon).
Linux: Ensure your user can access Docker without sudo (usermod -aG docker $USER, then re‑login).
Windows: Run the script from WSL2 (Ubuntu) for the best experience with kind/minikube; ensure Docker Desktop has the WSL2 backend enabled.

Quick usage recap

# Bring up default cluster (k90) and set namespace dev
./cluster.sh

# Bring up a named cluster
./cluster.sh demo up

# Tear it down
./cluster.sh demo down

You’re ready. Save this page, keep the YAML/Helm snippets handy, and start iterating.

Wrap-up & Next Steps

You now have:

A cluster you can spin up/down quickly.
A clean Deployment + Service in raw YAML.
A minimal Helm chart with values and releases.
A mental model to reason about controllers and desired state.

Post 1 (coming up):

Ingress vs. NodePort vs. LoadBalancer with local ingress addons.
Rolling updates & health probes (readiness/liveness/startup).
Config & Secrets (env vars, mounted files, externalized values).
Resource requests/limits & HPA basics.
Helm strategies: values layering, env directories, helmfile preview.

Kubernetes & Helm Blog Series: A Journey from Zero to Production

cloud-sky-ops — Sat, 20 Sep 2025 21:52:39 +0000

Kubernetes and Helm have become the backbone of modern DevOps practices, powering everything from small startups to enterprise-scale production environments. But the learning curve can feel steep, especially if you’re new. That’s why I’ve designed this 10-post series (starting from 0 as mentioned in the title, so effectively 11) to guide you step by step—from absolute foundations to advanced, production-ready strategies—so that both beginners and seasoned engineers can level up.

Why This Series?

For beginners: you’ll get clear definitions, simple labs, and practical examples that build confidence quickly.
For advanced engineers: you’ll find deeper dives, edge-case discussions, and proven practices used in real-world production clusters.
For everyone: each post has hands-on labs, cheat sheets, and a wrap-up that connects naturally to the next topic.

Series Structure

**Post 0 — Foundations: Zero to Base Setup

Get comfortable with the mental model of Kubernetes, YAML hygiene, kubectl basics, local clusters (kind/minikube), and Helm chart structure.

Post 1 — Multi-Tenancy & Security Baseline

Learn how Namespaces, ResourceQuotas, NetworkPolicies, and Pod Security Admission set the foundation for safe, shared clusters.

Post 2 — Reliability by Design: Probes, PDBs & Topology Spread

Design applications that survive rollouts, disruptions, and node failures with probes, PodDisruptionBudgets, and spread constraints.

Post 3 — Upgrades & Feature Gates: Safe Cadence

Master upgrade strategies, manage version skew, and safely evaluate new features.

Post 4 — Smart Scaling & Cost Control: HPA, KEDA & Karpenter

Balance elasticity and efficiency using Horizontal Pod Autoscaling, event-driven scaling, Cluster Autoscaler, and Karpenter.

Post 5 — Modern Traffic Management with Gateway API

Move beyond Ingress with the Gateway API: flexible routing, retries, mirroring, and multi-tenant gateways.

Post 6 — Helm Fundamentals Done Right

Build clean, reusable charts with sane defaults, values schema validation, library charts, and overlays.

Post 7 — Helm in CI/CD: Lint, Tests, Diff & Supply Chain Security

Treat charts as code: enforce linting, add unit tests, integrate diffs, push to OCI registries, and secure with provenance.

Post 8 — Progressive Delivery & GitOps

Ship confidently with Blue/Green and Canary patterns, GitOps controllers like Argo CD and Flux, and drift detection.

Post 9 — Operator’s Toolkit: Debugging & Power Moves

Sharpen your day-to-day: ephemeral debug containers, kubectl power tips, and Helm Day-2 operations.

Post 10 — What’s Next: Sidecars, eBPF, AI Gateways & Supply Chain Maturity

Explore the trends shaping Kubernetes: stable sidecars, eBPF-powered networking, AI routing, and SLSA supply-chain practices.

How to Follow Along

Start at Post 0 if you’re new, or jump in at any point if you’re experienced.
Each post builds on the previous, but they also stand alone as practical guides.
Expect plenty of copy-pasteable commands, diagrams to cement understanding, and mini-labs that run in less than 30 minutes.

Bottom line: by the end of this series, you’ll not only understand Kubernetes and Helm—you’ll be confident running, scaling, securing, and modernizing clusters in production. Start's September 23rd, 2025

Bash Solutions: NUL and process substitutions

cloud-sky-ops — Tue, 09 Sep 2025 20:52:13 +0000

As a DevOps engineer, I spend a lot of time trying to make CI/CD pipelines bulletproof. GitHub Actions is powerful, but for some cases you need to throw in some custom automation logic to bring your use case to completetion. Recently, I ran into one of those this should have been simple problems that ended up forcing me deep into Bash territory.

In this post, I’ll share how I solved it, the tricks I learned along the way, and why those tricks are now staples in my automation toolkit.

The Problem Scenario

I needed to build a GitHub Workflow that:

Loops through all the changed files in a PR.
Runs a service-specific script based on which directories were touched.
Ensures scripts are run only once per unique service.

Sounds easy, right? But here’s what I ran into:

Filenames and directories contained spaces, dashes, and special characters, which caused word-splitting nightmares.
Multiple directories had overlapping file changes, and I needed to deduplicate them cleanly before execution.

Naive attempts with for file in $(git diff …) blew up instantly when spaces or weird characters showed up. Deduplication with sort | uniq worked halfway, but it wasn’t reliable when integrated into the workflow.

I also thought of introducing path filters in the workflow trigger to target specific directories.

on:
  pull_request:
    paths:
      - "services/service-a/**"

But it would require updates to the workflow whenever new directories are added, so it didn't seem scalable.

I needed a bash-powered solution that was safe, idempotent, and CI-friendly.

Step-by-Step Troubleshooting & Resolution

Step 1: Getting the changed files

Inside the workflow, I grabbed the changed files with:

git diff --name-only origin/main...HEAD > changed_files.txt

That gave me a newline-delimited list. Great — until filenames with spaces appeared.

Step 2: Handling spaces and special characters

I realized I couldn’t rely on newlines alone. I switched to NUL-terminated strings, which led to the first crucial trick.

Step 3: Deduplicating directories

Next, I needed to strip each path down to its top-level directory, then deduplicate. A bit of awk and sort -u magic did the trick.

Step 4: Iterating safely in CI

Finally, I wrote a loop to iterate over each unique directory and run the associated script. The key was keeping it robust against any edge cases.

Advanced Bash Trick #1: NUL-Separated Processing

Before jumping into the fix, let’s talk about what this means.

Normally, when you run a command like git diff --name-only, it prints filenames separated by newlines (\n). That works most of the time — until you run into filenames with spaces, quotes, tabs, or even emojis. Bash sees those spaces and thinks they’re “separators” between different items, which can break loops in subtle ways.

That’s where NUL-separated processing comes in.

A NUL (\0) character is a special invisible character that represents “nothing.” It’s guaranteed never to appear in a valid filename on Unix-like systems.
If we use NULs instead of newlines as separators, Bash can handle structured data safely, no matter how strange the entries look.
Commands like git diff, xargs, and sort all have flags (-z or -0) to enable this mode.

Think of it like replacing a fragile delimiter (newline) with an unbreakable one (NUL).

Here’s the heart of the fix:

git diff --name-only -z origin/main...HEAD | \
  xargs -0 -n1 dirname | \
  awk -F/ '{print $1}' | sort -u -z > changed_dirs.txt

Why this works

-z makes git diff output NUL (\0) separated strings instead of newline-separated ones.
xargs -0 ensures even entries with spaces, quotes, or emojis (!) are handled safely.
By the time we hit sort -u -z, we have a clean, deduplicated list of changed top-level directories.

Real-world use cases

NUL-separated processing is useful well beyond CI workflows:

Iterating over structured data with spaces/special chars: e.g., parsing JSON keys piped into Bash.

  jq -r '.keys[]' file.json | tr '\n' '\0' | xargs -0 -n1 echo

Handling user input or filenames from external sources: bulk renaming photos with spaces in names.
Safely processing logs or configs: when entries might contain whitespace or unusual delimiters.

Advanced Bash Trick #2: Process Substitution

Let’s pause for a second — because this one looks weird the first time you see it.

In Bash, when you want to feed the output of one command into another, you normally use a pipe (|) or redirect to a temporary file. But sometimes, you need the output to “pretend” to be a file itself.

That’s where process substitution comes in.

It uses the syntax < <(command) to tell Bash: “Run this command, and treat its output as if it were a file I can read from.”
This avoids writing temp files to disk.
It makes loops and comparisons cleaner.

Think of it as creating a fake file on the fly, backed by a running process.

Here’s how I used it in the workflow loop:

while IFS= read -r -d '' dir; do
  echo "Running checks for $dir"
  ./scripts/run-checks.sh "$dir"
done < <(git diff --name-only -z origin/main...HEAD \
          | xargs -0 -n1 dirname \
          | awk -F/ '{print $1}' \
          | sort -u -z)

Why this works

< <(...) is process substitution, which feeds the output of a command directly into a loop as if it were a file.
No need for temporary files (changed_dirs.txt).
Works seamlessly inside GitHub Actions runners.

Real-world use cases

Process substitution shines in many contexts:

Streaming structured data directly into loops: e.g., looping through a filtered list of Kubernetes resources.

  while read pod; do kubectl logs "$pod"; done < <(kubectl get pods -o name)

On-the-fly comparisons:

  diff <(ls dir1) <(ls dir2)

Data processing pipelines: feeding transformed logs into analysis scripts without creating temp files.
Parallel workflows: redirecting different process outputs into the same loop for consolidated handling.

Final Workflow Snippet

Here’s the cleaned-up version that now powers my workflow:

jobs:
  detect-and-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run service-specific checks
        run: |
          while IFS= read -r -d '' dir; do
            echo "Running checks for $dir"
            ./scripts/run-checks.sh "$dir"
          done < <(git diff --name-only -z origin/main...HEAD \
                    | xargs -0 -n1 dirname \
                    | awk -F/ '{print $1}' \
                    | sort -u -z)

Key Takeaways

Don’t trust whitespace in CI — structured data can (and will) break naive loops. Always handle it safely.
NUL-separated processing (-z, xargs -0, read -d '') is your best friend for robustness.
Process substitution makes scripts cleaner, avoids temp files, and opens doors for powerful inline comparisons.
While GitHub’s path filters look appealing, they fall short in complex repos with multiple services. A Bash-driven approach offers the flexibility and resilience needed for real-world CI/CD.

Since adopting these patterns, I’ve reused them in multiple workflows — from linting to selective deployments — and they’ve held up every time.

Have you ever hit whitespace or multi-service workflow issues in GitHub Actions? I’d love to hear how you solved them in the comments! Thank you for your time.

Automating File Sync Across Repositories with a GitHub Action

cloud-sky-ops — Sun, 20 Apr 2025 14:36:19 +0000

Problem statement and introduction:

In many mid to large scale organisations software solutions are created using micro-services architecture. There you might encounter a bunch of repeated implementations when it comes to CI/CD processes. Managing consistency across multiple repositories can quickly become cumbersome. Whether it’s standardizing CI workflows, updating documentation, or synchronizing Dockerfiles, keeping everything in sync is crucial but often manual. In short, a bunch of repeated Ctrl c + Ctrl v.

That's where the newly published Sync Files to Multiple Repos via API action I developed comes into play. Licensed under MIT, this action is now available in GitHub Marketplace for free use. Built using Python and GitHub’s REST API, this action makes syncing files and directories across many repositories clean, efficient, and fully automated.

Sync Files to Multiple Repos via API
Source Code in GitHub

What this GitHub Action does?

This action allows you to synchronize specific directories or all directories in a repo across multiple GitHub repositories, without needing to clone or open PRs manually. It uses GitHub's REST API to:

Fetch default branches of target repositories
Create or update files in said repositories
Create new branches for changes (optional)
Open pull requests automatically (optional)

Key components of the action:

Here are five key components of the sync-files-multi-repo action:

1. Using GitHub API

I used GitHub API for file updates, fetching repository data and branch related operations in this action. This helped me learn about REST APIs in python and also gave me an alternative to the traditional approach of relying on multiple git commands.

2. Secure and Scoped Authentication

The action uses a Personal Access Token (PAT) injected in the workflow via GitHub secrets. it's important to highlight, when syncing .github/workflows, it requires the workflow scope. This showcases integration of secure and scoped authentication in GitHub Actions.

3. Full Directory Sync Support

Supports copying full directories from the source repository to destination repositories. It maintains subdirectory structure and makes configuration simple with just two inputs:

copy-from-directory
copy-to-directory

If not provided, it falls back to syncing the root directory in respective repos.

4. Flexible Sync Modes

You can choose to commit changes directly or open pull requests:

create-pull-request: true opens a PR on a new branch
false commits directly to the default branch

Great for balancing speedy automation against PR review compliances.

5. Simple Setup for Broad Adoption

Target repositories are listed in a plain sync-repos.txt file. The action provides sensible defaults, enabling even non-DevOps contributors to use it effectively. In future release the .txt file may be replaced with a more structured file like JSON or YAML.

Real-World Use Cases

Use Case	Description
Sync shared documentation	Keep `README.md`, `LICENSE`, etc., consistent across repos
Standardize CI/CD	Deploy the same GitHub workflows to all microservices
Update base Dockerfiles	Roll out updated base image configs across repos

Behind the Scenes: How It Works

The action is implemented in pure Python using standard libraries like os, base64, and requests. Some implementation highlights include:

Dynamic default branch detection:

  def get_default_branch(target_repo):
      response = requests.get(f"https://api.github.com/repos/{target_repo}", headers=HEADERS)
      return response.json().get("default_branch")

File uploads with Base64 encoding:

  response = requests.put(url, json=payload, headers=HEADERS)

Automated PR creation:

  response = requests.post(f"https://api.github.com/repos/{repo}/pulls", json=payload, headers=HEADERS)

Architecture Summary

Component	Details
Language	Python 3
API	GitHub REST v3
Auth	PAT via `Authorization` header
Branching	Timestamped feature branches

Conclusion

The sync-files-multi-repo GitHub Action is a robust, easy to use automation for cross-repository updates. As we discussed, it offers real value to engineering teams in multiple use cases.

If you’ve been managing updates across multiple repositories manually, give this action a spin and let me know your thoughts in the comments. Follow me for more such automations.

X Profile
LinkedIn

Thank You.

3 hacks to easily create and manage Argo CD applications

cloud-sky-ops — Wed, 09 Apr 2025 10:49:37 +0000

Introduction:

Argo CD is a powerful GitOps tool that helps you declaratively manage your Kubernetes applications. But let’s be real, getting everything set up efficiently can sometimes feel like assembling IKEA furniture without a manual. 😅

If you want to automate deployments while keeping your Kubernetes configurations clean and versioned, you’re in the right place! Here are three game-changing hacks that I used in my implementation of Argo CD for my DevOps portfolio. These strategies should make your Argo CD experience seamless, scalable, and smooth. Let's dive right in.

Hack 1: Use Helm to Manage Kubernetes Templates Like a Pro

Why Helm?

Managing raw Kubernetes manifests can get messy, real fast. Helm acts as the package manager for Kubernetes, helping you:

Template and reuse configurations instead of duplicating YAML files.
Use parameterized values for environment-specific configurations.
Maintain a clear version history for deployments.

With Helm, you can define once, reuse everywhere. Need to deploy a new version? Just update the values.yaml. No more editing multiple YAML files manually!

Quick example on how it works:

Instead of manually writing multiple deployment files, Helm allows you to define reusable Chart templates inside the helm-charts directory:

# node-app-helm/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{ .Release.Name }}-app
  template:
    metadata:
      labels:
        app: {{ .Release.Name }}-app
    spec:
      containers:
        - name: app
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          ports:
            - containerPort: 3000

If you need more info on what Helm has in store, you can read my introductory blog on Helm, it also covers the changes I made in my Argo CD project. Links to both are mentioned below:

Cruise-on-Kubernetes-with-Helm
Project Repository

Hack 2: Automate Helm Versioning in GitHub Actions using `sed`

The Problem

Argo CD tracks the main branch to determine the latest version of your app. But how do you automate updating the desired values in values.yaml and Chart.yaml every time a new Docker image is built?

The `sed` Solution

In the GitHub Actions workflow (build-and-run.yaml), we dynamically update values.yaml and Chart.yaml using sed. Here's how it works:

- name: Update Helm chart with new image tag
  run: |
    sed -i -E "s/(tag: \")([a-z0-9]*)(\")/tag: \"${{ github.sha }}\"/"  helm-chart-directory/values.yaml
    sed -i -E "s/(appVersion: \")([a-z0-9.]*)(\")/appVersion: \"${{ github.sha }}\"/" helm-chart-directory/Chart.yaml

Breaking Down the Command 🔍

sed -i: Edits the file in place.
[a-z0-9]: Matches the existing SHA-like string.
*: The * makes the alphanumeric SHA-like string optional, allowing for both SHA-like and blank values.
s|tag: \"[a-z0-9]*\"|tag: "${{ github.sha }}"|: Replaces the existing tag with the new GitHub SHA.
s|version: \"[0-9.]*\"|version: "1.0.${{ github.run_number }}"|: Updates Chart.yaml with a new version similarly.

Goes without saying this is not the only way to achieve this change. But the point I wanna drive home is automate this portion so there's no manual dependency to merge a pull request to update main branch.

Other alternatives:

Use yq commands to get the desired changes since you'd be dealing with YAML files. In my case the commands would look like:

IMAGE_TAG=${{ github.sha }} yq -i eval '.image.tag=env(IMAGE_TAG)' helm-chart-directory/values.yaml
IMAGE_TAG=${{ github.sha }} yq -i eval '.appVersion=env(IMAGE_TAG)' helm-chart-directory/Chart.yaml

Set the targetRevision field in the application.yaml to a desired branch where image tag is already available. This can be set using "--revision" flag in argocd CLI.

Example: This can work in a setup where you pre-test your image in a controlled environment before deploying it to Argo CD. Say your current built image is number n but the current deployed image is n-1. This way you can store the image on a dedicated branch which has a static name, something like ready-to-deploy, and update the tag manually/programmatically post testing.

You can manage a centralized repository for Argo CD to monitor. For this, you'd need to copy/sync helm manifests from one repository to another.
Th above approach is good for large micro-services architecture to manage helm charts in one place. You can try out this GitHub Marketplace Action I created to sync files between repositories.

Sync Files Across Repositories GitHub Action

I'm working on doing the same for the next release of my project and will share my strategies and implementation in coming blogs. Do follow along to grab advanced learning opportunity from this venture.

Hack 3: Use Argo CD CLI for Helm Manifest Rendering & Deployment

Argo CD CLI

Along with powerful web UI capabilities of Argo CD, the CLI has a broad area of use cases as well. In my implementation it helped easily automate the creation of the Argocd Application resource instead of creating an application.yaml file, we'll now explore the below commands regarding the same:

✅ Create applications programmatically
✅ Sync applications on-demand

How It Works

Post install verification script

Once the Kubernetes cluster (Minikube) is running inside GitHub Actions, and the installation for Argo CD is completed, the workflow verifies if all the pods created in the argocd namespace are in a 'Running' state, if not it'll wait and re-check in 30 seconds:

- name: Check if all pods are in "Running" state
  run: |
    while true; do
      echo -e "Printing current status of Argo CD Pods:\n"
      kubectl get pods -n argocd
      POD_STATUSES=$(kubectl get pods -n argocd -o=jsonpath='{.items[*].status.phase}')
      echo "POD_STATUSES: $POD_STATUSES"
      if [[ $(echo "$POD_STATUSES" | tr ' ' '\n' | sort -u) == "Running" ]]; then
          echo "All pods are in Running state."
          break
      else
          echo "Waiting for all pods to be in Running state...Re-checking in 30 seconds"
          sleep 30
      fi
    done

Log in to Argo CD after forwarding the port using kubectl:

Using the kubectl port-forward command shown below, the local Argo CD server is forwarded to port 8080 in the runner machine.
The & at the end of the command ensures the process is run in the background, AKA detached state, so the execution in the terminal is not halted.
The STD_OUT logs are re-directed to a file port-forward.log and this can be used for troubleshooting if something goes south.
yes command will continuously output y in the terminal which ensures the login commands gets a 'y' when prompted for (Yes/No). This can also be achieved by echo "y" instead of yes
The username is admin and password for first login is also fetched using argocd CLI.

- name: Login to ArgoCD
  run: |
    kubectl port-forward svc/argocd-server 8080:443 -n argocd > port-forward.log 2>&1 &
    sleep 3
    if ! grep -q "Forwarding from" port-forward.log; then
        echo "Port forwarding failed, check logs."
        cat port-forward.log
        exit 1
    fi    
    yes | argocd login localhost:8080 --username admin --password $(argocd admin initial-password -n argocd | head -n 1)

Create an Argo CD application using Helm:

- name: Create ArgoCD Application
  run: |
    argocd app create <application-name> \
      --repo https://github.com/<Organisation>/<repo-name>.git \
      --path <relative-path-of-kubernetes-manifests> \
      --dest-server <server-for-kubernetes-cluster> \
      --dest-namespace <desired-namespace> \
      --sync-policy automated

After filling the values for my project repository the command looked like:

- name: Create ArgoCD Application
  run: |
    argocd app create node-app \
      --repo https://github.com/cloud-sky-ops/node-app.git \
      --path node-app-helm \
      --dest-server https://kubernetes.default.svc \
      --dest-namespace default \
      --sync-policy automated

Sync the application to deploy the latest changes:

This step isn't required when --sync-policy is set to automated, however, it's a good to know because you might wanna disable auto-sync in production environments.

- name: Sync ArgoCD Application
  run: |
    argocd app sync node-app

Wrapping Up:

By combining Helm templating, GitHub Actions automation, and the Argo CD CLI, we’ve just built a fully automated Kubernetes deployment pipeline. Here’s a quick recap:

✅ Use Helm to template Kubernetes manifests.
✅ Automate versioning updates with sed in GitHub Actions.
✅ Use the Argo CD CLI to create, manage, and sync applications.

These three hacks make Kubernetes deployments faster, more predictable, and entirely Git-driven. 🚀

Want to see all this in action? Check out the node-app repository and start building your own automated GitOps workflow today!

Made it till the end? Drop your thoughts, suggestions, and queries in the comment section down below. Thank You!!

DEV Community: cloud-sky-ops

Post 6/10 — Helm Fundamentals Done Right: Chart Architecture, Values Schema, and Reuse

1) Executive Summary

2) Prereqs

3) Concepts

a) Chart layout & sane defaults

b) values.schema.json validation

c) Helpers/partials & library charts

d) Environment overlays strategy

4) Mini-Lab — From Raw Manifests to Chart

5) Cheatsheet

6) Pitfalls

7) Wrap-up — Next: Shipping Charts with Confidence: Lint, Tests, Diff, OCI & Provenance (Post 7)

Post 5/10 — From Ingress to Gateway API: Safer, Smarter Traffic Control

1) Executive Summary

2) Prereqs

3) Concepts

Ingress vs Gateway mental model

Gateway + HTTPRoute basics

TLS, Retries, Timeouts, Header Matching

Canary / Mirroring / Multi-tenant Gateways

4) Mini-Lab — Replace Ingress with Gateway/HTTPRoute + Canary + TLS

5) Cheatsheet

6) Pitfalls

7) Wrap-Up

Post 4/10 — Smart Scaling & Cost Control: HPA/KEDA + Cluster Autoscaler vs Karpenter

1️⃣ Executive Summary

2️⃣ Prereqs

3️⃣ Concepts

Requests / Limits & Rightsizing

HPA (Horizontal Pod Autoscaler)

🔁 KEDA ( Kubernetes Event-Driven Autoscaler )

Cluster Autoscaler (CA) Overview

Karpenter

4️⃣ Mini-Lab — HPA + CA vs Karpenter

5️⃣ Cheatsheet

6️⃣ Pitfalls

7️⃣ Wrap-Up

Diagram 1: Demand vs Capacity Timeline

Diagram 2: Provisioning Decision Tree

Post 3/10 — Safe Cadence: Kubernetes Upgrades, Version Skew, and Feature Gates

Executive Summary

Prereqs

Concepts

1️⃣ Support Windows & Version Skew

2️⃣ Upgrade Order: Control Plane → Nodes

3️⃣ Deprecation Checklist & API Migration

4️⃣ Feature Gate Evaluation in Pre-Prod

🧪 Mini-Lab — Plan a Minor Upgrade

Cheatsheet

Pitfalls

Diagram — Upgrade Workflow

✅ Wrap-Up

Post 2/10 — Reliability by Design: Probes, PodDisruptionBudgets, and Topology Spread Constraints

Executive Summary

Prereqs

Concepts

A. Probes: Liveness / Readiness / Startup

B. PodDisruptionBudget (PDB)

C. TopologySpreadConstraints

D. Rollout Strategies (Surge / maxUnavailable)

Mini-Lab

Step 1: Namespace + sample deployment

Step 2: Add probes, PDB, and topology constraints

Step 3: Simulate node drain and verify behavior

Bonus: Trigger a rollout:

Cheat Sheet Table

Pitfalls & Gotchas

Wrap-up & Bridge to Post 3

Diagram 1 — Rolling update timeline

Diagram 2 — Zone spread visualization

Mastering GitHub Actions: Insights and pitfalls of "if conditions"

The Essential if Conditions in GitHub Actions

When if: ${{ !cancelled() }} Bit Me Back

The Problem

DAG Construction & Pruning (root cause)

Example

Practical Example in GitHub Workflows

DAG in Practice: Why It Matters

The Fix

b) `values.schema.json` validation

The Essential `if` Conditions in GitHub Actions

When `if: ${{ !cancelled() }}` Bit Me Back

3) `kubectl` Basics