DEV Community: Naveed Ahmed

Apache Kafka Tutorial: KRaft, Docker & Consumer Groups

Naveed Ahmed — Wed, 04 Nov 2026 13:49:40 +0000

Originally published on Naveed Ahmed's Engineering Blog.

Apache Kafka is often described as a messaging system.

That's partly true, but understanding it as a distributed event log gives you a much clearer picture of how it works.

Producers write records to partitioned topics. Brokers store those records, and consumers read them at their own pace.

Unlike a traditional work queue, Kafka can retain events after they've been consumed, allowing different applications to process or replay them independently.

Let's walk through the key concepts and a small hands-on example.

1. How Apache Kafka Works

A simplified architecture looks like this:

Producer
   |
   v
Kafka Topic
   |
   +-- Partition 0
   +-- Partition 1
   +-- Partition 2
   |
   v
Consumer Group

Each partition is an ordered sequence of records.

Kafka uses offsets to identify record positions within a partition.

If you have three partitions, you can have up to three actively assigned consumers within one consumer group for that topic.

Adding more consumers than partitions doesn't create additional partition-processing parallelism.

2. What Is KRaft?

Modern Apache Kafka uses KRaft to manage cluster metadata without an external ZooKeeper deployment.

KRaft controllers form a quorum responsible for metadata decisions, while brokers handle client requests and partition storage.

This separation matters when you're investigating broker failures, partition leadership changes, or controller health.

3. Start Kafka with Docker

For a quick local test, you can start Kafka with the official Docker image:

docker run -d \
  --name kafka-local \
  -p 127.0.0.1:9092:9092 \
  apache/kafka:4.4.0

This example uses the image's default single-node configuration for development.

Create a topic:

docker exec kafka-local \
  /opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic orders.events \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

Inspect it:

docker exec kafka-local \
  /opt/kafka/bin/kafka-topics.sh \
  --describe \
  --topic orders.events \
  --bootstrap-server localhost:9092

This gives you a simple environment for learning partitions and offsets.

It isn't a highly available Kafka cluster.

4. Produce and Consume Events

Write a message:

echo '{"orderId":101,"status":"created"}' | \
docker exec -i kafka-local \
  /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders.events \
  --bootstrap-server localhost:9092

Read messages:

docker exec kafka-local \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --topic orders.events \
  --from-beginning \
  --timeout-ms 10000 \
  --bootstrap-server localhost:9092

You now have a local Kafka workflow using a producer, topic, and consumer.

5. Production Kafka Is a Different Problem

Running one broker on a laptop is easy compared with managing a replicated cluster.

In production, pay attention to:

Replication factor and minimum in-sync replicas
Broker disk capacity and I/O
Consumer group lag
Partition leadership
Authentication and authorization
KRaft controller quorum health
Recovery and upgrade procedures

For Kubernetes, operators such as Strimzi can help manage cluster lifecycle, but they don't remove the need for careful storage and capacity planning.

6. Investigating Consumer Lag

Consumer lag tells you how far a consumer group is behind the latest available partition offsets.

Use:

docker exec kafka-local \
  /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group inventory-service

The group must exist and have committed offsets for meaningful output.

If lag is increasing, check whether event production exceeds processing capacity.

Also investigate slow downstream dependencies, consumer exceptions, frequent rebalances, and partition counts.

Avoid resetting offsets without understanding the risk of skipping or replaying records.

Read the Complete Apache Kafka Guide

The original article goes deeper into Kafka storage internals, KRaft, Docker Compose, Kafka vs RabbitMQ, Spring Boot, Strimzi, and production runbooks.

Read Apache Kafka Architecture & Hands-On Production Guide

Final Thoughts

Kafka becomes much easier to operate once you understand how partitions, offsets, replication, and consumer groups work together.

The challenge isn't just sending messages. It's making the streaming platform reliable when workloads grow and failures happen.

What's the most difficult Kafka problem you've investigated in production?

Written by Naveed Ahmed — DevOps & Platform Engineering

Engineering Blog | GitHub

GitHub Actions OIDC AWS: Secure CI/CD Without Access Keys

Naveed Ahmed — Tue, 03 Nov 2026 08:03:20 +0000

Originally published on Naveed Ahmed's Engineering Blog.

Still storing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in GitHub Actions secrets?

It works, but it also means managing long-lived credentials, rotation, and the risk of those keys being exposed.

For workflows deploying to AWS, there's a better option: OpenID Connect (OIDC).

GitHub Actions can use OIDC federation to request temporary AWS credentials without storing permanent AWS access keys.

Here's how it works.

1. Understanding GitHub Actions OIDC

The authentication flow looks like this:

GitHub Actions
      |
      | Request signed OIDC JWT
      v
GitHub OIDC Provider
      |
      | JWT
      v
AWS STS
      |
      | Validate IAM role trust policy
      v
Temporary AWS Credentials
      |
      v
Deploy to AWS

AWS uses AssumeRoleWithWebIdentity to exchange the signed token for temporary credentials after validating the issuer, audience, and allowed subject.

2. Register the GitHub OIDC Provider in AWS

Configure an IAM OIDC identity provider with:

Provider URL:
https://token.actions.githubusercontent.com

Audience:
sts.amazonaws.com

You can configure it in the AWS IAM console, AWS CLI, or Terraform.

For production, keeping identity infrastructure in Terraform helps make the configuration reviewable.

3. Configure a Restricted IAM Trust Policy

This is the most important step.

A trust policy should identify the GitHub repository and deployment context that may assume the role.

For example, a branch-scoped policy condition can look like:

{
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:YOUR_ORG/YOUR_REPO:ref:refs/heads/main"
  }
}

This is the condition portion of a trust policy, not the complete policy document.

For repositories using GitHub's newer immutable OIDC subject format, adjust the value to match the actual token.

Avoid broad wildcards that unintentionally authorize additional repositories or workflows.

Also restrict the permissions attached to the role. A tightly scoped trust policy won't help much if the role has unnecessary administrator access.

4. Configure GitHub Actions

The workflow needs permission to request an OIDC token.

name: AWS OIDC Authentication

on:
  workflow_dispatch:

permissions:
  id-token: write
  contents: read

jobs:
  verify-aws:
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-oidc-role
          aws-region: us-east-1

      - name: Check caller identity
        run: aws sts get-caller-identity

Replace the example AWS account ID and IAM role.

For production, pin third-party and security-sensitive actions to reviewed commit SHAs.

The important point: id-token: write authorizes requesting an OIDC token. It does not grant AWS permissions by itself.

AWS evaluates the role trust policy before allowing role assumption.

5. Common OIDC Authentication Problems

If you encounter AccessDenied or a role-assumption failure, check:

The IAM identity provider exists in the correct AWS account.
The token audience matches the trust policy.
The subject claim matches the repository and deployment context.
The job has id-token: write.
The role ARN is correct.
The workflow is running from an allowed branch or protected environment.

If you use GitHub Environments, remember that the OIDC subject format differs from a branch-scoped subject.

6. Production Security Checklist

For a safer setup:

Restrict IAM trust to authorized repository identities.
Use protected branches and deployment environments.
Grant only necessary AWS permissions.
Keep development and production roles separate.
Review third-party GitHub Actions.
Monitor role assumption and AWS API activity through CloudTrail.
Avoid exposing temporary credentials in workflow logs.

OIDC removes the need to store permanent AWS keys for this authentication flow. It doesn't eliminate the need for secure workflow code and IAM design.

Read the Complete Tutorial

My full guide includes detailed OIDC architecture, AWS CLI and Terraform configuration, trust policy examples, GitHub Actions YAML, and troubleshooting.

GitHub Actions OIDC AWS: Full Secretless CI/CD Guide

Final Thoughts

Replacing static AWS keys with temporary credentials is a useful security improvement.

But the real protection comes from combining OIDC with strict IAM trust policies, least privilege, and protected deployment workflows.

Have you implemented GitHub Actions OIDC for your AWS deployments?

Written by Naveed Ahmed — DevOps & Platform Engineering

Engineering Blog | Kubernetes Mastery | GitHub

Local LLM for DevOps: Ollama, vLLM & Kubernetes Tutorial

Naveed Ahmed — Tue, 03 Nov 2026 06:57:06 +0000

Originally published on Naveed Ahmed's Engineering Blog.

What if you could analyze Kubernetes failures and review Terraform configurations using an AI model running on your own hardware?

That's one of the most useful applications of local LLMs for DevOps.

Instead of sending every prompt to a hosted model API, you can run an open-weight model inside a controlled workstation or internal network.

Two tools worth understanding are Ollama and vLLM.

1. Why Run an LLM Locally?

Local inference can help teams:

Keep more operational data within their own environment.
Work without relying on an external inference service.
Build internal tools for logs, infrastructure code, and runbooks.
Control model selection, deployment, and inference infrastructure.
Evaluate whether fixed infrastructure costs suit their workloads.

However, local deployment isn't automatically cheaper, faster, or more secure.

You still need to account for compute resources, model quality, access controls, and operational maintenance.

2. Ollama vs vLLM

Feature	Ollama	vLLM
Primary use	Local development and simple inference services	Higher-throughput model serving
Setup	Straightforward	More infrastructure planning
Common environment	Laptop, workstation, private server	GPU-backed servers and Kubernetes
API	Local HTTP API	OpenAI-compatible serving API
Best starting point	Individual DevOps workflows	Shared team inference service

Both can be useful, depending on the workload.

3. Run Ollama with Docker

For a CPU-based local test:

docker run -d \
  --name ollama \
  -p 127.0.0.1:11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama

Pull a coding-focused model:

docker exec ollama ollama pull qwen2.5-coder:7b

Then query the local API:

curl http://127.0.0.1:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:7b",
    "prompt": "Explain Kubernetes CrashLoopBackOff and suggest diagnostic commands.",
    "stream": false
  }'

The model needs to be downloaded before offline use. The first pull requires network access unless the model files are already available locally.

For GPU acceleration, follow the Docker and Ollama documentation for your hardware.

4. Local Kubernetes Incident Triage

A useful starting point is a read-only assistant that summarizes Pod failures.

Collect the evidence:

kubectl describe pod my-app-123 -n staging

kubectl logs my-app-123 \
  -n staging \
  --previous \
  --tail=100

After checking the output for credentials or sensitive information, send relevant excerpts to your local model.

Ask it to identify:

Likely failure causes
Relevant log lines
Missing evidence
Commands to verify the hypothesis
Recommended next steps

Treat the result as a troubleshooting suggestion, not a verified root cause.

5. Review Terraform Changes Privately

Another useful workflow is reviewing an infrastructure pull request.

For example:

terraform fmt -check
terraform validate
git diff -- '*.tf'

A local LLM can help explain the changes or identify suspicious patterns.

But don't rely on an LLM as your only validator.

Use deterministic checks, infrastructure policies, security scanning, and human review before merging or applying changes.

6. When to Use vLLM

Ollama is a convenient starting point, but a platform team may need a centralized inference service for many applications.

vLLM supports high-throughput model serving and an OpenAI-compatible API.

For Kubernetes, consider:

GPU node allocation and model memory
Resource limits and scheduling
Authenticated ingress or an internal API gateway
Monitoring and request quotas
Model versioning and persistence
Network isolation and secrets handling

Do not expose an inference endpoint publicly without an appropriate security boundary.

7. Production Safeguards

Local inference still needs security controls.

Use read-only Kubernetes identities where practical, sanitize logs, authenticate model endpoints, and isolate inference workloads.

Review AI-generated changes before deploying them.

A useful first milestone is an assistant that explains failures and proposes checks, not one that autonomously modifies production.

Read the Complete Local LLM for DevOps Guide

My full article covers model selection, Ollama Docker setup, vLLM on Kubernetes, low-resource hardware, and practical SRE playbooks.

Read Local LLM for DevOps: Complete Guide

Final Thoughts

Local LLMs aren't a replacement for DevOps fundamentals.

They're another tool that can make investigation, documentation, and infrastructure review more convenient when deployed with appropriate controls.

Have you tried Ollama or vLLM for DevOps work?

Written by Naveed Ahmed — DevOps & Platform Engineering

Engineering Blog | Kubernetes Mastery | GitHub

Binance Hack 2026: 6 Zero Trust DevOps Security Lessons

Naveed Ahmed — Sat, 31 Oct 2026 07:55:12 +0000

Originally published on Naveed Ahmed's Engineering Blog.

What would happen if an attacker gained access to a DevOps engineer's laptop in your organization?

Would they find a few development tools?

Or would they inherit cloud sessions, Kubernetes access, SSH credentials, and routes into sensitive production systems?

That question became especially relevant following the October 2026 Binance security incident.

Binance reported an estimated $6.3 billion impact. Its preliminary findings indicated that a compromised DevOps engineer's laptop may have provided access to a service container used to submit unauthorized withdrawal transactions.

The company also said that the investigation was ongoing and that it had not found evidence of wallet private-key theft.

The exact technical attack chain hasn't been fully established.

Still, the incident offers an opportunity to review how developer endpoints interact with production infrastructure.

1. Treat Developer Workstations as Part of Your Production Attack Surface

DevOps engineers often use laptops to access:

Cloud management APIs
Kubernetes control planes
CI/CD pipelines
Internal dashboards
SSH and bastion systems
Production troubleshooting tools

A compromised endpoint may expose active sessions or allow misuse of already authenticated access.

The goal shouldn't be to assume laptops will never be compromised.

Instead, build systems that limit what a compromised endpoint can do.

2. Use Zero Trust Access

Traditional VPN access can provide broad network connectivity.

For sensitive production systems, consider identity-aware access with device posture checks and per-application authorization.

A successful connection to the corporate network shouldn't automatically establish trust.

Evaluate identity, device health, operation sensitivity, and session context before granting access.

3. Reduce Long-Lived Credentials

Persistent cloud access keys and reusable administrative credentials increase the consequences of workstation compromise.

Prefer:

Identity federation
Short-lived cloud credentials
Scoped Kubernetes access
Strong MFA for privileged operations
Timely session revocation and rotation

Temporary credentials reduce exposure, but active sessions and stolen tokens can still be misused during their validity period.

4. Review Kubernetes RBAC

Production roles should not receive privileged capabilities by default.

Pay particular attention to:

pods/exec
pods/portforward
secrets
rolebindings
clusterrolebindings
deployments

A read-only monitoring identity generally shouldn't be able to launch shells in production Pods or modify workload permissions.

For example:

kubectl auth can-i create pods/exec \
  --namespace production

Run these checks using the relevant identity or impersonation configuration.

Also review inherited permissions and other roles bound to that identity.

5. Use Network Segmentation

Even when an attacker gains access to one service, the rest of the environment shouldn't automatically become reachable.

Kubernetes NetworkPolicies and compatible CNIs can help restrict traffic between workloads.

For sensitive services, consider:

Default-deny policies
Explicit ingress and egress rules
Workload identity
Mutual TLS where appropriate
Controlled access to signing and transaction services

Network policies are useful, but they don't replace application-level authorization.

6. Require Independent Authorization for Sensitive Actions

A single compromised identity should not have unlimited authority to execute high-impact transactions or infrastructure changes.

Depending on the workload, use:

Separation of duties
Independent approvals
Transaction velocity limits
Threshold authorization or signing controls
Policy-enforced execution gateways

Multi-party approval must be enforced by separate trust boundaries, not simply by two steps within the same compromised process.

7. Monitor Runtime Behavior

Security doesn't end after authentication.

Monitor for unusual events such as unexpected shell processes, suspicious API usage, unauthorized workload changes, or abnormal network connections.

Runtime tools such as Falco can help detect certain behaviors in containerized environments.

Combine runtime telemetry with cloud audit logs, endpoint detection, and incident response processes.

A Practical Incident Response Starting Point

If a privileged DevOps workstation is suspected of compromise:

Isolate the endpoint through the approved response process.
Preserve relevant forensic evidence.
Revoke or expire exposed sessions and credentials.
Review cloud and Kubernetes audit activity.
Investigate unauthorized changes and lateral movement.
Restore access from a trusted device and validate recovery.

Avoid relying on credential rotation alone if the attacker may still control active sessions or workloads.

Read the Complete Binance Hack DevOps Analysis

My full article explores the reported incident, workstation-to-cloud attack paths, Zero Trust access, Kubernetes hardening, runtime detection, and a practical incident response playbook.

Read the Full Binance Hack DevOps Security Guide

Final Thoughts

The most useful security question isn't whether your company uses Kubernetes, a VPN, or MFA.

It's whether a compromised identity or endpoint can cross too many trust boundaries without another independent control stopping it.

If an engineer's workstation were compromised today, which production permissions would concern you most?

Written by Naveed Ahmed — DevOps & Platform Engineering

Engineering Blog | Kubernetes Mastery | DevOps Interview Scenarios

AI Agent Guardrails: Cost Limits, Security & Production Circuit Breakers

Naveed Ahmed — Fri, 30 Oct 2026 07:00:17 +0000

Originally published on Naveed Ahmed's Engineering Blog.

AI agents can run commands, call cloud APIs, create infrastructure, and execute workflows with very little human intervention.

That makes them useful for DevOps and platform engineering.

But it also creates a serious question:

What happens when an agent keeps executing the wrong operation?

A poorly controlled agent could repeatedly retry a failed command, create unnecessary resources, or attempt a production change outside its intended scope.

System prompts alone aren't enough to prevent these problems.

That's why production AI agents need enforceable guardrails.

1. Why Prompt-Based Rules Aren't Enough

Imagine giving an agent the instruction:

"Never make more than five API calls."

That instruction communicates intent, but it doesn't create an actual execution limit.

The agent may still request additional calls.

A reliable control must be enforced by the application or tool-execution layer.

For example:

AI Agent
    |
    v
Tool Request
    |
    v
Policy Validation
    |
    +-- Allowed → Execute
    |
    +-- Denied → Reject & Log

The same principle applies to spending limits, permissions, and destructive operations.

2. The Four Layers of AI Agent Guardrails

I prefer dividing agent controls into four layers.

Layer	Purpose
Execution bounds	Limit iterations, retries, and runtime
Token and API quotas	Control requests, tokens, and concurrency
Financial controls	Monitor and restrict cost-generating actions
IAM and approval	Restrict permissions and approve high-risk operations

These layers address different failure modes and work best together.

3. Implement a Simple Python Circuit Breaker

A basic execution wrapper can reject requests when an agent exceeds its tool-call budget.

from time import monotonic

class AgentGuardrail:
    def __init__(self, max_calls=10, timeout=60):
        self.max_calls = max_calls
        self.deadline = monotonic() + timeout
        self.calls = 0

    def authorize(self):
        if monotonic() >= self.deadline:
            raise RuntimeError("Agent deadline exceeded")

        if self.calls >= self.max_calls:
            raise RuntimeError("Tool-call limit exceeded")

        self.calls += 1

Example usage:

guardrail = AgentGuardrail(max_calls=5)

def invoke_approved_tool(tool, *args, **kwargs):
    guardrail.authorize()
    return tool(*args, **kwargs)

This is a simplified demonstration, not a complete production implementation.

For real deployments, use centralized quotas, atomic counters, concurrency controls, tool-level authorization, and durable audit records.

The execution service must enforce those checks even when the model requests a different action.

4. AI Agent Cost Controls

When agents can create cloud resources or invoke paid APIs, cost management becomes an operational requirement.

Useful controls include:

Per-task token budgets
Maximum paid tool calls
Request-rate and concurrency limits
Cost estimates before resource creation
Application-level spending ceilings
Cloud billing alerts and automated remediation

Cloud budget notifications are valuable, but they may arrive after charges have accumulated.

They should not be treated as instantaneous hard caps.

Enforce immediate limits at the tool gateway or application layer.

5. Least-Privilege IAM and Human Approval

Avoid giving agents unrestricted shell access or administrative cloud credentials.

Instead, expose narrowly scoped operations such as:

get_pod_events
read_application_logs
query_cloud_metrics
get_deployment_history

Write operations should use separate identities and permissions.

High-impact actions, such as deleting a database or changing IAM policies, should require explicit approval and independent authorization checks.

A model's claim that an action is safe must not bypass your enforcement layer.

6. Production Checklist

Before running autonomous agents in production:

Limit execution time and retry count.
Restrict available tools.
Validate all tool parameters.
Enforce quotas and cost budgets.
Use short-lived, scoped credentials.
Require approval for sensitive changes.
Record audit events without exposing secrets.
Test failure and emergency-stop scenarios.

The goal is to contain mistakes before they become production incidents.

Read the Full AI Agent Guardrails Guide

My complete article covers the four-tier architecture, Python circuit breakers, and AWS, Google Cloud, and Azure cost-management patterns.

Read AI Agent Guardrails: Framework, Cost Limits & Production Architecture

Final Thoughts

AI agents can make operational workflows faster, but autonomous execution introduces risks that traditional chatbots don't have.

The right approach is to enforce safety outside the model.

Which guardrail would you implement first: tool permissions, execution limits, cost controls, or human approval?

Share your thoughts in the comments.

Written by Naveed Ahmed — Lead DevOps & Platform Architect

Engineering Blog | GitHub | Kubernetes Mastery

Vibe Coding vs Spec-Driven Development: GitHub Spec Kit Guide

Naveed Ahmed — Thu, 29 Oct 2026 13:47:02 +0000

Originally published on Naveed Ahmed's Engineering Blog.

AI coding assistants have changed how quickly developers can build applications.

Describe a feature, generate code, run it, and ask the AI to fix whatever doesn't work.

For a small prototype, that's often a great workflow.

But what happens when the project grows, requirements change, and several engineers need to maintain the same codebase?

That's where the difference between vibe coding and Spec-Driven Development becomes important.

1. What Is Vibe Coding?

Vibe coding typically follows a conversational workflow:

Prompt
   ↓
Generate Code
   ↓
Run Application
   ↓
Fix With Another Prompt
   ↓
Repeat

It can be useful for prototypes, experiments, and quickly testing ideas.

The challenge appears when important requirements remain unstated.

An AI assistant might choose an unexpected architecture, overlook edge cases, or modify unrelated code during later iterations.

These aren't inevitable outcomes, but they're risks worth managing.

2. What Is Spec-Driven Development?

Spec-Driven Development (SDD) puts requirements and architecture ahead of code generation.

Instead of starting with implementation, you establish a specification that developers and AI agents can use as a shared reference.

A typical workflow looks like this:

Requirements
     ↓
Technical Design
     ↓
Implementation Tasks
     ↓
Code Generation
     ↓
Tests and Review

A good specification includes:

Functional requirements
Acceptance criteria
API and data contracts
Security constraints
Architecture decisions
Testing expectations

The goal isn't to write more documentation for its own sake.

It's to make important decisions explicit before implementation begins.

3. GitHub Spec Kit

GitHub Spec Kit is an open-source toolkit for organizing specification-driven workflows with AI coding assistants.

It provides a CLI and reusable templates for project rules, feature requirements, technical planning, and implementation tasks.

You can install the Specify CLI using uv:

uv tool install specify-cli \
  --from git+https://github.com/github/spec-kit.git

Initialize a project:

specify init auth-service --ai claude

Inside your coding-agent workflow, you can then define requirements, create a plan, and break implementation into tasks.

The exact supported commands depend on the installed Spec Kit version and agent integration.

The important part is maintaining reviewable artifacts rather than depending entirely on chat history.

4. Why This Matters for DevOps

The same problem appears when AI generates Kubernetes manifests or Terraform configurations.

For example, asking:

"Deploy my API to Kubernetes"

doesn't specify:

CPU and memory requests
Security context
Readiness probes
Network access
Secrets management
Observability
Rollback requirements

A structured infrastructure specification makes these constraints visible before code is generated.

You might define:

deployment:
  replicas: 3
  resourceRequests:
    cpu: 250m
    memory: 512Mi
  security:
    runAsNonRoot: true
    allowPrivilegeEscalation: false
  readinessProbe:
    required: true

This is an illustrative specification, not a Kubernetes Deployment manifest.

Your tooling can use these requirements to generate and validate actual Kubernetes resources.

5. When Should You Use Each Approach?

Situation	Suggested approach
Quick prototype	Vibe coding
Exploring an idea	Vibe coding
Production API	Spec-driven workflow
Multi-developer project	Spec-driven workflow
Security-sensitive feature	Specification and review
Kubernetes infrastructure	Specification, policy checks, and tests

You can also combine the two approaches.

Prototype quickly, discover the requirements, then formalize them before building a maintainable production system.

Read the Complete GitHub Spec Kit Guide

My full article explores:

The limitations of unstructured AI coding
Vibe coding vs Spec-Driven Development
GitHub Spec Kit architecture
Installing and using Specify CLI
Project constitutions and technical plans
AI-assisted Kubernetes and Terraform workflows
Production validation and engineering guardrails

Read the Complete Vibe Coding vs Spec-Driven Development Guide

Final Thoughts

AI can help teams write code faster.

But speed doesn't replace requirements, architecture, security, or verification.

For prototypes, a conversational workflow may be enough.

For systems people depend on, specifications make collaboration and review easier.

Are you using vibe coding, GitHub Spec Kit, or another structured AI coding workflow?

Share your experience in the comments.

Written by Naveed Ahmed — Lead DevOps & Platform Architect

Engineering Blog | GitHub | Kubernetes Mastery

Scaling DevOps with AI: MCP, AI Agents & Platform Engineering

Naveed Ahmed — Thu, 29 Oct 2026 07:41:35 +0000

Originally published on Naveed Ahmed's Engineering Blog.

DevOps teams have been automating infrastructure for years.

We have Terraform, Kubernetes, CI/CD pipelines, and dozens of monitoring tools.

Yet engineers still spend a surprising amount of time investigating repetitive alerts, reviewing configuration changes, and collecting context during incidents.

AI agents may help reduce that workload.

But giving an AI agent access to production infrastructure introduces a different problem: how do you make sure it acts safely?

I recently explored this question while writing about scaling DevOps in the age of AI.

Here are the most important architectural lessons.

1. From Manual Operations to AI-Assisted Engineering

Traditional automation works well when the steps are known.

AI can help with tasks where engineers first need to collect and interpret context.

For example, an AI-assisted incident workflow might:

Collect Kubernetes Pod events
Examine container restart reasons
Query Prometheus metrics
Compare recent deployments
Summarize likely causes
Recommend a relevant runbook

That can reduce investigation effort, provided the output is checked against actual system evidence.

The important distinction is that proposing a fix and safely executing a fix are separate responsibilities.

2. MCP Tools Instead of Unrestricted Shell Access

Model Context Protocol (MCP) provides a standardized way for AI applications to work with external tools.

For DevOps teams, that might mean exposing narrowly scoped diagnostic operations instead of a general-purpose production terminal.

Examples include:

get_pod_events(namespace, pod)
get_deployment_history(namespace, deployment)
query_prometheus(metric, time_range)
get_recent_application_errors(service)

However, MCP doesn't automatically enforce security.

The server implementing these tools must apply authentication, authorization, parameter validation, rate limits, and auditing.

A read-only Kubernetes service account is a better starting point than cluster-admin credentials.

3. Use Three Levels of Production Guardrails

Not every infrastructure action carries the same risk.

I prefer separating automation into three levels.

Level	Example	Control
Read-only	Fetch Pod logs and metrics	Authenticated automation
Bounded change	Adjust replicas within approved limits	Deterministic policy and recovery checks
High-risk mutation	Delete data or alter IAM permissions	Explicit human approval

These controls must be enforced outside the language model.

You can use tools such as Open Policy Agent to evaluate structured requests, but a policy decision must still be enforced by the execution service.

Even a Pod restart can cause problems, so production change automation needs workload-aware safeguards.

4. AI-Assisted Kubernetes Incident Triage

Imagine an application starts returning 5xx errors immediately after deployment.

A diagnostic agent could gather:

kubectl get pods -n production

kubectl describe deployment api -n production

kubectl rollout history deployment/api -n production

kubectl get events -n production --sort-by=.lastTimestamp

It could then correlate that information with application logs, traces, and recent code changes.

The result might be a concise incident briefing containing:

What changed
Which services are affected
Relevant evidence
Possible causes
Suggested verification steps

That's useful even if the agent never executes a production mutation.

5. Platform Engineering Makes AI More Reliable

AI-generated infrastructure is only as useful as the standards around it.

Without reusable patterns, different services can end up with inconsistent security settings, readiness probes, resource limits, and monitoring.

An internal developer platform can provide approved templates, reusable modules, and controlled self-service workflows.

AI agents can work within those boundaries instead of generating a completely new infrastructure design for every request.

This is where platform engineering becomes important: it creates reliable building blocks for engineers and automation alike.

Read the Complete Guide

My full article covers:

Agentic platform engineering
MCP-based DevOps tools
Three-tier production guardrails
Autonomous Kubernetes incident triage
Self-healing workflow risks
Internal developer platforms
Practical career development for DevOps engineers

Read Scaling DevOps in the Age of AI: Full Architecture Guide

Final Thoughts

AI may change how DevOps work gets done, but it doesn't remove the need for engineering judgment.

The opportunity is to automate repetitive investigation, create reliable self-service systems, and reserve human attention for decisions that truly need it.

Would you allow AI to execute production changes, or keep it restricted to diagnosis and recommendations?

Written by Naveed Ahmed — Lead DevOps & Platform Architect

Engineering Blog | GitHub | Kubernetes Mastery

7 Kubernetes Deployment Strategies: Rolling, Canary, Blue/Green & More

Naveed Ahmed — Thu, 29 Oct 2026 06:57:14 +0000

Originally published on Naveed Ahmed's Engineering Blog.

Most Kubernetes teams start with RollingUpdate.

It works well for many applications, but there are situations where you need more control over traffic, rollback behavior, or the number of users exposed to a new release.

That's where different Kubernetes deployment strategies become useful.

Let's look at seven approaches and when to use them.

1. Rolling Update

RollingUpdate is the default Kubernetes Deployment strategy.

It gradually replaces old Pods with new ones.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 0

With enough available capacity and correctly configured readiness checks, this can help maintain application availability during a rollout.

Best for: Stateless applications with backward-compatible changes.

2. Ramped Rollout

A ramped rollout is a more controlled variation of RollingUpdate.

Instead of updating several Pods at once, you replace them in smaller batches.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0
minReadySeconds: 30

This limits the pace of Pod replacement and requires newly ready Pods to remain ready for a minimum period.

Best for: Applications that need cautious rollouts or time for warm-up and observation.

3. Recreate

Recreate terminates the old application Pods before creating the replacement Pods.

strategy:
  type: Recreate

It's simple, but it normally creates an availability gap.

Best for: Applications that cannot safely run old and new versions together, where downtime is acceptable.

4. Blue/Green Deployment

Blue/Green keeps two application environments:

Blue: Current production version
Green: New application version

You validate Green before switching production traffic to it.

If something goes wrong, traffic can be routed back to Blue, provided the old version remains compatible with shared dependencies.

Best for: Releases requiring controlled cutovers and fast traffic-based rollback.

5. Canary Deployment

Canary deployment gradually exposes the new application version to real users.

For example:

Stable Version: 95%
Canary Version: 5%

Monitor error rates, latency, and application metrics before increasing the new version's traffic share.

Argo Rollouts and compatible traffic-routing tools can automate progressive delivery.

Best for: High-traffic applications where limiting the initial blast radius matters.

6. A/B Testing

A/B testing routes different groups of users to different versions.

Unlike Canary, the primary goal is often to compare user behavior or product outcomes rather than simply validate release stability.

Best for: Feature experiments, beta releases, and user-segment testing.

7. Shadow Deployment

Shadow deployment sends a copy of production requests to a new application version.

Users still receive responses from the stable application.

This allows teams to observe how the new version behaves under realistic traffic.

Best for: Performance testing, infrastructure changes, and validating new services before cutover.

Be careful with database writes and other side effects from mirrored requests.

Which Kubernetes Deployment Strategy Should You Choose?

Strategy	Main advantage	Main trade-off
Rolling Update	Simple gradual release	Mixed application versions
Ramped	Slower, controlled replacement	Longer rollout
Recreate	Avoids concurrent versions	Downtime
Blue/Green	Controlled traffic switching	Duplicate capacity
Canary	Limits initial exposure	Routing and monitoring complexity
A/B Testing	User-segment experimentation	More complex routing
Shadow	Tests with mirrored requests	Extra load and side-effect risks

The best choice depends on application architecture, downtime tolerance, available capacity, and rollback requirements.

Production Checks That Matter

Before using any deployment strategy, verify:

Readiness and startup behavior
Available cluster resources
Application and database compatibility
Rollout monitoring
Rollback procedures
Error rates and latency
Traffic-routing configuration

Useful Kubernetes commands include:

kubectl rollout status deployment/my-app

kubectl describe deployment my-app

kubectl rollout history deployment/my-app

kubectl rollout undo deployment/my-app

A successful rollout status doesn't necessarily mean every user-facing feature is working correctly.

Read the Full Kubernetes Deployment Guide

In my complete article, I compare all seven strategies with architecture diagrams, deployment manifests, production considerations, and troubleshooting examples.

Read the Complete Guide: 7 Kubernetes Deployment Strategies and When to Use Each

Final Thoughts

There's no single deployment strategy that works best for every application.

RollingUpdate is a good default, but Blue/Green, Canary, and Shadow patterns can provide additional control when your requirements become more complex.

The important part is understanding the trade-offs before deploying to production.

Which Kubernetes deployment strategy does your team use most often, and why?

Written by Naveed Ahmed — Lead DevOps & Platform Architect

Engineering Blog | GitHub | Kubernetes Mastery

Ruby Deserialization Security: RCE Risks, Kubernetes Auditing & Fixes

Naveed Ahmed — Thu, 29 Oct 2026 05:20:39 +0000

Originally published on Naveed Ahmed's Engineering Blog.

🚨 Could a serialized object become a security problem in your Ruby application?

Ruby provides powerful object serialization features, but accepting serialized data from untrusted sources can introduce serious security risks.

For DevOps and SRE engineers, the challenge isn't just finding a vulnerable code path. It's identifying every service, container, and background worker that might process unsafe input.

Here's a practical approach.

1. Why Ruby Deserialization Matters

Ruby's Marshal module can serialize and reconstruct Ruby objects.

For example:

data = Marshal.dump({ user: "demo" })
result = Marshal.load(data)

This is suitable only when the serialized data comes from a trusted source.

Using Marshal.load with untrusted data can expose applications to dangerous object-loading behavior.

The key question is whether an attacker can control the input passed to the deserializer.

2. Where Should You Look?

Review applications and components that handle serialized objects:

Ruby on Rails services
Background workers and job queues
Cache and session integrations
Internal APIs receiving binary data
Third-party libraries that deserialize objects

Don't assume every Rails or Sidekiq deployment uses unsafe deserialization. Inspect your actual code paths and dependencies.

3. Audit Ruby Workloads in Kubernetes

Start by identifying your running application images:

kubectl get pods -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,IMAGES:.spec.containers[*].image'

Check a candidate Ruby container:

kubectl exec -n production deploy/rails-api \
  -c web -- ruby -v

This tells you which runtime is running, but does not establish whether an application is exploitable.

To inspect your codebase, search for deserialization calls:

grep -RInE 'Marshal\.(load|restore)' \
  --include='*.rb' app lib

Review each match to determine where the data originates.

4. Scan Container Images

Trivy can help identify known vulnerabilities in application images:

trivy image --scanners vuln \
  --severity HIGH,CRITICAL \
  registry.example.com/rails-api:production

For larger environments, use Software Bills of Materials (SBOMs) to track dependencies.

Remember that automated scanners may not detect application-level insecure deserialization patterns.

You need both dependency scanning and code review.

5. Production Hardening

A safer approach includes:

Avoiding Marshal deserialization of untrusted input.
Using JSON or another appropriate format for external data.
Validating application inputs.
Updating Ruby and application dependencies.
Rebuilding and redeploying affected containers.
Applying least-privilege security controls.
Monitoring suspicious behavior and unexpected crashes.

Ingress and WAF rules may provide additional protection for specific paths, but they cannot guarantee safety if unsafe deserialization occurs inside application workers.

Read the Complete Ruby Security Guide

My full article discusses Ruby2Shell claims, Ruby object deserialization, Kubernetes auditing, container scanning, and production hardening.

Read the Ruby2Shell Technical Guide

Note: The specific CVE-2026-113840 vulnerability details and claimed patch versions have not yet been independently confirmed through an official Ruby advisory.

Final Thoughts

A running container and a successful vulnerability scan don't necessarily mean your application is safe.

Security also depends on how data enters your application and what happens when that data is processed.

How does your team identify unsafe deserialization in Ruby applications?

Share your approach in the comments.

Written by Naveed Ahmed — Lead DevOps & Platform Architect

Engineering Blog | GitHub | LinkedIn

HTTP2Shell: Brotli Buffer Overflow Root Cause & Production Fix

Naveed Ahmed — Thu, 29 Oct 2026 03:53:45 +0000

Originally published on Naveed Ahmed's Engineering Blog.

🚨 A security vulnerability in a compression library might not sound like a major infrastructure problem. But when that library is used across applications, containers, and cloud runtimes, the impact can extend much further than expected.

That's what makes HTTP2Shell (CVE-2026-111497) important for DevOps, SRE, and security teams.

The vulnerability affects Google's Brotli decompression library and can lead to memory corruption when vulnerable software processes specially crafted compressed data.

Depending on the affected application and exploitability, consequences may include crashes, information disclosure, or remote code execution.

Here's what engineering teams should understand and how to respond.

1. What Is HTTP2Shell?

HTTP2Shell is an out-of-bounds write vulnerability in the native Brotli decoder.

Brotli is widely used for data compression, including web traffic and application workloads.

The vulnerability occurs when affected decoder versions process specially crafted compressed input, potentially writing beyond allocated memory boundaries.

A simplified failure path looks like this:

Untrusted Brotli Input
         |
         v
Vulnerable Brotli Decoder
         |
         v
Out-of-Bounds Memory Write
         |
         v
Memory Corruption
         |
         v
Potential Crash / Data Exposure / RCE

Despite its name, HTTP2Shell isn't limited to HTTP/2. The underlying issue involves Brotli decompression, regardless of how attacker-controlled data reaches the decoder.

An application is not necessarily exploitable simply because Brotli is installed. The relevant question is whether vulnerable decoding functionality processes untrusted input.

2. Which Applications Should You Check?

The vulnerability can affect applications and services that bundle or dynamically load vulnerable Brotli implementations.

Important environments to investigate include:

Node.js applications using vulnerable runtimes
Python applications with affected Brotli packages
.NET applications using affected decompression functionality
Docker images containing vulnerable native libraries
Kubernetes workloads running affected application images
Other software with embedded Brotli decoders

For Node.js, the October 2026 security updates include these fixed releases:

Node.js release line	Fixed version
22.x	22.23.4
24.x	24.22.1
26.x	26.12.1

Older, unsupported Node.js versions should be migrated to a maintained release.

Also remember that Linux distributions may backport security fixes without changing the upstream version number. Check vendor advisories rather than relying only on raw version strings.

3. How to Detect HTTP2Shell in Kubernetes

For production environments, start by identifying the container images and runtimes actually deployed.

List the images running in a namespace:

kubectl get pods -n production \
  -o custom-columns='POD:.metadata.name,IMAGES:.spec.containers[*].image'

Inspect a Node.js application's runtime:

kubectl exec -n production deploy/api-gateway \
  -c app -- node -p "process.version"

Check the embedded Brotli version:

kubectl exec -n production deploy/api-gateway \
  -c app -- node -p "process.versions.brotli"

These commands help identify candidate workloads for further investigation.

They don't prove exploitability on their own, and version checks should be interpreted alongside the relevant vendor advisories.

4. Scan Container Images with Trivy

Container image scanning is another useful part of the investigation.

For example:

trivy image --scanners vuln \
  --severity HIGH,CRITICAL \
  registry.example.com/api-service:production

Replace the image reference with your actual production image.

Review the findings for CVE-2026-111497 and affected Brotli-related packages.

A clean scan doesn't necessarily mean every embedded runtime is safe. Some statically bundled libraries and application dependencies may require additional inspection.

For larger environments, Software Bills of Materials (SBOMs) can help identify where affected dependencies are deployed.

5. Immediate Production Mitigation

If vulnerable workloads are identified, prioritize updating the affected runtime or library.

A practical remediation workflow is:

Identify applications processing untrusted Brotli-compressed input.
Review official vendor security advisories.
Upgrade vulnerable runtimes and dependencies.
Rebuild affected container images.
Redeploy the updated workloads.
Verify runtime versions and scan the rebuilt images.
Monitor applications for abnormal crashes and unexpected errors.

For Kubernetes environments, rebuilding the container is particularly important.

Updating a package on the host won't fix a vulnerable library already bundled inside an application image.

What if patching can't happen immediately?

Where feasible, temporarily disable Brotli decompression of untrusted input or block affected inbound compressed requests at the edge.

However, a WAF or ingress rule isn't a complete solution.

It may not protect applications decoding Brotli data from internal services, upstream responses, files, or other sources.

Treat edge filtering as a temporary defense while completing the actual patching process.

6. What DevOps Teams Can Learn

HTTP2Shell is another reminder that infrastructure security isn't limited to operating system packages.

Modern applications depend on libraries embedded in language runtimes, containers, and other software components.

A vulnerability deep inside one dependency can require changes across many services.

That's why production teams need:

Regular container image scanning
Dependency inventories and SBOMs
Supported runtime versions
Automated CI/CD security checks
Reliable container rebuild procedures
Clear emergency patching workflows

The goal isn't just to detect vulnerabilities. It's to know exactly which workloads are affected and how quickly they can be remediated.

📖 Read the Full HTTP2Shell Production Guide

I wrote a detailed technical guide covering the vulnerability's root cause, affected runtimes, Kubernetes triage, container scanning, and production mitigation strategies.

The complete article includes:

Brotli buffer overflow architecture
HTTP2Shell (CVE-2026-111497) root cause
Node.js, Python, and .NET considerations
Kubernetes workload auditing
Trivy image scanning
WAF and ingress mitigation approaches
Container rebuilds and CI/CD security checks

👉 Read the Complete HTTP2Shell: Brotli Buffer Overflow Root Cause & Production Fix

Final Thoughts

A library vulnerability can become an infrastructure-wide issue when the same component exists across dozens or hundreds of workloads.

For DevOps and SRE teams, the priority is straightforward:

Identify exposure, patch affected dependencies, rebuild containers, and verify what's actually running in production.

💬 How does your team handle emergency CVE patching across Kubernetes clusters?

Do you use Trivy, SBOMs, automated dependency updates, or another approach?

Share your experience in the comments.

Written by Naveed Ahmed — Lead DevOps & Platform Architect

🌐 Engineering Blog | 🐙 GitHub | 💼 LinkedIn

🛠️ SRE Interview Practice | ☸️ Kubernetes Mastery

How a Kubernetes Database Migration Caused a 502 Production Outage

Naveed Ahmed — Tue, 27 Oct 2026 05:33:05 +0000

Originally published on Naveed Ahmed's Engineering Blog.

A routine Kubernetes rolling update. A breaking PostgreSQL schema migration. And suddenly, a production API starts returning 502 Bad Gateway errors.

At 2:15 AM, an automated CI/CD deployment triggered what should have been a normal application update.

Instead, database queries started blocking, API latency increased, and PostgreSQL connections became exhausted.

The incident lasted around 24 minutes.

The real problem wasn't Kubernetes itself. It was a database migration that wasn't compatible with the old and new application versions running simultaneously.

Here's what happened and how to prevent it.

1. What Went Wrong During the Kubernetes Rolling Update?

Kubernetes rolling updates gradually replace old application pods with new ones.

During deployment, both versions can run at the same time.

Kubernetes Rolling Update

Application v1 (Old Pods)
        +
Application v2 (New Pods)
        |
        v
 Shared PostgreSQL Database

In this case, the deployment included a destructive database migration that removed a column still required by the older application pods.

Those pods continued querying the deleted column, causing database errors.

Key lesson: A Kubernetes rolling update doesn't guarantee zero downtime if your database schema isn't backward-compatible.

2. How PostgreSQL Lock Contention Caused 502 Errors

The schema migration also introduced PostgreSQL lock contention.

Operations such as dropping a column require an ACCESS EXCLUSIVE lock.

For example:

ALTER TABLE users DROP COLUMN full_name;

If existing transactions hold conflicting locks, the migration must wait.

New queries can then queue behind the waiting migration, causing a cascade:

Schema Migration
       ↓
Exclusive Lock Wait
       ↓
Blocked Database Queries
       ↓
Connection Pool Exhaustion
       ↓
Application Timeouts
       ↓
HTTP 502 / 5xx Errors

This explains why a database migration can disrupt API traffic even when Kubernetes pods appear to be running.

To investigate blocked PostgreSQL sessions:

SELECT
    pid,
    query,
    wait_event_type,
    wait_event,
    pg_blocking_pids(pid) AS blocking_pids
FROM pg_stat_activity
WHERE cardinality(pg_blocking_pids(pid)) > 0;

This helps identify queries waiting on other database sessions.

3. Why the Kubernetes Rollback Failed

Rolling back the application doesn't automatically reverse database schema changes.

For example:

kubectl rollout undo deployment/api

This restores the previous Deployment revision, but it doesn't restore deleted PostgreSQL columns.

If the older application still depends on the removed schema, errors can continue even after the rollback.

Key lesson: Application rollback and database rollback are separate operations. Both need a compatible recovery strategy.

4. The Solution: Expand and Contract Database Migrations

The safer approach is to introduce database schema changes gradually.

Phase 1: Expand

Add the new schema without removing existing columns.

ALTER TABLE users ADD COLUMN display_name TEXT;

Phase 2: Migrate

Deploy compatible application code and backfill existing data.

For large tables, use controlled batches rather than one massive transaction.

Phase 3: Switch

Update application reads and writes to use the new schema. Verify data consistency and wait until older application versions are no longer serving traffic.

Phase 4: Contract

Remove the old column in a later deployment.

ALTER TABLE users DROP COLUMN full_name;

This approach helps avoid breaking application compatibility during Kubernetes rolling updates. Destructive DDL can still require careful lock management.

5. Production Best Practices

A few safeguards can reduce the risk of database migration outages:

Use backward-compatible schema changes.
Run database migrations through a controlled CI/CD step or Kubernetes Job.
Avoid running migrations independently from every application pod.
Configure appropriate PostgreSQL lock_timeout values.
Monitor database connections, lock waits, API latency, and HTTP 5xx errors.
Test rollback compatibility before production releases.

For example, a bounded lock wait can prevent migrations from waiting indefinitely:

SET lock_timeout = '5s';

Choose timeout values based on your workload and migration strategy.

Read the Complete Kubernetes Database Migration Postmortem

In the full article, I break down:

The 24-minute production outage timeline
Kubernetes rolling update failure
PostgreSQL AccessExclusiveLock contention
Database connection pool exhaustion
Why the rollback didn't restore service
Expand and Contract migration strategies
Production troubleshooting and prevention

Read the Full Kubernetes Database Migration 502 Outage Postmortem

Final Thoughts

Kubernetes rolling updates can keep applications available during deployments, but they can't protect you from incompatible database changes.

The biggest lesson from this incident is simple:

Your database schema must support both old and new application versions during a rolling deployment.

Have you experienced database migration failures during Kubernetes deployments? How does your team handle zero-downtime schema changes?

Share your experience in the comments.

Written by Naveed Ahmed — Lead DevOps & Platform Architect

🌐 Engineering Blog | 🐙 GitHub | 💼 LinkedIn

🛠️ SRE Interview Scenarios | ☸️ Kubernetes Mastery

Caddy Web Server Tutorial: Architecture, Docker & Production Guide

Naveed Ahmed — Tue, 27 Oct 2026 03:53:32 +0000

Originally published on Naveed Ahmed's Tech Blog.

If you've worked with Nginx or Apache, you've probably spent some time configuring virtual hosts, setting up SSL certificates, troubleshooting reverse proxy rules, or fixing certificate renewal issues.

These are normal parts of managing web infrastructure, but they can become repetitive, especially when you're responsible for multiple services.

That's where Caddy caught my attention.

Caddy is an open-source web server and reverse proxy written in Go. One of its biggest advantages is that it handles HTTPS automatically, including obtaining and renewing TLS certificates.

You can also configure a reverse proxy with just a few lines.

But how does Caddy actually work? Is it secure enough for production? And should you consider it instead of Nginx or Apache?

I put together a detailed guide to explore these questions.

Here's a practical overview of what I covered.

1. What Is Caddy, and Why Should DevOps Engineers Care?

Caddy is a modern web server that can serve websites, handle HTTPS connections, and route incoming requests to backend applications.

It supports many features you'd expect from a production web server:

Automatic HTTPS and TLS certificate renewal
Reverse proxy and load balancing
HTTP/2 and HTTP/3 support
Simple Caddyfile configuration
Dynamic configuration through a REST API
Docker and container-based deployments

What makes Caddy interesting isn't that these features are impossible to implement with other web servers.

It's that several of them work with very little configuration.

For example, here's a basic reverse proxy configuration:

example.com {
    reverse_proxy localhost:3000
}

That's enough to tell Caddy to forward requests to an application listening on port 3000.

With a public domain, correct DNS records, and the required network access, Caddy can also provision and manage HTTPS certificates automatically.

For small applications and containerized services, this can save quite a bit of setup time.

2. Caddy vs. Nginx vs. Apache: Which One Makes Sense?

Nginx and Apache have been around for years, and there's a reason they're still widely used.

Both are mature, flexible, and proven in production environments.

Caddy takes a slightly different approach, with more emphasis on sensible defaults and simplifying common configuration tasks.

Here's how I look at the three:

Feature	Caddy	Nginx	Apache
Automatic HTTPS	Built-in	Usually requires additional setup	Usually requires additional setup
Configuration	Simple Caddyfile	Directive-based	Directive-based
Reverse proxy	Built-in	Built-in	Available through modules
HTTP/3	Supported	Supported in compatible builds	Depends on implementation
Dynamic configuration	REST API	Depends on edition and tooling	Typically file-based
Common use case	Simple HTTPS and reverse proxy setups	High-performance web serving and proxying	Flexible web hosting

I wouldn't say Caddy is automatically better than Nginx or Apache.

If you already have a well-managed Nginx environment, there's no reason to migrate just for the sake of using something newer.

But if you're starting a new project and want HTTPS and reverse proxy configuration to be straightforward, Caddy is worth considering.

The best choice still depends on your workload, operational requirements, and existing infrastructure.

3. How Secure Is Caddy?

Security is one of the more interesting parts of Caddy's architecture.

Because it's written in Go, Caddy benefits from memory safety features that help prevent certain classes of memory-related vulnerabilities common in manually managed memory environments.

However, that doesn't mean Caddy is free from vulnerabilities.

Like any web server, it still needs proper configuration, timely security updates, and operational monitoring.

For production environments, I'd pay particular attention to:

Running the service with minimal privileges
Restricting access to the administration API
Keeping Caddy and its modules updated
Protecting TLS certificate storage
Limiting unnecessary network exposure
Monitoring logs and reverse proxy errors

One important detail is Caddy's administration API, which normally listens on a local interface.

It's useful for dynamic configuration, but you shouldn't expose it publicly without appropriate security controls.

Automatic HTTPS is a helpful feature, but it doesn't replace good server security practices.

4. Installing Caddy on Ubuntu or Debian

Getting started is relatively simple.

For Ubuntu and Debian systems, Caddy provides an official package repository.

First, install the required packages:

sudo apt install -y \
  debian-keyring \
  debian-archive-keyring \
  apt-transport-https \
  curl \
  gnupg

Add the official repository signing key:

curl -1sLf \
  'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' \
  | sudo gpg --dearmor \
  -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg

Add the repository:

curl -1sLf \
  'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' \
  | sudo tee /etc/apt/sources.list.d/caddy-stable.list

Make the repository files readable:

sudo chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg

sudo chmod o+r /etc/apt/sources.list.d/caddy-stable.list

Install Caddy:

sudo apt update
sudo apt install -y caddy

Verify the installation:

caddy version
sudo systemctl status caddy --no-pager

The official package installs Caddy as a systemd service, making it easier to manage on Linux servers.

5. Using Caddy as a Reverse Proxy

Let's say you're running a Node.js application on port 3000.

Instead of exposing that application directly to the internet, you can put Caddy in front of it.

A simple Caddyfile looks like this:

app.example.com {
    encode zstd gzip

    reverse_proxy 127.0.0.1:3000
}

In this example, Caddy:

Receives requests for app.example.com
Handles HTTPS automatically when the domain and network are configured correctly
Enables response compression
Forwards requests to the Node.js application

Before applying changes, validate the configuration:

sudo caddy validate --config /etc/caddy/Caddyfile

Then reload Caddy:

sudo systemctl reload caddy

This is one of my favorite parts of Caddy.

A common reverse proxy setup doesn't need a large configuration file, and you don't have to manage a separate certificate renewal workflow for typical publicly accessible domains.

6. Running Caddy with Docker

If you use Docker regularly, Caddy can also run as a container.

For example, you can start it with:

docker run -d \
  --name caddy \
  -p 80:80 \
  -p 443:443 \
  -p 443:443/udp \
  -v "$PWD/Caddyfile:/etc/caddy/Caddyfile:ro" \
  -v caddy_data:/data \
  -v caddy_config:/config \
  caddy:2

This assumes you've created a Caddyfile in your current directory.

The named volumes are important because Caddy needs persistent storage for certificate data and configuration state.

For a real deployment, I'd recommend using Docker Compose and placing Caddy on the same Docker network as the backend services.

That way, you can route requests using container service names rather than exposing every application's ports publicly.

For example:

app.example.com {
    reverse_proxy app:3000
}

Here, app would be the name of a backend service reachable through the shared Docker network.

It's a clean approach for applications that need HTTPS without a complicated reverse proxy setup.

7. Would I Use Caddy in Production?

Yes, I'd consider it, particularly for environments where keeping the web server simple is a priority.

Some good use cases include:

Small and medium-sized web applications
Containerized services
Internal applications and developer platforms
Reverse proxies for APIs
Self-hosted tools and dashboards

But there are still things I'd check before deploying it:

Certificate storage: Make sure certificate data survives container restarts and redeployments.
Networking: Confirm DNS records, firewall rules, and ports 80 and 443 are configured correctly.
Monitoring: Collect access logs, error logs, and relevant operational metrics.
Security: Keep the admin API restricted and run the service with appropriate permissions.
Configuration management: Store the Caddyfile in version control and validate changes before deployment.

And, as with any infrastructure component, I'd test it against the actual application's requirements rather than choosing it based on benchmark numbers alone.

📖 Want to Go Deeper?

This post covers the basics, but there's more to consider when running Caddy in production.

I wrote a longer, hands-on guide that explores:

Caddy's architecture and request handling
Caddy vs. Nginx vs. Apache
Security considerations and hardening
Installing Caddy across different environments
Docker deployment and reverse proxy configuration
Automatic HTTPS and certificate management
Production configuration and troubleshooting

If you're evaluating Caddy for your next project, the complete tutorial should help you get started.

👉 Read the Complete Caddy Web Server Tutorial: Architecture, Docker & Production Guide

Final Thoughts

One thing I've learned while working with infrastructure is that more configuration doesn't necessarily mean a better setup.

Sometimes, reducing the number of moving parts makes systems easier to maintain.

That's what makes Caddy interesting to me.

It doesn't replace every use case for Nginx or Apache, but it offers a practical alternative, especially when automated HTTPS and simple reverse proxy configurations are priorities.

I'd be interested to hear how others are using it.

Have you tried Caddy in production? Do you prefer it over Nginx, Apache, or Traefik?

Share your experience in the comments. I'd love to hear what worked, what didn't, and whether you'd recommend it.

About the Author

I'm Naveed Ahmed, a Lead DevOps & Platform Architect. I write about DevOps, Linux, cloud infrastructure, containers, Kubernetes, and production engineering.

You can find more of my work here:

Thanks for reading, and happy building!