DEV Community: DevOps Daily

Secrets Management Best Practices with HashiCorp Vault

DevOps Daily — Mon, 22 Jun 2026 10:55:34 +0000

A database password leaks. Maybe it was committed to a private repo three years ago, maybe it sat in a CI log, maybe a contractor copied it into a Slack DM. You do not know, because that password has been valid the entire time and nobody rotated it. Now you are in an incident channel at 2am trying to figure out the blast radius of a credential that every service, every old laptop, and every backup job has used since 2023.

This is the problem HashiCorp Vault solves, and it is not the problem most teams use it for. Most teams install Vault, run it in dev mode, dump a pile of static key-value secrets into it, and call it done. That gives you an encrypted password store with a nicer API. Useful, but it leaves the worst part untouched: secrets that live forever and that no human can fully account for.

The real win with Vault is making secrets short-lived and generated on demand, so a leak has an expiry date measured in hours instead of years. This post shows how to run Vault for that: a production server that survives reboots, machine authentication that does not depend on root tokens, dynamic database credentials, and encryption as a service. Every command here is one you can run.

TLDR

Never run vault server -dev for anything real. It is in-memory and unsealed, so a restart wipes every secret.
Use auto-unseal (AWS KMS, GCP KMS, or another Vault) so a reboot does not need five humans with key shares.
Authenticate machines with AppRole, not long-lived root or service tokens.
Use dynamic secrets for databases. Vault creates a unique DB user per request with a short TTL and deletes it when the lease ends.
Use the transit engine for encryption as a service so your apps never touch the encryption keys.
Write least-privilege policies, turn on the audit log, and revoke leases when something goes wrong.

Prerequisites

A Linux host (or VM) where you can install the Vault binary
Vault 1.15 or newer (vault version to check)
A PostgreSQL database you can point Vault at for the dynamic secrets section
An AWS account with a KMS key if you want auto-unseal (optional but recommended)
Basic comfort with the command line and HCL config files

Stop running Vault in dev mode

Dev mode is the trap. You run one command and get a working Vault:

vault server -dev

==> Vault server configuration:
             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: inmem

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key.

Read that warning. Storage: inmem means every secret lives in RAM and disappears on restart. tls: disabled means traffic is plaintext. It starts unsealed, so anyone who reaches port 8200 owns it. Dev mode is for trying commands on your laptop, nothing else.

A production server needs three things dev mode skips: persistent storage, TLS, and a seal. Here is a real config.hcl using integrated Raft storage and AWS KMS auto-unseal:

# /etc/vault.d/vault.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file  = "/opt/vault/tls/vault.key"
}

# Auto-unseal: Vault asks KMS to decrypt its root key on boot.
# No more gathering humans with key shares after every restart.
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abc-12345"
}

api_addr     = "https://vault-1.internal:8200"
cluster_addr = "https://vault-1.internal:8201"
ui           = true

Start it and initialize once:

vault server -config=/etc/vault.d/vault.hcl &

export VAULT_ADDR="https://vault-1.internal:8200"
vault operator init -recovery-shares=5 -recovery-threshold=3

Recovery Key 1: vR2k9... (give to a different person than key 2)
Recovery Key 2: 8Lp4m...
Recovery Key 3: qW7nZ...
Recovery Key 4: 3xF8t...
Recovery Key 5: hT1bY...

Initial Root Token: hvs.CAESIJ...

Success! Vault is initialized

Recovery key initialized with 5 key shares and a key threshold of 3.

Because of auto-unseal you get recovery keys instead of unseal keys. Vault unseals itself on boot using KMS, and the recovery keys are only for emergencies like regenerating the root token. Split them across different people and store them offline. Never keep all of them in one place.

Now use that root token once to set up authentication and policies, then throw it away. Root tokens are for break-glass moments, not daily use.

vault login hvs.CAESIJ...

If you ever see this, your Vault restarted and could not reach its seal:

$ vault kv get secret/payments/stripe
Error making API request.
URL: GET https://vault-1.internal:8200/v1/secret/data/payments/stripe
Code: 503. Errors:
* Vault is sealed

A sealed Vault answers nothing. That is the whole point. Auto-unseal exists so this state heals itself instead of paging you.

Authenticate machines with AppRole, not tokens

A common mistake: generate a long-lived token, paste it into an app's environment, and forget it exists. Now you have the same forever-credential problem one layer up. If that token leaks, it works until someone notices.

For machines, use AppRole. The app proves its identity with a role_id (think username, not very secret) and a secret_id (think password, short-lived and delivered separately), and gets back a token scoped to exactly what it needs.

vault auth enable approle

# Create a role for the payments service.
vault write auth/approle/role/payments-api \
    token_policies="payments-api" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=24h \
    secret_id_num_uses=1

# role_id is stable and tied to the role.
vault read auth/approle/role/payments-api/role-id

Key        Value
---        -----
role_id    7b1c4e2a-9f3d-4a8e-b6c1-2d5f8e0a1b3c

The secret_id is the part that needs care. Generate it just before the app starts and hand it over once. With secret_id_num_uses=1 it works exactly one time, so a leaked secret_id in a log is already useless.

vault write -f auth/approle/role/payments-api/secret-id

Key                   Value
---                   -----
secret_id             d8a3...e91f
secret_id_accessor    4c2b...77a0
secret_id_ttl         24h

The app logs in with both and gets a short-lived token:

vault write auth/approle/login \
    role_id="7b1c4e2a-9f3d-4a8e-b6c1-2d5f8e0a1b3c" \
    secret_id="d8a3...e91f"

Key                  Value
---                  -----
token                hvs.CAESI...
token_duration       1h
token_renewable      true
token_policies       ["default" "payments-api"]

That token dies in an hour unless the app renews it. The pattern that delivers the secret_id securely (a sidecar, a cloud instance identity, or Vault Agent) is its own topic, but the rule is simple: the role_id can live in config, the secret_id should be freshly minted and single-use.

Dynamic database credentials

This is the feature that changes how you think about secrets. Instead of one shared database password that every service knows, Vault creates a brand new database user for each request, with a short TTL, and deletes it when the lease expires.

Enable the database engine and point it at PostgreSQL:

vault secrets enable database

vault write database/config/orders-db \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="orders-readonly" \
    connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/orders?sslmode=require" \
    username="vault-admin" \
    password="$ROOT_DB_PASSWORD"

The vault-admin account is the only static credential, and it is a privileged account Vault uses to create and drop other users. Now define a role that says what a generated user is allowed to do:

vault write database/roles/orders-readonly \
    db_name="orders-db" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
      GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

Ask for credentials:

vault read database/creds/orders-readonly

Key                Value
---                -----
lease_id           database/creds/orders-readonly/Qm9iY...
lease_duration     1h
lease_renewable    true
password           A1a-9Zx2Kp4Lq7Rt0Vn3
username           v-approle-orders-rea-x7Qd2bN9

That username did not exist a second ago. Run the command again and you get a different user with a different password. Each service instance, each request if you want, gets its own credentials. When the lease ends, Vault runs the revocation statement and the user is gone from PostgreSQL.

Here is why this matters in numbers. A static password sits valid until a human rotates it, which in practice means months or years. A dynamic credential with a one-hour TTL is useless to an attacker an hour after it leaks.

The shrink in exposure window is the entire reason to run Vault. If you take one thing from this post, make it this section.

Encryption as a service with the transit engine

Sometimes you do not want to store a secret, you want to encrypt application data: a customer's tax ID, a token, a column in your database. The wrong move is to ship an AES key to every app and hope nobody loses it. The transit engine keeps the key inside Vault and exposes encrypt and decrypt operations. Your app sends plaintext and gets ciphertext back. It never sees the key.

vault secrets enable transit
vault write -f transit/keys/orders-pii

Encrypt some data (transit takes base64 input):

vault write transit/encrypt/orders-pii \
    plaintext=$(echo -n "4111-1111-1111-1111" | base64)

Key            Value
---            -----
ciphertext     vault:v1:8SDd4HCQ9p7Hf2bxN0kZ...
key_version    1

Store vault:v1:8SDd... in your database. To read it back:

vault write transit/decrypt/orders-pii \
    ciphertext="vault:v1:8SDd4HCQ9p7Hf2bxN0kZ..."

Key          Value
---          -----
plaintext    NDExMS0xMTExLTExMTEtMTExMQ==

Base64-decode that and you are back to the card number. The v1 prefix is the key version, which means you can rotate the key with vault write -f transit/keys/orders-pii/rotate and old ciphertext still decrypts while new writes use the fresh key. No key ever leaves Vault, so an app compromise leaks data the app could already see, not the key that protects all of it.

Least-privilege policies and the audit log

Tokens are only as safe as the policy attached to them. The payments-api policy referenced earlier should grant exactly what the service needs and nothing more:

# payments-api.hcl
# Read dynamic DB creds for the orders database.
path "database/creds/orders-readonly" {
  capabilities = ["read"]
}

# Encrypt and decrypt PII, but not manage or export the key.
path "transit/encrypt/orders-pii" {
  capabilities = ["update"]
}
path "transit/decrypt/orders-pii" {
  capabilities = ["update"]
}

vault policy write payments-api payments-api.hcl

Notice what is missing. No database/creds/orders-admin, no transit/keys/* management, no wildcard paths. If the payments token leaks, the attacker can read orders and decrypt PII for an hour, and that is the ceiling. When a request asks for something outside the policy, Vault refuses:

$ vault read database/creds/orders-admin
Error reading database/creds/orders-admin: Error making API request.
URL: GET https://vault-1.internal:8200/v1/database/creds/orders-admin
Code: 403. Errors:
* 1 error occurred:
    * permission denied

Turn on the audit log before you put anything real in Vault. It records every request and response (secrets are HMAC'd, not stored in clear) so you can answer "who read this secret and when" during an incident:

vault audit enable file file_path=/var/log/vault/audit.log

And when you do have an incident, dynamic secrets give you a clean kill switch. Revoke every credential a database role ever issued in one command:

vault lease revoke -prefix database/creds/orders-readonly

All revocation operations queued successfully!

Every dynamic user that role created gets dropped from the database. Try doing that with a shared password that lives in forty places.

Where to go next

You now have the shape of a real Vault setup: a sealed, persistent server; AppRole for machines; dynamic database credentials; transit for encryption; tight policies; and an audit trail. The static KV store is still there when you need it, but it should be the exception, not the default.

Concrete next steps:

Replace one static database password with a dynamic role this week. Pick a low-risk read-only service and cut over. Seeing credentials expire on their own is what makes the model click.
Stand up a 3-node Raft cluster, not a single server. One Vault node is a single point of failure for every secret you own. Run vault operator raft list-peers to confirm the cluster.
Deploy Vault Agent to handle AppRole login and token renewal so your apps read a rendered file or env var instead of calling the Vault API directly.
Set short TTLs and test revocation. Run vault lease revoke -prefix against a staging role and confirm the users vanish from your database. Know the command works before you need it at 2am.
Ship the audit log to your SIEM so secret access shows up next to the rest of your security telemetry.

Start with step one. Turning a single forever-password into a one-hour credential is the smallest change that removes the largest class of secret leaks you have.

10 GitHub Repositories That Will Actually Teach You DevOps in 2026

DevOps Daily — Tue, 05 May 2026 18:11:07 +0000

There are roughly a thousand "top DevOps repos" listicles, and most of them are the same five awesome-lists in a different order. The problem with awesome-lists is that they are link directories. They tell you where to look, not what to do. If you want to actually get better at DevOps, you need a different shape of repo: ones with exercises, opinionated learning paths, hands-on demos, and source you can read and learn from.

So here are ten GitHub repositories that have moved real engineers from "I have heard of Kubernetes" to "I run it in production." We will start with the one we maintain on this site, then walk through the rest in order of star count, with notes on who each one is for and how to get the most out of it.

TLDR

#	Repo	Stars	Best for
1	The-DevOps-Daily/devops-daily	1k+	Tutorials, exercises, and quizzes across the stack
2	nilbuild/developer-roadmap	354k	Visual roadmap to plan your learning
3	bregman-arie/devops-exercises	82k	Interview prep and practice questions
4	kelseyhightower/kubernetes-the-hard-way	48k	Building Kubernetes from scratch
5	MichaelCade/90DaysOfDevOps	29k	A structured 90-day plan
6	milanm/DevOps-Roadmap	19k	Roadmap with linked study resources
7	ramitsurana/awesome-kubernetes	16k	Curated Kubernetes deep-dive material
8	dastergon/awesome-sre	13k	SRE-specific reading list
9	stefanprodan/podinfo	6k	A real microservice to deploy with GitOps
10	wmariuss/awesome-devops	4k	Broader DevOps tooling and practices

Star counts are pulled fresh from the GitHub API as of May 2026.

1. The-DevOps-Daily/devops-daily

github.com/The-DevOps-Daily/devops-daily. the source for everything you read on this site, fully open source.

We did not put ourselves at the top because we own the site. We put ourselves at the top because the way the repo is structured is a fast loop: every blog post, exercise, quiz, flashcard, checklist, and interview question is a markdown or JSON file you can read, fork, and PR into. If you find a typo, a broken command, or an outdated CLI flag, you can fix it. If you have a better explanation of how kubelet eviction works, you can add a card to the relevant flashcard deck.

How to use it:

Browse the content/ directory. Pick a topic you want to get better at and run through the exercise.
Use the quizzes for spaced retrieval. Repeat until you stop getting things wrong.
Submit a PR when you find something to improve. The maintainers (us) review fast and merge most of the time.

Best for engineers who learn by doing, contributing, and seeing the underlying source of every lesson.

2. nilbuild/developer-roadmap

github.com/nilbuild/developer-roadmap. 354k stars. Originally kamranahmedse/developer-roadmap, now under the nilbuild org. The DevOps roadmap is at roadmap.sh/devops.

This is a visual map of the skills, tools, and concepts that make up a DevOps career. It is the single best document on the internet for answering "what should I learn next?" without reinventing your own learning plan from scratch.

How to use it:

Open the DevOps roadmap. Identify the area you are weakest in.
Click any node to get a short explanation, links, and a checklist.
Mark items as you go. The site keeps your progress in localStorage if you do not sign up.

Best for people who feel scattered and want a single picture of the field.

3. bregman-arie/devops-exercises

github.com/bregman-arie/devops-exercises. 82k stars. Maintained by Arie Bregman, ex-Red Hat.

This repository is the reason a lot of engineers passed their DevOps interviews. It is hundreds of practical questions and exercises across Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, and more. Each topic has a mix of explanation questions ("What is X and when do you use it?") and hands-on exercises ("Write the Terraform module that does X").

How to use it:

Pick a topic. Try to answer the questions out loud or in writing without looking at the answers.
Star the ones you got wrong. Come back to them in a week.
Use it as a barometer. If you can answer most of the Kubernetes section without help, you know your Kubernetes is solid.

Best for interview preparation and finding gaps in your knowledge.

4. kelseyhightower/kubernetes-the-hard-way

github.com/kelseyhightower/kubernetes-the-hard-way. 48k stars. The repo description is honest: "Bootstrap Kubernetes the hard way. No scripts."

If you have only ever used gcloud container clusters create or eksctl, you have used Kubernetes. You have not learned it. This walkthrough has you stand up a control plane and worker nodes by hand, with TLS certificates you generated yourself, etcd you configured yourself, and a kubelet you registered yourself.

It is also a primary reason Kelsey Hightower has the reputation he has, which is its own kind of education.

How to use it:

Block out a weekend. The full walkthrough takes 6 to 10 hours the first time.
Do not copy commands. Type them. Read what they do before you run them.
When something breaks (and it will), debug it. That is the entire point.

Best for engineers who want a deep mental model of Kubernetes internals.

5. MichaelCade/90DaysOfDevOps

github.com/MichaelCade/90DaysOfDevOps. 29k stars. Three years of community-curated 90-day plans.

This started as one engineer's public learning project: 90 days, one DevOps topic per day, write what you learned. It exploded, and is now a structured tour through Linux, networking, programming, containers, Kubernetes, IaC, observability, databases, and serverless across three different yearly cohorts. The format is one folder per day with notes, diagrams, and links.

How to use it:

Treat it as a TV series, not a textbook. Watch one "episode" a day for 90 days.
Skip topics you already know. Spend extra time on the ones that feel uncomfortable.
Read previous cohorts' notes when you finish a day. The 2022, 2023, and 2024 versions cover slightly different angles on the same material.

Best for engineers early in their career who want a forced curriculum.

6. milanm/DevOps-Roadmap

github.com/milanm/DevOps-Roadmap. 19k stars. A different style of roadmap from #2.

Where the nilbuild roadmap is a visual node graph, this one is a long markdown document with curated links, books, courses, and YouTube videos for every step of the path. It is heavier on resources, lighter on the conceptual map.

How to use it:

Read the introduction. Identify which "phase" of the roadmap you are at.
Pick one resource per concept. Do not read all five linked resources for the same topic. Pick the format that matches how you learn best.
Use the prompts at the end of each section as a checklist before moving on.

Best for self-taught engineers building their own curriculum.

7. ramitsurana/awesome-kubernetes

github.com/ramitsurana/awesome-kubernetes. 16k stars. The most thorough Kubernetes-specific awesome-list.

If your day job is Kubernetes-heavy and you want to specialize, this is the link directory you want. It has sections for everything: storage, networking, monitoring, security, multi-cluster, GitOps, service mesh, FinOps. Each link is annotated.

How to use it:

Bookmark the page. Use it as a research starting point when you need to evaluate tools in a category.
Watch the commit log. New tools get added regularly, so it doubles as a "what is happening in Kubernetes" feed.

Best for Kubernetes-track engineers and platform teams researching tools.

8. dastergon/awesome-sre

github.com/dastergon/awesome-sre. 13k stars. The SRE-flavored cousin.

DevOps and SRE overlap, but the SRE side weights toward reliability theory, incident response, observability, and the social engineering of running production systems. This repo is the curated reading list for that side: books (Google's SRE book, Charity Majors' work), papers, postmortems, blog posts, conference talks, training courses.

How to use it:

Read at least one published postmortem a week. The "Postmortems" section is gold.
The conference talks list is more useful than most paid SRE courses.
Pair it with kelseyhightower/kubernetes-the-hard-way if your SRE work is on a Kubernetes platform.

Best for engineers moving into SRE or platform-engineering roles.

9. stefanprodan/podinfo

github.com/stefanprodan/podinfo. 6k stars. A small Go web app that exists to be deployed.

This one is different from the others. podinfo is not a learning resource in the read-and-take-notes sense. It is a real microservice (Go, REST + gRPC, metrics, tracing, health checks) that is purpose-built to be the demo target in tutorials. It is what every Flux, Argo CD, Linkerd, Istio, and Cilium tutorial uses when they need a service to deploy. If you want to actually try a GitOps tool end-to-end, you build the platform, point it at podinfo's helm chart, and ship.

How to use it:

Stand up a kind or k3d cluster locally.
Install Flux or Argo CD and point it at the podinfo chart.
Roll out a canary. Add Linkerd. Add Prometheus. Each thing you add lets you exercise a different platform skill on a service that already works.

Best for engineers who learn by deploying, not reading.

10. wmariuss/awesome-devops

github.com/wmariuss/awesome-devops. 4k stars. Smaller than awesome-kubernetes, broader in scope.

This is the everything-DevOps awesome list: chaos engineering, configuration management, container orchestration, log management, monitoring, package management, secret management, service discovery. The size of the list is approachable, which is its main strength. You can scroll the whole thing in 15 minutes and have a real mental map of the DevOps tooling landscape.

How to use it:

Read the section headings before clicking any links. The taxonomy itself is a learning aid.
When evaluating a new category of tool (say, you have to pick a secret manager), use this as your starting set rather than Googling.

Best for engineers who want a manageable map of the whole DevOps tools world.

How to Actually Use a List Like This

Lists are starting points, not learning plans. The mistake people make is to star all ten repos and never come back. Avoid that:

Pick exactly one starting repo today. If you have no plan, start with #2 (the roadmap) to get one. If you have a plan, start with #4 (kubernetes-the-hard-way) to deepen it. If you are interview-prepping, start with #3 (devops-exercises).
Block calendar time. "I will learn DevOps in my spare time" does not work. "I will spend Thursdays from 7 to 9 PM on the kubernetes-the-hard-way walkthrough" works.
Build something. Pick one of the awesome-list categories you do not understand (say, "service mesh") and use podinfo (#9) plus a tool from the list to build a working setup. You will learn more in two hours of building than two weeks of reading.
Teach what you learned. Write a blog post. Submit a PR to #1 with a flashcard you made. Give a brown-bag at work. Teaching is the fastest way to find the gaps in what you thought you knew.

Bookmark this page and come back when you finish one repo. The list is not going anywhere.

Key Takeaways

Awesome-lists are link directories, not learning plans. Pair them with hands-on repos like #1, #4, and #9.
Star counts are not the same as quality, but they are a decent first filter. Anything above 5k stars in this space has been read by enough people to be roughly trustworthy.
The single best learning loop is read → build → teach. Most engineers do step one, skip step two, and never reach step three. The repos in this list are picked to support all three.
Start one. Finish one. Do not collect ten tabs and never close any of them.
Contribute back. Every repo in this list takes PRs. Even small ones (typo fixes, broken-link fixes) count. They also get you GitHub history that future employers can see.

If we missed a repo you think belongs here, open an issue on our repo and tell us which one. We update this list when something deserves to be on it.

Claude Code Hidden Features You Probably Missed

DevOps Daily — Wed, 01 Apr 2026 17:21:58 +0000

Most people use Claude Code to write code, fix bugs, and maybe generate a commit message. That's fine, but you're leaving a lot on the table.

Boris Cherny, the creator of Claude Code, recently shared a thread on X about features that even daily users tend to overlook. Some of these genuinely changed how I work. Here's a rundown of the ones worth knowing about.

TLDR

Claude Code has mobile sessions, automated scheduling, voice input, parallel agents, git worktrees, hooks, and a browser extension. Most people use about 20% of what it can do.

Move Your Session Anywhere with /teleport

You can start a session on your laptop and pick it up on your phone. Or move it to the web. The /teleport command transfers your full session context between devices.

The reverse also works. If you're reviewing something on your phone during a commute, you can /teleport it back to your terminal when you sit down.

There's also /remote-control which lets you connect to a running session from another device without transferring it.

# On your laptop
/teleport

# On your phone or web - enter the code to pick up the session

This is useful when you kick off a long-running task on your workstation and want to check progress from your phone.

Automate Repetitive Tasks with /loop and /schedule

This one is a genuine workflow changer. You can tell Claude Code to run a task on a recurring schedule for up to a week.

# Review PRs every 30 minutes
/loop 30m review open PRs and post comments

# Run a health check every hour
/schedule every 1h check if the staging environment is healthy

Think about what you do repeatedly: reviewing PRs, checking CI status, monitoring deployments, updating dependencies. You can automate all of it without writing a single script.

Some practical examples:

Review all open PRs every morning at 9 AM
Monitor a Slack channel for feedback and create GitHub issues
Run your test suite after every push and report failures
Check for dependency updates weekly

Hooks for Deterministic Automation

Hooks let you run code at specific points in Claude Code's lifecycle. Unlike the AI-driven /loop command, hooks are deterministic - they always run the same way.

You configure them in your settings and they fire on events like:

Session start - set up your environment, load context
Before bash commands - validate or log commands before execution
On permission requests - auto-approve specific patterns
Continuous operation - keep Claude running without manual intervention

This is powerful for teams. You can enforce standards (like running linters before every commit) without relying on each engineer to remember.

Git Worktrees for Parallel Sessions

If you've ever wanted Claude to work on two different branches at the same time, worktrees make this possible. Each session gets its own isolated copy of the repo.

# Start a session in a worktree
claude --worktree

Why this matters: you can have Claude refactoring module A while simultaneously building feature B. Neither session interferes with the other.

This pairs well with /batch, which fans out work across dozens of parallel agents. Need to update 50 files? /batch can process them concurrently instead of one at a time.

Voice Input with /voice

You can dictate to Claude instead of typing. This sounds gimmicky until you try it for longer explanations.

/voice

It's particularly useful for:

Explaining complex requirements ("I need a migration that handles both the old and new schema formats, with a rollback path if...")
Code reviews ("Look at the authentication flow in this PR and tell me if...")
Brainstorming ("What's the best way to structure this API given these constraints...")

Typing detailed prompts takes time. Talking is faster for anything longer than a few sentences.

The Chrome Extension for Frontend Work

Claude Code has a Chrome extension that lets the AI see what your app looks like in the browser. Instead of describing UI bugs, Claude can verify its own output visually.

This closes the feedback loop for frontend work. Claude makes a change, checks the browser, adjusts if something looks off. You stop being the human screenshot tool.

/branch and --fork-session for Experiments

Want to try two different approaches to the same problem? /branch creates a copy of your current session so you can explore a different path without losing your progress.

# Fork the current session
/branch

# Or fork when starting
claude --fork-session <session-id>

This is like git branches but for your AI conversation. Try approach A in one branch, approach B in another, then pick the winner.

/btw for Side Questions

When Claude is working on a long task, you might have an unrelated question. Instead of interrupting the main task, /btw lets you ask a side question.

/btw what's the difference between SIGTERM and SIGKILL?

Claude answers your side question and goes right back to what it was doing. No context switching, no lost progress.

--bare for SDK Speed

If you're using Claude Code in scripts or CI pipelines, the --bare flag skips loading plugins and extra features, making startup up to 10x faster.

claude --bare -p "generate a migration for adding user roles"

This matters when you're calling Claude from automation scripts where every second counts.

--add-dir for Multi-Repo Work

Working across multiple repositories? You can give Claude access to all of them in a single session.

claude --add-dir ~/projects/api --add-dir ~/projects/frontend

Now Claude can see your API schema and your frontend code at the same time. No more copying types between repos or explaining your API structure manually.

Custom Agents with --agent

You can create custom agent configurations with their own system prompts and tool permissions.

claude --agent reviewer    # Uses your custom reviewer agent config
claude --agent deployer    # Uses your custom deployer agent config

Define these in your .claude/agents/ directory. Each agent can have different instructions, different tool access, and different behaviors. A code reviewer agent doesn't need write access. A deployment agent doesn't need to browse the web.

What This Means for DevOps

These features shift Claude Code from "AI code assistant" to "AI DevOps team member." The combination of scheduling, hooks, parallel sessions, and multi-repo access means you can automate workflows that previously required custom tooling.

Here's a realistic DevOps setup:

/schedule reviews all PRs every morning
Hooks enforce linting and security scanning on every session
Worktrees let you debug production while shipping features
--add-dir gives Claude access to your infra and app repos simultaneously
/loop monitors your staging environment and alerts you on issues

The key insight from Boris's thread: "There is no one right way to use Claude Code." The tool is intentionally flexible. Experiment with these features and build the workflow that fits your team.

Try It Out

If you haven't updated Claude Code recently, run:

claude update

Many of these features are recent additions. The mobile app, scheduling, and hooks in particular have been added in the last few months.

For more DevOps tools and guides, check out our exercises and quizzes to sharpen your skills.

This post was inspired by Boris Cherny's thread on X. Boris is the creator of Claude Code at Anthropic.

🎄 Advent of DevOps: 25 Days to Level Up Your DevOps Game!

DevOps Daily — Sun, 30 Nov 2025 22:00:00 +0000

Hey DevOps enthusiasts! 👋

Remember how exciting advent calendars were as a kid? Each day bringing a new surprise behind those little doors? Well, we're bringing that same excitement to the DevOps world, but instead of chocolate (sorry! 🍫), you're getting something even better: real-world DevOps skills that will make you a better engineer.

🎁 What is Advent of DevOps?

Think "Advent of Code" meets real-world DevOps challenges. Starting December 1st, we're releasing 25 daily hands-on challenges that cover everything you need to know to thrive in modern DevOps environments.

Each day unlocks a new practical challenge focusing on tools and techniques you'll actually use in production. No theory-heavy lectures, no boring slides—just pure, hands-on learning that you can apply immediately.

🚀 What's Inside?

Here's a taste of what you'll tackle over 25 days:

🐳 Containerization & Orchestration
⚙️ CI/CD & Automation
🏗️ Infrastructure as Code
🔒 Security & Observability
☁️ Cloud & Scaling

💡 Why Join?

🎯 Real-World Skills: Every challenge is based on actual scenarios you'll face in production

📈 Progressive Learning: Start easy, level up gradually. Whether you're a beginner or seasoned pro, there's something for you

🎮 Fun & Engaging: Gamified progress tracking makes learning addictive (in a good way!)

🌟 Community-Driven: Share solutions, learn from others, and grow together

⏰ Learn at Your Pace: Can't keep up daily? No problem! All challenges remain available year-round

🎄 How It Works

Pick Your Challenge: Start with Day 1 or jump to what interests you most
Get Hands-On: Each challenge includes clear tasks, starter code, and success criteria
Build & Learn: Complete the challenge at your own pace
Share & Celebrate: Post your wins and solutions with the community
Level Up: Review reference solutions and explanations to deepen your understanding

Each challenge includes:

✅ Clear task description
🎯 Success criteria
🔧 Starter code (when applicable)
💡 Solution & explanation
🔗 Additional resources

🌟 Join the Community

This isn't just about solo learning—it's about growing together!

Share your progress:

Follow us on X/Twitter: @thedevopsdaily
Use hashtag: #AdventOfDevOps
Share on LinkedIn, dev.to, wherever you hang out!

Contribute:

Found a cool solution? Share it!
Have ideas for challenges? We're open-source!
Check out our GitHub repo and contribute

🎯 Ready to Start?

Don't wait for December 1st to check it out—head over to the page now and get familiar with what's coming:

👉 devops-daily.com/advent-of-devops

Mark your calendar 📅, set your reminders ⏰, and get ready to transform your DevOps skills one day at a time!

🤔 Who Should Join?

DevOps Engineers looking to sharpen their skills
Developers wanting to understand the ops side better
System Administrators transitioning to DevOps
Students & Career Changers building practical experience
Anyone curious about modern infrastructure practices

No gatekeeping here, if you're interested in DevOps, you're welcome! 🙌

🎊 Let's Make This December Special

Learning doesn't have to be boring. It doesn't have to be stressful. And it definitely doesn't have to be lonely.

This December, join hundreds (thousands?) of DevOps practitioners around the world in leveling up together. One challenge at a time, one skill at a time, one day at a time.

See you on December 1st! 🎄✨

P.S. - Can't wait? Start exploring the challenges now at devops-daily.com/advent-of-devops. They're already live and ready for early birds! 🐦

P.P.S. - This is completely free, open-source, and community-driven. No paywalls, no upsells, just pure learning. If you find value, give us a star on GitHub and spread the word! ⭐

Follow DevOps Daily:

🐦 X/Twitter: @thedevopsdaily
💻 GitHub: The-DevOps-Daily/devops-daily
🌐 Website: devops-daily.com

Happy DevOps-ing! 🚀

Building a DDoS Attack Simulator to Understand Defense Strategies

DevOps Daily — Fri, 21 Nov 2025 09:53:22 +0000

I created an educational content piece for DevOps Daily and realized something: most explanations of DDoS attacks are either too abstract or too technical. We talk about "request floods" and "mitigation strategies," but it's hard to visualize what's actually happening.

So I built an interactive simulator to help bridge that gap.

The Problem with Learning About DDoS 📚

When you're reading about DDoS protection, you see phrases like "distributes load across multiple servers" or "rate limiting prevents abuse." But what does that actually mean when thousands of requests are hitting your infrastructure?

I wanted something that would help people - especially those newer to infrastructure work - actually see these concepts in action.

What the Simulator Does 🎮

You can try it here: devops-daily.com/games/ddos-simulator

It lets you simulate three common attack types:

HTTP Flood 🌊 - overwhelming with legitimate-looking requests
SYN Flood 🔄 - exploiting TCP handshake mechanics
UDP Flood 📦 - connectionless packet storms

The interesting part is watching how different defense mechanisms respond. You can toggle:

Firewall 🛡️ - blocks about 30% based on signatures
Load Balancer ⚖️ - reduces impact by 50%
Auto Rate Limit 🚦 - blocks high-frequency traffic

What I Learned Building It 💡

A few things became clear while working on this:

Attack intensity matters less than you'd think. The attack type and your defense configuration matter way more. A moderate SYN flood with no defenses is worse than an intense HTTP flood with proper rate limiting.

Single defenses aren't enough. This is obvious in theory, but seeing it play out makes it concrete. A firewall alone, or a load balancer alone, only gets you so far.

Visualization helps understanding. Watching the server health bar drop while packets animate across the screen creates an intuition that documentation doesn't.

Who Might Find This Useful ⚙️

If you're:

Learning about infrastructure security
Trying to explain DDoS concepts to your team
Deciding what protections to implement
Just curious how attacks and defenses interact

It might be helpful to play around with it for a bit.

What's Next 🚀

I'm planning to add more waves with additional attack vectors and defense mechanisms. Things like:

Application-layer attacks
CDN protection
Anycast routing
More realistic traffic patterns

If you have thoughts on what would be useful to include, I'd be interested to hear them.

The goal here is education, not creating chaos. Understanding how attacks work helps you build better defenses. 🛡️

If you try it out, let me know what you think or if anything is unclear.

Right-Sizing Kubernetes Resources with VPA and Karpenter

DevOps Daily — Fri, 22 Aug 2025 17:02:04 +0000

TLDR

Setting CPU and memory requests too high in Kubernetes wastes money and reduces cluster efficiency. This guide shows you how to identify overprovisioned workloads, use Vertical Pod Autoscaler (VPA) to right-size your pods, and implement Karpenter for smarter node scaling. You'll also learn to monitor costs and validate your improvements with real metrics.

When you set resource requests too conservatively in Kubernetes, your cluster reserves more capacity than workloads actually need. This leads to underutilized nodes and higher cloud bills. The problem gets worse at scale - imagine 200 pods each requesting 2 CPU cores but only using 200m. That's 400 reserved cores when actual demand is closer to 40 cores.

The solution involves right-sizing both your pods and nodes. You'll use monitoring data to understand actual usage, apply VPA to adjust pod requests automatically, and leverage Karpenter to provision nodes that match your workload requirements.

Prerequisites

Before you start, make sure you have:

A Kubernetes cluster (version 1.20 or higher) with metrics-server installed
kubectl configured with admin access to your cluster
Prometheus and Grafana deployed for monitoring (or similar observability stack)
Basic understanding of Kubernetes resource requests and limits

You'll also need the ability to install cluster-wide components like VPA and Karpenter.

Identifying Overprovisioned Workloads

The first step is understanding how your current workloads use resources compared to what they request. You can start with kubectl to get a quick snapshot of resource usage across your cluster.

# Check current resource usage for all nodes
kubectl top nodes

# View pod resource usage across all namespaces
kubectl top pods --all-namespaces --sort-by=cpu

# Get detailed resource requests vs usage for a specific namespace
kubectl describe nodes | grep -A 15 "Allocated resources"

These commands show you the gap between requested and actual resource usage. If you see pods consistently using 50Mi of memory while requesting 1Gi, or using 100m CPU while requesting 1000m, those are prime candidates for right-sizing.

For deeper analysis, you'll want historical data from Prometheus. Here are some key queries to run in your Grafana dashboard:

# CPU utilization percentage (actual usage vs requests)
(rate(container_cpu_usage_seconds_total{container!=""}[5m]) * 100) /
(container_spec_cpu_quota{container!=""} / container_spec_cpu_period{container!=""})

# Memory utilization percentage
(container_memory_working_set_bytes{container!=""} * 100) /
container_spec_memory_limit_bytes{container!=""}

# Top 10 pods with the highest request-to-usage ratio (biggest waste)
topk(10,
  (container_spec_cpu_quota{container!=""} / container_spec_cpu_period{container!=""}) /
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
)

Run these queries over a 2-week period to account for traffic variations and identify consistent patterns. Workloads running at 10-20% utilization with stable traffic are good candidates for optimization.

Installing and Configuring VPA

Vertical Pod Autoscaler analyzes your workloads and recommends optimal CPU and memory values. Start by installing VPA in your cluster.

# Clone the VPA repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Deploy VPA components
./hack/vpa-up.sh

This script installs three main components: the VPA recommender (analyzes usage), the updater (applies changes), and the admission controller (validates recommendations).

Next, create a VPA configuration for a workload you want to optimize. Start with recommendation mode to see suggested values before making changes.

# vpa-web-service.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-service-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: 'apps/v1'
    kind: Deployment
    name: web-service
  updatePolicy:
    updateMode: 'Off' # Only provide recommendations, don't auto-update
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        # Set boundaries to prevent extreme recommendations
        maxAllowed:
          cpu: '2'
          memory: '4Gi'
        minAllowed:
          cpu: '100m'
          memory: '128Mi'
        controlledResources: ['cpu', 'memory']

Apply the VPA configuration and wait for recommendations to generate:

kubectl apply -f vpa-web-service.yaml

# Wait a few minutes, then check recommendations
kubectl describe vpa web-service-vpa -n production

The output shows recommended values for CPU and memory under the Status section. VPA typically suggests values based on the 90th percentile of usage over the past 8 days, which provides a safety buffer while eliminating waste.

Applying VPA Recommendations Safely

Once you have solid recommendations, you can apply them gradually. Start with non-critical workloads and monitor for any issues.

# Update your deployment with VPA recommendations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-service
  template:
    metadata:
      labels:
        app: web-service
    spec:
      containers:
        - name: web-app
          image: nginx:1.21
          resources:
            requests:
              cpu: '250m' # Reduced from 1000m based on VPA recommendation
              memory: '512Mi' # Reduced from 2Gi based on VPA recommendation
            limits:
              cpu: '500m' # Set limits 2x requests for burst capacity
              memory: '1Gi'

After updating requests, monitor your workloads for at least a week. Watch for:

Increased pod restarts or OOMKilled events
Higher response times or error rates
Pods getting evicted under memory pressure

If everything runs smoothly, you can switch VPA to automatic mode:

# Update VPA to automatically apply changes
kubectl patch vpa web-service-vpa -n production --type='merge' -p='{"spec":{"updatePolicy":{"updateMode":"Auto"}}}'

In Auto mode, VPA will restart pods when it detects they need different resource allocations. Make sure you have proper PodDisruptionBudgets in place to maintain availability during updates.

Setting Up Karpenter for Node Optimization

While VPA optimizes individual pods, Karpenter optimizes your entire node infrastructure. Instead of fixed node groups, Karpenter provisions nodes dynamically based on your workload requirements.

First, install Karpenter in your cluster. The exact steps depend on your cloud provider, but here's the process for AWS EKS:

# Install Karpenter using Helm
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version "0.32.0" \
  --namespace "karpenter" \
  --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueueName=${CLUSTER_NAME}" \
  --wait

Next, create a NodePool that defines what types of nodes Karpenter can provision:

# karpenter-nodepool.yaml
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general-purpose
spec:
  # Template for nodes Karpenter will create
  template:
    metadata:
      labels:
        node-type: general-purpose
    spec:
      # Instance requirements - Karpenter will pick the best fit
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ['amd64']
        - key: karpenter.sh/capacity-type
          operator: In
          values: ['spot', 'on-demand'] # Allow both for cost optimization
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ['m6i.large', 'm6i.xlarge', 'm6i.2xlarge', 'r6i.large', 'r6i.xlarge']

      # Node configuration
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: general-purpose

      # Taints to control which pods can schedule here
      taints:
        - key: karpenter.sh/unschedulable
          value: 'true'
          effect: NoSchedule

  # Scaling and disruption policies
  limits:
    cpu: 1000 # Maximum CPU across all nodes in this pool
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s

Create the corresponding EC2NodeClass for AWS-specific configuration:

# karpenter-nodeclass.yaml
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: general-purpose
spec:
  # AMI and instance configuration
  amiFamily: AL2
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: '${CLUSTER_NAME}'
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: '${CLUSTER_NAME}'

  # Instance store configuration
  userData: |
    #!/bin/bash
    /etc/eks/bootstrap.sh ${CLUSTER_NAME}

  # Tags for cost tracking
  tags:
    Team: platform
    Environment: production

Apply both configurations:

kubectl apply -f karpenter-nodepool.yaml
kubectl apply -f karpenter-nodeclass.yaml

Karpenter will now monitor unschedulable pods and provision appropriately-sized nodes. When you deploy workloads with right-sized resource requests (thanks to VPA), Karpenter will select smaller, more cost-effective instances.

Monitoring Cost Impact

To validate your optimizations, you need visibility into resource costs. Kubecost provides detailed insights into how much each workload costs and how much capacity you're wasting.

Install Kubecost in your cluster:

# Add the Kubecost Helm repository
helm repo add kubecost https://kubecost.github.io/cost-analyzer/

# Install Kubecost with Prometheus integration
helm install kubecost kubecost/cost-analyzer \
  --namespace kubecost \
  --create-namespace \
  --set kubecostToken="your-token-here" \
  --set prometheus.server.global.external_labels.cluster_id="${CLUSTER_NAME}"

Access the Kubecost UI by port-forwarding:

kubectl port-forward -n kubecost deployment/kubecost-cost-analyzer 9090:9090

In the Kubecost dashboard, focus on these key metrics:

Efficiency scores: Shows the percentage of requested resources actually being used
Idle costs: Money spent on provisioned but unused resources
Right-sizing recommendations: Suggestions for adjusting requests and limits
Namespace costs: Helps identify which teams or applications drive costs

Track these metrics before and after implementing VPA and Karpenter to quantify your savings.

Real-World Optimization Example

Let's walk through optimizing a typical microservice deployment. You start with a Node.js API that was conservatively configured:

# Before optimization
resources:
  requests:
    cpu: '1000m'
    memory: '2Gi'
  limits:
    cpu: '2000m'
    memory: '4Gi'

After running this workload for two weeks, your monitoring shows:

Average CPU usage: 150m (15% of requests)
Average memory usage: 400Mi (20% of requests)
Peak CPU usage: 300m
Peak memory usage: 800Mi

Based on this data, VPA recommends:

# VPA recommendations (with safety buffer)
resources:
  requests:
    cpu: '200m' # Covers 99th percentile usage
    memory: '512Mi' # Accounts for memory spikes
  limits:
    cpu: '400m' # 2x requests for burst capacity
    memory: '1Gi' # Prevents OOM while allowing growth

The cost impact for 20 replicas of this service:

Before: 20 CPU cores, 40Gi memory requested
After: 4 CPU cores, 10Gi memory requested
Savings: 80% reduction in resource allocation

With Karpenter managing nodes, this workload now runs on smaller instances, further reducing costs by eliminating the need for oversized nodes.

Setting Resource Quotas and Guardrails

As you roll out right-sizing across your organization, implement quotas to prevent teams from reverting to oversized requests:

# namespace-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: backend-team-quota
  namespace: backend
spec:
  hard:
    requests.cpu: '50' # Total CPU requests across all pods
    requests.memory: '100Gi' # Total memory requests
    limits.cpu: '100' # Total CPU limits
    limits.memory: '200Gi' # Total memory limits
    pods: '100' # Maximum number of pods

You can also create LimitRanges to enforce reasonable defaults:

# limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: pod-limits
  namespace: backend
spec:
  limits:
    - type: Container
      default: # Default limits if not specified
        cpu: '500m'
        memory: '1Gi'
      defaultRequest: # Default requests if not specified
        cpu: '100m'
        memory: '256Mi'
      max: # Maximum allowed values
        cpu: '4'
        memory: '8Gi'
      min: # Minimum required values
        cpu: '50m'
        memory: '64Mi'

These guardrails help maintain optimization gains while giving teams flexibility within reasonable bounds.

Troubleshooting Common Issues

When implementing VPA and Karpenter, you might encounter some challenges. Here are solutions to the most common problems:

VPA recommendations seem too aggressive: VPA sometimes suggests very low values during low-traffic periods. Check that your monitoring data covers representative traffic patterns. You can also adjust the VPA algorithm:

spec:
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        controlledValues: RequestsOnly # Only adjust requests, leave limits alone
        mode: Auto

Karpenter nodes aren't scaling down: This usually happens when pods can't be evicted. Check for:

# Look for pods without PodDisruptionBudgets
kubectl get pods --all-namespaces -o wide | grep -v Terminating

# Check for pods using local storage or host networking
kubectl get pods --all-namespaces -o yaml | grep -A 5 hostNetwork

# Verify PodDisruptionBudgets allow eviction
kubectl get pdb --all-namespaces

Pods getting OOMKilled after VPA optimization: This indicates VPA recommendations were too low. Temporarily increase memory requests and check for memory leaks in your application:

# Check recent OOM events
kubectl get events --sort-by=.metadata.creationTimestamp | grep OOMKilled

# Monitor memory usage patterns
kubectl top pods --sort-by=memory --all-namespaces

You can make VPA more conservative by setting higher safety margins:

spec:
  resourcePolicy:
    containerPolicies:
      - containerName: web-app
        maxAllowed:
          memory: '2Gi' # Set a reasonable upper bound

Next Steps

Now that you have VPA and Karpenter working together, consider these additional optimizations:

Horizontal Pod Autoscaling: Combine with VPA to handle both vertical and horizontal scaling
Cluster Autoscaler tuning: If using multiple node provisioners, configure them to work together
Cost alerts: Set up notifications when resource costs exceed thresholds
Regular reviews: Schedule monthly reviews of VPA recommendations and cost reports

You can also explore more advanced Karpenter features like multiple NodePools for different workload types (CPU-intensive, memory-intensive, GPU workloads) and spot instance strategies for non-critical workloads.

The key is to treat right-sizing as an ongoing process. As your applications evolve and traffic patterns change, continue monitoring and adjusting to maintain optimal resource utilization.

The 5-Minute Kubernetes Cluster Health Check

DevOps Daily — Fri, 15 Aug 2025 10:30:08 +0000

TLDR

You can check your Kubernetes cluster's health in under 5 minutes using five key commands: checking node status, monitoring resource usage, reviewing pod health across namespaces, investigating problem pods, and examining cluster events. This quick routine helps catch issues before they escalate into critical problems.

Kubernetes is great until it's not. One bad node, a pod stuck in CrashLoopBackOff, or a resource spike can ruin your day. The good news? You don't need to spend an hour digging through dashboards to spot trouble early. With a few quick commands, you can get a solid read on your cluster's health in under 5 minutes.

Here's how to do it effectively.

Make Sure Your Nodes Are Happy

Start by checking the overall status of your cluster nodes. This gives you the foundation-level health of your infrastructure.

kubectl get nodes -o wide

This command displays all nodes in your cluster along with their detailed information. You'll see each node's status, roles, age, version, internal and external IPs, OS image, kernel version, and container runtime.

What you want to see:

STATUS should be Ready for all nodes
No mystery nodes suddenly showing up in your cluster
Roles, IPs, and ages that make sense for your environment

If you spot NotReady, that's your cue to dig deeper. A node in this state might be experiencing network issues, resource exhaustion, or kubelet problems.

Check Resource Usage at a Glance

Next, get a quick overview of resource consumption across your nodes to identify potential bottlenecks.

kubectl top nodes

This command shows CPU and memory usage for each node in your cluster. It provides both absolute values and percentages, making it easy to spot resource pressure.

Keep an eye out for:

CPU or memory regularly above 80% on any node
One node doing all the heavy lifting while others are barely working
Sudden spikes that don't match your expected workload patterns

No metrics-server running? Install it with this command:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

The metrics-server is essential for resource monitoring and is required for horizontal pod autoscaling to work properly.

Look at All Pods Across All Namespaces

Get a bird's-eye view of all pods running in your cluster to quickly identify any that are misbehaving.

kubectl get pods --all-namespaces

This command lists every pod across all namespaces, showing their current status, restart count, and age. It's like taking the pulse of your entire application ecosystem.

Healthy pods should be Running or Completed. If you see states like CrashLoopBackOff, ImagePullBackOff, Pending, or Error, note the namespace and pod name for further investigation.

Also watch the RESTARTS column closely. If a pod has restarted a dozen times in the last hour, something's definitely off. Frequent restarts often indicate:

Application crashes due to bugs or configuration issues
Failing health checks (readiness or liveness probes)
Resource limits being exceeded
Dependencies being unavailable

Zoom In on Problem Pods

When you spot problematic pods, dig deeper to understand what's causing the issues.

kubectl describe pod <pod-name> -n <namespace>

Replace <pod-name> and <namespace> with the actual values from your problem pods. This command provides detailed information about the pod's configuration, current state, and recent events.

Check for these common issues:

Events at the bottom (often the smoking gun that reveals the root cause)
Failing readiness or liveness probes that prevent the pod from receiving traffic
Image pull errors indicating registry access problems or incorrect image names
Resource limit issues where the pod exceeds its memory or CPU constraints

The events section is particularly valuable because it shows a chronological history of what happened to the pod, including scheduling decisions, volume mounts, and error conditions.

Check the Cluster's Event Log

Get insight into what's been happening across your entire cluster by examining recent events.

kubectl get events --sort-by=.metadata.creationTimestamp

This command shows cluster-wide events sorted by when they occurred, giving you a timeline of recent activity. Events provide context about system-level operations and can reveal patterns or issues that affect multiple components.

Events will tell you what's been happening behind the scenes:

Failed volume mounts that prevent pods from starting
DNS resolution errors affecting service communication
Scheduling issues when pods can't be placed on nodes
Node pressure warnings indicating resource constraints

Try k9s for a Better View

If you want something more interactive than command-line tools, give k9s a try. It's a terminal-based UI for Kubernetes that provides real-time cluster information in an intuitive interface.

k9s lets you browse resources, view logs, and drill into problems without typing long commands. You can navigate between different resource types using simple keystrokes, filter resources, and even perform actions like scaling deployments or deleting pods.

Once you try k9s, it's hard to go back to plain kubectl for exploratory tasks. It's particularly useful when you need to quickly jump between different namespaces or resource types during troubleshooting.

Five minutes a day is all it takes to stay ahead of most cluster problems. Make this health check part of your daily routine and you'll catch issues before they blow up and before your pager goes off at 3 a.m. Regular monitoring helps you understand your cluster's normal behavior, making it easier to spot anomalies when they occur.

What’s the Most Underrated DevOps Skill You’ve Learned (and How Did You Learn It)?

DevOps Daily — Tue, 05 Aug 2025 07:43:56 +0000

When we think about DevOps skills, we usually picture Kubernetes, Terraform, CI/CD pipelines, or cloud automation.

But some of the most valuable skills are the ones that never make it into a certification or a tech stack diagram.

It could be:

Staying calm during a production incident and knowing how to prioritize actions
Communicating effectively with teams under pressure
Spotting patterns in logs and metrics that others might miss
Finding ways to optimize cloud costs without slowing down delivery
Automating the boring stuff so you can focus on the real problems

For me, one of the most underrated skills I've learned is knowing when not to automate something. Sometimes the "manual but reliable" approach saves you from a lot of complexity and maintenance overhead later.

What about you?
What's the most underrated DevOps skill you've picked up along the way, and how did you learn it?

P.S. You might find some useful DevOps resources at devops-daily.com

What's the One DevOps "Best Practice" You Secretly Ignore (and Why)?

DevOps Daily — Wed, 30 Jul 2025 14:04:41 +0000

We've all read the books, followed the gurus, and tried to tick every box in the DevOps checklist.. but let’s be honest:

There's always that one best practice that just doesn’t work for your team, your stack, or your sanity.

Maybe you don't write as many tests as you should.
Maybe you still SSH into production (👀).
Maybe you use latest tags on your Docker images and pray.

No judgment here, just real talk from the trenches.

What's your "ignored" DevOps best practice, and why do you skip it?

Bonus points if you share how it's actually worked out for you.

🛠️ Posted by the team behind DevOps Daily

The Complete DevOps Roadmap for 2025 🚀

DevOps Daily — Sat, 26 Jul 2025 13:13:38 +0000

The DevOps landscape continues to evolve rapidly, and 2025 presents incredible opportunities for aspiring engineers. Organizations are increasingly adopting DevOps practices to deliver software faster, more reliably, and at scale. The demand for skilled DevOps professionals has never been higher.

Whether you're a developer looking to expand into operations, a system administrator aiming to modernize your skills, or a complete beginner drawn to this exciting field, this comprehensive roadmap will guide your journey to DevOps mastery.

Why DevOps in 2025? 🌟

DevOps represents a fundamental shift in how software is built, deployed, and maintained. It's not just about tools, it's about culture, collaboration, and continuous improvement. Here's why it matters:

🔄 Faster Delivery: Teams deploy multiple times per day instead of monthly releases
🛡️ Better Reliability: Automated testing and monitoring catch issues early
⚡ Improved Collaboration: Breaks down silos between development and operations
🔧 Enhanced Automation: Reduces manual work and human error
📈 Career Growth: High demand for skilled professionals across all industries

But beyond the benefits, DevOps offers intellectually rewarding work where you solve complex problems and see immediate impact on product delivery.

DevOps in the Age of AI: Why Infrastructure Matters More Than Ever 🤖

With AI transforming every industry, you might wonder: "Is DevOps still a smart career choice?" The answer is a resounding yes, and here's why:

🏗️ AI Runs on Infrastructure

Every AI application, from ChatGPT to autonomous vehicles, depends on robust, scalable infrastructure:

🚀 Model Training: Requires massive computational resources and distributed systems
⚡ Real-time Inference: Needs low-latency, highly available services
📊 Data Pipelines: AI models need continuous data flow and processing
🔄 Model Deployment: Rolling out AI models safely requires sophisticated CI/CD

🤝 AI Enhances DevOps (Doesn't Replace It)

Rather than replacing DevOps engineers, AI is becoming a powerful tool in our toolkit:

🔍 Intelligent Monitoring: AI helps predict system failures before they happen
🛠️ Automated Remediation: Smart systems can fix common issues automatically
📈 Resource Optimization: AI optimizes cloud costs and performance
🔐 Security Enhancement: AI-powered threat detection and response

🎯 The Human Element Remains Critical

While AI can automate many tasks, DevOps engineers provide irreplaceable value:

🧠 Strategic Thinking: Designing architecture and making technology choices
🔧 Complex Problem Solving: Debugging unique issues and system design
👥 Cross-team Collaboration: Bridging technical and business requirements
📋 Compliance & Governance: Ensuring systems meet regulatory requirements

🌐 Growing Complexity Requires Expertise

As AI adoption accelerates, infrastructure becomes more complex:

🔀 Multi-cloud Strategies: Managing resources across different providers
⚓ Container Orchestration: Running AI workloads at scale
🔒 Security Challenges: Protecting sensitive AI models and data
📊 Observability Needs: Understanding performance of AI-driven systems

The 9-Stage DevOps Learning Journey 🗺️

Stage 1: Master the Fundamentals 💻

Foundation Skills Every DevOps Engineer Needs

Before diving into advanced tools, you need rock-solid fundamentals:

🐧 Linux/Unix Systems: Command line proficiency is essential
📜 Shell Scripting (Bash): Automate repetitive tasks efficiently
🔀 Version Control (Git): Collaborate effectively with development teams
🐍 Basic Programming: Python or Go for automation scripts
🌐 Networking Fundamentals: Understand how services communicate

💡 Pro Tip: Don't rush this stage. These skills form the foundation for everything else.

What You'll Build: Personal development environment, system monitoring scripts, automated backup solutions.

Stage 2: Infrastructure as Code 🏗️

Manage Infrastructure Through Code

Infrastructure as Code (IaC) transforms how we manage infrastructure:

⚙️ Terraform: Industry standard for multi-cloud infrastructure
🔧 Ansible: Configuration management and application deployment
☁️ CloudFormation: AWS-native infrastructure provisioning
✅ Infrastructure Testing: Validate changes before deployment

Real Impact: Companies achieve consistent, reproducible deployments while reducing manual configuration errors.

What You'll Build: Multi-environment infrastructure, automated web application stacks, infrastructure testing pipelines.

Stage 3: Containerization & Orchestration 📦

Package and Orchestrate Applications

Container technology has revolutionized application deployment:

🐳 Docker Fundamentals: Package applications consistently
🔗 Container Networking: Understand service communication
⚓ Kubernetes: Orchestrate containers at scale
📋 Helm Charts: Simplify Kubernetes application deployment
🔒 Container Security: Protect your containerized workloads

Why It Matters: Containers solve the "it works on my machine" problem and enable consistent deployments across environments.

What You'll Build: Microservices e-commerce platform, container CI/CD pipeline, production-ready Kubernetes cluster.

Stage 4: CI/CD Pipelines ⚡

Automate Your Deployment Process

Continuous Integration and Deployment revolutionize software delivery:

🚀 GitHub Actions: Automate workflows directly in your repository
🔄 Jenkins: Build complex, enterprise-grade pipelines
🦊 GitLab CI: Integrated DevOps platform
🎯 ArgoCD: GitOps-style deployments
🧪 Testing Automation: Integrate quality gates

Game Changer: Teams can deploy changes safely and frequently, with automatic rollback capabilities.

What You'll Build: Multi-stage CI/CD pipeline, GitOps deployment system, blue-green deployment strategy.

Stage 5: Cloud Platforms ☁️

Master Modern Cloud Infrastructure

Cloud expertise is essential in today's landscape:

🌐 AWS Fundamentals: Learn the most widely adopted cloud platform
🔷 Azure Services: Microsoft's comprehensive cloud ecosystem
🔵 Google Cloud Platform: Strong in data and AI services
🌍 Multi-Cloud Strategy: Many organizations use multiple providers
💰 Cost Optimization: Control and reduce cloud spending

Industry Reality: Most organizations have moved to cloud-first strategies, making these skills essential.

What You'll Build: Multi-cloud architecture, serverless application suite, cost optimization dashboard.

Stage 6: Monitoring & Observability 📊

Ensure System Reliability and Performance

Observability provides visibility into system behavior:

📈 Prometheus & Grafana: Industry-standard metrics and visualization
📋 ELK Stack: Centralized logging and analysis
🔍 Distributed Tracing: Track requests across microservices
⚡ APM Tools: Application performance monitoring
🎯 SLO/SLI Design: Define and measure service reliability

Critical Importance: You can't improve what you can't measure. Monitoring prevents small issues from becoming major outages.

What You'll Build: Complete observability stack, SLO monitoring dashboard, performance analysis tools.

Stage 7: Security & Compliance 🛡️

Integrate Security Throughout the Pipeline

Security must be built-in, not bolted-on:

🔐 DevSecOps Practices: Shift security left in the development process
🛡️ Container Security: Secure runtime and images
🔑 Secrets Management: Handle credentials safely
📋 Compliance Automation: Automate SOC2, GDPR requirements
🔍 Security Scanning: Integrate vulnerability detection

Modern Approach: Security teams collaborate with development from day one, rather than reviewing at the end.

What You'll Build: Secure CI/CD pipeline, zero-trust network, compliance automation systems.

Stage 8: Database Management 🗄️

Handle Data Persistence and Reliability

Data management remains critical across all applications:

🗃️ SQL & NoSQL: Master both relational and document databases
🤖 Database Automation: Automate deployments and migrations
💾 Backup Strategies: Ensure data recovery capabilities
⚡ Performance Tuning: Optimize database performance
☁️ Cloud Databases: Leverage managed database services

Universal Need: Every application needs data persistence, making these skills valuable across all projects.

What You'll Build: Database migration pipeline, multi-database architecture, database monitoring system.

Stage 9: Continuous Learning 🎓

Embrace Lifelong Growth

Technology evolves rapidly, making continuous learning essential:

🌍 Open Source Contribution: Build your reputation in the community
✍️ Technical Writing: Share knowledge and build authority
👥 Mentoring: Guide others and develop leadership skills
🎤 Conference Participation: Stay current with industry trends
🛠️ Side Projects: Experiment with new technologies

Long-term Success: The most successful DevOps engineers are those who adapt and grow with the technology landscape.

What You'll Build: Open source contributions, technical blog series, mentorship programs.

Practical Learning Approach 🎯

1. Hands-On Projects Beat Theory 🔨

Don't just read about tools, build with them:

Set up a complete development environment
Create infrastructure across multiple cloud providers
Build and deploy a real application end-to-end
Implement comprehensive monitoring and alerting

2. Learn in Public 📢

Document your journey and help others:

Write blog posts about your learnings and challenges
Share code and configurations on GitHub
Participate in DevOps communities and forums
Help others troubleshoot problems you've solved

3. Focus on Problem-Solving 🧩

DevOps is about solving business problems:

Understand why tools exist, not just how to use them
Practice troubleshooting and debugging systematically
Learn to communicate with both technical and business stakeholders
Think about reliability, scalability, and maintainability

4. Embrace AI as a Tool 🤖

Learn to work alongside AI rather than compete with it:

Use AI-powered tools to enhance your productivity
Understand how to deploy and manage AI workloads
Learn about MLOps practices and AI model lifecycle management
Focus on the strategic and creative aspects that AI can't replace

Industry Trends to Watch in 2025 🔮

🏗️ Platform Engineering Rise

Organizations are investing in internal developer platforms to improve developer experience and reduce cognitive load.

🔄 GitOps Adoption

Git-based deployment workflows are becoming the standard for managing infrastructure and applications.

🤖 AI/ML Integration & Infrastructure Demands

AI is transforming DevOps in multiple ways:

🧠 AI-Powered Tools: Intelligent monitoring, predictive scaling, and automated incident response
🚀 MLOps Emergence: New discipline combining ML and DevOps practices
⚡ GPU Infrastructure: Managing specialized hardware for AI workloads
📊 AI Model Pipelines: Deploying and updating AI models safely at scale

🌱 Sustainability Focus

Green DevOps practices are becoming important as organizations focus on reducing their environmental impact, especially with energy-intensive AI workloads.

🔒 Security-First Mindset

Security considerations are moving earlier in the development lifecycle, making DevSecOps skills increasingly valuable, particularly for protecting AI models and data.

Your Next Steps 🚶‍♂️

Starting your DevOps journey can feel overwhelming, but remember: every expert was once a beginner. Here's how to begin:

📚 Start with the Fundamentals: Master Linux, Git, and basic programming
⏰ Practice Consistently: Dedicate time each day to hands-on learning
👥 Join Communities: Connect with other learners and experienced practitioners
🔨 Build Projects: Apply your knowledge to real-world scenarios
🔍 Stay Curious: Technology evolves rapidly, embrace continuous learning
📝 Document Everything: Keep notes and share your learning journey

Interactive Learning Resources 🎮

While this roadmap provides the structure, hands-on practice is essential. For an interactive experience with curated resources, practice labs, and detailed guidance for each skill, check out the complete DevOps roadmap.

The interactive version includes:

📚 Curated learning resources for each skill
💻 Hands-on project ideas with difficulty levels
🎯 Skills assessment and progress tracking
🔗 Direct links to tutorials, documentation, and practice platforms
🏆 Achievement badges and learning milestones
💡 Real-world examples and use cases

Conclusion 🎉

The DevOps field offers tremendous opportunities for those willing to invest in learning and skill development. Even in an AI-driven world, infrastructure expertise becomes more valuable, not less. As AI applications proliferate, they all depend on the robust, scalable systems that DevOps engineers build and maintain.

With the right roadmap and consistent effort, you can build a rewarding career that combines technical challenges with meaningful business impact. The rise of AI doesn't diminish the importance of DevOps, it amplifies it.

Remember, the goal isn't to master everything at once. Focus on building a strong foundation, then gradually expand your expertise. The industry rewards competence, problem-solving ability, and continuous learning, all qualities that define successful DevOps engineers.

The journey may seem long, but every step builds upon the previous one. Start where you are, use what you have, and do what you can. Your future self will thank you for starting today.

Start your journey now. The DevOps community is welcoming and always ready to help newcomers succeed! 🌟

What's your current position on this roadmap? Share your DevOps learning journey in the comments below! 💬

The 10 Most Common DevOps Mistakes (And How to Avoid Them in 2025)

DevOps Daily — Mon, 21 Jul 2025 11:00:00 +0000

DevOps isn't just about shipping code faster, it's about doing it smarter, safer, and saner. But let's be real: even the best teams make mistakes. Some are harmless. Others take down production on a Friday afternoon (yes, that Friday deploy).

Here are 10 common DevOps mistakes in 2025, how to avoid them, and a few moments that might hit a little too close to home.

1. Treating Infrastructure as Code Like a One-Off Script

You wrote Terraform once, it worked, and now it lives untouched in a dusty repo folder. That's not IaC, that's tech debt.

Avoid it:

Version control your IaC.
Apply formatting and linting.
Test it with tools like terraform plan or terratest.

2. Not Enforcing Version Control on CI/CD Configs

Your pipeline files are changing, but without versioning, there's no easy way to debug regressions.

Avoid it:

Store all CI/CD config files (like GitHub Actions, GitLab CI, etc.) in version control.
Treat pipeline logic like any other critical code.

3. Poor Secrets Management

Hardcoding secrets in code or using .env files without encryption is a fast way to land on HN for the wrong reasons.

Avoid it:

Use Vault, Doppler, AWS Secrets Manager, or SOPS.
Rotate secrets regularly.

4. No Rollback Strategy

You deploy. Something breaks. And there's no plan B.

Avoid it:

Use blue-green or canary deployments.
Automate rollbacks on failure.
Always have a rollback.sh or previous image ready.

5. Ignoring Observability Until It's Too Late

Monitoring isn't just about uptime. You can't fix what you can't see.

Avoid it:

Add metrics, logs, and traces from day one.
Use tools like Prometheus, Grafana, and OpenTelemetry.

6. Too Many Tools, Not Enough Integration

Your stack has 25 tools. None of them talk to each other. And your alert fatigue is real.

Avoid it:

Consolidate tools where possible.
Favor tools that integrate well with your existing stack.

7. Manual Approval for Every Tiny Change

A typo fix shouldn't need a 3-person review and a Slack war.

Avoid it:

Set up clear policies: auto-approve safe changes, gate critical ones.
Use GitHub environments, OPA, or custom bots to help.

8. No Documentation = Single Point of Failure

"Ask Alex, they built it." Alex is on vacation.

Avoid it:

Write docs as you go.
Use tools like Backstage, Docusaurus, or just plain Markdown.
Encourage a culture of async knowledge sharing.

9. Skipping Tests for Infrastructure Changes

You test app code, but deploy infra changes directly to prod? Bold.

Avoid it:

Use staging or preview environments.
Test IaC with checkov, terratest, or kitchen.

10. Forgetting Security in Your Pipelines

If your pipeline can deploy to prod, attackers might be able to as well.

Avoid it:

Use least privilege for pipeline credentials.
Run security checks like trivy, semgrep, and snyk.

Final Thoughts

DevOps is a journey. These mistakes are all lessons learned the hard way by teams around the world, and probably you, if you've been around long enough.

Want to avoid these mistakes before they cost you time, sleep, or your weekend? We're building checklists, guides, and battle-tested content at DevOps Daily. Come hang out.

PS: Got a DevOps horror story or lesson to share? Drop it in the comments or tag us on Twitter.

What's Your Go-To Stack for Personal Projects in 2025?

DevOps Daily — Fri, 18 Jul 2025 17:07:49 +0000

When you're building a side project in 2025, what's your default stack these days?

Are you still loving the reliability of Laravel or Ruby on Rails, or have you fully embraced Next.js, Bun, or something even more bleeding edge? Maybe you're mixing in tools like Supabase, Neon, or HTMX?

Curious to hear:

What's your go-to stack for quick MVPs or weekend builds?
Do you keep it simple or try to mirror production setups?
What are you hosting it on?

Been thinking about this a lot while working on something for DevOps Daily and it made me wonder what others are using this year.

Drop your stack below, someone might discover their next favorite combo from your setup!