DEV Community: James Joyner

DevOps as a Service Pricing: What Should Businesses Expect to Pay?

James Joyner — Mon, 29 Jun 2026 22:02:07 +0000

After 25 years of keeping production systems alive — building the automation, owning the pager, and helping companies stop bleeding money on preventable outages — the question I get asked most by founders and operations leads is blunt: "What is this going to cost me?"

The honest answer is the one nobody likes: it depends. But "it depends" isn't useful if you're trying to budget. So let me give you the real version — what drives the number, the pricing models you'll actually be quoted, and a simple way to figure out whether the spend pays for itself.

Why DevOps pricing varies so much

There's no sticker price on DevOps for the same reason there's no sticker price on "fixing my house." A one-bedroom condo and a 40-year-old farmhouse are different jobs. Three things move the number more than anything else:

Company size. A two-person startup with one Linux server and a single web app is a fundamentally different engagement than a 200-person company running multiple Kubernetes clusters across regions.
Infrastructure complexity. A static site on a single cloud VM is cheap to run. A microservices platform with service meshes, multiple databases, message queues, and compliance requirements is not.
Support expectations. "Help us when something breaks during business hours" and "24/7 on-call with a 15-minute response SLA" are priced an order of magnitude apart, because one of them owns someone's nights and weekends.

Before you can compare quotes, you have to be honest about which of those buckets you're actually in. A provider quoting you a low number may simply be assuming a smaller scope than the one you need.

The common pricing models

Most DevOps as a Service work is sold under one of five models. Each fits a different situation, and good providers will steer you toward the right one rather than forcing everything into their favorite.

Hourly / time-and-materials

You pay for hours worked, usually billed against a monthly cap.

When it fits: Small, well-defined tasks, ad-hoc help, or an early relationship where neither side knows the full scope yet.
Rough ballpark: Rates vary widely by region and seniority. The trap is that hourly incentivizes activity, not outcomes — a cheap hourly rate from someone who takes three times as long is not a bargain.

Monthly retainer

A fixed monthly fee buys you a block of capacity and ongoing ownership of your infrastructure.

When it fits: You have living infrastructure that needs continuous care — patching, monitoring, upgrades, small improvements — and you want a predictable line item.
Example: Ongoing Kubernetes version upgrades, Prometheus and Grafana tuning, and routine Ansible-driven patching of your Linux fleet are classic retainer work. The cluster doesn't stop needing attention, so neither does the engagement.

Project-based / fixed bid

A scoped deliverable for a fixed price.

When it fits: A clear, bounded build with a defined "done."
Example: A one-time Terraform plus GitLab CI/CD build-out — provision the cloud accounts, write the infrastructure as code, stand up the pipelines, Dockerize the apps, and hand it over — is naturally project-priced. You know what you're getting and what it costs before work starts.

Emergency / incident support

On-demand help when production is on fire, often at a premium rate or via a pre-paid response retainer.

When it fits: You run your own systems day-to-day but want a number to call when something serious breaks.
Reality check: This is the most expensive way to buy help per hour, because you're paying for someone to drop everything. It's insurance, not a maintenance plan — and it's far cheaper to prevent the incident than to buy emergency labor mid-outage.

Fully managed service

The provider owns your DevOps function end to end — infrastructure, pipelines, monitoring, security, on-call, the lot.

When it fits: You don't want to hire and retain an internal platform team, or you want to extend the small one you have.
Reality check: This is the highest monthly spend, but compare it against the loaded cost of hiring senior engineers, the recruiting time, and the bus-factor risk of a one-person internal team. Often it's cheaper and far less fragile than building the same capability in-house.

A healthy engagement often mixes models: a project-priced initial build-out, then a monthly retainer to run what was built.

What services actually move the price

Within any model, the scope of work is what sets the number. The big cost factors:

Cloud setup and infrastructure as code. Account structure, networking, and Terraform modules to make it all reproducible.
CI/CD pipelines. Building and maintaining GitLab CI/CD (or equivalent) so deploys are fast, repeatable, and safe.
Containers and orchestration. Docker images, registries, and Kubernetes — the single biggest complexity multiplier in modern infrastructure.
Monitoring and observability. Prometheus, Grafana, alerting rules, and dashboards. Good monitoring and alert generation is what turns a 3am outage into a 9am ticket.
Security. Secrets management, access control, network policy, vulnerability scanning, and hardening of your Linux servers.
Backups and disaster recovery. Tested restores — not just backups that exist on paper.
Incident response and on-call. The cost of someone being awake and accountable when things go wrong.
Automation. Ansible playbooks and scripting that replace manual, error-prone toil.
Compliance. SOC 2, HIPAA, PCI, and friends add audit, documentation, and control work that materially raises cost.

The more of these you need, and the higher the stakes, the higher the price. That's not padding — it's the actual work of keeping a real system running.

Why cheaper is not always better

Here's where my experience makes me opinionated: in production infrastructure, the cheapest quote is frequently the most expensive decision.

A low bid usually means one of a few things — a junior engineer learning on your dime, a scope that quietly excludes monitoring or backups, or a contractor who'll bolt something together and disappear before the technical debt comes due. You don't find out until the pipeline breaks at the worst possible moment, the backups turn out to be untested, or a security gap becomes an incident.

Infrastructure is one of those areas where you're not buying hours — you're buying the absence of disasters. That's hard to see on an invoice and very easy to feel in an outage.

What downtime actually costs

This is the framing that changes the conversation. Put a number on downtime and the "expensive" DevOps quote suddenly looks like a rounding error.

A simple cost-of-downtime model:

Downtime cost per hour = (Annual revenue / Business hours per year) + recovery labor + reputation/churn cost

Work a concrete example. Say a business does $5,000,000 in revenue a year and runs roughly 3,000 business hours. That's about $1,667 per hour in direct lost revenue — before you add the engineers pulled off roadmap work to firefight, the customers who churn, and the support load from a public incident. Call it $2,500–$4,000 an hour, conservatively.

Now consider what causes that downtime in shops without proper DevOps:

Failed deployments with no pipeline safeguards or rollback — a bad release that takes hours to unwind.
Poor monitoring that means you learn about the outage from angry customers instead of an alert, adding 30+ minutes of pure detection delay to every incident.
Manual, undocumented processes where only one person knows how to restore the service, and they're on vacation.

A single multi-hour outage can cost more than a year of competent monitoring and incident-response coverage. The DevOps spend isn't competing with zero — it's competing with the outages it prevents.

How AI changes the math

Part of why DevOps value-for-money has improved is that AI now removes a large slice of the repetitive labor that used to fill the bill.

Drafting and reviewing infrastructure as code. Terraform and Ansible scaffolding that used to take hours gets drafted in minutes, then reviewed by a human.
Pipeline and config generation. GitLab CI/CD configs, Dockerfiles, and Kubernetes manifests start from a solid AI-generated baseline instead of a blank file.
Monitoring setup. Generating sensible Prometheus alert rules and Grafana panels — historically tedious, easily templated work — is far faster with AI assistance.
Incident triage. Summarizing logs and correlating "what changed" compresses the slow part of an outage.

The key word is assisted — a human still owns every change to production. But a provider using AI well can deliver more per dollar, which means you get broader coverage for the same budget. If you want to see the kind of work this accelerates, our prompt library shows the patterns we lean on.

Starting lean: startups and small businesses

If you're early-stage, you do not need a fully managed enterprise engagement, and you shouldn't pay for one. Start with a lean package that covers the essentials and nothing you won't use yet:

A reproducible cloud setup with Terraform, so you're never clicking around a console by hand.
One clean CI/CD pipeline so deploys are boring and repeatable.
Basic monitoring and alerting on the handful of metrics that actually predict outages.
Tested backups.
A documented runbook so recovery doesn't depend on one person's memory.

That's a modest retainer or a small fixed-bid build-out, and it removes the failure modes that sink small companies. You add Kubernetes, deeper observability, and compliance work later — when the business actually needs them, not before. You can see how we structure tiers like this on our pricing page.

How to calculate ROI

Don't buy DevOps on vibes. Run the numbers. A usable formula:

ROI (%) = ((Value gained - Cost of service) / Cost of service) x 100

Where value gained is the sum of:

Downtime avoided — fewer outage hours × your cost-of-downtime-per-hour.
Engineering time reclaimed — hours your developers stop spending on infrastructure toil, at their loaded cost.
Faster delivery — features shipped sooner because the pipeline is fast and reliable.
Incidents prevented — the emergency-rate firefighting you never have to buy.

A worked example. Suppose a managed engagement costs $60,000 a year. Over that year it:

Prevents an estimated 20 hours of downtime at $3,000/hour = $60,000.
Frees two developers from ~5 hours/week of infra work — roughly $50,000 of reclaimed engineering time.
Speeds delivery enough to pull in revenue you'd otherwise have deferred — call it $30,000, conservatively.

That's $140,000 of value against $60,000 of cost:

ROI = (($140,000 - $60,000) / $60,000) x 100 = ~133%

Even if you halve every one of those estimates to be safe, you're still solidly positive. The exercise matters more than the exact figures — when you actually price the downtime you avoid and the time you reclaim, good DevOps consistently pays for itself.

The bottom line

DevOps as a Service pricing genuinely varies, and any provider who hands you a flat number without understanding your systems is guessing. But the framework is straightforward: know which size and complexity bucket you're in, pick the pricing model that fits the work, scope the services you actually need, and run the ROI math against the very real cost of doing nothing.

The mistake I see most often is treating DevOps as a cost line to minimize. It isn't. It's an investment in uptime, delivery speed, security, and the ability to scale without setting your infrastructure on fire. Price it against the outages, the lost engineering hours, and the deals you can't close because the platform won't hold — and the question stops being "what does this cost?" and becomes "what is it costing me not to have it?"

Cost figures and ranges here are illustrative. Build your own estimate from your real revenue, infrastructure, and risk profile before committing to a budget.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

Best DevSecOps Security Tools for CI/CD Pipeline Protection

James Joyner — Sun, 28 Jun 2026 05:05:55 +0000

I've spent twenty-five years building and securing deployment pipelines, and the single biggest shift in that time isn't a tool — it's where security lives. We used to bolt it on at the end, right before a release, when changing anything was expensive and everyone was already tired. That's backwards. DevSecOps is the correction: you move security checks left, into the pipeline, so problems surface when they're cheap to fix and the person who introduced them is still looking at the code.

This is a tour of the tool categories that matter, with representative (mostly open-source) examples and where each one fits in a real GitLab CI/CD or GitHub Actions pipeline. It is not a ranked ad. The right toolchain depends on your team size and how mature your infrastructure is, and I'll come back to that at the end.

What DevSecOps actually means

DevSecOps is "shift-left security": treating security as a property of the pipeline, not a gate at the end of it. Concretely, it means your CI runs the same checks a security reviewer would — scanning code, dependencies, containers, infrastructure definitions, and secrets — automatically, on every push, and fails the build when it finds something that genuinely matters.

The goal isn't to drown developers in findings. It's to catch the dangerous classes of mistakes early and consistently, so security becomes muscle memory instead of a quarterly fire drill.

Why the CI/CD pipeline is a prime target

Your pipeline is the most privileged thing in your engineering org and the least watched. It has credentials to your cloud, your registry, and production. That makes it a high-value target in ways teams routinely underestimate:

Supply-chain attacks. A compromised dependency or a malicious GitHub Action runs inside your build with your secrets in the environment. SolarWinds and the Codecov breach were both pipeline-level compromises, not application bugs.
Secrets sprawl. Pipelines are where API keys, cloud credentials, and registry tokens live. A leaked CI variable or a key hardcoded in a .gitlab-ci.yml is a direct path to your infrastructure.
Runner trust. Shared or self-hosted runners that build untrusted code can be poisoned. A pull request from a fork that triggers a privileged job is a classic foothold.
Artifact tampering. If nothing verifies that the image you deploy is the image you built, an attacker who can write to your registry can swap it.

Every category below is a defense against one or more of these.

SAST — Static Application Security Testing

What it does: scans your source code without running it, looking for vulnerable patterns — SQL injection, command injection, unsafe deserialization, hardcoded crypto mistakes.

Where it fits: early, on every merge request, as a fast job that runs before anything gets built.

Representative tools: Semgrep is my default — it's open-source, fast, and its rules read like the code they match, so writing a custom rule for your own footguns takes minutes. GitLab ships a built-in SAST analyzer you can enable with a single include in your .gitlab-ci.yml. For Python-specific work, Bandit is a lightweight option.

Practical tip: run SAST in diff mode on merge requests so it only flags new findings. Nothing kills adoption faster than a first run that reports 4,000 pre-existing issues and blocks everyone. Baseline the legacy debt, enforce on new code.

DAST — Dynamic Application Security Testing

What it does: attacks a running instance of your app the way an external scanner would — probing for XSS, injection, misconfigured headers, and exposed endpoints.

Where it fits: later in the pipeline, after you've deployed the build to an ephemeral staging environment.

Representative tools: OWASP ZAP is the open-source standard. Its baseline scan runs cleanly as a Docker container in a CI job — point it at your staging URL and it produces a report you can fail the pipeline on.

Practical tip: DAST is slower and noisier than SAST, so don't gate every merge request on a full active scan. Run a quick ZAP baseline scan on merge requests and a deeper scan nightly against staging.

SCA / dependency scanning

What it does: Software Composition Analysis inventories your third-party dependencies and flags ones with known CVEs. Most of the code in your app isn't yours, and this is where most exploitable vulnerabilities actually live.

Where it fits: on every build, and continuously in the background as new CVEs are published against dependencies you already shipped.

Representative tools: Trivy and Grype both scan lockfiles and dependency manifests for vulnerabilities, open-source and fast. For fixing — not just finding — Dependabot (GitHub-native) and Renovate (works everywhere, including GitLab) open automated PRs that bump vulnerable packages.

Practical tip: pair scanning with automated updates. Trivy tells you you're exposed; Renovate is what closes the gap before it rots into a quarter-long upgrade project. A scanner with no remediation path just generates anxiety.

Container image scanning

What it does: scans the layers of a built Docker image — OS packages and language dependencies baked into it — for known vulnerabilities. The friendly node:18 base image you pulled six months ago has accumulated CVEs since.

Where it fits: right after you build the image, before you push it to your registry.

Representative tools: Trivy again (it does both SCA and image scanning, which is why it's so widely deployed), Grype, and Clair, which underpins several registries' built-in scanning.

Practical tip: scan before the push and gate the push on the result. A concrete GitLab CI pattern:

container_scan:
  stage: scan
  image: aquasec/trivy:latest
  script:
    - trivy image --exit-code 1 --severity CRITICAL,HIGH "$IMAGE_TAG"

--exit-code 1 makes the job — and therefore the pipeline — fail on a critical finding, so a vulnerable image never reaches the registry in the first place.

Infrastructure as Code scanning

What it does: scans your Terraform, Ansible, CloudFormation, and Kubernetes manifests for insecure configuration before it provisions anything — public S3 buckets, security groups open to 0.0.0.0/0, unencrypted volumes, over-permissive IAM.

Where it fits: in the plan/validate stage, before terraform apply or ansible-playbook runs against real infrastructure.

Representative tools: Checkov is the broadest — it covers Terraform, Ansible, Kubernetes, and more out of the box. tfsec is Terraform-focused and fast (now converging with Trivy). KICS covers a wide spread of IaC formats.

Practical tip: scan the Terraform plan, not just the source, so you catch issues in the resolved configuration. In a GitHub Actions step:

- name: Checkov on Terraform plan
  run: |
    terraform plan -out=tfplan.binary
    terraform show -json tfplan.binary > tfplan.json
    checkov -f tfplan.json --quiet

This catches the misconfiguration before it becomes a public bucket someone finds for you.

Secrets detection

What it does: scans commits, history, and CI config for leaked credentials — AWS keys, tokens, private keys, passwords. This is the highest-leverage category for the effort involved, because a single leaked cloud key can cost you the whole environment.

Where it fits: everywhere — as a pre-commit hook on the developer's machine and as a CI job that scans the full diff.

Representative tools: Gitleaks and TruffleHog are the open-source workhorses. Run both through the pre-commit framework so secrets get caught before they ever hit the remote.

Practical tip: defense in depth. The pre-commit hook stops the honest mistake locally; the CI job catches the developer who skipped the hook with --no-verify. A combined Trivy-plus-Gitleaks GitLab job that fails the pipeline on a critical finding is one of the cheapest, highest-value things you can add this week.

Policy-as-code

What it does: lets you encode organizational rules as machine-checkable policy — "no container runs as root," "every image must come from our approved registry," "no resource without a cost-center tag" — and enforces them automatically.

Where it fits: two places. In CI, to validate manifests before merge. And at the Kubernetes admission layer, to block non-compliant workloads at deploy time.

Representative tools: OPA with Conftest for testing config files in CI (Terraform plans, Kubernetes YAML, Dockerfiles) against Rego policies. Kyverno for Kubernetes admission control, where policies are written as Kubernetes resources rather than a separate language — a gentler on-ramp for teams already fluent in YAML.

Practical tip: policy-as-code is how you turn "we agreed in a meeting that prod images must be signed" into something the cluster enforces. Start with one or two high-value policies in audit mode, watch what they'd block, then flip to enforce.

This is also the layer that defends against artifact tampering. Generate a signature at build time with cosign and gate the registry push — and the cluster admission — behind a valid signature. If the image isn't the one you built and signed, it doesn't deploy.

Runtime security monitoring

What it does: watches running workloads for suspicious behavior — a container spawning a shell, an unexpected outbound connection, a write to a sensitive path. This is the backstop for everything your pre-deploy scans missed.

Where it fits: in production, continuously, outside the pipeline — but it closes the DevSecOps loop by feeding real attack signals back to the people who build the pipeline.

Representative tools: Falco is the open-source standard, using eBPF to observe kernel-level syscalls with minimal overhead and alert on rule violations. The broader eBPF tooling ecosystem (Cilium, Tetragon) extends this into network policy and deep observability.

Practical tip: runtime security is your evidence that shift-left isn't perfect — and it never is. Treat a Falco alert as both an incident and a signal to add a check earlier in the pipeline so the same class of thing gets caught before deploy next time.

How to choose: team size and infrastructure maturity

You do not need all of the above on day one. Coverage you can't maintain is worse than honest gaps, because it breeds alert fatigue and trains your team to click past warnings.

Lean startup / small team. Pick the three highest-leverage, lowest-friction tools and run them as failing CI jobs: secrets detection (Gitleaks), dependency + image scanning (Trivy), and IaC scanning (Checkov) if you're running Terraform. That's maybe an afternoon of pipeline work and it eliminates the most common ways small teams get owned. Add Dependabot or Renovate so you're patching, not just panicking.

Mid-size, growing team. Layer in SAST in diff mode, DAST against staging, and image signing with cosign. Start introducing policy-as-code in audit mode so you understand your real posture before you enforce.

Mature org with regulatory pressure. Now the full stack earns its keep: enforced policy-as-code at admission, runtime monitoring with Falco, signed-and-verified artifacts end to end, and centralized reporting so security can see across teams. At this scale the integration and dashboards matter as much as the scanners.

The pattern is consistent: each category, run as a job that fails fast on real risk, beats a fancy tool that produces a report nobody reads. If you want to go deeper on pipeline patterns, our GitLab CI/CD prompts cover a lot of this ground.

Where AI fits — assistive, not authoritative

Modern security tooling generates a lot of output, and this is where AI genuinely earns its place. It's good at the reading and summarizing that wears humans down:

Triaging scan output. Paste a wall of Trivy or Semgrep findings and ask the model to group them, identify which CVEs are actually reachable in your code path, and rank by real exploitability. A list of 200 "HIGH" findings becomes "these 6 matter, here's why."
Explaining a finding. "What does this Checkov failure mean and what's the minimal Terraform change to fix it?" turns a cryptic policy ID into an actionable diff.
Drafting remediation. Generate the patched dependency version, the hardened Dockerfile, or the corrected security-group block — as a starting point you review.
Reading pipeline logs. Summarizing a failed, noisy CI job to find the one line that actually broke the build.

The non-negotiable rule, same as during an incident: AI summarizes and suggests; a human verifies and applies. A model will confidently mislabel a vulnerability's severity or propose a "fix" that doesn't compile. Use it to compress the toil, never to make the final security call. We keep a prompt library of these workflows, and the same judgment applies to AI-assisted infrastructure code review — assistive acceleration, human sign-off.

Conclusion

There is no single "best" DevSecOps toolchain. The best one is the one your team will actually use consistently. A perfect scanner that everyone learns to ignore protects nothing — coverage you don't act on is worthless. So start small: pick the tools that fit your pipeline, wire them in as jobs that fail fast on genuine risk, and earn the right to add the next layer. Secrets detection, dependency and image scanning, IaC checks, and signed artifacts will stop the overwhelming majority of how teams actually get compromised. Get those running and reliable first, keep the human in the loop on the AI-assisted parts, and grow the stack as your infrastructure — and your team's appetite — matures.

Security scan output and AI-generated remediations are assistive, not authoritative. Always verify findings and fixes against your own systems before applying them in production.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

Humanizing Artificial Intelligence for Log Analysis: Turning Raw Server Logs Into Clear DevOps Answers

James Joyner — Sat, 27 Jun 2026 19:21:04 +0000

It's 2:14 a.m. and my phone is buzzing because a customer's instance won't get a floating IP. The alert is one line. The truth is somewhere in about forty thousand lines spread across nova-compute, neutron-server, the OVS agent, and libvirtd — each with its own timestamp format, its own idea of what a "request" is, and its own favorite way of burying the actual error under a wall of stack traces. This is the part of the job nobody puts on a slide. You are not solving a hard problem yet. You are finding the problem, and finding it is grep, scroll, swear, repeat.

This is exactly where AI earns its keep — and exactly where most people misuse it. So let me be precise about what I mean by "humanizing AI" for log analysis, because the phrase has been beaten half to death by marketing.

AI reads. You decide.

Humanizing AI does not mean an autonomous bot that "handles the incident." It means using a language model for the one thing it is genuinely, freakishly good at: pattern-matching and correlating across huge volumes of text, and translating jargon into plain English. A model can read forty thousand lines faster than you can scroll one screen, notice that the req- ID in nova-compute shows up again in neutron-server 1.2 seconds later attached to a binding failure, and tell you that in a sentence.

What it must not do is pull the trigger. The model reads; you decide. It hands you ranked hypotheses and a command to verify them — never a "fix" it wants to apply on its own. You are still the engineer on the hook when the change goes sideways, so you stay the final decision-maker. Every workflow below is built around that rule. I wrote a fuller treatment of this on my site at devopsaitoolkit.com/blog/humanizing-artificial-intelligence-in-log-analysis, but the short version is: the human stays in the loop, and the loop is where the judgment lives.

Before anything else: redact

You are about to paste production logs into a model. Stop. Logs leak. They carry bearer tokens, Keystone auth tokens, DB connection strings, private IPs, customer email addresses, and the occasional password somebody logged "temporarily" in 2021. Treat every log line as hostile until you have stripped it.

I keep a redaction pass that runs before anything leaves the box:

journalctl -u nova-compute --since "10 min ago" --no-pager \
  | sed -E \
    -e 's/(password|passwd|secret|token|api[_-]?key)["'\'' :=]+[^ ,"]+/\1=REDACTED/gi' \
    -e 's/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/REDACTED_EMAIL/g' \
    -e 's/\b([0-9]{1,3}\.){3}[0-9]{1,3}\b/REDACTED_IP/g' \
    -e 's/Bearer [A-Za-z0-9._-]+/Bearer REDACTED/g' \
    > /tmp/nova-redacted.log

It is not perfect — no regex is — but it catches the obvious offenders, and it forces you to look at what you are about to share. Eyeball the output before you paste it.

Pro Tip: Build the redaction step into the same command that pulls the logs, never as a separate afterthought. The moment "redact later" becomes a step you do by hand, you will skip it at 2 a.m. when it matters most. A pipe that always redacts is a habit; a checklist item is a future incident.

Linux logs: journald and syslog

Start where most things start: the host. journalctl is your front door, and a model is far better at reading its firehose output than you are when you're tired.

journalctl -p err --since "today" --no-pager -o short-iso \
  | sed -E 's/Bearer [A-Za-z0-9._-]+/Bearer REDACTED/g' \
  > /tmp/host-errors.log

The mistake people make is pasting raw lines and asking "what's wrong?" That gets you a confident, useless summary. Give the model the shape of what you want: timeline, correlation, and a verification step. The prompt matters more than the model. I keep a running set of patterns for this in my journald-with-AI write-up, but the core ask is always the same: "Group these by service and time, tell me which errors are causes versus symptoms, and give me a command to confirm before I touch anything."

That last clause is the whole game. A model will happily tell you "restart the OVS agent." A humanized workflow makes it tell you how to check whether the OVS agent is actually the problem first.

Application and container logs: feed the right context

Container logs are where context discipline pays off. A crash-looping pod's current logs are often the least useful thing you can read — the interesting failure happened in the instance that already died. So you reach for the previous container:

kubectl logs deploy/payments -c api --previous --tail=500 \
  | sed -E 's/(authorization|cookie):.*/\1: REDACTED/gi' \
  > /tmp/payments-prev.log

Then pair it with the events, because the pod logs rarely tell you why Kubernetes killed the thing:

kubectl get events --field-selector involvedObject.name=payments-7d9f-abc \
  --sort-by=.lastTimestamp

Now you hand the model both: the previous container logs and the events. The events say "OOMKilled" or "readiness probe failed"; the application logs say what the process was doing in its last breath. The model's job is to connect those two stories. On its own, neither one is conclusive. Together they usually are — and if you only feed one, the model will hallucinate the other half. Garbage context in, confident nonsense out.

Pro Tip: When you share container logs with a model, always say which container and whether it's --previous or current. "Here are the logs" is a trap — the model can't see your kubectl flags, and a restart-loop's current logs versus its previous logs tell completely different stories. Label your evidence.

If your container logs live in Loki rather than kubectl, the same principle holds — you're just pulling from LogQL instead. I walked through that whole flow, including how to keep the model honest against a Loki backend, in reading Loki logs with AI. The pattern doesn't change with the backend; only the query language does.

Kubernetes events: the cheap, ignored goldmine

Events are the most under-read signal in a cluster. They expire, they're noisy, and they're written in half-jargon that's easy to skim past. That's precisely the kind of text a model parses well. Dump them wide and let it cluster:

kubectl get events -A --sort-by=.lastTimestamp \
  | grep -Ev "Normal\s+(Pulled|Created|Started|Scheduled)" \
  > /tmp/events.log

Ask the model to bucket these by root cause, not by namespace — "show me which events are likely the same underlying problem reported by different controllers." A human scanning this sees a thousand lines. The model sees three problems wearing a thousand costumes. You still decide which of the three is worth waking someone up for. If you want a deeper dive on the cluster-side patterns, I keep my Kubernetes material at devopsaitoolkit.com/categories/kubernetes-helm.

OpenStack: tracing one request across nova, neutron, and libvirt

Here's the 2 a.m. floating-IP problem, and it's the best example of why correlation beats reading. In OpenStack, a single API call fans out across services, and the only thread tying them together is the request ID. Find it, then chase it everywhere:

REQ="req-8f2c1a4e-..."   # pulled from the nova-api log for the failed call

grep -h "$REQ" \
  /var/log/nova/nova-compute.log \
  /var/log/neutron/neutron-server.log \
  /var/log/neutron/neutron-openvswitch-agent.log \
  | sort \
  | sed -E 's/\b([0-9]{1,3}\.){3}[0-9]{1,3}\b/REDACTED_IP/g' \
  > /tmp/req-trace.log

For the libvirt side, the request ID won't follow you — libvirtd and the per-instance QEMU logs key off the instance UUID and domain name instead, so you grab those by time window:

journalctl -u libvirtd --since "02:10" --until "02:16" --no-pager \
  > /tmp/libvirt-window.log

Now you've got a stitched-together story across four logs in three formats. A human can read this. It just takes twenty minutes of cross-referencing timestamps while a customer waits. The model does the cross-referencing in seconds and — crucially — tells you which service first reported a real failure versus which ones are just downstream noise reacting to it. That causal ordering is the hard part, and it's exactly what correlation-across-text is built for.

Here's what a genuinely humanized output looks like. Not a fix. Ranked hypotheses, confidence, and the next command to run:

Most likely (high confidence): The vif-plugged event from the OVS agent never reached nova-compute within the timeout. neutron-server logged the port as BUILD at 02:11:43 but never transitioned to ACTIVE. nova then timed out waiting for the network and rolled the instance back — so the floating-IP failure is a symptom, not the cause.
Verify before acting: openstack port show <PORT_ID> -f value -c status binding_vif_type — if status is DOWN or binding_failed, this hypothesis holds.

Less likely (medium confidence): OVS agent lost its connection to neutron-server (RabbitMQ hiccup). Check with journalctl -u neutron-openvswitch-agent --since 02:09 | grep -i "agent.*down\|AMQP".

Unlikely (low confidence): libvirt failed to attach the tap device. Only pursue if the port shows ACTIVE but the domain XML is missing the interface.

Read that again, because it models the behavior you want. It commits to an ordering. It separates cause from symptom. And every single hypothesis ends in a command you run to confirm or kill it — not an action the model took. You are still the one who runs openstack port show and reads the result with your own eyes. If you want more OpenStack-specific debugging patterns, that's a whole category on my site at devopsaitoolkit.com/categories/openstack.

Pro Tip: Demand the model rank by confidence AND mark each hypothesis as cause or symptom. The single most expensive mistake in an incident is chasing a loud symptom while the quiet root cause keeps burning. Forcing that label turns the model from a search engine into a triage partner — but you still own the triage.

Make the model show its work

The difference between a useful AI log workflow and a dangerous one comes down to one demand: show your reasoning and give me a command to verify. A model that says "it's the OVS agent, restart it" is a liability. A model that says "I think it's the OVS agent because these three log lines, and here's the openstack port show to confirm" is a colleague.

Build that demand into your prompt every time:

Quote the specific log lines you're reasoning from.
Rank hypotheses by confidence and label each as cause or symptom.
For each, give a read-only command that confirms or refutes it.
Never output a mutating command as a recommendation — only diagnostics.

That last rule keeps the human firmly in control. The model can suggest you look at something. It does not get to suggest you change something, because the change is your call, with your context, and your name on the change ticket. AI reads; you decide. If you want to see this wired up as an actual assistant rather than a copy-paste loop, I run a free one you can poke at over at devopsaitoolkit.com/dashboard/incident-response.

The part that's actually yours

Strip away the tooling and here's what's left. The model is a phenomenal reader and translator. It collapses forty thousand lines into three hypotheses, turns OpenStack's internal dialect into a sentence a sleep-deprived human can act on, and never gets bored on line thirty-nine thousand. That is real leverage, and refusing to use it out of purism is just making your nights longer.

But the decision is still yours. The model doesn't know that this customer is mid-migration and a "harmless" agent restart will nuke their in-flight transfer. It doesn't know your change-freeze window, your blast radius, or that the "obvious" fix burned you last quarter. That context lives in your head, and that's the irreducibly human part of the job. Humanizing AI means letting it do the reading so you have the energy left to do the deciding.

So redact your logs, feed the right context, demand the reasoning and the verification command — and then you run the command. That's the loop. That's the whole thing.

James Joyner IV runs devopsaitoolkit.com, where he writes about running production OpenStack, Kubernetes, and observability without losing his mind. Try the free AI Incident Response Assistant for ranked, verify-first log triage, and if you write about this stuff too, the Writing Humanizer pack keeps your prose sounding like you.

DevOps Security Best Practices Every Engineering Team Should Follow

James Joyner — Fri, 26 Jun 2026 14:01:11 +0000

I've spent 25 years securing Linux boxes, cloud accounts, CI/CD pipelines, and production clusters. The single most consistent lesson across all of it is this: the teams that get breached aren't the ones who lacked a security department. They're the ones who treated security as something a separate department would handle later.

Security is not a phase. It's not a gate at the end of the pipeline, and it's not a quarterly audit. It's a property of how you write infrastructure code, manage secrets, ship containers, and run production every single day. When security lives inside the daily workflow — in the merge request, the pipeline stage, the Terraform plan — it costs almost nothing. When it lives in a separate review at the end, it's expensive, late, and routinely skipped.

This is the checklist I'd hand a new engineering team. Everything here is defensive: hardening, detection, and recovery. Work through it section by section.

Why DevOps security belongs in the daily workflow

The whole premise of DevOps was to stop throwing work over the wall between dev and ops. Security is the last wall standing in most orgs, and it has to come down the same way: by moving the controls into the tools engineers already use.

Treat every pull/merge request as a security review surface, not just a code review.
Run security checks as pipeline stages that fail the build, not as advisory reports nobody reads.
Make the secure path the easy path — a hardened base image, a vetted Terraform module, a secrets helper — so engineers don't route around it.
Assign a security owner per service, not a security team for the whole company. Ownership beats oversight.
Measure mean-time-to-remediate for vulnerabilities the same way you measure deploy frequency.

If a control only exists in a wiki page, it doesn't exist. If it exists in the pipeline, it's real.

Secure access control and least privilege

Most incidents I've cleaned up came down to one over-privileged credential. Least privilege is boring and it's the highest-leverage thing on this list.

Default every IAM role, Kubernetes ServiceAccount, and Linux user to zero permissions, then add only what's needed.
Replace standing admin access with just-in-time elevation that expires automatically.
Scope cloud roles to specific resources and actions — no *:* policies, ever.
In Kubernetes, use RBAC Roles bound to namespaces rather than ClusterRole bindings wherever possible.
Separate human identities from machine identities. Humans get SSO; services get workload identity.
Audit who can sudo, who's in the docker group (that's root-equivalent), and who holds cloud admin — quarterly, in writing.

SSH key management and MFA

SSH is still how a huge amount of production gets touched, and it's still where credential hygiene quietly rots.

Disable password authentication entirely: PasswordAuthentication no and PermitRootLogin no in sshd_config.
Use per-user keys, never a shared key passed around in a chat thread.
Prefer short-lived SSH certificates from a CA over long-lived static keys; rotate the rest on a schedule.
Put a bastion/jump host in front of production and log every session through it.
Require MFA on every identity provider, VPN, and cloud console — phishing-resistant (WebAuthn/hardware keys) for anyone with production access.
Pull keys for departed team members the same day, and audit authorized_keys files for orphans.

Secrets management: API keys, passwords, and tokens

The fastest way to leak a secret is to commit it. The second fastest is to print it. Both are entirely preventable.

Never store secrets in git — not in code, not in .env, not in a "temporary" YAML file. Add a pre-commit secret scanner (gitleaks or trufflehog) to block it.
Centralize secrets in a real secrets manager: HashiCorp Vault, a cloud secrets manager, or equivalent.
For Kubernetes, use Sealed Secrets or an external-secrets operator so the cluster pulls from Vault at runtime — plain Secret objects are only base64, not encrypted.
Give every secret a rotation policy and an owner. Static credentials that never rotate are time bombs.
Inject secrets as runtime environment values or mounted files, not baked into container images or Terraform state.
Scan your git history, not just the current tree — a secret deleted in HEAD is still in the log until you rotate it.

CI/CD pipeline security

Your pipeline has credentials to everything. That makes it one of the highest-value targets you own, and it's frequently the least hardened.

Protect your main branches: require reviews, status checks, and signed commits before merge.
Mark CI/CD variables as protected and masked so they're only exposed on protected branches and never echoed to logs.
In GitLab CI, scope variables to environments and never echo a secret — masking helps, but the discipline of not printing it is what saves you.
Replace long-lived cloud keys in CI with short-lived credentials via OIDC. Let the pipeline exchange its identity for a temporary, scoped token instead of holding a static AWS_SECRET_ACCESS_KEY.
Pin and review your pipeline dependencies — third-party CI templates and actions run with your pipeline's privileges.
Require manual approval for production deploys, and make the deploy job itself least-privileged.

A leaked CI variable is a leaked production credential. Treat the pipeline config with the same care you'd treat root.

Container image scanning

A container is only as trustworthy as the layers underneath it. Most images ship with known CVEs the team never looked at.

Scan every image with Trivy (or Grype) as a GitLab pipeline stage before push, and fail the build on high/critical findings:

  container_scan:
    stage: test
    image: aquasec/trivy:latest
    script:
      - trivy image --exit-code 1 --severity HIGH,CRITICAL "$IMAGE_TAG"

Start from minimal base images (distroless or slim) to shrink the attack surface.
Run containers as a non-root user (USER in the Dockerfile) with a read-only root filesystem where possible.
Drop all Linux capabilities and add back only what's required.
Pin base images by digest, not by floating :latest tags, and rebuild regularly to pick up patches.
Sign images and verify signatures at admission so only your builds run in your cluster.

Infrastructure as Code security

IaC is where a one-line mistake becomes a fleet-wide misconfiguration. The good news: it's also where automated policy catches it before it ships.

Review Terraform and Ansible changes like application code — every change goes through a merge request with a human reviewer.
Run static analysis on IaC in the pipeline: tfsec/Checkov for Terraform, ansible-lint and kube-linter for the rest.
Adopt policy-as-code (OPA/Conftest or Sentinel) so rules like "no public S3 buckets" and "no 0.0.0.0/0 on port 22" are enforced automatically, not remembered by reviewers.
Protect and encrypt Terraform state — it contains secrets in plaintext. Use a remote backend with locking and access controls.
For Ansible, encrypt sensitive variables with Vault and avoid become where it isn't needed.
Diff the plan before every apply and require approval for changes to security groups, IAM, and networking.

If you want a structured second opinion on a risky module, an automated infrastructure code review catches the misconfigurations a tired reviewer skims past at the end of the day.

Patch management and vulnerability remediation

Unpatched systems are the most common root cause of breaches, and the least glamorous to fix. Make it routine so it isn't a decision.

Automate OS patching with unattended security updates on Linux, and rebuild container images on a cadence rather than letting them age.
Track your dependencies with SBOMs so you can answer "are we affected?" the day a CVE drops.
Subscribe to advisories for your stack and define an SLA: criticals patched in days, highs in a week or two.
Use Dependabot/Renovate to open dependency-bump PRs automatically and run them through your test suite.
Keep an inventory of every host, image, and cluster version — you can't patch what you don't know you run.

Monitoring, logging, and alerting for security events

You cannot respond to what you can't see. Detection is the difference between a contained incident and a postmortem that starts with "we think they were in for three months."

Enable Linux auditd and ship /var/log/auth.log, sudo events, and SSH activity to centralized, append-only storage.
Export security metrics to Prometheus and build Grafana dashboards plus alerts for anomalies: failed-login spikes, new sudo grants, unexpected outbound connections, root logins, container escapes.
Alert on auditd/SSH anomalies in Grafana — a burst of failed SSH from a new ASN, or a successful root login outside business hours, should page someone.
Turn on cloud audit logging (CloudTrail or equivalent) and alert on IAM policy changes, new access keys, and security-group edits.
Capture Kubernetes audit logs and alert on exec into production pods and changes to RBAC.
Keep logs immutable and retained long enough to investigate a slow-burn intrusion.

Backup and disaster recovery planning

Ransomware and fat-fingered terraform destroy have the same fix: backups you've actually tested. Untested backups are just hope.

Follow 3-2-1: three copies, two media types, one off-site and offline.
Keep at least one immutable/air-gapped copy that a compromised admin credential can't delete.
Encrypt backups at rest and control who can read and restore them.
Test restores on a schedule. A backup you've never restored is a guess, not a recovery plan.
Document RPO and RTO per service and verify your backup cadence actually meets them.
Back up the things people forget: Terraform state, Vault data, etcd, and database credentials.

Incident response preparation

The middle of an incident is the worst time to figure out your incident process. Prepare while it's calm.

Write a runbook: who's on call, how to declare an incident, where to communicate, and how to reach the right people fast.
Pre-define severity levels and the actions each triggers.
Keep break-glass credentials in a sealed, audited path — available in a crisis, logged when used.
Practice. Run a tabletop or game day at least quarterly so the steps are muscle memory.
Have communication templates ready — customer-facing and internal — so comms don't stall the investigation.
Draft blameless postmortems while the timeline is fresh, and turn action items into tracked work.

AI-assisted security checks — review, never blind trust

AI is genuinely useful for security work: it reads more YAML, Terraform, and logs than you can, and it's fast at spotting the misconfiguration buried in a 400-line diff. The rule is the same one I apply to everything an AI generates — it drafts, you decide.

Use AI to review IaC and pipeline configs for missing controls, over-broad permissions, and risky defaults — as a first-pass reviewer, not the final word.
Have it summarize scanner output and rank findings by real-world exploitability so your team fixes what matters first.
Let it draft hardening changes and policy rules, then read every line before you apply it — confident and correct are not the same thing.
Never paste live secrets, real hostnames, or customer data into a model. Scrub first.
Keep a human approving every change that touches IAM, networking, or production. AI accelerates the work; it doesn't own the risk.

If you want vetted starting points, our security & hardening prompts cover image scanning, Linux hardening, and IaC review, and the broader prompt library has the rest.

The bottom line

None of these practices are exotic. Least privilege, managed secrets, scanned images, policy-checked infrastructure, patched hosts, real monitoring, tested backups, and a rehearsed incident plan — every one is achievable this quarter, and every one is cheaper to build into the workflow than to bolt on after a breach.

And here's the part that doesn't show up on the engineering scorecard: secure DevOps is a competitive advantage. Customers run security questionnaires before they sign. Investors ask about your posture in diligence. Partners won't integrate with a platform they don't trust. The companies that win those deals are the ones who can show, not claim, that protecting customer systems is built into how they work every day.

Security isn't the tax you pay to ship. It's part of why people let you ship to them at all.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

Humanizing Artificial Intelligence for SRE Teams: Reducing Alert Fatigue With Smarter AI Guidance

James Joyner — Thu, 25 Jun 2026 14:27:13 +0000

The pager goes off at 3:11 a.m. It's the fifth time tonight, and it's the same alert: HighMemoryUsage on a node that's running a memory-mapped cache doing exactly what it was designed to do. You ack it half-asleep, knowing it'll fire again in twelve minutes. By the time the real incident shows up at 4:40 — a slow API degradation that's quietly eating your error budget — you're too fried to see it clearly. That's not a tooling failure. That's a design failure, and most of us have lived it.

I run production OpenStack, Kubernetes, Terraform, and the observability stack that watches all of it. I've spent more nights than I'd like fighting my own alerts. So when "AI for SRE" started showing up in every vendor deck, my first reaction was a tired no. The last thing on-call needs is an autonomous bot deciding to restart my database at 3 a.m. based on a hunch.

But there's a version of this that actually works, and it has nothing to do with autonomy. It's about using AI for the narrow set of things it's genuinely good at — clustering noise, summarizing storms, drafting hypotheses, surfacing the right runbook — while a human stays the final decision-maker. AI triages and proposes. You decide what pages a human and what gets fixed. That's what I mean by humanizing AI: not making the machine more human, but using the machine to keep the on-call human rested, focused, and in control.

The real problem is signal, not tooling

Alert fatigue isn't caused by too few dashboards. It's caused by alerts that don't map to a human decision. Every page should answer one question: does a person need to do something right now? If the answer is "no" or "not yet," it shouldn't page. We all know this. We violate it constantly because writing good alert rules is hard, and tuning them is a chore nobody prioritizes until they're drowning.

The honest baseline, before any AI enters the picture, is rule hygiene. If your alerts are symptom-based, tied to user-facing SLOs, and have sane thresholds, you've already eliminated most of the noise. I've written about this at length in designing alert rules that don't page you falsely, and the short version is: alert on what the user feels, not on what a single machine is doing. A node at 92% memory is not an incident. A checkout flow at 92% success rate is.

Here's a symptom-based, multi-window burn-rate alert — the kind that respects your error budget instead of paging on every blip:

groups:
  - name: slo-burn-rate
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          (
            job:slo_errors:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
            and
            job:slo_errors:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
          )
        for: 2m
        labels:
          severity: page
          slo: checkout-availability
        annotations:
          summary: "Checkout burning 30-day budget 14.4x too fast"
          runbook: "https://runbooks.internal/checkout-availability"

The 14.4 multiplier on a 0.1% error budget is the classic fast-burn threshold: at that rate you'd exhaust a 30-day budget in roughly two days, so it deserves a human now. Pair it with a slower window for the grind-it-down failures. If multi-window burn rate is new to you, this walkthrough is the most practical reference I keep handing to my team.

Pro Tip: Before you let AI touch a single alert, audit how many of your current pages are actionable. Pull a month of pages and bucket them: "I took an action," "I acked and ignored," "false positive." If more than a third land in the last two buckets, fix your rules first. AI applied to a noisy rule set just gives you faster, more confident noise.

Where AI earns its place: the alert storm

Rule hygiene gets you far, but it doesn't solve the 3 a.m. storm. When a top-of-rack switch flaps or a Kubernetes node goes NotReady, you don't get one clean alert. You get forty: pod restarts, probe failures, downstream latency, queue backups, dependent service timeouts. Each one is technically "real." Collectively, they're a wall of text that you have to mentally parse while half-conscious.

This is the single best place to put AI to work, because it's a summarization and clustering problem — exactly what large language models are good at. You feed the model the firing alerts, recent deploys, and relevant topology, and ask it to do what a senior SRE does instinctively: collapse the storm into a root cause plus its downstream effects.

A good humanized output looks like this:

Alert storm summary — 41 alerts, last 6 min
Most likely root cause (confidence: high): Node worker-prod-07 went NotReady at 03:08 UTC, 90s after the ingress-nginx rollout (deploy #4821). 38 of 41 alerts are pods evicted from or routing through this node.
Downstream effects (not separate incidents):

checkout p99 latency +400ms — pods rescheduling

payment-worker queue depth rising — consumers restarting

6x KubePodCrashLooping — all on worker-prod-07 Suggested next check: kubectl describe node worker-prod-07 and the kubelet logs around 03:07. Recommended runbook: node-notready-triage. This summary is advisory. No action has been taken.

That last line matters. The AI clustered 41 alerts into one root cause and three downstream effects, attached a confidence level, and pointed at the next diagnostic step. It did not cordon the node, restart anything, or roll back the deploy. It handed me a hypothesis and a starting point, and I decide whether it's right. That cuts my time-to-understanding from "read forty alerts" to "verify one claim" — which is exactly where you want AI to save you minutes, because those minutes directly improve your MTTA and MTTR. (If you want to be rigorous about which incident metrics actually matter, MTTA is the one that alert fatigue quietly wrecks.)

The mechanical clustering can — and should — live in Alertmanager. AI is the layer that explains the cluster in plain language; Alertmanager is the layer that suppresses the redundant pages so they never wake you in the first place.

Let Alertmanager do the suppression, let AI do the explaining

Don't ask the LLM to be your routing engine. Deterministic grouping and inhibition rules are reliable, auditable, and free. Use them to fold the storm down before it ever reaches a human, then let AI summarize what's left.

route:
  receiver: oncall-pager
  group_by: ['alertname', 'cluster', 'node']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = page
      receiver: oncall-pager
    - matchers:
        - severity = ticket
      receiver: ticket-queue

inhibit_rules:
  # If a node is NotReady, don't page on the pods it took down with it.
  - source_matchers:
      - alertname = KubeNodeNotReady
    target_matchers:
      - alertname =~ "KubePodCrashLooping|KubePodNotReady"
    equal: ['node']

That inhibition rule alone kills the most common storm pattern: a node failure dragging twenty pod alerts behind it. The equal: ['node'] clause scopes the suppression so a genuinely unrelated crashloop on a different node still pages. This is the deterministic floor. AI sits on top of it to explain the handful of alerts that remain, not to replace it. There's more nuance to getting this grouping right in cutting alert noise and designing alerts engineers actually trust, which pairs well with this approach.

Pro Tip: Treat every AI summary as a hypothesis with a confidence score, never a verdict. Require the model to emit a "next check" — a concrete command or query a human can run to confirm or refute it in under a minute. A confident-sounding root cause you can't quickly verify is worse than no suggestion at all, because it anchors your tired brain to the wrong path.

Prioritization: what actually deserves a human at 3 a.m.

Clustering tells you what is happening. Prioritization tells you whether it can wait until morning. This is the judgment call that burns out on-call engineers, and it's another place AI can genuinely help — as long as it ranks and recommends rather than decides.

The signal AI is good at synthesizing for prioritization:

Blast radius: how many users, services, or SLOs are affected, pulled from the alert labels and topology.
Budget velocity: is this burning error budget fast, slow, or not at all? A fast burn pages; a slow grind opens a ticket.
Business context: is checkout down at peak, or is an internal batch job late at 4 a.m.?
Recent change correlation: did a deploy, config push, or Terraform apply land in the last 30 minutes?

Feed those signals to the model and ask it to propose a severity and a routing decision with reasoning. The output should read like a recommendation you can override in one click:

Recommendation: open ticket, do not page (confidence: medium).
BatchExportLatencyHigh is firing, but it's an internal nightly export with no SLO and no user impact. Error budget for the user-facing API is untouched. No deploy correlated. Suggest ticket for the morning; re-evaluate if it's still firing after 2 retries.

You, the human, glance at that and either agree or say "actually that export feeds the 6 a.m. finance report, page me." The AI never had the business context that the export feeds finance — and that's the point. It proposes; you supply the judgment it can't have. Over time you encode that judgment back into the rules so the AI gets it right next time. For a broader survey of what's actually useful here, I keep a running list of the best AI tools for SRE teams and what each is realistically good for.

Surfacing the runbook, not writing the fix

When you do get paged for something real, the next time-sink is finding the right runbook and remembering the current state of the system. AI is good at retrieval and assembly: given the firing alert and its labels, it can surface the most relevant runbook, pull the last three related incidents, and assemble a quick situation brief.

What it should produce:

The runbook link that matches this alert's runbook annotation, plus any siblings.
A one-paragraph history: "This alert fired twice in the last 30 days; both were resolved by draining the node, mean resolution 14 minutes."
The current relevant state: recent deploys, open changes, who's already in the channel.

What it should not produce: the executed remediation. The model can draft the kubectl drain command and explain why. A human reads it, sanity-checks the node, and runs it. The gap between "here's the command" and "I ran the command for you" is the entire safety margin of this approach. Drain the wrong node and you've turned one incident into two.

Pro Tip: Wire your runbook links directly into alert annotations, like the runbook: field in the burn-rate example above. AI retrieval is dramatically more reliable when every alert already carries a canonical pointer — you're asking the model to fetch and summarize a known document, not to guess which runbook applies. Garbage retrieval in, confident-but-wrong brief out.

Closing the loop without a bot closing it for you

The part everyone skips is the follow-through. The storm ends, the incident resolves, and the action items — "tune that node alert," "add an inhibition rule," "fix the probe timeout" — evaporate into a Slack thread nobody reads again. Then the same storm wakes you next month.

AI helps here too, in its lane: it can draft the postmortem timeline from the alert and chat history, extract the action items, and propose the specific alert-rule changes that would have prevented the storm. But a human owns each item and a human merges the rule change. The discipline of actually closing the loop on incident action items is what turns last night's pain into next month's quiet. AI can draft the fix; it can't care that it ships. That's still on us.

This is the feedback loop that makes the whole system humane over time. Every overridden AI recommendation, every false page you tune out, every storm you cluster becomes encoded knowledge — better rules, better inhibition, better routing. The AI gets a sharper picture, the human gets quieter nights, and the error budget stays honest. If you want a deeper tour of the patterns, the incident response category on the site collects the workflows I actually run.

The line you don't cross

Let me be blunt about the boundary, because the vendors won't be. AI in your on-call loop should do four things: cluster noise, summarize storms, draft hypotheses, and surface runbooks. It should rank and recommend with explicit confidence levels and a verifiable next check. It should never silently take an action against production, never auto-close an incident, never decide on its own that something doesn't deserve a human.

The moment you hand the machine the keys to remediation, you've traded one kind of 3 a.m. terror — the noisy pager — for a worse one: the bot that did something you didn't sanction and now you're reverse-engineering its decision under pressure. Keep the human as the final approver. Use AI to make that human faster, calmer, and better-informed. That's the entire game.

The goal was never to remove the engineer from the loop. It's to make sure that when the pager does go off at 3 a.m. — and it will — it's for something that genuinely deserves you, with a clear summary, a ranked hypothesis, and the right runbook already in hand. That's a humane on-call. That's humanized AI.

James Joyner IV runs devopsaitoolkit.com, where he writes about keeping production systems and the humans who run them healthy. If you want to see this human-in-control approach in action, try the free AI Incident Response Assistant — it summarizes and triages, but you stay the one who decides.

Why AI Loves Ansible (And You Should Let It Help)

James Joyner — Thu, 25 Jun 2026 04:36:48 +0000

If you compare how well Claude handles Ansible against how well it handles, say, raw bash or kubectl YAML, Ansible wins by a wide margin. The reason isn't subtle: Ansible's shape — declarative, idempotent, modules-with-arguments — happens to map almost perfectly to how LLMs reason. They're good at producing structured output that fills in a known template, and that's what most Ansible tasks are.

This means AI-assisted Ansible work is the highest-leverage automation pairing I know of. If you only adopt AI for one infrastructure tool, make it Ansible.

What makes Ansible AI-friendly

Modules have published contracts

Every Ansible module has a documented argument spec: what's required, what's optional, what the defaults are. The model can fit your intent into the spec with high accuracy because the spec is finite and known.

Compare this to shell: there are a thousand ways to "create a user with a specific UID, member of these groups, with this shell, and a home directory in this location." In bash, every distro is slightly different. In Ansible, you use ansible.builtin.user with named arguments.

The model gets this right every single time.

Idempotency is the default

When a model generates a Python script, it has to think about "what if this is run twice." When it generates Ansible, most modules handle that for free. The model can write the task, ignore the re-run case, and produce something that works.

This means the cognitive load on both sides — model and human — is lower. You're describing the target state, not the procedure.

Roles and structure are predictable

roles/foo/{defaults,vars,tasks,handlers,templates,files,meta}/main.yml — every Ansible role looks the same. The model can scaffold a new role in seconds because the layout is fixed.

If you ask Claude to "create a new role for installing PostgreSQL 16 on Ubuntu 24.04 with default user postgres and a tuned postgresql.conf," you'll get a complete role structure with defaults/main.yml, tasks/main.yml, a Jinja template, and handlers/main.yml — all consistent, all in the right places. The structure is constrained enough that the model rarely improvises.

Use cases where AI shines for Ansible

Generating new roles from scratch

This is the killer app. You can describe a role in two sentences and get a 90%-done implementation. You then refine: add validation, adjust defaults, write a README.

I now treat "draft a new role with Claude" as the default first step. Even if I rewrite half of it, the structure saves me 20 minutes.

Converting shell scripts to playbooks

If you have a legacy bash script that provisions a server, pasting it into Claude with "convert this to an idempotent Ansible playbook using the appropriate modules" produces a usable result. The model knows when to use ansible.builtin.file, lineinfile, template, service, etc.

You'll need to verify the idempotency manually (run twice, expect 0 changes on the second run), but the conversion is mostly mechanical.

Refactoring playbooks to use FQCN

Ansible 2.10+ wants fully-qualified collection names: ansible.builtin.package instead of package. Old playbooks have hundreds of short-form references. AI is a perfect fit for this kind of mass refactoring — it knows the mapping and won't get bored.

Paste a 200-line playbook, ask for it back with FQCN throughout, and you're done in 30 seconds. Verify with ansible-lint.

Writing Molecule tests

Molecule scaffolding is repetitive — same molecule.yml, same converge.yml, same verify.yml structure for most roles. AI is great at generating the boilerplate. You describe what you want to test; the model writes the assertion playbook.

Jinja template generation

Jinja is just structured-enough that AI handles it well — generating templates for config files (nginx, postgres, sshd) from a description of the desired behavior. The model knows the configuration keys and the conditional structure.

Where AI struggles with Ansible

Variable precedence

Ansible's 21-layer variable precedence rules are not intuitive. The model will sometimes suggest putting a variable in vars/main.yml when you really want it in defaults/main.yml (the former overrides the latter). The result: users of your role can't override the variable they expected to.

Check: When the model puts something in vars/, ask "should this be overridable by the role user?" If yes, move to defaults/.

Custom facts and `set_fact` lifetime

The model sometimes uses set_fact for values that need to persist across plays, but doesn't add cacheable: true. The fact is then gone after the play ends, and the next play sees undefined.

Check: When you use set_fact for a value you need later, verify the lifetime is what you expect.

Vault integration

The model will sometimes generate playbooks that reference vault_db_password as a variable but don't include the lookup('community.hashi_vault.hashi_vault', ...) call or the Ansible Vault encrypted file. You have to wire up the secret source separately.

Check: For any sensitive variable in a generated playbook, verify there's an actual source for it (Vault encrypted file, external manager lookup, environment variable).

Distro-specific paths

The model defaults to Debian/Ubuntu conventions. If you run on RHEL, you'll sometimes get apt modules in tasks that should be using the package module (or distro conditionals).

Check: When generating playbooks for non-Debian systems, audit for apt, apt_repository, dpkg_selections, and ask for the abstraction (package) or the distro split.

A workflow that's been working for me

For a new role, my process now looks like this:

Describe the role to Claude in 2-3 sentences (purpose, target distros, key behaviors).
Generate the scaffolding: defaults/main.yml, tasks/main.yml, a template if needed, meta/main.yml with platforms.
Read every task. Look for the failure modes above (precedence, lifetime, Vault, distros).
Add Molecule tests. Have Claude scaffold molecule/default/, then write the assertions yourself or ask for them.
Run ansible-lint and Molecule. Fix what they catch.
Idempotence check. Run the role twice; second run should report 0 changed.
Refine the README. This is the one place I write from scratch — explaining the role to future-me.

This takes maybe 30 minutes for a moderately complex role. Without AI assistance, the same role would take me a couple of hours.

A note on safety

Ansible runs as root on production servers. Whatever the model generates, you are responsible for what it does. Two patterns I follow:

Check --check --diff before any real run. Dry-run the playbook in check mode; verify the diff matches what you expect.
Test on a sandbox host first. Especially for new roles. Don't trust the model with production until the role has run cleanly on a throwaway VM.

These are the same disciplines that apply to any infrastructure change. AI doesn't change the discipline; it just makes you faster at the parts before the change.

Why I think Ansible is the right entry point

If you're new to using AI for infrastructure work and want to pick one tool to start with, Ansible is the safest, highest-leverage choice. The structure makes the AI accurate. The idempotency makes mistakes recoverable. The module ecosystem covers most common cases.

By the time you've used AI to write a dozen Ansible playbooks, you'll have developed the intuition for what AI handles well and what needs human attention. That intuition transfers to harder tools — Terraform, Kubernetes, custom shell — where the cost of AI mistakes is higher.

For our full set of AI-driven Ansible workflows, see the IaC category — including ansible-vault-secrets-management and ansible-molecule-testing.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

Humanizing Artificial Intelligence for Platform Engineering: Helping Internal Developer Platforms Become Easier to Use

James Joyner — Tue, 23 Jun 2026 15:10:35 +0000

The Developer Who Just Wanted to Ship a Service

A backend developer on my team pinged me last week. She'd built a small Go service, it passed tests locally, and she wanted to get it running in our staging cluster. Simple ask. Then she opened the platform wiki.

Forty pages. A section on naming conventions. A page on which Terraform module to use for a Postgres instance, with a note at the bottom saying that page was "mostly current." A Helm values reference. A link to a deprecated runbook that someone forgot to delete. By the time she found the right golden-path template, she'd burned half a day and pinged three people in Slack. She didn't need a tutorial on our platform. She needed someone to walk her to the right door.

That gap is the actual problem with most internal developer platforms. The paved road exists. The templates exist. The policy is sound. But the cognitive load of finding the paved road is so high that developers route around it, copy a YAML file from another team, and ship something that violates four of our standards. The platform isn't failing on capability. It's failing on usability.

This is where I think AI earns its keep in platform engineering, and it's a narrower claim than the hype suggests. "Humanizing AI" here doesn't mean a chatbot that replaces your platform team. It means using AI as the friendly front door to your IDP: turning a plain-English request into the right golden-path template, scaffolding the boilerplate against an approved template, and explaining the cryptic platform error so the developer fixes it correctly instead of hacking around it. The guardrails, the templates, the approvals, and the humans who own them all stay exactly where they are. AI lowers the friction of self-service. It does not bypass the paved road.

Self-Service Is Only Self-Service If People Can Find It

The promise of an IDP is that a developer can provision what they need without filing a ticket and waiting two days. The reality, on most platforms I've seen, is a self-service portal that assumes the developer already knows the vocabulary of the platform. Which module? Which environment tier? Which approval gate applies to a service that touches PII?

A developer asking "I need a new Go service with Postgres" is asking a perfectly reasonable question in human terms. The platform answers in machine terms: pick a template ID, fill in fourteen variables, know in advance which of them are required. The translation layer between those two is exactly what an AI assistant is good at — and exactly where most teams have been doing the translation manually, in Slack, over and over, forever.

The trick is grounding. An ungrounded AI will happily tell your developer to drop in some Terraform it saw on a blog in 2021. That's worse than the wiki, because it's confidently wrong. A grounded assistant — one wired to your template catalog, your golden paths, your current docs via retrieval — recommends the paved road because the paved road is the only thing in its context.

Pro Tip: Don't fine-tune a model on your platform and call it done. Ground it with retrieval over your live template repo, your Backstage software catalog, and your current runbooks. Templates change weekly; a fine-tuned model freezes a snapshot. Retrieval means the assistant recommends what's actually approved today, not what was approved the day someone trained it.

From a Sentence to a Scaffold

Here's the concrete artifact. A developer types into the platform assistant:

I need a new Go service called billing-events with a Postgres database, staging and prod, and it needs to publish to our Kafka topic.

A grounded assistant doesn't generate freehand infrastructure. It maps that request onto the approved go-service-pg golden-path template in our catalog and produces the scaffold the platform team already blessed:

billing-events/
  catalog-info.yaml        # Backstage entity, owner team pre-filled
  cmd/main.go              # service skeleton, our logging + health endpoints
  Dockerfile               # from the approved base image, pinned digest
  helm/
    values.yaml            # resource limits from the "small" tier preset
  terraform/
    main.tf                # calls module "rds" { source = "registry/our-org/rds/aws" version = "4.2.1" }
    kafka.tf               # topic ACL via the approved kafka module
  .github/workflows/
    deploy.yaml            # the standard pipeline, not a bespoke one

The Terraform it emits isn't invented. It calls our internal module registry at a pinned version, with the variables the template requires:

module "billing_events_db" {
  source         = "registry.internal/our-org/rds/aws"
  version        = "4.2.1"
  instance_class = "db.t4g.medium"   # from the "small" tier
  multi_az       = true              # prod default, policy-enforced
  pii            = false             # tagged so policy bot knows the rules
}

Notice what the AI did and didn't do. It filled in the boilerplate, picked sane defaults from our tier presets, and tagged the resource so our policy engine can reason about it. It did not choose the module version, invent a new instance class, or skip multi_az. Those are constraints baked into the template, and the assistant is generating into the template, not around it. If the developer asks for something the template can't express — a database class we don't support, a region we don't run in — the right answer from the assistant is "that's not on the paved road, here's who to talk to," not a creative workaround.

If you're new to wiring AI into this kind of container-to-cluster flow, the walkthrough in from Dockerfile to first Kubernetes deployment with AI is a good ground-level starting point for what "generate against a known-good shape" looks like in practice.

Golden Paths Need a Tour Guide, Not Just a Map

A golden path is only golden if developers stay on it. The failure mode I see constantly: the platform team builds an excellent paved road, documents it beautifully, and then watches teams drift off it because at the moment of decision — 4 p.m., trying to ship before standup tomorrow — nobody reads the docs. They grab whatever's nearest.

An AI assistant grounded in your golden paths changes the economics of that decision. Staying on the paved road becomes the low-effort option, because the assistant does the lookup and the scaffolding for you. Drifting off becomes the high-effort option, because now you're writing Terraform by hand. That's the inversion you want. For years we've asked developers to do extra work to comply with the platform. Done right, AI makes compliance the path of least resistance.

This matters most for the workflows that are genuinely hard to reason about. GitOps is a great example: the model of "Git is the source of truth, a controller reconciles the cluster to match" is powerful but unintuitive for someone shipping their first service. An assistant that can explain why their change isn't live yet — "your manifest is merged, but Argo CD hasn't synced; here's the sync status and here's the app it belongs to" — is doing real developer support. If your platform runs this pattern, point people at the conceptual grounding in GitOps for infrastructure explained and the hands-on practical starting guide for Argo CD, and let the assistant answer the "but why isn't my thing deployed" questions against your actual setup.

Pro Tip: Make the assistant cite its source. Every recommendation should link back to the template, runbook, or catalog entry it came from. This does two things: it lets the developer verify the paved road for themselves, and it gives your platform team a signal — when the assistant keeps citing a stale doc, that's your cue to fix the doc. The AI becomes a usage tracker for your documentation.

Explaining the Cryptic Error Instead of Inviting a Hack

The highest-leverage place for AI in an IDP is the moment a developer hits a platform error they don't understand. This is where people hack. The pipeline fails with something opaque, the developer doesn't know what it means, and the fastest escape is to disable the check, add an || true, or copy a workaround from a Stack Overflow answer that has nothing to do with your platform.

A good assistant intercepts that moment. Here's the kind of output I want to see when a deploy fails our policy gate:

Your deploy to prod was blocked by the require-resource-limits policy.

Your helm/values.yaml sets a CPU request but no CPU limit. Our prod admission policy requires both — this prevents one service from starving the node. You don't need to disable the check.

The go-service-pg golden path ships a values-prod.yaml with the "small" tier preset that already includes compliant limits. You're missing it because this service was scaffolded before that preset existed. Copy the resources: block from the template here: [link to template]. After that, re-run the deploy — no override needed.

If you genuinely need higher limits than the preset allows, that's a capacity conversation with the platform team, not a policy bypass. Open a request here: [link].

That output respects the guardrail. It explains why the policy exists, points to the golden-path fix, and explicitly closes the door on the hack — while leaving a legitimate human-in-the-loop escalation path for the real exception. Compare that to the developer's default move, which is to find the flag that makes the red text go away. The AI didn't weaken the policy. It made the policy understandable, which is the only thing that makes a policy survive contact with a deadline.

There's a deeper point here about toil. Every one of these "what does this error mean" questions used to land in the platform team's Slack channel, and answering them is pure, repetitive toil — the same Postgres-module question, the same policy-gate question, fifteen times a month. Routing those to a grounded assistant frees your senior people to work on the platform instead of staffing a help desk for it. I've written more about that specific dynamic in identifying and eliminating toil with AI, and platform support is one of the cleanest examples of toil I know.

Deployment Workflows: Narrate, Don't Drive

I want to be precise about where AI sits in the deployment path, because this is where teams get nervous and they're right to. The assistant should narrate and assist the deployment workflow. It should not be the thing that pushes to prod.

The pattern that works: the developer commits to Git, the pipeline runs, the controller reconciles — your normal, audited, human-approved flow. The AI's job is to make that flow legible. "Here's where your change is. It's merged, it passed CI, it's waiting on the prod approval gate, which is owned by your team lead." When something goes wrong, it explains the failure in the terms above. When the developer asks "how do I roll back," it points them at the actual rollback procedure in your runbook — the real one, retrieved live, not a generic Kubernetes answer.

What it should never do is hold the credentials to apply infrastructure or promote a release on its own. The approval gates are the guardrail. A human owns the merge. A human owns the prod promotion. The AI is the assistant standing next to the control panel explaining what each button does — not the operator with its hands on the panel. If you build platform automation that AI helps developers operate, the Kubernetes operator pattern guide is worth reading for how to keep the reconciliation logic — the part that actually changes cluster state — in deterministic, audited controllers rather than in a probabilistic model.

Pro Tip: Treat the AI assistant's outputs as suggestions that flow into your existing review and policy gates, never as a side channel that skips them. The scaffold it generates should land as a pull request a human reviews. The Terraform it writes should still hit plan, policy-as-code, and approval. If your assistant can make a change that your normal pipeline couldn't, you've built a backdoor, not a front door.

The Human Platform Team Is the Point

It's tempting to read all of this as "AI replaces the platform team." It's the opposite. Every capability I've described depends on the platform team's work being good: the templates have to be right, the golden paths have to be real, the docs have to be current, the policies have to be sound. The AI is a multiplier on that work, not a substitute for it. Point it at a mediocre platform and you get a fast, confident way to spread mediocrity.

The work shifts, though. Less time spent answering the same onboarding question for the hundredth time. More time spent making sure the templates the assistant recommends are excellent, because now they reach every developer instantly. Your golden paths become higher-stakes in the best way — they're no longer docs that maybe get read; they're the thing the front door walks everyone toward. That's a better job. It's the platform-engineering job, freed from the help-desk tax.

"Humanizing AI" in this context means keeping humans in control of the road while using AI to make the road easy to walk. The developer who just wants to ship a Go service gets walked to the right template in thirty seconds instead of reading forty pages of wiki. The platform team keeps its guardrails, gains a tireless first-line guide, and gets back the hours it used to spend repeating itself. Nobody bypasses the paved road. The paved road just finally became the obvious one to take.

James Joyner IV runs devopsaitoolkit.com, where he writes about running AI alongside real production platforms. If you want to put grounded, human-in-control prompts to work on your own platform, start with the prompt library or the Writing Humanizer pack.

Vibe-Coded Infrastructure: How to Ship Fast Without Torching Production

James Joyner — Mon, 22 Jun 2026 19:05:48 +0000

You described what you wanted in plain English, the model wrote the Terraform, you ran apply, and it worked. No docs, no Stack Overflow, no fighting HCL syntax for an hour. That's vibe coding — building by describing intent and riding the model's output — and for infrastructure it is genuinely, addictively fast.

It's also how you accidentally terraform destroy a production VPC at 2 p.m. on a Tuesday.

I run production OpenStack, Kubernetes, and Terraform for a living, and I vibe-code a lot of it now. Here's the honest version of how to keep the speed without the smoking crater — the rules I actually follow.

Vibe coding is great at drafts and terrible at consequences

The thing a model is brilliant at is turning "I need a rate-limited NGINX reverse proxy in front of this service" into 40 lines of config in two seconds. The thing it has no idea about is that this service shares an upstream with the billing API, that your last outage came from exactly this kind of change, and that the "harmless" reload will drop in-flight connections during your peak hour.

The model writes code. It does not carry the consequences. You do. So the entire game of vibe-coding infrastructure safely is keeping the human on the hook for the blast radius while letting the machine do the typing.

Rule 1: Vibe the draft, never the apply

The single most important habit: the AI is allowed to propose changes. It is never allowed to make them. There's a world of difference between "here's the kubectl patch you'd run" and a bot that runs it for you.

Concretely, that means I demand a plan I can read before anything touches a real system:

terraform plan -out=tfplan
terraform show -json tfplan | <paste into the model>

"Read this plan. Tell me in plain English what changes, flag anything that destroys or replaces a resource, and rank the three riskiest changes. Don't tell me it's fine — tell me what could go wrong."

That force-replace buried on line 200 is exactly the thing vibe-coding glosses over and a careful read catches. I wrote up the full version of this — parsing a Terraform plan for AI-assisted review — but the one-liner is: the model reads the plan; you approve the apply.

Rule 2: Make destructive commands earn a second look

Vibe-coded shell scripts are where people get hurt fastest, because bash will cheerfully rm -rf "$DIR/" when $DIR is empty. Before I run anything a model handed me, it goes through a risk pass:

"Scan this script for anything destructive or irreversible — deletes, overwrites, force-pushes, drops, prod credentials. For each, tell me the blast radius and a safer version (dry-run flag, confirmation prompt, backup first)."

This catches the stuff vibe energy skips: missing set -euo pipefail, an unquoted variable that word-splits, a kubectl delete with no namespace scoping. I keep a whole pattern for catching risky shell commands before they run, and another for hardening a bash script with strict mode, traps, and back-out paths. Vibe-code the first draft; harden it before it runs once.

Rule 3: Stage everything the vibe touched

Vibe coding tempts you to skip the boring safety rails because the loop feels so fast. Don't. The fast loop needs the rails or it's just a faster way to break things:

Dry-run first. --dry-run=server, terraform plan, ansible --check. If the tool has a no-op mode, the vibe-coded change runs there first, every time.
Smallest blast radius. Apply to one node, one namespace, one non-prod env. Watch it. Then widen.
Back-out before apply. If you can't answer "how do I undo this in 60 seconds," you're not ready to apply it — no matter how confident the model sounded.

None of this slows the vibe down much. It just moves the "oh no" moment from production to a terminal where it's free.

Rule 4: Vibe-code the toil, hand-hold the crown jewels

Not all infrastructure is equal. I'll vibe-code a Grafana dashboard, a CI lint job, a one-off migration script, or a dev-env bootstrap with barely a glance — the downside is a wasted ten minutes. I will not vibe-code an IAM policy change, a database failover, a network ACL, or anything in the path of customer money without reading every line like it's a hostile PR.

Match your scrutiny to the blast radius. The model doesn't know which is which; you do. (If you're standing up an AI helper that runs alongside real systems, the same principle scales — I wrote about building an AI ops copilot with guardrails that proposes and never silently acts.)

Rule 5: Keep a prompt library so your vibes are reproducible

The dirty secret of good vibe coding is that it's not actually vibes — it's good prompts. A throwaway "fix my nginx config" gets you a throwaway answer. A prompt that says "act as a senior SRE, here's my config and the error, give me ranked causes and the nginx -t to verify before I reload" gets you something you can trust.

I keep the ones that work in a searchable prompt library — filterable by stack, difficulty, and whether they include production-safety guidance — so the next time I vibe-code a Postgres index or a Kubernetes rollout, I'm starting from a prompt that already bakes in the guardrails. Reproducible vibes beat lucky ones.

The honest takeaway

Vibe coding isn't the enemy of careful engineering — it's a power tool, and power tools are exactly as safe as the operator. Used well, you draft in seconds and spend your real attention on the 5% that can actually hurt you. Used badly, you ship confident nonsense at machine speed.

So: vibe the draft, read the plan, stage the apply, scope the blast radius, and keep a human's name on the change ticket. Do that and "vibe-coded" stops being a confession and starts being a workflow.

I write about running AI alongside real production infrastructure at devopsaitoolkit.com. New there? The start-here tour has the free toolkit and the incident assistant. And if your production is painful enough that vibe-coding won't fix it, I do fixed-price infrastructure audits.

AI for GitLab CI Authoring: Save Hours, Avoid Footguns

James Joyner — Sat, 20 Jun 2026 12:15:22 +0000

GitLab CI YAML is one of those formats where you can stare at it for an hour, get it 95% right, and have it fail with yaml: line 12: did not find expected key because of a tab character. AI assistants are very fast at this kind of work. They're also confidently wrong about specific GitLab features in ways that waste a lot of time if you don't know what to check.

After a year of letting Claude write a lot of my pipelines, here's what works and what doesn't.

What AI gets right consistently

Standard job shapes

"Write me a job that builds a Docker image, pushes to the GitLab Container Registry, and tags with the commit SHA and latest on the default branch." Type that into Claude and you get a working job in five seconds. The shape is well-established and the model has seen thousands of variations.

The same is true for:

Test jobs across languages (pytest, jest, go test, etc.)
Standard cache configurations
Standard artifact patterns
Basic rules: for branch / tag / MR pipelines

If you find yourself writing one of these from scratch, you're spending time that you don't need to spend.

Translating from other CIs

GitLab CI has obvious parallels to GitHub Actions, CircleCI, Jenkins declarative pipelines, etc. AI is excellent at translating between them. The structures rhyme; the model knows the dictionary.

If you're migrating from Actions to GitLab CI, paste the workflow and ask for the GitLab CI equivalent. You'll get something 80% right that you can refine.

Reviewing pipelines for inefficiency

This is the underrated use case. Paste your .gitlab-ci.yml and ask: "what's the critical path of this pipeline, and what's making it slow?" The model will spot things like:

"Your test job downloads node_modules from cache, but install-deps doesn't push to cache — your cache key is broken."
"Your build and deploy stages are sequential but build's artifacts aren't used by deploy — they can be parallel with needs:."
"Your rules:changes: doesn't include package-lock.json, so dependency changes don't retrigger tests."

These are real findings I've gotten from Claude on pipelines I thought I'd already optimized. Worth the five-minute review.

What AI gets wrong — and how to catch it

`rules:` vs `only/except` confusion

The model will sometimes mix them in the same job. GitLab silently ignores only: when rules: is also defined. The pipeline runs but the behavior isn't what you expect.

Check: Are you using rules: OR only:/except: in each job? Pick one. (Use rules: — only/except is legacy.)

`$CI_COMMIT_BRANCH` empty on MR pipelines

A common bug: you ask for "this job runs on the default branch" and you get:

rules:
  - if: $CI_COMMIT_BRANCH == "main"

This is correct for branch pipelines. It is empty on MR (merge_request_event) pipelines. If you have MR pipelines enabled, your job silently won't run when developers expect it to.

Check: Does your pipeline target both push events and MR events? If so, you probably want $CI_MERGE_REQUEST_TARGET_BRANCH_NAME or to handle both pipeline sources.

`needs:` referencing hidden jobs

Hidden jobs (prefixed with .) are templates — they don't execute. If you do needs: [".lint"], your job will fail with a confusing error because GitLab thinks you're depending on a job that doesn't exist.

Check: Every needs: entry should be a real job name, not a template.

Auto-apply rules that don't include the right branches

The model loves writing:

rules:
  - if: $CI_COMMIT_BRANCH == "main"
    when: always
  - when: never

This works on main but blocks the job on tags, on schedules, and on MR pipelines. Sometimes that's what you want. Often it's not.

Check: What pipeline sources do you expect this job to run in? List them, then verify your rules cover each.

Imaginary GitLab features

This is the most expensive AI failure mode. The model will sometimes generate syntax for features that don't exist:

A condition: field that's actually OPA/Conftest, not GitLab CI
An auto_retry: block that's GitHub Actions, not GitLab
A before_script: keyword that does exist but with different semantics than the model claims

Check: If you see a keyword you haven't seen before in GitLab docs, verify it exists. The lint endpoint (/api/v4/ci/lint) catches most of these, but some pass lint and just behave weirdly.

A workflow that catches the failures cheaply

I now do this for any non-trivial pipeline change:

Draft with AI. Describe the desired behavior in plain English; let the model write the YAML.
Read every line. Treat the output as a draft you'd write yourself.
Lint via the API.

   curl -s --header "PRIVATE-TOKEN: $TOKEN" \
       --header "Content-Type: application/json" \
       --data "{\"content\": $(cat .gitlab-ci.yml | jq -Rs .)}" \
       "$GITLAB_URL/api/v4/ci/lint" | jq

Run on a sandbox branch. Push to a branch that won't trigger deploys; verify the pipeline runs the jobs you expect, when you expect.
Diff against the existing pipeline. If the AI introduced changes you didn't ask for (a different cache key, a removed interruptible:), revert them.

Step 5 is the one most people skip. The model is good at writing YAML but not at preserving your previous decisions. If you don't diff, you'll lose your old cache strategy.

A practical example

Last month I needed to add a job that runs terraform plan on every MR and posts the output as a comment. Drafted with Claude in two minutes; it produced something like:

terraform-plan:
  image: hashicorp/terraform:1.9
  stage: plan
  script:
    - terraform init
    - terraform plan -out=tfplan -no-color
    - terraform show -no-color tfplan > plan.txt
    - |
      curl -X POST -H "PRIVATE-TOKEN: $GITLAB_API_TOKEN" \
          -d "body=$(cat plan.txt | jq -Rs .)" \
          "$CI_API_V4_URL/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

This is almost right. Two issues:

PRIVATE-TOKEN as a CI variable — using a personal access token for CI is the old pattern. Modern approach: use $CI_JOB_TOKEN for in-instance API calls. Saves rotation pain.
No terraform init -backend-config — works if the backend is configured in code, but if you have multiple environments using the same module, you'd want to specify which backend.

Both fixes are 30 seconds. Without the AI I'd have spent 15 minutes writing the curl invocation alone.

The bottom line

AI doesn't replace knowing GitLab CI. It removes the typing and the boilerplate so you can spend your attention on the parts that matter — the rules: logic, the cache keys, the secrets, the environment promotion.

Once you've internalized the failure modes above, the workflow becomes mostly automatic. You stop reading the boilerplate and start reading the rules. That's where the bugs live.

For the prompt set we use on GitLab CI specifically, see the GitLab CI/CD category — particularly gitlab-pipeline-optimization and gitlab-ci-rules-debugging.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

Humanizing Artificial Intelligence in DevOps Documentation: Making Runbooks Easier to Create and Use

James Joyner — Fri, 19 Jun 2026 21:23:54 +0000

The Runbook That Lied to Me at 3am

The pager went off at 3:14am for a wedged OpenStack Neutron agent. I did what any tired engineer does: I opened the runbook. It told me to restart a service that had been renamed eighteen months earlier, pointed at a Grafana dashboard that 404'd, and assumed a network topology we'd migrated off of two quarters back. The runbook wasn't just unhelpful. It was actively lying to me, and I burned twenty minutes trusting it before I gave up and went to read the source.

That's the real problem with documentation. It isn't that we don't write it. It's that the moment we finish writing it, it starts rotting, and the cost of keeping it fresh is high enough that nobody pays it until the document has already betrayed someone at 3am. A runbook your team doesn't trust is worse than no runbook, because no runbook at least forces you to think.

This is where AI actually earns its keep in a platform org, and not in the way the marketing decks suggest. AI is not going to own your documentation. It's going to do the tedious first-draft labor — turning a resolved incident, a chunk of shell history, or a deploy diff into a structured skeleton — so a human engineer can spend their scarce attention on the part that matters: verifying the commands, marking what's unproven, and editing the robotic tone out so the team actually reads it. AI drafts. You verify and sign off. That distinction is the whole game.

Why "Humanizing" AI Is the Job, Not a Slogan

Let me be precise about what I mean by "humanizing AI," because the phrase gets abused. I don't mean making AI sound human to fool a reader. I mean keeping a human in the loop as the editor and owner of record, and doing the unglamorous work of turning a competent-but-soulless machine draft into something a colleague trusts.

Two things break trust in AI-drafted docs, and both are fixable by a human pass:

Unverified claims stated with confidence. An LLM will happily tell you to run systemctl restart neutron-l3-agent whether or not that's the actual unit name on your boxes. It doesn't know. It's pattern-matching. So the human's first job is to run every command, in a safe environment, and confirm it does what the draft claims.
The robotic tone. Machine drafts read like a compliance memo: hedged, repetitive, weirdly formal, full of "it is important to note that." Engineers smell that instantly and stop reading. Editing for voice and concision isn't vanity — a doc people skim past doesn't get used. That cleanup pass is legitimate, valuable work, and it's exactly the "humanizing" angle that matters.

If you keep those two responsibilities firmly with a person, AI becomes a force multiplier instead of a liability generator. I've written more about the philosophy of runbooks engineers actually trust at 3am, but the short version is: trust is built by verification, and verification is a human act.

Troubleshooting Guides: Draft From the Incident You Just Resolved

The best time to write a troubleshooting guide is in the hour after you've fixed the thing, while the diagnostic path is still warm in your head. The worst time is never, which is the default. AI closes that gap because the raw material already exists in your terminal.

Here's the prompt pattern I use. I dump my actual shell history and the incident timeline at the model and constrain it hard:

You are drafting an internal troubleshooting guide for a platform team. I'll give you my shell history and a rough incident timeline. Produce a guide with these sections: Symptoms, Prerequisites/Access, Diagnosis Steps, Resolution, Rollback, Verification. Use the exact commands from my history — do not invent flags or paths. Wherever you infer something I didn't explicitly state, prefix that line with [ASSUMPTION] so I can verify it. Keep the prose tight; no filler.

That [ASSUMPTION] instruction is the single most important line in the prompt.

Pro Tip: Always force the model to explicitly tag anything it couldn't verify from your inputs. An LLM's failure mode is confident gap-filling. When you make it mark its own guesses, you convert an invisible risk into a visible checklist — every [ASSUMPTION] tag is a thing you personally confirm before the doc ships. The tags get deleted in editing, but they drive the verification pass.

Feed that prompt a history | grep neutron plus a three-line timeline, and you get back something like this — which is a genuinely good starting point, not a finished doc:

Diagnosis Steps

Confirm the agent is actually down, not just flapping: openstack network agent list --agent-type l3 --host compute-04

[ASSUMPTION] The L3 agent unit is neutron-l3-agent. Check status: systemctl status neutron-l3-agent

Tail the agent log for the RPC timeout signature: journalctl -u neutron-l3-agent --since "10 min ago" | grep -i timeout

Resolution

Restart the agent and confirm it re-registers with the server:
systemctl restart neutron-l3-agent && sleep 5 && openstack network agent list --host compute-04

Rollback

[ASSUMPTION] No rollback required for a restart. If the agent fails to re-register, fail the router over to a standby node before deeper debugging.

Notice what the AI did well: it structured the chaos of my history into a clean diagnostic flow and pulled the real commands I ran. Notice what it flagged: it didn't know my unit name, so it tagged it. When I verified, the unit was actually devstack@q-l3.service on the affected host — a thing I'd never have caught if the draft had just stated it confidently. That one assumption tag is the difference between a guide that helps and one that lies. I go deeper on this workflow in my piece on drafting runbooks from resolved incidents with AI.

Postmortems: AI Handles the Timeline, You Own the Blamelessness

Postmortems are where AI's drafting strength and its judgment weakness are both on full display. The mechanical parts — assembling a coherent timeline from Slack threads, alert timestamps, and deploy logs, then drafting an impact summary — are exactly the tedious work that delays postmortems for weeks. AI eats that for breakfast.

But the blameless part is not a tone setting you toggle. It's an editorial and cultural stance that a human has to own. I give the model the raw timeline and explicitly instruct it:

Draft a blameless postmortem. Describe what the system did and what signals were available, never what a person "failed" to do. Frame every human action as a reasonable decision given the information available at the time. Sections: Summary, Impact, Timeline, Contributing Factors, What Went Well, Action Items. Mark any causal claim you can't support from the inputs with [UNVERIFIED].

The model will get you 80% of the way to neutral language. The remaining 20% — catching the sentence that subtly implies the on-call engineer should have known better — is human work, every time. Blameless writing is a skill, and the editing pass is where you apply it. If you haven't internalized what separates a postmortem people read from one that gets filed and forgotten, this breakdown of blameless postmortems people actually read is worth your time.

Pro Tip: Never let AI write your Action Items unsupervised. The model loves to generate plausible-sounding remediation ("add more monitoring," "improve documentation") that sounds responsible and commits no one to anything. Real action items have an owner, a due date, and a verifiable definition of done. That's a leadership decision, not a text-generation task — strike every vague item the draft produces and replace it with something a person actually agreed to do.

The payoff is real, though. When the timeline and impact summary are drafted in ten minutes instead of taking the better part of a day, postmortems actually get written while the details are fresh, instead of being abandoned because everyone moved on to the next fire.

SOPs: Encode the Tribal Knowledge Before It Walks Out the Door

Standard operating procedures are the documents most likely to live entirely in one senior engineer's head — how we rotate the cluster certs, how we drain a node for maintenance, the precise dance to cut a new Terraform workspace without orphaning state. That knowledge is a bus-factor risk, and writing it down has always lost to more urgent work.

AI lowers the activation energy enough that it stops losing. I'll narrate the procedure out loud into a rough text file — half sentences, command fragments, the gotchas I remember — and have the model turn that mess into a structured SOP with numbered steps, prerequisites, and explicit "you are done when..." verification criteria.

The thing to watch here is that an SOP encodes policy, not just commands. The AI can format your steps beautifully and still produce something that violates your change-management rules because it doesn't know them. So the human pass on an SOP is checking two layers: are the commands correct, and does the procedure comply with how we're actually supposed to operate? The model can't see your org chart or your change board. You can.

I keep a small library of these drafting prompts so I'm not rewriting the scaffolding each time — collecting and reusing the prompts that work is half the productivity gain, because the quality of the draft is downstream of the quality of the prompt.

Deployment and Release Docs: Draft Straight From the Diff

Deployment documentation has a unique advantage: the source of truth is structured and machine-readable. A PR diff, a Terraform plan, a Helm values change — these are precise artifacts an AI can read directly, which means the draft starts from facts rather than recollection.

My workflow for release notes and deploy runbooks: pipe the diff or the terraform plan output to the model and ask for a deployment guide that includes pre-flight checks, the apply procedure, the blast radius, the rollback, and the post-deploy verification. Because the input is concrete, the hallucination rate drops sharply. The model isn't guessing what changed — it's reading it.

Pre-flight Checks

Confirm the target workspace: terraform workspace show returns prod-us-east.

[ASSUMPTION] This plan adds 2 nodes and modifies the ASG launch template; no destroys. Re-run terraform plan and confirm zero resources to destroy before applying.

Rollback

The launch template change is versioned. To roll back, point the ASG at the previous template version and trigger an instance refresh — do not terraform destroy.

Even here, the human verifies the blast radius claim. "No destroys" is the kind of statement that's true until a provider upgrade quietly makes a field force-new, and a misread there is how you turn a routine deploy into an outage. The AI gets you a structured, mostly-correct draft fast; you confirm the dangerous parts with your own eyes. For a fuller treatment of wiring this into a real pipeline, my 2026 guide to runbook automation with AI walks through the tooling end to end.

The Editing Pass Is Where Trust Is Built

I want to close on the part that gets skipped, because it's the part that actually decides whether your docs get used. Once the commands are verified and the assumptions are resolved, you still have a machine-shaped document on your hands — competent, structured, and slightly lifeless. It hedges too much. It repeats the obvious. It has that flat, over-formal register that makes an engineer's eyes glaze.

Editing that out is not cosmetic. A runbook nobody can stand to read is a runbook nobody reads, which puts you right back at 3am reading source code. So I do a final pass for voice: cut the filler, sharpen the warnings, add the one line of context only a human who lived the incident can add ("the agent flaps for about thirty seconds after restart — don't panic and restart it again"). That sentence is worth more than the entire generated scaffold, and only a person can write it.

That's the humanizing loop in full: AI drafts the structure from real artifacts, a human verifies every command and resolves every assumption, and then a human edits the robotic tone into something a tired colleague will actually trust and follow. Keep ownership with the person at every step and AI becomes the best documentation intern you've ever had — fast, tireless, and entirely supervised. Skip the human steps and you're just generating the next lie waiting to fire at 3am.

James Joyner IV runs devopsaitoolkit.com, where he writes about running production OpenStack, Kubernetes, and observability with AI in the loop. If your AI-drafted docs read like a robot wrote them, his Writing Humanizer pack is a toolkit for making machine drafts read like a human actually sat down and wrote them.

Securing AI-Generated Bash Scripts Before You Run Them

James Joyner — Thu, 18 Jun 2026 15:51:56 +0000

Bash is the easiest language for AI to write and the easiest language to get devastating output from. A 20-line script that "just cleans up old files" can recursively delete a home directory because the model assumed a variable would always be set. A "simple log shipper" can write your secrets to a remote server because the model used set -x for debugging and forgot to remove it.

I have run AI-generated bash that I should not have. Most engineers I know have too. After enough close calls, there's a short checklist that catches the worst of it. This is that checklist.

The five things to check before running any AI-generated bash

1. Does it start with a strict pragma?

The first lines of any non-trivial bash script should be:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

What each does:

set -e — exit on any command failure. Without this, a failure in line 5 doesn't stop the script from happily running lines 6-50.
set -u — error on undefined variables. This is the one that saves you from rm -rf $UNDEFINED/.
set -o pipefail — propagate failures through pipes. Without it, failing-command | grep something succeeds because grep succeeds.
IFS=$'\n\t' — sane field splitting. Defends against word-splitting bugs in filenames.

If the AI-generated script doesn't have these, add them and re-read the script. You'll often discover bugs the pragma now flags.

2. Is every variable expansion quoted?

# Wrong
rm -rf $TARGET_DIR

# Right
rm -rf "$TARGET_DIR"

The wrong version is what causes the "I deleted the root directory" stories. If $TARGET_DIR is empty or contains a space, the command becomes rm -rf (delete current directory) or rm -rf foo bar (delete two unintended things).

Models default to the wrong version about half the time because the right version is harder to write in chat ("escape the quotes!") and the wrong version is what most blogs show.

Fix: When reading AI bash, mentally check every $VAR for quotes. Add them if missing. This is the single biggest source of bash disasters.

3. What happens if a step fails partway through?

The AI will cheerfully write:

mkdir -p /opt/new-app
cd /opt/new-app
tar xzf $TARBALL
rm $TARBALL

What happens if tar xzf fails (corrupt tarball, full disk)? With set -e, the script stops. Good. Without set -e, it continues to rm $TARBALL and deletes your tarball with no backup.

For any state-changing script, ask yourself: at each step, what's the recovery path if the step fails? If the answer is "nothing automated," the script should at least not delete data before verifying the previous step succeeded.

The AI almost never thinks about this on its own.

4. Are secrets visible in logs?

The most common way AI-generated bash leaks secrets is via set -x:

set -x  # debugging
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/...

With set -x, every command is printed including the expanded variables. Your API token is now in the script's output, which is in your CI logs, which are visible to anyone with project access.

The fix is selective:

set +x  # disable trace
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/...
set -x  # re-enable

Or simply remove set -x once debugging is done. The model frequently leaves it in.

5. Does it run as root unnecessarily?

The AI will sometimes write sudo into every command, even ones that don't need it. Or it'll assume the script runs as root and use absolute paths that require root to write.

The principle: if a command can run as a non-root user, it should. The smaller the privileged surface, the smaller the blast radius.

This is especially important for scripts that download and execute code. A common pattern:

# Dangerous: privileged download + execute
sudo bash -c 'curl https://example.com/install.sh | bash'

# Safer: review then run
curl https://example.com/install.sh > install.sh
# READ install.sh
sudo bash install.sh

If the model generates the first pattern, replace it with the second. Always.

A real example

Last month I asked Claude to write a script that cleans up Docker images older than 30 days on a CI runner host. The first draft was:

#!/bin/bash

DOCKER_IMAGES=$(docker images --format '{{.ID}} {{.CreatedAt}}')
CUTOFF=$(date -d '30 days ago' +%s)

echo "$DOCKER_IMAGES" | while read ID DATE; do
    CREATED=$(date -d "$DATE" +%s)
    if [ $CREATED -lt $CUTOFF ]; then
        docker rmi $ID
    fi
done

Walking the checklist:

No strict pragma. Missing set -euo pipefail.
Unquoted $DOCKER_IMAGES, $ID, $DATE. Each one is a potential bug.
Failure handling. docker rmi fails if an image is in use. The script continues, marches through, and silently fails on every in-use image. We never know which were cleaned and which weren't.
No secrets (docker doesn't expose them here), but the script also doesn't log what it's doing, so you can't audit afterward.
No sudo, good — assumes the user has Docker socket access, which is reasonable.

The hardened version:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

CUTOFF=$(date -d '30 days ago' +%s)
REMOVED=0
SKIPPED=0

# Use --format with safer parsing
docker images --format '{{.ID}}|{{.CreatedAt}}' | while IFS='|' read -r ID DATE; do
    CREATED=$(date -d "$DATE" +%s)
    if [ "$CREATED" -lt "$CUTOFF" ]; then
        if docker rmi "$ID" 2>/dev/null; then
            echo "Removed: $ID"
            REMOVED=$((REMOVED + 1))
        else
            echo "Skipped (in use): $ID"
            SKIPPED=$((SKIPPED + 1))
        fi
    fi
done

echo "Cleanup complete. Removed: $REMOVED, Skipped: $SKIPPED."

This took two minutes of editing. Without the checklist, I might have run the original and noticed days later that disk usage hadn't really dropped because half the images were in use.

A small note on bash linting

shellcheck catches most of these issues automatically. If you adopt one tool from this article, make it shellcheck:

shellcheck cleanup-images.sh

It will flag unquoted variables, missing strict mode, and a dozen other patterns. AI-generated bash usually has at least one shellcheck warning.

I now run shellcheck on every script before I run the script itself. It's two seconds and catches things I'd miss.

When the AI gets it right

To be fair: the model is often perfectly capable of producing safe bash. If you prompt it explicitly — "write this with set -euo pipefail, quote every variable, fail loudly on errors" — you'll get a clean script.

The problem is that "write me a script that does X" without that prompt gets you the common form of the script, which is the unsafe form. So the rule of thumb:

Always include the safety requirements in the prompt. Or: always treat the output as a draft that needs hardening. Don't run any bash the AI wrote without one of those two disciplines.

The bottom line

Bash from AI is fast to produce and easy to read incorrectly. The checklist is short — strict pragma, quoted expansions, failure paths, secrets in logs, unnecessary privilege — and applying it takes a couple of minutes per script. The downside of skipping it is on the spectrum of "minor cleanup mistake" to "career incident." There's no excuse not to do the check.

For our prompts on bash specifically, see bash-script-code-review and the related linux-server-hardening prompt — both of which cover related territory.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

The Best AI Tools for DevOps Engineers in 2026

James Joyner — Wed, 17 Jun 2026 20:59:44 +0000

If you spend your day in a terminal, a YAML editor, or a Grafana tab — AI assistants in 2026 are no longer a curiosity. They're a real productivity layer. But not every tool is good at infrastructure work. After a year of daily use across Linux administration, OpenStack operations, Prometheus alert authoring, and Kubernetes debugging, here's the honest shortlist.

The criteria

We're not ranking on benchmark scores. We're ranking on infrastructure usefulness:

Reasoning over command output — can it actually read top, kubectl describe, or journalctl and find the real problem?
Safety — does it warn before suggesting destructive commands?
Long context — can it hold a 1,000-line .gitlab-ci.yml plus failing logs without losing track?
Terminal integration — can you use it without leaving your workflow?
Privacy and self-host options — for the engineers whose employers care.

The shortlist

1. Claude (Anthropic)

The current best general assistant for infrastructure reasoning. Long context handles enormous log dumps and Kubernetes manifests in one shot. It is consistently more cautious about destructive commands than alternatives — which matters when you're tired at 2am and tempted to copy-paste straight into prod.

Best for: Linux/OpenStack/Kubernetes troubleshooting, postmortem drafting, code review on infrastructure-as-code.

2. ChatGPT (OpenAI)

The broadest ecosystem. Strong code generation, plug-in support, and the largest community of shared prompts and patterns. For Ansible and Terraform generation, output quality is excellent. Slightly less cautious by default — you'll want to add safety constraints in your prompts.

Best for: Ansible/Terraform generation, ad-hoc scripting, learning new tools.

3. Cursor

If you live in an IDE, Cursor is what your IDE should have been. Native multi-file context, agent mode for repo-wide refactors, and tab-completion that actually understands your codebase. Especially strong for IaC repositories with many interconnected files.

Best for: Editing real codebases (Helm charts, Terraform modules, Python operators).

4. GitHub Copilot

The lowest-friction option. Inline completion just works, and the chat sidebar is genuinely useful for "explain this regex" or "what's this PromQL doing?" If your org already pays for GitHub, Copilot is essentially free upside.

Best for: Inline completion while editing YAML, Bash, Python.

5. Warp Terminal (with AI features)

The only entry on this list that isn't an AI assistant per se — it's a terminal that has AI built in. The killer feature: natural-language command suggestions in your shell, with safety previews. For Linux admins who don't want to alt-tab to a chat window every five seconds.

Best for: Terminal-native workflows where context-switching kills focus.

What we don't recommend (yet)

Generic LLM wrappers that promise "DevOps AI." Most are thin layers over the same APIs above, sometimes with worse safety defaults. Use the underlying tools directly.
Anything that requires uploading your ~/.ssh directory or production credentials. Be skeptical of "AI agents that run commands for you" without a clear sandbox model.

How to combine them

A pattern that works well in practice:

Claude or ChatGPT in a browser for deep diagnosis sessions (paste logs, walk through hypotheses, draft postmortems).
Cursor or Copilot in your editor for actually writing the fix.
Warp in the terminal for quick command lookups without switching context.

You don't need one perfect tool. You need a workflow where each tool plays to its strengths.

DEV Community: James Joyner

DevOps as a Service Pricing: What Should Businesses Expect to Pay?

Why DevOps pricing varies so much

The common pricing models

Hourly / time-and-materials

Monthly retainer

Project-based / fixed bid

Emergency / incident support

Fully managed service

What services actually move the price

Why cheaper is not always better

What downtime actually costs

How AI changes the math

Starting lean: startups and small businesses

How to calculate ROI

The bottom line

Best DevSecOps Security Tools for CI/CD Pipeline Protection

What DevSecOps actually means

Why the CI/CD pipeline is a prime target

SAST — Static Application Security Testing

DAST — Dynamic Application Security Testing

SCA / dependency scanning

Container image scanning

Infrastructure as Code scanning

Secrets detection

Policy-as-code

Runtime security monitoring

How to choose: team size and infrastructure maturity

Where AI fits — assistive, not authoritative

Conclusion

Humanizing Artificial Intelligence for Log Analysis: Turning Raw Server Logs Into Clear DevOps Answers

AI reads. You decide.

Before anything else: redact

Linux logs: journald and syslog

Application and container logs: feed the right context

Kubernetes events: the cheap, ignored goldmine

OpenStack: tracing one request across nova, neutron, and libvirt

Make the model show its work

The part that's actually yours

DevOps Security Best Practices Every Engineering Team Should Follow

Why DevOps security belongs in the daily workflow

Secure access control and least privilege

SSH key management and MFA

Secrets management: API keys, passwords, and tokens

CI/CD pipeline security

Container image scanning

Infrastructure as Code security

Patch management and vulnerability remediation

Monitoring, logging, and alerting for security events

Backup and disaster recovery planning

Incident response preparation

AI-assisted security checks — review, never blind trust

The bottom line

Humanizing Artificial Intelligence for SRE Teams: Reducing Alert Fatigue With Smarter AI Guidance

The real problem is signal, not tooling

Where AI earns its place: the alert storm

Let Alertmanager do the suppression, let AI do the explaining

Prioritization: what actually deserves a human at 3 a.m.

Surfacing the runbook, not writing the fix

Closing the loop without a bot closing it for you

The line you don't cross

Why AI Loves Ansible (And You Should Let It Help)

What makes Ansible AI-friendly

Modules have published contracts

Idempotency is the default

Roles and structure are predictable

Use cases where AI shines for Ansible

Generating new roles from scratch

Converting shell scripts to playbooks

Refactoring playbooks to use FQCN

Writing Molecule tests

Jinja template generation

Where AI struggles with Ansible

Variable precedence

Custom facts and set_fact lifetime

Vault integration

Distro-specific paths

A workflow that's been working for me

A note on safety

Why I think Ansible is the right entry point

Custom facts and `set_fact` lifetime

`rules:` vs `only/except` confusion

`$CI_COMMIT_BRANCH` empty on MR pipelines

`needs:` referencing hidden jobs