DEV Community: DevOps AI ToolKit

DevOps as a Service Pricing: What Should Businesses Expect to Pay?

James Joyner — Mon, 29 Jun 2026 22:02:07 +0000

After 25 years of keeping production systems alive — building the automation, owning the pager, and helping companies stop bleeding money on preventable outages — the question I get asked most by founders and operations leads is blunt: "What is this going to cost me?"

The honest answer is the one nobody likes: it depends. But "it depends" isn't useful if you're trying to budget. So let me give you the real version — what drives the number, the pricing models you'll actually be quoted, and a simple way to figure out whether the spend pays for itself.

Why DevOps pricing varies so much

There's no sticker price on DevOps for the same reason there's no sticker price on "fixing my house." A one-bedroom condo and a 40-year-old farmhouse are different jobs. Three things move the number more than anything else:

Company size. A two-person startup with one Linux server and a single web app is a fundamentally different engagement than a 200-person company running multiple Kubernetes clusters across regions.
Infrastructure complexity. A static site on a single cloud VM is cheap to run. A microservices platform with service meshes, multiple databases, message queues, and compliance requirements is not.
Support expectations. "Help us when something breaks during business hours" and "24/7 on-call with a 15-minute response SLA" are priced an order of magnitude apart, because one of them owns someone's nights and weekends.

Before you can compare quotes, you have to be honest about which of those buckets you're actually in. A provider quoting you a low number may simply be assuming a smaller scope than the one you need.

The common pricing models

Most DevOps as a Service work is sold under one of five models. Each fits a different situation, and good providers will steer you toward the right one rather than forcing everything into their favorite.

Hourly / time-and-materials

You pay for hours worked, usually billed against a monthly cap.

When it fits: Small, well-defined tasks, ad-hoc help, or an early relationship where neither side knows the full scope yet.
Rough ballpark: Rates vary widely by region and seniority. The trap is that hourly incentivizes activity, not outcomes — a cheap hourly rate from someone who takes three times as long is not a bargain.

Monthly retainer

A fixed monthly fee buys you a block of capacity and ongoing ownership of your infrastructure.

When it fits: You have living infrastructure that needs continuous care — patching, monitoring, upgrades, small improvements — and you want a predictable line item.
Example: Ongoing Kubernetes version upgrades, Prometheus and Grafana tuning, and routine Ansible-driven patching of your Linux fleet are classic retainer work. The cluster doesn't stop needing attention, so neither does the engagement.

Project-based / fixed bid

A scoped deliverable for a fixed price.

When it fits: A clear, bounded build with a defined "done."
Example: A one-time Terraform plus GitLab CI/CD build-out — provision the cloud accounts, write the infrastructure as code, stand up the pipelines, Dockerize the apps, and hand it over — is naturally project-priced. You know what you're getting and what it costs before work starts.

Emergency / incident support

On-demand help when production is on fire, often at a premium rate or via a pre-paid response retainer.

When it fits: You run your own systems day-to-day but want a number to call when something serious breaks.
Reality check: This is the most expensive way to buy help per hour, because you're paying for someone to drop everything. It's insurance, not a maintenance plan — and it's far cheaper to prevent the incident than to buy emergency labor mid-outage.

Fully managed service

The provider owns your DevOps function end to end — infrastructure, pipelines, monitoring, security, on-call, the lot.

When it fits: You don't want to hire and retain an internal platform team, or you want to extend the small one you have.
Reality check: This is the highest monthly spend, but compare it against the loaded cost of hiring senior engineers, the recruiting time, and the bus-factor risk of a one-person internal team. Often it's cheaper and far less fragile than building the same capability in-house.

A healthy engagement often mixes models: a project-priced initial build-out, then a monthly retainer to run what was built.

What services actually move the price

Within any model, the scope of work is what sets the number. The big cost factors:

Cloud setup and infrastructure as code. Account structure, networking, and Terraform modules to make it all reproducible.
CI/CD pipelines. Building and maintaining GitLab CI/CD (or equivalent) so deploys are fast, repeatable, and safe.
Containers and orchestration. Docker images, registries, and Kubernetes — the single biggest complexity multiplier in modern infrastructure.
Monitoring and observability. Prometheus, Grafana, alerting rules, and dashboards. Good monitoring and alert generation is what turns a 3am outage into a 9am ticket.
Security. Secrets management, access control, network policy, vulnerability scanning, and hardening of your Linux servers.
Backups and disaster recovery. Tested restores — not just backups that exist on paper.
Incident response and on-call. The cost of someone being awake and accountable when things go wrong.
Automation. Ansible playbooks and scripting that replace manual, error-prone toil.
Compliance. SOC 2, HIPAA, PCI, and friends add audit, documentation, and control work that materially raises cost.

The more of these you need, and the higher the stakes, the higher the price. That's not padding — it's the actual work of keeping a real system running.

Why cheaper is not always better

Here's where my experience makes me opinionated: in production infrastructure, the cheapest quote is frequently the most expensive decision.

A low bid usually means one of a few things — a junior engineer learning on your dime, a scope that quietly excludes monitoring or backups, or a contractor who'll bolt something together and disappear before the technical debt comes due. You don't find out until the pipeline breaks at the worst possible moment, the backups turn out to be untested, or a security gap becomes an incident.

Infrastructure is one of those areas where you're not buying hours — you're buying the absence of disasters. That's hard to see on an invoice and very easy to feel in an outage.

What downtime actually costs

This is the framing that changes the conversation. Put a number on downtime and the "expensive" DevOps quote suddenly looks like a rounding error.

A simple cost-of-downtime model:

Downtime cost per hour = (Annual revenue / Business hours per year) + recovery labor + reputation/churn cost

Work a concrete example. Say a business does $5,000,000 in revenue a year and runs roughly 3,000 business hours. That's about $1,667 per hour in direct lost revenue — before you add the engineers pulled off roadmap work to firefight, the customers who churn, and the support load from a public incident. Call it $2,500–$4,000 an hour, conservatively.

Now consider what causes that downtime in shops without proper DevOps:

Failed deployments with no pipeline safeguards or rollback — a bad release that takes hours to unwind.
Poor monitoring that means you learn about the outage from angry customers instead of an alert, adding 30+ minutes of pure detection delay to every incident.
Manual, undocumented processes where only one person knows how to restore the service, and they're on vacation.

A single multi-hour outage can cost more than a year of competent monitoring and incident-response coverage. The DevOps spend isn't competing with zero — it's competing with the outages it prevents.

How AI changes the math

Part of why DevOps value-for-money has improved is that AI now removes a large slice of the repetitive labor that used to fill the bill.

Drafting and reviewing infrastructure as code. Terraform and Ansible scaffolding that used to take hours gets drafted in minutes, then reviewed by a human.
Pipeline and config generation. GitLab CI/CD configs, Dockerfiles, and Kubernetes manifests start from a solid AI-generated baseline instead of a blank file.
Monitoring setup. Generating sensible Prometheus alert rules and Grafana panels — historically tedious, easily templated work — is far faster with AI assistance.
Incident triage. Summarizing logs and correlating "what changed" compresses the slow part of an outage.

The key word is assisted — a human still owns every change to production. But a provider using AI well can deliver more per dollar, which means you get broader coverage for the same budget. If you want to see the kind of work this accelerates, our prompt library shows the patterns we lean on.

Starting lean: startups and small businesses

If you're early-stage, you do not need a fully managed enterprise engagement, and you shouldn't pay for one. Start with a lean package that covers the essentials and nothing you won't use yet:

A reproducible cloud setup with Terraform, so you're never clicking around a console by hand.
One clean CI/CD pipeline so deploys are boring and repeatable.
Basic monitoring and alerting on the handful of metrics that actually predict outages.
Tested backups.
A documented runbook so recovery doesn't depend on one person's memory.

That's a modest retainer or a small fixed-bid build-out, and it removes the failure modes that sink small companies. You add Kubernetes, deeper observability, and compliance work later — when the business actually needs them, not before. You can see how we structure tiers like this on our pricing page.

How to calculate ROI

Don't buy DevOps on vibes. Run the numbers. A usable formula:

ROI (%) = ((Value gained - Cost of service) / Cost of service) x 100

Where value gained is the sum of:

Downtime avoided — fewer outage hours × your cost-of-downtime-per-hour.
Engineering time reclaimed — hours your developers stop spending on infrastructure toil, at their loaded cost.
Faster delivery — features shipped sooner because the pipeline is fast and reliable.
Incidents prevented — the emergency-rate firefighting you never have to buy.

A worked example. Suppose a managed engagement costs $60,000 a year. Over that year it:

Prevents an estimated 20 hours of downtime at $3,000/hour = $60,000.
Frees two developers from ~5 hours/week of infra work — roughly $50,000 of reclaimed engineering time.
Speeds delivery enough to pull in revenue you'd otherwise have deferred — call it $30,000, conservatively.

That's $140,000 of value against $60,000 of cost:

ROI = (($140,000 - $60,000) / $60,000) x 100 = ~133%

Even if you halve every one of those estimates to be safe, you're still solidly positive. The exercise matters more than the exact figures — when you actually price the downtime you avoid and the time you reclaim, good DevOps consistently pays for itself.

The bottom line

DevOps as a Service pricing genuinely varies, and any provider who hands you a flat number without understanding your systems is guessing. But the framework is straightforward: know which size and complexity bucket you're in, pick the pricing model that fits the work, scope the services you actually need, and run the ROI math against the very real cost of doing nothing.

The mistake I see most often is treating DevOps as a cost line to minimize. It isn't. It's an investment in uptime, delivery speed, security, and the ability to scale without setting your infrastructure on fire. Price it against the outages, the lost engineering hours, and the deals you can't close because the platform won't hold — and the question stops being "what does this cost?" and becomes "what is it costing me not to have it?"

Cost figures and ranges here are illustrative. Build your own estimate from your real revenue, infrastructure, and risk profile before committing to a budget.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

Best DevSecOps Security Tools for CI/CD Pipeline Protection

James Joyner — Sun, 28 Jun 2026 05:05:55 +0000

I've spent twenty-five years building and securing deployment pipelines, and the single biggest shift in that time isn't a tool — it's where security lives. We used to bolt it on at the end, right before a release, when changing anything was expensive and everyone was already tired. That's backwards. DevSecOps is the correction: you move security checks left, into the pipeline, so problems surface when they're cheap to fix and the person who introduced them is still looking at the code.

This is a tour of the tool categories that matter, with representative (mostly open-source) examples and where each one fits in a real GitLab CI/CD or GitHub Actions pipeline. It is not a ranked ad. The right toolchain depends on your team size and how mature your infrastructure is, and I'll come back to that at the end.

What DevSecOps actually means

DevSecOps is "shift-left security": treating security as a property of the pipeline, not a gate at the end of it. Concretely, it means your CI runs the same checks a security reviewer would — scanning code, dependencies, containers, infrastructure definitions, and secrets — automatically, on every push, and fails the build when it finds something that genuinely matters.

The goal isn't to drown developers in findings. It's to catch the dangerous classes of mistakes early and consistently, so security becomes muscle memory instead of a quarterly fire drill.

Why the CI/CD pipeline is a prime target

Your pipeline is the most privileged thing in your engineering org and the least watched. It has credentials to your cloud, your registry, and production. That makes it a high-value target in ways teams routinely underestimate:

Supply-chain attacks. A compromised dependency or a malicious GitHub Action runs inside your build with your secrets in the environment. SolarWinds and the Codecov breach were both pipeline-level compromises, not application bugs.
Secrets sprawl. Pipelines are where API keys, cloud credentials, and registry tokens live. A leaked CI variable or a key hardcoded in a .gitlab-ci.yml is a direct path to your infrastructure.
Runner trust. Shared or self-hosted runners that build untrusted code can be poisoned. A pull request from a fork that triggers a privileged job is a classic foothold.
Artifact tampering. If nothing verifies that the image you deploy is the image you built, an attacker who can write to your registry can swap it.

Every category below is a defense against one or more of these.

SAST — Static Application Security Testing

What it does: scans your source code without running it, looking for vulnerable patterns — SQL injection, command injection, unsafe deserialization, hardcoded crypto mistakes.

Where it fits: early, on every merge request, as a fast job that runs before anything gets built.

Representative tools: Semgrep is my default — it's open-source, fast, and its rules read like the code they match, so writing a custom rule for your own footguns takes minutes. GitLab ships a built-in SAST analyzer you can enable with a single include in your .gitlab-ci.yml. For Python-specific work, Bandit is a lightweight option.

Practical tip: run SAST in diff mode on merge requests so it only flags new findings. Nothing kills adoption faster than a first run that reports 4,000 pre-existing issues and blocks everyone. Baseline the legacy debt, enforce on new code.

DAST — Dynamic Application Security Testing

What it does: attacks a running instance of your app the way an external scanner would — probing for XSS, injection, misconfigured headers, and exposed endpoints.

Where it fits: later in the pipeline, after you've deployed the build to an ephemeral staging environment.

Representative tools: OWASP ZAP is the open-source standard. Its baseline scan runs cleanly as a Docker container in a CI job — point it at your staging URL and it produces a report you can fail the pipeline on.

Practical tip: DAST is slower and noisier than SAST, so don't gate every merge request on a full active scan. Run a quick ZAP baseline scan on merge requests and a deeper scan nightly against staging.

SCA / dependency scanning

What it does: Software Composition Analysis inventories your third-party dependencies and flags ones with known CVEs. Most of the code in your app isn't yours, and this is where most exploitable vulnerabilities actually live.

Where it fits: on every build, and continuously in the background as new CVEs are published against dependencies you already shipped.

Representative tools: Trivy and Grype both scan lockfiles and dependency manifests for vulnerabilities, open-source and fast. For fixing — not just finding — Dependabot (GitHub-native) and Renovate (works everywhere, including GitLab) open automated PRs that bump vulnerable packages.

Practical tip: pair scanning with automated updates. Trivy tells you you're exposed; Renovate is what closes the gap before it rots into a quarter-long upgrade project. A scanner with no remediation path just generates anxiety.

Container image scanning

What it does: scans the layers of a built Docker image — OS packages and language dependencies baked into it — for known vulnerabilities. The friendly node:18 base image you pulled six months ago has accumulated CVEs since.

Where it fits: right after you build the image, before you push it to your registry.

Representative tools: Trivy again (it does both SCA and image scanning, which is why it's so widely deployed), Grype, and Clair, which underpins several registries' built-in scanning.

Practical tip: scan before the push and gate the push on the result. A concrete GitLab CI pattern:

container_scan:
  stage: scan
  image: aquasec/trivy:latest
  script:
    - trivy image --exit-code 1 --severity CRITICAL,HIGH "$IMAGE_TAG"

--exit-code 1 makes the job — and therefore the pipeline — fail on a critical finding, so a vulnerable image never reaches the registry in the first place.

Infrastructure as Code scanning

What it does: scans your Terraform, Ansible, CloudFormation, and Kubernetes manifests for insecure configuration before it provisions anything — public S3 buckets, security groups open to 0.0.0.0/0, unencrypted volumes, over-permissive IAM.

Where it fits: in the plan/validate stage, before terraform apply or ansible-playbook runs against real infrastructure.

Representative tools: Checkov is the broadest — it covers Terraform, Ansible, Kubernetes, and more out of the box. tfsec is Terraform-focused and fast (now converging with Trivy). KICS covers a wide spread of IaC formats.

Practical tip: scan the Terraform plan, not just the source, so you catch issues in the resolved configuration. In a GitHub Actions step:

- name: Checkov on Terraform plan
  run: |
    terraform plan -out=tfplan.binary
    terraform show -json tfplan.binary > tfplan.json
    checkov -f tfplan.json --quiet

This catches the misconfiguration before it becomes a public bucket someone finds for you.

Secrets detection

What it does: scans commits, history, and CI config for leaked credentials — AWS keys, tokens, private keys, passwords. This is the highest-leverage category for the effort involved, because a single leaked cloud key can cost you the whole environment.

Where it fits: everywhere — as a pre-commit hook on the developer's machine and as a CI job that scans the full diff.

Representative tools: Gitleaks and TruffleHog are the open-source workhorses. Run both through the pre-commit framework so secrets get caught before they ever hit the remote.

Practical tip: defense in depth. The pre-commit hook stops the honest mistake locally; the CI job catches the developer who skipped the hook with --no-verify. A combined Trivy-plus-Gitleaks GitLab job that fails the pipeline on a critical finding is one of the cheapest, highest-value things you can add this week.

Policy-as-code

What it does: lets you encode organizational rules as machine-checkable policy — "no container runs as root," "every image must come from our approved registry," "no resource without a cost-center tag" — and enforces them automatically.

Where it fits: two places. In CI, to validate manifests before merge. And at the Kubernetes admission layer, to block non-compliant workloads at deploy time.

Representative tools: OPA with Conftest for testing config files in CI (Terraform plans, Kubernetes YAML, Dockerfiles) against Rego policies. Kyverno for Kubernetes admission control, where policies are written as Kubernetes resources rather than a separate language — a gentler on-ramp for teams already fluent in YAML.

Practical tip: policy-as-code is how you turn "we agreed in a meeting that prod images must be signed" into something the cluster enforces. Start with one or two high-value policies in audit mode, watch what they'd block, then flip to enforce.

This is also the layer that defends against artifact tampering. Generate a signature at build time with cosign and gate the registry push — and the cluster admission — behind a valid signature. If the image isn't the one you built and signed, it doesn't deploy.

Runtime security monitoring

What it does: watches running workloads for suspicious behavior — a container spawning a shell, an unexpected outbound connection, a write to a sensitive path. This is the backstop for everything your pre-deploy scans missed.

Where it fits: in production, continuously, outside the pipeline — but it closes the DevSecOps loop by feeding real attack signals back to the people who build the pipeline.

Representative tools: Falco is the open-source standard, using eBPF to observe kernel-level syscalls with minimal overhead and alert on rule violations. The broader eBPF tooling ecosystem (Cilium, Tetragon) extends this into network policy and deep observability.

Practical tip: runtime security is your evidence that shift-left isn't perfect — and it never is. Treat a Falco alert as both an incident and a signal to add a check earlier in the pipeline so the same class of thing gets caught before deploy next time.

How to choose: team size and infrastructure maturity

You do not need all of the above on day one. Coverage you can't maintain is worse than honest gaps, because it breeds alert fatigue and trains your team to click past warnings.

Lean startup / small team. Pick the three highest-leverage, lowest-friction tools and run them as failing CI jobs: secrets detection (Gitleaks), dependency + image scanning (Trivy), and IaC scanning (Checkov) if you're running Terraform. That's maybe an afternoon of pipeline work and it eliminates the most common ways small teams get owned. Add Dependabot or Renovate so you're patching, not just panicking.

Mid-size, growing team. Layer in SAST in diff mode, DAST against staging, and image signing with cosign. Start introducing policy-as-code in audit mode so you understand your real posture before you enforce.

Mature org with regulatory pressure. Now the full stack earns its keep: enforced policy-as-code at admission, runtime monitoring with Falco, signed-and-verified artifacts end to end, and centralized reporting so security can see across teams. At this scale the integration and dashboards matter as much as the scanners.

The pattern is consistent: each category, run as a job that fails fast on real risk, beats a fancy tool that produces a report nobody reads. If you want to go deeper on pipeline patterns, our GitLab CI/CD prompts cover a lot of this ground.

Where AI fits — assistive, not authoritative

Modern security tooling generates a lot of output, and this is where AI genuinely earns its place. It's good at the reading and summarizing that wears humans down:

Triaging scan output. Paste a wall of Trivy or Semgrep findings and ask the model to group them, identify which CVEs are actually reachable in your code path, and rank by real exploitability. A list of 200 "HIGH" findings becomes "these 6 matter, here's why."
Explaining a finding. "What does this Checkov failure mean and what's the minimal Terraform change to fix it?" turns a cryptic policy ID into an actionable diff.
Drafting remediation. Generate the patched dependency version, the hardened Dockerfile, or the corrected security-group block — as a starting point you review.
Reading pipeline logs. Summarizing a failed, noisy CI job to find the one line that actually broke the build.

The non-negotiable rule, same as during an incident: AI summarizes and suggests; a human verifies and applies. A model will confidently mislabel a vulnerability's severity or propose a "fix" that doesn't compile. Use it to compress the toil, never to make the final security call. We keep a prompt library of these workflows, and the same judgment applies to AI-assisted infrastructure code review — assistive acceleration, human sign-off.

Conclusion

There is no single "best" DevSecOps toolchain. The best one is the one your team will actually use consistently. A perfect scanner that everyone learns to ignore protects nothing — coverage you don't act on is worthless. So start small: pick the tools that fit your pipeline, wire them in as jobs that fail fast on genuine risk, and earn the right to add the next layer. Secrets detection, dependency and image scanning, IaC checks, and signed artifacts will stop the overwhelming majority of how teams actually get compromised. Get those running and reliable first, keep the human in the loop on the AI-assisted parts, and grow the stack as your infrastructure — and your team's appetite — matures.

Security scan output and AI-generated remediations are assistive, not authoritative. Always verify findings and fixes against your own systems before applying them in production.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

DevOps Security Best Practices Every Engineering Team Should Follow

James Joyner — Fri, 26 Jun 2026 14:01:11 +0000

I've spent 25 years securing Linux boxes, cloud accounts, CI/CD pipelines, and production clusters. The single most consistent lesson across all of it is this: the teams that get breached aren't the ones who lacked a security department. They're the ones who treated security as something a separate department would handle later.

Security is not a phase. It's not a gate at the end of the pipeline, and it's not a quarterly audit. It's a property of how you write infrastructure code, manage secrets, ship containers, and run production every single day. When security lives inside the daily workflow — in the merge request, the pipeline stage, the Terraform plan — it costs almost nothing. When it lives in a separate review at the end, it's expensive, late, and routinely skipped.

This is the checklist I'd hand a new engineering team. Everything here is defensive: hardening, detection, and recovery. Work through it section by section.

Why DevOps security belongs in the daily workflow

The whole premise of DevOps was to stop throwing work over the wall between dev and ops. Security is the last wall standing in most orgs, and it has to come down the same way: by moving the controls into the tools engineers already use.

Treat every pull/merge request as a security review surface, not just a code review.
Run security checks as pipeline stages that fail the build, not as advisory reports nobody reads.
Make the secure path the easy path — a hardened base image, a vetted Terraform module, a secrets helper — so engineers don't route around it.
Assign a security owner per service, not a security team for the whole company. Ownership beats oversight.
Measure mean-time-to-remediate for vulnerabilities the same way you measure deploy frequency.

If a control only exists in a wiki page, it doesn't exist. If it exists in the pipeline, it's real.

Secure access control and least privilege

Most incidents I've cleaned up came down to one over-privileged credential. Least privilege is boring and it's the highest-leverage thing on this list.

Default every IAM role, Kubernetes ServiceAccount, and Linux user to zero permissions, then add only what's needed.
Replace standing admin access with just-in-time elevation that expires automatically.
Scope cloud roles to specific resources and actions — no *:* policies, ever.
In Kubernetes, use RBAC Roles bound to namespaces rather than ClusterRole bindings wherever possible.
Separate human identities from machine identities. Humans get SSO; services get workload identity.
Audit who can sudo, who's in the docker group (that's root-equivalent), and who holds cloud admin — quarterly, in writing.

SSH key management and MFA

SSH is still how a huge amount of production gets touched, and it's still where credential hygiene quietly rots.

Disable password authentication entirely: PasswordAuthentication no and PermitRootLogin no in sshd_config.
Use per-user keys, never a shared key passed around in a chat thread.
Prefer short-lived SSH certificates from a CA over long-lived static keys; rotate the rest on a schedule.
Put a bastion/jump host in front of production and log every session through it.
Require MFA on every identity provider, VPN, and cloud console — phishing-resistant (WebAuthn/hardware keys) for anyone with production access.
Pull keys for departed team members the same day, and audit authorized_keys files for orphans.

Secrets management: API keys, passwords, and tokens

The fastest way to leak a secret is to commit it. The second fastest is to print it. Both are entirely preventable.

Never store secrets in git — not in code, not in .env, not in a "temporary" YAML file. Add a pre-commit secret scanner (gitleaks or trufflehog) to block it.
Centralize secrets in a real secrets manager: HashiCorp Vault, a cloud secrets manager, or equivalent.
For Kubernetes, use Sealed Secrets or an external-secrets operator so the cluster pulls from Vault at runtime — plain Secret objects are only base64, not encrypted.
Give every secret a rotation policy and an owner. Static credentials that never rotate are time bombs.
Inject secrets as runtime environment values or mounted files, not baked into container images or Terraform state.
Scan your git history, not just the current tree — a secret deleted in HEAD is still in the log until you rotate it.

CI/CD pipeline security

Your pipeline has credentials to everything. That makes it one of the highest-value targets you own, and it's frequently the least hardened.

Protect your main branches: require reviews, status checks, and signed commits before merge.
Mark CI/CD variables as protected and masked so they're only exposed on protected branches and never echoed to logs.
In GitLab CI, scope variables to environments and never echo a secret — masking helps, but the discipline of not printing it is what saves you.
Replace long-lived cloud keys in CI with short-lived credentials via OIDC. Let the pipeline exchange its identity for a temporary, scoped token instead of holding a static AWS_SECRET_ACCESS_KEY.
Pin and review your pipeline dependencies — third-party CI templates and actions run with your pipeline's privileges.
Require manual approval for production deploys, and make the deploy job itself least-privileged.

A leaked CI variable is a leaked production credential. Treat the pipeline config with the same care you'd treat root.

Container image scanning

A container is only as trustworthy as the layers underneath it. Most images ship with known CVEs the team never looked at.

Scan every image with Trivy (or Grype) as a GitLab pipeline stage before push, and fail the build on high/critical findings:

  container_scan:
    stage: test
    image: aquasec/trivy:latest
    script:
      - trivy image --exit-code 1 --severity HIGH,CRITICAL "$IMAGE_TAG"

Start from minimal base images (distroless or slim) to shrink the attack surface.
Run containers as a non-root user (USER in the Dockerfile) with a read-only root filesystem where possible.
Drop all Linux capabilities and add back only what's required.
Pin base images by digest, not by floating :latest tags, and rebuild regularly to pick up patches.
Sign images and verify signatures at admission so only your builds run in your cluster.

Infrastructure as Code security

IaC is where a one-line mistake becomes a fleet-wide misconfiguration. The good news: it's also where automated policy catches it before it ships.

Review Terraform and Ansible changes like application code — every change goes through a merge request with a human reviewer.
Run static analysis on IaC in the pipeline: tfsec/Checkov for Terraform, ansible-lint and kube-linter for the rest.
Adopt policy-as-code (OPA/Conftest or Sentinel) so rules like "no public S3 buckets" and "no 0.0.0.0/0 on port 22" are enforced automatically, not remembered by reviewers.
Protect and encrypt Terraform state — it contains secrets in plaintext. Use a remote backend with locking and access controls.
For Ansible, encrypt sensitive variables with Vault and avoid become where it isn't needed.
Diff the plan before every apply and require approval for changes to security groups, IAM, and networking.

If you want a structured second opinion on a risky module, an automated infrastructure code review catches the misconfigurations a tired reviewer skims past at the end of the day.

Patch management and vulnerability remediation

Unpatched systems are the most common root cause of breaches, and the least glamorous to fix. Make it routine so it isn't a decision.

Automate OS patching with unattended security updates on Linux, and rebuild container images on a cadence rather than letting them age.
Track your dependencies with SBOMs so you can answer "are we affected?" the day a CVE drops.
Subscribe to advisories for your stack and define an SLA: criticals patched in days, highs in a week or two.
Use Dependabot/Renovate to open dependency-bump PRs automatically and run them through your test suite.
Keep an inventory of every host, image, and cluster version — you can't patch what you don't know you run.

Monitoring, logging, and alerting for security events

You cannot respond to what you can't see. Detection is the difference between a contained incident and a postmortem that starts with "we think they were in for three months."

Enable Linux auditd and ship /var/log/auth.log, sudo events, and SSH activity to centralized, append-only storage.
Export security metrics to Prometheus and build Grafana dashboards plus alerts for anomalies: failed-login spikes, new sudo grants, unexpected outbound connections, root logins, container escapes.
Alert on auditd/SSH anomalies in Grafana — a burst of failed SSH from a new ASN, or a successful root login outside business hours, should page someone.
Turn on cloud audit logging (CloudTrail or equivalent) and alert on IAM policy changes, new access keys, and security-group edits.
Capture Kubernetes audit logs and alert on exec into production pods and changes to RBAC.
Keep logs immutable and retained long enough to investigate a slow-burn intrusion.

Backup and disaster recovery planning

Ransomware and fat-fingered terraform destroy have the same fix: backups you've actually tested. Untested backups are just hope.

Follow 3-2-1: three copies, two media types, one off-site and offline.
Keep at least one immutable/air-gapped copy that a compromised admin credential can't delete.
Encrypt backups at rest and control who can read and restore them.
Test restores on a schedule. A backup you've never restored is a guess, not a recovery plan.
Document RPO and RTO per service and verify your backup cadence actually meets them.
Back up the things people forget: Terraform state, Vault data, etcd, and database credentials.

Incident response preparation

The middle of an incident is the worst time to figure out your incident process. Prepare while it's calm.

Write a runbook: who's on call, how to declare an incident, where to communicate, and how to reach the right people fast.
Pre-define severity levels and the actions each triggers.
Keep break-glass credentials in a sealed, audited path — available in a crisis, logged when used.
Practice. Run a tabletop or game day at least quarterly so the steps are muscle memory.
Have communication templates ready — customer-facing and internal — so comms don't stall the investigation.
Draft blameless postmortems while the timeline is fresh, and turn action items into tracked work.

AI-assisted security checks — review, never blind trust

AI is genuinely useful for security work: it reads more YAML, Terraform, and logs than you can, and it's fast at spotting the misconfiguration buried in a 400-line diff. The rule is the same one I apply to everything an AI generates — it drafts, you decide.

Use AI to review IaC and pipeline configs for missing controls, over-broad permissions, and risky defaults — as a first-pass reviewer, not the final word.
Have it summarize scanner output and rank findings by real-world exploitability so your team fixes what matters first.
Let it draft hardening changes and policy rules, then read every line before you apply it — confident and correct are not the same thing.
Never paste live secrets, real hostnames, or customer data into a model. Scrub first.
Keep a human approving every change that touches IAM, networking, or production. AI accelerates the work; it doesn't own the risk.

If you want vetted starting points, our security & hardening prompts cover image scanning, Linux hardening, and IaC review, and the broader prompt library has the rest.

The bottom line

None of these practices are exotic. Least privilege, managed secrets, scanned images, policy-checked infrastructure, patched hosts, real monitoring, tested backups, and a rehearsed incident plan — every one is achievable this quarter, and every one is cheaper to build into the workflow than to bolt on after a breach.

And here's the part that doesn't show up on the engineering scorecard: secure DevOps is a competitive advantage. Customers run security questionnaires before they sign. Investors ask about your posture in diligence. Partners won't integrate with a platform they don't trust. The companies that win those deals are the ones who can show, not claim, that protecting customer systems is built into how they work every day.

Security isn't the tax you pay to ship. It's part of why people let you ship to them at all.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

Why AI Loves Ansible (And You Should Let It Help)

James Joyner — Thu, 25 Jun 2026 04:36:48 +0000

If you compare how well Claude handles Ansible against how well it handles, say, raw bash or kubectl YAML, Ansible wins by a wide margin. The reason isn't subtle: Ansible's shape — declarative, idempotent, modules-with-arguments — happens to map almost perfectly to how LLMs reason. They're good at producing structured output that fills in a known template, and that's what most Ansible tasks are.

This means AI-assisted Ansible work is the highest-leverage automation pairing I know of. If you only adopt AI for one infrastructure tool, make it Ansible.

What makes Ansible AI-friendly

Modules have published contracts

Every Ansible module has a documented argument spec: what's required, what's optional, what the defaults are. The model can fit your intent into the spec with high accuracy because the spec is finite and known.

Compare this to shell: there are a thousand ways to "create a user with a specific UID, member of these groups, with this shell, and a home directory in this location." In bash, every distro is slightly different. In Ansible, you use ansible.builtin.user with named arguments.

The model gets this right every single time.

Idempotency is the default

When a model generates a Python script, it has to think about "what if this is run twice." When it generates Ansible, most modules handle that for free. The model can write the task, ignore the re-run case, and produce something that works.

This means the cognitive load on both sides — model and human — is lower. You're describing the target state, not the procedure.

Roles and structure are predictable

roles/foo/{defaults,vars,tasks,handlers,templates,files,meta}/main.yml — every Ansible role looks the same. The model can scaffold a new role in seconds because the layout is fixed.

If you ask Claude to "create a new role for installing PostgreSQL 16 on Ubuntu 24.04 with default user postgres and a tuned postgresql.conf," you'll get a complete role structure with defaults/main.yml, tasks/main.yml, a Jinja template, and handlers/main.yml — all consistent, all in the right places. The structure is constrained enough that the model rarely improvises.

Use cases where AI shines for Ansible

Generating new roles from scratch

This is the killer app. You can describe a role in two sentences and get a 90%-done implementation. You then refine: add validation, adjust defaults, write a README.

I now treat "draft a new role with Claude" as the default first step. Even if I rewrite half of it, the structure saves me 20 minutes.

Converting shell scripts to playbooks

If you have a legacy bash script that provisions a server, pasting it into Claude with "convert this to an idempotent Ansible playbook using the appropriate modules" produces a usable result. The model knows when to use ansible.builtin.file, lineinfile, template, service, etc.

You'll need to verify the idempotency manually (run twice, expect 0 changes on the second run), but the conversion is mostly mechanical.

Refactoring playbooks to use FQCN

Ansible 2.10+ wants fully-qualified collection names: ansible.builtin.package instead of package. Old playbooks have hundreds of short-form references. AI is a perfect fit for this kind of mass refactoring — it knows the mapping and won't get bored.

Paste a 200-line playbook, ask for it back with FQCN throughout, and you're done in 30 seconds. Verify with ansible-lint.

Writing Molecule tests

Molecule scaffolding is repetitive — same molecule.yml, same converge.yml, same verify.yml structure for most roles. AI is great at generating the boilerplate. You describe what you want to test; the model writes the assertion playbook.

Jinja template generation

Jinja is just structured-enough that AI handles it well — generating templates for config files (nginx, postgres, sshd) from a description of the desired behavior. The model knows the configuration keys and the conditional structure.

Where AI struggles with Ansible

Variable precedence

Ansible's 21-layer variable precedence rules are not intuitive. The model will sometimes suggest putting a variable in vars/main.yml when you really want it in defaults/main.yml (the former overrides the latter). The result: users of your role can't override the variable they expected to.

Check: When the model puts something in vars/, ask "should this be overridable by the role user?" If yes, move to defaults/.

Custom facts and `set_fact` lifetime

The model sometimes uses set_fact for values that need to persist across plays, but doesn't add cacheable: true. The fact is then gone after the play ends, and the next play sees undefined.

Check: When you use set_fact for a value you need later, verify the lifetime is what you expect.

Vault integration

The model will sometimes generate playbooks that reference vault_db_password as a variable but don't include the lookup('community.hashi_vault.hashi_vault', ...) call or the Ansible Vault encrypted file. You have to wire up the secret source separately.

Check: For any sensitive variable in a generated playbook, verify there's an actual source for it (Vault encrypted file, external manager lookup, environment variable).

Distro-specific paths

The model defaults to Debian/Ubuntu conventions. If you run on RHEL, you'll sometimes get apt modules in tasks that should be using the package module (or distro conditionals).

Check: When generating playbooks for non-Debian systems, audit for apt, apt_repository, dpkg_selections, and ask for the abstraction (package) or the distro split.

A workflow that's been working for me

For a new role, my process now looks like this:

Describe the role to Claude in 2-3 sentences (purpose, target distros, key behaviors).
Generate the scaffolding: defaults/main.yml, tasks/main.yml, a template if needed, meta/main.yml with platforms.
Read every task. Look for the failure modes above (precedence, lifetime, Vault, distros).
Add Molecule tests. Have Claude scaffold molecule/default/, then write the assertions yourself or ask for them.
Run ansible-lint and Molecule. Fix what they catch.
Idempotence check. Run the role twice; second run should report 0 changed.
Refine the README. This is the one place I write from scratch — explaining the role to future-me.

This takes maybe 30 minutes for a moderately complex role. Without AI assistance, the same role would take me a couple of hours.

A note on safety

Ansible runs as root on production servers. Whatever the model generates, you are responsible for what it does. Two patterns I follow:

Check --check --diff before any real run. Dry-run the playbook in check mode; verify the diff matches what you expect.
Test on a sandbox host first. Especially for new roles. Don't trust the model with production until the role has run cleanly on a throwaway VM.

These are the same disciplines that apply to any infrastructure change. AI doesn't change the discipline; it just makes you faster at the parts before the change.

Why I think Ansible is the right entry point

If you're new to using AI for infrastructure work and want to pick one tool to start with, Ansible is the safest, highest-leverage choice. The structure makes the AI accurate. The idempotency makes mistakes recoverable. The module ecosystem covers most common cases.

By the time you've used AI to write a dozen Ansible playbooks, you'll have developed the intuition for what AI handles well and what needs human attention. That intuition transfers to harder tools — Terraform, Kubernetes, custom shell — where the cost of AI mistakes is higher.

For our full set of AI-driven Ansible workflows, see the IaC category — including ansible-vault-secrets-management and ansible-molecule-testing.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

AI for GitLab CI Authoring: Save Hours, Avoid Footguns

James Joyner — Sat, 20 Jun 2026 12:15:22 +0000

GitLab CI YAML is one of those formats where you can stare at it for an hour, get it 95% right, and have it fail with yaml: line 12: did not find expected key because of a tab character. AI assistants are very fast at this kind of work. They're also confidently wrong about specific GitLab features in ways that waste a lot of time if you don't know what to check.

After a year of letting Claude write a lot of my pipelines, here's what works and what doesn't.

What AI gets right consistently

Standard job shapes

"Write me a job that builds a Docker image, pushes to the GitLab Container Registry, and tags with the commit SHA and latest on the default branch." Type that into Claude and you get a working job in five seconds. The shape is well-established and the model has seen thousands of variations.

The same is true for:

Test jobs across languages (pytest, jest, go test, etc.)
Standard cache configurations
Standard artifact patterns
Basic rules: for branch / tag / MR pipelines

If you find yourself writing one of these from scratch, you're spending time that you don't need to spend.

Translating from other CIs

GitLab CI has obvious parallels to GitHub Actions, CircleCI, Jenkins declarative pipelines, etc. AI is excellent at translating between them. The structures rhyme; the model knows the dictionary.

If you're migrating from Actions to GitLab CI, paste the workflow and ask for the GitLab CI equivalent. You'll get something 80% right that you can refine.

Reviewing pipelines for inefficiency

This is the underrated use case. Paste your .gitlab-ci.yml and ask: "what's the critical path of this pipeline, and what's making it slow?" The model will spot things like:

"Your test job downloads node_modules from cache, but install-deps doesn't push to cache — your cache key is broken."
"Your build and deploy stages are sequential but build's artifacts aren't used by deploy — they can be parallel with needs:."
"Your rules:changes: doesn't include package-lock.json, so dependency changes don't retrigger tests."

These are real findings I've gotten from Claude on pipelines I thought I'd already optimized. Worth the five-minute review.

What AI gets wrong — and how to catch it

`rules:` vs `only/except` confusion

The model will sometimes mix them in the same job. GitLab silently ignores only: when rules: is also defined. The pipeline runs but the behavior isn't what you expect.

Check: Are you using rules: OR only:/except: in each job? Pick one. (Use rules: — only/except is legacy.)

`$CI_COMMIT_BRANCH` empty on MR pipelines

A common bug: you ask for "this job runs on the default branch" and you get:

rules:
  - if: $CI_COMMIT_BRANCH == "main"

This is correct for branch pipelines. It is empty on MR (merge_request_event) pipelines. If you have MR pipelines enabled, your job silently won't run when developers expect it to.

Check: Does your pipeline target both push events and MR events? If so, you probably want $CI_MERGE_REQUEST_TARGET_BRANCH_NAME or to handle both pipeline sources.

`needs:` referencing hidden jobs

Hidden jobs (prefixed with .) are templates — they don't execute. If you do needs: [".lint"], your job will fail with a confusing error because GitLab thinks you're depending on a job that doesn't exist.

Check: Every needs: entry should be a real job name, not a template.

Auto-apply rules that don't include the right branches

The model loves writing:

rules:
  - if: $CI_COMMIT_BRANCH == "main"
    when: always
  - when: never

This works on main but blocks the job on tags, on schedules, and on MR pipelines. Sometimes that's what you want. Often it's not.

Check: What pipeline sources do you expect this job to run in? List them, then verify your rules cover each.

Imaginary GitLab features

This is the most expensive AI failure mode. The model will sometimes generate syntax for features that don't exist:

A condition: field that's actually OPA/Conftest, not GitLab CI
An auto_retry: block that's GitHub Actions, not GitLab
A before_script: keyword that does exist but with different semantics than the model claims

Check: If you see a keyword you haven't seen before in GitLab docs, verify it exists. The lint endpoint (/api/v4/ci/lint) catches most of these, but some pass lint and just behave weirdly.

A workflow that catches the failures cheaply

I now do this for any non-trivial pipeline change:

Draft with AI. Describe the desired behavior in plain English; let the model write the YAML.
Read every line. Treat the output as a draft you'd write yourself.
Lint via the API.

   curl -s --header "PRIVATE-TOKEN: $TOKEN" \
       --header "Content-Type: application/json" \
       --data "{\"content\": $(cat .gitlab-ci.yml | jq -Rs .)}" \
       "$GITLAB_URL/api/v4/ci/lint" | jq

Run on a sandbox branch. Push to a branch that won't trigger deploys; verify the pipeline runs the jobs you expect, when you expect.
Diff against the existing pipeline. If the AI introduced changes you didn't ask for (a different cache key, a removed interruptible:), revert them.

Step 5 is the one most people skip. The model is good at writing YAML but not at preserving your previous decisions. If you don't diff, you'll lose your old cache strategy.

A practical example

Last month I needed to add a job that runs terraform plan on every MR and posts the output as a comment. Drafted with Claude in two minutes; it produced something like:

terraform-plan:
  image: hashicorp/terraform:1.9
  stage: plan
  script:
    - terraform init
    - terraform plan -out=tfplan -no-color
    - terraform show -no-color tfplan > plan.txt
    - |
      curl -X POST -H "PRIVATE-TOKEN: $GITLAB_API_TOKEN" \
          -d "body=$(cat plan.txt | jq -Rs .)" \
          "$CI_API_V4_URL/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

This is almost right. Two issues:

PRIVATE-TOKEN as a CI variable — using a personal access token for CI is the old pattern. Modern approach: use $CI_JOB_TOKEN for in-instance API calls. Saves rotation pain.
No terraform init -backend-config — works if the backend is configured in code, but if you have multiple environments using the same module, you'd want to specify which backend.

Both fixes are 30 seconds. Without the AI I'd have spent 15 minutes writing the curl invocation alone.

The bottom line

AI doesn't replace knowing GitLab CI. It removes the typing and the boilerplate so you can spend your attention on the parts that matter — the rules: logic, the cache keys, the secrets, the environment promotion.

Once you've internalized the failure modes above, the workflow becomes mostly automatic. You stop reading the boilerplate and start reading the rules. That's where the bugs live.

For the prompt set we use on GitLab CI specifically, see the GitLab CI/CD category — particularly gitlab-pipeline-optimization and gitlab-ci-rules-debugging.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

Securing AI-Generated Bash Scripts Before You Run Them

James Joyner — Thu, 18 Jun 2026 15:51:56 +0000

Bash is the easiest language for AI to write and the easiest language to get devastating output from. A 20-line script that "just cleans up old files" can recursively delete a home directory because the model assumed a variable would always be set. A "simple log shipper" can write your secrets to a remote server because the model used set -x for debugging and forgot to remove it.

I have run AI-generated bash that I should not have. Most engineers I know have too. After enough close calls, there's a short checklist that catches the worst of it. This is that checklist.

The five things to check before running any AI-generated bash

1. Does it start with a strict pragma?

The first lines of any non-trivial bash script should be:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

What each does:

set -e — exit on any command failure. Without this, a failure in line 5 doesn't stop the script from happily running lines 6-50.
set -u — error on undefined variables. This is the one that saves you from rm -rf $UNDEFINED/.
set -o pipefail — propagate failures through pipes. Without it, failing-command | grep something succeeds because grep succeeds.
IFS=$'\n\t' — sane field splitting. Defends against word-splitting bugs in filenames.

If the AI-generated script doesn't have these, add them and re-read the script. You'll often discover bugs the pragma now flags.

2. Is every variable expansion quoted?

# Wrong
rm -rf $TARGET_DIR

# Right
rm -rf "$TARGET_DIR"

The wrong version is what causes the "I deleted the root directory" stories. If $TARGET_DIR is empty or contains a space, the command becomes rm -rf (delete current directory) or rm -rf foo bar (delete two unintended things).

Models default to the wrong version about half the time because the right version is harder to write in chat ("escape the quotes!") and the wrong version is what most blogs show.

Fix: When reading AI bash, mentally check every $VAR for quotes. Add them if missing. This is the single biggest source of bash disasters.

3. What happens if a step fails partway through?

The AI will cheerfully write:

mkdir -p /opt/new-app
cd /opt/new-app
tar xzf $TARBALL
rm $TARBALL

What happens if tar xzf fails (corrupt tarball, full disk)? With set -e, the script stops. Good. Without set -e, it continues to rm $TARBALL and deletes your tarball with no backup.

For any state-changing script, ask yourself: at each step, what's the recovery path if the step fails? If the answer is "nothing automated," the script should at least not delete data before verifying the previous step succeeded.

The AI almost never thinks about this on its own.

4. Are secrets visible in logs?

The most common way AI-generated bash leaks secrets is via set -x:

set -x  # debugging
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/...

With set -x, every command is printed including the expanded variables. Your API token is now in the script's output, which is in your CI logs, which are visible to anyone with project access.

The fix is selective:

set +x  # disable trace
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/...
set -x  # re-enable

Or simply remove set -x once debugging is done. The model frequently leaves it in.

5. Does it run as root unnecessarily?

The AI will sometimes write sudo into every command, even ones that don't need it. Or it'll assume the script runs as root and use absolute paths that require root to write.

The principle: if a command can run as a non-root user, it should. The smaller the privileged surface, the smaller the blast radius.

This is especially important for scripts that download and execute code. A common pattern:

# Dangerous: privileged download + execute
sudo bash -c 'curl https://example.com/install.sh | bash'

# Safer: review then run
curl https://example.com/install.sh > install.sh
# READ install.sh
sudo bash install.sh

If the model generates the first pattern, replace it with the second. Always.

A real example

Last month I asked Claude to write a script that cleans up Docker images older than 30 days on a CI runner host. The first draft was:

#!/bin/bash

DOCKER_IMAGES=$(docker images --format '{{.ID}} {{.CreatedAt}}')
CUTOFF=$(date -d '30 days ago' +%s)

echo "$DOCKER_IMAGES" | while read ID DATE; do
    CREATED=$(date -d "$DATE" +%s)
    if [ $CREATED -lt $CUTOFF ]; then
        docker rmi $ID
    fi
done

Walking the checklist:

No strict pragma. Missing set -euo pipefail.
Unquoted $DOCKER_IMAGES, $ID, $DATE. Each one is a potential bug.
Failure handling. docker rmi fails if an image is in use. The script continues, marches through, and silently fails on every in-use image. We never know which were cleaned and which weren't.
No secrets (docker doesn't expose them here), but the script also doesn't log what it's doing, so you can't audit afterward.
No sudo, good — assumes the user has Docker socket access, which is reasonable.

The hardened version:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

CUTOFF=$(date -d '30 days ago' +%s)
REMOVED=0
SKIPPED=0

# Use --format with safer parsing
docker images --format '{{.ID}}|{{.CreatedAt}}' | while IFS='|' read -r ID DATE; do
    CREATED=$(date -d "$DATE" +%s)
    if [ "$CREATED" -lt "$CUTOFF" ]; then
        if docker rmi "$ID" 2>/dev/null; then
            echo "Removed: $ID"
            REMOVED=$((REMOVED + 1))
        else
            echo "Skipped (in use): $ID"
            SKIPPED=$((SKIPPED + 1))
        fi
    fi
done

echo "Cleanup complete. Removed: $REMOVED, Skipped: $SKIPPED."

This took two minutes of editing. Without the checklist, I might have run the original and noticed days later that disk usage hadn't really dropped because half the images were in use.

A small note on bash linting

shellcheck catches most of these issues automatically. If you adopt one tool from this article, make it shellcheck:

shellcheck cleanup-images.sh

It will flag unquoted variables, missing strict mode, and a dozen other patterns. AI-generated bash usually has at least one shellcheck warning.

I now run shellcheck on every script before I run the script itself. It's two seconds and catches things I'd miss.

When the AI gets it right

To be fair: the model is often perfectly capable of producing safe bash. If you prompt it explicitly — "write this with set -euo pipefail, quote every variable, fail loudly on errors" — you'll get a clean script.

The problem is that "write me a script that does X" without that prompt gets you the common form of the script, which is the unsafe form. So the rule of thumb:

Always include the safety requirements in the prompt. Or: always treat the output as a draft that needs hardening. Don't run any bash the AI wrote without one of those two disciplines.

The bottom line

Bash from AI is fast to produce and easy to read incorrectly. The checklist is short — strict pragma, quoted expansions, failure paths, secrets in logs, unnecessary privilege — and applying it takes a couple of minutes per script. The downside of skipping it is on the spectrum of "minor cleanup mistake" to "career incident." There's no excuse not to do the check.

For our prompts on bash specifically, see bash-script-code-review and the related linux-server-hardening prompt — both of which cover related territory.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

The Best AI Tools for DevOps Engineers in 2026

James Joyner — Wed, 17 Jun 2026 20:59:44 +0000

If you spend your day in a terminal, a YAML editor, or a Grafana tab — AI assistants in 2026 are no longer a curiosity. They're a real productivity layer. But not every tool is good at infrastructure work. After a year of daily use across Linux administration, OpenStack operations, Prometheus alert authoring, and Kubernetes debugging, here's the honest shortlist.

The criteria

We're not ranking on benchmark scores. We're ranking on infrastructure usefulness:

Reasoning over command output — can it actually read top, kubectl describe, or journalctl and find the real problem?
Safety — does it warn before suggesting destructive commands?
Long context — can it hold a 1,000-line .gitlab-ci.yml plus failing logs without losing track?
Terminal integration — can you use it without leaving your workflow?
Privacy and self-host options — for the engineers whose employers care.

The shortlist

1. Claude (Anthropic)

The current best general assistant for infrastructure reasoning. Long context handles enormous log dumps and Kubernetes manifests in one shot. It is consistently more cautious about destructive commands than alternatives — which matters when you're tired at 2am and tempted to copy-paste straight into prod.

Best for: Linux/OpenStack/Kubernetes troubleshooting, postmortem drafting, code review on infrastructure-as-code.

2. ChatGPT (OpenAI)

The broadest ecosystem. Strong code generation, plug-in support, and the largest community of shared prompts and patterns. For Ansible and Terraform generation, output quality is excellent. Slightly less cautious by default — you'll want to add safety constraints in your prompts.

Best for: Ansible/Terraform generation, ad-hoc scripting, learning new tools.

3. Cursor

If you live in an IDE, Cursor is what your IDE should have been. Native multi-file context, agent mode for repo-wide refactors, and tab-completion that actually understands your codebase. Especially strong for IaC repositories with many interconnected files.

Best for: Editing real codebases (Helm charts, Terraform modules, Python operators).

4. GitHub Copilot

The lowest-friction option. Inline completion just works, and the chat sidebar is genuinely useful for "explain this regex" or "what's this PromQL doing?" If your org already pays for GitHub, Copilot is essentially free upside.

Best for: Inline completion while editing YAML, Bash, Python.

5. Warp Terminal (with AI features)

The only entry on this list that isn't an AI assistant per se — it's a terminal that has AI built in. The killer feature: natural-language command suggestions in your shell, with safety previews. For Linux admins who don't want to alt-tab to a chat window every five seconds.

Best for: Terminal-native workflows where context-switching kills focus.

What we don't recommend (yet)

Generic LLM wrappers that promise "DevOps AI." Most are thin layers over the same APIs above, sometimes with worse safety defaults. Use the underlying tools directly.
Anything that requires uploading your ~/.ssh directory or production credentials. Be skeptical of "AI agents that run commands for you" without a clear sandbox model.

How to combine them

A pattern that works well in practice:

Claude or ChatGPT in a browser for deep diagnosis sessions (paste logs, walk through hypotheses, draft postmortems).
Cursor or Copilot in your editor for actually writing the fix.
Warp in the terminal for quick command lookups without switching context.

You don't need one perfect tool. You need a workflow where each tool plays to its strengths.

Auditing Kubernetes Manifests With AI: A Practical Workflow

James Joyner — Tue, 16 Jun 2026 04:31:15 +0000

A senior K8s engineer I work with audits manifests faster than I read them. He's seen so many patterns that "missing readinessProbe on a Deployment that takes 45 seconds to start" jumps off the page. Most of us don't have that pattern library memorized — and increasingly, we don't need to. AI assistants have read more Kubernetes manifests than any human ever will.

The catch: a generic "review this YAML" prompt produces generic noise. You need to direct the model toward the categories of issues that actually matter in your environment.

The two mistakes everyone makes

Mistake 1: Asking for "a security review." You'll get a bullet list of every possible concern, ranked alphabetically, with no signal about which matter. You'll skim, dismiss, and learn nothing.

Mistake 2: Pasting one manifest. Real Kubernetes problems live in the interaction between resources — a Deployment's readiness probe and a Service's selector, a NetworkPolicy and the actual app traffic. One YAML in isolation hides most of the bugs.

The fix for both is the same: give the model a bounded scope and enough context to reason about interactions.

A workflow that works

Step 1: Pick the audit dimension

Pre-decide what you're checking for. Different prompts for different dimensions:

Resource limits & QoS — are requests/limits set, does QoS match intent, are limits realistic
Probes & lifecycle — readiness, liveness, startup, preStop, terminationGracePeriodSeconds
Security context — runAsNonRoot, capabilities, readOnlyRootFilesystem, seccomp
Network exposure — NetworkPolicy, Service type, Ingress rules
Reliability — PodDisruptionBudget, topology spread, replica count
State & storage — PVC access modes, retention policies, backup tags

Mixing dimensions in one review produces wishy-washy output. Pick one, get a clean answer, move on.

Step 2: Paste the manifest + related context

For a workload review, paste:

The Deployment / StatefulSet / DaemonSet
Its Service(s) and Ingress
Any NetworkPolicies that match its labels
The HPA if relevant
The ConfigMaps and Secrets it references (sanitize first)

For YAML this is usually under 500 lines, well within any model's context window. The model can now reason about interactions, not just isolated fields.

Step 3: Use a directive prompt

The big difference between "tell me about this YAML" and a useful review is the instruction format. Compare:

Review this Kubernetes manifest.

versus:

You are reviewing a production Deployment + Service + NetworkPolicy bundle. For each finding, give: (1) severity (critical/high/medium/low), (2) the exact field path that's wrong, (3) one sentence on why it matters, (4) the corrected YAML snippet. Focus only on probes, lifecycle, and graceful shutdown. Ignore documentation/comments.

The first prompt produces an essay. The second produces a list of fixable issues.

Step 4: Verify before applying

This is where most reviews go wrong. The model is right most of the time. It's wrong some of the time, often in ways that look correct.

Common AI failure modes in K8s review:

Hallucinated field names — spec.template.spec.terminationGracePeriod (it's terminationGracePeriodSeconds)
Outdated API versions — policy/v1beta1 PodDisruptionBudget (removed in 1.25)
Wrong defaults claimed — claiming failureThreshold defaults to 1 when it's 3
Misreading the use case — recommending runAsNonRoot: true for a workload that legitimately needs root

For every "fix" the model suggests, glance at the official K8s docs for that field. This adds 30 seconds per finding and catches the wrong ones. Without this step, you will apply changes that break things.

A real example

Here's a Deployment I reviewed last week:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
spec:
  replicas: 2
  selector:
    matchLabels: { app: payments }
  template:
    metadata:
      labels: { app: payments }
    spec:
      containers:
      - name: app
        image: registry.example.com/payments:v3.1.0
        ports:
        - containerPort: 8080
        env:
        - name: DB_URL
          value: postgres://payments-db:5432/payments
        resources:
          limits:
            cpu: "2"
            memory: "2Gi"
        readinessProbe:
          httpGet: { path: /healthz, port: 8080 }
          initialDelaySeconds: 5

I asked Claude to review for probes and graceful shutdown only. The findings:

No requests, only limits → pod gets BestEffort QoS, first to be evicted under pressure. Set requests equal to or below limits.
initialDelaySeconds: 5 → Java/Spring apps typically need 30-90 seconds to start. Add startupProbe with longer threshold.
No livenessProbe → kubelet won't restart if the app deadlocks. Mirror readinessProbe with looser thresholds.
No terminationGracePeriodSeconds → defaults to 30s; for a payment service with in-flight requests, this is borderline. Set to 60s.
No preStop hook → SIGTERM hits immediately; load balancers may still send traffic for ~10s after pod marked Terminating. Add sleep 15 preStop.

All five were real, all five were fixable in two minutes of YAML editing. The model didn't tell me about anything irrelevant. That's because I scoped the prompt to "probes and graceful shutdown only."

The big one — #5 — is something I've personally been bitten by twice. The model wouldn't have prioritized it without the directive prompt.

What about Kyverno / OPA / Pod Security Admission?

Yes, you should run those too. They catch consistent issues at admission time. They don't catch issues that require judgment: "is 30 seconds enough graceful shutdown for this specific service?" Policy enforcement is a floor; AI review is a directed second opinion above that floor.

I run both. Kyverno catches "no securityContext at all" before it ever lands. AI review catches "readinessProbe path doesn't match what the app exposes" — something only a human (or an AI imitating one) would notice.

A starter prompt

If you want a template, here's the one I use most:

You are reviewing a Kubernetes workload bundle for production readiness. Focus only on: probes (readiness, liveness, startup), terminationGracePeriodSeconds, preStop hooks, and rolling update strategy. For each finding produce: severity, exact field path, why it matters in one sentence, corrected YAML. Ignore everything else (security context, network policies, resource limits — those are separate reviews). The workload is [serves HTTP at /api on port 8080 / consumes from a queue / batch processor that runs N hours].

The bracketed context at the end is what makes the review accurate for your workload. Without it, the model assumes a generic web service.

For our full prompt library on Kubernetes review, see the Kubernetes & Helm category — especially kubernetes-yaml-security-review and kubernetes-resource-limits-tuning.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

How to Use Claude to Troubleshoot Linux Servers

James Joyner — Sun, 14 Jun 2026 21:15:59 +0000

Claude is genuinely useful for production Linux troubleshooting — when you use it right. Here's the workflow that works, after a year of using it on real incidents across Ubuntu, RHEL, and Rocky.

The mental model: Claude is a senior pair, not an oracle

The mistake most engineers make on day one: they paste a 5-line error message and expect a fix. Claude can do better than that — but only if you give it the same context you'd give a senior engineer joining your incident bridge.

A senior engineer would want:

What OS and version?
What does this server do?
What changed recently?
What's the actual symptom?
What command output have you already gathered?

Give Claude that, and the quality of analysis changes completely.

The workflow

Step 1: Establish context with a system prompt

Use our Linux Server Troubleshooting Prompt as your system prompt, or paraphrase: "You are a senior Linux sysadmin. Rank root-cause hypotheses by probability. Recommend safe diagnostics first. Label destructive commands as DANGEROUS."

Step 2: Paste structured context, not noise

Good:

OS: Ubuntu 22.04, kernel 5.15
Role: production MySQL replica, 64GB RAM, 16 cores
Recent changes: kernel upgrade 6 hours ago
Symptom: server load average 40+, MySQL replication lag growing, queries timing out

$ uptime
 14:22:01 up 6:02,  4 users,  load average: 41.23, 38.51, 35.04

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           62Gi        58Gi       1.2Gi       128Mi       3.1Gi       1.8Gi

$ iostat -xz 2 3
[...]

Bad:

my server is slow can you help

Step 3: Let it ask follow-up questions

The good prompts in our library tell Claude to ask for missing data before guessing. When it asks "can you share dmesg | tail -50 and vmstat 1 5?" — that's a feature, not a flaw. Give it the data.

Step 4: Validate suggested commands before running

Claude will sometimes suggest a command with subtly wrong syntax, a destructive flag, or a path that doesn't exist on your distro. Read every suggestion before running. Never paste straight into a root shell.

Step 5: Keep the conversation alive

Claude's long context means you can run a 30-minute diagnostic session in one thread, paste new output as you gather it, and the model retains the full diagnostic context. This is the single biggest workflow win versus older AI tools.

What Claude is good at

Reading command output you don't fully understand (strace, perf, tcpdump summaries).
Drafting awk/sed/grep one-liners for log analysis.
Explaining why a specific kernel parameter or sysctl is set.
Suggesting what to look at next when you're stuck.
Drafting the incident summary after you've fixed it.

What Claude is not good at

Real-time anything — it can't see your live metrics.
Distinguishing between two plausible root causes when both fit the symptoms (it'll guess).
Telling you what's normal for your environment. You have to provide that baseline.

A real-world example

A production server's load average suddenly spiked. Pasting top, iostat -xz 2 3, and dmesg | tail -50 into Claude with our prompt template, it immediately flagged: "%iowait is 78%, await on /dev/sda is 320ms, and dmesg shows 'task X blocked for more than 120 seconds.' The disk subsystem is saturated, not CPU. Investigate which process is doing heavy I/O: iotop -oP -d1 will show the writer in 1-second intervals."

That's exactly the diagnosis we wanted, framed with the evidence — in seconds.

Companion resources

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

How to Choose the Right DevOps as a Service Provider

James Joyner — Sat, 13 Jun 2026 19:26:40 +0000

I've spent 25 years building, breaking, and scaling production infrastructure — long enough to watch "DevOps" go from a conference buzzword to a thing companies now rent by the month. That shift is real, and for a lot of teams it's the right call. But the gap between a great DevOps as a Service provider and a bad one is enormous, and the marketing pages all read the same.

So this is the article I wish more buyers had: what DevOps as a Service actually means, when it beats hiring, and how to tell — before you sign — whether the people you're talking to have ever been on-call at 3am.

What DevOps as a Service actually means

DevOps as a Service (DaaS) is outsourcing the engineering function that builds and runs your delivery pipeline and infrastructure, rather than hiring that function in-house. A provider takes ownership of some or all of: your CI/CD, your cloud environments, your observability, your automation, and the on-call response when something breaks.

It is not a single tool, and it is not a one-time project. A consultancy that drops a Terraform repo and disappears is not DaaS. The "as a Service" part means there's an ongoing operational relationship — someone is responsible for your systems on Tuesday at 2am, not just during the engagement.

Done well, you get the output of a seasoned platform team — Linux fundamentals, Kubernetes, Docker, infrastructure-as-code, pipelines, monitoring — without carrying that whole team on payroll.

Why companies outsource DevOps instead of hiring

Hiring a full in-house DevOps team is the "right" answer that's often the wrong answer in practice. Here's why teams rent it instead.

Cost. A single senior DevOps/SRE hire in a competitive market is expensive — and you need more than one for real on-call coverage. Add recruiting time, ramp-up, benefits, and the risk of a bad hire, and the fully-loaded number gets large fast. A provider amortizes senior talent across clients, so you pay for the expertise without paying for the bench.

Speed to maturity. A good provider has already built the Terraform modules, the GitLab CI templates, the Prometheus alert libraries, the backup runbooks. You're buying an opinionated, battle-tested baseline instead of inventing it. That can compress a year of platform work into weeks.

On-call coverage. Sustainable 24/7 on-call needs roughly six to eight engineers in a healthy rotation. Most companies under a certain size simply cannot staff that without burning people out. Providers spread the rotation across a larger team, so nobody's carrying a pager every single night.

Hard-to-hire seniority. The engineers who can debug a gnarly Kubernetes networking issue, reason about etcd, and also write clean Terraform are rare and they know it. They're hard to attract and harder to retain at a non-tech company. DaaS is often the only realistic way for a mid-sized business to get that caliber of person near its infrastructure.

What's usually included

Scope varies, but a full-spectrum provider should be able to own all of these. When you evaluate one, map their offering against this list and find out exactly where the lines are.

CI/CD — pipeline design, build/test/deploy stages, and crucially, a real rollback path.
Cloud infrastructure — provisioning and managing your environments as code (Terraform or equivalent), with sane network and IAM design.
Monitoring and observability — Prometheus, Grafana, logs, and alert rules that page a human only when a human is actually needed.
Automation — configuration management with Ansible, scripted runbooks, and elimination of manual toil.
Security — secrets management, least-privilege access, patching, and image scanning baked into the pipeline.
Incident response — a defined process, on-call rotation, and blameless postmortems, not just "we'll look at it."
Backups and disaster recovery — and, more importantly, tested restores. A backup you've never restored is a rumor.
Cost optimization — right-sizing, autoscaling, spot/reserved strategy, and killing the zombie resources nobody owns.

Questions to ask before you hire a provider

This is the part that separates the real operators from the slide decks. Don't ask "do you do Kubernetes?" — everyone says yes. Ask for specifics and watch how fast and how concretely they answer.

"Show me your Terraform module structure and how you handle state." Real teams have an opinion about remote state, locking, workspace-vs-directory layout, and blast-radius isolation. Vague answers here mean they're winging your infrastructure.
"Walk me through a real GitLab CI pipeline you run, including the rollback path." A deploy story with no rollback story is half a pipeline. I want to hear how they revert a bad release in minutes, not hours.
"How do you wire Prometheus alert rules to avoid pager fatigue?" The right answer involves symptom-based alerting, for: durations, severity routing, and ruthless deletion of noisy alerts. If every blip pages everyone, nobody responds to the one that matters.
"What does your on-call rotation look like, and what's your real response time?" Get the rotation size, escalation policy, and the SLA in writing. "We're very responsive" is not an SLA.
"How do you manage secrets and access?" Listen for a vault, short-lived credentials, and least privilege — not secrets in environment files or a shared password manager.
"When did you last test a restore from backup, and how long did it take?" The hesitation tells you everything.
"How do you handle configuration drift?" Ansible, immutable images, drift detection — there should be a system, not heroics.
"What happens to our infrastructure if we leave you?" A confident provider hands you clean, documented IaC and walks away gracefully. Lock-in is a choice they make, and you should know it up front.
"Who specifically will be on our account, and what's their production background?" You're buying judgment. Find out whose judgment.

Red flags to avoid

A few patterns that, in my experience, reliably predict pain.

Buzzword density with no specifics. If they can't move from "we leverage cloud-native synergies" to "here's how we structure a Helm chart" in one question, walk.
No rollback story. Anyone can deploy. Operators can un-deploy under pressure.
ClickOps in the cloud console. If they're configuring your production environment by hand instead of in code, you have no reproducibility and no audit trail.
Everything is "automated by AI." AI helps. AI does not own your incident at 2am. A provider hiding thin staffing behind AI claims is a serious risk (more on this below).
Alert noise as a feature. Hundreds of alerts is not observability; it's a team that's trained itself to ignore the dashboard.
No postmortems, or blame-heavy ones. A team that doesn't write honest postmortems isn't learning, and you'll pay for the same outage twice.
They won't show you anything real. Sanitized examples are fine. "We can't show you any of our work" usually means there isn't much to show.
Deep lock-in by design. Proprietary wrappers around standard tools, undocumented infra, contracts that punish leaving — all signs they're protecting revenue, not your uptime.

Why real production experience beats buzzwords

Here's the thing the marketing won't tell you: tools are easy, judgment is hard. Anyone can terraform apply. The value is in the engineer who knows not to apply at 4:55pm on a Friday, who recognizes the failure mode three layers down, who's restored a database under pressure and remembers exactly how it went wrong last time.

That judgment only comes from having run real production systems and felt the consequences. When you evaluate a provider, you're not really buying their Kubernetes skills — those are table stakes. You're buying scar tissue. You want the team that's debugged the keepalived VIP flap, the etcd disk-pressure cascade, the Docker layer that quietly doubled image size and blew out the build cache. Ask for war stories. The good ones light up; the pretenders get vague.

How AI fits — and where it doesn't

I'm bullish on AI in DevOps, and I build with it daily. Used right, it's a genuine force multiplier: it can summarize a wall of logs faster than any human, draft Terraform and Ansible boilerplate, propose PromQL, correlate a timeline of "what changed," and write the first pass of a postmortem. That's real leverage, and a modern provider should be using it.

But there's a hard line, and it's the same one I draw on my own systems: AI reads and reasons; humans run commands. During an active incident, AI proposes a risk-classified, safest-first plan and a human executes every step. The model never touches production. If a provider tells you their AI auto-remediates your prod environment unattended, that's not maturity — that's an outage waiting for a confident-but-wrong suggestion.

The right framing is AI as a very fast, very well-read junior engineer sitting next to a senior who owns the keyboard. It compresses the slow parts of the work without replacing the judgment that keeps you up. If you want to see what that looks like in practice, our AI incident-response workflows and prompt library are built around exactly that human-in-the-loop principle.

So when you evaluate a provider's AI claims, ask the same question you'd ask about any tool: where's the human, and what's the blast radius if the AI is wrong?

How a good provider actually pays for itself

The reason this model works isn't just cheaper labor — it's better outcomes in three places that show up directly on your books.

It saves money. Cost optimization is continuous work most teams never get to: right-sizing nodes, tuning autoscaling, buying reserved capacity, deleting orphaned volumes and idle environments. A provider doing this routinely often saves more on cloud spend than they cost. The infrastructure-as-code discipline also prevents the expensive mistakes — the hand-clicked resource nobody can reproduce, the security group left wide open.

It reduces downtime. Better alerting means you catch degradation before customers do. Tested restores mean a disaster is an inconvenience, not a company-ending event. A defined incident process with real on-call coverage means the response starts in minutes. Downtime is one of the most expensive things a business buys without meaning to, and maturity here directly buys it back.

It speeds up deployments. A solid GitLab CI pipeline with automated testing and a clean rollback path turns deploys from a scary quarterly event into a boring daily one. Teams that deploy confidently ship faster, and shipping faster is usually the whole point. The fastest way to slow down engineering is to make every release terrifying; good DevOps makes it dull.

Where to go from here

Be honest with yourself about where your infrastructure actually stands. Can you deploy and roll back in minutes, or does a release ruin someone's afternoon? Do your alerts mean something, or has your team learned to ignore them? If your primary database died right now, do you know — not hope, know — that you can restore it? Is there a real on-call rotation, or one exhausted person who's secretly the single point of failure?

If those questions made you wince, you're not behind — you're normal. Most teams are running far less maturity than they think, and trying to close that gap by hiring slowly, one expensive senior at a time, while production keeps moving. DevOps as a Service exists precisely so you don't have to win that hiring war before you can move fast.

Take an honest inventory this week. Score yourself on pipelines, observability, incident response, and recovery. Wherever you find a gap that's quietly costing you money, downtime, or velocity, that's where a good provider earns their fee many times over. The teams that move fastest aren't the ones with the most engineers — they're the ones who got serious about maturity before the outage forced the conversation. Decide which kind you want to be, and move while it's still your choice.

Evaluate any provider against your own systems and constraints. The right answer depends on your scale, your risk tolerance, and how much production maturity you already have in-house.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

ChatGPT vs Claude for Infrastructure Engineers

James Joyner — Thu, 11 Jun 2026 15:21:51 +0000

Both ChatGPT and Claude are excellent. But they have different strengths, and infrastructure engineers feel those differences more than most users — because we deal with long logs, multi-file configurations, and operations where being almost right can mean being very wrong.

Here's a side-by-side from a year of daily use on real infrastructure work.

Long-context reasoning over logs and manifests

Winner: Claude.

Claude's long context window means you can paste a 2,000-line kubectl describe pod, the full Deployment manifest, and your last 50 events without losing fidelity. ChatGPT can handle long contexts too, but in practice it's more likely to summarize or "forget" earlier details mid-conversation.

For diagnostic workflows where you keep pasting more output as you gather it, Claude's behavior is meaningfully better.

Safety with destructive commands

Winner: Claude (slightly).

Without explicit prompting, Claude is more likely to flag destructive commands (rm -rf, DROP TABLE, nova reset-state, kubectl delete) with caveats. ChatGPT will too — but is more likely to just hand you the command without extra emphasis.

If you use either tool in production troubleshooting, bake the safety constraints into your prompt (our prompt library does this). Don't rely on default behavior.

Code generation: Ansible, Terraform, Bash, Python

Roughly tied. Different defaults.

ChatGPT tends toward more "modern" Terraform (newer providers, recent syntax) and is slightly faster to produce a working playbook from scratch.
Claude tends toward more cautious, conventional output with better comments and more attention to idempotency.

For infrastructure-as-code review, Claude usually catches more subtle issues. For first-draft generation, ChatGPT is often a hair faster.

PromQL and observability queries

Roughly tied.

Both can write correct PromQL with rate(), histogram_quantile(), and label aggregation. Both occasionally hallucinate metric names if you don't paste your /metrics output. The deciding factor is your prompt quality, not the model.

Postmortem drafting

Winner: Claude.

Claude's prose is consistently more readable, less marketing-flavored, and more naturally blameless. ChatGPT tends to slip into corporate phrasing that engineers find grating ("leveraged our learnings to enhance reliability").

Ecosystem and integrations

Winner: ChatGPT.

Far larger ecosystem of plugins, GPTs, and shared prompts. If you want a tool that integrates with everything else you use, ChatGPT wins.

Pricing

Both are roughly comparable for individual use. Both offer free tiers with rate limits. Teams pricing varies by org needs.

Which should you use?

The honest answer: both, for different tasks.

Claude for diagnostic sessions, postmortems, sensitive prod work, and IaC review.
ChatGPT for fast scaffolding, plugin-heavy workflows, and broad community templates.

If you can only pick one and you do mostly production troubleshooting, pick Claude. If you can only pick one and you do mostly greenfield IaC scaffolding, ChatGPT is fine — your prompt quality matters more than the model.

Companion resources

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

How DevOps Engineers Can Use AI to Triage Production Incidents Faster

James Joyner — Mon, 08 Jun 2026 19:49:29 +0000

The pager goes off at 02:14. Checkout latency is up, error rate is climbing, and you have three dashboards, a wall of logs, and a half-awake brain. The fix, once you know what's wrong, is usually fast. The expensive part is the triage — the first fifteen minutes of "what is actually broken, and what changed?"

That triage window is exactly where AI helps most, and exactly where it's most dangerous if you let it run commands. This is how to use it to go faster without handing it the keys to production.

The rule that makes AI safe during an incident

AI reads and reasons. Humans run commands.

During an active incident you are sleep-deprived and time-pressured — the worst possible state to paste a command you don't fully understand. So draw a hard line: AI is allowed to look at evidence and propose a plan. It is never allowed to execute anything. Every command it suggests goes through your eyes and your hands.

In practice that means you treat the model like a very fast, very well-read junior SRE sitting next to you: it can summarize, correlate, hypothesize, and draft commands — but you're the one with the keyboard, and you read each command before it runs.

If you only take one thing from this article, take that.

Step 1: Turn the firehose into a summary

The first thing AI is genuinely great at is reading more text than you can at 2am. Paste in the raw material and ask for structure, not answers:

The firing alerts (name, severity, labels, duration)
A representative slice of error logs
Recent deploy / change history
The relevant dashboard values (p99 latency, error rate, saturation)

Then prompt it deliberately:

"Here are the alerts, logs, and recent changes for an active production incident. Summarize what's happening in 5 bullets, list the top 3 hypotheses ordered by likelihood, and for each hypothesis give me the single read-only command that would confirm or rule it out. Do not suggest any command that changes state."

That last sentence matters. Left unconstrained, models love to suggest kubectl rollout restart as step one. You want the diagnostics first.

Step 2: Make it order commands by blast radius

A good incident AI prompt forces a risk classification on every suggested command. Ask it to label each one:

safe — pure read-only: kubectl get, journalctl, ss, ip, cat, grep, promtool query
caution — shells in or makes a small change: kubectl exec, docker exec, editing non-prod config
destructive — restarts, deletes, scale-to-zero, firewall changes, migrations, restores

Then it must order them safest-first. You work top-down and you stop the moment you have a diagnosis. The number of incidents that get worse because someone reached for a destructive "fix" before confirming the cause is depressingly high — a forced safest-first ordering is a cheap guardrail against that.

Tip: keep your standard incident prompt in a snippet manager or a prompt library so you're not authoring it at 2am. We keep a set of AI incident-response prompts for exactly this.

Step 3: Correlate "what changed" automatically

Most incidents are caused by a change. The model is good at lining up a timeline if you give it the raw inputs: the alert start time, the last few deploys, config changes, and infra events. Ask:

"The latency spike started at 02:09 UTC. Here is the deploy log and the config-change history for the last 6 hours. What changed closest to 02:09, and what's the mechanism by which it could cause this symptom?"

This is where AI routinely beats a tired human: it doesn't get tunnel vision on the service you think is the problem. It will notice the keepalived VIP change, the connection-pool tweak, or the cert that rotated — the boring change three layers down that you'd have found 20 minutes later.

Step 4: Draft comms while you investigate

Incident comms are a tax you pay in attention you don't have. Hand them to the model:

"Write a status-page update for a degraded-checkout incident, customer-facing, no internal jargon, no root cause speculation, ~3 sentences. Then write a one-line internal update for the incident channel with current severity and what we're checking."

You get a customer update and an internal update in seconds, both in the right register. You skim, adjust a word, post. The investigation never stops to write prose.

Step 5: Let it draft the postmortem from the timeline

When the incident is resolved, the timeline is freshest and you're most likely to actually write it down. Paste the incident-channel scrollback and the command history and ask for a blameless postmortem draft: summary, timeline, root cause, impact, what went well, what to improve, and action items. You're editing a draft instead of facing a blank page — which is the difference between a postmortem that gets written and one that doesn't.

What NOT to do

A few failure modes worth naming:

Don't paste secrets. Scrub tokens, passwords, internal hostnames, and customer data before anything goes into a model. Treat the prompt like a screenshot you might accidentally post in a public channel.
Don't let it invent metrics. If you ask for PromQL and you haven't given it your real metric names, it will confidently make them up. Give it your metric names or tell it to use clearly-marked placeholders.
Don't trust a confident command. "Confident" and "correct" are unrelated in language models. The safest-first ordering exists precisely so a wrong-but-confident suggestion is read-only.
Don't skip the human review for "obvious" fixes. The obvious fix at 2am is how the incident gets a second act.

Where this fits in your workflow

You don't need a platform to start — a saved prompt and a scratch buffer get you most of the value tonight. The structure is what matters: summarize the firehose, hypothesize with read-only confirmations, correlate the timeline, draft the comms, and let the human run every command.

If you want the structured version of this flow — paste your symptoms and logs, get a risk-classified, safest-first plan plus a postmortem draft — that's exactly what we built the AI Incident Response Assistant for. But the technique stands on its own. Steal the prompts, keep the human on the keyboard, and reclaim the first fifteen minutes.

Generated incident plans and commands are assistive, not authoritative. Always verify recommendations against your own systems before running anything in production.

This article was originally published on DevOps AI ToolKit — practical AI workflows for cloud engineers.

DEV Community: DevOps AI ToolKit

DevOps as a Service Pricing: What Should Businesses Expect to Pay?

Why DevOps pricing varies so much

The common pricing models

Hourly / time-and-materials

Monthly retainer

Project-based / fixed bid

Emergency / incident support

Fully managed service

What services actually move the price

Why cheaper is not always better

What downtime actually costs

How AI changes the math

Starting lean: startups and small businesses

How to calculate ROI

The bottom line

Best DevSecOps Security Tools for CI/CD Pipeline Protection

What DevSecOps actually means

Why the CI/CD pipeline is a prime target

SAST — Static Application Security Testing

DAST — Dynamic Application Security Testing

SCA / dependency scanning

Container image scanning

Infrastructure as Code scanning

Secrets detection

Policy-as-code

Runtime security monitoring

How to choose: team size and infrastructure maturity

Where AI fits — assistive, not authoritative

Conclusion

DevOps Security Best Practices Every Engineering Team Should Follow

Why DevOps security belongs in the daily workflow

Secure access control and least privilege

SSH key management and MFA

Secrets management: API keys, passwords, and tokens

CI/CD pipeline security

Container image scanning

Infrastructure as Code security

Patch management and vulnerability remediation

Monitoring, logging, and alerting for security events

Backup and disaster recovery planning

Incident response preparation

AI-assisted security checks — review, never blind trust

The bottom line

Why AI Loves Ansible (And You Should Let It Help)

What makes Ansible AI-friendly

Modules have published contracts

Idempotency is the default

Roles and structure are predictable

Use cases where AI shines for Ansible

Generating new roles from scratch

Converting shell scripts to playbooks

Refactoring playbooks to use FQCN

Writing Molecule tests

Jinja template generation

Where AI struggles with Ansible

Variable precedence

Custom facts and set_fact lifetime

Vault integration

Distro-specific paths

A workflow that's been working for me

A note on safety

Why I think Ansible is the right entry point

AI for GitLab CI Authoring: Save Hours, Avoid Footguns

What AI gets right consistently

Standard job shapes

Translating from other CIs

Reviewing pipelines for inefficiency

What AI gets wrong — and how to catch it

rules: vs only/except confusion

$CI_COMMIT_BRANCH empty on MR pipelines

needs: referencing hidden jobs

Auto-apply rules that don't include the right branches

Imaginary GitLab features

A workflow that catches the failures cheaply

A practical example

The bottom line

Securing AI-Generated Bash Scripts Before You Run Them

The five things to check before running any AI-generated bash

1. Does it start with a strict pragma?

Custom facts and `set_fact` lifetime

`rules:` vs `only/except` confusion

`$CI_COMMIT_BRANCH` empty on MR pipelines

`needs:` referencing hidden jobs