How an alert, ten browser tabs, and a Slack ping actually get resolved when AI is in the loop — and where I still don't trust it.
An alert fires.
I open Grafana. Then CloudWatch. Then the logs. Then kubectl describe on the pod that's misbehaving. Then GitHub, to see what merged. Then Argo CD, to see what actually rolled out.
Ten tabs in, trying to hold six timelines in my head at once, someone drops into the channel:
Do we know what happened yet?
That moment is the real job. Not the syntax. Not remembering the exact kubectl flag. The job is correlating scattered signals fast enough to form a hypothesis worth testing. That's the part where AI has changed how I work. It didn't take the troubleshooting away from me. I'm still doing all of it. It just shortened the gap between "something is wrong" and "this is probably where I should look."
I don't use AI as a replacement for understanding Kubernetes, AWS, Terraform, Linux, networking, databases, or CI/CD. I use it as another tool in the workflow, one that helps me get from a problem to a testable hypothesis faster.
My AI usage today broadly splits across three areas:
- ChatGPT — communication, research, reasoning, and technical analysis
- Claude and Claude Code — coding, Kubernetes, scripts, configurations, and troubleshooting
- AWS DevOps Agent — AWS infrastructure investigation, resource analysis, troubleshooting, and optimization
Each tool has a slightly different role. The part that actually matters isn't having access to AI. It's knowing where it's useful, what context to give it, and when its output needs to be challenged. None of them makes a production decision for me.
One habit before I get into the tools: I'm careful about what I paste into any of them. Config with real hostnames, account IDs, or anything secret-shaped stays out.
ChatGPT: the part of DevOps nobody warns you about
People underestimate how much of this job is communication.
I'll finish a technical investigation and then have to explain it — to a developer, a manager, a vendor, or the wider team — without either dumping raw metrics on someone who doesn't need them or oversimplifying to the point of being wrong.
A recent example: I was looking at EKS control-plane logging. I'd already done the actual work — figured out that api, audit, and authenticator logs were adding up to roughly 20–25 GB of ingestion per cluster per month, that ingestion was the real cost driver rather than storage, and that the sensible levers were trimming which log types we actually needed and dropping retention from 90 days to 30.
That investigation was mine. What I didn't want to spend twenty minutes on was rewriting the summary three times until it read cleanly for a Slack thread.
So I hand ChatGPT the findings and let it tighten the explanation. It isn't helping with the investigation. It's helping with the communication overhead after the investigation.
The second way I use it is more interesting: as a second pair of eyes on my own reasoning.
Say I've got a c7i.2xlarge sitting mostly idle and I'm thinking of dropping it to c7i.xlarge. The weak version of the question is "should I downsize this?" — because it just tells me what I already decided.
The version I actually get value from is:
What am I missing before I make this change?
And it'll nudge me toward network throughput, EBS bandwidth, short-lived peaks that a 30-day average hides, workload patterns, rollback path. Half of those I'd check anyway. But every so often it names the one I would've skipped, and that's the whole point.
I stopped asking AI to make the call and started asking it to find the hole in mine.
Claude and Claude Code: reducing the search space
I use Claude for the code-shaped work — Kubernetes manifests, Bash and Python scripts, CI/CD config. That part's useful but not that interesting; generating a Deployment YAML was never the hard part.
Claude Code is where it gets useful, because a Kubernetes problem almost never lives in one file.
The behavior I'm chasing is usually smeared across Helm templates, values.yaml, values-prod.yaml, a ConfigMap, a Secret, a GitHub Actions workflow, and whatever Argo CD is actually syncing. Previously I'd grep my way through all of it by hand, holding the thread in my head the whole time.
Now I can ask, against the real repo:
Find where DATABASE_HOST is defined and explain how it reaches the production Deployment.
Or:
Compare dev and prod and show me anything that affects memory limits or scheduling.
Here's a concrete one. A pod keeps restarting. I run the usual:
kubectl describe pod <pod>
kubectl logs <pod>
kubectl logs <pod> --previous
kubectl get events
And I find:
Last State: OOMKilled
Exit Code: 137
I already know the direction — this is memory. What used to eat my time wasn't forming that hypothesis; it was the file-hopping to test it. Where's the current limit set? Is a prod override quietly bumping it down? Did a recent change touch the Helm values? Claude Code walks that trail with me instead of me opening eight files to reconstruct it.
It's also confidently wrong sometimes. Once it pointed me at a limit in values.yaml and told me that was the effective value in prod — except Argo CD was syncing that service from a different repo, so the file it was reading never applied. The answer looked right. It just didn't know about the source it couldn't see. That's exactly why the output is a lead, not a verdict: it reduces the search space, and then I confirm what actually applies.
AWS DevOps Agent: correlating telemetry I'd otherwise eyeball
This is the most infrastructure-facing part of how I work.
Worth being precise here, because it's a capable product and it's easy to undersell. AWS DevOps Agent can go a long way on its own: it investigates incidents autonomously, correlates telemetry with code and deployment data, and produces root-cause analysis and mitigation plans without me driving each step. I read those as a strong lead, not a verdict. The part I lean on most is the correlation, because that's where it saves me the tab-juggling. The root-cause call and anything that touches production, I still confirm and own myself.
AWS already collects an enormous amount of operational data. Finding the graphs was never the problem. The problem is figuring out what changed at the same time as the thing that's on fire.
RDS is the clearest example. A CPU spike shows up, and normally I'd be pulling up CPUUtilization, DatabaseConnections, FreeableMemory, read/write IOPS, latency, DiskQueueDepth, Performance Insights, and the RDS events feed — one tab at a time.
Say the picture looks like this:
10:00 CPU: 20% Connections: 100
10:05 CPU: 85% Connections: 450
10:10 CPU: 95% Connections: 700
The useful observation isn't "CPU hit 95%." Anyone can read that off a graph. The useful observation is that CPU and connections climbed together, and that turns a symptom into a hypothesis. Maybe replicas scaled up. Maybe connection pooling fell over. Maybe a scheduled job kicked in, or queries got slower, or a retry loop started hammering the database.
Correlation isn't proof. I still have to confirm which of those it actually was — and this is where Performance Insights earns its place: top SQL by load told me the time was going into a single unindexed query, which ruled out the retry-loop and pooling theories fast. But having the correlation surfaced for me is the difference between starting an investigation and starting it fifteen minutes late.
The same correlation habit feeds straight into cost work. Picture a c7i.2xlarge running at ~5% average CPU, ~15% peak, memory under 15%. The lazy move is to downsize on CPU alone. Instead I want CPU, memory, network, EBS/disk, and the peak-workload periods considered together, because low CPU with high memory is a completely different story.
One catch that's easy to miss: halving the instance size also halves the network and EBS bandwidth ceilings, so that 15% peak has to be re-checked against the smaller instance's limits, not the current one's. When the signals genuinely agree that c7i.2xlarge → c7i.xlarge is safe, and I've validated it, AI has directly helped me cut spend while keeping real headroom. That's more honest than saying "AI helps with cost optimization." It helped me confidently run the workload smaller.
The thing all three have in common
Strip away the tools and here's what actually changed.
Before:
Alert → metrics → logs → docs → repo search → config comparison → hypothesis
Now:
Alert → collect evidence → AI helps correlate/search → hypothesis → validate → fix
The steps didn't disappear. The evidence still has to be collected, the hypothesis still has to be validated, the fix is still mine to ship. What compressed is the messy middle: the correlating, the searching, the file-hopping.
Where the human stays in the loop
I don't hand production to any of this.
If a right-sizing analysis says an instance can shrink, I don't just run terraform apply. If Claude generates a manifest, I read every line of it. If the AWS agent points at a root cause, I still go confirm the metrics say what it thinks they say.
My loop is boring on purpose:
AI suggestion → understand it → validate it → make the change → monitor → roll back if needed
The trap is easy to fall into: create Kubernetes Deployment, copy the YAML, ship it. But if you can't explain your own resources.requests, your probes, or your affinity rules, AI just made you faster at deploying things you don't understand. That's more dangerous, not less.
AI makes syntax, searching, and retrieval cheap. If the syntax is cheap now, the judgment about whether to run it is the part that's actually worth paying an engineer for.
What moved is how long I spend getting from "something is wrong" to "this is probably where I should look." That gap used to be most of the incident. Now it's a smaller part of it, and I'd rather spend my time on the part that actually needs an engineer.
One question for the DevOps, SRE, cloud, and platform folks: where has AI actually created more work for you than it saved — a wrong root cause you chased, a confident answer that was subtly off, a place you still don't trust it? (And if you've got a genuine win, I'll take that too.)
Top comments (0)