It's 3 a.m. Your database connection pool exhausts. The site goes down. PagerDuty wakes up whoever's on call.
They jump on a Slack huddle, grep through CloudWatch logs, and find a Lambda function stuck in an infinite retry loop. Someone uses the AWS CLI to throttle its concurrency to 0. The database recovers. They scale the Lambda back up and go back to sleep.
The incident is over. The real pain shows up 48 hours later, when engineering leadership asks for the RCA and an updated runbook, and nobody remembers the exact sequence of commands they ran at 3 a.m. The post-mortem comes out vague, the runbook is half-finished, and six months later a different on-call engineer has to solve the same outage from scratch.
Every engineering org has some version of this problem. The fix doesn't have to involve a tired human trying to reconstruct their own memory two days later. Your infrastructure already watched what happened. It just needs to write it down.
Here's how to build that using AWS Systems Manager, CloudTrail, Amazon Bedrock, and Amazon Q Business.
The pipeline
The idea is to stop relying on human memory and instead mine AWS's own audit trail for what broke and how it got fixed.
SSM Incident Manager tracks the incident from the moment the alarm fires, giving you a clean start and end timestamp: the time window everything else is built around.
Meanwhile, AWS is already recording everything that happens inside that window. CloudWatch captures the error traces and latency spikes that triggered the alarm in the first place. CloudTrail logs every API call the on-call engineers make while they're fixing it. When someone throttles a Lambda, CloudTrail records a PutFunctionConcurrency event with their IAM user attached to it.
Once the incident is marked resolved in SSM, it fires an event to EventBridge, which kicks off the RCA pipeline automatically. A Lambda function pulls the CloudWatch error logs and the filtered CloudTrail events and passes them to Claude 3.5 Sonnet on Bedrock, with a prompt roughly like:
"You are a Principal SRE. Analyze these CloudWatch error logs to find the root cause. Then look at the sequence of CloudTrail API calls the engineers ran to stop the outage. Write a blameless RCA explaining what broke, and a step-by-step runbook for fixing it if it happens again."
A second Lambda takes that output, formats it as Markdown, and opens a pull request against your infra repo, something like docs: Add Runbook for DB Connection Pool Exhaustion. Once it's merged, Amazon Q Business indexes the new runbook into your RAG setup. Next time the same alarm fires, Q can drop a message straight into the incident channel: "I see the DB pool is exhausted. Last time this happened, Sarah fixed it by throttling the checkout Lambda. Here's the runbook. Want me to execute it?"
Why this is cheap
A decent RCA and runbook takes an engineer maybe two hours to research, write, and format. At a blended rate of $75/hour, that's $150 per incident, plus whatever context-switching costs you don't see on the invoice.
Running the equivalent through Bedrock: about 20,000 tokens of CloudTrail and CloudWatch JSON into Claude 3.5 Sonnet at $3/million input tokens costs roughly $0.06. A 2,000-token Markdown RCA back out at $15/million output tokens costs about $0.03. Call it $0.09 total.
The cost savings are the easy part to sell. The bigger point is that this captures institutional knowledge that normally evaporates the moment the incident ends: the "how Sarah actually fixed it" that usually only lives in one person's head.
Where this gets messy
A few things to get right before this works in practice:
CloudTrail is noisy. A busy AWS account throws off thousands of events a minute, and you can't hand all of that to an LLM. The Lambda that gathers data needs to filter for readOnly: false events and scope them to the IAM users who were actually paged, otherwise you're burying the signal in normal background API traffic.
CloudTrail only sees the control plane. It logs AWS API calls (scaling a DynamoDB table, restarting an ECS task) but not the data plane, meaning it won't see what an engineer typed into a SQL shell to fix a corrupted row. If your team uses SSM Session Manager to get into instances, you can configure it to stream terminal input/output to CloudWatch, which at least gets you the bash and SQL commands they actually ran.
The AI's output is a draft, not a finished document. It can misread why an engineer took a particular action, so the pipeline should open a pull request rather than commit straight to main. The human still reviews it the next morning, fixes a few sentences, and merges. The model does most of the drafting; a person is still the one who signs off.
The point
We tend to think of generative AI as a coding assistant or a support-ticket bot, and mostly ignore what it can do for operations. Wiring Bedrock up to AWS's own audit logs turns your infrastructure into something that watches what your engineers do and writes it down, instead of leaving that entirely to whoever happens to still be awake at 3 a.m.
How is your team handling RCAs and runbooks today? Curious if anyone's tried something like this. Drop it in the comments.
Top comments (1)
Great Knowledge