DEV Community

Cover image for Building an Agentic On-Call Copilot on AWS, in Public (Part 1: Why and Scope)
joshyfruit
joshyfruit

Posted on

Building an Agentic On-Call Copilot on AWS, in Public (Part 1: Why and Scope)

#aws #ai #automation #aiops

I build agents for fun and break them to find out where they crack. This is Project 1 of 3 I'm doing this year on Amazon Bedrock, AWS Strands, Bedrock AgentCore, and Kiro, and I'm posting the whole build, gotchas included.

The short version

Most "AI for DevOps" demos summarize a log and stop there. I want the opposite. Incident Copilot reads a real production alert, pulls the runbooks and past incidents that relate to it, reasons about a likely root cause, and recommends the next thing I should check. It asks me before it runs anything.

What I'm building

Incident Copilot takes one production alert and runs an agent loop:

  1. Classify and summarize the alert.
  2. Retrieve runbooks and historical postmortems with RAG.
  3. Propose a root-cause hypothesis.
  4. Suggest one or two read-only diagnostic actions, like checking recent deploys or querying the error-rate metric.
  5. Wait for me to approve.
  6. Run the approved tools and return a summary plus a recommended next step.

Version one does no auto-remediation. I chose that on purpose.

The decision that makes or breaks it: cut the scope

I have two to four hours a week for this. The thing that kills side projects isn't a model
that reasons poorly. It's a scope so big that I quit in week three. So the MVP stays small:

  • One alert type, high API latency. Not "any incident."
  • Read-only tools. The agent looks. It does not touch.
  • I approve before any tool runs.

The fun part is the safety rails, not despite them. An on-call agent that can run kubectl delete without asking would be the kind of thing that ends up in a postmortem of its own. Scoping the agent and gating its actions is the puzzle I actually wanted to solve here.

The architecture

Incident Response Workflow

Alert -> Strands agent (Bedrock / Claude reasoning)
   1. classify and summarize the alert
   2. retrieve runbooks and past incidents (RAG)
   3. propose a root-cause hypothesis
   4. suggest read-only diagnostic calls -> [HUMAN APPROVAL GATE]
   5. run the approved tools
   -> incident summary plus a recommended next step
Enter fullscreen mode Exit fullscreen mode

AWS Strands runs the loop and the tool calls. Amazon Bedrock with Claude does the reasoning at steps 1, 3, and 4. RAG grounds step 2 so the hypothesis cites a real runbook. Bedrock AgentCore is where I deploy this in August, which gives me the production story.

What I learned already

Writing the scope down before I wrote any code killed three features I would have sunk a week into. The build log earns its keep twice over: every decision and gotcha I jot down feeds the weekly posts and the final Devpost writeup. The content comes from the build rather than competing with it for my evenings.

Next

Week 2: the Strands agent skeleton and the first tool contract. I'll show the code.

Top comments (0)