The Problem
Our cloud operations team handles hundreds of Kubernetes alert triage cases every month. Each case follows the same pattern: navigate monitoring dashboards across regions, write PromQL queries from memory, cross-reference audit logs, correlate findings, and draft a customer communication — all in separate tools with no AI assistance.
For API server latency alerts alone, each investigation took 30–60 minutes of manual work. The investigation steps were well-defined in a runbook. The problem wasn't knowledge — it was automation.
The Approach: Spec First, Code Second
This is where Kiro's spec-driven development changed everything.
Instead of jumping into code, I started by defining the what — the full investigation workflow as a specification. Three layers, each building on the last.
What I Built
Layer 1: MCP Tool
Built a custom MCP tool that accepts natural language and translates it to PromQL queries against our metrics backend.
- Auto-discovers the correct monitoring workspace across 16 regions
- Translates natural language to PromQL — no more memorizing query syntax
- Supports instant queries, range queries, and all Prometheus query types
Engineers no longer need deep Prometheus expertise to investigate alerts.
Layer 2: Custom Kiro Agent
Configured a Kiro agent with:
- The MCP tool for metric queries
- A context file — patterns for all 20 alert types, visualization guidelines, and an incremental analysis workflow
The context file is key. It gives the agent domain knowledge about how to investigate each alert type, not just where to query. Think of it as encoding years of engineering expertise into a structured reference the agent loads automatically.
Layer 3: Agent SOP
An end-to-end autonomous investigation workflow. Give it a case ID, and it:
- Extracts alert context (cluster, region, alert time)
- Queries metrics to find the latency spike window
- Identifies the affected component
- Searches audit logs for the specific slow API calls
- Extracts caller identity and calculates duration
- Determines root cause (single slow caller vs. broad overload)
- Drafts a structured response following the exact team template
The SOP ties it all together — the MCP tool provides the data, the agent context provides the knowledge, and the SOP defines the workflow.
Why Spec-Driven Development Mattered
Complexity management — The investigation involves multi-source reasoning: time-series metrics + audit log events + communication templates. Defining each layer's contract upfront prevented scope creep and kept the pieces composable.
Reducing hallucination — Every statement in the generated output is grounded in actual tool results. The spec mandates copying exact values from tool output — not paraphrasing. This was a design decision made at the spec level, not an afterthought.
Testability — Each layer is testable independently. I could validate the MCP tool returns correct metrics without testing the full SOP. Each SOP step produces a concrete, verifiable artifact before the next begins.
Iteration speed — When the audit log search hit pagination limits on busy clusters, I could update just that section of the spec and re-implement without touching the metric query layer.
The Results
| Metric | Before | After |
|---|---|---|
| Time per investigation | 30–60 min | <5 min |
| Monthly hours saved | 35+ hours | Fully automated |
| PromQL expertise required | Yes — significant barrier | No — natural language |
| Idea to production | — | <2 months |
Three specs. Three layers. One automated pipeline.
Key Takeaways
Spec-first doesn't mean slow — Less than 2 months from concept to production. The spec prevented rework that would have cost more time.
Layer your automation — MCP tool → Agent with context → SOP. Each layer is independently useful, and together they're more than the sum of parts.
Well-defined runbooks are specs waiting to be automated — If your team has a manual process documented step-by-step, you're already halfway there.
Grounding eliminates hallucination by design — When the spec says "use the exact value from tool output," the agent can't fabricate data. Design this constraint in, don't bolt it on.
Context files are underrated — A good context file turns a general-purpose agent into a domain expert. 366 lines of patterns covering 20 alert types saved weeks of prompt engineering.
What's Next
Scaling this pattern to cover all alert types — moving from engineer-initiated investigation to fully proactive, automated triage. The same MCP tool + agent + SOP architecture applies to any alert with a well-defined runbook.
Top comments (0)