DEV Community: Ilia Dubovskii

AI Root Cause Analysis for Incidents: What It Can and Can't Do

Ilia Dubovskii — Fri, 23 Oct 2026 22:40:02 +0000

An AI agent that investigates an alert can save the on-call engineer the worst part of the first half hour: opening ten dashboards, scrolling the deploy log and asking in the channel who changed what. It can't be trusted to decide what's true, or to change production on a hunch. This guide covers where the line falls in practice: what an AI investigation should collect in the first minutes, how it should rank causes, why it should propose rather than act, how to confirm a fix actually worked, how to get post-mortems out of it that people trust, and how to measure the result without fooling yourself.

The short version

AI is good at the investigation: gathering evidence from many sources at once and correlating it with recent changes. It is weak at judgement, novel failures and incidents with more than one cause.
Every hypothesis should come with a confidence, the evidence behind it, and a list of what was checked and ruled out.
Give the agent read-only credentials. It proposes; a human approves. Only actions that are verifiable, reversible and contained should ever run on their own, one at a time.
An incident is over when the metric that triggered it is back to normal, not when someone ran a command. The agent should watch that metric and say if the fix didn't work.
Build post-mortem drafts from timestamped evidence, mark what's inferred, keep them blameless, and have a human merge them.
Mean time to restore (MTTR) is a noisy number. Look at distributions and per-incident detail, not a single average.

1. What AI root cause analysis can and can't do

Incident response has two kinds of work in it. The first is investigation: find out what's affected, what changed, which signals moved and when. It's mechanical, it's parallel, and it's where the first twenty minutes usually go. The second is judgement: decide what's actually going on, whether to roll back, who to wake up, what to tell customers. An AI agent with access to your telemetry can do most of the first and should do very little of the second.

What it does well:

Breadth and speed. It can query metrics, logs, traces, the deploy history and the feature flag audit log at once, where a person opens them one tab at a time.
Correlation with change. Most incidents follow a change: a deploy, a flag, a config edit, a migration, a scaling event. Lining those up against the moment a metric moved is exactly the kind of tedious cross-referencing a model does reliably.
Recognising known patterns. Connection pool exhaustion, a missing index, an expired certificate, a dependency that's down. If it has happened before, in your systems or in general, the agent will often name it quickly.
Writing it down. A clear summary in the channel, and later a first draft of the timeline, saves real time.

Where it struggles:

Novel failures. Without a recognisable pattern or an obvious recent change, a model tends to produce a plausible story rather than say "I don't know".
Anchoring. Models latch onto the first hypothesis that fits and stop looking, which is a real risk for incidents with more than one contributing cause.
Missing context. It doesn't know that the marketing campaign started at 9:00, or that the payment provider emailed a maintenance notice, unless that information is somewhere it can read.
Trade-offs. Whether to roll back a release that also contains a security fix is a business decision, not a pattern match.

This isn't only a cautious vendor's opinion. In August 2025, ClickHouse tested four frontier models on anomalies injected into the OpenTelemetry demo application, with access to the observability data through an MCP server. Their conclusion was blunt: "Autonomous RCA is not there yet." Models improve quickly, so treat any single benchmark as a snapshot. The design conclusion still holds: let the agent find and organise the evidence, and let a human decide.

2. What to collect in the first minutes

The quality of an AI investigation is set by what it can see. Before you worry about the model, make sure the agent can reach these, with read-only access, and that it collects them for every alert:

The alert and the real start time. The alert fires when a threshold is crossed, which is often several minutes after things started going wrong. The agent should find the onset in the metric itself and use that, minus a margin, as the start of every other search window.
What the service depends on, and what depends on it. Take this from APM traces and your service catalog, not a hand-drawn diagram. It tells the agent which databases, queues and downstream services to check, and who else is affected.
Changes before the onset. Deploys, feature flag changes (with who and what percentage), config and secret changes, infrastructure-as-code applies, database migrations, scheduled jobs, autoscaling events. This list finds the cause more often than any anomaly detector.
Dependency health. Database CPU, connections and slow queries; queue depth; cache hit rate; error rates and latency of the services called; third-party status pages.
New log signatures. Not "all errors", but error messages that are new or sharply more frequent than in the baseline window.
Traces of slow or failing requests. Where the time actually goes. This often separates the cause from the symptom.
Similar past incidents. Previous incidents for the same service or with the same signature, and how they were resolved.
Owners and on-call. Who owns each service involved, so the agent can suggest who to page.

The output of this step isn't an answer. It's a set of facts, each with a timestamp and a link back to its source, which is what makes the next two steps checkable.

3. Ranking causes: confidence, evidence and "ruled out"

An investigation that returns one confident sentence is dangerous, because a wrong answer looks exactly like a right one. A useful first evaluation has a structure that lets the on-call engineer check it in a minute:

Ranked hypotheses, each with a plain confidence level (high, medium, low) rather than a falsely precise percentage.
Evidence for each one, as specific facts with links: "flag rolled from 10% to 100% at 09:41", "database CPU 31% to 97% at 09:43", "93% of slow spans wait on this call". If a hypothesis has no evidence the engineer can click, it shouldn't be ranked first.
What was ruled out, and why. "No deploys in six hours. Payment provider latency normal. Network metrics at baseline." This is the most underrated part. It stops the team spending fifteen minutes on the reflexive "was it the deploy?", and it shows what the agent actually checked, so a gap is visible.
Cause versus symptom. An exhausted connection pool is usually a symptom of slow queries, not a cause. The agent should say which it thinks each item is.
What would change its mind. For example: "if CPU doesn't fall within five minutes of turning the flag off, this hypothesis is wrong".

Treat correlation with suspicion, including when the agent finds it. A change that happened shortly before the onset is a strong lead, not proof. The proof comes later, when you undo the change and the metric recovers.

4. Read-only by default: propose, don't act

The asymmetry decides this. If the agent's hypothesis is wrong, you lose a few minutes reading it. If the agent acts on a wrong hypothesis, it can turn one incident into two: a rollback that removes a fix, a restart that drops in-flight work, a scale-up that hides the real problem and runs up the bill. The vendors in this space mostly agree. Rootly says every change its AI SRE proposes "requires explicit human sign-off before execution". incident.io says the only change its Investigations product can make "is a pull request you review and merge yourself". PagerDuty describes its SRE Agent as performing "approved remediation".

In practice that means:

Read-only credentials, scoped to observability data, the deploy history and the flag audit log. Not a production admin role "for convenience".
Remediation as a proposal: a specific action, its expected effect, how to undo it, and a button. A separate, narrowly permissioned path carries out the approved action, and logs who approved it.
Code changes as pull requests, which go through the same review as any other change.

There's a narrow exception. Some actions are safe enough to run without waiting, if they pass three tests: the outcome is verifiable (you can tell from a metric whether it worked), the action is reversible (one step puts things back), and the blast radius is contained (it touches one component). Turning off a feature flag that was changed ten minutes ago can pass. Failing over a database doesn't. Even then, grant autonomy one action at a time, after the team has watched the agent propose that exact action correctly many times. Our autonomous DevOps framework covers this test and the gate that enforces it in more detail.

5. Confirming recovery on the metric that fired

"Flag turned off" isn't the end of an incident. The end is when the metric that started it is back to normal and stays there. This step is where many incident processes, human or automated, go soft: the action is taken, the channel goes quiet, and nobody checks whether the error rate really came down or just moved somewhere else.

An AI investigation is well placed to close this loop, because it already knows which metric fired and what normal looks like:

Define "restored" before acting. For example: the error rate below the alert threshold, and close to its baseline, for ten consecutive minutes.
Watch the cause signals as well as the symptom. If the hypothesis was a slow query, database CPU should fall first, then latency, then errors. If errors drop but CPU doesn't, something else is going on.
Set a deadline and say so. "If checkout isn't back within ten minutes, I'll report that and reopen the other hypotheses." A fix that didn't work is evidence too.
Record the times. Onset, alert, first evaluation, action, recovery. These become the post-mortem timeline and the honest version of your MTTR.

This is also what makes the agent improve over time. Each incident ends with a check of whether its top hypothesis was right, which is the number to watch when you decide how much to trust it.

6. Post-mortems you can trust

AI-drafted post-mortems are now a standard feature: incident.io, Rootly, FireHydrant and PagerDuty all offer some form of them. The draft is the easy part. Making it trustworthy takes a few rules:

Build the timeline from evidence, not from chat. Timestamps should come from the metric, the deploy log and the flag audit trail. Chat messages are useful for what people decided and why, but they record when someone noticed something, not when it happened.
Separate what was observed from what was inferred. "CPU reached 97% at 09:43" is a fact. "The flag caused the slow query" is a conclusion, confirmed or not by what happened when the flag was turned off. Write it that way.
Keep it blameless. Google's SRE book puts it plainly: a blameless postmortem must "focus on identifying the contributing causes of the incident without indicting any individual or team". An AI draft can quietly break this by writing "Maria rolled the flag to 100%" under root cause. The change belongs in the timeline. The cause is in the system: a query without an index, and no check of database impact before a flag reaches all traffic.
Action items are suggestions. Mark them as "for the team to confirm", with suggested owners. Only people should commit people to work.
Review it like code. Opening the draft as a pull request in the repository where your post-mortems live gives you review, history and an approver by default. incident.io's own docs call its AI draft "a starting point, not a finished document", which is the right framing whatever tool writes it.

The quiet benefit is coverage. Small incidents rarely get a write-up because nobody has the time. When the draft costs nothing, more of them get reviewed, and the patterns in small incidents are often what prevents the big ones.

7. Measuring MTTR honestly

Mean time to restore is the obvious metric for an incident agent, and it's the metric we build around, because it's what the business feels. It's also easy to misuse. Two credible sources are worth reading before you put it on a slide:

Google's report Incident Metrics in SRE (Štěpán Davidovič, 2021) uses simulation to show that statistics like MTTR are "poorly suited for decision making or trend analysis in the context of production incidents".
Courtney Nash's work on the VOID, an open database of public incident reports, argues "how unreliable MTTR can be" (SREcon22 Americas). Incident durations are heavily skewed, so a mean says little about a typical incident.

That doesn't mean you can't measure. It means measuring carefully:

Use the time to restore for each incident, and look at the distribution (median and a high percentile), not a monthly mean.
Define the start as the onset in the metric, not the alert, and the end as the confirmed recovery from section 5. Otherwise improving the alert threshold looks like improving response.
Measure the parts the agent controls: time from alert to first evaluation, and how often its top-ranked cause was right. Those are directly attributable. A change in overall MTTR over a few months isn't.
Compare like with like. A quarter with one long outage will swamp everything else. Read the individual incidents.
Don't let the number drive behaviour. If people start closing incidents early to keep MTTR down, the metric has stopped measuring anything.

8. Buy or build?

For most teams, buying is the right first move. These are the products we'd look at first, described from their own sites (checked October 2026):

Datadog Bits AI SRE (called Bits Investigation in Datadog's current docs) investigates alerts as they fire, forms hypotheses from Datadog telemetry and suggests code fixes. The natural choice if Datadog already holds most of your data.
incident.io has an AI SRE that investigates when an incident is declared, plus AI post-mortem drafts. Its only change to your systems is a pull request you merge.
Rootly AI SRE ranks evidence-backed hypotheses with confidence scores and suggests fixes, with human sign-off on every change.
PagerDuty offers an SRE Agent (triage, diagnosis, approved remediation) and a Scribe Agent that captures incident conversations for post-incident reviews.
FireHydrant offers AI incident summaries and AI-assisted retrospectives. Freshworks has announced it is acquiring FireHydrant.
Resolve.ai positions itself as an AI SRE that goes on call and mitigates incidents, closer to the autonomous end of the range.

A custom system makes sense in narrower cases: your evidence lives across tools that no single vendor reads well; your runbooks and approval rules are specific and need to be enforced exactly; telemetry can't leave your environment, so the agent must run in your own cloud; or you want the full loop described here (investigate, propose, approve, verify on the metric, draft the post-mortem) tuned to one metric you care about.

9. FAQ

How accurate is AI root cause analysis?

It depends heavily on your data and the kind of incident. For failures that follow a recent change or match a known pattern, an agent with good access to telemetry often names the likely cause quickly. For novel or multi-cause incidents it is much weaker: in ClickHouse's August 2025 test of four frontier models, the conclusion was that autonomous root cause analysis is not there yet. Measure it yourself by recording, for every incident, whether the agent's top-ranked cause turned out to be right.

Should an AI incident agent have write access to production?

No, not by default. Give it read-only credentials to observability data, deploy history and the feature flag audit log, and have it propose remediation that a human approves. The only exceptions worth considering are actions that are verifiable, reversible and contained, such as turning off a feature flag that was just changed, granted one action at a time after the team has seen the agent propose it correctly many times.

What data does an AI incident investigation need?

The alert and the metric behind it, service dependencies from APM traces or a service catalog, every change before the onset (deploys, feature flags, config, infrastructure applies, migrations), dependency health, new log signatures, traces of failing requests, similar past incidents, and service owners. Missing any of these creates blind spots, and the agent should say which sources it could not check.

Can AI write the post-mortem?

It can write a good first draft, and most incident management tools now offer this. Build the timeline from timestamped evidence rather than chat, separate what was observed from what was inferred, keep the draft blameless, and present action items as suggestions. A human should review and merge it, for example as a pull request.

Is MTTR a good metric for AI incident response?

It is the metric the business feels, but a mean is misleading because incident durations are heavily skewed. Google's report Incident Metrics in SRE and Courtney Nash's work on the VOID both show how unreliable MTTR can be for trend analysis. Track time to restore per incident, look at the median and a high percentile, and also measure what the agent directly controls: time to first evaluation and how often its top cause was right.

Should we buy an AI SRE tool or build our own?

Buy first if your telemetry mostly lives in one platform and a vendor's workflow fits your on-call process: Datadog, incident.io, Rootly, PagerDuty, FireHydrant and Resolve.ai all offer AI investigation or post-mortem features. Build, or have one built, when your evidence is spread across tools no single vendor reads well, when telemetry can't leave your environment, or when you need your own runbooks and approval rules enforced exactly.

Written by Ilia Dubovskii, founder of Beamreach. Vendor descriptions were checked against each vendor's own site on 10 October 2026. Products in this space change monthly, so if something here no longer matches, the vendor's site wins. Tell us and we'll fix this page.

Originally published at https://beamreach.ai/docs/ai-root-cause-analysis-incidents.html.

The verifiable, reversible, contained test from section 4 is the core of my talk Autonomous DevOps: Engineering Systems That Don't Need Our Attention at KCD Porto x DevOpsDays Portugal (19–20 November 2026). Talk page: https://beamreach.ai/talks/autonomous-devops.html

Autonomous DevOps: A Framework for Deciding What AI Agents May Do Alone

Ilia Dubovskii — Fri, 23 Oct 2026 22:39:40 +0000

Autonomous DevOps is an operating model in which agents continuously observe cloud and pipeline telemetry, decide what needs changing, and act inside declared boundaries without a human approving each instance, while the system stays observable, auditable and reversible. Humans set the boundary, not the instance.

The short version of this guide: autonomy is not a property of the agent. It is a property of the operation. An operation can run without human attention when it is verifiable, reversible and contained, and those three things can be scored from facts about your infrastructure rather than guessed.

What is autonomous DevOps?

For two decades DevOps has been about automation: CI/CD pipelines, Infrastructure as Code, observability platforms, deployment systems. All of it moved execution off the human. None of it moved attention off the human. Someone still reviews the dashboards, investigates the alerts, decides whether an infrastructure change is safe, maintains the honeypots and keeps the Terraform aligned with reality. That after-deploy work is what we call Continuous Maintenance: the stage CI/CD never automated.

The distinction that matters is between automation and autonomy:

	Automation	Autonomy
Who decides that it happens	A human	The system, inside declared boundaries
Who decides when	A human, or a schedule a human set	The system, from what it observes
Human attention needed	To trigger it and to check it	To set and revise the boundary
What a human approves	Each instance	A class of operations

So autonomous DevOps is not "an AI agent with production credentials". It is a way of engineering operational systems so that routine decisions no longer need our limited focus, while humans remain responsible for strategy, architecture and exceptions. Autonomy does not mean giving AI unrestricted control. It means identifying the operations that can be trusted to run alone, and building the machinery that enforces that decision.

Large language models matter here for one reason: they can assemble a fix for situations nobody wrote a rule for. That widens what an agent can do far beyond a scripted runbook. It also means the set of possible actions can no longer be listed in advance, so it can't be made safe by narrowing it to a few typed commands. It has to be made safe by deciding, case by case, whether each operation qualifies. That decision is the subject of this guide.

Attention is the scarce resource

Opinion. I think attention, not compute and not headcount, is now the binding constraint on running cloud-native systems. Here's the argument:

Automation scaled execution. Execution is now close to free.
Every automation we added created something new that needs watching: a pipeline, a dashboard, an alert rule, a Terraform state, a posture report.
So the number of decisions per engineer per week went up, not down. I call that attention debt: the accumulated volume of decisions a system needs from humans per unit of time. Every new alert rule and dashboard adds to it, and it compounds quietly.
Attention doesn't scale the way compute does. It's serial, it breaks under interruption, and it degrades with volume. Alert fatigue is what attention bankruptcy looks like.
So the next gain isn't "execute more". It's "decide less".

Two things from my own estate made this concrete for me. My IaC coverage was roughly half, which meant half the infrastructure was invisible to the tool that was supposed to manage it, and nobody was assigned to notice. And the posture scanners I run (Prowler, Trivy, Gitleaks, Checkov) produce findings faster than any person can triage them. The scan was never the bottleneck. Reading the output was.

AI coding tools make this worse before they make it better. They generate more infrastructure change, faster, and review capacity stays flat. In my view AI is both a cause of the review backlog and the only realistic response to it, which is exactly why the boundary has to be engineered rather than assumed.

Autonomy is a property of the operation: verifiable, reversible, contained

Most discussion of AI agents in DevOps asks whether the agent is good enough. That's the wrong unit. The same agent can be perfectly safe planting a decoy credential and dangerous resizing a production database. What changes isn't the agent. It's the operation.

An operation can run without human attention when it has three properties:

Verifiable

The system can prove, mechanically and without a human reading it, that the operation achieved its intent. Not "did it run" but "did it do the right thing". Example: after an import, terraform plan shows no changes.

Reversible

There is a known, cheap, bounded path back to the prior state. Example: reverting a PR that added Terraform import blocks changes state, not infrastructure. Deleting a database has no such path.

Contained

The maximum damage if the operation is wrong is known in advance and small. This is blast radius. Example: a decoy credential has no dependents; nothing in production reads it.

These aren't a slogan. Each one can be measured from facts you already have: how many resources an operation touches and of what types, whether they're tagged production, whether the change class is additive or destructive, what the live metrics say about load right now, whether compliance tags are attached. Once you score operations instead of agents, a different set of questions becomes answerable:

Operation	Verifiable	Reversible	Contained
Plant or remove a canary credential via Terraform PR	High	High	High
Import an unmanaged resource into Terraform	High (plan is a verifier)	Very high	High (no runtime effect)
Route a low-risk finding to an auto-queue	Medium	High	High
Downsize a production instance	Medium (savings are estimates)	Medium	Low to medium
Delete a resource with data on it	Medium	Low	Low

This also reframes the useful engineering work. It isn't "build a smarter agent". It's building the machinery that decides which operations qualify, and making that decision auditable.

The autonomy gate and the dial

In a real system the framework isn't a slide. It's a scoring function and a threshold. Here is how it works in Beamreach Compass, described in prose.

Score the operation. Every proposed operation gets a risk score built from five weighted components: blast radius, data loss, reversibility, operational load and compliance. Each component is computed from facts about the infrastructure (resource counts and types, environment tags, the change class, live CloudWatch metrics, compliance tags), not from the model's opinion of itself.
Score the urgency separately, because "risky" and "time-sensitive" are different questions.
Apply the gate. High-risk operations always require human approval, and so do moderately risky ones when they're also urgent. An operation may run unattended only if autopilot is enabled, it didn't trip the approval rule, and both its risk and urgency are at or below the ceilings the team configured.
Turn the dial. The team expresses how much autonomy it grants with one number: the maximum risk level allowed to run unattended. Everything else is derived. Unattended execution can be switched off entirely. When it's on, the default ceiling only admits the lowest-risk operations.

Raising the dial is an explicit configuration change made by a person, saved in the workspace config. It's not something the agent negotiates or adjusts for itself. Lowering it takes effect the moment trust is lost.

The useful mental model: the framework is a threshold you turn up as trust is earned, and turn down the moment it isn't. That's what separates delegated autonomy from an agent that's just running unattended.

Containing the model itself

The gate decides whether an operation may proceed unattended. A second, independent layer decides what the model can physically do, and that layer doesn't depend on the model behaving. In Compass the LLM runtime works under a restricted execution contract:

Boundary	Mechanism
Filesystem	Working directory pinned to a checked-out repository; no host access
Tools	Explicit allow-list of file and shell tools, plus optional scoped lookups
Cloud	Read-only lookups. There is no write path to live infrastructure.
Termination	Hard turn limit, bounded repair attempts, and a loop limit with a defined fallback. Loops end by construction.
Output	A git diff, delivered as a pull request. There is no other way out.
Intent	System prompt: work only inside the repository, don't perform live infrastructure mutations, stop and explain if unsure

The last row is the weakest guardrail and the first five are the real ones. A prompt is a preference. A missing IAM permission is a boundary. Never rely on the model to respect a rule you could have enforced structurally.

Security challenges of autonomous DevOps agents

This is where most of the security risk lives, and it's worth listing plainly:

Inherited credentials. An agent running on an engineer's session has the engineer's permissions. That's a loophole, not an operating model. Give agents their own narrow identity.
Prompt injection through data. Agents read findings, logs, tickets and repository content, and any of it can carry instructions. Containment matters more than filtering: an injected instruction that can only produce a diff in a PR is a review problem, not an incident.
Unbounded loops and spend. Agents don't get tired, so a stuck task can burn tokens indefinitely. Turn and loop limits are guardrails, not tuning knobs.
Silent failure. Some operations fail by omission: a suppressed alert that should have fired. These need the most careful scoring (see the noise example below).
Prompt-only rules. Any boundary that exists only in a system prompt should be treated as absent.

The design goal is to make the failure mode boring. If the model does something nobody anticipated, the worst case should be a bad diff in a PR that nobody merges. For the organisational side of this (trust, control, cost) see obstacles for autonomous software.

Trust Engineering, and how teams actually adopt autonomy

Trust Engineering is the discipline of making an autonomous system's boundaries explicit, measurable and inspectable, so a human can extend or withdraw autonomy as a deliberate configuration change rather than an act of faith. The scoring function, the gate, the dial and the containment layer are all Trust Engineering. So is being able to answer, after the fact, why a given operation was allowed to run.

Opinion. My bet is that most teams will adopt autonomy per operation class, not per team and not per tool. Nobody will "turn on the AI". They'll let canary placement run alone first, then Terraform imports, then low-risk finding triage, and keep rightsizing and deletions behind approval for a long time. The adoption curve is a risk threshold being raised one notch at a time, each notch justified by a track record on the notch below.

In practice that looks like this:

Observe. Let the agent find, score and propose. Every change goes through a PR a human merges. You're calibrating your trust in the scores, not the agent's prose.
Pick one operation class that scores high on all three properties and enable unattended runs for it alone, at the lowest ceiling.
Watch the outcomes, not the volume. Did anything need reverting? Did anything get wrongly suppressed?
Raise the ceiling one step, or don't. Lower it immediately when something surprises you.

Opinion. Two more hypotheses I hold and want challenged. First, autonomy's real value is that it makes expertise modular, distributable and actionable: a version of "good" for a given operation that you can subscribe to and inspect, instead of rebuilding it in every team. Second, the moat isn't the agent. Agents are becoming a commodity. The assembled context is not: a queryable map of the estate where every fact carries provenance, confidence and freshness. Without that map, none of the scores above could be computed.

Opinion. I also think self-hosting multiplies trust rather than being a feature: when the runtime is inside your boundary, "what is it doing with my data" stops being part of the conversation.

Four worked examples

Each example follows the same pattern: what the operation is, how it scores on the three properties, whether it earns autonomy, and what actually exists. Using the same shape every time is the point. It's what makes this a framework rather than a set of demos.

1. Honeypot and canary management

Operation: decide from the infrastructure map where a canary belongs, plant it, and later decommission it.

Scores: verifiable is high (the asset exists in state or it doesn't; an alert fires or it doesn't). Reversible is high (decommissioning is a PR that deletes the resource). Contained is high (a fake secret has no dependents).

Verdict: the best autonomy candidate in the system, and the one people find most surprising, because "security" and "unattended" sound contradictory. You shouldn't trust the agent with security assets. You can trust that this particular operation is contained.

What exists: map-driven placement ranking, Terraform generation, plant and decommission PRs, and alert ingestion authenticated through IAM, so there's no shared secret to leak. Scheduled rotation of canaries doesn't exist yet. More on cloud honeypots and canary credentials.

2. IaC coverage and drift

Operation: discover resources that exist in AWS but not in Terraform, generate import blocks and HCL, and open a PR.

Scores: verifiable is high, because terraform plan is a mechanical verifier. Reversible is very high, because an import changes state, not infrastructure. Contained is high: no runtime effect.

Verdict: earns autonomy for discovery and proposal. Merging stays human, because the diff encodes a decision about who owns the resource and where it lives in the codebase.

What exists: discovery, coverage and drift calculation, HCL generation and import PRs, implemented end to end. More on IaC coverage and Terraform drift detection.

3. Infrastructure rightsizing (the one that fails)

Operation: find waste, attribute it to a Terraform module and owner, propose the change.

Scores: verifiable is medium (savings are estimated, not proven, until the bill arrives). Reversible is medium (you can scale back up, but not instantly and not without impact). Contained is low to medium (it touches live, load-bearing resources).

Verdict: rightsizing should not be autonomous, and saying so is what makes the framework credible. The score rules it out. That's the framework working, not failing.

What exists: FinOps findings attributed to a Terraform module and owner, proposed as PRs. The reversibility component scores downsizes as harder to undo than additive changes, which pushes them toward human approval. More on AWS cost optimization with Compass.

4. Observability noise suppression

Operation: score incoming findings for risk and urgency, auto-queue the trivial ones, escalate the rest.

Scores: reversible is high (suppression is a routing decision, not a mutation). Contained is high (no infrastructure is touched). Verifiable is only medium: you find out you suppressed something wrongly by missing it, which is a poor feedback loop.

Verdict: earns autonomy, with the caveat that its failure mode is silent. This is the example to be most careful with.

What exists: the triage scoring and auto-queue path described in section 4.

What's still hard: verifiability

The framework says verifiable, reversible and contained. The scoring function in Compass today measures containment and reversibility. It doesn't score verifiability at all. I'd rather say that here than have someone find it in the code.

Today, verification is after the fact and human: someone reviews the PR. For an operation to be genuinely unattended, the system needs a mechanical answer to "did this achieve its intent?" That could be a re-scan confirming the finding is gone, a plan diff that matches the proposal, or a canary that fires on demand. The IaC example shows it's solvable for some operation classes, because terraform plan already is that verifier. For others, like rightsizing and alert suppression, it's an open problem.

Opinion. I think verifiability will turn out to be harder than reversibility or containment, and it's where the field is weakest right now, mine included. Everyone builds governance first because buyers ask for approval gates. Almost nobody asks for verification.

Other open problems I don't have good answers to yet:

Are the weights right? I don't know. They're calibrated on my own estate and they're configuration, not doctrine. They need testing against other people's environments.
Auditability of the gate itself. The action is a reviewable PR, but the reasoning behind each gate decision (the risk breakdown and thresholds at that moment) should be as inspectable as the action. That's work in progress.
Cost. Turn and loop limits bound token spend, but budget-aware planning isn't solved.

Frequently asked questions

What is autonomous DevOps?

Autonomous DevOps is an operating model in which agents continuously observe cloud and pipeline telemetry, decide what needs changing, and act inside declared boundaries without a human approving each instance, while staying observable, auditable and reversible. Humans set the boundary, not the instance. In practice it means deciding, per operation, whether that operation is verifiable, reversible and contained enough to run without human attention.

What's the difference between autonomous DevOps and autonomous CloudOps?

Autonomous DevOps is the general operating model and the framework for deciding which operations an agent may run unattended. Autonomous CloudOps (autonomous cloud operations) is that model applied to the cloud infrastructure plane: cost, Terraform drift, IaC coverage, security posture and compliance findings. The autonomous cloud operations guide covers those workloads.

Are autonomous DevOps agents safe?

An agent is not safe or unsafe in general; an operation is. An agent is safe to run unattended on operations whose damage is small and known in advance, that have a cheap path back, and whose result can be checked mechanically. Everything else should go through human approval. Safety also depends on containing the model itself: no write path to live infrastructure, allow-listed tools, hard turn limits, and output limited to a reviewable pull request.

What are the security challenges of autonomous DevOps agents?

The main ones are: agents inheriting a human operator's broad credentials; prompt injection through findings, logs or repository content the agent reads; unbounded loops and token spend; silent failure modes such as wrongly suppressed alerts; and relying on instructions in a prompt instead of structural limits. The mitigation is to enforce boundaries structurally (missing permissions, no live-mutation path, allow-listed tools, turn limits, diff-only output) and to treat the prompt as the weakest guardrail, not the main one.

How do agentic DevOps tools improve deployment rollback?

They do not make rollback magic. What helps is designing reversibility in before the agent acts: if every change is a Terraform or Git diff delivered as a pull request, rollback is a revert of a known commit. A useful agentic tool also scores reversibility before acting, so deletes and downsizes, which are harder to undo, are pushed toward human approval, while additive or state-only changes can qualify to run unattended.

What are AI guardrails for DevOps?

AI guardrails for DevOps are the limits that decide whether an agent may act and what it can physically do. Two independent layers matter: a decision layer that scores each operation for risk from infrastructure facts and compares it to a threshold the team sets, and a containment layer around the model runtime, such as a pinned repository checkout, allow-listed tools, no write access to live infrastructure, turn limits and pull-request-only output.

Will autonomous DevOps replace SREs?

No. It moves SREs from approving routine instances to setting boundaries: deciding which operation classes qualify for autonomy, tuning the risk threshold, handling exceptions, and owning architecture. The judgement does not disappear; it moves up a level and gets applied once per class of operation instead of once per change.

Can I apply this framework without Beamreach Compass?

Yes. The framework is a way of scoring operations, not a product feature. Any team can classify its operations by verifiability, reversibility and containment, compute a risk level from infrastructure facts, and gate unattended execution on a threshold that a human controls. If you want it engineered into your own environment, that is what our custom autonomous systems engineering does; no Compass required.

How to Reduce Alert Fatigue in Prometheus and Alertmanager: Measure, Cut, Verify

Ilia Dubovskii — Fri, 23 Oct 2026 22:38:45 +0000

Most advice on alert fatigue stops at "make alerts actionable". This guide is the working version for teams on Prometheus and Alertmanager: how to count the alerts that actually reach a person, which config changes cut them (grouping, inhibition, routing, for: durations), which runbooks are safe to automate, how to run a weekly noise review, and how to check afterwards that you didn't throw away the alert that mattered. Every config snippet was checked against the Prometheus and Alertmanager documentation.

The short version

Track one number: alerts that reach a human per week, broken down by alert rule. Label each page with what happened next.
Group by the labels that identify one problem (group_by), and inhibit symptom alerts while their cause is firing (inhibit_rules with a correct equal list).
Page only for severity="page". Everything else goes to a ticket queue or a channel, never to a phone.
Stop flapping with for: and keep_firing_for:, and test rule changes with promtool test rules.
Automate a runbook only when its result can be checked, undone and kept small.
Review the noisiest rules weekly and change them by pull request. After every change, check that real incidents would still have paged.

1. Measure the noise before you touch a rule

"We have too many alerts" is a feeling. To fix it you need a number that goes down when things improve and doesn't go down when you simply break alerting. The one that works: alerts that reach a human per week. That means pages, phone calls and messages that someone is expected to look at, not everything Prometheus evaluates.

There are three places to get it, from easiest to most complete:

Your paging tool. PagerDuty, Opsgenie and Grafana IRM all keep a history of incidents or alert groups you can pull from the UI or the API. Count them per week and group them by alert name.
Alertmanager's own metrics. alertmanager_notifications_total counts notifications sent, labelled by integration (for example pagerduty or slack). sum by (integration) (increase(alertmanager_notifications_total[7d])) gives a weekly total per channel. It counts notifications, not alert groups, so repeats and resolved messages are included. Use it for the trend, not as an exact count.
A webhook log. Add a webhook_configs receiver that writes every notification payload to a file or table. The payload includes each alert's labels, startsAt and fingerprint, so you can count per rule and per firing.

To see how much time each rule spends firing, Prometheus already has the data. It writes a synthetic ALERTS series with alertstate="firing" at every rule evaluation, so the number of samples is the time spent firing divided by the evaluation interval:

# Samples spent firing per rule over the last week.
# Multiply by your rule evaluation interval to get firing time.
topk(20, sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[7d])))

Then add the part no tool gives you: what happened after each page. For a few weeks, have whoever is on call tag every page with one of four outcomes: acted (did something that changed the system), looked, nothing to do, duplicate (another page already covered it) or already known. A rule's actionability rate is acted divided by total pages. It's a rough measure, and that's fine. A rule that paged 40 times last month with 2 actions is obvious without decimals.

2. Sort every paging rule: page, ticket, automate or delete

The Google SRE book's chapter on monitoring has the best short checklist for this. For each rule that can page, ask whether it detects a condition that is urgent, actionable and user-visible that nothing else detects; whether you'll ever ignore it because it's benign; whether the action could wait until morning or be safely automated; and whether other people are being paged for the same issue. The chapter puts it in one line: "Every page should be actionable."

The answers put each rule into one of four buckets:

Bucket	Test	Where it goes
Page	Users are affected or soon will be, and a person must act now	On-call, with a runbook link
Ticket	Needs fixing, but tomorrow is fine (a certificate expiring in 14 days, a disk at 70%)	Ticket queue, deduplicated
Automate	The response is always the same, and its result can be checked	A runbook that runs and verifies, paging only if it fails
Delete	Nobody has acted on it in months, or another alert always fires with it	Removed, after the check in section 9

Prefer paging on symptoms users feel (error rate, latency, failed checkouts) over causes (CPU, memory, a single pod restarting). Cause alerts are useful as context and as tickets. As pages, they're how one bad deploy turns into 30 notifications.

3. Group alerts that describe one problem

Alertmanager batches alerts with the same values for the labels in group_by into one notification. Many configs group too finely, or use group_by: ['...'], which the docs say "effectively disables aggregation entirely". Pick the labels that identify one problem from the responder's point of view, usually cluster, namespace or service, plus the alert name:

route:
  receiver: ticket-queue          # anything not matched below never pages
  group_by: [cluster, namespace, alertname]
  group_wait: 30s                 # wait for the rest of the group before the first notification
  group_interval: 5m              # how often to send updates about a group that changed
  repeat_interval: 4h             # re-send an unchanged, still-firing group
  routes:
    - receiver: oncall-pager
      matchers:
        - severity="page"
    - receiver: team-channel
      matchers:
        - severity="info"

Three details the docs spell out and teams tend to miss:

group_wait also decides whether inhibition works. If it's too short, the first notification goes out before the alert that should inhibit it has arrived. 30 seconds (the default) is a reasonable floor when you rely on inhibition.
repeat_interval is rounded up to a multiple of group_interval, so set it as one.
Child routes inherit group_by, group_wait, group_interval and repeat_interval from the parent unless they set their own. A team route can group by service while the root groups by cluster.

Grafana Alerting works the same way: notification policies have a Group by option and the same three timers, and by default group by alertname and grafana_folder.

4. Inhibit symptoms while the cause is firing

Grouping merges alerts that look alike. Inhibition handles alerts that look different but share a cause: a node runs out of disk, and you also get evictions, crash loops and 5xx errors from everything scheduled on it. An inhibit rule mutes the target alerts while a source alert is firing, as long as the labels in equal match:

inhibit_rules:
  # A critical alert hides the warning version of the same alert.
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: [alertname, cluster, service]

  # Disk pressure on a node hides pod symptoms on that same node.
  # Alert names are examples: use the names your rules actually produce.
  - source_matchers:
      - alertname="NodeDiskPressure"
    target_matchers:
      - alertname=~"KubePodEvicted|KubePodCrashLooping"
    equal: [cluster, node]

The second rule is where most inhibition goes wrong, so check two things.

Both alerts must carry every label in equal. Many pod-level alerts don't have a node label, because the metric they're built on doesn't. If the source has node="ip-10-2-14-7" and the target has no node at all, the values differ and nothing is inhibited. The fix is to add the label in the rule's expression (for example by joining with kube_pod_info), not to drop node from equal.

Missing labels count as equal. The Alertmanager docs say: "if all the label names listed in equal are missing from both the source and target alerts, the inhibition rule will apply." An equal list with a typo in a label name can therefore mute every matching target in every cluster. Keep the source and target matchers narrow, and test the rule (section 9).

Inhibited alerts aren't deleted. They still show in Prometheus and in the Alertmanager API as suppressed, with the inhibiting alert listed. That matters when you check you lost nothing.

5. Route non-urgent alerts to tickets, not phones

The cheapest change on this list: make paging opt-in. With the routing tree above, only severity="page" reaches the on-call receiver. Everything else lands in a ticket queue through a webhook:

receivers:
  - name: oncall-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-key
  - name: ticket-queue
    webhook_configs:
      - url: https://alert-to-ticket.internal.example/hook
        send_resolved: true
  - name: team-channel
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: '#ops-alerts'

The webhook service should create one ticket per alert group (use the payload's groupKey), comment on it on repeats instead of opening a new one, and close it when the alert resolves. That turns the cert-expiring-in-14-days alert from six pages a week into one ticket.

Check routing changes without deploying them: amtool check-config alertmanager.yml validates the file, and amtool config routes test --config.file=alertmanager.yml severity=page service=payments prints which receiver a given label set would reach.

6. Tune thresholds with for: and keep_firing_for:

A rule that fires for 40 seconds and resolves on its own is flapping. Two fields fix most of it:

groups:
  - name: payments
    rules:
      - alert: PaymentsHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="payments", code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="payments"}[5m])) > 0.02
        for: 10m              # must be true for 10 minutes before it fires
        keep_firing_for: 5m   # stays firing 5 minutes after it stops being true
        labels:
          severity: page
        annotations:
          summary: Payments 5xx above 2% for 10 minutes
          runbook_url: https://runbooks.internal.example/payments-5xx

for keeps the alert in the pending state until the condition has held for that long, so short spikes never notify. keep_firing_for (Prometheus 2.42 and later) keeps it firing for a while after the condition clears, so a metric that dips below the threshold for one evaluation doesn't produce a resolve and a fresh page. Grafana alert rules have the same idea as a pending period.

Don't guess the durations. Use the firing-time query from section 1 and your incident history: if real incidents for this rule always lasted longer than 15 minutes and the noise always cleared in under 3, a for of 5 to 10 minutes costs you little detection time. Write the decision down as a unit test so the next person can see why:

# payments_test.yml, run with: promtool test rules payments_test.yml
rule_files:
  - payments_rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      # 3-minute spike to 5% errors, then normal: must NOT fire.
      - series: 'http_requests_total{job="payments", code="500"}'
        values: '0+0x10 0+5x3 15+0x20'
      - series: 'http_requests_total{job="payments", code="200"}'
        values: '0+100x40'
    alert_rule_test:
      - eval_time: 20m
        alertname: PaymentsHighErrorRate
        exp_alerts: []

Add a second test with a sustained error rate that must fire. The pair stops anyone loosening the rule until it misses a real outage.

7. Which runbooks are safe to automate

If the response to an alert is always the same (clear old logs, restart one stuck worker, rotate one node), a person doing it at 2 a.m. adds delay and little judgement. But automatic remediation is only safe when three things are true. We call them verifiable, reversible and contained:

Verifiable. You can tell from a metric whether it worked, ideally the same metric that raised the alert. "Disk on the node below 80% within 5 minutes" is verifiable. "Restarted the service" is not; it only says the command ran.
Reversible. If it was the wrong move, you can undo it. Restarting a stateless pod is reversible in practice. Deleting data or failing over a primary database usually isn't.
Contained. It acts on a bounded scope with hard limits: one node at a time, at most three runs an hour, never during a deploy, never on more than 10% of a pool.

Wire it so the runbook runs, waits, reads the triggering metric again, and if the metric hasn't recovered, stops and pages a person with what it tried. An automated fix that fails quietly is worse than a page. Log every run where the team will see it, so the weekly review can count automated fixes too: a rule fixed automatically 30 times a week is a root cause waiting for a ticket.

8. The weekly noise review

Alert rules decay as the system under them changes, so tuning once doesn't hold. A 30-minute weekly slot with the people who were on call does. A format that works:

The number (5 minutes). Alerts that reached a human this week versus last, and how many were acted on.
The top five rules by pages (20 minutes). For each: how many pages, how many acted on, how long it fired. Decide one of: keep, retune, demote to ticket, automate, delete. Write down why.
Owners (5 minutes). Every decision becomes a pull request against the rules repo with a named owner, merged by the following week.

Two rules keep it honest. Changes go through pull requests, never through a silence that someone forgets to remove. And nobody mutes a rule because it's annoying without saying what would catch the problem instead.

9. Check you didn't lose signal

You can always reduce alerts by alerting on less. The number from section 1 can't tell you whether you did that, so every change needs a second check.

Replay real incidents. Take the last few months of real incidents, especially those found by a page. For each, ask whether the changed rules would still have paged, and how much later. Unit tests make this concrete: feed the incident's metric shape into promtool test rules and assert the alert still fires.
Watch suppressed alerts for a week. Prometheus's ALERTS series isn't affected by Alertmanager inhibition or routing. Compare what fired in Prometheus with what was delivered. Any alert that was inhibited while its source alert was not about the same problem is a bad equal list.
Count incidents found the wrong way. Track incidents first reported by customers, support or another team. If that count rises after a tuning round, you cut too deep, whatever the page count says.
Demote before deleting. Move a doubtful rule to severity="ticket" for a month first. If nobody acts on any of those tickets either, delete it.

10. Where products and agents fit

Everything above works with open-source Prometheus and Alertmanager, and for many teams it's enough. The commercial tools automate parts of it:

PagerDuty offers Intelligent Alert Grouping (machine-learned grouping of related alerts into one incident) and Auto-Pause Incident Notifications (holding notifications for transient alerts so they can resolve on their own) in its AIOps add-on, or as Signal Intelligence in its PD Reliability Platform plans. PagerDuty Runbook Automation can run diagnostics and remediation from incidents.
Datadog Event Management groups related alerts into cases, with Intelligent Correlation (machine learning using service topology) and Pattern-based Correlation (rules you define).
BigPanda correlates alerts from many monitoring tools into incidents and is aimed at large enterprise IT operations teams.
Opsgenie stopped being sold on 4 June 2025 and its support ends on 5 April 2027. Atlassian moves its alerting and on-call features into Jira Service Management, so plan that migration rather than new Opsgenie tuning.

What none of them do for you is the judgement in sections 2, 7, 8 and 9: deciding which rule is wrong, fixing it in your repo, and proving nothing was lost.

11. FAQ

What is alert fatigue?

Alert fatigue is what happens when on-call engineers get so many alerts that lead nowhere that they stop treating each one as urgent. Pages get acknowledged without being read, and the alert that matters is missed or answered late. The fix is fewer alerts reaching people, not faster responses to all of them.

How do I reduce the number of alerts from Alertmanager?

Start by counting pages per alert rule for a few weeks and noting which ones anyone acted on. Then group alerts that describe one problem with group_by, inhibit symptom alerts while their cause is firing with inhibit_rules, route anything non-urgent to a ticket queue instead of the pager, and add for: durations to rules that flap. Review the noisiest rules weekly and change them by pull request.

What is the difference between grouping and inhibition in Alertmanager?

Grouping merges alerts that have the same values for the labels in group_by into one notification, for example nine crash-looping pods in one namespace. Inhibition mutes alerts of a different kind while a related source alert is firing, for example pod evictions on a node that is out of disk. Grouping reduces repeats; inhibition removes symptoms of a cause you already know about.

Why is my Alertmanager inhibit rule not working?

The usual cause is the equal list. Both the source and target alerts must have the same value for every label in it, and if the target alert doesn't carry a label such as node at all, the values differ and nothing is inhibited. A group_wait that's too short is the other cause: the target's notification goes out before the source alert arrives. Check the labels on both alerts in the Alertmanager UI or with amtool.

How long should the for: duration be on a Prometheus alert?

Long enough that short spikes which clear on their own never fire, and short enough that a real incident still pages in time. Look at how long this rule's noisy firings lasted compared with real incidents, and pick a value between them. Add keep_firing_for (Prometheus 2.42 and later) if the alert resolves and re-fires when the metric hovers around the threshold.

Which runbooks are safe to automate?

Runbooks whose result you can check with a metric, that you can undo, and that act on a small, bounded scope, such as clearing old logs on one node or restarting one stateless worker. The automation should re-read the metric that triggered the alert after it runs and page a person if the metric hasn't recovered. Anything that deletes data or fails over a database should stay with a human.

How do I know I didn't lose important alerts after tuning?

Replay recent real incidents against the changed rules, ideally as promtool unit tests, and confirm each one would still have paged. Compare what fired in Prometheus with what Alertmanager delivered for a week to catch inhibit rules that hide too much. Track incidents first reported by customers or other teams: if that number rises after a tuning round, you cut too deep.

Written by Ilia Dubovskii, founder of Beamreach. Config was checked against the Prometheus and Alertmanager documentation, and product details against each vendor's own site, on 10 October 2026. If something here no longer matches the docs, the docs win. Tell us and we'll fix this page.

Originally published at https://beamreach.ai/docs/reduce-alert-fatigue-alertmanager.html.

The verifiable, reversible, contained test from section 7 is the core of my talk Autonomous DevOps: Engineering Systems That Don't Need Our Attention at KCD Porto x DevOpsDays Portugal (19–20 November 2026). Talk page: https://beamreach.ai/talks/autonomous-devops.html

AWS Canary Credentials: How to Set Up an IAM Honeytoken

Ilia Dubovskii — Fri, 23 Oct 2026 22:37:52 +0000

A canary credential, or AWS honeytoken, is a real AWS access key that can do nothing and that nobody legitimate will ever use. You plant it where an attacker looks for credentials. When someone uses it, CloudTrail records the call and you get an alert that carries very little doubt, and that points at exactly one place: wherever you planted that key. This guide builds one from scratch in Terraform, explains the AWS details that most write-ups get wrong, and compares the free and commercial alternatives honestly.

The short version

Create an IAM user with no console access, an explicit deny-all policy and one access key.
Alert on any CloudTrail event from that user. If you use EventBridge, the rule needs state = "ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS" (AWS provider 5.27.0 or later). Without it, you'll miss sts:GetCallerIdentity, which is usually the attacker's first call.
EventBridge rules are regional, so cover every enabled region or use a multi-region trail with a CloudWatch Logs metric filter as a backstop.
Anyone holding the key can learn its AWS account ID without touching your logs. Decide deliberately which account the canary lives in.
Plant one key per location, under names your own tooling won't load automatically.

1. What a canary credential is, and why it's the cheapest certainty

A honeypot is any decoy built to be touched only by an intruder. A honeytoken is a decoy piece of data, such as a credential, a document or a database row, rather than a whole decoy system. A canary token is a honeytoken that alerts you when it's used. An AWS canary credential is all three: a real IAM access key (AKIA…) attached to a user that is explicitly denied everything.

The point isn't the decoy, it's the quality of the signal. Most alerts measure a symptom: CPU, egress, error rate, cost. They fire after something has started to go wrong, against a baseline that legitimate work (and attackers) move around in. That's why a human has to triage them, and why automating a response to them is risky. If one alert in a thousand is an attack, an automatic kill switch is wrong 999 times.

A canary trip measures intent. Someone found a credential, decided it was worth stealing, and used it. There's no baseline to tune, and because each key lives in exactly one place, the key ID tells you where it was taken from. That's a signal you can act on immediately, including automatically, as long as the action is one you can undo.

Canaries aren't the only high-confidence signal. A GuardDuty finding for instance credentials used outside AWS, or a Falco rule for a shell spawned in a production container, can be just as trustworthy, and if you trust them you should automate on them too. The rule is to automate on certainty. Canaries are simply the cheapest way to manufacture certainty: no agents, they work on ECS Fargate and Lambda, and one takes an afternoon. They don't replace your alerts. Keep both.

2. Your options: canarytokens.org, Thinkst Canary, DIY, Compass

Option	What you get	Enough when…
canarytokens.org (free, by Thinkst)	An "AWS API key" token in about a minute. Thinkst owns the AWS account and the detection, and alerts come by email or webhook. Their docs say alerts can take between 2 and 30 minutes.	You want a first canary today and don't mind the key living in a well-known account. TruffleHog now recognises these keys offline, so a careful attacker using it won't trip them.
Thinkst Canary (commercial)	A mature product: hardware, virtual and cloud Canaries plus a managed console for tokens, including AWS keys. TruffleHog says it deliberately doesn't fingerprint the paid service's accounts.	You want canaries across the whole estate (network, Windows, cloud) with a vendor who has done this for years.
DIY with Terraform (this guide)	A key in an account you choose, alerting through your own CloudTrail, EventBridge and SNS. You own every moving part.	You have a handful of high-value places to cover and someone who will keep the region coverage and placements up to date.
Beamreach Compass	Suggests placements from a map of your AWS infrastructure, plants them as Terraform PRs in your own repo and account, and routes trips to Slack or PagerDuty. Today only the IAM canary is available. The rest is in development or on the roadmap; see the honest status.	You want placement and lifecycle managed through code review and wired into an automated response. Disclosure: we make it.

If you're unsure, start with one free token today and build the DIY version this week. The rest of this guide is the DIY version.

3. How detection works in AWS: the facts that matter

The whole design rests on a few AWS behaviours. Each one below is checked against AWS documentation, which is linked.

CloudTrail records calls made with the key, including denied ones. A call refused by the deny-all policy is still an authenticated API call. It's logged with errorCode (for example AccessDenied), plus sourceIPAddress, userAgent and userIdentity.accessKeyId. (CloudTrail record contents.)
sts:GetCallerIdentity can't be denied, and it is logged. AWS says no permissions are required and that an explicit deny doesn't stop it (API reference). CloudTrail captures all IAM and STS API calls (IAM user guide). This is the "whoami" call that attackers and secret scanners make first. Your canary succeeds at it, which helps it look real, and the call still lands in your logs.
EventBridge ignores read-only calls unless you opt in. CloudTrail sends "AWS API Call via CloudTrail" events to the default event bus. A rule in the normal ENABLED state matches write events but not read-only management events such as Get*, List*, Describe* and GetCallerIdentity. Only a rule in the ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS state matches those (EventBridge: events via CloudTrail, read-only management events). In Terraform this is the state argument of aws_cloudwatch_event_rule, added in AWS provider 5.27.0 (it replaces the deprecated is_enabled). This is the single most common bug in DIY AWS canaries: a rule that only catches the attacker's second, mutating call, if they ever make one.
You need a trail. EventBridge only receives CloudTrail-sourced events if a trail with logging is enabled. Event history alone isn't enough (same page).
Events land in the region that was called. EventBridge rules are regional. A call to a regional endpoint such as sts.eu-west-1.amazonaws.com is logged in eu-west-1. Calls to the global STS endpoint are logged in us-east-1 (IAM user guide). The attacker chooses the region, not you.
Forwarding read-only events needs the special state twice. If you forward events from one region's default bus to another bus, both the forwarding rule and the destination rule need ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS (same page).
CloudTrail isn't instant. AWS says CloudTrail typically delivers events to S3 and CloudWatch Logs within an average of about 5 minutes, and that this isn't guaranteed (CloudTrail to CloudWatch Logs). One vendor's measurement of S3 delivery found an average around 2.5 minutes and a 99th percentile just over 5 (Tracebit). AWS doesn't publish a latency figure for the EventBridge path, so measure your own (section 9).

4. Step 1: the canary user and key

Use one IAM user per placement. When a key trips, the user name and key ID should map to exactly one location. Name it like your other service users. Anything visible in AWS (name, path, description, tags) can be read by whoever holds a credential with iam:Get* or iam:List*, so none of it should say "canary".

terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.27, < 6.0" # 5.27.0 added aws_cloudwatch_event_rule.state
    }
  }
}

# Home region. Global-endpoint IAM/STS calls are logged here.
provider "aws" {
  region = "us-east-1"
}

data "aws_caller_identity" "current" {}

locals {
  account_id  = data.aws_caller_identity.current.account_id
  canary_name = "svc-backup-reporting" # match your real naming style
  home_bus    = "arn:aws:events:us-east-1:${local.account_id}:event-bus/default"
}

resource "aws_iam_user" "canary" {
  name = local.canary_name
  path = "/service/"
  # No aws_iam_user_login_profile: no console password, API key only.
  # No tags or descriptions that give the game away.
}

resource "aws_iam_user_policy" "deny_all" {
  name = "backup-reporting"
  user = aws_iam_user.canary.name
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{ Effect = "Deny", Action = "*", Resource = "*" }]
  })
}

resource "aws_iam_access_key" "canary" {
  user = aws_iam_user.canary.name
}

output "canary_access_key_id" {
  value = aws_iam_access_key.canary.id
}

output "canary_secret_access_key" {
  value     = aws_iam_access_key.canary.secret
  sensitive = true
}

Two notes. First, the secret ends up in Terraform state in plain text. For a key that can do nothing that's usually acceptable, and anyone who steals it from state trips it. If your policy forbids it, use the pgp_key argument or create the key outside Terraform. Second, an explicit Deny wins over any Allow, so even if someone later attaches a policy to this user by mistake, the key stays useless.

5. Step 2a: fast alerts with EventBridge, in every region

Match on the user's ARN rather than the key ID. If an attacker exchanges the key for a session token, later calls carry a temporary ASIA… key ID but the same user ARN. The home-region rule sends matches to an SNS topic. Every other region gets a copy of the rule that forwards matches to the home region's default bus.

variable "alert_email" {
  type = string
}

resource "aws_sns_topic" "alerts" {
  name = "ops-notifications"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email # click the confirmation link AWS sends
}

# Let EventBridge rules and CloudWatch alarms in this account publish.
data "aws_iam_policy_document" "alerts" {
  statement {
    sid       = "EventBridgePublish"
    actions   = ["sns:Publish"]
    resources = [aws_sns_topic.alerts.arn]
    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }
    condition {
      test     = "ArnLike"
      variable = "aws:SourceArn"
      values   = ["arn:aws:events:us-east-1:${local.account_id}:rule/*"]
    }
  }
  statement {
    sid       = "CloudWatchAlarmPublish"
    actions   = ["sns:Publish"]
    resources = [aws_sns_topic.alerts.arn]
    principals {
      type        = "Service"
      identifiers = ["cloudwatch.amazonaws.com"]
    }
    condition {
      test     = "ArnLike"
      variable = "aws:SourceArn"
      values   = ["arn:aws:cloudwatch:us-east-1:${local.account_id}:alarm:*"]
    }
  }
}

resource "aws_sns_topic_policy" "alerts" {
  arn    = aws_sns_topic.alerts.arn
  policy = data.aws_iam_policy_document.alerts.json
}

locals {
  canary_pattern = jsonencode({
    "detail-type" = ["AWS API Call via CloudTrail"]
    detail = {
      userIdentity = {
        arn = [aws_iam_user.canary.arn]
      }
    }
  })
}

# Home region rule. It also matches events forwarded from other regions.
resource "aws_cloudwatch_event_rule" "canary_home" {
  name          = "backup-reporting-activity"
  state         = "ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS"
  event_pattern = local.canary_pattern
}

resource "aws_cloudwatch_event_target" "canary_home_sns" {
  rule = aws_cloudwatch_event_rule.canary_home.name
  arn  = aws_sns_topic.alerts.arn
}

# Role EventBridge uses to forward events from other regions to us-east-1.
resource "aws_iam_role" "events_forwarder" {
  name = "events-forwarder"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "events.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "events_forwarder" {
  name = "put-events-home-bus"
  role = aws_iam_role.events_forwarder.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "events:PutEvents"
      Resource = local.home_bus
    }]
  })
}

# Repeat this block for EVERY region enabled in the account.
# eu-west-1 shown. List yours with: aws account list-regions --region-opt-status-contains ENABLED ENABLED_BY_DEFAULT
provider "aws" {
  alias  = "eu_west_1"
  region = "eu-west-1"
}

resource "aws_cloudwatch_event_rule" "canary_eu_west_1" {
  provider      = aws.eu_west_1
  name          = "backup-reporting-activity"
  state         = "ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS"
  event_pattern = local.canary_pattern
}

resource "aws_cloudwatch_event_target" "canary_eu_west_1" {
  provider = aws.eu_west_1
  rule     = aws_cloudwatch_event_rule.canary_eu_west_1.name
  arn      = local.home_bus
  role_arn = aws_iam_role.events_forwarder.arn
}

Wrap the per-region pair in a small module and instantiate it once per region with a provider alias. (AWS provider v6 adds a per-resource region argument that lets you for_each over regions instead. This guide sticks to v5.) Email is the simplest target to show. In practice, subscribe your paging tool to the topic, or point the rule's target at an SQS queue or Lambda that posts to Slack.

EventBridge still needs a trail that is logging (section 3). Most accounts already have one. If yours doesn't, the trail in step 2b covers it. Configure it for both read and write management events. We haven't verified whether a write-only trail still feeds read-only events to EventBridge, so don't rely on it.

6. Step 2b: the backstop, a multi-region trail with a metric filter

A multi-region trail sends events from every enabled region to one CloudWatch Logs log group (AWS docs). A metric filter on the canary's ARN, plus an alarm, catches calls in any region, including regions you forget to add rules for. It's slower, because it waits for CloudTrail's delivery to CloudWatch Logs. If you already have a multi-region trail, add the CloudWatch Logs settings to it instead of creating a second one. CloudTrail bills additional copies of management events (pricing).

locals {
  trail_name = "management-events"
  trail_arn  = "arn:aws:cloudtrail:us-east-1:${local.account_id}:trail/${local.trail_name}"
}

resource "aws_s3_bucket" "trail" {
  bucket_prefix = "audit-logs-"
}

resource "aws_s3_bucket_public_access_block" "trail" {
  bucket                  = aws_s3_bucket.trail.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "trail_bucket" {
  statement {
    sid       = "AWSCloudTrailAclCheck"
    actions   = ["s3:GetBucketAcl"]
    resources = [aws_s3_bucket.trail.arn]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }
  statement {
    sid       = "AWSCloudTrailWrite"
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.trail.arn}/AWSLogs/${local.account_id}/*"]
    principals {
      type        = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceArn"
      values   = [local.trail_arn]
    }
  }
}

resource "aws_s3_bucket_policy" "trail" {
  bucket = aws_s3_bucket.trail.id
  policy = data.aws_iam_policy_document.trail_bucket.json
}

resource "aws_cloudwatch_log_group" "trail" {
  name              = "cloudtrail/management-events"
  retention_in_days = 90
}

resource "aws_iam_role" "trail_to_logs" {
  name = "cloudtrail-to-cloudwatch-logs"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "cloudtrail.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "trail_to_logs" {
  name = "write-trail-log-group"
  role = aws_iam_role.trail_to_logs.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["logs:CreateLogStream", "logs:PutLogEvents"]
      Resource = "${aws_cloudwatch_log_group.trail.arn}:*"
    }]
  })
}

resource "aws_cloudtrail" "main" {
  name                          = local.trail_name
  s3_bucket_name                = aws_s3_bucket.trail.id
  is_multi_region_trail         = true
  include_global_service_events = true
  enable_log_file_validation    = true
  cloud_watch_logs_group_arn    = "${aws_cloudwatch_log_group.trail.arn}:*"
  cloud_watch_logs_role_arn     = aws_iam_role.trail_to_logs.arn

  event_selector {
    read_write_type           = "All" # read-only calls like GetCallerIdentity included
    include_management_events = true
  }

  depends_on = [aws_s3_bucket_policy.trail, aws_iam_role_policy.trail_to_logs]
}

resource "aws_cloudwatch_log_metric_filter" "canary" {
  name           = "backup-reporting-activity"
  log_group_name = aws_cloudwatch_log_group.trail.name
  pattern        = "{ $.userIdentity.arn = \"${aws_iam_user.canary.arn}\" }"

  metric_transformation {
    name      = "BackupReportingActivity"
    namespace = "Ops"
    value     = "1"
  }
}

resource "aws_cloudwatch_metric_alarm" "canary" {
  alarm_name          = "backup-reporting-activity"
  namespace           = "Ops"
  metric_name         = "BackupReportingActivity"
  statistic           = "Sum"
  period              = 60
  evaluation_periods  = 1
  threshold           = 1
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [aws_sns_topic.alerts.arn]
}

An alarm notifies when it moves into ALARM, not on every matching event. A second trip while it's still in alarm won't page again, so use the EventBridge path for per-event detail and treat the alarm as the "something happened somewhere" backstop.

	2a: EventBridge rules	2b: Trail → Logs → metric filter
Latency	Doesn't wait for log-file delivery. AWS publishes no figure, so measure it.	CloudTrail to CloudWatch Logs averages about 5 minutes per AWS (not guaranteed), plus the alarm period.
Region coverage	Only regions where you deployed a rule	Every enabled region, through one multi-region trail
Event detail in the alert	The full CloudTrail event	A metric crossing a threshold. Query Logs for detail.
Cost driver	Small. Canary events are rare, though cross-region forwarding is billed per event.	CloudWatch Logs ingests every management event in the account, which can add up in busy accounts
Main failure mode	A region without a rule, or a rule missing the read-only state	Slow, and no per-event paging

If you can afford both, run both: 2a for speed, 2b so a missed region can't hide a trip.

7. Account-ID fingerprinting: which account should own the canary?

Anyone holding an access key ID can find out which AWS account owns it, without ever using the key. sts:GetAccessKeyInfo returns the account ID for any key ID (API reference), and the call is logged only in the caller's account, not yours (Hacking the Cloud). The account ID can also be decoded offline from the key ID itself, a technique published by Aidan Steele and later refined by Tal Be'ery. That's how TruffleHog spots canarytokens.org keys without tripping them.

So the account a canary lives in is part of the disguise:

In the same account as the workload. The key matches the account ID an attacker sees everywhere else in the environment (ARNs, task metadata), so it looks like it belongs. The cost is a deny-all IAM user in production. It's harmless, but it's one more principal to explain to auditors.
In a dedicated canary account. This keeps production clean and makes the canary easy to manage centrally. The cost is that its account ID matches nothing else the attacker has found. Give the account a believable name in your organisation and don't reuse it for anything that would make it look empty.
In a third party's account (canarytokens.org). This is the easiest option, and also the easiest for tools to recognise.

There's no universally right answer. Pick one deliberately, and write the choice down next to the Terraform.

8. Where to plant canaries, and where not to

Good places, where an attacker who got in would look:

CI runners: a pipeline variable with a plausible but non-default name, such as BACKUP_AWS_ACCESS_KEY_ID and BACKUP_AWS_SECRET_ACCESS_KEY.
Build boxes and bastions: a named profile in ~/.aws/credentials, for example [backup-prod]. Not [default].
Containers (ECS, Fargate, Kubernetes): extra env vars in the task definition, or a decoy .env file in the image. Malware routinely dumps the environment, and on Fargate this is the canary with no agent at all.
Secrets stores: a decoy secret named like your real ones, holding the canary key.
Old internal repos, wikis and runbooks: places where leaked keys realistically sit for years.

Bad places, which cause false trips or outages:

Standard variable names in a workload that uses an AWS SDK. The SDK default credential chain reads AWS_ACCESS_KEY_ID before the container or instance role. You'd silently replace the task's real credentials with a deny-all key, break the service, and trip the canary yourself.
A [default] profile on any machine where the CLI, Terraform or an SDK runs.
Anywhere your own secret scanner verifies keys. Scanners that validate AWS keys typically call GetCallerIdentity. Either exclude the canary locations or recognise your scanner's user agent in the alert.
Public repos, unless you want to measure the internet. Public credentials are found and tried by bots, and AWS itself may react to keys exposed publicly. You'll get trips that say nothing about your own environment.
Backups or images you share with vendors, where a legitimate third party may reasonably test what they find.

One key per location, always. When it trips, the key is the address.

9. Testing it safely

Tell whoever is on call first. Run the test from a machine whose IP you know, in a subshell so the credentials don't leak into your session:

(
  export AWS_ACCESS_KEY_ID="$(terraform output -raw canary_access_key_id)"
  export AWS_SECRET_ACCESS_KEY="$(terraform output -raw canary_secret_access_key)"
  unset AWS_SESSION_TOKEN AWS_PROFILE
  date -u +%H:%M:%S

  # 1. Read-only, can't be denied: succeeds. Proves the read-only state works.
  aws sts get-caller-identity --region us-east-1

  # 2. Denied call: fails with AccessDenied, and is still logged.
  aws s3api list-buckets --region us-east-1

  # 3. A regional endpoint in another region: proves multi-region coverage.
  aws sts get-caller-identity --region eu-west-1 \
    --endpoint-url https://sts.eu-west-1.amazonaws.com
)

You should get three alerts, or one alarm for path 2b. Note the time each arrives and compare it with the event's eventTime. That difference is your latency, which is the only number worth quoting. If test 1 produces nothing, the rule is missing ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS. If test 3 produces nothing, eu-west-1 has no rule. To confirm CloudTrail saw the calls regardless of alerting:

aws cloudtrail lookup-events --region us-east-1 \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA...

Event history is regional, so run the lookup with --region eu-west-1 as well to see the third call.

Repeat the test after every change to regions, trails or alert routing, and on a schedule. A canary that silently stopped alerting is worse than none, because you believe you're covered.

10. What to do when it trips

Read the event: eventName, sourceIPAddress, userAgent, awsRegion. Rule out your own test or your own scanner in under a minute. Everything else is real until proven otherwise.
Go to the source, not the key. The key tells you where it was planted: that runner, that task definition, that laptop. Whatever could read the canary could also read the real credentials stored next to it.
Contain that source. Stop or isolate the workload (on ECS, stopping the task lets the service replace it from a clean image). Revoke active sessions for the role it ran as, and rotate any real secrets it could reach.
Leave the canary key active for now. It can't do anything, and every further call it makes shows you what the attacker is trying. Deactivating it immediately also tells them they've been seen.
Afterwards, burn it and plant a fresh one. A used canary is a known canary.

Steps 2 and 3 are good candidates for automation, because the signal is certain and the action is reversible and contained. Our autonomous DevOps guide sets out that test (verifiable, reversible, contained) in more detail.

11. Limits and what we couldn't verify

EventBridge latency. AWS publishes no figure for CloudTrail-to-EventBridge delivery. Measure it as in section 9.
Write-only trails. We haven't confirmed whether a trail set to write-only management events still feeds read-only events to EventBridge. Use read_write_type = "All".
Unlogged APIs. Security researchers have repeatedly found AWS API calls that weren't recorded by CloudTrail and could reveal the caller's identity in an error message. AWS has fixed reported cases, but a careful attacker may still validate a key without tripping it. A canary makes detection very likely, not guaranteed.
Delivery delays. CloudTrail occasionally delivers late. Delayed events carry an addendum with reason DELIVERY_DELAY.
Coverage. A canary only covers where it's planted. It tells you nothing about an attacker who never looks there. That's why you keep your other alerts.

12. FAQ

What is an AWS canary token?

An AWS canary token is a real AWS access key that exists only to be stolen. It belongs to an IAM user that is denied every action, so it can't do any harm, but any API call made with it is recorded by CloudTrail and can trigger an alert. Because nobody legitimate uses it, an alert is high-confidence by design. Because each key is planted in exactly one place, the key ID also tells you where it was stolen from.

Is an AWS honeytoken the same as a canary credential?

In practice, yes. A honeytoken is any decoy piece of data, and a canary credential is a honeytoken in the form of a credential, wired to alert when used. An AWS honeypot is the broader term and can also mean decoy servers, buckets or databases.

Does sts:GetCallerIdentity trip an IAM canary?

It's logged by CloudTrail, and it succeeds even with a deny-all policy because it needs no permissions. Whether it trips your alert depends on the alerting. An EventBridge rule only matches it if the rule's state is ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS (Terraform: the state argument of aws_cloudwatch_event_rule, AWS provider 5.27.0 or later), in the region where the call was made. A multi-region trail feeding a CloudWatch Logs metric filter catches it in any region, more slowly.

How quickly will I get the alert?

It depends on the path, and you should measure your own. AWS says CloudTrail typically delivers events to S3 and CloudWatch Logs within an average of about 5 minutes, without a guarantee. canarytokens.org documents 2 to 30 minutes for its AWS key tokens. AWS doesn't publish a latency figure for CloudTrail events reaching EventBridge, so time your own test calls against each event's eventTime.

Can a canary key be used to do damage?

Not if it's built as described. The user has no console password and an explicit Deny on every action, and an explicit deny overrides any allow added later. The one thing it can do is call sts:GetCallerIdentity, which reveals the user's ARN and account ID. Choose names and the owning account with that in mind.

Should I use canarytokens.org or build my own?

Use canarytokens.org if you want a canary in the next five minutes. It's free and run by Thinkst. Build your own if you need the key to live in an account you control, because the free tokens sit in known Thinkst accounts that secret scanners such as TruffleHog recognise offline. Build your own too if you want alerts in your own pipeline. Thinkst Canary, the commercial product, and Beamreach Compass are the managed options.

How do I detect leaked AWS keys that aren't canaries?

Enable GuardDuty, which flags credentials used from unusual places, including EC2 instance credentials used outside AWS. Run secret scanning on your repositories and CI logs, review IAM access key last-used data and remove unused keys, and move workloads to short-lived role credentials. Canaries complement this. They catch the act of looking for credentials in places you choose, even when the real keys are never touched.

Written by Ilia Dubovskii, founder of Beamreach. AWS behaviours were checked against AWS documentation on 4 October 2026. AWS changes, so if something here no longer matches the docs, the docs win. Tell us and we'll fix this page.

Originally published at https://beamreach.ai/docs/aws-canary-credentials.html.

I'm giving a five-minute Ignite on this idea, One Fake Password Is Worth More Than 10,000 Alerts, at DevOpsDays Barcelona on 14 November 2026. Talk page: https://beamreach.ai/talks/one-fake-password.html

DEV Community: Ilia Dubovskii

AI Root Cause Analysis for Incidents: What It Can and Can't Do

1. What AI root cause analysis can and can't do

2. What to collect in the first minutes

3. Ranking causes: confidence, evidence and "ruled out"

4. Read-only by default: propose, don't act

5. Confirming recovery on the metric that fired

6. Post-mortems you can trust

7. Measuring MTTR honestly

8. Buy or build?

9. FAQ

How accurate is AI root cause analysis?

Should an AI incident agent have write access to production?

What data does an AI incident investigation need?

Can AI write the post-mortem?

Is MTTR a good metric for AI incident response?

Should we buy an AI SRE tool or build our own?

Autonomous DevOps: A Framework for Deciding What AI Agents May Do Alone

What is autonomous DevOps?

Attention is the scarce resource

Autonomy is a property of the operation: verifiable, reversible, contained

Verifiable

Reversible

Contained

The autonomy gate and the dial

Containing the model itself

Security challenges of autonomous DevOps agents

Trust Engineering, and how teams actually adopt autonomy

Four worked examples

1. Honeypot and canary management

2. IaC coverage and drift

3. Infrastructure rightsizing (the one that fails)

4. Observability noise suppression

What's still hard: verifiability

Frequently asked questions

What is autonomous DevOps?

What's the difference between autonomous DevOps and autonomous CloudOps?

Are autonomous DevOps agents safe?

What are the security challenges of autonomous DevOps agents?

How do agentic DevOps tools improve deployment rollback?

What are AI guardrails for DevOps?

Will autonomous DevOps replace SREs?

Can I apply this framework without Beamreach Compass?

Related reading

How to Reduce Alert Fatigue in Prometheus and Alertmanager: Measure, Cut, Verify

1. Measure the noise before you touch a rule

2. Sort every paging rule: page, ticket, automate or delete

3. Group alerts that describe one problem

4. Inhibit symptoms while the cause is firing

5. Route non-urgent alerts to tickets, not phones

6. Tune thresholds with for: and keep_firing_for:

7. Which runbooks are safe to automate

8. The weekly noise review

9. Check you didn't lose signal

10. Where products and agents fit

11. FAQ

What is alert fatigue?

How do I reduce the number of alerts from Alertmanager?

What is the difference between grouping and inhibition in Alertmanager?

Why is my Alertmanager inhibit rule not working?

How long should the for: duration be on a Prometheus alert?

Which runbooks are safe to automate?

How do I know I didn't lose important alerts after tuning?

AWS Canary Credentials: How to Set Up an IAM Honeytoken

1. What a canary credential is, and why it's the cheapest certainty

2. Your options: canarytokens.org, Thinkst Canary, DIY, Compass

3. How detection works in AWS: the facts that matter

4. Step 1: the canary user and key

5. Step 2a: fast alerts with EventBridge, in every region

6. Step 2b: the backstop, a multi-region trail with a metric filter

7. Account-ID fingerprinting: which account should own the canary?

8. Where to plant canaries, and where not to

9. Testing it safely

10. What to do when it trips

11. Limits and what we couldn't verify

12. FAQ

What is an AWS canary token?

Is an AWS honeytoken the same as a canary credential?

Does sts:GetCallerIdentity trip an IAM canary?

How quickly will I get the alert?