Nobody Could Prove It Wasn't Us

An SRE reference pattern for the one question that stalls every multi-region incident - and where AI actually belongs

---

In September 2025, in one of my applications, latency in one region started climbing. Not a cliff, not an outage. Steadily worse, on a multi-tenant SaaS platform in financial services running active-active across two AWS regions, with Aurora Global Database underneath and Route 53 geolocation routing on top.

Every dashboard we owned was green. Target groups healthy. Pods running. Database connections fine. And yet a meaningful share of users in one geography were having a bad time, and Route 53 geolocation kept faithfully routing them straight back into the region that was hurting them, because geolocation routing does not know anything about pain. It knows where you are.

So, we sat on a bridge and asked the only question that mattered.

Is this us, or is this them?

If it was the cloud provider, the answer was to shift traffic out of the region. If it was our own change, shifting traffic out was the worst available move: we would carry the problem with us, abandon a healthy region, concentrate all load onto the survivor, and still be broken, with fewer options and a database failover we could not undo.

We could not tell. We could not even tell whether we were looking at a DNS problem or an application problem. And so:

Nobody wanted to make the call, because nobody could prove it wasn't us.

We argued about it for the better part of two hours. We rolled back the most recent change first, because it was the reversible move and because it would at least eliminate one of the two worlds. It changed nothing. That told us something, but it told us slowly, and we were still guessing.

The cause turned out to be on the provider’s side. We had spent two hours proving a negative, and the only thing that would have shortened those two hours was evidence we had never arranged to have on hand.

That is not a tooling gap. We could shift traffic in minutes. The runbook existed. The routing controls were provisioned. What we could not do was produce, quickly and defensibly, a reason to use them.

This article is what I built in response: the architecture, where AI sits inside it, where it emphatically does not, and what the dry runs taught us. It is written as a reference pattern. If you run active-active anywhere, you should be able to lift it.

Why this is not another agentic AI post

Fair question before you spend ten minutes here: why read this when every feed is already full of AI operations content?

Because nearly all of that content is the same artifact - an agent with tools and credentials, handed a goal, looping until it declares itself done. Impressive in a demo. In infrastructure the demo is not the hard part. The hard part is being wrong at three in the morning with production traffic in your hands.

This is not conventional AI bolted onto a chat box. It is a language model placed inside an infrastructure decision path, and almost every design choice here exists to constrain it:

• It holds no credentials and calls no APIs. It cannot act. It reads a structured evidence bundle and returns a structured verdict.
• It answers one question - is it us or is it them - because that is the question that actually stalls the incident, and the one no dashboard answers.
• Deterministic policy decides whether anything happens. Versioned, reviewed like code, auditable afterwards. Never a model.
• Irreversible actions stay human, permanently, regardless of how good the model gets.

The honest framing: we are not automating the decision. We are automating the evidence the decision needs. That is a much smaller claim than the one the agentic posts make, and it is the one that survives contact with a real incident.

So if you are here for an autonomous agent that resolves outages while you sleep, this will disappoint you, and it should. What follows is the narrower thing that actually works, including the parts of it that failed.

Why this question is genuinely hard

It would be convenient if this were a monitoring problem. It isn’t, instead its gray failure.

A gray failure is one in which the system's own failure detectors do not perceive a problem that the system's applications are actively suffering from.

Not up. Not down. The system's internal model of its own health has diverged from what its users are experiencing. Your health check passes and your customer's request fails, and both are true at the same time.

Both readings are correct. Neither is complete.

Which is why the vocabulary we all use - healthy, unhealthy, in-service - cannot describe the incidents that actually hurt. A region that goes dark is easy: every alarm fires and the decision makes itself. The expensive ones are where the region is mostly fine, nothing has breached a threshold, and the humans have to adjudicate.

But gray failure only explains why detection is hard. The second problem underneath it is what actually stalls the bridge call.

Our change and their infrastructure produce the same symptoms. A bad deploy that reached one region first, and a regional impairment in that same region, look identical from the outside. Same latency. Same error skew. Same graphs. Opposite correct action.

No additional metric resolves that, because the disambiguating evidence does not live in your metrics at all. It is scattered across deployment history, dependency graph, provider health feed, replication lag, and whether you can still launch capacity. Nobody correlates those under time pressure at three in the morning.

The same problem, in public

Ours was a private version of a problem that went public a month later. On 20 October 2025, AWS published a post-event summary describing the same dynamic at a scale everybody could see: a race condition in DynamoDB's internal DNS automation left a regional endpoint with no IP addresses. The region was never down - existing instances kept serving, new ones could not launch.

Attribution is the long pole. The failure began at 11:48 p.m. AWS identified the DNS state as the source at 12:38 a.m. Fifty minutes, inside AWS, by the team that wrote the system, with complete internal telemetry. Be honest about how long that takes the rest of us from outside.

And at 9:36 a.m., AWS manually disabled its own automatic Network Load Balancer failover. Network propagation was lagging, so health checks failed, so NLB removed capacity that was not actually broken. The health checks were not wrong - the observation path was degraded, not the targets. The signal was correct and the conclusion was harmful. It stayed off for four and a half hours.

A human had to switch off the automation during the exact incident that automation existed for, because it could not distinguish "this target is broken" from "my ability to observe this target is broken."

That is the design constraint for everything that follows.

What we tried first

Three things, in order, none of which worked. They are worth listing because they are the obvious answers and you will be offered all three.

Tighter thresholds produce alerts, not answers. They fire on the good region during normal variance and say nothing whatsoever about cause. We ended up with a louder version of the same argument.

More dashboards - per-region, per-tenant, per-endpoint - were genuinely useful for the post-mortem and useless during the incident. The bottleneck was never information volume. It was correlation under time pressure, by tired people, across systems that do not share a query language.

A better runbook was a useful failure. We wrote a decision tree, then tried to follow it during a drill and found that the first branch - "has there been a recent change to this service?" - took eleven minutes to answer, because the deployment history lived somewhere nobody on the bridge had open. The runbook was not wrong. It was correct and unexecutable, because it assumed evidence was at hand that was not.

Which reframes the whole thing. This is an evidence assembly problem wearing a decision-making costume.

The decomposition

The design starts by refusing to hand the decision to a model. Instead, split it into three questions that have three genuinely different kinds of answer.

• Where is the impairment? A statistics question. Deterministic and explainable.
• Why is it happening - our change, or their infrastructure? An evidence synthesis question, spread across systems that do not talk to each other. This is the hard part.
• Whether to act? A policy question. Versioned rules, reviewed like code, auditable afterwards. Never a model.

Once the questions are separated, it becomes obvious that only one of them wants a language model, and it is the middle one.

The AI occupies exactly one stage. It never holds credentials and never calls an API.

The first stage collects evidence, and the choice of signals is most of the value. Seven collectors, each a small Lambda, each returning structured JSON and never prose:

• Synthetic canaries run from outside the region - the direct antidote to differential observability. If you take one thing from this article, take this one.
• Multi-window SLO burn rate per region and per tenant, and the provider health feed (AWS Health API) read programmatically rather than by refreshing a status page.
• Recent change events - deployments, flags, config pushes, timestamped. This is the "is it us" input, and the one most teams cannot produce quickly.
• Dependency graph from distributed tracing, to establish blast radius.
• Aurora Global Database replication lag, which gates whether evacuation is safe at all.
• Control-plane liveness - can we launch a node right now? This catches the region that serves existing traffic perfectly and cannot scale, which is exactly the failure mode in the public example above.
• Capacity headroom in the surviving region - can it absorb the shift?

The second stage is deterministic statistics: is one region a genuine outlier, or is this normal variance? Not a model - it has to be explainable to an auditor and identical every time it runs on the same inputs.

What the AI actually does

The model gets exactly one job: read the structured evidence bundle and return a structured verdict. Correlation across heterogeneous, partially contradictory sources under time pressure happens to be the thing language models are genuinely good at. It does nothing else.

The output schema is the whole contract:

{
  "impairment_scope":  "none | single-az | regional-partial | regional-full",
  "attribution":       "our-change | provider-infrastructure |
                        dependency | unknown",
  "confidence":        0.0 - 1.0,
  "recommended_action":"none | rollback | partial-shift | full-evacuation",
  "reversibility":     "two-way-door | one-way-door",
  "evidence": [
    { "claim": "...", "source": "canary|health|deploy|trace|lag|capacity",
      "observation": "...", "timestamp": "..." }
  ],
  "dissenting_evidence": [ ... ]
}

Four constraints make this safe, and all four are load-bearing.

1. No free text drives control flow. Downstream code reads enumerated fields only. Nothing parses a sentence for intent.
2. No tools, no credentials. The model cannot call an API, shift traffic, or read anything that was not handed to it. If it were fully compromised, the worst it could do is return a wrong opinion into a policy gate that is designed to distrust it.
3. Every claim carries a citation back to a raw observation. A verdict you cannot audit months later is worthless in a regulated environment.
4. Dissenting evidence is mandatory. The model must report what contradicts its own conclusion. This mattered more than expected - see the failures section.

The prompt-injection surface here is real and easy to miss. Log lines and error strings can contain attacker-influenced text. That is why evidence is structured and enumerated before it reaches the model, and why the model's output can never select an action directly.

Where it runs

There is an obvious failure mode in everything described so far: if the system that decides to evacuate a region can itself be taken down by that region's impairment, you have built an expensive way to do nothing at the worst possible moment.

The recovery path must not share fate with the thing it is recovering.

So the agent never runs in the region it is judging. Identical stacks run in at least two other regions and must agree before anything happens. Quorum is not for accuracy; it is for fate-sharing. A single witness that happens to be correlated with the impaired region is not a witness.

This is AWS's own static stability principle - rely on data planes, not control planes, during recovery - applied to the decision layer rather than the data layer. It is the same reason ARC's routing-control data plane is spread across five regions.

What it is allowed to do

The last question is authority, and the answer has nothing to do with how confident the model is. It has to do with whether the action can be undone.

Autonomy is earned through reversibility, not through model confidence.

Shifting Route 53 weights is a two-way door: if we are wrong, we shift back and the cost is latency and embarrassment. Failing over Aurora Global Database is a one-way door. It carries a recovery-point consequence that no amount of changing your mind reverses.

Those two must never sit in the same authority class, no matter how good the model gets. In our design the database failover is not automatable at any confidence level. The system's job there is to arrive with the evidence pre-assembled so a human can decide in two minutes instead of forty.

Underneath the policy gate sits one more layer worth adopting even if you take nothing else: ARC safety rules. Assertion and gating rules make states like "every cell disabled at once" structurally impossible at the API level. Even a fully compromised decision system cannot black-hole everything, because the platform itself refuses.

What the dry runs showed

Being precise about status, because it matters: this architecture has been designed and walked through manually against real historical incident data and a small number of staged scenarios. It is not running unattended in production. What follows is what the dry runs suggested, not what a year of production has proven.

The headline observation is that the full loop - detect, assemble, attribute, decide, mitigate - collapsed to a small number of minutes, against a previous state where the attribution argument alone could run far longer.

In the dry runs the full cycle came in at roughly two to three minutes, against the two hours we spent on the original bridge call. I want to be careful with that comparison. The two hours were people arguing under real pressure with incomplete information; the two minutes were a rehearsed scenario where the ground truth was already known. It is a directional result, not a benchmark, and I would not present it as one.

The mechanism of the improvement was not intelligence. It was that the evidence was already assembled. The eleven minutes we once spent finding out whether anything had been deployed recently became a field in a JSON document that was already on screen when the humans arrived.

Two second-order effects mattered more than the headline number. The argument changed shape - instead of debating what was happening, people debated whether the evidence supported the verdict, which is bounded, on-screen, and terminates. And somebody was finally willing to make the call, because a scored verdict with citations and a named dissenting view distributes that responsibility onto something reviewable. That is a social outcome, not a technical one, and it may be the most valuable thing the system does.

What went wrong

Four things, all of which changed the design.

Flapping during partial recovery. Early logic would have recommended shifting back the moment metrics improved, then out again when they dipped. Real recoveries are not monotonic. Fixed with hysteresis, a minimum dwell time, and by treating failback as a separate, more conservative decision than evacuation - you are moving toward a region that was recently sick.

The model produced a confident, wrong root cause. Asked why, it would generate a plausible narrative even from thin evidence. The fix was structural, not prompt engineering: split what and where - deterministic, gates action - from why - model-generated, advisory, never gating on its own. Mandatory dissenting evidence came out of this too, and it visibly reduced overconfident verdicts.

Route 53 geolocation does not evacuate cleanly. Geolocation routing pins users by where they are, not by where things are healthy. Shifting traffic away is not one weight change; it means reconsidering the geolocation policy itself, with real latency consequences for the users being moved. Model that cost explicitly, or the agent will happily recommend a shift that makes user experience worse in a different way.

Evidence volume nearly defeated the whole thing. The first version passed too much raw material to the model. Summarise and structure aggressively first. Raw logs are exactly the unstructured, attacker-influenced, high-volume material the design is supposed to keep out.

The reusable pattern

If you want to lift this, here is the order I had build it in. Each step is independently useful, which matters - you should get value before you get to the AI.

1. Out-of-region synthetic canaries first. Before any model. If your only health signal originates inside the region being assessed, everything downstream inherits that blind spot.
2. Make change events queryable in seconds. Deployments, flags, config, timestamped, reachable by an API call. Highest-value input to the "is it us" question, and most organisations cannot produce it under pressure.
3. Add deterministic outlier detection across regions. No model. Explainable, repeatable, auditable. Answers where.
4. Assemble the evidence bundle - structured JSON, all sources, one document. Stop here for a while. A large part of the pain disappears at this step, before any intelligence is involved at all.
5. Add the model for attribution only, with the schema above, in shadow mode. It writes to a log and touches nothing. Let it be wrong in private for a long time.
6. Score it against ground truth every time something real happens. This also keeps the path warm - an evacuation runbook exercised twice a year is a fallback path, and fallback paths fail when you finally need them.
7. Add the policy gate and human approval. The model recommends, a person clicks, the evidence is attached to the decision record.
8. Only then, and only for two-way doors, consider letting it act alone - behind quorum, confidence, and a hard capacity-headroom precondition.

The principle underneath all of it

Statistics tell you where. The model tells you why. Policy decides whether. And a human owns every door that only opens one way.

The version of this that fails is the one where an autonomous agent holds credentials and reasons its way to a traffic shift. That is not a resilience system. That is a faster, more confident version of the worst on-call engineer you have ever worked with. We have already seen how that goes when automation acts decisively on a signal it cannot validate.

The version that works is narrower and much less exciting. It assembles evidence nobody had time to assemble, offers an opinion with its reasoning attached and its own counter-evidence listed, and then gets out of the way of a decision that remains, deliberately, ours.

We did not need a system that could decide. We needed a system that could tell us whether it was us.

---

Thanks for reading. If you run active-active anywhere, I would genuinely like to hear how you answer the "is it us or is it them" question today.

Devjyoti Giri - AVP - AI Innovations

© 2026 Devjyoti Giri. All rights reserved. This article may not be reproduced or republished, in whole or in part, without the author's written permission. The views expressed are the author's own and do not necessarily represent those of any employer or client. All incident details have been generalised and contain no confidential or client-identifying information.