Most incidents aren't mysteries. They're changes. Here's how we built a service that captures every production change, attaches it to the system it touched, and turned the hardest question in incident response into a one-line query.
The 2 a.m. question
It's the middle of the night. A service's error rate is climbing, latency is fanning out, and a handful of engineers are now in a call that nobody wanted to join. The dashboards are red. The traces are interesting. But the question that actually decides how long this incident lasts isn't what is broken — it's what changed?
For a long time, that question was the most expensive part of our incident response. Not because the answer was unknowable, but because it was scattered. A code deployment lived in one tool. A feature-flag flip lived in another. A DNS or edge-config tweak lived with the team that owned the CDN. An infrastructure change sat in a cloud audit log nobody had open. A cache failover was a line in a console that no one thought to check until an hour in.
So we'd do the forensic thing. We'd ping teams: "Did anyone deploy?" "Did anyone touch the config?" "Was there a failover?" Each question was a context switch, a wait, and a maybe. Reconstructing a simple timeline of everything that changed around the moment things went wrong could eat the first hour of an incident — the most valuable hour there is.
The uncomfortable truth underneath all of this: we had excellent observability for state, and almost none for change. We could see what the system was doing. We couldn't see what we had done to it.
The insight: incidents are mostly changes wearing a costume
If you look back across a quarter of incidents, a pattern jumps out. The overwhelming majority are triggered by a change someone made — a deploy, a flag, a config push, an infra edit, a failover. The system rarely breaks on its own. Something changed, and the blast radius found its way to a user.
That reframes the whole problem. If most incidents are caused by a change, then the single highest-leverage thing you can do for detection and diagnosis is to capture every change, timestamp it, and put it exactly where engineers are already looking when something goes wrong.
Not in a separate "change log" tool that becomes another tab nobody opens. In the observability platform itself, attached to the specific service or entity the change touched. So that when an engineer pulls up an unhealthy service, the changes that hit it are right there on the timeline — overlaid on the latency and error graphs, in the same view, at the same timestamps.
That idea became Odin Eye.
Odin Eye in one sentence
Odin Eye is a change-intelligence pipeline: a set of lightweight collectors that watch every meaningful source of production change, a durable queue that fans them in, and a service that normalizes each change and pushes it into Dynatrace as an event attached to the affected entity — where it becomes queryable with DQL alongside everything else we monitor.
Here's the shape of it:
The architecture is deliberately boring in the best way. Every design choice optimizes for one thing: a change should never get lost on its way to becoming context.
How a change becomes context
1. Collectors at the edge of every source
Each source of change gets a small, single-purpose collector that knows how that source speaks. These are intentionally thin — they translate, they don't reason.
- Deployments flow in with the richest detail of all — not just "a deploy happened," but individual deployment steps, and crucially the failed, aborted, and rolled-back ones. A rollback at 02:14 is often the single most important line in an incident timeline.
- The CloudTrail collector is triggered whenever new audit logs land, fans out via notification, and filters the firehose down to the changes that actually matter — a curated allowlist of infrastructure-mutating actions (instance termination, database modification, security-group edits, secret changes, load-balancer deletion, and dozens more). For the most sensitive, destructive operations it additionally checks whether a human initiated the action, so routine automation doesn't drown out the changes a person made by hand. It even captures whether the action succeeded or failed, and attaches the error details when it didn't.
- The Cloudflare collector watches for DNS and zone configuration changes and forwards them as they're applied.
- The dynamic-configuration collector captures feature-flag and runtime-configuration changes from our internal configuration platform — the kind of "flip a value in production" change that is famously invisible and famously the cause of incidents.
- The application-configuration collector emits an event whenever new application configs are promoted through the delivery pipeline, stamped with who triggered it and the exact commit.
- The managed-cache collector polls for cache failover events on a short interval, filters out test failovers, and forwards the real ones.
- The mobile-releases collector emits an event when a mobile app release ships — a new version submitted or promoted, or a staged rollout's percentage moved — so client-side changes land on the same timeline as everything server-side. Mobile is often exactly where a "nothing changed on our side" incident actually began.
The collectors live in different languages and runtimes (serverless functions, scheduled jobs, pipeline steps) because they live next to very different sources. What they share is an output contract.
2. A common contract and a durable fan-in
Every collector emits the same envelope:
{
"source": "<which kind of change>",
"timestamp": "<when it happened, UTC>",
"title": "<human-readable summary>",
"sender": "odin-eye",
"payload": { "...who made the change, what changed, and any source-specific details..." }
}
…and drops it onto a single SQS queue. That queue is the backbone of the whole system. It decouples a dozen heterogeneous producers from one consumer, absorbs bursts, and — because it's durable — guarantees that a change emitted during the chaos of an incident doesn't evaporate if the consumer is briefly busy. Failed messages retry with backoff and ultimately land in a dead-letter queue rather than disappearing silently. No change gets dropped on the floor.
One field in that envelope does outsized work: sender is always odin-eye. Every record the pipeline produces carries the same sender tag, so a single filter pulls back everything we've ever captured — the who and the what of each change live in the payload, but the front door is one field:
// Everything Odin Eye has recorded, newest first
fetch events
| filter sender == "odin-eye"
| sort timestamp desc
3. Normalize, attribute, and attach
Odin Eye itself — a Go service — consumes the queue and does the part that makes all of this useful: it figures out which entity each change belongs to and attaches the change there.
This attribution step is the heart of the product. A change that isn't attached to anything is trivia. A change attached to the exact service that just started erroring is a diagnosis. Odin Eye routes each event by its source to a specialized mapper, resolves the target service — from a service tag carried on the event, or a known mapping — and emits a Dynatrace event (CUSTOM_CONFIGURATION for config-style changes, CUSTOM_DEPLOYMENT for deploys) with an entity selector that pins it to the right service, device, or environment.
It also guards against a subtle real-world problem: large estates accumulate duplicate or near-duplicate entities, and an event attached to the wrong copy is worse than no event. Odin Eye keeps a cache of resolved entities so changes land on the canonical one. The result is that a change shows up on the timeline of the thing it actually touched — not a lookalike, not a generic bucket.
Why "attach it to the entity" is the whole game
Plenty of organizations have a change log. Far fewer have change context in the place engineers already look. That difference is everything.
Because Odin Eye writes changes as first-class events on the entities themselves, two things become true at once:
They appear visually, in context. Open an unhealthy service and the change markers sit right on the latency and error charts. The eye does the correlation for you: the error rate elbows up at 02:14, and there's a deployment marker at 02:13. You're not cross-referencing five tools; you're reading one graph.
They become queryable with DQL. Every change is now data you can slice. A few illustrative queries (shapes, not exact schema):
// Everything that changed on a given service in the last hour
fetch events
| filter event.kind == "CUSTOM_CONFIGURATION" or event.kind == "CUSTOM_DEPLOYMENT"
| filter affected.entity == "the-service-in-trouble"
| filter timestamp > now() - 1h
| sort timestamp desc
// Every failed, aborted, or rolled-back deployment across production today
fetch events
| filter event.kind == "CUSTOM_DEPLOYMENT"
| filter deployment.state in ("FAILED", "ABORTED", "ROLLBACK")
| summarize count(), by:{ affected.entity }
// Did any Cloudflare or dynamic-config change land right before this problem opened?
fetch events
| filter source in ("cloudflare", "dynamic-config")
| filter timestamp between problem.start - 10m and problem.start
The question that used to take an hour of pinging humans — "what changed around this service right before it broke?" — is now a query that returns in seconds. And it's the same query language engineers already use for metrics, logs, and traces, so there's nothing new to learn.
See it on the service screen
Because every change is a first-class event on the entity, it surfaces on that entity's own screen — no separate tool, no extra tab. On any service (or host, process, or environment), the timeline carries CUSTOM_DEPLOYMENT and CUSTOM_CONFIGURATION markers right next to its golden signals, and opening a marker reveals the change's payload: the title, the source, and the who and what behind it.
A service's event timeline: deployments and configuration changes, attributed by Odin Eye, overlaid on the same view engineers already watch during an incident.
That's the payoff made tangible. An engineer investigating a spike never has to leave the service screen to ask "what changed?" — the deployment marker at 02:13 and the configuration change beside it are already in frame, each one click away from the full detail of who did it and what it touched.
The impact: from forensics to a first-class signal
The headline result is blunt:
The investigative phase that used to be a scavenger hunt across tools and teams collapsed into reading a single, unified change timeline overlaid on the system's own health signals.
But the qualitative shifts matter just as much:
- A shared language. "What changed?" stopped being a round of interrogations and became a shared artifact everyone in the incident channel could see at once.
- The invisible became visible. Flag flips, Cloudflare changes, cache failovers, mobile releases, infra edits made by hand — the changes that used to be discovered late, by accident, an hour into the call — now announce themselves on the right timeline the moment they happen.
- It's blameless by construction. Odin Eye tracks changes, not people. The framing is "a configuration changed and here's the blast radius," not "who broke production." That keeps the focus where it belongs: on understanding and recovering fast.
Built to grow: any new source is a small adapter
The most important property of Odin Eye isn't any single source — it's that any new source of change can be tracked simply by adding a new collector, with no change to the core. Adding the next one is cheap. Because every collector speaks the same envelope and the consumer routes by a source field into a config-driven mapper, onboarding a brand-new kind of change doesn't require re-architecting anything. You write a thin collector that emits the common contract, add a small mapping that says "this source attaches to that kind of entity," and the change starts showing up on timelines.
That extensibility is the difference between a one-off integration and a platform. As new systems are introduced — new internal platforms, new managed services, new categories of change — they slot into the same pipeline. The set of things we can answer "what changed?" about only ever grows.
Lessons for anyone building change intelligence
If you're considering something similar, a few things we'd underline:
- Change is a first-class observability signal, not an afterthought. Treat it with the same seriousness as metrics, logs, and traces. The payoff during incidents is disproportionate.
- Attribution is the product. Capturing changes is easy; attaching them to the right entity is what turns data into diagnosis. Invest there, including the unglamorous work of de-duplicating entities.
- Decouple producers from the consumer with a durable queue. Heterogeneous sources, one contract, one resilient buffer. It makes the system both robust and easy to extend.
- Meet engineers where they already are. The win wasn't a new tool; it was putting change inside the tool people already open at 2 a.m., queryable in the language they already speak.
- Keep collectors dumb and the contract strict. Thin translators at the edges plus one well-defined envelope is what keeps a multi-source system from collapsing into special cases.
Closing
We didn't make production change less risky — change is the job. What we changed is our relationship to it. The next time a service goes red at 2 a.m., the first question is still "what changed?" — but now it's answered before the call has finished filling up, by a query instead of an investigation.
That's the quiet superpower of change intelligence: it doesn't prevent the incident, it gives you back the hour you used to spend just figuring out where to look.
Top comments (2)
Great write-up! The detail that stood out to me is the CloudTrail collector filtering for human-initiated actions on destructive operations, that's exactly the signal-to-noise problem that kills most change-log attempts. Also, capturing failed/aborted/rolled-back deploys is underrated; a rollback at 02:14 really is often the whole story.
Thanks Ramez! Cloudtrail is by default scattered because in most cases you have multiple accounts and in order to create a collector to gather these. data you need to bring all the accounts logs together in single place then add the collector on top of it. At the end, the goal is to bind any cloud event to the respective entity in realtime and that's what odin-eye provided.