DEV Community: Omar Reda

What Changed? How We Turned Production Change Blindness into a less than 10-Minute Answer

Omar Reda — Fri, 03 Jul 2026 21:53:27 +0000

Most incidents aren't mysteries. They're changes. Here's how we built a service that captures every production change, attaches it to the system it touched, and turned the hardest question in incident response into a one-line query.

The 2 a.m. question

It's the middle of the night. A service's error rate is climbing, latency is fanning out, and a handful of engineers are now in a call that nobody wanted to join. The dashboards are red. The traces are interesting. But the question that actually decides how long this incident lasts isn't what is broken — it's what changed?

For a long time, that question was the most expensive part of our incident response. Not because the answer was unknowable, but because it was scattered. A code deployment lived in one tool. A feature-flag flip lived in another. A DNS or edge-config tweak lived with the team that owned the CDN. An infrastructure change sat in a cloud audit log nobody had open. A cache failover was a line in a console that no one thought to check until an hour in.

So we'd do the forensic thing. We'd ping teams: "Did anyone deploy?" "Did anyone touch the config?" "Was there a failover?" Each question was a context switch, a wait, and a maybe. Reconstructing a simple timeline of everything that changed around the moment things went wrong could eat the first hour of an incident — the most valuable hour there is.

The uncomfortable truth underneath all of this: we had excellent observability for state, and almost none for change. We could see what the system was doing. We couldn't see what we had done to it.

The insight: incidents are mostly changes wearing a costume

If you look back across a quarter of incidents, a pattern jumps out. The overwhelming majority are triggered by a change someone made — a deploy, a flag, a config push, an infra edit, a failover. The system rarely breaks on its own. Something changed, and the blast radius found its way to a user.

That reframes the whole problem. If most incidents are caused by a change, then the single highest-leverage thing you can do for detection and diagnosis is to capture every change, timestamp it, and put it exactly where engineers are already looking when something goes wrong.

Not in a separate "change log" tool that becomes another tab nobody opens. In the observability platform itself, attached to the specific service or entity the change touched. So that when an engineer pulls up an unhealthy service, the changes that hit it are right there on the timeline — overlaid on the latency and error graphs, in the same view, at the same timestamps.

That idea became Odin Eye.

Odin Eye in one sentence

Odin Eye is a change-intelligence pipeline: a set of lightweight collectors that watch every meaningful source of production change, a durable queue that fans them in, and a service that normalizes each change and pushes it into Dynatrace as an event attached to the affected entity — where it becomes queryable with DQL alongside everything else we monitor.

Here's the shape of it:

The architecture is deliberately boring in the best way. Every design choice optimizes for one thing: a change should never get lost on its way to becoming context.

How a change becomes context

1. Collectors at the edge of every source

Each source of change gets a small, single-purpose collector that knows how that source speaks. These are intentionally thin — they translate, they don't reason.

Deployments flow in with the richest detail of all — not just "a deploy happened," but individual deployment steps, and crucially the failed, aborted, and rolled-back ones. A rollback at 02:14 is often the single most important line in an incident timeline.

The CloudTrail collector is triggered whenever new audit logs land, fans out via notification, and filters the firehose down to the changes that actually matter — a curated allowlist of infrastructure-mutating actions (instance termination, database modification, security-group edits, secret changes, load-balancer deletion, and dozens more). For the most sensitive, destructive operations it additionally checks whether a human initiated the action, so routine automation doesn't drown out the changes a person made by hand. It even captures whether the action succeeded or failed, and attaches the error details when it didn't.

The Cloudflare collector watches for DNS and zone configuration changes and forwards them as they're applied.

The dynamic-configuration collector captures feature-flag and runtime-configuration changes from our internal configuration platform — the kind of "flip a value in production" change that is famously invisible and famously the cause of incidents.

The application-configuration collector emits an event whenever new application configs are promoted through the delivery pipeline, stamped with who triggered it and the exact commit.

The managed-cache collector polls for cache failover events on a short interval, filters out test failovers, and forwards the real ones.

The mobile-releases collector emits an event when a mobile app release ships — a new version submitted or promoted, or a staged rollout's percentage moved — so client-side changes land on the same timeline as everything server-side. Mobile is often exactly where a "nothing changed on our side" incident actually began.

The collectors live in different languages and runtimes (serverless functions, scheduled jobs, pipeline steps) because they live next to very different sources. What they share is an output contract.

2. A common contract and a durable fan-in

Every collector emits the same envelope:

{
  "source":    "<which kind of change>",
  "timestamp": "<when it happened, UTC>",
  "title":     "<human-readable summary>",
  "sender":    "odin-eye",
  "payload":   { "...who made the change, what changed, and any source-specific details..." }
}

…and drops it onto a single SQS queue. That queue is the backbone of the whole system. It decouples a dozen heterogeneous producers from one consumer, absorbs bursts, and — because it's durable — guarantees that a change emitted during the chaos of an incident doesn't evaporate if the consumer is briefly busy. Failed messages retry with backoff and ultimately land in a dead-letter queue rather than disappearing silently. No change gets dropped on the floor.

One field in that envelope does outsized work: sender is always odin-eye. Every record the pipeline produces carries the same sender tag, so a single filter pulls back everything we've ever captured — the who and the what of each change live in the payload, but the front door is one field:

// Everything Odin Eye has recorded, newest first
fetch events
| filter sender == "odin-eye"
| sort timestamp desc

3. Normalize, attribute, and attach

Odin Eye itself — a Go service — consumes the queue and does the part that makes all of this useful: it figures out which entity each change belongs to and attaches the change there.

This attribution step is the heart of the product. A change that isn't attached to anything is trivia. A change attached to the exact service that just started erroring is a diagnosis. Odin Eye routes each event by its source to a specialized mapper, resolves the target service — from a service tag carried on the event, or a known mapping — and emits a Dynatrace event (CUSTOM_CONFIGURATION for config-style changes, CUSTOM_DEPLOYMENT for deploys) with an entity selector that pins it to the right service, device, or environment.

It also guards against a subtle real-world problem: large estates accumulate duplicate or near-duplicate entities, and an event attached to the wrong copy is worse than no event. Odin Eye keeps a cache of resolved entities so changes land on the canonical one. The result is that a change shows up on the timeline of the thing it actually touched — not a lookalike, not a generic bucket.

Why "attach it to the entity" is the whole game

Plenty of organizations have a change log. Far fewer have change context in the place engineers already look. That difference is everything.

Because Odin Eye writes changes as first-class events on the entities themselves, two things become true at once:

They appear visually, in context. Open an unhealthy service and the change markers sit right on the latency and error charts. The eye does the correlation for you: the error rate elbows up at 02:14, and there's a deployment marker at 02:13. You're not cross-referencing five tools; you're reading one graph.
They become queryable with DQL. Every change is now data you can slice. A few illustrative queries (shapes, not exact schema):

   // Everything that changed on a given service in the last hour
   fetch events
   | filter event.kind == "CUSTOM_CONFIGURATION" or event.kind == "CUSTOM_DEPLOYMENT"
   | filter affected.entity == "the-service-in-trouble"
   | filter timestamp > now() - 1h
   | sort timestamp desc

   // Every failed, aborted, or rolled-back deployment across production today
   fetch events
   | filter event.kind == "CUSTOM_DEPLOYMENT"
   | filter deployment.state in ("FAILED", "ABORTED", "ROLLBACK")
   | summarize count(), by:{ affected.entity }

   // Did any Cloudflare or dynamic-config change land right before this problem opened?
   fetch events
   | filter source in ("cloudflare", "dynamic-config")
   | filter timestamp between problem.start - 10m and problem.start

The question that used to take an hour of pinging humans — "what changed around this service right before it broke?" — is now a query that returns in seconds. And it's the same query language engineers already use for metrics, logs, and traces, so there's nothing new to learn.

See it on the service screen

Because every change is a first-class event on the entity, it surfaces on that entity's own screen — no separate tool, no extra tab. On any service (or host, process, or environment), the timeline carries CUSTOM_DEPLOYMENT and CUSTOM_CONFIGURATION markers right next to its golden signals, and opening a marker reveals the change's payload: the title, the source, and the who and what behind it.

A service's event timeline: deployments and configuration changes, attributed by Odin Eye, overlaid on the same view engineers already watch during an incident.

That's the payoff made tangible. An engineer investigating a spike never has to leave the service screen to ask "what changed?" — the deployment marker at 02:13 and the configuration change beside it are already in frame, each one click away from the full detail of who did it and what it touched.

The impact: from forensics to a first-class signal

The headline result is blunt:

The investigative phase that used to be a scavenger hunt across tools and teams collapsed into reading a single, unified change timeline overlaid on the system's own health signals.

But the qualitative shifts matter just as much:

A shared language. "What changed?" stopped being a round of interrogations and became a shared artifact everyone in the incident channel could see at once.
The invisible became visible. Flag flips, Cloudflare changes, cache failovers, mobile releases, infra edits made by hand — the changes that used to be discovered late, by accident, an hour into the call — now announce themselves on the right timeline the moment they happen.
It's blameless by construction. Odin Eye tracks changes, not people. The framing is "a configuration changed and here's the blast radius," not "who broke production." That keeps the focus where it belongs: on understanding and recovering fast.

Built to grow: any new source is a small adapter

The most important property of Odin Eye isn't any single source — it's that any new source of change can be tracked simply by adding a new collector, with no change to the core. Adding the next one is cheap. Because every collector speaks the same envelope and the consumer routes by a source field into a config-driven mapper, onboarding a brand-new kind of change doesn't require re-architecting anything. You write a thin collector that emits the common contract, add a small mapping that says "this source attaches to that kind of entity," and the change starts showing up on timelines.

That extensibility is the difference between a one-off integration and a platform. As new systems are introduced — new internal platforms, new managed services, new categories of change — they slot into the same pipeline. The set of things we can answer "what changed?" about only ever grows.

Lessons for anyone building change intelligence

If you're considering something similar, a few things we'd underline:

Change is a first-class observability signal, not an afterthought. Treat it with the same seriousness as metrics, logs, and traces. The payoff during incidents is disproportionate.
Attribution is the product. Capturing changes is easy; attaching them to the right entity is what turns data into diagnosis. Invest there, including the unglamorous work of de-duplicating entities.
Decouple producers from the consumer with a durable queue. Heterogeneous sources, one contract, one resilient buffer. It makes the system both robust and easy to extend.
Meet engineers where they already are. The win wasn't a new tool; it was putting change inside the tool people already open at 2 a.m., queryable in the language they already speak.
Keep collectors dumb and the contract strict. Thin translators at the edges plus one well-defined envelope is what keeps a multi-source system from collapsing into special cases.

Closing

We didn't make production change less risky — change is the job. What we changed is our relationship to it. The next time a service goes red at 2 a.m., the first question is still "what changed?" — but now it's answered before the call has finished filling up, by a query instead of an investigation.

That's the quiet superpower of change intelligence: it doesn't prevent the incident, it gives you back the hour you used to spend just figuring out where to look.

Going Extreme With Extreme Programming

Omar Reda — Sun, 21 Feb 2021 16:27:29 +0000

In this article, we are going to discuss one of the software development agile methodologies which is Extreme Programming (XP).

For those who don't know enough information about agile, you can check this quick introduction before proceeding with the article.

What is Agile? 🤔

The word ‘agile’ means

Able to move your body quickly and easily.
Able to think quickly and clearly.

In business, ‘agile’ is used for describing ways of planning and doing work wherein it is understood that making changes as needed is an important part of the job. Business ‘agility’ means that a company is always in a position to take account of the market changes.

In software development, the term ‘agile’ is adapted to mean ‘the ability to respond to changes from Requirements, Technology and People.’

Content

Origins Of Extreme Programming
What Is Extreme Programming ?
When We Could Apply XP ?
Values & Practices
Extreme Programming Rules
Roles In XP

Origins Of Extreme Programming 🗺

Extreme Programming (XP) was first used on the Chrysler Comprehensive Compensation Program (C3), which was initiated in the mid-'90s and switched to an XP project when Kent Beck was brought on to the project to improve the performance of the system.

In October 1999, Kent Beck published Extreme Programming Explained, detailing the entire method for others, and shortly thereafter the official website was launched as well.

What Is Extreme Programming ? 🧐

XP is a set of engineering practices. Developers have to go beyond their capabilities while performing these practices. That’s where the “extreme” in the framework’s title comes from. To get a better understanding of these practices, we’ll start with describing XP’s lifecycle and the roles engaged in the process.

Similar to other Agile Methods of development, Extreme Programming aims to provide iterative and frequent small releases throughout the project, allowing both team members and customers to examine and review the project’s progress throughout the entire Software Development Life Cycle (SDLC).

When We Could Apply XP ? 😯

The general characteristics where XP is appropriate were described by Don Wells.

Dynamically changing software requirements.
Risks caused by fixed time projects using new technology.
Small, co-located extended development team
The technology you are using allows for automated unit and functional tests.

Values & Practices ⭐️🚀

Values 🤝

The five values of XP are communication, simplicity, feedback, courage, and respect and are described in more detail below.

Communication Everyone on a team works jointly at every stage of the project.
Simplicity Developers strive to write simple code bringing more value to a product, as it saves time and effort.
Feedback Team members deliver software frequently, get feedback about it, and improve a product according to the new requirements.
Courage Programmers objectively evaluate their own results without making excuses and are always ready to respond to changes.
Respect Every person assigned to a project contributes to a common goal.

Practices 🏋️‍♂️

The core of XP is the interconnected set of software development practices listed below. While it is possible to do these practices in isolation, many teams have found some practices reinforce the others and should be done in conjunction to fully eliminate the risks you often face in software development.

The XP Practices have changed a bit since they were initially introduced. The original twelve practices are listed below.

The Planning Game

Small Releases

Metaphor

Simple Design

Test-Drived Development

Refactoring

Pair Programming

Collective Ownership

Continuous Integration

40-hour week

On-site Customer

Coding Standard

I'm going to discuss some of these practices that are important to be understood.

Pair Programming 👨‍💻👩‍💻

This practice requires two programmers to work jointly on the same code. While the first developer focuses on writing, the other one reviews code, suggests improvements and fixes mistakes along the way. Such teamwork results in high-quality software and faster knowledge sharing but takes about 15 percent more time. In this regard, it’s more reasonable trying pair programming for long-term projects.

Test-Driven Development 🔍

Is it possible to write a clear code quickly? The answer is yes, according to XP practitioners. The quality of software derives from short development cycles that, in turn, allow for receiving frequent feedback. And valuable feedback comes from good testing. XP teams practice test-driven development technique (TDD) that entails writing an automated unit test before the code itself. According to this approach, every piece of code must pass the test to be released. So, software engineers thereby focus on writing code that can accomplish the needed function.

Code Refactoring 🛠

To deliver business value with well-designed software in every short iteration, XP teams also use refactoring. The goal of this technique is to continuously improve code. Refactoring is about removing redundancy, eliminating unnecessary functions, increasing code coherency, and at the same time decoupling elements. Keep your code clean and simple, so you can easily understand and modify it when required would be the advice of any XP team member.

Continuous Integration 🖇

Developers always keep the system fully integrated. XP teams take iterative development to another level because they commit code multiple times a day, which is also called continuous delivery. XP practitioners understand the importance of communication. Programmers discuss which parts of the code can be reused or shared. This way, they know exactly what functionality they need to develop. The policy of shared code helps eliminate integration problems. In addition, automated testing allows developers to detect and fix errors before deployment.

Extreme Programming Rules ♻️

Don Wells published the first XP rules in 1999 to counter claims that extreme programming doesn’t support activities that are necessary to software development, such as planning, managing, and designing. From planning to testing the software, follow these basic steps for each iteration.

Planning 📉

The first stage is when the customer meets the development team and presents the requirements in the form of user stories to describe the desired result. The team then estimates the stories and creates a release plan broken down into iterations needed to cover the required functionality part after part. If one or more of the stories can’t be estimated, so-called spikes can be introduced which means that further research is needed.

Managing 👨‍🏫

At this stage, the project manager will set the team up to succeed in this methodology. Everyone needs to work collaboratively and effectively communicate to avoid any slipups. This stage involves:

Creating an open workspace for your team.
Setting a sustainable pace (i.e. determining the right length for iterations).
Scheduling a daily standup meeting.
Measuring project velocity (the amount of work getting done on your project).
Reassigning work to avoid bottlenecks or knowledge loss.
Changing the rules if XP isn’t working perfectly for the team.

Designing 👨‍🎨

is actually a part of the planning process, but can be set apart to emphasize its importance. It’s related to one of the main XP values that we’ll discuss below — simplicity. A good design brings logic and structure to the system and allows to avoid unnecessary complexities and redundancies.

Coding 👨‍💻👩‍💻

Is the phase during which the actual code is created by implementing specific XP practices such as coding standards, pair programming, continuous integration, and collective code ownership (the entire list is described below).

Testing 💣

Is the core of extreme programming. It is the regular activity that involves both unit tests (automated testing to determine if the developed feature works properly) and acceptance tests (customer testing to verify that the overall system is created according to the initial requirements).

🧔👱‍♀️ Roles In XP 👨‍🦱👩‍🦰

Customers are expected to be heavily engaged in the development process by creating user stories, providing continuous feedback, and making all the necessary business decisions related to the project.
Programmers or developers are the team members that actually create the product. They are responsible for implementing user stories and conducting user tests (sometimes a separate Tester role is set apart). Since XP is usually associated with cross-functional teams, the skill set of such members can be different.
Trackers or managers link customers and developers. It’s not a required role and can be performed by one of the developers. These people organize the meetups, regulate discussions, and keep track of important progress KPIs.
Coaches can be included in the teams as mentors to help with understanding the XP practices. It’s usually an outside assistant or external consultant who is not involved in the development process, but has used XP before and so can help avoid mistakes.

Agility principles are becoming increasingly popular as they prove their effectiveness. Even though extreme programming is not the most widespread methodology, it offers a lot of sensible practices that can benefit software development and are worth considering for implementation in your projects.

References

That's it, I hope this article was beneficial to you and you enjoyed reading it. Furthermore, feel free to contact me on email from here Gamil if you want to ask about anything, and you can find me on LinkedIn, My Website.

Undergraduate experience in corporates and SMEs

Omar Reda — Tue, 03 Nov 2020 22:34:27 +0000

In this article, I will try to cover the experience you can get from joining a corporate or SME (small and medium-sized enterprises) company as an undergraduate software engineer, and the information provided in this article is based on my real personal experience in 3 different scale companies ... So let's get started.

Points of Comparison

Environment
Your Impact as Team Member
Learning Curve
Recognition
Perks
Income
Tips & Advices

Environment 🏘

Corporate: The first trivial thing about joining a corporate is the formal environment that meets the worldwide standards, from every aspect you will learn how to act in a formal way like sending and receiving formal emails, your outfit at work, the way you communicate with team members and managers, the fixed working hours, everything is organized and runs within the system of the company, also you will find a well-structured hierarchy for the company and a clear vision for your career path.

SMEs: The environment in SMEs is more friendly and more flexible than corporates, due to the small number of employees compared to a number of employees at corporates you can easily make many friends, in some SMEs the whole company could be friends and know each other, so you will surely find more flexible system than that of the corporates and that will make you act yourself more, and one more thing that is added as a bonus for working in SMEs is that you become more familiar with business and how it grows, and what are the market needs, and how to sell the company's idea or product to the customers.

Your Impact as Team Member 🚣🚣🚣‍♀️

Corporate: You will find yourself working by default on large scale projects that have an impact on society, you will work in a large team with different backgrounds and experience levels, and that will make you learn so much whether from technical or non-technical perspectives, and that's because employees in corporates always focus on getting promoted and reach higher levels in their career path, so they learn about many things and they take in their consideration the non-technical stuff especially the managerial content, as it will help them to manage and lead teams and get promoted faster.

SMEs: You will work on different scale projects some of them are large ones and some are not, and you will have more time to learn on your own as the intensity of the work won't be like the intensity in corporates, and of course you will have a team with different backgrounds and experience levels, but won't be as large as the corporates teams, and the variety will be also less than that you can find in corporates teams.

Learning Curve 📈

Corporate: Your learning curve is affected mainly by 2 points the first one we previously mentioned above when we talked about experienced team members and the technical or non-technical information you will learn from them, the second point which is mainly found in corporates is the access you got to enroll in a huge amount of online courses whether provided by the company you work for or any other online courses provider, and you will find this point in most of the corporates once you get your company domain email you will get the access on many courses and e-books offered online.

SMEs: Your learning curve is affected also by 2 points the first one is also the experience you got from working with the team members, and the second point is that you will search and learn about topics that you need at work if you aren't working in a company or within a team it will be difficult and confusing to select which topic should I learn about, and is that information is important or not, and this issue will be solved once you have a team you work with when you join a company which can provide you with guidance regarding what to learn about in order to finish your job and this guidance is very precious.

"ALWAYS KEEP YOUR LEARING CURVE RISING"

Recognition 🏆

Corporate: Working in a famous corporation could turn out to be very useful for your future career opportunities. Let’s face it! It looks good on a resume! Brand recognition has a heavy weight on anyone’s resume.

SMEs: It’s a bit harder to get this kind of recognition with SMEs, but not impossible. Usually, when they hit an important milestone, they do not become recognized brands instantly. It might take years until people will get to acknowledge them and automatically identify their activities.

Now The Fun Part 🎮🎯

Perks ✨

Corporate: Working in a corporation will ensure you some more significant perks. You will get a life and social insurance, gym membership, transportation reimbursements and many other benefits. In fact, corporations have the tendency to design their hiring positions around these advantages.

SMEs: You will feel very relaxed within the atmosphere of SMEs, The employees also enjoy many perks like free food and drinks, and a play area to release the work tension and enjoy your free time and make new friendships with the company's employees from different departments.

Income 💵

Corporate: All of the corporations have a standardized income that is brought to the average level of the market and sometimes higher. It may vary according to the market’s value and it can also go through some minor changes as you advance.

SMEs: The amount of resources is limited, so you cannot expect a high income. Moreover, sometimes it is going to be lower than the average income of the market, and also sometimes it is going to be very close to the average income of the market, and despite that it may sound a bit discouraging, but as an intern your main goal is not to get more money but to gain more knowledge, and as this knowledge increases the money will come definitely by time.

Small tips and advices to you as students

First, try to get the most out of any experience you get in while you are a student whether you joined a corporation or SME company, because what will you learn will be very beneficial to you when you start searching for a job as a full-time employee.
Second, try to make as many relationships as you can with people at work because this could help you to get a job in the future if you have proven your competence, and this will happen by getting a recommendation after you graduate from college to join back the company you worked at as intern.
Third, I mentioned this before in the income section, but I want to mention it again, don't search for money at the early beginning of your life, first learn and gain more knowledge and improve your resume with quality experiences, and trust me money will come to you after that.
Forth, don't you ever stop learning, this career is about learning new things every day to keep up with the market changes, so the most valuable weapon in your career journey is the ability to learn.