DEV Community: Nico Krijnen

Programming as Theory Building

Nico Krijnen — Mon, 07 Sep 2026 09:29:29 +0000

Picture, you join a new team working on a big system. Everybody who knew anything has left, either to find greener grass or to enjoy a well deserved pension. You and the team struggle to build new features for the system or to adapt functionality to match changes in legislation. Not to mention the trouble it is to figure out what to fix when things go wrong. At the same time, the business that you support is screaming for innovation and pushing for more and more changes.

Recognize this situation? Ever experienced it yourself?

A world full of legacy systems

“Legacy. What is a legacy? It’s planting seeds in a garden you never get to see.”

– Lin-Manuel Miranda, “Hamilton”

Legacy, the thing that you are remembered for, typically the word has a positive meaning… how come that in tech the word “Legacy” has such a bad connotation? When we call out a legacy system, we usually mean: code without tests (Michael Feathers) or code you “got” from somebody else, or code that you’re scared to touch.

However, there is a reason these legacy systems are still around. In almost all cases, that system still brings in money or is somehow still valuable. If it did not bring any value anymore, wouldn’t it be decommissioned? There must be something in these systems that makes them survive, where other systems did not.

How systems become “Legacy”

So legacy systems are those that have become hard or scary to change. In my experience, that not because something is wrong with the code or technology. The major contributing factor is usually that the knowledge about the system has left the organization. And then I don’t mean the documentation, but the people that built, maintained and ran the system. When those people are gone, you know that nobody else is going to be happy touching that thing.

The value of software

Code is like a mapping of desired real world behavior to a program that can be executed by a machine. So where is the value of a system, is that in that code?

Over the past years I’ve talked a lot at conferences and training workshops about the value of code. How I used to explain it is that the value of software is to be easy to change. It’s in the name: soft = flexible. Which contrasts with hardware, which is quite hard to change once you’ve soldered everything together. Pretty much all the things that we do to improve our way of writing code boil down to making the code easier to understand for the next person working on it (or yourself in 6 months). Our primary objective for building maintainable systems is to make code easy to change. In a sense, to make the “legacy” that we leave behind be a positive one.

Then, about a year ago, I got drawn deep into the rabbit hole by this video on The Value of Source Code by Philomatics. It stated that the value of a software system does not have a lot to doe with the actual source code. Instead, the real value is the ability to quickly change the system, and that the key thing that you need for that is people with a “ mental theory” of why the system works the way it does. Without that theory, no person is able to confidently make any changes.

For me, this way of looking at it immediately made sense. When I look back at observations I made on the numerous software systems I worked on over the years, everything just fell into place. I also realized that this fundamental shift in thinking about software systems has wide implications. Or as Maria Rey puts it so adequately:

“This theory is learned through immersion, through building and debugging and negotiating constraints. You cannot download it; you acquire it by living with the system until the system inhabits you. This is an uncomfortable idea, because as soon as we say this, the economic picture around code trembles: the most valuable part of the program, the part that determines whether it can be safely changed, does not appear on the balance sheet. It is invisible, embodied, and therefore precarious.”

– Maria Rey

That all brought me down the track of diving into this topic, talking about it at conferences and writing blogs like these about it.

Programming as Theory Building

This is not some new AI-era insight either. Peter Naur, the N from BNF and a Turing Award winner for his work on Algol 60, already wrote about this in 1985 in his article “Programming as Theory Building”.

Naur’s point is simple, and once you see it you cannot really unsee it. A program is not just the source code sitting in a repo. The real thing is the theory people build in their heads while working on it. Why it works that way. Why it was designed that way. What tradeoffs were made. What parts are safe to touch and what parts will bite you.

What made that argument so sharp is the world he was arguing against. The dominant view back then treated programming as a kind of production process. Get the requirements in, have programmers translate them, get code and documentation out. In that picture, the human beings in the middle are mostly interchangeable. If the specs are good enough, anyone should be able to pick up the work and continue.

Input       =  Requirements & Observing real world
Programmers  =  Replaceable "machines"
Output      =  Code & Docs

What Naur saw, and what I keep seeing on teams now, is that this is just not how software behaves in practice. Programs are human constructions. The important part is not only the artifact, but the understanding wrapped around it. Specs help. Docs help. Tests help. But none of them carry the full story on their own. They only really work when people still understand what problem the system is solving and why earlier decisions were made.

Input       =  Observing real world, situational experience
Programmers  =  Build a mental theory
Output      =  Theory: mapping between real world & code
Code        =  Side-product

This was not just philosophy for Naur. He got there by watching real projects. And that part matters, because the production view sounds reasonable right up until you have to live inside an actual system. On paper, changing software looks cheap. You edit some text. You run the build. You ship.

In real teams, the expensive part is figuring out whether the change is actually safe, whether you really understood the situation, and whether you understand what side effects will occur.

When he applied the Theory Building view to his observations, the actual outcomes matched exactly with what the view would predict. Software transferred from one team to another with code, docs AND human consulting resulted in success. A similar transfer took place a while later. This time only the code and docs got transferred WITHOUT human contact between the teams. The result: an unmaintainable system.

The effect was not only visible in the ability to change a system, he also saw it around the ability to deal with unexpected situations. I advise you to read Naur’s original text for more details on what he described.

As I already mentioned, to me this immediately clicked. Having people on the team that deeply understand a system makes a huge difference for how effective the team is going to be at adapting it and dealing with incidents. I’m sure that anyone with sufficient experience on software projects will feel the same.

Can’t we just document the theory?

Ok, so if that mental theory is so important, why don’t we just write it down in documentation?

Seems like the obvious answer. The trouble is: you can’t.

Writing down your mental model of a system is the same as asking someone to write down how to ride a bike and then asking someone else who never rode a bike to learn it from the text. This is the nature of Tacit Knowledge, it is learned by doing, not by reading. When you learn cycling, you “build a feel” for the balance. Reading an essay on how gravity works is not going to help you. The same goes for someone learning the finer touches of playing a violin, or the intuition on how to react that’s built up by basket ball players.

	Tacit Knowledge	Explicit Knowledge
Definition	Personal, experiential know-how that’s hard to articulate.	Formal, documented knowledge that can be written down, codified, or stored.
How it’s gained	Through practice, intuition, trial-and-error, observation.	Through reading, lectures, databases, manuals.
How it’s shared	Mentoring, storytelling, shadowing, demonstration, practice.	Books, reports, procedures, checklists, training materials.
Examples	Riding a bike, a chef’s “feel” for seasoning, a leader’s intuition in a crisis.	Recipes, standard operating procedures (SOPs), user manuals, financial reports.
Transfer difficulty	Hard to capture in words and usually requires personal interaction.	Relatively easy. You can just copy, store, or transmit the written or digital record.
Value	Enables creativity, innovation, and expert judgment.	Provides structure, consistency, and scalability.

The theory of a system is tacit knowledge. This is “The Art” of doing something.

The code, tests and the docs are explicit knowledge. These are “the instructions” for doing something.

“The Art” of doing something is not something you can write down. It must be passed person-to-person and learnt through experience and social interaction.

“The conclusion seems inescapable that at least with certain kinds of large programs, the continued adaptation, modification, and correction of errors in them, is essentially dependent on a certain kind of knowledge possessed by a group of programmers who are closely and continuously connected with them.”

– Peter Naur, Programming as Theory Building

So what survived from the age of Naur?

Ok, nice theory from the 80s, why doesn’t anybody seem to talk about this and why does it seem like our industry is just completely ignoring these quite fundamental insights?

What I find interesting is that our industry never fully chose one side. We kept a lot of the formalist instinct, and that is not a bad thing. We like tests, contracts, static analysis, CI, typed code, all of that. At the same time, a lot of the ways good teams actually work came much closer to Naur. Pairing. Reviews. Domain conversations. Keeping developers close to incidents and operations. All of that only makes sense if human understanding is part of the product.

Looking back, our industry never fully chose one side. We kept a lot of the formalist instinct, and that is not a bad thing. We like tests, contracts, static analysis, CI, typed code, all of that. This is the production view, though less mathematical than its original advocates envisioned: an emphasis on rigorous tools and processes, from static analysis and type systems to automated testing and CI/CD pipelines.

At the same time, a lot of the ways good teams actually work came much closer to Naur. Pairing. Reviews. Domain conversations. Keeping developers close to incidents and operations. These practices recognize that human and social factors are critical. Agile, DevOps, Domain-Driven Design, sociotechnical engineering, all reflect that insight. Straight from the Agile Manifesto: "documentation is secondary to working software and team communication", Alistair Cockburn even included Naur's article in his book "Agile Software Development."

So some of the thoughts of Naur made it through, but clearly not enough. After all, we're still looking at software creation as a production pipeline (we literally call it that) and we're seeing way too many places where teams operate as "feature factories".

What this means in the age of AI

Ok, so the obvious pushback is that this was the 80s and we now live in a very different world. AI can regenerate code quickly, so readable code matters less nowadays, right?!

We live in a world that is under constant change. The real value of software is not preserving source files, but preserving the ability to adapt behavior when the world changes (which it will). AI helps us immensely in producing code quicker, but it does not remove the need for a human theory of why the system exists, what tradeoffs were made and what must not break.

Ok, all fine, but we are doing spec-based-development now. So all these things are nicely written down in specs and if the code gets ugly, we can just regenerate the complete system from those spec, right?!

Well… the trouble is that the mental theory, the “art” of the system, cannot be fully written down. Specs are incomplete artifacts too. Just getting all the relevant expert understanding from the right people into a spec is hard in the first place. And even then, they still need interpretation. Code has a strict syntax. Specs are written in natural language, and natural language is inherently ambiguous. The hardest part is usually not writing the code from the spec, but knowing which spec is wrong, stale, underspecified, politically negotiated, or missing.

Rewriting a codebase every three months is only safe if the team still understands the domain and the consequences of change. AI makes syntax cheap. It does not make judgment cheap.

If anything, faster code generation increases the premium on shared understanding, because a team can now produce more change than it can truly comprehend. I’m hearing some people referring to this as “cognitive debt”, and if we keep using AI the way we are doing now, that debt is going to rise at an alarming rate.

Knowledge walking out the door

It is clear that the survival of human mental models in a team is way more important than we treat them in today’s industry. Where Naur warned us that programmers are not replaceable cogs in a machine, the current industry trend is to treat developers as just that: code monkeys that can be replaced with cheaper AI.

At the same time, developers job-hopping every few years has been a reality for quite some time. I was curious what causes lie behind that, to help figure out what we can do about this. What I found is quite shocking.

I couldn’t find any definitive source, but taking numbers from various surveys and reports the following causes jump out:

Career growth / Desire for new challenges To me this feels like the most logical and common reason I hear for people switching jobs. With percentages between 10 and 20, reality seems that this accounts for a fairly low percentage.
Poor leadership These have me rather worried. Management and poor leadership being at the same size as career growth. Clearly a topic that requires attention.
Burnout / Overwork The numbers for this one vary wildly, but just as the previous one. That this one is even in here is not ok.
Aging / Retirement Fairly low numbers here for the moment, but this is a demographic train coming our way, and no way to stop it. A lot of knowledge about the systems running today are in the heads of the more experienced engineers in our organizations. When they leave, their knowledge leaves with them, and as we already concluded, no amount of documentation is going to prevent that.

How legacy feeds itself

The most intriguing numbers are those for Technical debt / Legacy as reason for developer turnover. These numbers are rather high. Which, to be honest makes total sense. If a system is already hard to work with, the people on the team feel that pain every day. Every change takes longer than it should. Every incident takes too much energy to solve. And the less confidence people have in the system, the more stressful the work becomes.

That is where the vicious circle kicks in. Legacy systems drive people away. When people leave, their theory of the system leaves with them. With less theory on the team, the system becomes harder to change. A harder system drives more people away. And the cycle repeats, each turn making the system a little more opaque, a little more fragile, a little more “legacy.”

Legacy, then, is not just technical decay. It is a self-reinforcing loop of knowledge loss. It is what remains when the system outlives the shared theory that once made it changeable.

And that also explains something important. When engineers avoid touching certain parts of a system, it is usually not because they are lazy or unwilling. It is because nobody can still explain with confidence what those parts are really doing, why they were built that way, and what might break when you change them.

This vicious circle is already bad enough on its own. But there are forces outside of it that are accelerating the loss of theory even further.

Why keeping knowledge alive is getting harder

That leaves me with a bigger question. With all the tools, all the process improvements, all the focus on documentation and collaboration, why does knowledge continuity seem more fragile now instead of less?

I think at least part of the answer lies in two forces that are reshaping the way we build software. Remote work has removed many of the small, informal moments where tacit knowledge used to spread: overheard conversations, quick questions, spontaneous pairing, the little bits of context you absorb just by being around people who know the system well. At the same time, AI-assisted delivery lets us produce more code, faster, while reducing how much understanding humans are forced to build along the way.

The theory lives in human minds. Our tools increasingly let produce vast amounts of code without building understanding, what exactly are we optimizing for? Fast delivery now, unmaintainable systems in two years? Is that the bargain we are making without realizing it?

Enough about the problem at hand. I’m a practical guy, so I propose we don’t let that happen. AI is a great tool, and at the same time we have some serious work ahead of us to prevent uncontrolled cognitive debt.

Enough there to dig into for the next post.

MCP Apps - Finally a UI paradigm that speaks the language of intent

Nico Krijnen — Sat, 21 Mar 2026 00:00:00 +0000

Earlier this month I spoke at the Dutch AI Conference. After my own talk, I attended a talk by Glenn Reyes about MCP-UI. While listening, something profound struck me, something that I think not many others may have realized. So I figured this insight is worth writing down and sharing.

Glenn argued that due to the text-based nature of our interactions with LLMs and agents, we are throwing away decades of UX expertise. MCP-UI has been an attempt to change that, allowing interactive visual components to be provided through MCP (Model Context Protocol), an open standard that gives AI assistants the ability to connect to external tools and take real action, not just generate text. Development of MCP-UI went quite fast and currently, interactive HTML-based interfaces are an official part of the MCP Apps standard.

MCP Apps solve the how do we make agent UIs better problem. But sitting in that audience, a different question grabbed me: how do you design what those agent interactions should be? Because building a beautiful MCP App that solves the wrong problem is still solving the wrong problem.

That question has been answered before. Not by the AI community, but by a different group of practitioners who’ve spent the past decade perfecting collaborative techniques for exactly this: understanding complex behavior, modeling intent, and building shared language between the people who know the domain and the people who write the code. These two worlds have barely met. So let’s see if we can bring these communities together!

Troublesome state

UIs have always been built around state. Lists, forms, search bars, toggles, modals, loading spinners, all of these require state to be managed. Many frameworks have been built in attempts to tame state complexity (Redux, MobX, signals), but the fundamental model stays the same: UI reacts to data, components reflect state. Even with these frameworks, UI code is still hard to reason about, hard to test, and far away from the language of the business.

Language of the business

What do I mean with “language of the business”, and why is that important? Developers (and their coding agents) write the code. To build the right thing, they need to understand the what and why behind what they are building or changing. Otherwise they solve the wrong problem…

The understanding that is needed typically lives with non-technical people: the users of the system or the people who know how things really work. They don’t speak your technical language and they shouldn’t have to. What you need is to build deep understanding by using a shared non-technical language. More on that later…

Intent

What struck me during Glenn’s talk was that MCP-UI and MCP Apps don’t manage state in the usual way that we’ve been doing that for frontend components. Instead, the primitive is the message or tool call, not the component. The UI becomes a surface for expressing goals, not navigating menus. Instead of “fill form, submit, handle error state”, the interaction is “user states intent, agent invokes tool, result surfaces”.

In MCP Apps, every meaningful interaction is a named action with context. Things like: “SearchProducts”, “PlaceOrder”, “SummarizeDocument”. Sounds familiar? That’s because it’s very close to the commands used in EventStorming. Think of an MCP tool call like agent’s way of issuing a command to the outside world. A command is simply a request to do something. It captures what someone wants to happen in a name and with the information needed to do it. Commands model intent directly, they express “I want this to happen” rather than “set this field to that value”. They carry meaning, not mechanics.

A command like “ApproveExpenseReport” says why something needs to happen, not just how the data changes. When modeled correctly, commands speaks the language of the business, not the language of the database.

Designing Intent-based AI interactions

Let’s start with a concrete example. Imagine you’re building an MCP App for a warehouse operations team. Order pickers need to collect items from shelves and get them to the loading dock. A picker says “plan my picks for the next batch” and the agent needs to respond with an optimized walking route through the warehouse. You can’t describe a spatial route in text. This domain needs a visual interface. So how do you design it?

Instead of the usual: “let the dev team dream up how this will work”, let’s do some collaborative modeling and build some deeper understanding.

You get the warehouse team into the room and run an EventStorming session. You start by asking: “What are all the things that happen during a pick run?” Sticky notes go up: PickOrderReceived, RouteOptimized, ItemPicked, CartFull, BatchCompleted. Standard stuff, roughly what you expected.

Then the warehouse supervisor adds one: ObstructionReported. You ask what that means. Turns out aisle blockages happen every day: pallet spills, restocking crews. When a planned route is blocked, pickers currently find out by walking into the obstruction and rerouting on the fly, or by shouting across the floor. This was nowhere in the requirements.

You keep going. ColdChainTimerStarted appears: frozen items have a 12-minute window from cold storage to the insulated shipping container. HazmatSegregationViolationDetected: cleaning chemicals and food products can never share a cart. The process is far richer than anyone expected. Commands pop up: PlanPickRoute, PickItem, ReportObstruction, SplitCart, SubstituteItem. Each maps directly to an MCP tool definition, named in the language of the warehouse team.

Then something interesting happens. When the supervisor explains how obstruction rerouting works, sticky notes aren’t enough. She grabs a marker and draws the floor plan on the whiteboard next to the event timeline. Aisles, shelving sections, cold storage tucked in the back corner, the loading dock. She marks where today’s restocking is blocking aisle 6 and draws the detour path a picker would take. Someone grabs colored markers: orange zones for hazmat storage, a blue snowflake near cold storage, red X’s for blockages.

You zoom in on the tricky parts with Example Mapping. You pick PlanPickRoute and explore the rules:

Rule: cold chain : “Given a batch that includes frozen items, when the route is planned, then cold storage items must be picked last to stay within the 12-minute window.”
Rule: hazmat segregation : “Given a batch with both cleaning chemicals and food products, when the route is planned, then the cart must be split and items picked in separate runs.”
Edge case: “What if the only route to cold storage is blocked?” The supervisor pauses. “Good question. I think… the picker skips the cold items and comes back for them after the obstruction clears? We’ve never actually decided that.”

The sketch on the wall becomes the MCP App: a floor plan with aisles, an optimized route line, obstructions in red, color-coded item categories. The events and commands from the timeline map to tool definitions. The visual the supervisor drew to explain her world is the visual the picker will use every day. The modeling session didn’t just uncover what the interactions should be, it produced the UI too.

Which techniques did you say?

Understanding how things work and building systems that fit into that is something we’ve been doing for a long time already, so it’s no surprise that there are plenty of tools that we can use to do that effectively. I think the following 4 have a an exceptionally good use for designing AI agent interactions, especially when using MCP and MCP Apps.

EventStorming

This collaborative modeling technique helps you understand system behavior. It maps business processes as a storyline of domain events, triggered by commands from actors. This is exactly how an AI-driven UI interaction flows.

With EventStorming, the initial focus is only on events: facts that have happened. That is not by accident. When you think primarily about command and actions, it is easy to fall into the trap of modeling how things work. They model intention: things may still end up different than intended. The big benefit of starting with events is that they force you to focus on desired outcomes, not on how you get there. That gives you the freedom to find better ways to do things, to find a simpler command trigger for an event, still arriving at the same outcome. Both EventStorming and Event Modeling also include sketching UI interactions directly on the wall: wireframes, screens, visual cues that show what the user sees at each step. For MCP Apps, those sketches naturally become the design for custom visual components.

Domain Storytelling

More suitable for interview style, domain storytelling captures how actors work together to achieve goals through concrete work stories. These stories help you understand how the domain actually works. Again, this is a very natural fit for mapping the intent-flows of AI agent and MCP interactions, told in the language of the domain. When you use this technique, you draw an easy to understand visual story of what the person you’re talking to is explaining. The key here is the quick feedback and validation. If you misunderstood something, they will instantly point out the mistake in your drawing or point out edge cases not yet covered.

Event Modeling

Very similar to EventStorming, also modeling the intent (commands), the facts (events) and the views (read models). This feels like an almost perfect conceptual match for MCP tool calls, conversation history and context windows.Event Modeling organizes parts of the flow as vertical slices, each slice a complete flow from intent to state change to updated view. The key being that each slice is an isolated piece of behavior that can be implemented as one unit of work. This makes estimation and planning a breeze. As each slice is isolated, it can also be implemented in parallel with other slices without waiting for the rest of the system to be done. Productivity-wise, this is rather interesting now that we can put many AI agents to work in parallel to each build a piece of a system. Slices also map nicely to how MCP Apps work, each slice essentially one tool interaction: what the user asks for, what happens and what they see as a result.

Example Mapping

The more concrete you can make it, the easier it is to spot edge cases or incorrect assumption. Example Mapping is an excellent tool to quickly explore the rules around a specific command or intent: given this situation, when this happens, then this should be the outcome. Perfect to capture exactly what an MCP tool should do and how it should respond in different scenarios.

Nowadays, you can take the raw output of an example mapping session, give it to your coding assistant and turn it into a test-suite. And with good behavioral tests and the right architecture guidelines, your coding agent will have a breeze creating the implementation for you too.

What I find most interesting is how all of these techniques are not just similar to how MCP tools work, conceptually they map almost one-to-one.

The real superpower

All these are not just modeling tools, they are collaborative discovery tools. They work so well at uncovering complex behavior because they put experts and developers in the same room : working side by side without any intermediaries. This direct contact creates shared understanding before a single line of code is written. It also creates that insight very quickly and much deeper than with written specs. Result: high focus during implementation, with very few questions or points of confusion. And when the inevitable unforeseen edge case emerges, the personal connections that developers have built up with the experts mean that they know exactly who to call.

Contrast that with the traditional approach: BA → Specs → PO → Story → Tasks → Dev/AI → Code. Where every handoff is a translation loss: the telephone-game effect in action. By the time the requirements reach the implementor, they are abstract, ambiguous, full of assumptions and often wrong in ways nobody notices until late. The typical result is expensive rework, long feedback cycles and clarity arriving at the worst possible moment.

The language aspect matters even more for our interaction with AI agents, where the “UI” is largely language. If you get the intent vocabulary wrong, the whole thing feels broken to it’s users. This focus on precise, shared language has its roots in Domain-Driven Design and is now more important than ever. Getting the vocabulary right is no longer just a modeling concern, it’s a usability concern. I guess that deserves a blog of its own.

What this all means for you

AI agents are becoming part of how we build, design and use software. Intent-based interaction through MCP fit naturally to command and event based design tools. Collaborative modeling techniques like EventStorming, Domain Storytelling, Event Modeling, and Example Mapping have been perfected over the past decade for exactly this kind of problem: understanding what a system should do by working directly with the people who know. If you want to build MCP Apps that actually solve the right problems, start by getting the right people in the room.

If you’d like a hand getting started, whether it’s facilitating a good modeling session or designing MCP tool interactions that speak your domain’s language, reach out. I’d love to help.

Traceable Tests

Nico Krijnen — Fri, 27 Jan 2023 00:00:00 +0000

Originally published on Luminis on 27 Jan 2023.

Have you ever spent hours or days trying to figure out why some API test is failing? Whenever something like that happens to me, my immediate thought is: what could have helped me find this problem faster?

Unit tests are much better at this, when they fail, you just run them again locally, set a breakpoint if you have to, and find out what failed. But with tests that run against your API (yes, you need those too!) it can be much harder to figure out exactly what code got executed. There are all sorts of boundaries that make this hard, especially when those tests run as part of your pipeline (yes, you need to do that!). You can’t just set a breakpoint in your pipeline and even if you have logging in your backend, how do you know which log entries resulted from which test scenario?

I’ve been playing with the idea of linking backend logs to tests for some time. Years back, I had a plan to add the name of the test to a custom HTTP header when calling the API from tests. In the backend I could add that value to the MDC context that we attach to log entries. The result would be that for each log entry, you could see which test triggered it.

Déjà vu

Somehow I never got around to implementing that idea. Last year though, I helped out a team working on an API-only service that ran into the very same problem. Their system is comprised of a set of microservices and some of the API calls go through several of these services to handle the request. Next to unit tests, one of their other automated validations is a regression test suite that runs against the API to validate expected behavior.

So far, so good, and that test suite was really valuable. It allowed the team to quickly implement new functionality without spending a lot of time checking that existing functionality kept working.

You know what’s coming now… whenever a test would fail, it was a nightmare to figure out why.

Luckily, I had just finished hooking up all the services to an Elastic Cloud environment, with APM-powered tracing and all the good stuff. The traces gave excellent insight into what happened inside and between the microservices. However, it was still really hard to link a specific test failure to a set of traces.

OpenTelemetry FTW

I still had that idea from before in my head, and when I saw the team struggling with these failing tests, a new idea popped in: what if I linked the traces to tests?

The typical method to pass trace IDs across service calls is by using a trace header. OpenTelemetry provides the traceparent header for that, and the Elastic APM agent being used in the services picks that up out-of-the-box.

My first idea was to return the traceparent header in the API responses and then log them with each test. However, that’s not really how the OpenTelemetry traceparent header is supposed to be used. You’re supposed to include the header in the calling request. So that’s what I tried next.

Show me the code!

First thing I did was locate how the test suite was making the API calls. Turned out this was done with HttpClient, which offers a nice way to inject additional headers:

import static TraceInstrumentationInit.openTelemetry;
import static TraceInstrumentationInit.tracer;

public class HttpClientTraceInstrumentation implements HttpProcessor {
 @Override
 public void process(HttpRequest request, HttpContext context) {
 injectTraceHeaders(request);
 }

 @Override
 public void process(HttpResponse response, HttpContext context) {
 }

 private static void injectTraceHeaders(HttpRequest request) {
 // TODO
 }

 public static HttpClient registerTraceInstrumentation(
 DefaultHttpClient httpClient) {
 HttpClientTraceInstrumentation traceInstrumentation =
 new HttpClientTraceInstrumentation();
 httpClient.addRequestInterceptor(traceInstrumentation);
 httpClient.addResponseInterceptor(traceInstrumentation);
 return httpClient;
 }
}

Ok, so that gives us a way to inject the traceparent header. First I figured I could generate the trace ID myself. However, it turns out the traceparent value is not just a trivial UUID, it has a very specific structure and I wouldn’t advise trying to generate it yourself.

Luckily, OpenTelemetry has excellent SDKs that make it easy to generate and inject these headers. So we need a dependency on the OpenTelemetry API and SDK:

<dependencyManagement>
 <dependencies>
 <dependency>
 <groupId>io.opentelemetry</groupId>
 <artifactId>opentelemetry-bom</artifactId>
 <version>1.18.0</version>
 <type>pom</type>
 <scope>import</scope>
 </dependency>
 </dependencies>
</dependencyManagement>

<dependencies>
 <dependency>
 <groupId>io.opentelemetry</groupId>
 <artifactId>opentelemetry-api</artifactId>
 </dependency>
 <dependency>
 <groupId>io.opentelemetry</groupId>
 <artifactId>opentelemetry-sdk</artifactId>
 </dependency>
</dependencies>

Next, we had to initiate a trace for each scenario, and a sub-span for each API call being made.

After looking at the codebase, we saw we could add that to an already present base class used by all tests to report scenario results to TestRail. Personally I’m not a big fan of using base classes for tests as it can be hard to maintain a sensible hierarchy and the use of inheritance creates tight coupling between tests, which we don’t want. In JUnit for example, I would use @ExtendWith to keep such responsibilities separated and easier to re-use and to avoid the use of static instances as seen here. But we didn’t want to overhaul the complete test suite just to introduce this, better not to mix a big refactoring like that. The main goal at this point was to validate that traces could work.

As you see, we also passed the traceId to TestRail, more on that later.

public class Hooks extends BaseSteps {
 private static final TestRailReporter testRailReporter =
 TestRailReporterFactory.get();
 private static final ScenarioTraceInstrumentation instrumentation =
 new ScenarioTraceInstrumentation();

 @Before
 public void setup(Scenario scenario) {
 instrumentation.startTrace(scenario);
 }

 @After
 public void teardown(Scenario scenario) {
 String traceId = instrumentation.endTrace();
 testRailReporter.reportResultToTestRail(scenario, traceId);
 }
}

For every scenario we start a new span, including attributes to link it to the scenario. We also do some rudimentary logging for every scenario, making it easy to spot in the logs which scenario ran at which point and what the trace ID was for that scenario run.

import static TraceInstrumentationInit.tracer;

public class ScenarioTraceInstrumentation {
 private final ThreadLocalSpan spanState = new ThreadLocalSpan();

 public void startTrace(Scenario scenario) {
 Span span = spanState.start(
 tracer.spanBuilder(scenario.getName())
 .setSpanKind(SpanKind.CLIENT)
 );
 span.setAttribute("test.scenario_id", scenario.getId());

 System.out.println("|---------\n"
 + "| SCENARIO: " + scenario.getId() + " - " + scenario.getName() + "\n"
 + "| trace.id: " + span.getSpanContext().getTraceId());
 }

 public String endTrace() {
 String traceId = spanState.traceId();
 spanState.end();
 return traceId;
 }
}

To keep track of span state, I used a ThreadLocal, exposing a simple API to start and end the span that we could use from the ScenarioTraceInstrumentation and HttpClientTraceInstrumentation. As you see I did get lazy a bit here, using a Pair to store both the Span and Scope, using a record for that would make the code easier to read.

class ThreadLocalSpan {
 private final ThreadLocal<Pair<Span, Scope>> state = new ThreadLocal<>();

 Span start(SpanBuilder spanBuilder) {
 Span span = spanBuilder.startSpan();
 Scope scope = span.makeCurrent();
 state.set(Pair.of(span, scope));
 return span;
 }

 String traceId() {
 return state.get().getLeft().getSpanContext().getTraceId();
 }

 void end() {
 Pair<Span, Scope> pair = state.get();
 pair.getRight().close();
 pair.getLeft().end();
 }
}

The OpenTelemetry SDK also needs some initialization. Keeping it simple for this first experiment, and since some of the other code was static already, we just created two package scoped statics to be used from the ScenarioTraceInstrumentation and HttpClientTraceInstrumentation.

class TraceInstrumentationInit {
 static final OpenTelemetry openTelemetry = initOpenTelemetry();
 static final Tracer tracer = openTelemetry.getTracer("regressionTest");

 private static OpenTelemetry initOpenTelemetry() {
 SdkTracerProvider sdkTracerProvider = SdkTracerProvider
 .builder().build();
 OpenTelemetrySdk sdk = OpenTelemetrySdk.builder()
 .setTracerProvider(sdkTracerProvider)
 .setPropagators(ContextPropagators.create(
 W3CTraceContextPropagator.getInstance()
 ))
 .build();
 Runtime.getRuntime().addShutdownHook(
 new Thread(sdkTracerProvider::close)
 );
 return sdk;
 }
}

And finally, we could put all the pieces together in the HttpClientTraceInstrumentation, see below.

We create an additional span for the request [1] and then inject the trace header into the request [2].

The OpenTelemetry SDK code is very generic, allowing you to inject headers into pretty much anything, from HTTP requests to messages with all sorts of client libraries. The extremely generic code does make it a bit hard to read, but essentially you need to pass it a method that accepts two string arguments, the header name and header value. The HttpRequest from HttpClient has exactly such a method: setHeader [3].

Last but not least, we also log every request being made [4], giving a nice overview in the test logs of what API calls are being made for each scenario.

import static TraceInstrumentationInit.openTelemetry;
import static TraceInstrumentationInit.tracer;

public class HttpClientTraceInstrumentation implements HttpProcessor {
 private static final String W3C_TRACEPARENT_HEADER = "Traceparent";

 private static final TextMapPropagator textMapPropagator =
 openTelemetry.getPropagators().getTextMapPropagator();

 private static final TextMapSetter<HttpRequest> setter =
 HttpRequest::setHeader; // [3] http client setHeader method

 private final ThreadLocalSpan spanState = new ThreadLocalSpan();

 @Override
 public void process(HttpRequest request, HttpContext context) {
 Span span = spanState.start(
 tracer.spanBuilder("/")
 .setSpanKind(SpanKind.CLIENT)
 );
 try (Scope ignored = span.makeCurrent()) {
 // [1] add span details
 span.setAttribute(HTTP_METHOD, request.getRequestLine().getMethod());
 span.setAttribute(HTTP_URL, request.getRequestLine().getUri());
 span.setAttribute("component", "http");

 injectTraceHeader(request);
 logRequest(request);
 }
 }

 @Override
 public void process(HttpResponse response, HttpContext context) {
 spanState.end();
 }

 private static void injectTraceHeader(HttpRequest request) {
 // [2] let opentelemetry sdk propagate any required headers
 textMapPropagator.inject(Context.current(), request, setter);
 }

 private static void logRequest(HttpRequest request) {
 System.out.println( // [4]
 "|- " + request.getRequestLine() + "\n" +
 "| ^- " + request.getLastHeader(W3C_TRACEPARENT_HEADER));
 }
}

Everything coming together

Now when tests are run, their logs clearly show each of the scenarios, their trace ID and the API calls being invoked. And for every API call, the complete traceparent header is shown. With these details being propagated to our backend, we can now look up all the calls for a scenario in Elastic, or even search there for the span ID to locate the trace for a specific API call.

|---------
| SCENARIO: T2748705 - Validate if method PUT was added to controller and it's possible to use it
| trace.id: d28a64332015de5d324cb3e0f0380eba
|- PUT http://.../api/...
| ^- traceparent: ...
...
...
|---------
| SCENARIO: ...
| trace.id: ...
|- GET http://.../api/...
| ^- traceparent: ...
...
...

Integrate existing tools

At this client, the team used TestRail to centrally view reports of all test runs. To find the cause for a failing test, wouldn’t it be nice if we could skip sifting through logs? As you saw before, we had the trace ID available when we posted results to TestRail, so we added a link there to take you directly to the corresponding trace in Elastic.

Results!

When you follow that link to Elastic, it shows you all the API calls (transactions) that were part of that test scenario (trace).

From there you can navigate to each API call and look at individual traces. Elastic shows a lot of useful info, like a latency distribution chart that makes it super easy to quickly select all slow calls or all failing calls.

There is also a dedicated errors section, which allows you to quickly locate details for any errors that occurred as part of the scenario traces.

Tip: If you want to take a closer look at what’s in these kind of traces, there is a nice free Elastic demo environment where you can play around with traces in the APM section.

Conclusion

With just a little bit of code, we managed to reduce the amount of time needed to pinpoint test failures dramatically. Which shows again that optimizing for quick work loops and short feedback cycles are crucial for improving software delivery performance.

Move Faster with New Feature Flag Capabilities in AWS

Nico Krijnen — Mon, 13 Dec 2021 00:00:00 +0000

Originally published on Luminis on 13 Dec 2021.

I always get excited about things that allow you to bring new ideas to production faster. And it seems we are in luck! During the yearly AWS re:Invent conference that took place in Las Vegas, AWS launched several services that help you do just that!

Business around us is changing more and more rapidly. Year by year, the State of Devops research confirms that high performing companies excel in adapting to those changes by optimizing their ability to experiment, deliver quickly and gather feedback directly. You need to try many ideas and quickly find out which are successful and which are not. That ability to make many small changes reduces risk and increases the ability to innovate.

The new AWS services around feature flags enable organizations to achieve these goals. I will explain these briefly and show you how you can leverage them.

Moving fast

So what ingredients do you need to go fast? First, you have to optimize your organization and teams so they can come up with diverse ideas. Make sure you enable them to try those out and get feedback quickly without having to go through committees to get approval.

Second, on the engineering side, you need to learn to do everything in small steps. Integrate all your changes with the rest of your team often, at least daily. This way of working is called trunk-based development. The key here is never letting go of high quality and keeping your software in a constant state of being able to deploy it to production. To achieve that, you need a (continuous) delivery pipeline that automates all the steps needed to get from commit to deploy. As excellently explained by Dave Farley, the key responsibility of that pipeline is not to ensure that the software is 100% correct – that is not realistic. Instead, the pipeline needs to do all that it can to figure out if something might be broken or misbehaving. That means you’ll need at least a solid automated test suite that provides you with confidence that on the functional side everything works. Next to that, you can add validations to your pipeline for things like performance, code quality, and any other aspects that are important for your application.

If you do everything in small steps, does that mean you cannot have big innovations? No, of course it doesn’t. Big innovations never happen in one go. They are the combination of smaller increments, even the failed ones that you learned from. Like Edison’s light bulb, which was the result of thousands of failed experiments.

Feedback, as early as possible

A common trick is to separate the deployment of changes from the release of a feature. By using feature flags, you can include the new code you are writing without the feature showing up or being active. With all the code being deployed continuously, the act of “releasing” a feature simply becomes toggling the feature flag.

This way of working helps greatly in catching mistakes or wrong assumptions in your ideas early, while it is still easy to try another approach. It is much better to discover that your idea does not work after a few hours than finding out weeks later, when you’ve already invested a lot of time. You also make it much easier to experiment with new features much earlier in their development, maybe just exposing them to a few interested beta users and receive feedback before you make the feature generally available.

Enter feature flags

I always like keeping things simple. The easiest way to implement feature flags is to just add some booleans and ‘if statements’ in your code and allow these to be toggled between the different environments you have. For example, showing the new feature you are working on in your staging environment, but hiding and deactivating it in production. To be honest, in many cases, a simple mechanism like this is more than sufficient.

Do realize there is a tradeoff when using feature flags. Having these branches in your code means you add complexity and you make your application less predictable. In many cases the benefits greatly outweigh the downside, as long as you keep it under control. Don’t let hundreds of feature flags linger around. Be responsible, be a good craftsman and keep your codebase clean. Once a feature is released, clean up all traces of the flag. That way you minimize complexity and keep your codebase lean and mean and easy to change. After all, you want to keep doing more experiments and keep delivering value in a sustainable way.

Depending on how long your build pipeline takes, having to change booleans in code may mean that it takes a while to toggle a feature in production. This may be ok in some cases, but when it turns out that a feature doesn’t play nice on production yet, you want to toggle it off again quickly. What if you could toggle it in real-time and let someone else handle part of the complexity around feature flags?

Launch 1: AWS AppConfig – Feature Flags

AWS AppConfig has been around since 2019 and allows you to roll out configuration changes to your applications in real-time, without having to deploy code or take your application out of service. It can validate and then roll out configuration changes gradually while monitoring your application and rolling back changes in case of errors, minimizing impact to users.

Recently AWS launched AWS AppConfig Feature Flags, which builds on AppConfig and makes it easier to roll out new configurations of your feature toggles. You can now setup and toggle feature flags directly from the AWS Console. Versions of your feature flags configuration are tracked and as with config changes, you can manage and monitor the gradual deployment to your environments.

As feature flags are just booleans, there is no validation around them. You can also add custom attributes to flags. An example of what I would use these for is creating flags that are only enabled for specific beta tenants. For these attributes you can set up some simple validation rules in the UI. That may not be as powerful as writing a validation lambda as you can do for normal AppConfig validation, but it sure is a lot simpler and probably sufficient. And don’t forget, less handcrafted code means you reduce risk.

Apart from feature flags, you can also use this feature to toggle operational aspects of your application quickly. For example, in case of an incident you can temporarily switch off non-critical functionality or behavior to reduce load and keep the essentials of your service working, allowing your team to focus on resolving the incident.

That all sounds fantastic, but it does mean that releases of features are no longer tracked in your central git history, instead they are changed through the AWS Console. Luckily, the Console does track versions of your configurations and shows you the history of feature flags and configuration deployments that have been done to each of your environments. So, it may not be in one place anymore, but you can still audit when features went live or were rolled back. Just be aware that you are making this tradeoff.

Launch 2: AWS CloudWatch – Evidently

Where AppConfig Feature Flags is geared towards simplifying the technicality of managing feature flags and their deployment, Evidently is all about connecting the data you collect for experiments and A/B tests around feature roll outs that you do with smaller sets of users before making the feature generally available.

Evidently allows you to set up multiple variants of a feature and test these variants on different subsets of users at the same time. Once you have set up and are running such an experiment, Evidently also helps in statistically analyzing which variant is performing better, based on data collected during the experiment, giving you confidence in making the right choices.

In today’s fast paced world, you have to continuously reinvent your business to remain relevant. To do so you have to innovate and Evidently really helps you with tools to apply that mindset of running experiments and basing your decisions on the data that you gather. In today’s fast paced world, you have to continuously reinvent your business to remain relevant. To do so you have to innovate and Evidently really helps you with tools to apply that mindset of running experiments and basing your decisions on the data that you gather.

Launch 3: Amazon CloudWatch RUM (Real User Monitoring)

Now obviously, to make a service like Evidently work well, you need to collect the right data. Data that helps you prove the effectiveness and desired behavior of the features you build. To help with this, AWS launched RUM, another service under the CloudWatch umbrella.

CloudWatch is all about monitoring, but so far that was mostly oriented towards raw infrastructure and backend metrics of apps running inside AWS. RUM enhances that with capabilities to monitor web application performance, aimed at user experience. You inject some JavaScript into your web application frontend which then allows you to anonymously collect page load and layout performance metrics, JavaScript errors and client-side experienced HTTP errors.

Where traditional CloudWatch metrics are somewhat raw, RUM feels more like Google Analytics, giving you out-of-the-box dashboards that provide visibility into your application’s reliability, performance, and end user satisfaction. The service performs analysis on the collected data and breaks it down so you can gain insights about which areas of your application’s user experience can be improved.

On the observability side, CloudWatch RUM integrates with AWS X-Ray, so you can view traces and segments for end user requests. You can even see your clients in CloudWatch ServiceLens service maps.

Pricing

As usual with AWS, pricing for these services are based on usage, which means you can try these out on a small scale and then evaluate if the benefits they bring you are worth the cost. Once you use these services in production, always put up some cost monitoring so you remain aware of what you are spending. Below you will find an overview of the pricing for these services. For details, look at their pricing page.

AWS AppConfig Feature Flags are charged by the number of times you request and receive config changes:

$0.0000002 per configuration request via API Calls
$0.0008 per configuration received

AWS CloudWatch Evidently has a first time free trial that includes 3 million Evidently events and 10 million Evidently analysis units per account. After that you pay:

$5 per 1 million events (user actions & assignment events)
$7.50 per 1 million analysis units

AWS CloudWatch RUM allows you to control costs by configuring it to only analyze a limited percentage of user sessions. There is a first time free trial that includes 1 million RUM events per account. After that you pay:

$1 for every 100,000 events collected (page view, JavaScript error, HTTP error)

Alternatives

Compared to products that specialize in this field like LaunchDarkly or Optimizely, these AWS services may feel somewhat bare bones. This is no coincidence, as Werner Vogels said: AWS services are designed to be building blocks, not frameworks.

AWS may not be as feature rich as some of these others but having it all integrated into the rest of your service stack can be a big benefit. The low barrier to entry means you can start using it today and get value out of it quickly, or switch to something else later if you discover that your needs have grown.

And don’t forget, if you don’t need all the experimentation statistics or real-time flag controls, just keep it simple and add a few booleans in your code.

Conclusion

It is great to see that AWS is serious about giving their users the keys they need for innovation. What I personally find most exciting is that with this combination of services, AWS makes it easier for all of us to adopt that mindset of doing experiment-based and data-driven roll outs.

If you need help to introduce that mindset of experimentation, don’t hesitate to reach out!

Learn more

If you want to learn more about these AWS services, check out the official blogs: