The systems we test are evolving quickly, and I’m realizing that the way we think about testing has to evolve with them.
New products, new capabilities, new technology, and new ways of thinking about what we build and how we deliver it. As the systems around me have changed, I’ve found my thinking about testing changing with them.
That doesn’t mean throwing away what works. It means building on a strong foundation while adapting to new technologies and architectures. The environment is evolving, and the way I think about meaningful coverage is evolving with it. My time in Test has taught me to question assumptions, look beyond the obvious, and think about the difference between how something is supposed to work and how it actually behaves. What I’m starting to see more clearly now is just how far beyond the individual feature that thinking can extend.
A Test Passing Doesn’t Necessarily Mean You’re Covered
It’s easy to look at a feature and start identifying test cases. Can I create something? Can I modify it? Can I retrieve it? Can I delete it? Those are important behaviors to validate, but there’s another question underneath them: What exactly did we prove?
Imagine a fairly ordinary record moving through a system. It gets created, information is added or changed, and eventually it appears somewhere else. From the outside, the testing problem can look pretty straightforward. Create the record, modify it, and verify the result. But that only proves the visible behavior.
Where did the information originate? What happened before it reached its destination? Which parts of the system participated? Were there transformations along the way? Did every field arrive correctly? What happened when something changed? What needed to remain synchronized? What happens when one part of that chain behaves differently than expected? Suddenly, you’re no longer looking at a handful of test cases. You’re looking at a system.
And that distinction has been changing the way I think about coverage.
From Feature Thinking to System Thinking
I’ve spent a lot of my career thinking about testing from the perspective of behavior. Give the system an input, observe what it does, and compare the result against what should have happened. That mindset is still incredibly valuable, and I don’t see that changing.
But as the systems around me have become more distributed and interconnected, I’ve found myself widening the boundary around what I consider “the thing” being tested. A user might experience one feature, but behind that feature may be services, APIs, data stores, integrations, events, and handoffs all working together. Testing the visible result can tell me whether the destination looks right, but it doesn’t necessarily tell me whether the entire journey behaved correctly.
That has pushed me toward a phrase I keep coming back to: meaningful coverage. To me, meaningful coverage isn’t simply more tests, the prettiest automation report, or a checkbox beside every feature. It’s coverage that represents how the system actually works and gives us confidence in the behavior that matters.
Sometimes the Most Important Work Happens Before the Test
This has become increasingly important as I’ve worked with larger, more established systems. Mature systems accumulate history. They evolve across years, teams, technologies, and changing requirements. Knowledge becomes distributed across code, documentation, architecture, and often the people who have worked with those systems over time.
No single person has to know every inner working of a system to test it effectively. But we do need to recognize when our mental model has gaps before deciding how something should be tested and spend more time understanding the system around it. That means asking questions, talking to people who understand different parts of the system, learning where a behavior begins and where it goes, identifying important boundaries and handoffs, and determining what actually needs to be proven.`
Only then can we make a better decision about where testing provides the most value. Sometimes the highest-leverage testing work happens before a test is ever written. It comes from building the right mental model of the system first.
Automate Where Automation Gives You Leverage
My thinking around automation has been evolving too. Manual exploratory testing still matters. Give me a completely new product and one of the first things I want to do is use it like a human being. I want to click around, try things, ask questions, and find the weird edges nobody anticipated. Humans are really good at that kind of exploration.
Once we understand a behavior well enough, though, we're increasingly interested in a different question: What is the most stable and efficient layer at which we can continuously prove this? Sometimes that may be the user interface. Sometimes it isn’t. If an API or another stable interface allows us to validate the underlying behavior directly, that can provide faster feedback, greater consistency, and the ability to exercise a much larger portion of the system continuously.
That doesn’t eliminate other forms of testing. It lets each kind of testing do the job it’s actually good at. Human judgment can focus on exploration, automation can provide repeatability and scale, and deeper validation can happen where the architecture gives us better points of observation.
The goal isn’t automation for automation’s sake. The goal is confidence.
Coverage Is Only the Beginning
There’s another idea I’ve been exploring that I’m especially excited about. What if automated testing isn’t only about producing a pass or fail? Every automated execution also produces information, and over time, many automated workflows produce a lot of information.
That changes the question from “How do we automate more tests?” to “How do we turn continuous testing into continuous visibility?”
Think about this as a progression. First, establish meaningful automated coverage and useful feedback. Then improve how those signals are collected, organized, correlated, and visualized. Once that foundation exists, capabilities like observability, trend analysis, better diagnostics, faster investigation, and potentially AI-assisted analysis become much more interesting.
One idea has become especially important to me while thinking through all of this:
The intelligence isn’t the foundation. The architecture is.
AI can be an incredibly useful layer. It can help investigate, summarize, correlate, and potentially surface things a human might not notice quickly. But if the underlying signals are poor, the coverage doesn’t represent the system, or the architecture doesn’t provide meaningful context, adding intelligence on top doesn’t magically fix the foundation.
The foundation has to be useful first. Once it is, the tools we layer on top can compound its value.
Build the Foundation, Then Let It Grow
That realization has influenced how I think about the larger testing strategy. Becoming less interested in designing one enormous solution that attempts to solve everything immediately. I’d rather build useful layers by establishing meaningful coverage, getting fast feedback, improving visibility, and learning from what the system is telling us. From there, the next layer can be built based on what we’ve actually learned rather than what we assume we’ll need.
That approach feels especially appropriate when the technology around you is changing quickly. You don’t need to know exactly what the final system will look like on day one. You need a foundation strong enough that whatever you build next has somewhere useful to stand.
Maybe that’s the larger shift in my thinking. Testing has always been about more than writing test cases, but what I’m seeing more clearly now is how much it can become an exercise in building confidence across an entire system. The more interconnected that system becomes, the more important it is to understand what we’re actually proving and whether our coverage represents reality rather than simply the parts that are easiest to observe.
I don’t have every answer, and I don’t think anyone working with complex systems ever does. But the questions I’m asking are getting broader. Less “What tests should I write for this feature?” and more “What do I need to understand and continuously validate to have confidence in this system?”
That’s a much bigger question, and right now, I think it’s a much more interesting one.
I’d genuinely like to hear how other engineers and test teams think about this as their systems evolve. Where do you draw the boundary between testing a feature and testing the system behind it? And what does meaningful coverage mean to you?
Top comments (0)