DEV Community: Jarrod Cabarubio

A Roadmap Is a Hypothesis

Jarrod Cabarubio — Wed, 02 Sep 2026 16:18:58 +0000

What building an AI Control Plane taught me about planning, evidence, and knowing when to recalibrate

What building in an emerging design space taught me about planning, evidence, and knowing when to recalibrate

There is a fascinating moment that can happen when you are building in territory that is still being defined.

You begin with everything you should have: research, architectural thinking, deliberate milestones, and a roadmap built around the best understanding you have at the time.

Then you start building.

And somewhere along the way, the work gives you something you could not have had when you made the plan.

Something no amount of careful planning could have manufactured ahead of time.

Evidence that does not exist yet.

For the past several months, I have been building an AI Control Plane as part of a larger personal project called Synthesizer.

At a high level, the Control Plane is about governing how AI driven work is allowed to happen. It creates explicit boundaries around capabilities, policy, approvals, execution, and evidence while keeping architectural authority separate from the engines that eventually perform the work.

It is an exciting problem because many of the engineering fundamentals are familiar, but the design space around them is still emerging. There are established ideas to learn from, but far fewer established patterns for the exact kind of system I am trying to build.

That means the roadmap matters.

It also means every milestone has the potential to teach me something the previous milestone could not have known.

The moment the question changed

Around the third major phase of the Control Plane, I started seeing part of the implementation differently.

Nothing had mysteriously built itself. The system had not somehow raced ahead while I was not looking. I had simply accumulated more evidence than I had when I designed the roadmap.

Weeks of implementation, testing, architectural review, persistence work, recovery thinking, and boundary validation had changed what I understood about the subsystem in front of me.

I found myself shifting away from the question I had been using to measure progress.

What is left on the roadmap?

And toward a different one.

What responsibility has not actually been fulfilled?

That was the aha moment.

The implementation had reached a point where I could evaluate the subsystem by what it was responsible for doing rather than by how many planned tasks remained.

Once I looked at it that way, I realized my understanding of where I was in the roadmap needed to change.

A little bit under the hood

This phase was focused on execution history and persistence.

If the Control Plane is going to govern work, it needs durable evidence of what happened. So I had been working through questions around persistence, inspection, recovery, and how execution history should behave as the larger system evolves.

By the time I reached this point, those ideas were no longer architectural assumptions on paper. They had been implemented, tested, and exercised enough that I could evaluate the subsystem against its actual responsibility.

That changed the conversation.

When I compared what the implementation had taught me with what remained on the roadmap, I realized that continuing exactly as planned would mean making decisions about future needs before those needs had given me enough evidence to design them responsibly.

The roadmap was not wrong.

My understanding was simply better than it had been when I wrote it.

When the territory is newer than the map

I think this matters even more when you are working in an emerging technical space.

When mature patterns exist, a roadmap can lean on years of accumulated industry experience. When the territory itself is still developing, some decisions necessarily begin as informed assumptions.

That does not mean improvising the architecture as you go. It means being disciplined enough to recognize the difference between an assumption and evidence.

There is a big difference between changing direction because a new idea feels exciting and recalibrating because implementation has materially changed what you know.

One is churn.

The other is learning.

That is the line I want to get better at recognizing.

A roadmap deserves respect, not blind loyalty

I do not think a roadmap should be treated casually. It creates direction, protects scope, forces decisions, and gives the work a coherent sequence.

Most of the time, the right move is to follow it.

But when real implementation exposes something the original plan could not have known, that moment deserves attention.

The standard cannot simply be, "I thought of something better."

The standard has to be closer to, "I now have material evidence that changes the assumptions behind this decision."

That is a much higher bar.

And when that bar is met, changing your mind is not weakness.

It is a signal that you are listening.

The roadmap did its job

This is the part that changed how I think about the whole experience.

I did not reach this point because the roadmap failed.

The roadmap carried me far enough to create the evidence that improved my understanding of the roadmap.

It gave me enough structure to build deliberately. Implementation then gave me enough knowledge to recalibrate deliberately.

The architectural responsibility stayed intact. The source of truth stayed intact. The seams stayed open. Future capabilities remained possible.

I just had better information now than I had when I originally planned the work.

A roadmap is a hypothesis. Implementation is the experiment.

When engineering teaches architecture back

There is a feedback loop here that I find fascinating.

At the beginning of a project, architecture teaches engineering. It defines boundaries, responsibilities, contracts, and the direction of travel.

Then engineering begins producing evidence.

If I am paying attention, eventually that evidence starts teaching the architecture back.

Not because the system autonomously decides what it should become. The control and judgment still belong with the person building it.

The implementation simply gives me information I could not possess before I built it.

I build. I observe. I test assumptions. I learn. Then I decide whether the architecture should move.

That feedback loop is becoming one of my favorite parts of this project.

Architecture creates direction. Implementation creates evidence. Evidence refines architecture.

Be a student of your craft

There is a mindset underneath all of this that I want to keep.

Be a student of your craft.

A student can be experienced. A student can be highly technical. A student can have strong opinions and carefully designed plans.

What a student cannot afford to become is finished.

Working in an emerging space makes that impossible to ignore. The tools are changing. The workflows are changing. The design space is expanding. Things I did a few months ago are already being challenged by things I have learned since.

That does not make the earlier work foolish. It is how the later understanding became possible.

Plan seriously. Build carefully. Test the assumptions. Then give new evidence enough respect to change your mind when it has earned the right to.

Ride the wave

There is a phrase I use in my own life that keeps finding its way into how I think about building.

Ride the wave.

For me, that does not mean chasing novelty or rewriting the plan every time something new appears.

It means being comfortable being uncomfortable when something real is changing underneath you.

Sometimes the right move is to stay the course. Sometimes the right move is to reset the table and choose again with better information.

The difficult part is learning the difference.

When the evidence earns the change, make the better decision and keep moving.

Ride the wave with style.

What I am carrying forward

I am not leaving this phase believing roadmaps matter less.

I think they matter more.

But I am starting to see a roadmap less as a promise about exactly what the future will contain and more as the best technical hypothesis I can form before the future gives me evidence.

That feels especially important when I am building in territory that is still being explored.

The goal is not to predict every future need perfectly.

The goal is to build with enough discipline that when reality teaches me something better, the architecture can absorb the lesson without losing its principles.

That is what happened here.

I planned. I built. I learned enough to see the next decision differently.

So I recalibrated.

Not because the roadmap failed.

Because it got me far enough to know more than I knew when I wrote it.

A question for other builders

If you are building in an emerging technical space, I would genuinely like to hear where you draw this line.

When has implementation given you enough evidence to change a plan you once felt confident about? And how did you distinguish a meaningful discovery from the ordinary temptation to keep redesigning?

If you have applied a similar idea in your own work, tell me what the system taught you.

I am still a student of this craft too.

# Beyond the Passing Test: Rethinking What Coverage Really Means

Jarrod Cabarubio — Thu, 20 Aug 2026 04:58:05 +0000

The systems we test are evolving quickly, and I’m realizing that the way we think about testing has to evolve with them.

New products, new capabilities, new technology, and new ways of thinking about what we build and how we deliver it. As the systems around me have changed, I’ve found my thinking about testing changing with them.

That doesn’t mean throwing away what works. It means building on a strong foundation while adapting to new technologies and architectures. The environment is evolving, and the way I think about meaningful coverage is evolving with it. My time in Test has taught me to question assumptions, look beyond the obvious, and think about the difference between how something is supposed to work and how it actually behaves. What I’m starting to see more clearly now is just how far beyond the individual feature that thinking can extend.

A Test Passing Doesn’t Necessarily Mean You’re Covered

It’s easy to look at a feature and start identifying test cases. Can I create something? Can I modify it? Can I retrieve it? Can I delete it? Those are important behaviors to validate, but there’s another question underneath them: What exactly did we prove?

Imagine a fairly ordinary record moving through a system. It gets created, information is added or changed, and eventually it appears somewhere else. From the outside, the testing problem can look pretty straightforward. Create the record, modify it, and verify the result. But that only proves the visible behavior.

Where did the information originate? What happened before it reached its destination? Which parts of the system participated? Were there transformations along the way? Did every field arrive correctly? What happened when something changed? What needed to remain synchronized? What happens when one part of that chain behaves differently than expected? Suddenly, you’re no longer looking at a handful of test cases. You’re looking at a system.

And that distinction has been changing the way I think about coverage.

From Feature Thinking to System Thinking

I’ve spent a lot of my career thinking about testing from the perspective of behavior. Give the system an input, observe what it does, and compare the result against what should have happened. That mindset is still incredibly valuable, and I don’t see that changing.

But as the systems around me have become more distributed and interconnected, I’ve found myself widening the boundary around what I consider “the thing” being tested. A user might experience one feature, but behind that feature may be services, APIs, data stores, integrations, events, and handoffs all working together. Testing the visible result can tell me whether the destination looks right, but it doesn’t necessarily tell me whether the entire journey behaved correctly.

That has pushed me toward a phrase I keep coming back to: meaningful coverage. To me, meaningful coverage isn’t simply more tests, the prettiest automation report, or a checkbox beside every feature. It’s coverage that represents how the system actually works and gives us confidence in the behavior that matters.

Sometimes the Most Important Work Happens Before the Test

This has become increasingly important as I’ve worked with larger, more established systems. Mature systems accumulate history. They evolve across years, teams, technologies, and changing requirements. Knowledge becomes distributed across code, documentation, architecture, and often the people who have worked with those systems over time.

No single person has to know every inner working of a system to test it effectively. But we do need to recognize when our mental model has gaps before deciding how something should be tested and spend more time understanding the system around it. That means asking questions, talking to people who understand different parts of the system, learning where a behavior begins and where it goes, identifying important boundaries and handoffs, and determining what actually needs to be proven.`

Only then can we make a better decision about where testing provides the most value. Sometimes the highest-leverage testing work happens before a test is ever written. It comes from building the right mental model of the system first.

Automate Where Automation Gives You Leverage

My thinking around automation has been evolving too. Manual exploratory testing still matters. Give me a completely new product and one of the first things I want to do is use it like a human being. I want to click around, try things, ask questions, and find the weird edges nobody anticipated. Humans are really good at that kind of exploration.

Once we understand a behavior well enough, though, we're increasingly interested in a different question: What is the most stable and efficient layer at which we can continuously prove this? Sometimes that may be the user interface. Sometimes it isn’t. If an API or another stable interface allows us to validate the underlying behavior directly, that can provide faster feedback, greater consistency, and the ability to exercise a much larger portion of the system continuously.

That doesn’t eliminate other forms of testing. It lets each kind of testing do the job it’s actually good at. Human judgment can focus on exploration, automation can provide repeatability and scale, and deeper validation can happen where the architecture gives us better points of observation.

The goal isn’t automation for automation’s sake. The goal is confidence.

Coverage Is Only the Beginning

There’s another idea I’ve been exploring that I’m especially excited about. What if automated testing isn’t only about producing a pass or fail? Every automated execution also produces information, and over time, many automated workflows produce a lot of information.

That changes the question from “How do we automate more tests?” to “How do we turn continuous testing into continuous visibility?”

Think about this as a progression. First, establish meaningful automated coverage and useful feedback. Then improve how those signals are collected, organized, correlated, and visualized. Once that foundation exists, capabilities like observability, trend analysis, better diagnostics, faster investigation, and potentially AI-assisted analysis become much more interesting.

One idea has become especially important to me while thinking through all of this:

The intelligence isn’t the foundation. The architecture is.

AI can be an incredibly useful layer. It can help investigate, summarize, correlate, and potentially surface things a human might not notice quickly. But if the underlying signals are poor, the coverage doesn’t represent the system, or the architecture doesn’t provide meaningful context, adding intelligence on top doesn’t magically fix the foundation.

The foundation has to be useful first. Once it is, the tools we layer on top can compound its value.

Build the Foundation, Then Let It Grow

That realization has influenced how I think about the larger testing strategy. Becoming less interested in designing one enormous solution that attempts to solve everything immediately. I’d rather build useful layers by establishing meaningful coverage, getting fast feedback, improving visibility, and learning from what the system is telling us. From there, the next layer can be built based on what we’ve actually learned rather than what we assume we’ll need.

That approach feels especially appropriate when the technology around you is changing quickly. You don’t need to know exactly what the final system will look like on day one. You need a foundation strong enough that whatever you build next has somewhere useful to stand.

Maybe that’s the larger shift in my thinking. Testing has always been about more than writing test cases, but what I’m seeing more clearly now is how much it can become an exercise in building confidence across an entire system. The more interconnected that system becomes, the more important it is to understand what we’re actually proving and whether our coverage represents reality rather than simply the parts that are easiest to observe.

I don’t have every answer, and I don’t think anyone working with complex systems ever does. But the questions I’m asking are getting broader. Less “What tests should I write for this feature?” and more “What do I need to understand and continuously validate to have confidence in this system?”

That’s a much bigger question, and right now, I think it’s a much more interesting one.

I’d genuinely like to hear how other engineers and test teams think about this as their systems evolve. Where do you draw the boundary between testing a feature and testing the system behind it? And what does meaningful coverage mean to you?

DEF CON Was Different This Time, Maybe Because I Was Different... <☠/>

Jarrod Cabarubio — Sat, 15 Aug 2026 17:28:46 +0000

The first time I went to DEF CON, I spent most of my time doing challenges. That was the mission. Find something interesting, sit down, start digging, and see if I could figure it out. I loved it. There is something incredibly satisfying about disappearing into a problem for a few hours and eventually getting to the other side of it.

This year was different.

I still love that deep-dive side of DEF CON. CTFs and technical challenges are part of what makes this community what it is, and there is a reason so many incredibly talented people spend hours working through them.

But this year I realized that what was scarce for me wasn't screen time. I can always find time to go deep on a technical problem. What I couldn't recreate at home was being physically surrounded by this enormous concentration of knowledge, creativity, technology, companies, researchers, builders, hackers, and people obsessed with things I may have never even heard of.

I wanted to see what was out there.

So I walked.

Following Curiosity

DEF CON is difficult to explain to somebody who has never been. Yes, it's a cybersecurity conference, but that description doesn't really capture it.

There are villages dedicated to specific areas of security, people working through CTFs and challenges, hardware being torn apart, locks being picked, wireless signals being investigated, social engineering, AI, physical security, embedded systems, privacy, infrastructure, and countless things that seem to exist somewhere between all of those categories.

What struck me this year wasn't any single technology. It was the density of ideas.

You can walk ten feet and find somebody who knows an incredible amount about something you barely knew existed. Walk another ten feet and it happens again. There is something energizing about realizing how much you don't know.

Instead of making me want to retreat into the things I already understood, it made me want to keep moving.

I started following whatever caught my attention.

Sometimes it was security. Sometimes it was a product or a demonstration. Sometimes somebody had built something that immediately made me wonder what else you could do with it.

That's where my brain tends to get me into trouble.

One idea becomes another idea. That one connects to a problem I've seen somewhere else. Then I'm wondering whether the same approach could work in an entirely different environment.

Before long, I wasn't just looking at what people had built.

I was thinking about what could be built next.

More Than the Technology

Something else caught my attention that I wasn't expecting: how companies were presenting themselves.

My company is entering a period where marketing is becoming increasingly important. Marketing isn't my job, but apparently my brain doesn't care much about job descriptions when it finds an interesting problem.

So I started watching.

What made somebody stop at one booth and walk past another? How were companies explaining complicated technology? What made a demonstration memorable? How did they turn something deeply technical into something another human being could understand in thirty seconds?

I started looking at DEF CON partly as an engineer, partly as somebody interested in security, partly as somebody interested in products, and apparently partly as an accidental marketing researcher.

That ended up being one of my favorite parts of the trip.

It also made me realize that the things I find interesting increasingly seem to live between disciplines rather than comfortably inside one of them.

The Conversations

Then there were the people.

I met a penetration tester working in the same state where my company operates. I unexpectedly met the social media person from one of our channel partners. I talked with vendors, builders, security people, and people working on technologies that sent me down entirely new trains of thought.

Those conversations mattered.

Not because every conversation resulted in some immediate opportunity. Most didn't. They mattered because talking to somebody who sees the world through a different technical lens can change the way you look at your own problems.

That's hard to reproduce sitting behind a computer.

And this was probably where my experience diverged most from my first DEF CON.

The first time, I wanted to solve what was in front of me.

This time, I wanted to discover what I didn't know was there.

Neither approach is better. They're just different ways of using the same incredible environment.

For me, this year, the second one felt right.

Paying Attention to Where My Mind Goes

I've been thinking about that since coming home.

I'm a test engineer, and testing has shaped the way I think more than I probably realized. It teaches you to question assumptions, look for what isn't obvious, understand how systems interact, and investigate the distance between how something is supposed to work and how it actually works.

I love that mindset.

But I'm also starting to notice that my curiosity doesn't stop at testing.

Lately I've been increasingly interested in designing solutions, architecture, AI, scaling systems, communicating ideas, connecting technologies, and taking something that exists in one place and imagining what it could become somewhere else.

I'm not ready to put a title on that.

Actually, I don't think I want to.

I'd rather pay attention to it.

There is a difference between deciding who you want to become and noticing what you naturally move toward when nobody is telling you where to go.

DEF CON gave me an interesting opportunity to observe the second one.

Nobody told me which booth to visit. Nobody assigned me a project. Nobody told me which conversations to have or which ideas were relevant to my job.

I had an enormous playground of technology and knowledge in front of me and complete freedom to choose where I spent my attention.

Looking back, where that attention went might be one of the most valuable things I learned on the trip.

I'm documenting this journey as it happens, the things I'm building, what I'm learning about AI and systems, the people and ideas that challenge the way I think, and occasionally experiences like DEF CON that move the whole thing in an unexpected direction.

Someday I want to be able to look backward and see how the pieces connected.

This was one of them.

One more for the books. Until next time, DEF CON.

The problem was never intelligence, it was continuity.

Jarrod Cabarubio — Tue, 04 Aug 2026 22:40:37 +0000

Building an AI Operating Layer
Entry 2
The Problem Was Never Intelligence. It Was Continuity

Somewhere along the way I realized I had been asking the wrong question.
I kept obsessing over intelligence.
Bigger models.
Better reasoning.
More capable agents.
Better tools.

It seemed obvious. If the models kept getting smarter, everything else would fall into place.
Then one day it hit me.

None of that matters if the next conversation has to start over.

Think about that for a second.
You spend two weeks finally understanding a problem.
You make decisions. You reject bad ideas. You discover patterns. The architecture starts making sense.

Then a new model jumps in.
Or another engineer joins the project.
Or six months pass.

And everyone starts rebuilding the same understanding all over again.

That felt crazy to me.

I thought memory was the answer.
Save the chats.
Store the documents.
Add vector search.
Summarize everything.

Those ideas help, but they don't solve the deeper problem.

Memory stores information.
It doesn't tell you what still matters.

That was the turning point.

I stopped thinking about memory and started thinking about continuity.

What if the important thing wasn't remembering everything?
What if it was preserving understanding?

That single question quietly changed the architecture.
Today the project looks very different.
Instead of asking how to give an AI more information, I'm asking how to prepare the right understanding before any reasoning begins.

That has led us into knowledge models, governed context, project reconstruction, and architectural boundaries.

Some of those ideas will survive.
Some probably won't.

That's okay.
This journal isn't here to prove I was right.
It's here to document how the thinking evolves.

Where I am today
I no longer think the hardest problem is building a smarter AI.

I think the harder problem is making sure understanding survives.
If we can solve that, maybe intelligence stops starting over.

Maybe every project gets a little wiser than the last.
I don't know yet.

But I think that's a question worth chasing.

Building an AI Operating Layer - Episode 1: Why I Didn't Start Sooner

Jarrod Cabarubio — Wed, 29 Jul 2026 21:04:03 +0000

Building an AI Operating Layer

Episode 1
Why I Didn't Start Sooner

Most engineering projects begin with an idea. This one began with a question.

For months I found myself watching the explosion of AI tools, frameworks, models, and agent platforms. Every week there seemed to be another breakthrough, another library, and another opinion about where everything was headed. I could have started building immediately. Part of me thought I should have.

But I realized something, and it kept bothering me. I wasn't afraid of writing code. I was afraid of solving the wrong problem.

When a new technology appears, it's easy to jump straight into implementation. Pick a framework. Choose a model. Build something. Ship it. I didn't want to start there because I had a feeling there was a much bigger picture that I wasn't seeing yet.

So I waited.

I spent my time reading, experimenting, asking questions, and trying to understand how all of these pieces connected. The more I learned, the more I realized I wasn't actually interested in building another AI application.

What fascinated me was the system behind the systems.

What happens when you stop looking at models, memory, orchestration, tools, policies, and execution as separate ideas and start seeing them as parts of a much larger ecosystem?

That question became the beginning of this project.

This isn't a story about predicting the future. It's a story about trying to understand it. I'm sure some of my assumptions will be wrong. I'm sure parts of this architecture will change. If they do, you'll see that too.

I don't want this journal to only show the polished results. I want it to capture the discoveries, the wrong turns, the redesigns, and the moments where a better idea replaces an old one.

At the center of this journey is a project I'm calling the AI Operating Layer. Today it's mostly architecture, documentation, research, prototypes, and a growing collection of ideas. Maybe that's exactly where projects like this should begin.

I'm not interested in rushing toward a finished product. I'm interested in building understanding first. If the understanding is solid, everything that follows has a stronger foundation.

One realization has already changed the way I think about this project. I originally thought I was building a sophisticated synthesis engine. Now I think I'm trying to understand what it means to build a living knowledge ecosystem that can continuously learn, integrate new capabilities, and grow over time.

Whether that idea survives the journey is something we'll discover together.

This isn't a launch announcement. It isn't a product reveal.

It's an engineering journal.

Welcome to the journey.