The System That Was Never Challenged
I worked on a system that, on the surface, was well-architected.
It had structure. It had clear intent. It solved the problem it was designed to solve.
And in a lot of ways, it was good work.
It also had an N+1 problem.
On top of that, it was running, on average, an additional 60 or so queries per page load.
And there was enough “magic” baked into it that, over time, only a couple of us really understood how it worked.
None of that happened because the engineer who built it didn’t know what they were doing.
Quite the opposite.
They were capable. Thoughtful. Intentional.
The system didn’t fail because of a lack of skill.
It failed because it was never truly challenged.
Red Team vs Blue Team
Before I go too far, it’s probably worth calling out the obvious question:
If there’s a red team… what’s the blue team?
In this context, the blue team is the group responsible for the design itself — the people building the architecture, writing the technical design document, and making the initial set of decisions.
They’re the ones solving the problem.
The red team exists to challenge how that problem is being solved.
And if you’ve ever had a TDD red-teamed and absolutely shredded by the Navy’s N6… everything else tends to pale in comparison.
In a lot of places, that challenge gets folded into a design review.
A TDD or TAD walkthrough where stakeholders, engineers, and leadership all sit in a room and talk through the design.
On paper, that sounds like the right place for it.
In practice, it usually isn’t.
Those meetings tend to be high-level.
There’s pressure to keep things moving.
To stay focused.
To not get bogged down in edge cases or deep technical rabbit holes.
And that’s not necessarily wrong — those meetings serve a purpose.
But it also means the harder questions…
The uncomfortable ones…
The “what happens when this breaks?” conversations…
often don’t get asked.
Or don’t get asked deeply enough.
That’s why I prefer to separate them.
A blue team session to design the system.
And a dedicated red team session to pressure-test it.
Different goals. Different mindset. Different conversations.
One builds.
The other challenges.
And you need both.
Building the Right Red Team
In security, we talk about red teams.
Groups whose job is to actively probe, challenge, and break systems before someone else does.
Engineering doesn’t always have an equivalent.
But it should.
Over time, I’ve found that the most effective red teams aren’t random.
I try to be intentional about who’s in the room — or at least what perspectives are represented.
At a minimum, I want:
- someone strong on the backend — how the system is structured and behaves
- someone who understands the frontend — how it’s actually used and where it breaks in real interaction
- someone with a deep understanding of the database — how queries behave and where performance issues hide
- someone security-minded — thinking about abuse, edge cases, and failure modes
And depending on the system, I’ll also try to bring in:
- someone with an operations or SRE mindset — thinking about failure, recovery, observability, and “what happens at 2am”
- someone approaching it from a maintenance perspective — “how hard is this to live with in six months?”
Sometimes one person can cover multiple areas.
I also try to be mindful of how many people I’m pulling into this.
In most cases, I’ll cap an initial red team at around four people.
That tends to be the sweet spot — enough perspective to meaningfully challenge the design, without unnecessarily interrupting half the team.
If I can get the right coverage with three, I’ll do that.
And if I can only get three, I’ll take three.
That’s where having people who can span multiple disciplines becomes really valuable.
The goal isn’t to make this heavy.
It’s to make it effective.
One of the more surprising things I’ve learned is how valuable it is to include mid-level engineers in these conversations.
Not because they know less — but because they see things differently.
They’re often the ones closest to the day-to-day work.
They’re the ones most likely to be maintaining what gets built.
And they ask questions that more senior engineers sometimes skip right past.
The kind of questions that sound simple…
until you realize they’ve just exposed something you hadn’t considered at all.
More than once, those have been the questions that mattered most.
What Red Teaming Actually Looks Like
Because most design problems don’t look like problems at the beginning.
They look like decisions.
Reasonable ones.
Individually justifiable.
Even clever.
Until they aren’t.
Until load increases.
Until someone new joins the team.
Until a bug shows up in a place no one expects.
Until the system becomes something only its author can safely modify.
That’s where red teaming comes in.
Not as criticism.
Not as bureaucracy.
But as a deliberate practice:
“Let’s try to break this before reality does.”
In practice, that often looks like questions:
- What happens under load?
- Are we introducing N+1 queries?
- How many queries does this page actually generate?
- Where are we relying on implicit behavior or “magic”?
- How long would it take a new engineer to understand this?
- What assumptions are we making that might not hold?
None of these questions are particularly difficult.
But they’re easy to skip when you’re focused on getting something built.
Why It Matters
That’s the trap.
Because building is visible.
Shipping is measurable.
Thinking ahead… isn’t.
And to be clear — this isn’t about slowing things down.
It’s about avoiding the kind of complexity that slows you down later.
I’ve found that some of the best engineering conversations happen when someone is willing to say:
“I don’t think this breaks yet… but I think it will.”
That’s not negativity.
That’s experience.
And this is where tools — including AI — can help.
Not as decision-makers, but as pressure testers.
A quick way to ask:
- “What am I missing?”
- “Where does this fall apart?”
- “What assumptions am I making?”
But just like anything else, that only works if you’re willing to question the answers you get back.
Because at the end of the day, red teaming isn’t about tools.
It’s about mindset.
Closing Thoughts
Good engineers build systems that work.
Great engineers build systems that survive scrutiny.
If there’s one thing I’ve learned, it’s this:
Most of the issues that hurt the most later…
were visible earlier.
We just didn’t ask the right questions.
Or we didn’t ask them soon enough.
Red teaming isn’t about catching everything.
It’s about catching enough early that the system stays understandable, maintainable, and resilient over time.
Because systems don’t break all at once. They erode. Quietly. Until one day, they’re too complex to safely change. And by then, it’s already expensive.
So ask the questions.
Invite the pushback.
Pressure-test the design.
Before reality does it for you.
Top comments (0)