The architecture is correct.
You read it twice looking for the flaw and there is not one. The tradeoffs are handled. The failure modes are addressed. If you had a team of eight it would be the right answer.
You have four people. One of them understands Flink, she is on call every third week, and she has been talking to a recruiter. You cannot build this, and nothing in the recommendation is wrong.
🔍 What was missing was not technical
There is a well-tested finding underneath this, and it is older than most of the tools involved.
In 1968 Conway observed that a system's structure ends up mirroring the communication structure of the organisation that built it. That sat as folklore for decades until MacCormack, Rusnak and Baldwin tested it properly, comparing products built by loosely-coupled and tightly-coupled organisations. They found strong support: in every pair they examined, the loosely-coupled organisation produced a significantly more modular product.
Microsoft went further in a 2008 study of Windows Vista, measuring organisational structure against defects. Their finding was blunt. The organisational measures were statistically significant predictors of how failure-prone a component would be.
So the shape of your organisation is not context around the architecture. It is one of the inputs that determines whether the architecture works. An answer that does not have it is missing a variable, not a nicety.
This is a different gap from the one in the last post. There I was talking about what you cannot change: your orchestrator, your language, what your team knows how to write. That is about whether you can build the thing. This is about whether you can still be running it in two years, which is a separate question with different inputs.
🧠 Why only seniors can supply this
Nobody else on the team has the information.
A mid-level engineer knows their own workload. They do not know that the platform team is losing two people in Q1, that the Flink migration was already attempted in 2023 and abandoned, or that your director has told you no new headcount until the reorganisation settles. You know all of it, and you are the only one who does.
The reason it does not get written down is that it does not feel like part of a technical question. When you ask about handling late-arriving events, the on-call rotation seems like a separate topic, filed under management. So you supply every technical constraint you have and none of the ones that will actually decide this.
There is a second reason and it is more awkward. Some of these facts are uncomfortable to state plainly. "One person understands this system and she may leave" is a real fact about your architecture and it also reads as an admission. It is easier to describe the technology than to describe the situation.
✅ Three lines about running it, not building it
On call: 4 engineers, weekly rotation, ~2 pages/week now
Bus factor: 1 person understands the streaming layer
Slack: no capacity for a new system until Q2 at the earliest
On call is the most useful line and the one people never write. It converts an architecture question into an operations question, which is what it becomes about a month after launch. A design that adds three new alerting surfaces to a rotation already running at two pages a week is a different proposal from the same design on a quiet rotation.
Bus factor changes recommendations more than almost anything else. An answer that is correct when three people understand the system may be reckless when one does. It moves the right answer toward boring technology and away from the thing that is technically better.
Slack is the line that stops you receiving a plan for a team that does not exist. Not headcount. Whether the people you have could take on another system without dropping something.
Notice none of these mention skill. That is covered by asking what your team knows, and it is a different question. Skill is what they can write. This is what they can carry.
One thing worth adding to important questions: ask what this design would cost to operate, separately from what it costs to build. That is not a normal question and you will not get the answer unless you ask it. It is where most of these decisions actually get made.
⚖️ The objection worth taking seriously
The weak version is "that is a people problem, not an architecture problem." The mirroring research answers it: the organisation predicts the defects, so it is an architecture input.
The strong version is harder. State your limits and you will get advice that works around them, forever.
There is a measured behaviour behind this. Models tend to accept a user's premises rather than challenge them, which researchers call framing sycophancy. So "we have one Flink engineer and no headcount" produces the best available design for a team with one Flink engineer and no headcount. What it will not produce is the sentence you might actually need, which is that you are attempting something your team is not staffed to run, and that the answer is a hiring decision rather than a design decision.
You are also the person who could change that. A mid-level engineer stating the same constraints is describing weather. A lead doing it is describing something they have some power over. Encoding your organisation's current limits into every technical decision is one way a team stays exactly the size and shape it is, permanently, while the architecture quietly bends to fit.
The Conway's Law literature has a line about this that is worth keeping. When you are fixing a broken organisation or a broken architecture, fix the organisation first, because the architecture will follow it either way.
So the honest version of the practice is two questions rather than one. Ask the constrained version, because you have to ship something. Then ask separately: given this problem, what team would you need to run this properly, and how far is that from what I have? You will not always act on the answer. But that gap is the actual finding, and stating your constraints is how you make sure you never see it.
🎯 The takeaway
Three lines about running it:
1. On call. How many people, how often, how noisy already.
2. Bus factor. How many understand each part. One is a design input.
3. Slack. Whether anyone could take on another system this quarter.
Then ask separately what team this design assumes. The distance between that and your team is the real answer, and you will only get it if you ask for it on its own.
The architecture was correct. It was correct for an organisation you do not have, and nothing in it was going to tell you that.
Top comments (0)