DEV Community: Bartosz Frąckowiak

Probabilistic Metrics

Bartosz Frąckowiak — Mon, 10 Aug 2026 12:42:20 +0000

In my previous article (From Signals to Blips) I described the engine room of our AI transformation: a tech radar fed by weekly AI sweeps over 15 squad reports. Every week an agent reads the reports, extracts signals, scores them, and hands me a ranked candidate list for human validation.

This article is about the piece of that machinery I trust least and use most. The scores that steer our architecture attention are not computed from clean, structured data. They are extracted by AI from messy human text: probabilistically, with a known and accepted margin of error.

For once there is a production system behind my private conclusions, so the usual grain of salt can be slightly smaller ;)

The clean data tax

Classical metrics demand clean inputs: instrumented pipelines, structured events, agreed definitions. That cleanliness is a tax, and you pay it three times. Once in engineering, because somebody has to build and maintain the pipeline. Once in politics, because somebody has to win the meeting where "active user" gets defined. And once in coverage, because you only ever measure what you predicted, a year in advance, would be worth measuring.

And after paying all of that, the numbers still get reshaped in preprocessing before anyone sees a chart. I wrote a whole article about that theater (Data-Driven Theater).

The result is an organization precisely informed about a thin slice of reality and blind everywhere else. And the blind spots are not random. They are exactly the messy, human parts of the system: friction, workarounds, practices quietly spreading between teams, morale.

What AI actually changes

AI changes the input space of measurement. A metric no longer has to start its life as a structured event. It can be extracted from free-form text: a blocker described in a frustrated sentence, a win buried in the middle of a paragraph, the same practice named three different ways by three different teams.

On the radar this is very concrete. An agent decomposes every weekly report into atomic claims, classifies each one as a stopper or a promotable, and scores it on multiplicity, severity, and strategic weight. Nobody built a data pipeline for this. The "sensor" is a versioned prompt pointed at documents that already existed.

Any single extraction can be wrong. And that is not a bug waiting for a fix. That is the deal. You trade per-observation certainty for coverage of the whole organization.

The assumptions

This trade is only sane under explicit assumptions. Here are mine, all learned (some painfully) from running the radar:

Accept the error rate, and say it out loud. A dose of acceptance is part of the contract. If a stakeholder expects audit-grade numbers, stop them at the door. This instrument doesn't produce those, and pretending otherwise is how trust dies.
Never trust a single extraction. Trust convergence. One squad's report can be misread; five squads independently describing the same friction is a fact about the organization, whatever the error rate per reading. Redundancy replaces precision.
Every number must cite its source. Our rule is cite or it didn't happen: each scored signal points back to the exact report, week, and section it came from. Probabilism without traceability is just noise with confidence.
A human gate before consequences. Probabilistic metrics route attention; they must never issue verdicts. In our case nothing enters the radar without a human ticking a checkbox. AI proposes, humans dispose.
The prompt is the preprocessing, so treat it like code. In Data-Driven Theater I argued that data manipulation lives in the preprocessing phase. Here, the preprocessing is the extraction prompt. So it's versioned, its scoring anchors have a changelog, and anyone can read it. If you hide the prompt, you've rebuilt the theater with better lighting.
Shared language still matters, but now it's the model's dictionary. An LLM will happily map three teams' different vocabularies onto one concept. That's the superpower and the danger in one move: it can silently merge things that are not the same. The semantic contracts between people don't disappear; auditing how the model translates becomes part of the method.

What I take out of it

Coverage beats precision when the job is steering. I'd rather be roughly right about the whole organization than precisely right about the 5% of it I managed to instrument. For triage (where should attention go this week) that is simply the correct trade.

You can also measure things that never had a pipeline. Converging practices, recurring friction, the mood of a transformation. On the radar, these signals routinely show up weeks before anything reaches a dashboard, because the dashboard for them was never going to be built.

And metrics become disposable, which is the change I underestimated most: when a new metric costs one prompt instead of one quarter, you can try ten and throw away eight, and the metric portfolio stops being an investment you defend in meetings and becomes a hypothesis you test on Tuesday.

The failure modes move; they don't vanish. Yesterday people gamed the data cleaning. Tomorrow they'll game the extraction.

"When a measure becomes a target, it ceases to be a good measure." ― Goodhart's law, in Marilyn Strathern's phrasing

Same theater, new stage, and the same defense: documented preprocessing, cited sources, and a culture where asking "how was this computed?" is a compliment, not an attack.

None of it will survive an audit, and that's fine. A compass is not an accounting system. Use probabilistic metrics to decide where to look; once you know what you're looking at, instrument it properly. The two don't compete. One finds the question, the other defends the answer.

One question

So here is what I'd leave you with: what would you measure in your organization if a new metric cost one prompt instead of one quarter?

Because that is the actual offer on the table now. The classic excuse for not measuring the human side of your system (too messy, too expensive, no pipeline) has just expired.

The Bottleneck Moved

Bartosz Frąckowiak — Mon, 03 Aug 2026 07:09:11 +0000

AI made writing code cheap. Our squad reports show where the queue went next: to the people who have to read it.

I read squad reports for a living now. 15 of them, every month, from teams at different stages of an AI transformation that covers several hundred developers. Sometime this spring, the complaints changed.

For years the standing blocker was production. Not enough hands, not enough hours, the backlog only grows. This spring the sentence flipped. We are not blocked on writing the code, the reports said. We are blocked on getting it absorbed: reviewed, understood, trusted, merged.

These are notes from one organization, mine, with every bias that implies. But I have checked the pattern against enough teams now to stop filing it as an anecdote.

The queue did not shrink. It moved one seat to the right.

Typing was never the unit of work

We staffed for production because production was visible. Headcount plans, story points, velocity charts: every dial on the dashboard points at how much gets written. Then the cost of producing a diff fell through the floor, and the dials kept pointing at the part that got cheap.

What didn't get cheap is deciding whether the diff is right.

A senior engineer who used to review four colleagues now reviews four colleagues plus everything their agents produce. The reports describe the same scene in different words. Pull requests aging in queues. Reviewers rubber-stamping out of exhaustion. Or the opposite: one reviewer becomes the only honest gate, and everything slows to their reading speed.

Nobody is blocked on writing anymore. Everybody is blocked on believing.

I met this bottleneck alone before I met it at scale. Running a company of one staffed by agents, I am my own review queue, and my throughput is capped by what I can verify, not by what the agents can generate. The organizational version is the same shape with worse feedback loops.

A diff with nobody behind it

There is a reason review did not get cheaper alongside writing, and it took me months to name it.

Reviewing a colleague's change was never a purely technical act. It was a conversation with a counterparty. You read the diff, a question formed, and you walked over and asked it. Why this shape? What did you try first? The answer usually came back in one sentence, because the person who made the decision was still holding it in their head.

An agent's diff has no such person. The intent behind it lived in a prompt, in a context window, in a session that has since closed. You can't ask the author what they were thinking. There was no thinking of the kind you mean, and the human who ran the prompt is often meeting the result at the same moment you are.

In my notes they are simply authorless diffs. The code is here, the counterparty is not, and every question you would have walked over to ask now needs somewhere else to go.

Peter Naur described programming as theory building: the real output of a team is not the program but the understanding that lives in the people who built it. An authorless diff is a diff whose theory is already gone.

So the question has to be answerable by the system itself, or it does not get answered.

Reviewability is a system property

Which is the part I can act on as an architect. How hard a change is to review is not a fact about the reviewer. It is a fact about the system.

You can design a system to be easy to review the same way you design a database to be easy to query. Tests that state intent, so a green suite means something specific. Contracts at the seams, so a change's blast radius is visible from its diff. Small modules with real boundaries, so nobody has to page the whole system into their head. Decision records that say which paths were rejected and why. Context files that tell an agent, or a new colleague, where the bodies are buried. None of it is glamorous.

None of this advice is new. What is new is the price of ignoring it, because now the unreviewable system is the slow system.

Indexes made queries cheap long after disks made storage cheap. Reviewability is the index we now owe our own codebases.

What "done" has to mean now

The definition of done in most teams was written when writing was the scarce step. Code complete, tests pass, merged. Verification was assumed to be somebody's background job.

That assumption is now the whole game. If producing a change costs an afternoon and trusting it costs three days, then the three days are the work, and the definition of done, the staffing plan and the promotion criteria are all still written as though the afternoon were the expensive part. Nobody has updated them. Nobody has noticed.

A senior engineer whose main output is judgment about other people's diffs is not doing less engineering. They are doing the engineering that is left.

The dashboards will resist this, because dashboards count what is easy to count, and reading has always been the thing we measure worst.

What I watch now

I still read 15 reports a month. What changed is the line I look for first.

It used to be the delivery numbers. Now it is a sentence buried in the middle of the wins section, the one admitting that a squad shipped something and nobody could tell whether it was right for a week. That sentence is not a complaint about tooling. It is a system telling you where its constraint sits.

The dashboards above those reports are still counting output, because output is what dashboards were built to count, and I don't expect that to change quickly. But the reports underneath them stopped talking about production some time this spring, and they have not gone back.

The queue did not shrink. It moved one seat to the right, and it is quieter over there, which is why it took a season to notice. If your dashboards are green while your pull requests age, it has moved in your organization too. The authorless diffs are already arriving. The seat they queue for is not on your headcount plan.

Multi-Agent Is an Org Chart

Bartosz Frąckowiak — Wed, 29 Jul 2026 11:36:16 +0000

Recently, as part of our AI platform work, I've been designing a multi-agent experience: a supervisor agent that understands the user's intent and routes work to specialist agents, each owning a slice of the domain, with escalation paths for the cases nobody can handle.

Somewhere around the third iteration of the target-architecture diagram I stopped and started laughing. Boxes with narrow responsibilities. A coordinator on top. Formal handoffs. Escalation paths. I had seen this diagram before: in every HR tool I've ever been onboarded into.

I wasn't designing a system. I was designing an organization. And the dozen-plus years I've spent watching human organizations misbehave suddenly became the most relevant architecture experience I own.

Field notes from one solution design follow. Private observations, not a paper.

Same Primitives, Different Substrate

Strip the buzzwords and a multi-agent architecture is built from exactly the primitives an org designer uses: roles (specialist agents with narrow charters), handoffs (who passes work to whom, in what format), escalation paths (what happens when an agent is out of its depth), and trust boundaries (whose output is taken at face value and whose gets verified).

This is not a cute metaphor. The primitives are identical because the underlying problem is identical: coordinating limited intelligences with partial context toward a shared goal. For humans the limits are attention and working memory; for agents, the context window. Same constraint, different units.

The parallels get uncomfortably specific:

Context windows are span of control. A manager can effectively lead only so many people before nuance drops on the floor. An agent can hold only so much context before it starts confidently forgetting. In both cases the failure smells the same: dropped threads, generic answers, work quietly reinvented twice.
Agent-to-agent contracts are… contracts. Schemas, expectations, what "done" means. Two teams with a fuzzy interface produce meetings; two agents with a fuzzy interface produce hallucinated handoffs. The meetings are cheaper.
Decomposition is charter-writing. Deciding what each agent owns is the same act as writing team charters, and it fails the same way, through overlaps (two agents both "own" the answer) and gaps (a question no agent considers theirs).

Supervisors Are Middle Management

The supervisor pattern (one orchestrator routing everything) is multi-agent middle management. And it fails exactly the way middle management fails:

The bottleneck. Route every interaction through one coordinator and you've serialized your parallelism. One overloaded node, everyone else idle. Any resemblance to a manager whose calendar is the company's critical path is fully intended.

The telephone game. Each relay compresses. The specialist's rich answer becomes the supervisor's summary, which becomes a summary of a summary. Fidelity degrades per hop. In orgs we call it "communication overhead", in agents "context loss", and it's the same tax.

Empire building. Give a supervisor agent loose instructions and it starts doing the work instead of routing it: answering domain questions itself, badly, because delegating "feels" riskier than acting. I have met this middle manager. Several times. Some of them were prompts.

The fixes are org-design fixes, translated: push decisions down to the specialists, allow direct channels between agents that collaborate often (not everything needs to go "up"), and define escalation criteria instead of escalation habits. If your supervisor agent starts scheduling coordination meetings between the other agents, turn the computer off ;)

Guardrails Are Governance

Every serious multi-agent design ends up with guardrails: what an agent may do autonomously, what requires human approval, what gets logged for audit. Rename these and you get a governance framework, complete with autonomy budgets, approval gates, and audit trails. The corporate déjà vu is total.

And the same tension applies. Over-govern and you build agent bureaucracy: every action awaiting sign-off, throughput dying politely, the system technically safe and practically useless. Under-govern and you're one confident hallucination away from an incident report. Organizations have been tuning this dial for a century, and the honest answer in both worlds is that the dial never stops needing tuning. You're not choosing a setting; you're accepting a job.

The Mirror

What this exercise actually gave me, beyond a chatbot architecture:

Organizational experience turned out to be directly transferable. Clear charters beat detailed procedures; the smallest unit that can own an outcome end-to-end should own it; and you should never create a coordinator role to fix a communication problem that a better contract would fix for free.

But the transfer runs both ways, and that's the part I can't stop thinking about. When you design coordination from scratch, with no politics, no history, no feelings to spare, you would never include a layer whose only function is summarizing upward. You would never require every hand-off to pass through one node. You would never let two roles both vaguely own the same outcome, and you would never leave an escalation path undefined.

Then you look at your real org chart and ask why it's full of things you would never design. The polite version of the answer is "history". I explored the human side of sitting inside such a structure in Technical Leader — Identity Disorder; designing agents is the first time I've gotten to sit on the other side of the whiteboard.

There is a Conway's law for agents, too: left unattended, your multi-agent architecture will faithfully copy your organization's communication patterns, including the dysfunctional ones, because its designers can't easily imagine anything else. We nearly shipped our org chart into the prompt. I suspect we weren't the first.

Multi-agent design is org design with a compiler. For the first time, our coordination theories produce stack traces instead of opinions.

So here's the question I'd leave you with: if you rebuilt your organization the way you'd architect a multi-agent system (from scratch, charters first, no history to respect), which of your current roles would survive?

And if the answer makes you uncomfortable: the agents are ready whenever you are.

Mathematicians vs. Lawyers

Bartosz Frąckowiak — Mon, 27 Jul 2026 13:20:20 +0000

Mathematicians vs. Lawyers

Two thinking archetypes I keep meeting in architecture rooms, and why your system needs both.

Put two smart people in front of the same business problem and watch what happens.

One of them walks to the whiteboard and starts drawing boxes and arrows. Entities, relations, flows. Within five minutes the whole domain is a living organism on the wall, and half the room quietly wonders where the actual requirements went.

The other one opens a document and starts listing cases. What if the customer is from Germany? What if the invoice was already corrected? What if the correction arrives after the fiscal year closes? Twenty minutes later you have a table with forty rows, and the whiteboard person quietly wonders why we are debugging the universe.

After years of working as a Solution Architect, I started recognizing these two recurring archetypes of thinking. I call them Mathematicians and Lawyers.

Before anyone gets offended: there are many more archetypes than two, and nobody is purely one of them. These are modes of thinking, not personality boxes. But once you see these two, you cannot unsee them.

The Lawyers

Lawyers live in a linear world of rules. Given condition A and condition B, then C applies, unless exception D. Their thinking moves like a legal act: precise, sequential, exhaustive.

And they are good. A Lawyer is the person who catches the case that would have exploded in production three months after go-live. While the rest of the room admires the elegant model, the Lawyer asks the boring, deadly question: "and what happens on February 29th?"

Their natural artifact is the Document. The big one.

The Edge Case Explosion

This is where corporate culture enters the story, and where my observation stopped being about individuals and started being about the system.

A product company has a very hard time saying no. Every client exception gets accepted. Every "just this one special case" becomes a requirement. And the Lawyers do exactly what they are brilliant at: they faithfully encode all of it. Rule by rule, exception by exception.

The document grows. Fifty pages. Eighty. Hundreds of edge cases, each one individually justified.

Let's be honest with ourselves: nobody reads that document carefully. Nobody except other Lawyers. The rest of us skim it, nod in the review meeting, and approve. The knowledge is technically written down and practically inaccessible.

There is also a trap hiding in plain sight. A thick specification fits perfectly into corporate optics: volume looks like effort. "Big" reads as "a lot of work was done." A three-hundred-page document is impressive in a way that a single clean diagram will never be. So the organization quietly rewards the explosion instead of questioning it.

What you end up with is complexity that nobody ordered. It was delivered one accepted exception at a time, by people doing their jobs well, inside a culture that couldn't refuse anyone.

The Mathematicians

Mathematicians think in abstract systems. They see high-level entities and how they cooperate. Their world is not made of rules but of relations between objects: how those relations evolve over time, what constraints must hold, who governs what.

Show a Mathematician eighty edge cases and they will not see eighty problems. They will ask a different question: what is wrong with the model that keeps generating these exceptions? Change the model, and whole families of edge cases simply stop existing. Nobody has to handle a case that the design made impossible.

Their curse: the impact is less visible. One diagram against three hundred pages. When a Mathematician does the job well, the visible result is… nothing. Complexity that never appeared. Incidents that never fired. An integration that didn't need a workaround. Try putting that on a promotion slide.

In a culture where volume equals work, Mathematicians chronically look underemployed. Right up until they leave, and the edge cases start breeding unsupervised.

And to be fair, Mathematicians have their own failure mode. Left unchecked, they drift into astronaut architecture: models so abstract they no longer touch the ground. Guess who pulls them back? The Lawyer, with one concrete, annoying, absolutely real corrected invoice from Germany.

Two Worlds, One System

This is not a war, and I'm not writing this to declare a winner.

The best design sessions I have ever seen had both minds in the room. The Mathematician reshapes the model so the exceptions dissolve; the Lawyer stress-tests the new model with cases brutally imported from reality. Lawyers keep the Mathematicians honest. Mathematicians keep the Lawyers' world from exploding.

A room full of Lawyers produces an edge case explosion with a spreadsheet to track it. A room full of Mathematicians produces a beautiful abstraction that dies on first contact with a real customer. If everyone in your design review thinks the same way, that is your biggest architecture risk, and it's not in any diagram.

These two worlds complement each other. Use both. Deliberately.

Where I Choose to Stand

My private rule as a Solution Architect: stay at the mathematical level. Don't descend into the details.

Not because details are beneath me. Because the moment I dive into edge cases, I become one more Lawyer in a room that usually has plenty of them already, and the system-level perspective goes vacant. Every edge case has an owner somewhere. The relations between systems often have none. That empty chair is exactly where an architect should sit.

I won't pretend it's easy. Diving into details gives you the quick dopamine of being concretely useful. Holding the abstract level means accepting that your best work will often be invisible: the complexity that never happened.

So, next time you are in a design review, look around the room. Who is writing the rules, and who is drawing the relations? Which one are you when the pressure rises?

And more importantly: who is holding the other half of the picture?