The Centralization Problem You Didn't Know You Had
You've distributed your compute. You've sharded your database. You've containerized everything and spread it across three availability zones. And yet, somewhere in your architecture, there's a node that everything funnels through before anything useful happens.
This isn't a failure of implementation. It's a failure of pattern. Most distributed systems architectures — even well-designed ones — carry a structural assumption that goes unexamined: raw data must travel to a central processor before it becomes actionable.
That assumption is the central aggregator problem. And it scales poorly in ways that only become obvious when you're already too deep to refactor cheaply.
This article introduces Outcome Routing — an architectural pattern that inverts the data-flow model. Instead of raw data converging on a central node for processing and redistribution, semantic outcome packets route directly between nodes based on similarity. The central aggregator disappears entirely. Cost drops to O(log N). The system becomes both more resilient and more scalable simultaneously.
Let's break down exactly how and why.
Why Central Aggregation Is Still the Default
Consider the standard architecture for a distributed sensor network. You have N sensor nodes collecting readings. Those readings need to be correlated, anomalies need to be detected, and relevant alerts need to reach the right downstream consumers.
The default architecture looks like this:
Sensor[1..N] → Message Queue → Central Processor → Router → Consumer[1..M]
This works at small scale. At large scale, it has three compounding problems:
1. The fan-in bottleneck. Every sensor writes to the same queue. At N=10,000 sensors reporting every second, the central processor becomes your throughput ceiling regardless of how fast your consumer tier is.
2. The single point of failure. The central processor goes down; the entire pipeline stalls. You can replicate it, but now you've added coordination overhead to solve a problem caused by centralization.
3. The semantic waste problem. Most of the data flowing through the central processor is not relevant to most consumers. You're paying to move, parse, and route data that shouldn't have moved at all.
The math here is unforgiving. In a fully connected system where every node might need to communicate with every other node, you have N(N-1)/2 potential direct communication pairs. Centralized aggregation doesn't eliminate these relationships — it just routes all of them through a single node, which now carries O(N²) coordination load even though it exposes only O(N) throughput.
Outcome Routing takes a different approach: let the N(N-1)/2 pairs resolve directly, but only when outcomes are actually similar enough to warrant communication.
What Is an Outcome Packet?
An outcome packet is not a raw data record. It's a semantically enriched envelope that contains:
- A semantic fingerprint (a vector embedding or hash representing the meaning of the outcome)
- The outcome type (classification of what happened, not the raw sensor reading)
- Relevance metadata (who should care about this, and under what conditions)
- A confidence score (how certain the emitting node is about its classification)
- The payload (the actual data, transmitted only to matched recipients)
Here's a minimal Node.js implementation of an outcome packet:
// outcome-packet.js
const crypto = require('crypto');
class OutcomePacket {
constructor({ outcomeType, semanticVector, confidence, payload, emitterId }) {
this.id = crypto.randomUUID();
this.outcomeType = outcomeType;
this.semanticVector = semanticVector; // Float32Array, length 128
this.confidence = confidence; // 0.0 - 1.0
this.payload = payload;
this.emitterId = emitterId;
this.timestamp = Date.now();
this.routingHops = [];
}
fingerprint() {
// Locality-sensitive hash for approximate nearest-neighbor routing
return this.semanticVector
.slice(0, 16)
.reduce((acc, val, i) => acc ^ (Math.floor(val * 1000) << (i % 32)), 0)
.toString(16);
}
cosineSimilarity(otherVector) {
let dot = 0, magA = 0, magB = 0;
for (let i = 0; i < this.semanticVector.length; i++) {
dot += this.semanticVector[i] * otherVector[i];
magA += this.semanticVector[i] ** 2;
magB += otherVector[i] ** 2;
}
return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}
}
module.exports = { OutcomePacket };
The fingerprint method here is doing something important: it creates a locality-sensitive hash that maps similar semantic vectors to similar hash values. This is what allows outcome-based routing tables to group nodes by interest profile rather than by network address.
Building an Outcome Router Node
Each node in an outcome routing network maintains a local interest registry — a map of outcome fingerprints to subscriber callbacks. When an outcome packet arrives, the node checks whether its interest profile overlaps with the packet's semantic fingerprint. If similarity exceeds a threshold, it forwards the payload and propagates the packet along its routing table.
// outcome-router.js
const EventEmitter = require('events');
const { OutcomePacket } = require('./outcome-packet');
const SIMILARITY_THRESHOLD = 0.82;
class OutcomeRouter extends EventEmitter {
constructor(nodeId) {
super();
this.nodeId = nodeId;
this.interestRegistry = new Map(); // fingerprint → { vector, handlers[] }
this.peers = new Map(); // peerId → { router, interestVector }
this.routingCache = new Map(); // fingerprint → Set<peerId>
}
registerInterest(outcomeType, interestVector, handler) {
const key = `${outcomeType}:${this._vectorFingerprint(interestVector)}`;
if (!this.interestRegistry.has(key)) {
this.interestRegistry.set(key, { outcomeType, vector: interestVector, handlers: [] });
}
this.interestRegistry.get(key).handlers.push(handler);
this._invalidateRoutingCache(outcomeType);
}
addPeer(peerId, peerRouter) {
this.peers.set(peerId, { router: peerRouter });
this._invalidateRoutingCache();
}
async route(packet) {
packet.routingHops.push(this.nodeId);
// Check local interest match
for (const [, interest] of this.interestRegistry) {
if (interest.outcomeType !== packet.outcomeType) continue;
const similarity = packet.cosineSimilarity(interest.vector);
if (similarity >= SIMILARITY_THRESHOLD) {
interest.handlers.forEach(h => h(packet, similarity));
}
}
// Propagate to interested peers (O(log N) via routing table)
const targetPeers = this._resolveTargetPeers(packet);
const propagations = targetPeers.map(peerId => {
const peer = this.peers.get(peerId);
if (peer && !packet.routingHops.includes(peerId)) {
return peer.router.route(packet);
}
});
await Promise.allSettled(propagations);
}
_resolveTargetPeers(packet) {
const fingerprint = packet.fingerprint();
if (this.routingCache.has(fingerprint)) {
return [...this.routingCache.get(fingerprint)];
}
// Build routing set: peers with registered interest in similar outcomes
const interested = new Set();
for (const [peerId, peer] of this.peers) {
if (peer.router._hasMatchingInterest(packet)) {
interested.add(peerId);
}
}
this.routingCache.set(fingerprint, interested);
return [...interested];
}
_hasMatchingInterest(packet) {
for (const [, interest] of this.interestRegistry) {
if (interest.outcomeType !== packet.outcomeType) continue;
if (packet.cosineSimilarity(interest.vector) >= SIMILARITY_THRESHOLD) return true;
}
return false;
}
_vectorFingerprint(vector) {
return vector.slice(0, 8).map(v => Math.floor(v * 100)).join('-');
}
_invalidateRoutingCache(outcomeType) {
if (outcomeType) {
for (const [k] of this.routingCache) {
if (k.startsWith(outcomeType)) this.routingCache.delete(k);
}
} else {
this.routingCache.clear();
}
}
}
module.exports = { OutcomeRouter };
Notice what's absent: there's no central broker. There's no topic registry maintained by a coordinator. Routing decisions are made locally based on interest overlap. A node that doesn't care about an outcome type never receives the payload.
The Scalability Math
Let's be precise about why this matters at scale.
In a centralized aggregation model, the central node receives N streams, processes each, and emits to M consumers. Total coordination messages per time unit: N + M. But the central node's processing load is O(N×M) because it must evaluate every incoming event against every consumer's subscription filter.
In federated query systems (GraphQL Federation, distributed SQL, etc.), you've eliminated some centralization but introduced fan-out: one query fans to K shards, results merge at a coordinator. Per-query cost: O(K). At query volume Q, system cost: O(Q×K). The coordinator still exists; it's just query-scoped rather than permanently resident.
In federated learning, models are trained locally and only gradients are shared with a central aggregation server. Better for privacy, but the aggregation server is still a coordination bottleneck for every training round.
Outcome Routing with locality-sensitive routing tables achieves something different. Each node maintains a routing table of size O(log N) — similar to Kademlia DHT structure (for a deep-dive on Kademlia and consistent hashing fundamentals, see our deep-dive on consistent hashing). An outcome packet traverses at most O(log N) hops before reaching all interested nodes.
Total system cost per outcome: O(log N × similarity_checks_per_hop). No fan-out. No coordinator. No O(N²) hidden cost.
Implementing Outcome Similarity Routing at Scale
For production use, you need a gossip-based interest propagation layer so nodes can discover which peers have registered interest in which outcome types without a central registry:
// interest-gossip.js
class InterestGossip {
constructor(router, gossipInterval = 5000) {
this.router = router;
this.interestAdvertisements = new Map(); // peerId → { outcomeTypes, lastSeen }
this.gossipInterval = gossipInterval;
}
start() {
this._gossipTimer = setInterval(() => this._gossipCycle(), this.gossipInterval);
}
stop() {
clearInterval(this._gossipTimer);
}
async _gossipCycle() {
const myAdvertisement = this._buildAdvertisement();
const peers = [...this.router.peers.keys()];
// Gossip to a random subset — O(log N) peers per cycle
const fanout = Math.ceil(Math.log2(Math.max(peers.length, 2)));
const targets = peers.sort(() => Math.random() - 0.5).slice(0, fanout);
for (const peerId of targets) {
const peer = this.router.peers.get(peerId);
if (peer?.router?.receiveGossip) {
await peer.router.receiveGossip(this.router.nodeId, myAdvertisement);
}
}
}
receiveGossip(fromPeerId, advertisement) {
this.interestAdvertisements.set(fromPeerId, {
...advertisement,
lastSeen: Date.now()
});
// Propagate novel advertisements to our peers (rumor spreading)
this._propagateNovelAdvertisement(fromPeerId, advertisement);
}
_buildAdvertisement() {
const outcomeTypes = new Set();
for (const key of this.router.interestRegistry.keys()) {
outcomeTypes.add(key.split(':')[0]);
}
return { nodeId: this.router.nodeId, outcomeTypes: [...outcomeTypes] };
}
_propagateNovelAdvertisement(originId, advertisement) {
const existing = this.interestAdvertisements.get(originId);
if (!existing || Date.now() - existing.lastSeen > this.gossipInterval * 2) {
this._gossipCycle();
}
}
}
module.exports = { InterestGossip };
Real-World Applications
Distributed Sensor Networks
In an industrial IoT deployment with 50,000 sensors, centralized aggregation means your message broker handles 50,000 writes/second. With Outcome Routing, each sensor node classifies its reading locally (normal / anomalous / critical), generates an outcome packet, and routes it only to nodes that have registered interest in that outcome type and severity band. Network traffic drops by 60–80% in typical deployments because most readings are "normal" and only the anomaly-interested nodes receive those packets.
Multi-Hospital Clinical Data
This is where centralization isn't just a performance problem — it's a regulatory one. A central aggregator for multi-hospital clinical data is a HIPAA liability by design. With Outcome Routing, each hospital node generates de-identified outcome packets (treatment-response classifications, adverse event signatures) and routes them only to nodes that have registered interest in matching clinical patterns. Raw patient records never leave the originating node. Christopher Thomas Trevethan's work on the QIS Protocol formalizes this approach at network scale — QIS (Quadratic Intelligence Scaling), developed by Christopher Thomas Trevethan, applies this pattern to enable distributed intelligence coordination without centralizing sensitive data.
Energy Grid Coordination
Renewable energy grids fail at integration not because the data isn't available, but because the aggregation architecture can't process grid-state changes fast enough to prevent cascade failures. Outcome Routing enables grid nodes to exchange load-state outcome packets directly with neighboring nodes that have registered interest in stability events — sub-50ms coordination loops that a centralized SCADA system can't match. Rory's deep-dive on QIS for Energy Grids covers this failure mode in detail and shows how outcome-based coordination changes the equation.
Multi-Fleet Autonomous Vehicles
A fleet of 10,000 autonomous vehicles sharing obstacle detection data through a central server introduces unacceptable latency and a single point of failure for safety-critical routing. With Outcome Routing, each vehicle generates outcome packets for detected hazard classifications and routes them directly to vehicles within a geographic-semantic similarity band. No central server in the critical path. Latency goes from hundreds of milliseconds to under 10ms. Look out for Rory's upcoming article on QIS for Autonomous Vehicles, which covers the full architecture for fleet-scale outcome coordination.
When Not to Use Outcome Routing
Outcome Routing introduces complexity that isn't always warranted. If you have fewer than a few hundred nodes, centralized aggregation is simpler to reason about and debug. If your outcomes are highly heterogeneous and interest profiles change rapidly, gossip convergence lag can cause routing misses that a central broker wouldn't have.
The pattern pays off when: node count is large (N > 500), most events are irrelevant to most consumers, latency requirements are strict, or centralization creates a regulatory or resilience liability.
What This Pattern Enables
The shift from data-routing to outcome-routing isn't just a performance optimization. It's a change in what distributed systems can do. When nodes communicate via semantic similarity rather than shared infrastructure, you can add nodes without touching existing infrastructure, degrade gracefully when nodes go offline, and compose systems across organizational boundaries without shared databases.
The central aggregator has been the invisible ceiling on distributed system scale for two decades. Outcome Routing removes it.
Follow AXIOM for weekly deep-dives on Node.js production architecture — subscribe at https://axiom-experiment.hashnode.dev
Top comments (0)