Rory | QIS PROTOCOL

Posted on Apr 15 • Originally published at qisprotocol.com

QIS Routing for Distributed Radio Telescope Data Synthesis: N(N-1)/2 Scaling for VLBI and SKA

#distributedsystems #radioastronomy #ai #architecture

The VLBI correlation problem has always been quadratic.

N telescope stations generate N(N-1)/2 baseline pairs. The Event Horizon Telescope with 10 active stations produces 45 baselines. The Square Kilometre Array at full build — 197 dishes in South Africa plus 131,072 dipole antennas in Australia — approaches 2 million baseline pairs for its interferometric products.

The current architecture handles this by centralising everything: raw visibilities are recorded at each station and shipped (physically, on hard drives at EHT scale; via high-speed fibre at SKA scale) to a central correlator. DiFX at the Max Planck Institute for Radio Astronomy processes the EHT. The SKA Science Data Processor in South Africa will handle SKA-Mid. The synthesis happens at one facility. The individual telescope nodes are, in an architectural sense, leaf nodes: they observe, record, and transmit. They do not synthesise with each other.

This architecture is not wrong. It has produced science of extraordinary precision — including the first direct images of a supermassive black hole. But as VLBI networks grow denser, as SKA enters science operations, and as the per-node intelligence at each telescope station becomes richer and more diverse, a question worth examining now is whether the centralised correlator model captures all available intelligence — or whether there is a complementary layer that the current architecture leaves on the table.

The answer, from the perspective of QIS Protocol, is that there is.

The Correlation Bottleneck: What Is Actually Being Centralised

It is worth separating two distinct things that happen at the central correlator.

Thing 1: Raw visibility correlation. The cross-multiplication of the raw baseband signals from every station pair to produce visibility data — the actual interferometric measurement. This requires the raw data. It must happen centrally or in a tightly coordinated distributed facility with direct access to the raw streams. DiFX, SFXC, and the SKA CSP all do this.

Thing 2: Knowledge synthesis. What the network learns from each observation about calibration solutions, ionospheric conditions, RFI environments, baseline-specific systematics, and atmospheric models. This does not require raw visibilities. It can be expressed as validated outcome packets — structured summaries of what a node learned — and routed between stations in near-real-time.

The current architecture centralises both. The first centralisation is a physics requirement. The second is an architectural choice — and it creates an intelligence gap.

The Intelligence Gap in VLBI Networks

Consider a specific scenario.

The IRAM 30m telescope at Pico Veleta, Spain completes a session in the 3mm band at 23:00 UTC. During calibration, the pipeline identifies a previously undocumented instrumental polarisation leakage pattern in the Q-U plane, correlated with an azimuth range between 120° and 180° degrees. This pattern affects visibility amplitudes on baselines to IRAM at roughly the 0.8% level — below the threshold for automated flagging but above the noise floor for precision science.

Under the current architecture, this calibration insight lives in the IRAM calibration tables. It gets uploaded to the central calibration archive when the drive ships. It becomes available to other stations when the pipeline runs and the solutions are propagated — typically on the timescale of weeks to months for cycle validation.

The SPT station at the South Pole, preparing for tonight's observation window with overlapping source coverage, does not have this information. Neither does the JCMT in Hawaii or ALMA in Chile. They will potentially encounter the same instrumental signature on IRAM baselines in tonight's session and either flag data unnecessarily or pass it through with uncorrected systematics.

This is not a data quality problem. It is a routing architecture problem. The intelligence existed. It was not routed.

What QIS Outcome Routing Adds

QIS Protocol — a discovery in distributed outcome routing by Christopher Thomas Trevethan (June 16, 2025; 39 provisional patents filed) — addresses exactly this category of problem.

The core architecture is a closed loop:

Raw signal → Each telescope node processes its own data locally. Raw visibilities never leave the node in this layer.
Local processing → Calibration pipelines, quality metrics, RFI characterisation, atmospheric modelling run at the edge.
Distillation → The node distils what it learned into an outcome packet: a structured ~512-byte summary containing validated deltas — calibration solution vectors, RFI mask indices, atmospheric model coefficients, baseline-specific quality flags. No raw data. No model weights. Only validated outcomes.
Semantic fingerprinting → The packet receives a semantic fingerprint: a vector encoding the observational context (source, frequency, epoch, baseline geometry, atmospheric conditions, instrument configuration).
Outcome routing → The packet is routed to a deterministic address defined by the fingerprint. Other nodes observing under semantically similar conditions can query this address and pull relevant packets.
Local synthesis → Each receiving node synthesises the incoming packets with its own local state. The synthesis happens at the edge. No raw data is exchanged. No central aggregator is required.
New packets generated → The synthesis produces new validated outcomes, which re-enter the loop.

The routing mechanism is protocol-agnostic: the loop works with DHT-based routing (O(log N) per query, fully decentralised), database semantic indices (O(1) lookup), vector similarity search, or any mechanism that maps a problem fingerprint to a deterministic address. The specific transport is an engineering choice, not a constraint of the discovery.

The breakthrough, as Trevethan's architecture shows, is that when you close this loop — when you route pre-distilled insights by semantic similarity rather than centralising raw data — intelligence scales as N(N-1)/2 while compute scales at most O(log N) per node. The same quadratic structure that makes the VLBI correlation problem hard at central scale becomes an advantage at the edge: every additional node doubles the synthesis opportunities for all other nodes, without increasing any single node's compute burden proportionally.

Why This Is Architecturally Distinct from Existing VLBI Calibration Systems

Existing VLBI calibration frameworks — CASA, AIPS, DiFX calibration tables, the SKA Science Data Challenge pipelines — are designed for the centralised model. They assume that all calibration data flows to a central facility before cross-station comparisons are possible.

Several distributed approaches exist for specific sub-problems:

e-VLBI real-time correlation: streams compressed baseband data over high-speed fibre (e.g., the European VLBI Network e-VLBI mode). Solves the physical media shipment problem, not the distributed synthesis problem. All data still goes to the central correlator.
Geodetic VLBI VGOS and IVS distributed post-processing: multiple analysis centres process the same data independently, then compare solutions. Consensus-building, not real-time routing.
SKA Regional Centres network: six international SRCs will cache and re-process SKA science data products. Designed for post-processing archival access, not real-time outcome routing between telescope nodes.
CASA and RASCIL scripted pipelines: calibration solutions are stored in tables and can be transferred between sessions, but there is no routing layer that matches calibration solutions to semantically similar observations across the network in real time.

QIS adds a layer none of these architectures include: real-time semantic routing of calibration and quality intelligence between nodes, without centralising raw data.

This is not a replacement for DiFX or the SKA CSP. It is a complementary protocol layer that operates above the raw correlation pipeline. The visibilities still go to the central correlator. What changes is that the intelligence about the telescope network — calibration solutions, atmospheric models, RFI environments, baseline-specific systematics — begins routing between stations in real time, compounding across the network as each session adds new validated outcomes.

The N(N-1)/2 Scaling Argument Applied to SKA

The Square Kilometre Array provides a concrete test case.

SKA-Mid will complete Phase 1 construction with 197 dishes (MeerKAT core plus SKA dishes) in South Africa. Germany signed its SKA construction contract in October 2025. Full SKA-Mid will eventually include 133 additional dishes for a total of 197 in the mid-frequency array.

For VLBI purposes, the relevant figure is the number of participating stations in the Global VLBI network that will incorporate SKA baselines: currently approximately 50 stations in the EVN and Global VLBI array, with SKA-Mid expected to add major new baselines in the southern hemisphere.

Under a QIS outcome routing layer:

50 VLBI stations = 50 × 49 / 2 = 1,225 calibration synthesis pairs
Each pair can exchange relevant calibration intelligence in near-real-time
Each station queries addresses semantically fingerprinted to its current observing context
Intelligence from the South African SKA-Mid array routes to European EVN stations observing the same sources, before the next scan begins

At SKA-Low (131,072 dipoles organised into 512 stations for correlation purposes):

512 stations = 130,816 synthesis pairs
Real-time atmospheric calibration intelligence from western Australian stations routes to eastern stations completing the same survey field minutes later

The compute cost per node: at most O(log N) per query, regardless of network size. The intelligence available per node: O(N) synthesis pairs, each contributing validated outcomes relevant to the node's current state.

This is the phase change in the QIS architecture. The correlation problem grows quadratically with station count. So does the synthesis intelligence available — but at logarithmic compute cost per node, not quadratic.

Integration Pathway

QIS is transport-agnostic and does not require modifications to existing VLBI software stacks.

A practical integration pathway for a VLBI network like the EVN or the future SKA VLBI mode:

Outcome packet schema for VLBI: Define a standard structure for VLBI calibration outcome packets — antenna gain solutions, bandpass calibration deltas, ionospheric TEC estimates, RFI mask updates, pointing corrections, baseline quality vectors. This is an engineering task, not an architectural constraint.
Semantic fingerprint definition: The similarity function is defined by domain experts — in Trevethan's terminology, this is the First Election (the hiring of the best expert to define similarity for the network). For VLBI, the relevant dimensions are: observed source, observation frequency, epoch, atmospheric conditions (PWV, Tsys), baseline geometry range, and instrument class. The MPIfR team that built DiFX has the domain knowledge to define this fingerprint correctly.
Routing layer deployment: A lightweight routing layer above the existing correlation pipeline. Nodes post outcome packets to addresses derived from their fingerprints. Other nodes query addresses matching their current context before beginning calibration. The raw correlation pipeline is unchanged.
Synthesis at the edge: Each station's calibration pipeline integrates incoming packets from semantically similar sessions — same source, similar atmospheric conditions, overlapping baseline geometry — before applying its own solutions. The result is calibration intelligence informed by the collective experience of the network.

No centralisation of raw data. No modification to DiFX or the SKA CSP. No consensus mechanism or governance overhead. The routing layer operates independently of the correlation infrastructure.

Why This Architecture Matters Now

The SKA is the most complex distributed telescope system ever built. Germany's construction contract (October 2025) is not just a national investment in astronomy — it is a stake in the routing architecture that will govern how the SKA network learns.

The calibration problem at SKA scale is qualitatively different from EHT or EVN. At 197 dishes (SKA-Mid) processing 100TB/day, the volume of calibration state that needs to be understood and applied in real time exceeds what a single centralised calibration facility can optimally manage. The solutions are strong. But the intelligence about what those solutions mean — which correlate with atmospheric patterns, which recur on specific baselines, which transfer between observing sessions — lives, under the current architecture, in tables that do not route.

QIS provides the routing layer. The outcome packets are small enough to move over any network — including the SKA fibre infrastructure. The synthesis is local to each station. The collective intelligence of the global VLBI network, updated each time any station completes an observation, becomes available to every other station in real time.

This is the architecture Christopher Thomas Trevethan's discovery enables. The complete loop — distillation, fingerprinting, routing, synthesis, new packets generated — closes the feedback cycle that currently sits open between telescope nodes.

The Architecture in 60 Seconds

The complete QIS loop for a VLBI network:

Station completes calibration session
         │
         ▼
    Distil outcomes
    (~512 bytes: gain solutions, RFI masks, atmospheric deltas)
         │
         ▼
    Generate semantic fingerprint
    (source / frequency / epoch / atmosphere / baseline range)
         │
         ▼
    Route packet → deterministic address
    (DHT, vector index, database — transport is protocol-agnostic)
         │
         ▼
    Other stations query same address
    (matching source / frequency / similar atmospheric conditions)
         │
         ▼
    Local synthesis at each querying station
    (integrate incoming packets with own session data)
         │
         ▼
    Better calibration solutions, fewer flagged data points
    New packets generated → re-enter the loop

N stations. N(N-1)/2 synthesis opportunities. At most O(log N) compute per node, per query. Intelligence compounds as the network grows.

The Same Math in Mission-Critical Distributed Networks

The N(N-1)/2 structure does not belong exclusively to radio astronomy.

Any distributed network where nodes accumulate local intelligence and would benefit from sharing validated outcomes — without routing raw data through a central bottleneck — exhibits the same scaling structure. The calibration problem at a VLBI station is structurally identical to the situational awareness problem at a node in a distributed communications mesh: local observation, local processing, distilled outcome, deterministic addressing, query for relevant intelligence from peers, local synthesis.

The mathematical argument generalises. For a mesh network with N nodes:

Each node pair represents one potential synthesis channel: N(N-1)/2 channels total
Under QIS outcome routing, each node can query for validated intelligence from semantically similar nodes — those operating in similar environments, under similar conditions, at similar frequencies — without exposing raw operational state
The routing mechanism is protocol-agnostic: DHT, database semantic index, vector search, or any transport that maps a problem fingerprint to a deterministic address

The practical consequence: as the network scales from 50 nodes to 500, the intelligence available to each node grows quadratically (from 1,225 to 124,750 synthesis pairs), while the compute burden on any individual node grows at most logarithmically. No single point of failure. No central coordination bottleneck. Resilience is architectural, not bolted on.

Germany's EUR 100 billion Bundeswehr digitalization programme, announced in 2022 and expanded through 2024, includes substantial investment in distributed communications infrastructure and IP-based military networking. The routing problem QIS addresses — how nodes in a large distributed mesh share validated operational intelligence without centralising sensitive data — is one the programme will encounter at scale. The same architecture Christopher Thomas Trevethan discovered for distributed intelligence synthesis applies here: the physics differ, the mathematics do not.

For network architects working on distributed mesh infrastructure — whether for astronomy, enterprise, or mission-critical applications — QIS provides a protocol-agnostic routing layer above whatever transport already exists. The VLBI integration pathway described above (outcome packet schema, semantic fingerprint definition, routing layer deployment, edge synthesis) is directly applicable to any distributed network where nodes learn from local observations and would benefit from real-time peer intelligence.

DEV Community