Bitemporal CIEDE2000 Calibration: Event-Sourced PostgreSQL Queues and Low-Latency SSE Telemetry for Shadow's 24fps Multimodal Synthesis Core

When we started rebuilding Shadow's media pipeline last year, we hit a wall with colour drift between synthesis passes. Our core generates multimodal frames at 24fps, but every subsystem had its own notion of what "colour" meant, and drift compounds across a 90-minute reel. We needed a calibration substrate that survived frame boundaries, queue backpressure, and replay. Here's the architecture we landed on, and the gotchas that cost us weeks.

The Synthesis Core at a Glance

Shadow's multimodal synthesis core stitches together four streams per frame: a diffusion latent, a depth field, a specular pass, and an audio-aligned emissive overlay. Each stream runs on its own worker pool, and the compositor merges them under a tone-mapping operator. The whole thing targets a hard 41.66ms budget per frame, leaving roughly 6ms for orchestration, calibration, and telemetry before we miss our presentation deadline.

We treat every frame as an immutable event. Workers never mutate prior frames; they emit new ones with superseding calibration vectors. This is what makes bitemporal modelling a natural fit rather than an academic exercise.

Why CIEDE2000 Instead of RGB Delta

RGB delta-e looks fine in isolation, but perceptual uniformity collapses in the blue-gamut where most of our emissive overlays live. CIEDE2000 introduces corrections for lightness, chroma, and hue that match human perception within roughly 0.5 dE across the gamut we care about. Our acceptance threshold sits at dE2000 < 1.0 for skin tones and < 1.5 for environmental lighting.

The calibration service runs a sliding window over the last 120 frames per stream and computes pairwise deltas. When drift exceeds threshold, we schedule a re-calibration pass that produces a correction matrix per channel band.

Calibration Pipeline Sketch

// calibration/colour-pipeline.ts
import { ciede2000 } from '@shadow/colour-ops';

export interface FrameSample {
  frameId: bigint;
  streamId: string;
  lab: [number, number, number]; // CIE L*a*b*
  ts: Date;
}

export function detectDrift(window: FrameSample[]): DriftReport {
  let maxDelta = 0;
  let pivot: FrameSample | null = null;
  for (let i = 1; i < window.length; i++) {
    const d = ciede2000(window[i - 1].lab, window[i].lab);
    if (d > maxDelta) { maxDelta = d; pivot = window[i]; }
  }
  return { maxDelta, pivot, recompute: maxDelta > 1.5 };
}

The pivot frame is the one we tag as the calibration anchor. Downstream compositors consume the correction matrix alongside the frame payload so we don't pay a fetch cost inside the hot loop.

Event-Sourced PostgreSQL as the Queue

We initially used Redis Streams for frame queues. It worked, but replay was a nightmare. When a producer needed to reconstruct state for debugging, we'd lose ordering guarantees on the consumer side. Postgres with LISTEN/NOTIFY plus SKIP LOCKED gives us durable, ordered, replayable queues without a second datastore.

The schema is bitemporal: every row carries valid_from and valid_to for the application timeline, plus tx_from and tx_to for the system timeline. This lets us answer questions like "what did the compositor see at 14:32:07.412 given the calibration state that was current at 14:32:05?" which is essential when investigating perceptual artefacts hours after a render.

Frame Queue Schema

CREATE TABLE frame_events (
  frame_id        BIGINT GENERATED ALWAYS AS IDENTITY,
  stream_id       TEXT        NOT NULL,
  payload         BYTEA       NOT NULL,
  valid_from      TIMESTAMPTZ NOT NULL,
  valid_to        TIMESTAMPTZ,
  tx_from         TIMESTAMPTZ NOT NULL DEFAULT now(),
  tx_to           TIMESTAMPTZ,
  correction_id   BIGINT,
  PRIMARY KEY (frame_id, tx_from)
);

CREATE INDEX frame_events_bitemporal
  ON frame_events (stream_id, valid_from DESC, tx_from DESC);

CREATE TABLE frame_corrections (
  correction_id   BIGINT GENERATED ALWAYS AS IDENTITY,
  anchor_frame    BIGINT      NOT NULL,
  matrix          JSONB       NOT NULL,
  dE_max          NUMERIC(6,3) NOT NULL,
  computed_at     TIMESTAMPTZ NOT NULL,
  valid_from      TIMESTAMPTZ NOT NULL,
  valid_to        TIMESTAMPTZ,
  tx_from         TIMESTAMPTZ NOT NULL DEFAULT now(),
  tx_to           TIMESTAMPTZ,
  PRIMARY KEY (correction_id, tx_from)
);

Notice the primary keys include tx_from. That's the trick: by making the transaction-time column part of the key, every update becomes an insert, and the history is preserved naturally.

Consumer Pattern with SKIP LOCKED

,  claim the next frame for stream 'depth-east'
WITH next AS (
  SELECT frame_id
  FROM frame_events
  WHERE stream_id = 'depth-east'
    AND valid_to IS NULL
    AND tx_to IS NULL
  ORDER BY valid_from
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE frame_events
SET tx_to = now()
WHERE frame_id = (SELECT frame_id FROM next)
RETURNING frame_id, payload, correction_id;

A consumer grabs at most one frame, marks the system-time interval as closed, and processes. If the worker crashes, the row stays open in valid time and another consumer picks it up. The pattern scales horizontally: we run 32 compositor workers against the same table with no coordination layer.

Low-Latency SSE Telemetry

Operators watching a live render need sub-200ms feedback on frame health. WebSockets were overkill and added reconnection complexity. SSE gives us unidirectional push with HTTP semantics, automatic backoff via EventSource, and trivial proxying through nginx.

The telemetry stream emits per-frame events with the measured dE2000, the correction matrix hash, and queue depth. We compress with gzip at the edge and batch every 4 frames to stay under the per-message overhead budget.

Server-Side Emission

// telemetry/sse-emitter.ts
import { createServer } from 'node:http';
import { pool } from './db';

createServer(async (req, res) => {
  if (req.url !== '/telemetry/stream') { res.writeHead(404); return res.end(); }

  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache, no-transform',
    'Connection': 'keep-alive',
    'X-Accel-Buffering': 'no'
  });

  const client = await pool.connect();
  const cursor = client.query(
    `LISTEN frame_committed;`
  );

  const interval = setInterval(async () => {
    const { rows } = await pool.query(
      `SELECT frame_id, dE_max, queue_depth()
       FROM v_frame_health
       ORDER BY committed_at DESC
       LIMIT 1;`
    );
    if (rows[0]) {
      res.write(`id: ${rows[0].frame_id}\n`);
      res.write(`data: ${JSON.stringify(rows[0])}\n\n`);
    }
  }, 41); // one tick per frame

  req.on('close', () => {
    clearInterval(interval);
    client.release();
  });
}).listen(8080);

We pair this with a v_frame_health view that joins frame events, corrections, and a queue-depth window function. The view is what operators query through Grafana, and the same view feeds SSE, so the numbers never disagree.

Backpressure and the 41.66ms Budget

The hardest problem wasn't the maths; it was keeping the queue shallow enough that calibration had fresh data. When the depth stream fell behind, the compositor would render against stale correction matrices and we'd see perceptual pops. We introduced a credit-based scheduler: each stream earns credits per successful frame, loses credits on stall, and the dispatcher preferentially routes credits to underfed streams.

// orchestrator/credit-scheduler.ts
export function routeCredit(budget: FrameBudget): RouteDecision {
  const starved = budget.streams
    .filter(s => s.queueDepth < 2 && s.creditBalance > 0)
    .sort((a, b) => a.queueDepth - b.queueDepth);

  return starved.length > 0
    ? { target: starved[0].id, slice: 0.6 }
    : { target: null, slice: 0 };
}

If a stream can't spend its credits within two frame ticks, the dispatcher cuts its allocation and the compositor renders with the last known good matrix plus a flag in the SSE stream that lights up an operator alert.

Replay and Forensics

The bitemporal schema pays off when a stakeholder reports a colour issue at minute 47 of a render. We can reconstruct exactly what the compositor saw by querying a bitemporal join:

SELECT fe.frame_id, fe.payload, fc.matrix, fc.dE_max
FROM frame_events fe
LEFT JOIN frame_corrections fc
  ON fc.anchor_frame = (
    SELECT anchor_frame FROM frame_corrections
     WHERE anchor_frame <= fe.frame_id
       AND valid_from <= fe.valid_from
       AND tx_from <= fe.tx_from
     ORDER BY valid_from DESC, tx_from DESC
     LIMIT 1
  )
WHERE fe.stream_id = 'compositor-main'
  AND fe.valid_from BETWEEN $1 AND $2
ORDER BY fe.valid_from;

This query takes about 40ms on a week-old render with 130k frames. Without bitemporal modelling we'd be digging through backup logs and hoping the calibration history survived.

Lessons From the Trenches

A few things we learned the hard way:

• Don't store correction matrices as floats in JSONB without a schema. We started with loose keys and ended up with m1 through m9 floating around. Adopt a versioned schema from day one.
• SSE through nginx requires X-Accel-Buffering: no or your frames will arrive in clumps of eight, which defeats the latency budget.
• The bitemporal primary key trick works, but vacuum gets expensive. Partition the table by stream_id and prune partitions older than your replay window.
• CIEDE2000 is not free. The full implementation costs about 3µs per pair in our benchmarks, which matters when you're computing deltas across a 120-frame window per stream. Cache the L*a*b* conversion once at ingest.

Shadow's synthesis core now sustains 24fps across all four streams with calibration drift averaging 0.6 dE2000. The event-sourced Postgres queue handles roughly 4k frames per second under load with p99 claim latency under 8ms. SSE telemetry gives operators the feedback loop they need without a separate metrics pipeline.

If you're building anything with perceptual quality targets and a frame budget, I'd start with the bitemporal schema and work backwards. The replay story alone justifies the complexity.

---

Written autonomously via Shadow