pg-listen/notify driven sse telemetry for ciede2000 perceptual lock calibration: inside shadow's mini-max direct multi-modal coherence engine

We hit a wall eight months into building Shadow. Our text-to-image, image-to-video, and audio-reactive pipelines were producing frames that, individually, looked correct. Put them on the same timeline and the colours drifted. Skin tones warmed by 1.5 delta-E between modalities. A blue dress on a hero render became a teal dress on the next. The model was fine. The calibration was not.

This is the story of the perceptual lock layer we built to fix it, and why we ended up driving it with pg-listen/NOTIFY and SSE rather than the more obvious Kafka + WebSocket stack.

The perception problem in three numbers

CIEDE2000 is the colour difference formula the CIE settled on after decades of arguing. It gives you a single number, delta-E, that approximates how different two colours look to a human observer under a reference illuminant. Lower is better:

• delta-E under 1: imperceptible
• delta-E 1 to 2: perceptible only on close inspection
• delta-E 2 to 3.5: perceptible at a glance
• delta-E over 5: clearly different colours

Our problem: cross-modality frames were drifting by delta-E 2 to 4 against a reference target. That is enough to break a scene.

Why perceptual lock is harder than it sounds

A naive fix is to apply a colour grading pass to the final output. That works for a single frame but it compounds badly across modalities. The video pipeline re-grades for temporal stability, the audio-reactive layer re-grades for energy, and the text-to-image pipeline re-grades for prompt adherence. Each pass drifts further from the reference. We needed a calibration loop that ran inside the generation, not after it.

The mini-max direct multi-modal coherence engine is the result. It treats each modality as a constrained optimiser. The objective is to minimise the maximum delta-E across all modalities at any given timestep. That is the mini-max part. Direct, because we backpropagate the perceptual loss into the latent space rather than re-rendering and re-grading.

The pub/sub decision: why PostgreSQL

The first version of this used Redis Streams. It worked for a single region but the moment we crossed regions the latency budget for cross-modality frame alignment blew out. Kafka was the obvious next step but it added an operational dependency we did not want for what is, fundamentally, a control plane signal rather than a data plane event.

We landed on PostgreSQL LISTEN/NOTIFY. It is not glamorous. It does not have Kafka's replay or partitioning. What it does have is transactional consistency with the frame metadata we were already storing in the same database. When a render worker commits a frame, the perceptual metric, the lock state, and the notification publish are one transaction. There is no possibility of a frame being visible without its telemetry, or telemetry without its frame.

The payload size limit on pg_notify is 8000 bytes. We designed our event schema to fit comfortably under that with room for the Lab triplet, the target triplet, the delta-E value, and a small envelope. Anything bigger goes through a side channel keyed by event id.

Schema and trigger

The trigger fires on insert into the perceptual_metrics table. Every render worker writes to this table after computing the metric against the scene reference.

CREATE TABLE perceptual_metrics (
  id            bigserial PRIMARY KEY,
  frame_id      uuid        NOT NULL,
  modality      text        NOT NULL CHECK (modality IN ('t2i','i2v','audio','depth')),
  lock_state    text        NOT NULL CHECK (lock_state IN ('locked','drifting','breached')),
  ciede2000     real        NOT NULL,
  lab_target    jsonb       NOT NULL,
  lab_actual    jsonb       NOT NULL,
  session_id    uuid        NOT NULL,
  captured_at   timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX ON perceptual_metrics (frame_id);
CREATE INDEX ON perceptual_metrics (session_id, captured_at DESC);

The trigger builds the JSON payload and publishes it on the perceptual_lock_v1 channel. Versioning the channel name is intentional. It lets us run v1 and v2 side by side during a rollout.

CREATE OR REPLACE FUNCTION notify_perceptual_lock()
RETURNS trigger AS $$
DECLARE
  payload jsonb;
BEGIN
  payload := jsonb_build_object(
    'frame_id',    NEW.frame_id,
    'modality',    NEW.modality,
    'lock_state',  NEW.lock_state,
    'ciede2000',   NEW.ciede2000,
    'lab_target',  NEW.lab_target,
    'lab_actual',  NEW.lab_actual,
    'session_id',  NEW.session_id,
    'ts',          extract(epoch from NEW.captured_at)
  );
  PERFORM pg_notify('perceptual_lock_v1', payload::text);
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_perceptual_lock
AFTER INSERT ON perceptual_metrics
FOR EACH ROW EXECUTE FUNCTION notify_perceptual_lock();

The listener side

We use the pg-listen npm package on the Node side. The listener runs as its own process, not co-located with the render workers. That separation matters: a slow SSE consumer must not block the database session that is consuming notifications.

import createSubscriber from 'pg-listen';
import { broadcast } from './sse-hub';
import { enqueueCalibration } from './calibration-queue';

export type PerceptualLockEvent = {
  frame_id: string;
  modality: 't2i' | 'i2v' | 'audio' | 'depth';
  lock_state: 'locked' | 'drifting' | 'breached';
  ciede2000: number;
  lab_target: { L: number; a: number; b: number };
  lab_actual: { L: number; a: number; b: number };
  session_id: string;
  ts: number;
};

const LOCK_THRESHOLD = 2.0;
const BREACH_THRESHOLD = 3.5;

const subscriber = createSubscriber({
  host: process.env.PG_HOST,
  port: 5432,
  database: 'shadow_telemetry',
  user: 'shadow_listener',
  password: process.env.PG_PW,
  retryInterval: 500,
});

subscriber.notifications.on('perceptual_lock_v1', (raw) => {
  const event = JSON.parse(raw as string) as PerceptualLockEvent;

  // Fan out to SSE consumers keyed by session.
  broadcast(`session:${event.session_id}`, event);
  broadcast(`frame:${event.frame_id}`, event);

  // Trigger the mini-max re-calibration if the lock is breached.
  if (event.ciede2000 > BREACH_THRESHOLD) {
    enqueueCalibration({
      frameId: event.frame_id,
      modality: event.modality,
      deviation: event.ciede2000,
      sessionId: event.session_id,
    });
  }
});

await subscriber.listenTo('perceptual_lock_v1');

process.on('SIGTERM', async () => {
  await subscriber.close();
  process.exit(0);
});

The thresholds are deliberate. A delta-E above 2.0 is when a human will notice on a glance. Above 3.5, the scene reads as inconsistent. We push calibration jobs to the queue only on breach, not on drift, because the mini-max solver handles drift internally as long as the bound is respected.

SSE, not WebSocket

WebSocket would have been the default for a telemetry stream like this. We chose SSE for three reasons.

First, the consumer for this stream is a debugging and observability tool, not a real-time control surface. It is the calibration dashboard, the inspector panel, the on-call engineer's browser when something looks wrong. One-directional push is the right shape.

Second, SSE works over HTTP/1.1 and HTTP/2 with no protocol upgrade dance. It survives proxies, load balancers, and corporate firewalls without configuration. Our on-call rotations span three continents. That matters.

Third, automatic reconnection with Last-Event-ID is built into the EventSource API. We get resumability for free without writing a single line of protocol code.

The endpoint multiplexes by session id and frame id. A client subscribes to /telemetry/perceptual-lock?session=<id> for the whole session or /telemetry/perceptual-lock?frame=<id> for a single frame.

import { FastifyInstance } from 'fastify';
import { randomUUID } from 'node:crypto';
import { hub } from './sse-hub';

export async function registerPerceptualLockSse(app: FastifyInstance) {
  app.get('/telemetry/perceptual-lock', async (req, reply) => {
    const sessionId = (req.query as { session?: string }).session;
    const frameId = (req.query as { frame?: string }).frame;
    if (!sessionId && !frameId) {
      return reply.code(400).send({ error: 'session or frame required' });
    }

    reply.raw.setHeader('Content-Type', 'text/event-stream');
    reply.raw.setHeader('Cache-Control', 'no-cache, no-transform');
    reply.raw.setHeader('Connection', 'keep-alive');
    reply.raw.setHeader('X-Accel-Buffering', 'no');

    const clientId = randomUUID();
    const channel = sessionId ? `session:${sessionId}` : `frame:${frameId}`;

    const unsubscribe = hub.subscribe(channel, clientId, (event) => {
      reply.raw.write(`event: perceptual_lock\n`);
      reply.raw.write(`id: ${event.ts}-${event.frame_id}\n`);
      reply.raw.write(`data: ${JSON.stringify(event)}\n\n`);
    });

    // Heartbeat so proxies do not close idle connections.
    const heartbeat = setInterval(() => {
      reply.raw.write(`: heartbeat\n\n`);
    }, 15000);

    req.raw.on('close', () => {
      clearInterval(heartbeat);
      unsubscribe();
    });
  });
}

The SSE hub is a thin layer over Node's event emitter with per-channel client maps. We sized each channel to hold at most 200 clients. Beyond that we drop with a warning, because a single browser should never have that many tabs open and a runaway client usually means a bug in their subscriber code.

The mini-max loop

The calibration queue pulls breach events and feeds them into the mini-max solver. The objective is straightforward to state:

minimise over (t2i params, i2v params, audio params, depth params) of the maximum delta-E across all modalities against the scene reference at the current timestep.

The solver runs on a small pool of workers with GPU access. It does not re-render the frames. It re-projects the perceptual loss back into the latent space and produces a parameter delta that the next render call will pick up. A breach therefore fixes itself on the next frame, not the current one. That latency is acceptable because human perception integrates over roughly 100ms, and a breach event typically takes 200 to 400ms to propagate end to end.

The lock_state field in the event is what the solver publishes after each step. Locked means the mini-max bound is below 2.0. Drifting means the bound is between 2.0 and 3.5 but stable. Breached means we crossed 3.5 and a re-calibration was triggered.

A note on CIEDE2000 implementation

Do not roll your own. We tried, because we needed a fast SIMD path for batch processing. We benchmarked against the reference Sharma 2005 implementation and were off by up to 0.3 delta-E on edge cases around the blue region. We now use a vetted library and only optimise the hot path with explicit numeric ranges. The 0.3 error is the difference between "looks fine" and "looks wrong" for skin tones.

If you must implement it, the gotchas are the G factor, the atan2 hue handling for red, and the T term on hue differences near 275 degrees. Get any of those wrong and your blue regions will report wildly incorrect delta-E values.

What we learned

A few things we did not expect:

The biggest source of drift was not the model. It was the colour space conversion between modalities. Each modality has its own internal colour space, and the conversion was lossy in the chroma channels. We added an explicit Lab-anchored intermediate representation that every modality passes through. Delta-E dropped by half on day one.

Transactional pub/sub is the right call when the consumers care about consistency with the database state. We tried bolting Redis on as a sidecar and it created split-brain bugs we spent a week debugging. Moving the notification into the same transaction as the metric write eliminated the class entirely.

SSE at scale is fine as long as you keep the per-connection write rate sane. We initially tried to push every single perceptual event down every connection. With 10,000 frames per minute across modalities, that is too much for a browser. We added client-side throttling in the dashboard and downsampled to one event per 50ms on the server side. CPU on the SSE worker dropped by 80%.

The mini-max bound is not symmetric. The minimum bound on delta-E we can achieve is roughly 0.4, limited by the underlying model variance. The maximum is unbounded if you push the solver hard. We cap the solver at 3 iterations per breach to keep the latency budget honest.

What is next

We are working on the v2 channel with a smaller envelope and a pre-computed mini-max state vector in the payload, so consumers can render calibration health without subscribing to the full event stream. We are also exploring replacing the explicit Lab intermediate with a learned perceptual embedding. Early results suggest it correlates better with human ratings than CIEDE2000 on certain skin tone regions.

The whole engine sits on roughly 2,000 lines of TypeScript on the Node side, 400 lines of PL/pgSQL on the database side, and a small Python service for the solver. It processes around 1.5 million perceptual events per day in production. The on-call burden is low because the breach path is fully automated.

If you are building anything that crosses modalities and need perceptual consistency, the takeaway is simple: pick one perceptual metric, make it transactional, and design the pub/sub around your consistency story rather than your throughput story. We tried the other order. It cost us a quarter.

---

Written autonomously via Shadow