DEV Community: Bryan Clark

Read Puppeteer console errors off the argument handles, not msg.text()

Bryan Clark — Mon, 21 Sep 2026 16:12:51 +0000

TL;DR — Puppeteer's msg.text() is a rendering of a console argument,
not the argument. Log an object and you get [object Object] — or, from a
minified bundle, [object Ae] — with every URL and field inside it gone. If
anything downstream classifies console errors by substring, read the arguments
instead: msg.args() → arg.evaluate(…). That's async, so collection has to
be separated from judgement. Jump to the fix.

slackwater-web is an
offline-first tide and current PWA, so a big share of its Puppeteer smoke suite
runs with the network deliberately cut and asserts that nothing broke. "Nothing
broke" means: no pageerror, and no console.error that isn't expected
offline noise. Expected noise is a real category here — with the network down,
Chrome itself logs a resource-load failure for every external fetch the app
makes and honestly handles, and none of those are app errors.

So there's a classifier. It matches on the hosts we know are external:

function isChsFetchNoise(text) {
  return (
    text.includes(IWLS_HOST) ||        // tide prediction API
    text.includes(SEASCAPE_HOST) ||    // bathymetry tiles
    /blocked by CORS policy/.test(text) ||
    /Failed to load resource: net::ERR_(FAILED|INTERNET_DISCONNECTED|NAME_NOT_RESOLVED|CONNECTION_REFUSED|TIMED_OUT|ADDRESS_UNREACHABLE)/.test(text)
  );
}

Every clause reads a string. That's the whole bug.

The problem

The map check went red in CI and only in CI:

SMOKE FAILED: /map offline reload reported errors:
console.error: [object Ae]
console.error: [object Ae]
console.error: [object Ae]
console.error: [object Ae]

Four lines, no information in any of them. Ae is a two-letter token out of a
minified bundle. There's no host to look up, no stack, no status code — nothing
to decide whether this is the map politely failing to reach a tile server with
the network cut (fine, expected, that's the whole point of the check) or the
app actually breaking (not fine).

The listener that produced them was the obvious one, copied six times across
the suite:

page.on("console", (msg) => {
  if (msg.type() === "error" && !isChsFetchNoise(msg.text())) {
    errors.push(`console.error: ${msg.text()}`);
  }
});

Diagnosis: `text()` is a description, not the value

A console argument lives in the browser. Puppeteer holds a JSHandle — a
reference into the page — and ConsoleMessage.text() gives you the browser's
own short description of each argument joined together. For a string that
description is the string. For an object it's whatever Chrome would print as a
one-line label.

Fifteen lines of Puppeteer will show you the difference. Log four things and
read each one both ways:

p.on("console", async (msg) => {
  if (msg.type() !== "error") return;
  const viaEval = await Promise.all(
    msg.args().map((a) =>
      a.evaluate((o) => (o instanceof Error ? `${o.name}: ${o.message}` : String(o))),
    ),
  );
  console.log(JSON.stringify({ text: msg.text(), viaEval }));
});

await p.evaluate(() => {
  class Ae { constructor(url) { this.url = url; this.status = 404; } }
  class Be extends Error { constructor(u) { super("Not Found: " + u); this.url = u; } }
  console.error(new Ae("https://tiles.example.org/a.pbf"));
  console.error(new Be("https://tiles.example.org/b.pbf"));
  console.error(new Error("Failed to fetch"));
  console.error({ plain: "object", url: "https://tiles.example.org/x" });
});

Output, on Chrome 151 with puppeteer-core 25.3.0:

{"text":"[object Ae]","viaEval":["[object Object]"]}
{"text":"Be: Not Found: https://tiles.example.org/b.pbf","viaEval":["Error: Not Found: https://tiles.example.org/b.pbf"]}
{"text":"Error: Failed to fetch","viaEval":["Error: Failed to fetch"]}
{"text":"[object Object]","viaEval":["[object Object]"]}

Line one is the CI failure, reproduced in a file with no app in it. A class
instance Chrome doesn't describe as an Error renders as
[object <ClassName>], and after minification the class name is Ae. The
url property that was sitting right there on the object never appears in the
rendered text at all — so isChsFetchNoise had literally nothing to match on,
and a fetch failure we expect under a cut network got counted as an app error.

Note line two as well: even for an Error subclass the prefix is the minified
constructor name, not err.name. text() is a debugging convenience. It
was never a data source, and nothing in its shape tells you when it has dropped
the field you're keying on.

Why it was green locally

The tempting explanation is a browser difference. It wasn't one. CI resolved
Chrome-for-Testing 151.0.7922.71; the laptop that ran the same suite green
was on 151.0.7922.109. Same major, same minor, three days apart.

The real reason is duller and much more common: the assertion ran before the
errors existed.

await mapPage.reload({ waitUntil: "domcontentloaded" });
await mapPage.waitForSelector(".map-canvas .maplibregl-canvas", { timeout: 10_000 });
if (mapErrors.length) throw new Error(/* … */);   // green

MapLibre puts its canvas in the DOM as soon as it has a GL context — well
before it has finished trying, and failing, to fetch tiles and glyphs over a
network that isn't there. On a fast local machine the check sampled an empty
array roughly two seconds early and passed. On a slower CI runner the failures
landed inside the window, got collected, and were unreadable. One bug hid the
other: the suite had to get slower before it could show me it was also
blind.

What I tried first

Add the missing host to the allowlist. Can't — that's the point. The host
is inside an object the classifier never sees. There is no string to add.

Widen the filter to swallow the unreadable shape.

// no
/^\[object \w+\]$/.test(text) ||

This is the version that ships. It's green, it's one line, and it converts
every future object-shaped console error in the app — including real ones —
into permitted noise. A filter whose match is "I couldn't read this" isn't a
filter.

Reach for jsonValue(), which is the answer most search results give:

const args = await Promise.all(msg.args().map((a) => a.jsonValue()));

It's better than text() and still wrong for this. jsonValue() serialises,
so an Error comes back as its enumerable own properties only:

{"name":"AJAXError","url":"https://tiles.openwaters.io/12/34/56.pbf"}

The message — which is where MapLibre puts the status and the failing URL —
isn't an enumerable own property, so it's gone. On a plain new Error("boom")
you get {}. It happens to carry the URL here because that error also stores it
as a field; rely on that and the next library breaks you.

Only assert pageerror and stop filtering console errors at all. Genuinely
tempting, and it would have stayed green forever. It also deletes the assertion
that catches an app that logs its way through a broken offline path without
throwing — which is exactly the failure mode an offline-first PWA has.

The fix

Resolve each argument in the page, where it's still a real object:

function watchErrors(page) {
  const pending = [];
  page.on("pageerror", (err) =>
    pending.push(Promise.resolve({ kind: "pageerror", raw: err.message })));
  page.on("console", (msg) => {
    if (msg.type() !== "error") return;
    pending.push(
      Promise.all(
        msg.args().map((arg) =>
          arg
            .evaluate((o) => (o instanceof Error ? `${o.name}: ${o.message}` : String(o)))
            .catch(() => null),
        ),
      )
        // Falls back to the rendered text when the page closed before the
        // handles could be read, or when there are no arguments at all
        // (browser-level "Failed to load resource" lines).
        .then((parts) => parts.filter(Boolean).join(" ") || msg.text())
        .catch(() => msg.text())
        .then((raw) => ({ kind: "console.error", raw, filterable: true })),
    );
  });
  return async () => {
    const seen = await Promise.all(pending);
    return seen
      .filter((e) => !e.filterable || !isChsFetchNoise(e.raw))
      .map((e) => `${e.kind}: ${e.raw}`);
  };
}

Read that same failing line now and it says:

Error: AJAXError: Not Found (404): https://tiles.openwaters.io/…

— which isChsFetchNoise classifies correctly, because the host is finally in
the string.

Three things in there are load-bearing beyond the evaluate:

pageerror is never filterable. An uncaught exception is always a failure, whatever it says. Only console errors get classified.
Both .catch()es fall back to msg.text(). A handle can't be read after its page closes, and browser-level lines like Failed to load resource: net::ERR_FAILED arrive with no arguments at all — msg.text() is the correct source for those, and it's the reason that regex clause still earns its place.
The return value is a settle(), not an array. Resolving a handle is a round trip to the browser, so the listener can't append synchronously any more. Collection and judgement are now two separate moments, and every call site has to await the second one:

const settleMapErrors = watchErrors(mapPage);
await mapPage.reload({ waitUntil: "domcontentloaded" });
await mapPage.waitForSelector(".map-canvas .maplibregl-canvas", { timeout: 10_000 });
// The canvas appears well before the map has finished failing its external
// fetches, so asserting the instant it exists passed for the wrong reason —
// the errors simply hadn't been logged yet. Settle first, then judge.
await new Promise((resolve) => setTimeout(resolve, 2_000));
const mapErrors = await settleMapErrors();
if (mapErrors.length) throw new Error(`/map offline reload reported errors:\n${mapErrors.join("\n")}`);

The async fix forced the shape that fixes the race. That's a nice accident, not
a design — but it's the reason to prefer a collector object over an array you
push into: an array tempts you to read it at the wrong instant, and a settle()
can't be read at all until you've awaited it.

The ordering the offline blocks need

Same suite, adjacent trap. Cutting the network with CDP and then asserting
requires the steps in one specific order, and two of the six exist purely
because getting them wrong produces a green run.

// 2. cut it
const cdp = await page.createCDPSession();
await cdp.send("Network.enable");
await cdp.send("Network.emulateNetworkConditions", {
  offline: true, latency: 0, downloadThroughput: -1, uploadThroughput: -1,
});

// 3. prove it — a real request to an origin that must not resolve
const networkState = await page.evaluate(() =>
  fetch("https://example.com/" + Math.random()).then(() => "UP").catch(() => "down"),
);
if (networkState !== "down") {
  throw new Error(`network emulation did not take effect: fetch reported "${networkState}"`);
}

// 4. only NOW attach the listeners
const settleErrors = watchErrors(page);

Don't accept navigator.onLine as the proof. It answers "does this device
have a network interface", never "can I reach anything" — a captive portal
returns true all day. Under CDP emulation it's worse than merely unhelpful:
whether the flag follows the emulated state is a browser-and-transport detail
that has changed across versions and setups, and has read true on a cut
network in this project before. On the combination I re-measured today
(Chrome 151, puppeteer-core 25.3.0) it reported false correctly, both via
CDP and via page.setOfflineMode(true) — which is exactly why it's a bad
oracle. A signal that's right on your machine, this week, and silently wrong on
the runner is the same failure class as everything else in this post. A fetch
that must reject is evidence. A flag is a claim.

Attach the listeners after the proof fetch, not before. The proof fetch is
supposed to fail — that's its job — and Chrome logs its own resource-load error
for it regardless of the .catch() on the JS side. Attach a console listener
first and your very first collected "app error" is the test's own probe. It'll
be the top line of a failure report about something else entirely.

The bonus finding: a third origin nobody had written down

Once the errors were legible, they immediately said something new. The
allowlist had two external hosts on it — the tide API and the bathymetry tile
host. The real map was hitting three.

A MapLibre style sheet has a glyphs field, and the vendor style we load points
it at MapLibre's public demo font server — a different origin from the one
serving the style. Add a label layer and you're fetching glyph files from a host
nobody in the project ever chose:

const GLYPH_HOST = "demotiles.maplibre.org";   // NOT the tile host

That's a real finding, not just test bookkeeping: an offline-first app was
quietly depending on a third party's demo infrastructure to draw place names.
It had been true for weeks. Nobody knew, because for weeks the only evidence
was a line reading [object Ae].

One more, in the same batch: when the network is cut hard enough that no
response comes back at all, MapLibre re-wraps the rejection as a bare
Error: Failed to fetch carrying no URL — its AJAXError, the one that does
name the host, only exists once a response actually arrived. There is nothing
left to attribute it to, so that one gets an anchored exact match rather than a
substring:

/^Error: Failed to fetch$/.test(text) ||

Anchored, so Error: Failed to fetch https://… from anywhere else still fails
the smoke.

Why it matters

The general rule is narrower than "console errors are unreliable":

Never classify on a rendering. msg.text(), a log line, a formatted table,
a CLI's pretty output — every one of them is a lossy projection chosen for a
human reader, and every one of them can change between versions without
breaking a single type. If a decision depends on a field, get the field. Here
the field was one arg.evaluate() away the whole time.

Two traps sit next to it:

A filter that can't parse its input must fail, not pass. The one-line regex that swallows [object Ae] is available at every step of this story and it's the wrong move every time. If your classifier gets something it can't read, that's a red test, not a matched rule.
Waiting for the first sign of life is not waiting. waitForSelector on the artifact that appears earliest is the most common way to build a check that samples before the system has finished. Wait for the last thing, or settle over a window and judge afterwards.

And the meta-lesson: this suite was green for two months. It went red only when
a slower machine let the errors land inside the assertion window — the fastest
way to find out a check is blind is to make it slower.

I'm building the software for an all-electric charter catamaran mostly in
public — SignalK plugins, MCP servers, and this offline tide app that has to
work with no signal in the Salish Sea. The smoke suite above lives in
scripts/smoke.mjs
(32bfe09).

Related: Porting a tide engine to Swift with the original as the test oracle — what it looks like when a check does examine the result · Monitor the delivery path, not just the alarm — the same silence, one layer out

Adding a type column to a shared registry breaks every consumer's filter

Bryan Clark — Sun, 20 Sep 2026 05:27:10 +0000

A shared data package grew an optional kind: "tide" field, with absent meaning "current" for backwards compatibility. Every consumer that enumerated the registry had a hand-rolled "which rows are gates" filter written before that field existed, so the new rows swept straight in — ten no live id for Victoria-style throws per poll cycle on the boat server, and one false name drift? warning per build in another repo. The real fix isn't a better filter in each consumer. Jump to the fix: export the selection from the package that owns the data.

The problem

Two symptoms, three weeks apart, in two repos that share one dependency.

The SignalK currents plugin, on the boat server, on every poll cycle:

no live id for Tofino
no live id for Victoria
no live id for Sooke
no live id for Fulford Harbour
no live id for Port Renfrew
no live id for Campbell River
no live id for Point Atkinson
no live id for Vancouver
no live id for Port Alberni
no live id for Owen Bay

And the harmonic-constituent bundler, on every build:

registry gate chs-malibu-rapids found no live IWLS station (name drift?)

Neither is fatal, which is exactly why both ran for weeks. The throw drops the
station, so /currents still served the right 20 gates; the bundler still emitted
a valid bundle. The cost is noise — a wasted fetch attempt per station per cycle,
real station errors buried under fake ones, and a drift warning that now fires on
a station that cannot drift, training you to ignore it when a real CHS rename
lands.

Diagnosis

The dependency is @sailingnaturali/station-corrections
— station identity (name, position, provider) for the Salish Sea tide and current
stations, published as a plain JSON artifact so non-JS consumers can read it
without npm. It was currents-only by accident: current gates were simply the
first data whose identity we owned, not a decision anyone made.

Then it grew a second class. CHS tide reference ports arrived so a gate could name
its companion port:

chs-owen-bay:
  name: Owen Bay
  context: Sonora Island
  position: [50.310, -125.223]
  provider: chs
  kind: tide

The schema change was written to be maximally compatible:

// kind is optional, not required: the 19 existing gates predate the
// distinction and an absent kind reads as "current", so the resolver
// applies that default (one line) rather than backfilling every entry.

Optional-with-a-default is the right call for the data. For consumers it is a
trap, because their filters are written in terms of what the registry happened to
hold on the day they were written. Both of them said the same thing:

// signalk-currents
return Object.entries(data)
  .filter(([, e]) => e.provider === 'chs')
  .map(/* → a station we will fetch a live current series for */);

// chs-constituents
for (const [key, entry] of Object.entries(data)) {
  if (entry.provider !== provider) continue;
  // → a name overlay we will match against the live IWLS station list
}

Both answer "which rows are current gates?" with "the ones from this provider."
True when written. Silently false the moment the registry grows a class.

The registry today is 31 rows in four shapes: 19 plain current gates (kind
absent), 10 tide ports (kind: tide), 1 NOAA gate, and one oddity that matters —
a derived gate. Malibu Rapids is a genuine current gate that CHS publishes no
current station for at all; its slack comes from Point Atkinson's high/low water
plus a lag:

chs-malibu-rapids:
  name: Malibu Rapids
  position: [50.1626, -123.8515]
  provider: chs
  kind: current          # still a gate…
  derived:               # …but there is no series of its own to fetch
    reference: chs-point-atkinson
    hwLagMinutes: 25
    lwLagMinutes: 35

So there are two ways to be in the registry and not be something you can fetch a
live current series for — a wrong kind, and a right kind with a derived
block. Each consumer discovered exactly one of them.

What we tried (and why it failed)

1. Exclude the new kind, in the consumer

chs-constituents, the day the tide ports landed. Ten false drift warnings per
build, so:

// Tide reference ports (2.2.0+) have no current series; carrying them
// into the overlay makes the no-live-station drift warning fire 10x/build.
if (entry.kind && entry.kind !== "current") continue;

Warnings per build: 10 → 1. The survivor is Malibu. This filter has no opinion
about derived, because on the day it was written Malibu did not exist yet — it
was added to the registry the next day. A gate that is derived precisely
because CHS publishes no station for it can never match a live IWLS station, and
the code warns on exactly that condition. One false warning, every build, for
three weeks.

2. Exclude derived gates, in the other consumer

signalk-currents, the next day, against the other new class — Malibu would have
been swept into the fetch list and thrown every cycle:

.filter(([, e]) => e.provider === 'chs' && e.derived === undefined)

The exact complement of fix #1. Two consumers of one registry, 24 hours apart,
each excluding the class that had just arrived in front of it, neither noticing
they were the same bug. signalk-currents still swept in all ten tide ports —
nearly three more weeks of ten throws per poll cycle on the boat server before
anyone read that log.

3. Denylist the tide ports

The obvious patch, shipped as 0.11.1:

.filter(([, e]) => e.provider === 'chs' && e.derived === undefined
  && e.kind !== 'tide')

Correct today, and the wrong shape. The bug was the registry growing a kind this
repo had never seen — and !== 'tide' re-acquires that bug the next time it
grows one. The regression test writes down the shape:

it('skips a kind it has never seen, rather than assuming it is a gate', () => {
  const future = { 'chs-somewhere': { name: 'Somewhere', position: [49, -123], provider: 'chs', kind: 'wave' } };
  expect(registryChsStations(future as never)).toEqual([]);
});

The denylist fails it. An allowlist passes:

.filter(([, e]) => e.provider === 'chs' && e.derived === undefined
  && (e.kind === undefined || e.kind === 'current'))

4. Two correct filters, in two repos

Now both consumers agreed. The policy still lived in two places, expressed in two
different styles, in two languages' worth of idiom — and the next reader of that
registry would write a third copy from whatever it happened to hold that month.
Four fixes in, nothing had changed about why it kept breaking.

The fix

Move the selection into the package that owns the data:

export function currentGates({
  registry = new Map(Object.entries(bundledRegistry)),
  provider,
  includeDerived = false,
} = {}) {
  const gates = new Map();
  for (const [id, record] of registry) {
    if (provider !== undefined && record.provider !== provider) continue;
    if ((record.kind ?? "current") !== "current") continue;   // allowlist
    if (!includeDerived && record.derived !== undefined) continue;
    gates.set(id, record);
  }
  return gates;
}

Both consumers delete their filters and ask:

// signalk-currents
return [...currentGates({ registry: new Map(Object.entries(data)), provider: 'chs' })]
  .map(/* … */);

// chs-constituents
const gates = currentGates({ registry: new Map(Object.entries(data)), provider });
for (const [key, entry] of gates) { /* … */ }

kind is the registry's own editorial classification. Only the package that adds
a class knows what the existing consumers meant by "gate" — so it should be the
one that decides, once, instead of every reader rediscovering the split from a
production symptom.

Why it matters, and the traps nearby

An optional column with a default is a breaking change for enumerators. Semver
calls it a minor: nothing was removed, no field changed meaning, every old row
still parses. But every consumer that iterates rows carries an implicit
WHERE clause it never wrote down, and a new row type silently violates it. If
you ship a shared data artifact, "we added a type column" belongs in the release
notes next to the breaking changes, whatever the version number says.

A denylist re-acquires the bug it just fixed. kind !== 'tide' encodes "the
classes I know about today." The failure mode being patched was not knowing
about a class. Prefer kind is current, and prove it with a test that passes a
kind you invented.

Type the shared selector by what it reads, not by your record. The first
release declared registry?: Registry — the package's own full station type.
chs-constituents builds its overlay on a deliberately narrow {name, provider, kind} record whose fixtures omit position to prove the overlay reads no
position. Adopting the selector meant widening a type to satisfy a signature
rather than a caller, so the follow-up release made it generic over the caller's
shape:

/** The three fields currentGates reads. */
export interface GateSelectable { provider?: string; kind?: string; derived?: unknown; }

export function currentGates<T extends GateSelectable = RegistryStation>(options?: {
  registry?: Map<string, T>;
  provider?: string;
  includeDerived?: boolean;
}): Map<string, T>;

A shared selector that forces its callers to widen their own types is not
adoptable, and un-adopted selectors leave the hand-rolled filters in place. Note
that tsc is the only thing that catches this class of regression, so it's worth
a compile-only surface test that instantiates the narrow shape.

Only the enumerators are at risk. A third consumer reads the same registry and
never broke: currents-mcp looks stations up by key, because each vault pass
names its own station: id. A new row class is invisible to code that asks for a
row by name.

A warning that fires on a station missing by definition is worse than no
warning. The name drift? check exists to catch a real CHS rename detaching a
curated gate from its live station — a failure that would otherwise be silent. One
guaranteed false positive per build is enough to teach you to scroll past it. The
value of that last fix isn't the log line it removes; it's the signal it restores.

Close

All of this runs on a boat: a SignalK server serving tidal-current predictions to
an agent that answers questions about when a pass goes slack. The registry, the
plugin, and the constituent bundler are public —
station-corrections,
signalk-currents,
chs-constituents — and
currentGates() is now the only place any of them decides what a gate is.

Related: Offline tidal currents from harmonic constituents — what the gates in this registry are actually for — and two false walls in NOAA's currents API, the other half of where station identity comes from.

Four ways a delivery-health alarm silently doesn't work

Bryan Clark — Wed, 16 Sep 2026 16:04:50 +0000

Four independent bugs in one small delivery-health check. The headline: mqtt.js only invokes the publish callback on PUBACK, so during a broker outage it fires neither success nor failure and a consecutive-failure counter never moves — the monitor sleeps through the exact outage it was written to report. Fix: record a failure yourself when client.connected === false, then publish anyway. Jump to the fix.

Five weeks ago I published Monitor the delivery path, not just the alarm. The argument: a notification pipeline is exercised only when something is already wrong, so it needs its own pulse — count consecutive delivery failures, raise a deliveryFailed notification past a threshold, and surface it on a channel that doesn't depend on the lane that's down.

I still believe every word of it. This post is what happened when I re-implemented that same pattern in the plugin that replaced the relay, shipped it, and then read the whole branch back with fresh eyes. The health check was broken four independent ways. None of them were in the alarm-routing paths — the sirens worked fine. All four were in the layer whose entire job is to tell me when the sirens don't.

Problem: the watcher looked right and did nothing

The successor is signalk-notification-router — one subscription to notifications.*, three outputs: MQTT (which the dashboard and the voice agent read), a Telegram siren for alarm/emergency, and an agent webhook for alert/warn. Delivery health is tracked per lane, so MQTT dying doesn't claim Telegram is failing:

const failures = new Map();   // lane -> consecutive failure count
const raised = new Set();     // lanes currently carrying a deliveryFailed alarm

function recordResult(lane, ok) {
  if (ok) {
    failures.set(lane, 0);
    if (raised.has(lane)) { raised.delete(lane); setDeliveryFailed(lane, false); }
    return;
  }
  const n = (failures.get(lane) || 0) + 1;
  failures.set(lane, n);
  if (n >= failureThreshold && !raised.has(lane)) {
    raised.add(lane);
    setDeliveryFailed(lane, true);
  }
}

That is the July post's design, faithfully. It shipped in 1.0.0 with 53 passing tests. Every test I had thought to write was green — and the reason they were green is that each of the four bugs lives in a seam between two components I had tested separately: the counter and the MQTT client, the notification tree and the publish path, the plugin lifecycle and the in-memory mirror of the tree, the alarm and the human.

A whole-branch review after shipping is what found them. Not a bug report — nothing reported it, which is the point.

Diagnosis: four places one alarm can stop

Walk the path an alarm has to travel and the failures line up in order:

lane send outcome → recordResult() → raised guard → notifications tree
                                   → MQTT publish → dashboard + voice → human

The outcome never arrives, so the counter never moves. (mqtt.js)
The counter moves and the tree is written, but the row never leaves the tree. (the loop guard)
The tree is written, and can then never be un-written. (the restart latch)
The row reaches the dashboard, which is the only place it can ever reach. (visual-only)

Each one alone makes the health check useless. I had all four.

Blind spot 1: mqtt.js only calls back on PUBACK

The MQTT lane's outcome came from the publish callback:

function publishMqtt(topic, envelope) {
  if (!mqttClient) return;
  mqttClient.publish(topic, JSON.stringify(envelope), { qos: 1, retain: true }, (err) =>
    recordResult('mqtt', !err)
  );
}

This reads like every callback in Node: fires with an error, or fires without one. It isn't. From the mqtt.js README, on client.publish(topic, message, [options], [callback]):

callback - function (err, packet), fired when the QoS handling completes, or at the next tick if QoS 0. An error occurs if client is disconnecting.

"When the QoS handling completes" means on PUBACK at QoS 1 (PUBCOMP at QoS 2). The client is mqtt@5; in lib/client.js, an offline publish never reaches the wire at all:

// _sendPacket()
if (!this.connected) {
  this._storePacket(packet, cb, cbStorePut);   // queue it, and return
  return;
}

and _storePacket puts the QoS-1 packet in the outgoing store, where the callback waits for an ack that cannot arrive until the broker comes back. The callback isn't errored. It isn't dropped. It's deferred, indefinitely.

So during a broker outage the MQTT lane records neither a success nor a failure. failures.get('mqtt') stays at 0, the threshold is never crossed, and deliveryFailed.mqtt cannot raise. A health counter that only counts completed deliveries measures nothing during an outage, because an outage produces no completions.

This is the July post's own lesson biting from the other end. There I wrote that a quiet boat produces no sends, so no failures, so a false all-clear. Here the boat is noisy — alarms are firing, publishes are being called — and it still produces no failures, because failure was defined as "the callback said so" and the callback is on the far side of the broken thing.

What I tried

Await the publish instead. mqtt v5 has publishAsync, and a promise reads better than a callback:

try { await mqttClient.publishAsync(topic, body, { qos: 1, retain: true }); recordResult('mqtt', true); }
catch (e) { recordResult('mqtt', false); }

Identical semantics. The promise settles on the same PUBACK, so it simply hangs instead of not calling back. Prettier, equally blind.

Race the callback against a timer. There is no per-publish timeout option, so this means hand-rolling one per message — and then holding the bag when the deferred callback does fire on reconnect, twenty minutes later, against a message you already recorded as failed. Double-counting a recovering lane into a fresh alarm is worse than the bug.

Drop to QoS 0 with queueQoSZero: false. This genuinely does produce an immediate error callback while offline — _storePacket's last branch fires cb(new Error('No connection to broker')). It also throws away the thing QoS 1 buys: real alarms queue and flush on reconnect. Trading alarm delivery for alarm observability is exactly backwards.

The fix is not to make the callback trustworthy. It's to stop asking the callback a question it can't answer, and ask the client directly.

Blind spot 2: the loop guard sat one line too early

The July post's hardest-won rule was: never route the health alarm through the failing path. The router obeyed it with an early continue at the top of the delta handler:

const path = v.path.slice('notifications.'.length);
// Never route our own delivery-path alarms back out — they must not
// loop through the very lane that is failing.
if (path.startsWith(SELF_PREFIX)) continue;

Correct intent, wrong line. Everything downstream of that continue includes the MQTT publish:

publishMqtt(`${prefix}/${path}`, buildEnvelope(row, position()));   // ← never reached for self rows
newlyActive.push(row);                                             // ← the push lanes (correctly skipped)

MQTT is not a push lane here. It is the only path from the SignalK notifications tree out to the dashboard and the voice agent — the two "independent channels" the whole design leans on. Skipping it meant deliveryFailed.mqtt (and .telegram, and .hook) reached exactly one audience: somebody already logged into the SignalK admin UI, looking at the notifications tree, which is not a thing anyone does on a boat at 02:00.

The alarm was real, correct, well-tested, and invisible.

The test that covered this was green, and its name is the giveaway:

✔ the plugin never routes its own deliveryFailed alarm

It asserted the alarm didn't reach Telegram or the webhook. It never asserted the alarm reached anything. A negative test with no positive counterpart will happily pass on a dead system.

Blind spot 3: an admin-UI config save latched the alarm forever

raised is an in-memory mirror of what's currently alert in the notifications tree. plugin.start reset it along with everything else, which looks like ordinary hygiene:

failures.clear();
raised.clear();      // ← this one

The SignalK notifications tree is server state; it survives a plugin restart. The in-memory Set does not. And saving config in the SignalK admin UI calls stop() then start() — no server restart, no page reload, just a save button. Sequence:

1. the mqtt lane fails 3× → raised = {mqtt}, tree shows deliveryFailed.mqtt = alert
2. operator saves any config change → SignalK calls stop(), then start()
3. start() clears raised     → raised = {},  tree still shows alert
4. the broker comes back, a publish succeeds → recordResult('mqtt', true)
5. `if (raised.has(lane))` is false → setDeliveryFailed(lane, false) never runs
6. tree shows alert forever — until the whole SignalK server restarts

A latched-on alarm on a recovered lane is arguably worse than no alarm: it trains you to ignore the one indicator whose entire value is that it's normally quiet.

What I tried: rebuilding raised on start by reading the tree back with app.getSelfPath('notifications.notificationRouter.deliveryFailed…'). It works, and it's more "correct" in that it re-derives state from the source of truth. It's also more code, one more API dependency, and one more thing to get wrong at startup — to fix a bug whose cause was clearing a set that had no business being cleared. failures should reset (a stale failure streak shouldn't count toward a fresh threshold). raised shouldn't. Deleting one line beats adding ten.

Blind spot 4: visual-only, on purpose

The fourth isn't a bug. It's the design constraint that makes the other three matter, and it doesn't go away when they're fixed:

value: active
  ? {
      state: 'alert',
      method: ['visual'],      // never ['sound'], never a push
      message: `${lane} delivery path failing — notifications are not reaching it`,
      timestamp: new Date().toISOString(),
    }
  : { state: 'normal', method: [], message: '' },

Routing keys on the notification's own method array, so ['visual'] is what stops the health alarm from being classified into a push lane and looping straight back through the lane that's failing. It's deliberate and it's right.

It also means the health alarm can never page anybody. It is a light on a panel. If the Telegram lane is down and nobody opens the dashboard, the fact that Telegram is down is displayed, correctly, to an empty room. The July post said to surface the alarm on an independent channel; what it didn't say clearly enough is that independent and reachable are two different properties, and a self-hosted stack usually only has enough independent paths to guarantee one of them.

I don't have a fix for this one. I have a boundary: the health alarm reports lane failures to whatever is watching, and something outside this plugin — a phone, a person, a second provider — has to be the thing that watches. Naming the residual risk beats pretending the fourth channel exists.

The fix

1 — ask the client, not the callback. One line before the publish, and still publish, so QoS-1 alarms queue and flush on reconnect:

function publishMqtt(topic, envelope) {
  if (!mqttClient) return;
  // mqtt.js only calls the publish callback on PUBACK, so an unreachable
  // broker would otherwise record neither success nor failure — the health
  // counter would sit silent through the exact outage it exists to report.
  // Still publish: qos-1 messages queue and flush on reconnect.
  if (mqttClient.connected === false) recordResult('mqtt', false);
  mqttClient.publish(topic, JSON.stringify(envelope), { qos: 1, retain: true }, (err) => {
    try {
      recordResult('mqtt', !err);
    } catch (e) {
      try { app.error(`mqtt publish callback error: ${e.message}`); } catch {}
    }
  });
}

(=== false, not !mqttClient.connected — a test double or a client mid-handshake without the property shouldn't be read as offline. And the callback is invoked directly by mqtt.js, not through a promise chain, so a throw inside it takes down the process rather than losing one outcome.)

2 — move the guard past the publish. Self rows publish to MQTT like any other row; only push routing is skipped:

const isSelf = path.startsWith(SELF_PREFIX);
// …
publishMqtt(`${prefix}/${path}`, buildEnvelope(row, position()));
// Our own delivery-path alarms still publish to MQTT (the only path from the
// notifications tree to voice/dashboard) but are never routed back out through
// classify/route — that would loop through the very lane that is failing.
if (isSelf) continue;
newlyActive.push(row);

Safe by two independent mechanisms, not one: method: ['visual'] means classify() can't assign it a lane anyway, and an MQTT failure can't re-raise past the raised guard.

3 — don't clear the mirror of state you didn't clear.

failures.clear();
// Do NOT clear `raised`: the notifications tree survives a plugin restart, so
// forgetting a raised lane here means the next success never clears the tree.

4 — no code. Documented as a boundary, in the README and above.

Every fix TDD'd, red before green, 53 → 63 tests:

✔ an unreachable MQTT broker still records failures and raises deliveryFailed.mqtt (Finding 3)
✔ a connected test double without a `connected` property is treated as connected
✔ the delivery-health alarm itself reaches MQTT (Finding 1)
✔ the plugin never routes its own deliveryFailed alarm, but still publishes it to MQTT
✔ a raised alarm survives a restart and still clears on recovery (Finding 2)
✔ a throwing app.error does not produce an unhandled rejection, and later deliveries still work (Finding 5)

Note the second test name in that renamed pair. the plugin never routes its own deliveryFailed alarm became …, but still publishes it to MQTT. The bug was in the clause that wasn't there.

Shipped as @sailingnaturali/signalk-notification-router 1.2.0. Anyone who ran 1.0.0–1.1.0 is affected by the first two.

Why it matters / gotchas

Your monitor's failure modes are correlated with the thing it monitors. This is the whole post in one line. Three of the four bugs are the same mistake wearing different clothes: the health check was built out of the same components as the delivery path, so the outage took both down together. The publish callback is unavailable precisely when the broker is; the health alarm's only exit is the transport whose failure it reports. Independence has to be designed in and then checked, or you've built a smoke detector wired to the same circuit as the stove.
"Fires on success or error" is an assumption, not a contract. Check what your client actually promises. mqtt.js says "when the QoS handling completes", and completion requires the broker. Two caveats if you copy this fix: with queueQoSZero: false an offline QoS-0 publish does error back immediately (a different branch entirely), and MQTT.js#1561 documents a race where a QoS-0 callback can be dropped rather than deferred — closed as not planned. Don't state the deferral as a universal guarantee; state it for QoS 1 at default settings, which is what it is.
The plugin lifecycle is a failure mode. Saving config in the SignalK admin UI is a stop() + start(). Any in-memory structure that mirrors server-side state — the notifications tree, a resource provider's cache, a subscription list — has to survive that, or reset both sides. Half a reset is a latch. (plugin.start is now also self-guarding: it calls plugin.stop() first, so a start() without a preceding stop() can't leave two live subscriptions and a double siren.)
A negative test passes on a dead system. never routes its own alarm was true of the correct implementation, of the broken one, and of a plugin that does nothing at all. Every "X must not reach Y" assertion needs its "X must reach Z" partner in the same test, or the day Z breaks, nothing turns red.
Ship, then read the whole branch back. No user reported any of this; the failure mode of a broken monitor is silence. The four were found by re-reading a shipped branch end to end, with the specific question "how does each of these paths not work?" — which is a different exercise from code review, and the only one that catches a bug whose symptom is nothing happening.

Close

This is the alerting layer of an all-electric charter catamaran, where the difference between a silent alarm and a working one is the entire value of the system. The router — the health check, the four fixes, and the tests — is MIT and open: github.com/sailingnaturali/signalk-notification-router.

Related: Monitor the delivery path, not just the alarm — the health check this post re-implements and breaks; a zero-dependency SignalK relay — the plugin this router replaced; the NMEA 2000 alarm-fatigue problem — why the notification's own method array decides routing.

A nearest-neighbour matcher turned a timing lag into a fake direction bug

Bryan Clark — Fri, 11 Sep 2026 15:40:45 +0000

TL;DR — If your comparison against golden data matches each expected event
to the nearest computed event of any kind and then asserts the kind, you have
coupled two independent error dimensions. A timing lag will surface as a
wrong-label failure and your oracle will confidently blame the wrong thing.
Match same-kind-to-same-kind for timing, and test direction separately at the
expected timestamp. Jump to the fix.

We validate our offline tidal-current engine against the hydrographic
authority's own published current predictions: for each published maximum flood
or maximum ebb, does the model produce the same event, at about the same time,
at about the same speed, and pointing the same way? A validation run over the
Canadian stations came back with two stations flagged as having a reversed
flood axis — the model claiming flood where the authority says ebb. That is
the one failure you cannot ship. Both stations were quarantined.

Both stations were fine. The matcher was broken.

This post is about the matcher — how you compare model output against golden
data. It is the sequel to
porting a tide engine to Swift with the original as the test oracle,
which is about where golden data comes from. Same test suite; the bug is in the
other half.

The problem: a label-flip failure that wasn't

The failing assertion looked like this, and its output is what got two stations
quarantined:

✕ maxEbb at 2026-07-14T14:12:00Z: engine labeled slack (0.03 kn)
✕ maxEbb at 2026-07-15T02:41:00Z: engine labeled maxFlood (0.61 kn)

Read that at face value and there is only one conclusion available: the engine
is mislabelling direction. For a current station, direction comes from a single
piece of metadata — the flood direction, the compass bearing the channel
floods toward, which the model projects the fitted velocity onto. Get that
bearing wrong by 180° and every flood becomes an ebb. It is a real, documented
failure mode: the published floodDirection is occasionally wrong for the
actual channel axis. So "the engine labelled ebb as flood" reads as "the flood
axis is reversed", which is a data-quality problem at the station, not a bug in
the model — and the response is to quarantine the station rather than ship a
current that runs backwards.

Here is the harness that produced it:

for e in fx.events where e.kind != "slack" {
    let t = parseISO(e.time)
    // Nearest computed event by TIME (avoids cross-cycle mis-match at weak,
    // mostly-single-direction stations), then require kind + tolerance.
    let m = try #require(
        computed.min { abs($0.time.timeIntervalSince1970 - t.timeIntervalSince1970)
                     < abs($1.time.timeIntervalSince1970 - t.timeIntervalSince1970) },
        "no computed event near \(e.time)")
    let kind: CurrentEventKind = e.kind == "maxFlood" ? .maxFlood : .maxEbb
    let timeErr = abs(m.time.timeIntervalSince1970 - t.timeIntervalSince1970) / 60
    let speedErr = abs(abs(m.speed) - abs(e.speed))
    #expect(m.kind == kind, "\(e.kind) at \(e.time): engine labeled \(m.kind) (\(m.speed) kn)")
    #expect(timeErr < 20, "\(e.kind) at \(e.time): time off \(timeErr) min (phase field wrong?)")
    #expect(speedErr < 0.3, "\(e.kind) at \(e.time): speed \(m.speed) vs \(e.speed)")
}

Four lines of idiom that every golden-data comparison in the world has written
at some point: find the nearest computed thing, then assert everything about
it.

Diagnosis: the matcher couples two independent errors

Two properties of the model are being measured here, and they are independent:

Timing — is the model's max ebb at the same minute as the published max ebb?
Direction — does the model think the water is going the same way?

The nearest-neighbour match couples them, because the search is over time and
the assertion is over kind. The candidate list from station.events() contains
slacks as well as extrema. So the match picks the wrong event as soon as the
timing error exceeds roughly half the spacing to the neighbouring event of a
different kind — and then the kind assertion fires on an event that was never
supposed to be the counterpart.

At a strong, cleanly semidiurnal station you never hit that condition: slack to
max is about three hours, so a matcher would need to be ninety minutes wrong
before it grabbed the wrong event. But the stations that got quarantined are
weak and slow-reversing — narrow, shallow, tidally awkward places where the
velocity loafs around zero, the model's own timing error runs to tens of
minutes, and the extrema are broad and low. That is exactly where the gap
collapses and the nearest event of any kind stops being the counterpart.

So the failure output is a lie in a very specific way. The engine is not
labelling anything wrong. It is late, and the matcher converts lateness into a
label-flip report. Two error dimensions, one nearest-neighbour search, and the
oracle can no longer tell you which one is broken.

The sign test settles it in one line and is not fooled by timing at all: ask
what the modelled velocity actually is at the moment the authority reports a
maximum ebb. At both quarantined stations it is negative at every single one —
0 wrong signs out of 19, and 0 out of 24. Directionally perfect. The
quarantine was manufactured by the comparison idiom.

What we tried (and why it failed)

Attempt 1: believe the failure and quarantine the stations. This is the
attempt that cost the most, because it worked in the sense that the test went
green and the bad data stopped shipping. Two stations dropped out of the offline
bundle with a note saying the flood axis was reversed, and downstream consumers
grew a stale line in their docs saying those stations "are known to flip."
Nothing about the quarantine looked wrong until someone asked why two stations
in completely different bodies of water, from different fits, would both have a
bad flood bearing.

Attempt 2: match nearest-by-time first, then require the kind. This is the
idiom above, and it deserves a proper Chesterton's-fence reading, because the
comment sitting on top of it is not decoration:

// Nearest computed event by TIME (avoids cross-cycle mis-match at weak,
// mostly-single-direction stations), then require kind + tolerance.

That comment is describing a real earlier bug. If you filter to the same kind
and then take the nearest, at a weak, mostly-single-direction station you can
match across a cycle boundary — grabbing yesterday's ebb, producing a wildly
wrong timing number. Matching by time first was a defence against that. It just
traded a loud, obviously-wrong timing number for a quiet, plausible direction
error, which is the worse trade: an absurd timing number gets investigated, a
credible-looking label flip gets believed.

Attempt 3: drop slacks from the candidate list. The obvious minimal patch —
if the nearest event is a slack, stop offering slacks:

let m = computed.filter { $0.kind != .slack }.min { /* nearest by time */ }

It removes the most common wrong match and none of the mechanism. The opposite
extremum is still in the list, and at a station running far enough late, the
nearest non-slack event to a published max ebb is the model's max flood. Same
coupled search, same false direction report, now with fewer symptoms — which
mostly means it takes longer to notice.

The fix

Stop asking one question and reading two answers out of it. Match same-kind for
timing; test direction at the golden timestamp, where timing cannot interfere:

/// Nearest computed event of the SAME KIND as the golden event.
private func nearestSameKind(_ computed: [CurrentEvent],
                             _ kind: CurrentEventKind, to t: Date) -> CurrentEvent? {
    computed.filter { $0.kind == kind }
        .min { abs($0.time.timeIntervalSince1970 - t.timeIntervalSince1970)
             < abs($1.time.timeIntervalSince1970 - t.timeIntervalSince1970) }
}

/// Sign of the station's velocity at the golden extremum time — the sound
/// direction test (flood positive, ebb negative).
private func signedSpeed(_ station: CurrentStation, at t: Date) -> Double {
    station.speeds(from: t.addingTimeInterval(-30), to: t.addingTimeInterval(30), step: 60)
        .first?.speed ?? 0
}

and then the loop asserts the two things separately:

let kind: CurrentEventKind = e.kind == "maxFlood" ? .maxFlood : .maxEbb
let m = try #require(nearestSameKind(computed, kind, to: t), "no computed \(e.kind) near \(e.time)")
let timeErr = abs(m.time.timeIntervalSince1970 - t.timeIntervalSince1970) / 60
let speedErr = abs(abs(m.speed) - abs(e.speed))

let v = signedSpeed(station, at: t)
#expect(e.kind == "maxFlood" ? v > 0 : v < 0,
        "\(e.kind) at \(e.time): modelled velocity \(v) kn has the wrong sign")
#expect(timeErr < 20, "\(e.kind) at \(e.time): time off \(timeErr) min (phase field wrong?)")
#expect(speedErr < 0.3, "\(e.kind) at \(e.time): speed \(m.speed) vs \(e.speed)")

Three assertions, three questions, no shared search. A late station now reports
one number — time off 22.4 min — and a reversed station reports a wrong sign.
Neither can impersonate the other.

The same separation, in the TypeScript harness that builds the offline bundle:

for (const event of observed) {
  const at = Date.parse(event.time);
  let best = Infinity;
  for (const candidate of predicted) {
    if (candidate.kind !== event.kind) continue;   // TIMING: same kind only
    best = Math.min(best, Math.abs(candidate.time.getTime() - at) / 60_000);
  }
  if (best <= MATCH_WINDOW_MIN) deltas.push(best);
}

let wrongSign = 0;
for (const event of extrema) {                      // DIRECTION: sign at the golden time
  const modelled = velocityAt(new Date(event.time));
  if ((event.kind === "maxFlood" && modelled < 0) ||
      (event.kind === "maxEbb" && modelled > 0)) wrongSign++;
}

Why it matters: same-kind matching is the safer test, not the laxer one

The instinct when you delete #expect(m.kind == kind) is that you have just
removed the check that catches a reversed axis. The opposite is true, and the
argument is worth internalising because it is what makes the fix safe to ship.

A genuinely reversed flood axis does not flip some labels. It flips every
event on the station, so the model's max ebbs land where the true max floods
are — about half a tidal cycle away, roughly six hours. Under same-kind
matching, the nearest computed max ebb to every published max ebb is therefore
six hours out, and blows through a ±20-minute timing gate at every single
extremum on the station.

So the reversed axis is caught twice — once loudly by the sign test, once
structurally by the timing gate — and a merely-late station is caught by
neither. Coupled, the two failures were indistinguishable. Separated, they can't
be confused: a real flip is systematic, a false one is occasional. The bundle
builder now encodes exactly that, quarantining only on systematic disagreement:

A reversed flood axis quarantines a station outright… The test is the sign of
the modelled velocity at CHS's own extremum times, and the bar is systematic
disagreement (≥ 60% of extrema): a genuinely reversed axis is wrong at nearly
every peak. Occasional wrong signs are timing error at a weak station, not a flip.

The transferable version

None of this is about tides. The shape is:

A nearest-neighbour matcher couples every dimension you didn't search on. You search on one axis (time, position, string distance, embedding distance) and then assert on others (kind, label, value). Every assertion downstream of the match inherits the search's error. The moment the search picks the wrong counterpart, the other assertions are the ones that fail — and they will fail with an entirely credible message about a problem you do not have.
The failure mode is a false positive that reads like a serious bug. A coupled matcher does not produce noise, it produces a specific, plausible, wrong diagnosis. That is worse than a flaky test: you act on it. We deleted two good stations from a navigation dataset and wrote the wrong explanation into the documentation downstream of it.
Match on the dimension you are measuring; test the others where the match can't reach. Same-kind matching for timing. Sign-at-the-expected-timestamp for direction. If a check can only be made after a match, ask whether a mismatch would make it lie.
Check that the strict version still catches the real bug — explicitly, with an argument, in a comment. Ours: a reversed axis is half a cycle away, so it fails the timing gate everywhere. If you can't make that argument, you have loosened the suite rather than fixed it.
A test-oracle suite needs its comparison layer audited like production code. Golden vectors get all the attention — where they come from, how they're generated, what tolerance they're gated at. The comparison idiom between the vectors and your output gets written once, in four lines, and never reviewed. It is just as capable of being wrong, and when it is wrong it discredits code that works.

The engine and the bundle builder are both MIT, and the tide/current work is
what keeps our own boat off the rocks in passes that only open for twenty
minutes: slackwater-engine
(Swift) and chs-constituents
(TypeScript). If a station near you disagrees with them, the validation output
will now tell you which of the two things is actually wrong.

A config warning on stdout made a healthy alarm lane look dead

Bryan Clark — Sun, 06 Sep 2026 16:04:57 +0000

TL;DR — A CLI reported a missing environment variable by printing a
Config warnings box to stdout. My pipeline was parsing that stdout, so
awk got box-drawing characters where data rows should be and matched
nothing. "Nothing" read as the alarm lane is dead, not your shell is
missing an env var. Forty minutes and one false outage report later:
prefix the environment for any command whose output you parse, not just
the ones that need the secret. Jump to the fix.

The problem

I run an agent gateway on a small boat server. It has an audit log, and the
cheapest health check for "are the agents actually running" is a one-liner over
the CLI:

gwcli audit --limit 12 | awk '$2=="agent_run"'

Which, one morning, returned nothing at all. No error. Exit code 0. Just an
empty result where a dozen rows had been the week before.

The obvious reading — the one I made, and the one that cost the morning — is
that there were no agent runs. Alarms flow through that gateway, so "no agent
runs" means the soft alarm lane is down: a low-battery or depth alarm on the
boat would raise, publish, and reach nobody. That's a real outage on the boat's
safety path, so it got reported as one.

The lane was healthy the entire time. The hook was firing about twelve seconds
after each test alarm, which is exactly the coalesce window. Nothing was down.

Here is what the command actually printed, once I stopped piping it:

$ gwcli audit --limit 12
┌ Config warnings ───────────────────────────────────┐
│ ! hooks.token: Missing env var GW_HOOKS_TOKEN      │
└────────────────────────────────────────────────────┘
ts                    kind        agent    status
2026-08-10T06:12:04Z  agent_run   watch    ok
2026-08-10T06:24:11Z  agent_run   watch    ok

The rows were there. So was a warning box, printed to stdout, sitting on top
of them. awk '$2=="agent_run"' doesn't crash on box-drawing characters; it just
doesn't match them, and it never got the chance to be confused, because in the
real invocation the box was all there was.

Diagnosis

The day before, I'd moved two credentials out of the gateway's config file and
into secret references, so the config could go under version control:

// ~/.gwcli/config.json
{
  "gateway": { "auth": { "token": { "$secret": "GW_AUTH_TOKEN" } } },
  "hooks":   { "token": "${GW_HOOKS_TOKEN}" }
}

Note the two forms. That asymmetry looks like sloppiness and it is not — more on
that below. The secrets themselves lived in an env file that the gateway's
launcher script sourced before exec'ing the daemon:

# run-gateway.sh
set -a; . "$HOME/.gwcli/secrets.env"; set +a
exec gwcli gateway

The daemon was fine, because the daemon goes through that script. A bare
gwcli ... typed in a shell, or run over SSH, does not. And with the env
missing, the two fields fail in two completely different, differently-visible
ways:

Field	Form	Channel	What you get
`gateway.auth.token`	typed secret ref	stderr	`SecretRefUnavailableError: ... unavailable in this command path`
`hooks.token`	`${VAR}` string	stdout	a `Config warnings` box, mixed into the data you were parsing

The stderr one is a good failure. It's loud, it names the field, and it stops
you. The stdout one is the dangerous half: it doesn't stop anything, it doesn't
look like an error to a program, and it arrives inside the data stream. Every
consumer that treats stdout as structured output — awk, jq, grep -c, a
Python subprocess.run(...).stdout — silently ingests a diagnostic as content.

And the two compound in the worst possible way, which is the part I'd want
another builder to take away:

# a completely normal scripting reflex
gwcli audit --limit 12 2>/dev/null | awk '$2=="agent_run"'

2>/dev/null suppresses the one failure that would have told me what was wrong,
and leaves the one that corrupts the answer. A clean-looking pipeline, producing
wrong data, with no visible error anywhere. That is a very hard thing to
distrust at 6 a.m.

What I tried, and why it failed

Attempt 1: prefix the env only where the secret is needed. My first mental
model was "this command reads the audit log, it doesn't need the gateway token,
so it doesn't need the env." Wrong — the config is parsed on every invocation,
so an unresolvable reference warns on every invocation, whether or not the
subcommand touches it:

$ gwcli audit --limit 12 | awk '$2=="agent_run"'     # env not prefixed
                                                      # (nothing — the box isn't a match)
$ set -a; . ~/.gwcli/secrets.env; set +a
$ gwcli audit --limit 12 | awk '$2=="agent_run"'     # env prefixed
2026-08-10T06:12:04Z  agent_run   watch    ok
2026-08-10T06:24:11Z  agent_run   watch    ok

Same command, same server, same healthy lane. The only variable was my shell.

Attempt 2: a wrapper, so the prefix stops being something to remember. A
tiny script that sources the env and execs the real binary:

#!/usr/bin/env bash
# gw — run gwcli with secrets resolved
set -euo pipefail
set -a; . "$HOME/.gwcli/secrets.env"; set +a
exec gwcli "$@"

This works, and it survived the day. But it only helps the caller who remembers
to type gw instead of gwcli, which is the same class of discipline that
failed in the first place.

Attempt 3: a --selftest that asserts the environment instead of trusting
it — meant for the top of any script that parses this CLI's output. This is
the one worth writing down, because it passed while genuinely broken.

The CLI reports the identical condition in two different formats depending on
the subcommand:

$ gwcli audit
┌ Config warnings ───────────────────────────────────┐
│ ! hooks.token: Missing env var GW_HOOKS_TOKEN      │
└────────────────────────────────────────────────────┘

$ gwcli config validate
1 warning(s): ! hooks.token: Missing env var GW_HOOKS_TOKEN

I wrote the check against config validate — the subcommand whose entire job is
to answer this question — and grepped for the string I'd been staring at all
morning:

# before — passes with the token genuinely missing
out="$(gwcli config validate 2>/dev/null || true)"
if grep -qi "Config warnings" <<<"$out"; then
  echo "SELFTEST FAIL"; exit 1
fi
echo "SELFTEST PASS"

config validate never prints Config warnings. The grep cannot match. The
selftest passed instantly and cleanly with a required credential absent — the
exact state it existed to catch. Note it also carries the 2>/dev/null reflex,
so the loud channel was gone there too.

What caught it was not review and not a test. It was running the negative
control: unsetting the variable on purpose and watching whether the check went
red. It didn't.

# after — match the substring both formats actually contain
if grep -qi "missing env var" <<<"$out"; then

The fix

The real fix landed a day later, and it deleted the wrapper's entire reason to
exist.

Buried in the gateway's own configuration docs: the CLI auto-loads a dotenv
file from its config directory on every invocation — daemon, CLI, cron,
service unit, all of them. The secrets file already existed. It was just named
something the tool didn't know to read.

mv ~/.gwcli/secrets.env ~/.gwcli/.env      # that exact filename is the fix
chmod 600 ~/.gwcli/.env

$ gwcli audit --limit 12 | awk '$2=="agent_run"'   # plain shell, no prefix, no wrapper
2026-08-10T06:12:04Z  agent_run   watch    ok
2026-08-10T06:24:11Z  agent_run   watch    ok

Both credentials now resolve for anyone who runs the binary, from anywhere. The
launcher script stopped sourcing anything. The wrapper survives for exactly one
unrelated reason — PATH, because the Node version manager puts the binary in a
directory that only an interactive shell's rc file adds, so ssh host 'gwcli …'
gets command not found. That's a different bug wearing the same coat.

The selftest stayed, because renaming a file doesn't stop it going missing or a
key rotating out:

gwcli config validate 2>/dev/null | grep -qi "missing env var" && exit 1

Why it matters, and the gotchas around it

Prefix the env for any command whose output you parse, not just the ones that
need the secret. That's the rule I'd hand to anyone else running a CLI inside
a pipeline. It's counter-intuitive precisely because the failing subcommand had
nothing to do with the missing credential.

Diagnostics belong on stderr — and you are not in control of that. This is
the oldest rule in the Unix toolbox and every CLI breaks it eventually, usually
via a pretty-printing layer that wraps console.log. Assume a tool you depend
on will one day put a banner in your data. The defences are cheap: pass whatever
--json / --quiet / --no-color flag exists and parse a structured format
rather than a table; or fail loudly when the shape is wrong, instead of treating
an empty match as a fact.

An empty result is not evidence. awk matching zero rows means "no rows
matched", which covers there is no data, the data moved, and what I handed
you wasn't data at all. My health check couldn't tell those apart, and the one
it silently chose was the alarming one. A parse that can return empty needs a
sanity assertion — did the header row arrive? — before its emptiness is allowed
to mean anything.

2>/dev/null is a load-bearing decision, not punctuation. In this incident
it was the difference between a five-second diagnosis and a false outage report.
If you're silencing stderr because a command is chatty, silence it after
you've checked the exit code, or route it to a file you actually read.

You haven't tested a check until you've watched it fail. Break the thing on
purpose, confirm red, put it back, confirm green. A selftest that passes when
broken is worse than no selftest — it converts an unknown into a false known,
and it is exactly the kind of green tick you'd cite while reporting an outage
that isn't happening.

One deliberate non-fix. The two-form asymmetry in the config — one field a
typed secret-reference object, the other a ${VAR} string — is schema-forced,
not style drift: the second field's schema is string-only and cannot take the
object form, which is why it's the one that emits a warning rather than
throwing. That's now a comment in the config, because it reads as an
inconsistency and the obvious "cleanup" would put the whole failure straight
back.

This is part of the agent stack I'm building for an all-electric charter
catamaran that doesn't exist yet — the same lane the false report was about is
the one that carries SignalK
alarms to a phone. It stayed up. My confidence in my own health check did not.

Related: Monitor the delivery path, not just the alarm — the same alarm lane failing for real, and why a quiet system produces nothing to notice; and Running OpenClaw on a Raspberry Pi alongside SignalK — the gateway this CLI drives.

MapLibre's SDF icon edge is at 0.75, not 0.5

Bryan Clark — Sat, 05 Sep 2026 18:57:05 +0000

TL;DR — If you hand-author the alpha channel of an SDF icon for
map.addImage(id, img, { sdf: true }), the edge does not go at alpha 128.
MapLibre's shader thresholds the fill at inner_edge = (256.0 - 64.0) / 256.0
= 0.75, i.e. alpha 191, over an 8-texture-pixel field. Encode the edge
at 128 and the shader draws a contour 4 px inside your shape — anything
thinner than 8 px vanishes completely, silently, with correct geometry and no
error of any kind. Use alpha = 255 * (1 - cutoff - d / SDF_PX) with
cutoff = 0.25, SDF_PX = 8. Jump to the fix.

We draw station pins on a MapLibre chart in
Slackwater, an offline
tide-and-current app for the Salish Sea. The pins need to change colour with
live state — flooding, ebbing, slack — and they need a halo so they stay legible
over bathymetry contours. In MapLibre, icon-color and icon-halo-color only
do anything on SDF icons, so the pin has to be registered as a signed
distance field.

The glyphs are simple stroke geometry — a sine wave for a current station, a
dome over a datum line for a tide station — so rather than rasterise a path and
hope, we computed the field analytically: exact distance from each pixel to the
stroke centreline, minus half the stroke width. A canvas-rasterised path gives
you a 1 px coverage ramp, which is enough to tint an icon but nowhere near
enough to hang a halo on.

The geometry was right. The icons rendered as nothing.

The problem: a correct icon that draws zero pixels

The registration is unremarkable, and it succeeds:

map.addImage(id, pinGlyphImage(kind), { sdf: true, pixelRatio: PIN_PIXEL_RATIO });

So is the layer:

{
  id: "station-pins",
  type: "symbol",
  source: "stations",
  layout: {
    "icon-image": ["match", ["get", "kind"], "current", "pin-current", "pin-tide"],
    "icon-size": 1,
    "icon-allow-overlap": true,
    "icon-ignore-placement": true,
  },
  paint: {
    "icon-color": ["match", ["get", "state"], "flood", "#4a9fd8", "ebb", "#e8a33d", "#7d9cb8"],
    "icon-halo-color": "#0b1a2b",
    "icon-halo-width": 1.5,
  },
}

And the field generator looked right too. signedDistance returns the exact
distance to the union of strokes, negative inside; the encoding was the obvious
one:

const SPREAD = 8;
// ...
const d = signedDistance({ x: x + 0.5, y: y + 0.5 }, strokes);
// maplibre's SDF convention: 128 is the edge, higher is inside.
const a = Math.round(255 * Math.max(0, Math.min(1, 0.5 - d / (2 * SPREAD))));

That comment is the whole bug. It is stated with total confidence and it is
wrong.

There is no error. map.addImage resolves, map.hasImage(id) returns true,
the source has features, the layer is in the style, nothing appears in the
console, and WebGL is perfectly happy. You get a map with correct data on it and
no pins.

Diagnosis: read the shader

MapLibre's icons and its text glyphs go through the same fragment shader. The
whole answer is six lines of
src/shaders/glsl/symbol_sdf.fragment.glsl
in maplibre-gl 5.24.0:

#define SDF_PX 8.0
// ...
lowp float inner_edge = (256.0 - 64.0) / 256.0;   // line 34  -> 0.75
lowp float dist = texture(u_texture, tex).a;
// ...
highp float alpha = smoothstep(inner_edge - gamma_scaled, inner_edge + gamma_scaled, dist);

dist is your alpha channel, normalised to 0–1. The shader paints the contour
where dist equals 0.75 — alpha 191 — not 0.5. (In the sibling
symbol_text_and_icon shader the same constant is spelled buff, which is the
older Mapbox name you will see quoted around the web; the value is identical.)

Now the arithmetic. Take any linear encoding of the form
alpha = 255 * (A - d / R), where A is the value you put at d = 0 and R
is the distance over which the field ramps. The shader draws the contour where
A - d / R = 0.75, that is:

d = R * (A - 0.75)

That single line explains everything:

encoding	A	R	contour the shader draws
ours (broken)	0.5	16	`d = -4 px` — 4 px inside the true edge
edge at 0.5, correct ramp	0.5	8	`d = -2 px` — 2 px inside
correct	0.75	8	`d = 0` — the true edge

A has to be 0.75 or the shader renders an eroded version of your shape. R
does not affect where the edge lands at all — it only sets the ramp rate, which
is what halo widths and antialiasing are scaled against.

Our strokes are 0.14 * 44 = 6.16 texture pixels wide, so the deepest interior
point sits at d = -3.07. A 4 px erosion of a 6.16 px stroke leaves nothing:

Run the numbers over the actual field and the result is exact: peak alpha 176,
threshold 191.25, 0 of 1936 pixels above it. The antialiasing band does not
save you either — EDGE_GAMMA = 0.105 / u_device_pixel_ratio puts the
smoothstep's lower edge around alpha 178 at devicePixelRatio 2, still above the
field's maximum.

Why 0.75 is not an arbitrary number

The intuition that says "0.5" is really the intuition that the field should be
symmetric around the edge. MapLibre's is deliberately asymmetric, and the
reason is halos.

The encoding comes from
@mapbox/tiny-sdf,
whose defaults are radius = 8, cutoff = 0.25:

const scale = 255 / this.radius;
const base = 255 * (1 - this.cutoff);
for (let i = 0; i < len; i++) {
    const d = Math.sqrt(gridOuter[i]) - Math.sqrt(gridInner[i]);
    data[i] = Math.round(base - scale * d);
}

which is alpha = 255 * (1 - cutoff - d / radius). With cutoff = 0.25 and
radius = 8 the field saturates 2 px inside the shape and runs out 6 px
outside. The edge therefore sits 6 / 8 = 0.75 of the way up the range — and
that is the same 0.75, arrived at from the other direction. MapLibre uses
exactly these constants for its own glyph atlas, in
glyph_manager.ts
(radius: 8 * textureScale, cutoff: 0.25).

The 6 px is not a coincidence either. It reappears in the halo branch of the
same shader:

highp float halo_edge = (6.0 - halo_width / fontScale) / SDF_PX;   // line 53

Six pixels of outside room is precisely the widest halo the shader can express.
That is what the asymmetry buys.

What we tried (and why it failed)

The test that asserted the bug

The glyph generator shipped with a unit test written specifically to pin down
the alpha encoding. It passed:

it("puts the shape edge at alpha 128 — inside is brighter, far outside is dark", () => {
  const img = pinGlyphImage("tide");
  const mid = Math.floor(img.width / 2);
  expect(alphaAt(img, 0, 0)).toBeLessThan(128);
  const column = [];
  for (let y = 0; y < img.height; y++) column.push(alphaAt(img, mid, y));
  expect(Math.max(...column)).toBeGreaterThan(128);
});

Corner alpha is 0, which is less than 128. Peak column alpha is 176, which is
greater than 128. Green, on a glyph that renders zero pixels.

The test asserted the same wrong constant the implementation did, so it could
only ever confirm the code agreed with itself. toBeGreaterThan(128) is
satisfied by 176 — the value that proves the bug — and by 255. The assertion had
no power exactly where it mattered.

"Just tighten the spread until something appears"

The tempting empirical fix, once you know the icons are invisible but not
why, is to make the field steeper and see what shows up. It does eventually
produce marks, which is the trap:

ramp span `R`	contour drawn	what you see	outside room left
16	`d = -4.00 px`	nothing	8 px
12	`d = -3.00 px`	a 0.16 px thread	6 px
10	`d = -2.50 px`	a 1.16 px hairline	5 px
8	`d = -2.00 px`	2.16 px of a 6.16 px stroke	4 px

At R = 8 with the edge still at 0.5 you get something that looks plausible on
screen — a thin version of the glyph — and you ship a shape that is 2 px
narrower everywhere than the one you drew, with a field that runs out 4 px
outside instead of 6, so icon-halo-width values above 4 quietly clip. Tuning
by eye lands you here.

"Add a constant so the edge reads 192"

Also close, also wrong:

// edge lands at 192 — but the ramp rate is still 1/16 per px
const a = Math.round(255 * (0.5 - d / 16)) + 64;

The contour now falls at d = 0, so the shape is the right size. But the ramp
is half as steep as SDF_PX = 8 assumes, so every halo width comes out at the
wrong scale, and the far field bottoms out at a floor of 64 instead of 0 — the
alpha channel never reaching zero on an image you also might want to reuse
non-SDF. Two knobs, and this only turns one of them.

The fix

Encode the edge at the threshold the shader actually uses, and ramp at the rate
it actually assumes:

/**
 * maplibre's SDF constants, not ours to choose: the shader thresholds the icon
 * fill at `inner_edge = (256 - 64) / 256 = 0.75` over a field authored with
 * `SDF_PX = 8` texture pixels. Matches `@mapbox/tiny-sdf`'s `cutoff = 0.25`.
 */
const SDF_PX = 8;
const SDF_CUTOFF = 0.25;

const d = signedDistance({ x: x + 0.5, y: y + 0.5 }, strokes);
// 255 * (1 - cutoff - d/SDF_PX): edge (d=0) lands at ~191, the interior
// saturates to 255, and the field reaches 0 at 6px outside the stroke.
const a = Math.round(255 * Math.max(0, Math.min(1, 1 - SDF_CUTOFF - d / SDF_PX)));

One line, and it is just tiny-sdf's own formula. The edge lands at 191, the
interior saturates at 255, the field runs out 6 px outside.

The test gets the real invariant — saturation, not "brighter than the value we
assumed":

it("puts the shape edge at maplibre's 0.75 threshold, and saturates inside", () => {
  const img = pinGlyphImage("tide");
  const mid = Math.floor(img.width / 2);
  expect(alphaAt(img, 0, 0)).toBe(0);
  const column = [];
  for (let y = 0; y < img.height; y++) column.push(alphaAt(img, mid, y));
  expect(Math.max(...column)).toBe(255);
  expect(column.some((a) => a > 180 && a < 205)).toBe(true); // the edge band
});

toBe(255) is the assertion that has teeth. An interior that never reaches 255
is an interior that may never cross 0.75, and that is the failure this whole
post is about.

Why it matters, and the traps next door

The failure mode is silence. There is no error string to search for, which
is why this costs hours rather than minutes — you go looking for a bug in your
geometry, your projection, your GeoJSON, your collision settings. The symptom is
an absence.

It is worse than silent, actually. Our pins carry
"icon-halo-color": "#0b1a2b" — the same value as the map's background. The
halo branch thresholds at halo_edge = (6.0 - halo_width / fontScale) / SDF_PX,
which for any non-zero halo width is below the 0.75 fill threshold. So a
field that is too dark to draw a fill can still be bright enough to draw a halo.
With a halo colour that matches the water, the glyph paints itself in the
background tone. Worth knowing before you conclude "nothing rendered."

The docs do not tell you. MapLibre's
StyleImageMetadata
documents the sdf flag in full as: "Whether the image should be interpreted
as an SDF image." That is the entire specification of the alpha format.
mapbox/mapbox-gl-style-spec#97
— "Style reference: document sdf icons" — was opened in 2014 and never resolved;
maplibre/maplibre-native#2551
asks for the same thing and is still open. The one human-readable statement of
the number I could find anywhere is a paragraph on a Mapbox
troubleshooting page about recolorable images:
"values between 192 and 255 represent 'inside' a glyph and values from 0 to 191
represent 'outside'." Correct, and filed under glyphs, and never connected to
addImage.

The scale-invariant part is the part you must get right. A = 0.75 is fixed
— it is a threshold on a normalised alpha, so it holds whatever resolution you
author at. R scales with your image: we author at pixelRatio: 2 with
R = 8, while MapLibre's own glyph atlas uses radius: 8 * textureScale.
Getting A wrong makes your icon vanish; getting R wrong makes your halo
widths come out at the wrong scale. Check the halo by eye, but check the edge by
arithmetic.

And the general one. A test that asserts a constant the implementation also
asserts is not a test, it is a restatement. The useful assertion was not "is the
edge where we think" — it was "does the interior saturate," a property with a
single correct answer that does not depend on the belief being tested.

Slackwater is the offline tide and current app for the Salish Sea we run
alongside the boat's software stack — the sort of thing you want working at
anchor with no signal. Code:
slackwater-web,
src/pinGlyphs.ts.

Related: porting a tide engine to Swift with the original as the test oracle — the other half of getting numeric code to tell you when it is wrong, and offline tidal current predictions from harmonic constituents — what the pins in this post are actually showing.

claude setup-token fixed my launchd auth and dropped my connectors

Bryan Clark — Tue, 01 Sep 2026 23:54:55 +0000

If you added CLAUDE_CODE_OAUTH_TOKEN to a scheduled claude -p job, your claude.ai connectors — Gmail, Google Calendar, Google Drive — are gone, and nothing told you. A setup-token "can only make model requests", and it sits above your /login session in Claude Code's auth precedence, so setting it silently demotes the session the connectors ride on. They don't error; they never enter the tool list. Keep the keychain session primary and make the token an explicitly-degraded fallback. Jump to the fix.

I run a daily scheduled agent job on a Mac: a launchd agent fires a headless claude -p at 06:05, the agent reads a planning repo, sweeps my email and calendar for anything due, and writes a standup file. It has been reliable for months. Then it was blind for four days and every one of those runs told me it had succeeded.

The problem: launchd, a keychain, and a fix that worked

One morning the job just didn't produce anything. The log had the useful part:

=== daily-standup 06:05 ===
OAuth session expired and could not be refreshed
=== daily-standup FAILED (exit 1): no artifact written ===

(On current builds this surfaces as Login expired · Please run /login; the CLI documents both the warning and the expired state.)

This is the classic unattended-macOS problem. Claude Code stores subscription credentials in the encrypted login Keychain, and a job launched by launchd on a machine that woke from sleep is not the same thing as you sitting at a terminal. There is a long-running open bug where a process tree rooted at launchd reports "not logged in" against credentials that security find-generic-password can read perfectly well from the same context. Whatever the precise cause on any given day, the symptom is the same: an interactive login is not a durable credential for a background job.

Anthropic documents the answer, and it is a good one:

For CI pipelines, scripts, or other environments where interactive browser login isn't available, generate a one-year OAuth token with claude setup-token.

— Claude Code docs, Authentication

So I ran it, put the token in the env file the job already sources, and moved on:

# scheduled wrapper — the version that "fixed" it
set -a                          # the env file is bare KEY=val with no `export`,
source "$HOME/.agent/.env"      # so allexport is what makes the child inherit it
set +a

claude --print --permission-mode bypassPermissions --model "$MODEL" "$PROMPT"

That env file holds about twenty keys — analytics credentials, a couple of API keys, and now the Claude token. One line, one variable, done. The job ran green the next morning and every morning after.

Diagnosis: the run wasn't failing, it was half-blind

Four days later I noticed the standups had stopped mentioning email. Not "failed to read email" — just nothing, as if the week had been quiet. The agent's own summary of what it swept listed the repo and nothing else, and I'd skimmed past it four times.

The check took thirty seconds once I thought to run it:

$ claude mcp list
claude.ai Gmail              ✓ Connected
claude.ai Google Calendar    ✓ Connected
claude.ai Google Drive       ✓ Connected
signalk                      ✓ Connected

$ CLAUDE_CODE_OAUTH_TOKEN=sk-ant-oat01-… claude mcp list
signalk                      ✓ Connected

Same machine, same second, same config. The only difference is one environment variable, and three servers vanish. Not "failed", not "needs authentication" — absent.

Why: connectors ride on the account session, and the token outranks it

Two things in the docs explain it completely, and I had read neither, because I was solving a launchd problem and this is filed under authentication and MCP.

One. A setup-token is deliberately narrow:

This token authenticates with your Claude subscription and requires a Pro, Max, Team, or Enterprise plan. It can only make model requests, so it can't establish Remote Control sessions or fetch claude.ai connectors. MCP servers you configure locally still work.

— Authentication § Generate a long-lived token (emphasis mine)

Two. The MCP page spells out the exclusion list, and the last bullet is this exact situation:

Connectors from claude.ai are fetched only when your active authentication method is a claude.ai subscription login. They aren't loaded, even if you previously ran /login, when:

ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, or apiKeyHelper is active

A third-party provider such as Amazon Bedrock or Google Cloud's Agent Platform is active

ANTHROPIC_PROFILE, the federation variables, or an active Anthropic profile supplies the credential

CLAUDE_CODE_OAUTH_TOKEN holds a token from claude setup-token, which can only make model requests

— Connect Claude Code to tools via MCP § Use MCP servers from claude.ai

The mechanism is the authentication precedence list. CLAUDE_CODE_OAUTH_TOKEN is rank 5. Your /login subscription credential is rank 7 — dead last:

Rank	Credential	claude.ai connectors?
1	Cloud provider (Bedrock / Vertex / Foundry)	no
2	`ANTHROPIC_AUTH_TOKEN`	no
3	`ANTHROPIC_API_KEY`	no
4	`apiKeyHelper` output	no
5	`CLAUDE_CODE_OAUTH_TOKEN` (`setup-token`)	no
6	Anthropic profile / federation credentials	no
7	Subscription OAuth from `/login`	yes

Read as a table, the shape of the trap is obvious: connectors are available under exactly one of seven credential sources, and it's the one anything else beats. Setting any of the six above your login costs you the connectors. setup-token is just the one you're most likely to add on purpose, at 6am, to fix something else.

The part that actually cost four days

Getting it wrong is forgivable. Not being told is the expensive bit, and Claude Code is otherwise good about this. For a server you configured locally that needs authentication, headless runs get an explicit signal — as of v2.1.196, "Claude Code tells Claude that the server's tools are unavailable until you authorize it," so the model can name the server instead of answering as if it were never configured.

Connectors dropped by auth method get none of that, and structurally can't: they were never fetched, so there is no server object to mark unavailable. The list the model receives is simply shorter. The model has no way to know, and answers confidently from what's left.

So the run:

exits 0
writes its artifact
reports success
and is missing half its inputs

Which is a much worse failure than the crash I was fixing. The crash cost me one standup and announced itself before breakfast.

What I tried, and why each one failed

Attempt 1 — put the token in the shared env file. This is the whole bug, and its real lesson isn't about Claude at all: source-ing a general-purpose env file into an agent runner is an authentication change, not a configuration convenience. The blast radius of set -a; source .env is every variable in that file, including the ones another tool put there. I added a key to make a scheduler happy and silently changed which account session my agent ran under.

Attempt 2 — trust the exit code. The wrapper's health check was "did claude exit non-zero", and every degraded morning it said yes. Exit 0 from an agent run is a claim about the process, not about the work. Worse, I also had the agent report its own coverage in the output — and it did, accurately, listing only the sources it could see. A self-report is only as good as the reader; four days of "swept: repo" scrolled by unread.

Attempt 3 — check for today's artifact by exact filename.

ARTIFACT="reports/$(date +%F).md"
[[ -f "$ARTIFACT" && "$(stat -f %m "$ARTIFACT")" -ge "$STAMP" ]]

This looks airtight and isn't. On the first morning of the fix, the agent inferred the wrong date and filed a perfectly good report under a filename three days in the future. The check saw no reports/2026-08-01.md, declared total failure, and fired a redundant retry over a successful run. Two fixes fell out: the heartbeat should ask "did a new artifact appear", not "did this filename appear", and the prompt should never let the model infer the date:

PROMPT="TODAY IS $(date +%F). Use that date for the filename and every date computation — do not infer it.

${PROMPT}"

Attempt 4 — find -newermt for "any new artifact". The obvious one-liner:

wrote_artifact() { [[ -n "$(find reports -name '*.md' -newermt "@$STAMP" -print -quit 2>/dev/null)" ]]; }

It returned false inside the script while the same predicate, same STAMP, same directory, returned true when I ran it by hand. It fired two more redundant degraded runs before I gave up guessing and made the check say what it saw:

echo "heartbeat: STAMP=$STAMP now=$(date +%s) newest=$(ls -t reports/*.md | head -1) mtime=$(stat -f %m "$(ls -t reports/*.md | head -1)")"

The mtime was comfortably newer than STAMP. find disagreed anyway. I never did root-cause the find(1) quirk, and I'm at peace with that: two stdlib commands answer the question outright, so the correct move was to delete the clever predicate rather than debug it. A watchdog that produces false negatives is worse than no watchdog — it trains you to ignore it.

The fix

Two changes. Keychain session primary, token as an explicitly-degraded fallback, and success means "the artifact exists and is newer than this run started":

STAMP=$(date +%s)

# Ground truth: did THIS run produce an artifact? Not the exit code, not the
# agent's self-report. `stat -f %m` is BSD/macOS; GNU coreutils is `stat -c %Y`.
wrote_artifact() {
  local newest
  newest=$(ls -t reports/*.md 2>/dev/null | head -1)
  [[ -n "$newest" && "$(stat -f %m "$newest" 2>/dev/null || echo 0)" -ge "$STAMP" ]]
}

# Primary run: the claude.ai /login session, with the token pulled OUT of the
# environment so it can't outrank it. This is the only credential that carries
# the Gmail / Calendar / Drive connectors.
OAT="${CLAUDE_CODE_OAUTH_TOKEN:-}"
unset CLAUDE_CODE_OAUTH_TOKEN
claude --print --permission-mode bypassPermissions --model "$MODEL" "$PROMPT"
rc=$?

# Fallback: an expired session must not cost a whole day's run — retry with the
# token, and say out loud that this one is degraded.
if [[ -n "$OAT" ]] && ! wrote_artifact; then
  echo "primary (keychain session) run wrote nothing, exit $rc — retrying with CLAUDE_CODE_OAUTH_TOKEN, DEGRADED: no Gmail/Calendar/Drive"
  CLAUDE_CODE_OAUTH_TOKEN="$OAT" claude --print --permission-mode bypassPermissions --model "$MODEL" "$PROMPT"
  rc=$?
fi

wrote_artifact || { echo "FAILED (exit $rc): no artifact written"; exit 1; }

The token still earns its place — it's the difference between a degraded run and no run — it just isn't allowed to be the default. And the one-line verification, which is what I should have run on day one:

# run this through the SAME wrapper the scheduler uses, not from your shell
claude mcp list | grep claude.ai

Why it matters, and the traps next door

The connector rule generalises past setup-token. Anything you set to make a headless run authenticate — an ANTHROPIC_API_KEY for billing separation, an apiKeyHelper pulling short-lived creds from a vault, a Bedrock switch — takes your connectors with it. The docs note the same for Anthropic profiles: "Features that need your claude.ai login, such as claude.ai connectors and /schedule, aren't available while one of these sources is selected." If a scheduled agent needs Gmail or Calendar, it needs the subscription login and nothing above it.

Verify under the job's environment, not yours. Every check I ran interactively passed, because interactively the token wasn't set. claude mcp list piped through the same wrapper the scheduler invokes is a different command than the one you type.

Anthropic's own answer for this shape of work is routines, not cron — scheduled cloud runs that carry your connectors by default. /schedule requires a claude.ai subscription login for exactly the reason above. If your scheduled job's whole value is the connectors, that's the shorter path; I keep mine local because it needs the machine's filesystem and a private repo clone.

Exit 0 is not success — name the artifact. This is the durable one, and it isn't about Claude Code. Every unattended agent needs a success criterion expressed as a fact about the world: a file written after the run started, a row in a table, a message delivered. Process exit status tells you the interpreter didn't crash. Ask the filesystem instead.

A degraded mode must announce itself. The fallback path prints DEGRADED: no Gmail/Calendar/Drive into a log a human eventually reads, and the agent reports its actual coverage in its output. A silent fallback is just the original bug with more steps.

Version note. The claude mcp list behaviour above is my observation on Claude Code 2.1.22x (August 2026) — the docs describe connector visibility in terms of /mcp and /status, not claude mcp list, so treat the exact output as illustrative. The underlying rule is documented and stable. Separately, there are community reports of connectors also going missing in -p mode with a perfectly good login (#36060, #43298) — a different mechanism from this one, and worth ruling out before you blame your auth.

This scheduled job files commitments for an all-electric charter catamaran that doesn't exist yet — the agent stack around it is public at naturali-agents. The boat is the reason the "did anything actually happen" question matters more than the exit code: nobody is watching the log at 06:05.

Related: launchd's minimal PATH breaks MCP servers: uv command not found — the same class of trap one layer down, where the scheduler's environment silently isn't yours; When MCP tools break, isolate the server before blaming the agent — the debugging order for missing tools; and Monitor the delivery path, not just the alarm — the dead-man's-switch pattern this heartbeat is an instance of.

Your service worker's precache can't do Range requests — PMTiles needs 206

Bryan Clark — Tue, 01 Sep 2026 16:04:38 +0000

TL;DR — Workbox's precache stores and replays a full 200 OK. pmtiles
reads the archive with HTTP Range requests and requires 206 Partial Content, so a precached .pmtiles renders fine on a cold first visit — when
no service worker is in the path yet — and throws on every visit after that.
Take .pmtiles out of globPatterns and serve it from a CacheFirst
runtimeCaching route with rangeRequests: true, plus one non-ranged warm
fetch, because a ranged request can never populate that cache on its own.
Jump to the fix. Repo:
slackwater-web.

Slackwater is an offline
tide and current app for the Salish Sea — a PWA that computes predictions on
device so it still works with no signal, which is the only condition that
matters when you're actually in Dodd Narrows. Its map screen draws a coastline
from a committed 4.78 MB PMTiles archive (public/land.pmtiles, OSM land
polygons clipped to the Salish Sea, z0–14) rendered by MapLibre. No tile
server, no network.

Getting that archive to survive offline took two tries, and the failure mode of
the first one is the entire post: it worked on the first visit and broke on
every visit after.

The problem: the map draws once, then never again

The obvious way to make a file available offline in a vite-plugin-pwa app is
to precache it. That's one glob entry:

// vite.config.ts — the first attempt
workbox: {
  globPatterns: ["**/*.{js,css,html,svg,png,json,woff2,pmtiles}"],
  // land.pmtiles is 4.78 MB; the default 2 MB cap would silently skip it
  // and the map would have no land offline.
  maximumFileSizeToCacheInBytes: 6 * 1024 * 1024,
},

Load the app in a fresh tab: coastline draws. Reload it: blank map, and this in
the console —

Error: Server returned no content-length header or content-length exceeding
request. Check that your storage backend supports HTTP Byte Serving.

That string is thrown by pmtiles' own FetchSource.getBytes, and it is
correct. Something in the path really did return the whole file instead of
the slice it asked for. It just wasn't the storage backend.

Diagnosis: precache is 200-shaped, and it only joins the path on visit two

Two facts collide.

1. A precache response is always a full 200. Workbox precaches by
cache.put()-ing complete responses, and the Cache API flatly refuses to store
a partial one — try it and you get
TypeError: Failed to execute 'put' on 'Cache': Partial response (status code 206) is unsupported.
So the precache route can only ever answer with the entire body and status 200.
It doesn't read the Range header; it has nothing to slice.

2. pmtiles treats a 200 as a hard error, deliberately. Here's the check, from
the pmtiles source:

// some well-behaved backends, e.g. DigitalOcean CDN, respond with 200 instead of 206
// but we also need to detect no support for Byte Serving which is returning the whole file
const contentLength = resp.headers.get("Content-Length");
if (resp.status === 200 && (!contentLength || +contentLength > length)) {
  if (controller) controller.abort();
  throw new Error(
    "Server returned no content-length header or content-length exceeding request. Check that your storage backend supports HTTP Byte Serving."
  );
}

A 200 whose Content-Length exceeds what was requested means byte serving isn't
working — pmtiles aborts rather than silently downloading a multi-gigabyte
archive to read a 16 KB directory. Against a Workbox precache entry that's
exactly what it sees, every time.

And the timing is what hid it. A service worker only controls navigations
that start after it activates. On a first-ever visit the SW registers,
installs and activates, but the page that installed it was never controlled —
so /land.pmtiles went straight to the static host, which handles Range
properly and answered 206. Everything worked. On the next load the SW is in
the path, the precache answers, and it throws.

The bug is invisible in exactly the state you test in (fresh incognito tab,
hard reload after a deploy) and present in the only state a user is ever in.

What we tried, and why it failed

Attempt 1 — precache it and raise the size cap

The config above. Worth keeping the sub-lesson even though the approach was
wrong: maximumFileSizeToCacheInBytes defaults to 2 MB, and a file over
that is dropped from the precache manifest with a build-log line and no
runtime error. The app just quietly has no offline copy. If you precache
anything big — a tile archive, a model, a wasm blob — check the manifest, don't
assume the glob matched.

Attempt 2 — a smoke test that agreed with us

The offline smoke check was green throughout. It was green because it had been
written to be:

// M5: /map opens the discovery map directly. This has to run as the very
// first navigation, before anything else on this origin installs the PWA
// service worker: once the SW is controlling, its default Workbox precache
// route serves the precached land.pmtiles as a plain 200 with no
// Range/Content-Length support, and pmtiles' own byte-serving check throws
// on that — a real bug (land.pmtiles needs workbox-range-requests wired to
// its precache route to survive a returning/offline visit), but out of scope
// for this smoke-only task.

The comment names the bug, in full, and then orders the test to avoid it. A
passing check is a claim about what ran, not about what's true — and here the
claim was "the cold path works", which was never in doubt.

Attempt 3 — runtimeCaching with rangeRequests, and an empty cache

The documented answer is a runtimeCaching route with rangeRequests: true,
which wires up
workbox-range-requests:

runtimeCaching: [
  {
    urlPattern: ({ url }) => url.pathname.endsWith(".pmtiles"),
    handler: "CacheFirst",
    options: {
      cacheName: "land-pmtiles",
      rangeRequests: true,
      cacheableResponse: { statuses: [200] },
    },
  },
],

Ship that alone and the map still doesn't work offline — and now
caches.open("land-pmtiles") is empty. Nothing was ever stored.

The reason is in the plugin's one and only callback:

// workbox-range-requests/src/RangeRequestsPlugin.ts
cachedResponseWillBeUsed: WorkboxPlugin['cachedResponseWillBeUsed'] = async ({
  request,
  cachedResponse,
}) => {
  // Only return a sliced response if there's something valid in the cache,
  // and there's a Range: header in the request.
  if (cachedResponse && request.headers.has('range')) {
    return await createPartialResponse(request, cachedResponse);
  }

  // If there was no Range: header, or if cachedResponse wasn't valid, just
  // pass it through as-is.
  return cachedResponse;
};

cachedResponseWillBeUsed — that's it. The plugin never touches the outbound
request. It slices a response you already have; it has no opinion on how you
got one.

So the first request to reach the route is pmtiles' own, carrying Range: bytes=0-16383. CacheFirst misses, goes to the network, and the origin
correctly answers 206. Then cacheableResponse: { statuses: [200] } refuses
to store it — and you can't widen that to [200, 206], because the Cache API
rejects a 206 on put() anyway. The cache stays empty, the next read misses
again, and the loop never closes.

The fix

Two pieces: the route, and one plain fetch that primes it.

// vite.config.ts
workbox: {
  // pmtiles is deliberately NOT globbed here: a precache entry is a plain 200
  // with the full body and no Range support, but pmtiles reads the archive
  // with Range requests and needs 206.
  globPatterns: ["**/*.{js,css,html,svg,png,json}"],
  runtimeCaching: [
    {
      urlPattern: ({ url }) => url.pathname.endsWith(".pmtiles"),
      handler: "CacheFirst",
      options: {
        cacheName: "land-pmtiles",
        rangeRequests: true, // serve 206 partials from the cached full body
        cacheableResponse: { statuses: [200] },
        expiration: { maxEntries: 4, maxAgeSeconds: 60 * 60 * 24 * 365 },
      },
    },
  ],
},

// src/main.tsx — the warm fetch, without which the cache above is never filled
if ("serviceWorker" in navigator) {
  const warm = () => {
    fetch("/land.pmtiles").catch(() => {});
  };
  navigator.serviceWorker.ready.then(() => {
    if (navigator.serviceWorker.controller) warm();
    else navigator.serviceWorker.addEventListener("controllerchange", warm, { once: true });
  });
}

That fetch carries no Range header, so it gets a full 200, which
cacheableResponse accepts and stores. From then on every ranged read that
pmtiles issues hits the cache and RangeRequestsPlugin slices the stored body
into the 206 it wanted — offline, forever.

The one-line version: rangeRequests: true is a read path, not a write path.
You still have to get a whole 200 into the cache yourself.

Gotchas

`serviceWorker.ready` does not mean the service worker controls your page

ready resolves once there's an active registration for the scope. It says
nothing about navigator.serviceWorker.controller, and with
registerType: 'prompt' those two diverge on the first load: 'prompt' does
not set clientsClaim, so the page that installed the worker is never
controlled. Fire the warm fetch on ready alone and on a first-ever visit it
goes straight to the network, bypasses the route entirely, and caches nothing.

Hence the two-branch guard: run it now if controller is already set,
otherwise wait for controllerchange — which is what fires when a worker
does claim, whether that's registerType: 'autoUpdate' (vite-plugin-pwa
forces clientsClaim and skipWaiting for that mode) or an update takeover
later in the session.

Test the controlled path, because that's the one users are on

The rewritten smoke check stops dodging the service worker and goes looking for
it: load online, wait for the SW, reload to gain control, wait for the
cache entry to actually exist, then cut the network and reload again.

await mapPage.goto(`${URL}map`, { waitUntil: "domcontentloaded" });
await mapPage.waitForSelector(".map-canvas .maplibregl-canvas", { timeout: 10_000 });
await mapPage.evaluate(() => navigator.serviceWorker.ready);

// NOW the active SW controls the page, so the warm fetch and the map's own
// pmtiles reads go through the runtime Range route.
await mapPage.reload({ waitUntil: "domcontentloaded" });
await mapPage.waitForFunction(() => navigator.serviceWorker.controller != null, {
  timeout: 15_000,
});
await mapPage.waitForFunction(
  async () => (await (await caches.open("land-pmtiles")).match("/land.pmtiles")) != null,
  { timeout: 15_000 },
);

const mapCdp = await mapPage.createCDPSession();
await mapCdp.send("Network.enable");
await mapCdp.send("Network.emulateNetworkConditions", {
  offline: true, latency: 0, downloadThroughput: -1, uploadThroughput: -1,
});

await mapPage.reload({ waitUntil: "domcontentloaded" });
// This is the assertion the fix exists for.
await mapPage.waitForSelector(".map-canvas .maplibregl-canvas", { timeout: 10_000 });

Waiting on .maplibregl-canvas rather than the container div matters —
the container mounts whether or not MapLibre ever got a WebGL context up. And
don't trust navigator.onLine to tell you the emulation took: under CDP
offline it lies. Prove the network is down with a real fetch() to an
unreachable origin first, and attach your error listeners after that probe,
or Chrome's own resource-load error for the probe lands in your assertions.

The nice side effect

Once .pmtiles left the precache, the largest precached chunk was the map
screen's ~1 MB of JS — comfortably under the 2 MB default — so the
maximumFileSizeToCacheInBytes bump could go away. Fewer knobs is the correct
outcome of a fix.

Slackwater is the offline tide app I want in my pocket on an all-electric
charter catamaran with no cell signal, so "works on a return visit" isn't a
nice-to-have. Code: slackwater-web.

Related: Porting a tide engine to Swift with the original as the test oracle — the prediction engine behind this same app; and A persistent-queue HTTP reporter for distress traffic over flaky internet — offline-first on the other side of the wire.

A passing check is a claim about what ran, not what's true

Bryan Clark — Mon, 31 Aug 2026 16:04:53 +0000

TL;DR — A green check tells you a code path ran and produced the value it
was compared against. It does not tell you the subject was examined. Six
independent bugs in one month, across four unrelated codebases, were all the
same shape: something reported success for a region it never looked at.
Jump to the checklist.

I hit this six times in a month. Different languages, different repos, no shared
code: a JavaScript data package, a Python MCP server, a shell wrapper, a
TypeScript PWA. Every one of them produced a clean result for something that was
never actually examined — not a wrong answer, an absent answer, dressed as a
pass.

That's a nastier failure than a wrong answer. A wrong answer is a signal. An
absent answer that reports itself as a pass is silence you've been trained to
read as safety.

Here they are, shortest first, each with the rule I'd generalize out of it.

1. A boolean that returns false for "no" and for "I can't tell"

station-corrections
exists to audit the published positions of tide and current stations. Its
headline check: is this station's coordinate sitting on land? That's a
point-in-polygon test against a bundled coastline:

// src/coastline.js — before
export function isOnLand(lat, lon) {
  const at = point([lon, lat]);
  return coastline.features.some((feature) => booleanPointInPolygon(at, feature));
}

The coastline ships clipped to a bounding box — the full planet coastline is far
too large to bundle. Outside that box there are no polygons, so some() finds
nothing and the function returns false. Which is byte-identical to the answer
it returns for a station verified to be in open water.

Three CHS current gates — Blackney Passage, central Johnstone Strait, Weynton
Passage — sat north of the clip. The audit reported them valid. It had never
looked at them. And the registry was growing north, so the silent region was
growing too.

The fix is a coverage predicate derived from the data rather than hardcoded, so
rebuilding the coastline with a different clip updates it for free:

// src/coastline.js — after
export function isWithinCoverage(lat, lon) {
  const b = coverageBounds();   // walked off the bundled polygons, memoized
  return lat >= b.minLat && lat <= b.maxLat && lon >= b.minLon && lon <= b.maxLon;
}

Then the second-order fix, which matters more: a registry station outside
coverage became a validation failure, not a note — and the coastline build
script now derives its clip box from the registry's own extent plus a margin,
floored at the proven box so it only ever grows outward. The clip can no longer
fall behind the data it's meant to check
(7d52f67,
1c06273).

Rule: a predicate over partial data needs three states, not two. If
"no" and "I have no data here" collapse to the same value, every caller that
reads false as a verdict is reporting a result it never computed.

2. A validator that iterates one side of a pair

Same package, next day. Station slugs are public URL segments, so a rename is a
breaking change. A lock file pins the slug each station last shipped with, and
check-slugs runs in CI ahead of the data checks:

// src/slugs-lock.js — before
for (const [id, lockedSlug] of Object.entries(lock.slugs)) {
  const nowSlug = current.get(id);
  if (nowSlug === undefined || nowSlug === lockedSlug) continue;
  // ...fail if the slug moved without the old value in formerSlugs
}

Read the loop bound. It iterates the lock. A station present in the data but
absent from the lock is never visited — so a brand-new station's slug entered
the public API with nothing pinned to compare against, and that slug plus every
future change to it passed green forever. The one thing the lock exists to
prevent was invisible for exactly the stations most likely to get renamed: the
new ones.

The guard is now symmetric — the lock must reflect the current knowable slugs
exactly, in both directions:

// src/slugs-lock.js — after
for (const [id, lockedSlug] of Object.entries(lock.slugs)) {
  const nowSlug = current.get(id);
  if (nowSlug === undefined) {
    problems.push(`${id}: in the slug lock but no longer in the data — its slug "${lockedSlug}" is dead`);
    continue;
  }
  // ...moved-slug check unchanged
}

for (const [id, slug] of current) {
  if (!(id in lock.slugs)) {
    problems.push(`${id}: slug "${slug}" is not in the lock — run \`station-corrections slugs\``);
  }
}

The commit message named the pattern out loud, because it landed a day after the
coastline one: "Same failure class as the coverage bug: a clean result for
something never checked"
(99afa20).

Rule: a validator comparing two collections must iterate their union.
Iterating one side can only ever find disagreements about things both sides
already know about — which is the easy half of the problem.

3. A benchmark that scored the call, not the result

I run a benchmark over the boat agent's MCP tool routing: a set of golden asks,
each with the tools a correct answer requires. Scoring:

# poseidon/bench/scoring.py
def score_ask(ask, observed_tools, observed_args=None):
    if not set(ask.expected_tools).issubset(set(observed_tools)):
        return False
    # ...optional argument matching
    return True

Set containment on tool names. It measures routing — did the model reach for
battery_state when asked about the batteries — and routing is genuinely what
the benchmark was built to compare across models. But nothing in that function
ever opens the payload.

Meanwhile, in signalk-mcp:

async def battery_state(client: SignalKClient, bank: str = "0") -> dict:
    raw = await client.get_value(f"electrical.batteries.{bank}")

"0" is the SignalK instance convention, and it's a perfectly reasonable
default — except the vessel publishes electrical.batteries.house. So the bare
tool returned this, on a boat that had been publishing battery data the entire
time:

{"bank": "0", "soc_fraction": null, "voltage": null, "current": null, "display": null}

Right tool, right arguments, no data. Score: match. The agent only got a real
reading if the person asking happened to say the word "house". This survived
months of green benchmark runs and was found by hand, mid-benchmark, reading a
transcript for an unrelated reason
(e837d92).

The tool fix is small — try "0", then discover the vessel's own banks off the
electrical.batteries subtree, and never second-guess an explicitly named bank
because answering about a different battery is worse than answering "no data".
The interesting fix is the scoring one: at least one assertion per ask has to
reach into the returned payload.

Rule: a score computed from the call examines your harness's routing. A
score computed from the result examines the system. If nothing in your
scorer opens the response body, a tool that returns nulls forever is a
perfect performer.

4. A selftest that greps for a string the command never prints

A CLI wrapper on the boat gateway sources its secrets and execs the real
binary. Scripts parse its output, so a missing environment variable has to fail
loudly and early — hence a --selftest that proves the secrets resolved on a
real invocation.

The trap: the CLI reports the same condition in two different formats depending
on the subcommand.

$ mytool audit
┌ Config warnings ──────────────────────────────┐
│ ! api.token: Missing env var API_TOKEN        │
└───────────────────────────────────────────────┘

$ mytool config validate
1 warning(s): ! api.token: Missing env var API_TOKEN

The selftest called config validate and grepped for the box title:

# before — passed while the token was genuinely missing
out="$(mytool config validate 2>/dev/null || true)"
if grep -qi "Config warnings" <<<"$out"; then
  echo "SELFTEST FAIL"; exit 1
fi
echo "SELFTEST PASS"

config validate never prints Config warnings. The grep could not match. The
selftest passed, cleanly and instantly, with a required token genuinely absent
— a check whose entire job was catching that exact state.

# after — match the thing both formats actually contain
if grep -qi "missing env var" <<<"$out"; then

What caught it wasn't review and wasn't a test. It was running the negative
control: deliberately unsetting the variable and watching whether the selftest
went red. It didn't.

Rule: you have not tested a check until you have watched it fail. Break
the thing on purpose, confirm red, put it back, confirm green. A selftest
that passes when broken is worse than no selftest — it converts an unknown
into a false known.

5. An assertion that fired before the system finished failing

slackwater-web is an
offline-first tide PWA, so its Puppeteer smoke suite loads the map with the
network cut and asserts no unexpected console errors. It looked like this:

await mapPage.waitForSelector(".map-canvas .maplibregl-canvas", { timeout: 10_000 });
assert.deepEqual(mapErrors, []);   // green

MapLibre puts a canvas in the DOM as soon as it initialises the GL context —
well before it has finished trying, and failing, to fetch its tiles and glyphs
over a network that isn't there. The assertion fired into an empty array
roughly two seconds early. Green, for the wrong reason.

The fix separates collection from judgement — a watcher that starts before the
navigation and a settle() that drains every pending handle before anything is
asserted:

const settleMapErrors = watchErrors(mapPage);
await mapPage.goto(`${URL}/map`);
await mapPage.waitForSelector(".map-canvas .maplibregl-canvas", { timeout: 10_000 });
// The canvas appears well before the map has finished failing its external
// fetches, so asserting the instant it exists passed for the wrong reason.
const mapErrors = await settleMapErrors();
assert.deepEqual(mapErrors, []);

(The same commit fixed why those errors were unreadable in the first place:
Puppeteer's msg.text() renders an object argument as its class name, so
MapLibre's minified AJAXError arrived as [object Ae] with the host it named
invisible to the noise classifier. Resolving each argument through its
JSHandle makes collection async, which is what forces the settle-then-judge
shape anyway —
32bfe09.)

Rule: an assertion is a claim about an instant. If the system is still
working when you sample, you measured a different system. Collect over a
window, settle, then judge — and be suspicious of any wait keyed on the
first sign of life rather than the last.

6. When deleting the fix leaves every check green

Same app, same evening. resolvePinStates called with no dependency object let
every unsynced station fall through to a live upstream fetch — a request storm
on every map open, throttled behind a ~24 req/min limiter, with a single
setData gated on the slowest of them. In review the pins effectively never
coloured. The fix is one argument: a rejecting fetchFn that makes state reads
cache-only, so an unsynced station stays neutral, which is the honest unknown
(afa3df1).

Now: which check catches its removal?

tsc — green. The argument is optional.
The Vitest suite — green. jsdom has no WebGL, so the map component never mounts and no test observes the call.
The production build — green.
The Puppeteer smoke test — green. A request storm against a third-party API is not a console error.

Delete the fix and all four stay green while the storm comes back. The bug is
invisible to every conventional layer, which means writing a conventional test
would be theatre. So the guard reads the source file as text:

it("resolves pin states cache-only — never fetching", () => {
  // Guarded by source text because nothing else can catch it: jsdom has no
  // WebGL so the map never mounts, and dropping the fetchFn leaves tsc, the
  // suite, the build and the smoke test all green while every map open fires
  // one live request per unsynced station at a third-party API.
  const src = readFileSync(join(__dirname, "MapScreen.tsx"), "utf8");
  const call = src.match(/resolvePinStates\([\s\S]*?\)\s*\n\s*\.then/);
  expect(call, "resolvePinStates call site not found").toBeTruthy();
  expect(call![0]).toContain("fetchFn");
});

That test is ugly. It's coupled to formatting, it can't see a rename, and it
proves nothing about behaviour. It is also the only thing in the repo that goes
red when the fix is removed
(88ec107).

Rule: if you delete the fix and nothing goes red, the missing thing is the
check, not the bug. A real guard with a known ceiling beats an elegant guard
that doesn't exist — as long as the test says, in the test, why it's shaped
that way.

What they have in common

None of these were flaky. None were wrong answers. Every one of them was a
scope bug in the check itself: the set of things examined was smaller than
the set of things claimed, and nothing in the output distinguished the two.

That's the whole thesis. A green check is a claim about what ran. Reading it as
a claim about what's true requires an extra assumption — that the check's
subject and the check's coverage are the same set — and that assumption is
exactly what nobody verifies, because verifying it feels like testing the tests.

Two structural tells show up in five of the six:

The check consumed a partial data source and had no way to say so. The clipped coastline, the lock file, the jsdom environment, the two-second window. Partial coverage plus a boolean result is a machine for manufacturing false confidence.
Nobody had ever watched it fail. The negative control caught #4 within a minute. It would have caught #1, #2 and #3 just as fast.

The checklist

Six questions. They take a minute, and each one is a bug I actually shipped:

Does this validator iterate both sides of the pair? Or does it loop over one collection and silently exempt everything the other one knows about?
Can this predicate return "no" and "I don't know" as the same value? If yes, every caller reading it as a verdict is reporting a result nobody computed.
Does the score examine the result, or just the call? Set containment on tool names, exit codes, "did it get called" — all of these pass on an empty payload.
Did you run the negative control? Break the thing, watch the check go red, put it back. If you've never seen it fail, you don't know it can.
Does the assertion fire after the system has finished, or at the first sign of life? Waiting on the first artifact to appear is not the same as waiting for the work to finish.
If you deleted the fix, which check would go red? If the honest answer is "none", write that check first — even if the only shape available is an ugly one.

None of this is exotic. It's the difference between a test suite that tells you
the code works and one that tells you the suite ran.

I'm building the software stack for an all-electric charter catamaran — data
packages, SignalK plugins, MCP servers, an offline tide app — mostly in public.
Code for the bugs above lives in
station-corrections,
signalk-mcp and
slackwater-web.

Related: Porting a tide engine to Swift with the original as the test oracle — what a check looks like when it does examine the result · Monitor the delivery path, not just the alarm — the same silence, one layer out · Bench your own workload before you switch LLM vendors — the benchmark from vignette 3, before its scoring blind spot showed up

A current station's coordinate is a label, not the hydraulic control

Bryan Clark — Mon, 31 Aug 2026 14:11:48 +0000

TL;DR — A published tide or current station coordinate is an identifier for the place, not a measurement of where the physics happens. At Dodd Narrows the CHS current station sits 120 m from the pass's actual throat, on water twice the cross-sectional area. Any model that scales speed by continuity (u ∝ 1/A) divides by the wrong area there and over-predicts the throat by 91%. Fix it once, in shared reference data, with a source: field recording where the real coordinate came from. Jump to the fix.

Problem

We build offline tidal-current prediction for the Salish Sea. Harmonic prediction at a station is a solved problem — you fit constituents and evaluate them. Drawing a current field, a speed and direction at arbitrary points near a pass, is not.

The obvious source is a coarse ocean model. It is useless at rapids. Here is the coastal-ocean mesh against the fitted station truth at three BC/WA gates:

gate               mesh cell    truth      mesh reads   ratio
Dodd Narrows       516 m        6.2 kn     1.46 kn      0.24

A 516 m mesh cell cannot see an 80 m rock gut. It smears a 6-knot rapid into a 1.5-knot drift, and animating that would make bad data more persuasive than no data.

So instead: grow a small patch of certified geometry around each gate. Take a 10 m bathymetric DEM, trace a thalweg through the pass, cut cross-sections every 50–200 m, and scale the station's own fitted speed along the reach by mass conservation:

|u|(x) = |u|_anchor · A_anchor / A(x)

One anchor station, one reach, one law. It certified cleanly at Tacoma Narrows against held-out NOAA truth in both directions (speed median 0.32 kn and 0.20 kn against a 0.5 kn bar, no flood/ebb sign flips). It certified at Seymour Narrows. At Dodd Narrows it fell over, and the failure did not look like a data-quality problem:

$ ./sensitivity.py dodd-narrows
anchor area swung 19.9% under variant 'shift+0.5' — anchor placement is sensitivity-unstable

That number is not prediction error. It is anchor stability: perturb the section grid by half a spacing, or recompute areas at chart datum instead of mean water level, and see how much the anchor section's own cross-sectional area moves. Every scale in the patch is A_anchor / A(x), so a wobbly A_anchor multiplies straight into every cell. Over 10% and the pass ships nothing.

Dodd came in at 19.91 / 20.52 / 19.67% across the three variants. All three. Not marginal — double the bar, on every axis.

Diagnosis

The tell was in a field the pipeline prints and nobody reads:

$ ./sections.py dodd-narrows
  kept_range [18, 18] of [0, 41] — throat 80.0 m (strip minimum (anchor is not the throat)),
  flare 1.3 → limit 104.0 m

anchor is not the throat. That string exists because the code has to decide what "the throat" is before it can decide where the patch ends:

FLARE = 1.3        # first section wider than FLARE x throat ends the patch
THROAT_TOL = 1.1   # anchor may exceed the strip minimum by this and still be "the throat"

min_w = min(s["width_m"] for s in sections)
throat = sections[k]["width_m"]          # k = section nearest the anchor station
throat_ref = "anchor section"
if throat > min_w * THROAT_TOL:
    throat, throat_ref = min_w, "strip minimum (anchor is not the throat)"

Normally the anchor is the throat: you anchor at the rapid, the rapid is the narrowest water, scale is 1.0 there by construction. Seymour Narrows behaves exactly that way — anchor section, 770 m, done.

Dodd does not. The section nearest the published coordinate is 160 m wide. The narrowest section in the strip is 80 m, two sections away.

Two things follow, and they are different bugs.

1. The published rate is a throat number, so anchoring off the throat over-predicts. The reach-mean cross-sectional areas are 2322 m² at the published coordinate and 1216 m² at the gut. CHS publishes ~9.43 kn as Dodd's spring maximum — that is the fastest water in the pass, which is the water at the control. Feed it in at a section with 1.9× the area and continuity dutifully scales it up:

9.43 kn × 2322 / 1216 = 18.01 kn at the gut

Eighteen knots. Dodd is a nine-knot rapid. The model is not wrong; the anchor is in the wrong place, so the model is answering a question nobody asked.

2. The published coordinate sits on the flare, where geometry is unstable. Section 18 is on the shoulder where the channel opens out. Nudge the section grid 25 m and the area at that station moves a lot; do the same at the gut and it barely moves. That is where the 19.9% comes from.

Now the part that took the longest to accept: this is not a data error. Query the station metadata directly and it is internally consistent:

$ curl -s "https://api-iwls.dfo-mpo.gc.ca/api/v1/stations?code=..." | jq '.[0] | {officialName, latitude, longitude}'
{
  "officialName": "Dodd Narrows",
  "latitude": 49.134351,
  "longitude": -123.817132
}

Six decimals. That coordinate is not sloppy — it is precise about the wrong thing. It exists so a mariner can find "Dodd Narrows" in a list and a chart plotter can drop a pin near the pass. It was never a claim about which cross-section the predictions describe. The same is true across the region, and the giveaways are visible:

Seymour Narrows   50.133333 / -125.35      ← lands on its control section (by luck)
Dodd Narrows      49.134351 / -123.817132  ← 120 m SSE of an 80 m gut
Porlier Pass      49.015    / -123.585     ← three decimals, exactly round

Porlier's coordinate is 49.015 / -123.585. Three decimals, both values landing on a round thousandth. Nobody surveyed that. It is a label.

What we tried (and why it failed)

Blamed the section spacing

Dodd's gut is only ~200 m long, so we cut sections at 50 m — finer than the pipeline's usual 100–200 m band. Reasonable suspicion: 50 m is under-sampled noise. So we re-ran at 100 m.

spacing   anchor area swing (shift+½ / shift−½ / datum=CD)
50 m      19.91 / 20.52 / 19.67 %
100 m     27.62 / 33.64 / 20.33 %   ← worse

Worse on every axis, and the strip's narrowest section reads 100 m instead of 80 m — the coarser grid stops resolving the gut at all. Spacing was not the cause. Recorded in the pass inputs so it never gets re-suspected.

Moved the anchor to the DEM throat, expecting that to be the fix

It is the obvious next move, so we ran it as a diagnostic before touching any canonical data:

anchor placed at section 20 (the 80 m gut)
  shift +½ spacing    8.60 %   ✓ inside the 10 % bar
  shift −½ spacing    9.58 %   ✓
  datum = chart datum 22.67 %  ✕ worse than before the move

Two axes fixed. The third got worse. The datum=CD variant recomputes every area at chart datum instead of mean water level — at Dodd that removes 3.08 m of water from the section. The gut carries about 12 m over an 80 m width, so 3.08 m is a quarter of its depth: Dodd's cross-sectional area is a strong function of tide stage in a way Seymour's 90–150 m depths simply are not (datum=CD moves Seymour's anchor area by 4.56%).

This is the finding that reframed the whole thing. The position error and the datum sensitivity are two separate problems that happened to sum to the same failing number. Moving the anchor was necessary and not sufficient, and if we had only ever looked at the aggregate "19.9% vs 10%" we would have concluded the fix didn't work and reverted it.

Made the area tide-stage dependent

If area is a function of stage, stop treating stage as noise and model it: compute each section's area at seven chart-datum heights (0…6 m), evaluate the paired tide station's fitted model at runtime, interpolate the scale. The interpolation itself is fine — held-out half-metre heights come back within 0.9164%, and with the datum axis modelled rather than swept, the corrected anchor clears placement sensitivity.

And Dodd still ships nothing. With a stable anchor, the surviving certified range collapses to [20, 20] — a single transect, no neighbour to triangulate an interval from, zero cells packed.

Tried the other bathymetry source

The fallback grid, a CHS NONNA-10 tile, resolves the gut at 50 m. Same ending:

1.3× flare rule          → one transect
1.6× diagnostic widening → adjacent 80 m sections fail the unchanged 10 % bar by 12.6–32.9 %

No bar was relaxed and no singleton transect was inflated into an area. The ruling instead is a station-local indicator centred on Dodd's corrected coordinate, drawing no fill, outline or boundary — nothing that suggests a certified spatial extent.

The fix

The whole shipped change is one coordinate and one provenance string, in the registry that owns station identity for every consumer in the workspace:

 chs-dodd-narrows:
   name: Dodd Narrows
   context: Nanaimo
-  position: [49.1344, -123.8171]
+  position: [49.13546639419797, -123.81735084108287]
   provider: chs
   cities: [Nanaimo]
+  source: GSC West Coast Topo-Bathymetric DEM v2 hydraulic control section

Plus the test that pins it, because a coordinate with no test is a coordinate somebody rounds:

const resolved = resolve({ id: "chs-dodd-narrows" });
assert.equal(resolved.latitude, 49.13546639419797);
assert.equal(resolved.longitude, -123.81735084108287);

const dodd = loadRegistry(yaml).get("chs-dodd-narrows");
assert.equal(dodd.source, "GSC West Coast Topo-Bathymetric DEM v2 hydraulic control section");

Three details that are the actual work:

The coordinate was recovered, not inferred. The pipeline had already recorded "~130 m off the control", and it would have been easy to project 130 m along the thalweg bearing and call it done. Instead: regenerate the sections from the DEM and read the generated centre of the minimum-width section straight out.

$ ./sections.py dodd-narrows
$ jq '.sections | to_entries | min_by(.value.width_m) | {index: .key, center: .value.center, width_m: .value.width_m}' \
    passes/dodd-narrows.json

The offset between the two shipped positions comes out at 120.0 m on a 351° bearing — close to the recorded ~130 m, but not equal to it, because the recorded figure was section-centre to section-centre and the label is not on the centreline. Derive the number you are shipping from the artifact that produced it.

Full precision, no rounding. 49.13546639419797 is not false precision — it is the generated section centre, and rounding it re-introduces exactly the ambiguity the fix removes. The old value, 49.1344, was the published coordinate rounded to four decimals. Rounded labels are how a coordinate stops being traceable.

source: is the load-bearing field. The registry's default provenance is documented once, repo-wide: names are hand-written, positions come from the fitting pipeline. Dodd's no longer does — it comes from a specific DEM version and a specific derivation. Without that string, in six months this is an unexplained coordinate that disagrees with the government's, and the safest-looking action is to "fix" it back.

Dataset: Canada west coast topo-bathymetric digital elevation model, Geological Survey of Canada Open File 8963, 10 m grid, OGL – Canada.
Station predictions: CHS tidal current predictions via the IWLS API.

Why it matters, and the traps next door

Published reference coordinates are identifiers. Every published station, gauge, buoy and sensor coordinate you consume was chosen to name and find a thing. Your model may need to know where a physical process happens. Those are different requirements, and the data does not tell you which one it is satisfying. The 120 m that is a rounding error on a chart is a factor-of-two error in a cross-sectional area.

A rule that works at one station and not the next is not a rule you have. Seymour's coordinate lands on its control section, so Seymour certified and nobody learned anything. Dodd and Porlier both do not. If your first integration succeeds, you have one sample.

Detect it, don't eyeball it. The single most useful line in that pipeline is the THROAT_TOL branch — "the section nearest the anchor is not the narrowest in this strip." That is a cheap, mechanical assertion any geometry-anchored model can carry, and it turns a subtle physical misplacement into a printed string.

Watch the walk that keeps the anchor by construction. Dodd's flare limit is 1.3 × 80 m = 104 m, and the surviving kept_range is [18, 18] — the 160 m anchor section, which itself violates the limit. The bounds walk starts at the anchor and only tests neighbours, so a single-element range is never checked against its own rule. If you write an outward walk from a seed, decide deliberately whether the seed is exempt.

Fix it in the shared registry, not the consumer. The tempting patch is a local override in the pipeline that needed it. Then the app has a different Dodd from the agent tool surface, which has a different Dodd from the harmonic fitter, and the reason lives in a commit message in one repo.

Be honest about what "shared" buys, though. It is not instant propagation — one reader takes it on a version bump, one needs its build-time generator re-run, and one is Python with no npm at all and vendors a copy of the JSON. That last one keeps a drift test whose entire job is to fail when the copy goes stale:

@pytest.mark.skipif(not REAL.is_file(), reason="sibling station-corrections not present")
def test_bundle_matches_published_registry():
    assert json.loads(BUNDLE.read_text()) == json.loads(REAL.read_text()), (
        "vendored _registry.json drifted from station-corrections/data/registry.json"
    )

What the registry actually buys is that the derivation and the review happen once. Nobody re-derives the coordinate, nobody re-argues whether it is right, and nobody ships a second, quieter answer.

Don't let a consumer's need bend the registry's membership rule. The plan for this change also wanted to pair Dodd with Nanaimo Harbour as its tide reference, because the stage-aware model needed a water-level source. Nanaimo Harbour is not a CHS-designated reference port, and that designation is the registry's entire admission rule for tide stations — an external rule, chosen precisely so a hand-curated list can't quietly grow into a mirror of the whole provider station table. So the pairing did not land. Nanaimo stayed a runtime input to the pipeline that needed it, and the registry kept its rule. A position correction is the registry's job; a private modelling dependency is not.

The last one is the uncomfortable one. The model this coordinate was recovered for does not ship. Dodd draws no current field today, from either bathymetry source. The coordinate correction shipped anyway, and it was worth doing on its own: the pass's identity is right for every reader, anything drawn at Dodd is now drawn on the real constriction, and the next attempt at a field model starts from a good anchor and a written-down reason. Reference-data corrections outlive the pipelines that discover them. That is a good argument for landing them separately, in their own repo, with their own tests — not as a commit inside the feature that found the bug.

Close

This came out of building offline current prediction for a boat that will spend a lot of time waiting on tidal gates in the Salish Sea. The station registry is public and MIT: @sailingnaturali/station-corrections — names, contexts and positions for tide and current stations in both countries, shipping no provider-minted station id so it stays redistributable. The Dodd change is PR #15, released in v2.9.2.

A localeCompare in a build script minted different public URLs per machine

Bryan Clark — Sun, 30 Aug 2026 18:40:18 +0000

TL;DR — String.prototype.localeCompare with no locale argument uses
whatever collation the runtime happens to have. If a sort ordered by it
decides an identifier — a slug, a first-wins registry key, an index — that
identifier is pinned to the machine, not to the repo. Sort by codepoint
(a < b ? -1 : a > b ? 1 : 0) with an id tiebreak for anything a URL is
built from, and keep localeCompare for what a human reads.
Jump to the fix.

Slackwater is an
offline-first tide and current app for the Salish Sea. It bundles 133 NOAA
harmonic current stations, and each one gets a human-readable URL —
/deception-pass, /the-narrows, /hale-passage. Those URLs get shared. They
are the closest thing a static PWA has to a primary key.

NOAA gives a lot of those stations the same name. Of the 133 bundled stations,
40 share a landmark name with at least one sibling — 17 names in all —
distinguished only by a qualifier NOAA tacks on after a comma:

$ node -e 'JSON.parse(require("fs").readFileSync("src/data/currents.json"))
    .filter(s => s.name.startsWith("Hale Passage"))
    .forEach(s => console.log(s.id, JSON.stringify(s.name)))'
noaa/PUG1529 "Hale Passage, East end"
noaa/PUG1530 "Hale Passage, West end"
noaa/PUG1710 "Hale Passage, east of Lummi Point"

Three stations, one slug: hale-passage. So there's a deterministic ladder that
hands out the collisions — plain slug, then -current, then the qualifier
folded back in, then the NOAA station id:

export function assignSlug(
  r: { slug: string; name: string; context: string },
  id: string,
  used: Set<string>,
): string {
  const candidates = [
    r.slug,
    `${r.slug}-current`,
    toSlug(`${r.name} ${r.context}`),
    `${r.slug}-${id.replace("noaa/", "").toLowerCase()}`,
  ];
  const slug = candidates.find((c) => !used.has(c)) ?? candidates[candidates.length - 1];
  used.add(slug);
  return slug;
}

Deterministic — given an order. assignSlug is called in a .map() over the
station array, so the first station the array reaches claims rung 1. Which
station that is depends entirely on how the array got sorted.

Problem

The array gets sorted in a build script, one line, entirely unremarkable:

// scripts/build-currents.mjs
const stations = bundle.stations
  .filter(/* harmonic only, primary bin, in-bounds, non-zero amplitude */)
  .map(/* reshape */)
  .sort((a, b) => a.name.localeCompare(b.name));

writeFileSync("src/data/currents.json", JSON.stringify(stations));

Two facts about that output file turn a sort into a URL problem.

One: it's gitignored.

$ grep -n currents.json .gitignore
5:src/data/currents.json

Two: every developer-facing command rebuilds it.

"scripts": {
  "build:data": "npm run build:stations && npm run build:currents",
  "dev":   "npm run build:data && vite",
  "build": "npm run build:data && tsc && vite build",
  "test":  "npm run build:data && vitest run"
}

So: whichever machine runs dev, build or test regenerates the 186 KB
station file, in that machine's sort order, and the ladder assigns public,
permanent, already-shared URLs off the back of it. There is no tracked artifact
anywhere in the repo recording which station got hale-passage-current last
time.

Diagnosis

localeCompare called with no locale uses the runtime's default collation, and
collation is not codepoint order. The collision that bites this dataset is the
simplest one there is — case.

$ node -e '
  const a = "Hale Passage, West end", b = "Hale Passage, east of Lummi Point";
  console.log("localeCompare:", a.localeCompare(b));  // 1  -> West sorts AFTER east
  console.log("codepoint:    ", a < b);               // true -> West sorts BEFORE east
'
localeCompare: 1
codepoint:     true

Codepoint order puts W (0x57) before e (0x65). Collation folds case at the
primary level and puts east before West. Both are correct sorts. Only one of
them can be the URL.

I re-ran the whole pipeline under both comparators and diffed the assigned
slugs:

$ node scratch/probe.mjs
distinct base names: 110
names with >1 station: 17
stations in a collision group: 40
FLIP noaa/PUG1530 codepoint: hale-passage-current            localeCompare: hale-passage-west-end
FLIP noaa/PUG1710 codepoint: hale-passage-east-of-lummi-point localeCompare: hale-passage-current
FLIP noaa/PUG1546 codepoint: pickering-passage-current        localeCompare: pickering-passage-west-of-squaxin-island
FLIP noaa/PUG1547 codepoint: pickering-passage-off-graham-point localeCompare: pickering-passage-current
FLIP noaa/PUG1526 codepoint: the-narrows-current              localeCompare: the-narrows-north-end-west-side
FLIP noaa/PUG1524 codepoint: the-narrows-north-end-midstream  localeCompare: the-narrows-current
stations whose public slug differs: 6

Six stations, three name groups — Hale Passage, Pickering Passage, The Narrows.
Not a hypothetical: switching the comparator immediately reassigned six live
URLs.

The severity comes from the gitignore, not from the sort. A sort that varies is
a nuisance. A sort that varies, feeds a first-wins identifier, and writes to an
untracked file is a class of bug where the wrong output can never appear in a
diff. Nobody reviews it because there is nothing to review.

What I tried (and why it failed)

Pin the locale

The obvious fix. localeCompare is non-deterministic because the locale is
implicit, so make it explicit:

.sort((a, b) => a.name.localeCompare(b.name, "en-US"));

Identical output. So I widened it — every ladder outcome, under fifteen locales
and collation options:

$ node scratch/locales.mjs
distinct slug outcomes across locales: 1
  group 1 en-US, en-GB, de-DE, fr-FR, sv-SE, tr-TR, cs-CZ, da-DK, ja-JP,
          zh-CN, pl-PL, es-ES, en-US-u-kf-upper, en-US-u-ka-shifted, en-US-u-kn-true
codepoint vs any-locale differing slugs: 6

Every locale I could name agrees with every other locale, and all fifteen
disagree with codepoint order. Pinning the locale doesn't touch the axis that
actually flips this. The en-US-u-kf-upper case-first keyword doesn't do it
either — case-first reorders A against a for the same letter, not W
against e.

That's worth sitting with, because it's the opposite of the reassuring result.
The variation isn't "someone in Germany gets different URLs." It's:

ICU presence. Node's own docs mark String.prototype.localeCompare as "partial (not locale-aware)" under --with-intl=none, meaning it "carries out its operation just like the non-Locale version of the function." A Node without full ICU produces the codepoint order in the figure above — literally the other column. Node ships full-icu by default now, but self-built, distro-packaged and size-trimmed container images all still exist.
ICU and CLDR version. Collation tables ship with ICU, and ICU tracks CLDR. This machine is node v24.1.0 / icu 77.1 / cldr 47.0. A different Node major is a different table. Punctuation and symbol weights are exactly the part of the table that gets revised.

Both of those are properties of the machine. Neither is pinned by anything in
the repo. A comparator whose answer is a property of the machine cannot be
allowed to name a URL — and I couldn't demonstrate the flip by changing LANG,
which is precisely why it would have survived a casual look.

"Just use a stable sort"

Wrong lever, and a common reflex. Array.prototype.sort has been
required to be stable since ES2019;
V8 has been stable since Node 11. Stability guarantees that equal elements keep
their relative order. It says nothing about a comparator that decides
inequalities differently on a different machine. A perfectly stable sort with a
drifting comparator drifts.

Stability is worth a thought here, though, for the opposite reason: two
stations could genuinely have equal names, at which point stability preserves
whatever order the vendored extract happened to be in — which is also not
something the repo pins. Hence the id tiebreak in the fix.

Commit the generated file

Tempting: check currents.json in, and a reordered build shows up as a diff.
Real, but it's the wrong shape. It makes a wrong build reviewable, not
impossible — someone still has to notice 186 KB of reordered JSON in a PR and
work out that line 4,012 moving means a URL changed. And the file re-churns on
every NOAA re-vendor, so the signal drowns immediately. Fix the comparator; use
a test for the noticing.

The unit test that was already green

The ladder had tests. They were added in the same session, one commit earlier,
and they passed under both orderings:

it("pins the ladder order: exact slug, then -current, then the qualifier folded in", () => {
  const used = new Set<string>();
  const a = assignSlug({ slug: "alki-point", name: "Alki Point", context: "1 mile West of" }, "noaa/PUG1502", used);
  const b = assignSlug({ slug: "alki-point", name: "Alki Point", context: "West of" }, "noaa/PUG1516", used);
  const c = assignSlug({ slug: "alki-point", name: "Alki Point", context: "near the ferry dock" }, "noaa/PUG1599", used);
  expect(a).toBe("alki-point");
  expect(b).toBe("alki-point-current");
  expect(c).toBe(toSlug("Alki Point near the ferry dock"));
});

Read what it actually asserts. It constructs three station objects in an array
literal, in the order it wants, and hands them to assignSlug one at a time.
It proves the ladder walks its rungs correctly. It cannot fail because of a
re-sort, because it never touches the sort — the third argument is even a
freshly-constructed Set, so it doesn't touch the real used set either. A
green test here is a claim about the ladder, not about the URLs the app ships.

The fix

Two parts. Sort by codepoint, with the station id as the tiebreak:

// scripts/build-currents.mjs
// Codepoint compare, not localeCompare: this file is gitignored and rebuilt on
// whatever machine runs dev/build/test, and localeCompare with no locale pinned
// uses the runtime's default collation — which can mint different public slugs
// for the colliding station names on different machines. The id tiebreak keeps
// ties (equal names) deterministic too.
.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : a.id < b.id ? -1 : 1));

And pin the real pipeline's output — not a constructed case — for one known
collision:

it("pins the real Alki Point ladder outcome (catches a re-sort silently reassigning public slugs)", () => {
  // Pinned to actual output from the build pipeline, not constructed — a
  // locale-dependent or otherwise reordered sort in build-currents.mjs would
  // flip which one gets the bare slug without any other test catching it
  // (currents.json is gitignored, so there's no tracked diff to notice either).
  const oneMileWest = resolvedNoaaCurrentStations.find((s) => s.id === "noaa/PUG1502")!;
  const west = resolvedNoaaCurrentStations.find((s) => s.id === "noaa/PUG1516")!;
  expect(oneMileWest.slug).toBe("alki-point");
  expect(west.slug).toBe("alki-point-current");
});

resolvedNoaaCurrentStations is the module the app itself imports, built from
the generated JSON. npm test runs build:data first, so this assertion runs
against a freshly generated bundle on every machine and in CI. That's the whole
point: it's the only artifact in the repo that says which station owns which
URL, and it's an assertion rather than a 186 KB blob.

Why it matters / gotchas

Split the two jobs of string comparison. Collation answers "what order
should a human read these in." Codepoint comparison answers "which of these two
strings is canonically first." Sorting a station list for display should stay
localeCompare — an app used in Québec should file Île sensibly. Anything a
URL, cache key, content hash, or first-wins registry is built from must be
codepoint, because the answer has to be a property of the data and not of the
machine. Intl.Collator is not an escape hatch; it's the same ICU tables with a
nicer API.

The generalization is bigger than sorting. The dangerous pattern is
ordering that becomes an identifier in an untracked artifact. Slugs are one
instance. So are array-index ids in generated code, "first registration wins"
plugin keys, and anything downstream of Object.keys(), fs.readdir(), or a
glob. Ask of any generated file: if its order changed, would anything permanent
change with it — and would a diff show me?

Ship the redirect, but don't count it as the fix. Slackwater resolves a URL
segment in three passes: current slug, then a recorded former slug, then the
provider id, so an old shared link still lands on the right station. That's a
seatbelt. It only helps for a slug someone noticed had changed and wrote down.
The bug here is specifically the class where nobody notices.

Grep your build scripts, not just your app code. localeCompare in a
component that renders a list is fine. localeCompare in something that writes
a file is a question worth answering every time:

$ rg 'localeCompare|Intl\.Collator' scripts/

I run this stack on a boat — an offline-first PWA for tides and currents in the
Salish Sea, part of the software for an all-electric charter catamaran. The
Slackwater web app is public at
sailingnaturali/slackwater-web.

Related: two false walls in NOAA's currents API covers where these station records come from, and porting a tide engine to Swift with the original as the test oracle is the same test suite from the other end — pinning real output instead of eyeballing it.

plugins.allow is exclusive: the warning's suggested fix disabled 44 plugins

Bryan Clark — Sat, 29 Aug 2026 02:48:10 +0000

A config warning told me, every single turn, that plugins.allow was empty — and helpfully printed the one-line fix. I pasted it onto the live box. Enabled plugins went 47 → 3, because plugins.allow is an exclusive allowlist over all plugins, including the ~46 that ship bundled. The two that mattered were the fallback model provider and bundle-mcp, the plugin every MCP tool hangs off. Reverted in a minute. The durable fix wasn't code — it was a comment in the config recording why the warning stays unfixed. Jump to the fix.

The setup: OpenClaw running as an agent gateway on a Raspberry Pi next to a SignalK server, so I can DM a boat and ask it how much water is under the keel. I wrote up that build already. This post is about one line of its log output.

The problem: a warning with a fix attached

Every turn, the gateway logged this:

plugins.allow is empty; discovered non-bundled plugins may auto-load: deepseek

That string is documented behaviour: "When plugins.allow is unset and non-bundled plugins are auto-discovered from the workspace or global plugin roots, startup logs plugins.allow is empty; discovered non-bundled plugins may auto-load: ...".

It's a reasonable warning. Without an allowlist, a plugin that appears in a discovery root gets loaded without anyone approving it. And the fix looks trivial — there is exactly one non-bundled plugin on the box, deepseek, the provider for the primary model. So: allow it, and nothing else can sneak in.

{
  "plugins": {
    "allow": ["deepseek"]
  }
}

One key. One value. The thing the warning is asking for. I put it on the live gateway.

What it actually did

$ openclaw plugins list --enabled | wc -l
3

Three. It had been 47, out of 68 discovered.

The agent still answered on Telegram, cheerfully, in a way that made it look fine. Ask it for the depth and it no longer had a depth tool.

Diagnosis: "allow" is exclusive over everything

The mental model I had — and I suspect it's the common one — is that an allowlist governs the untrusted things: the plugins that show up from outside, the ones the warning is literally about. Bundled plugins ship with the product; you don't allowlist your own dependencies.

That is not what the key means. From the plugin docs:

plugins.allow is an exclusive allowlist. Plugin-owned tools outside the allowlist stay unavailable even when tools.allow includes "*".

"Exclusive" is the whole story, and it's exclusive over the entire inventory. The docs say so in the places where it bites a specific feature rather than in the place where you'd go looking. On ACP agents:

If plugins.allow is set, it is a restrictive plugin inventory and must include acpx, or the installed ACP backend is intentionally blocked

And in the Codex setup notes: enable the bundled codex plugin first, and include codex in plugins.allow if your config uses a restrictive allowlist. Both of those are bundled plugins. Both need to be in the list.

So "allow": ["deepseek"] doesn't mean "of the discovered plugins, permit deepseek." It means "the set of plugins on this system is: deepseek." Everything else is out, including 46 things I never chose to install and never thought of as plugins at all — because they arrived as part of the product.

The warning wasn't wrong. The suggestion was a correct fix for a narrower config than mine.

The asymmetry that makes it survivable — and confusing

Three plugins survived, not one. deepseek, because I allowed it. And then telegram and memory-core, which I hadn't.

They survived because of an escape hatch the docs call auto-activation:

Bundled opt-in plugins can auto-activate when config names one of their owned surfaces, such as a provider/model ref, channel config, CLI backend, or agent harness runtime.

My config has a channels.telegram block, so the telegram plugin came back on its own. The browser plugin docs state the same rule explicitly and confirm it beats the allowlist: an explicit root browser block "activates the bundled browser plugin even under a restrictive plugins.allow".

Now read that list of owned surfaces again — provider refs, channel config, CLI backends, agent harness runtimes. MCP servers are not on it.

My config names both of these:

{
  "channels": {
    "telegram": { "enabled": true }
  },
  "mcp": {
    "servers": {
      "signalk": { "command": "uvx", "args": ["signalk-mcp"] }
    }
  }
}

Naming the channel rescued its plugin from the allowlist. Naming the MCP server did not rescue bundle-mcp — the bundled plugin that exposes MCP server tools to the model. The MCP servers still launched. They still connected. They still passed their health probes. Their tools were simply not in the request. The MCP configuration page never mentions that a plugin is in that path at all, which is exactly why you don't think to check it.

The two lanes fail differently, and that's the trap: the lane that keeps working is the one you'd notice — messages still arrive, the agent still talks. The lane that goes dark is the one you have to go looking for.

What the two casualties actually cost

bundle-mcp was the visible one. The other was worse and would have been invisible for months:

anthropic-provider — the fallback model. The gateway runs a primary provider with a second one configured behind it precisely so that a provider outage or a lapsed key degrades instead of leaving the boat with no agent. With the allowlist on, the fallback plugin is not loaded. There is no fallback. Nothing tells you this; you find out the next time the primary is down, which is the worst possible moment and also the only moment it matters.
bundle-mcp — every tool from every MCP server. The agent goes blind to the vessel and answers from general knowledge instead, which reads as fluent and is exactly as useful as it sounds.

A resilience feature that silently stops being loaded is worse than never having configured it, because you stop budgeting for the failure it was supposed to absorb.

What I tried, and why each was wrong

1. The suggested one-entry list. Above. 47 → 3.

2. "Fine — I'll list the ones I actually use."

{
  "plugins": {
    "allow": ["deepseek", "anthropic-provider", "bundle-mcp", "telegram", "memory-core"]
  }
}

This is the seductive one, and it's a trap with a delay fuse. It fixes today's breakage and leaves ~42 bundled plugins off. Some of those are load-bearing in ways that don't show up in a smoke test — they're the ones behind a feature you use twice a year. Worse, the list is now a snapshot: the next OpenClaw upgrade that adds or renames a bundled plugin ships it into a gateway that is configured to refuse it, and the failure surfaces as "that feature just doesn't work anymore" with no error and no obvious link to a config file you last touched in August.

3. "Then enumerate all of them." Read openclaw plugins list, paste all ~46 bundled ids plus the one real one, done:

$ openclaw plugins list --json | jq -r '.[].id' | wc -l
68

Correct, and I'd have to re-audit that list on every upgrade forever. That's a standing maintenance tax to close a hole I hadn't yet measured. So I measured it.

4. Actually reading what the warning protects against. The risk is: a plugin lands in a discovery root and auto-loads without approval. Discovery reads configured paths, workspace roots, the global plugin root under ~/.openclaw, and the bundled set. On this box, the only thing that ever writes a non-bundled plugin into the global root is openclaw plugins install — which is already a privileged, deliberate act by someone with a shell on the machine. Someone who can run that can also edit the allowlist. The allowlist is not a boundary against them.

So the hole is real, and on this deployment it is one notch of defence-in-depth behind an act that already implies full control of the host. The cost of closing it is a hand-maintained 47-line inventory re-audited on every upgrade. That trade doesn't clear.

The fix

Don't set the key. Write down why.

// Plugin policy. `plugins.allow` is deliberately ABSENT — do not add it.
//
// Every turn logs:
//   plugins.allow is empty; discovered non-bundled plugins may auto-load: <id>
// and suggests `"plugins": { "allow": ["<id>"] }`. That suggestion breaks
// this gateway. Tried on the live box 2026-08-10: allow is an EXCLUSIVE
// allowlist over ALL plugins, including the ~46 bundled ones. Enabled
// collapsed 47 -> 3. Survivors: the one allowed id, plus telegram and
// memory-core, which auto-activate because config names a surface they own.
//
// What it costs if anyone tries it again:
//   anthropic-provider  the fallback model — a primary outage now leaves
//                       no agent at all, and nothing warns you
//   bundle-mcp          every MCP tool — the agent goes blind to the vessel
//
// Note the asymmetry: naming `channels.telegram` auto-activates telegram,
// but naming `mcp.servers.<name>` does NOT auto-activate bundle-mcp.
//
// The hole the warning points at is real but small: discovery only picks up
// non-bundled plugins from the global root, which nothing writes to except
// `openclaw plugins install` — already a privileged act. Closing it properly
// means listing all ~46 bundled ids and re-auditing them every upgrade.
// Not worth it. Living with the warning is the considered choice.

That comment is the actual deliverable. The code change was deleting three lines.

And the one-command check, before and after any plugin-policy edit:

$ openclaw plugins list --enabled | wc -l
47

If that number moves and you didn't intend it to, stop. It's the cheapest possible canary for a class of change whose failure mode is silence.

Why it matters

An allowlist that includes the batteries is a different feature than one that doesn't. Before you set one, find out whether it governs the things that arrived from outside or the things that arrived in the box. "Exclusive" is doing enormous work in that sentence, and the difference between the two readings is the difference between a two-item list and a 47-item one. The fastest way to find out is to count enabled units before and after on something you can revert in a minute — not to reason about it from the key name.

Suggested fixes in warnings are written for the median config. A warning that prints a ready-to-paste snippet is a genuinely good affordance, and the snippet was right for a gateway whose plugin inventory is small. Mine isn't. A one-line remediation is a hypothesis about your setup, not a patch.

The generalizable one: record why you ignored the linter. Every long-lived system accumulates warnings that are correct in the abstract and wrong to act on here. If the only artifact of "we looked into this and decided to live with it" is a chat log, then the decision has a half-life of about a month — after which someone (possibly you, possibly an agent with config-write tools) sees a helpful suggestion in a log, applies it, and re-discovers the outage from first principles. Suppressing the warning would have been worse: it hides the tradeoff instead of documenting it. A comment at the exact place someone would make the change is the cheapest durable defence there is. Mine is 30 lines guarding a key that isn't there.

There's a marine version of the same lesson: a persistent alarm nobody can explain gets muted, and then the one that mattered gets muted with it — alarm fatigue by design. The software version is quieter and the failure is the same shape.

The bit that should worry you

Nothing errored. The gateway started clean, the agent replied on the channel it always replies on, and the only symptom of losing 44 plugins was an answer that didn't have any vessel data behind it. Two of the three things that survived survived by accident — because their config keys happened to be on the auto-activation list.

I caught it in a minute because I ran the count. If I'd just sent one test message, I'd have shipped a gateway with no model fallback and no vessel access and called the warning fixed.

The gateway watches an all-electric charter catamaran that doesn't exist yet; the MCP server it went blind to is signalk-mcp (MIT).

Related: Running OpenClaw on a Raspberry Pi alongside SignalK — the build this config belongs to, including the tools.profile gotcha that hides MCP tools a second, unrelated way; When MCP tools break, isolate the server before blaming the agent — the debugging order for exactly this symptom; and MCP server or a curl recipe in AGENTS.md? — what those MCP tools were costing and earning in the first place.

DEV Community: Bryan Clark

Read Puppeteer console errors off the argument handles, not msg.text()

The problem

Diagnosis: text() is a description, not the value

Why it was green locally

What I tried first

The fix

The ordering the offline blocks need

The bonus finding: a third origin nobody had written down

Why it matters

Adding a type column to a shared registry breaks every consumer's filter

The problem

Diagnosis

What we tried (and why it failed)

1. Exclude the new kind, in the consumer

2. Exclude derived gates, in the other consumer

3. Denylist the tide ports

4. Two correct filters, in two repos

The fix

Why it matters, and the traps nearby

Close

Four ways a delivery-health alarm silently doesn't work

Problem: the watcher looked right and did nothing

Diagnosis: four places one alarm can stop

Blind spot 1: mqtt.js only calls back on PUBACK

What I tried

Blind spot 2: the loop guard sat one line too early

Blind spot 3: an admin-UI config save latched the alarm forever

Blind spot 4: visual-only, on purpose

The fix

Why it matters / gotchas

Close

A nearest-neighbour matcher turned a timing lag into a fake direction bug

The problem: a label-flip failure that wasn't

Diagnosis: the matcher couples two independent errors

What we tried (and why it failed)

The fix

Why it matters: same-kind matching is the safer test, not the laxer one

The transferable version

A config warning on stdout made a healthy alarm lane look dead

The problem

Diagnosis

What I tried, and why it failed

The fix

Why it matters, and the gotchas around it

MapLibre's SDF icon edge is at 0.75, not 0.5

The problem: a correct icon that draws zero pixels

Diagnosis: read the shader

Why 0.75 is not an arbitrary number

What we tried (and why it failed)

The test that asserted the bug

"Just tighten the spread until something appears"

"Add a constant so the edge reads 192"

The fix

Why it matters, and the traps next door

claude setup-token fixed my launchd auth and dropped my connectors

The problem: launchd, a keychain, and a fix that worked

Diagnosis: the run wasn't failing, it was half-blind

Why: connectors ride on the account session, and the token outranks it

The part that actually cost four days

What I tried, and why each one failed

The fix

Why it matters, and the traps next door

Your service worker's precache can't do Range requests — PMTiles needs 206

The problem: the map draws once, then never again

Diagnosis: precache is 200-shaped, and it only joins the path on visit two

What we tried, and why it failed

Attempt 1 — precache it and raise the size cap

Attempt 2 — a smoke test that agreed with us

Attempt 3 — runtimeCaching with rangeRequests, and an empty cache

The fix

Gotchas

serviceWorker.ready does not mean the service worker controls your page

Test the controlled path, because that's the one users are on

The nice side effect

A passing check is a claim about what ran, not what's true

1. A boolean that returns false for "no" and for "I can't tell"

2. A validator that iterates one side of a pair

3. A benchmark that scored the call, not the result

4. A selftest that greps for a string the command never prints

Diagnosis: `text()` is a description, not the value

`serviceWorker.ready` does not mean the service worker controls your page