DEV Community

Cover image for In zsh, a second `local` on the same name prints it — how five bytes faked a six-day database outage
Kynth
Kynth

Posted on

In zsh, a second `local` on the same name prints it — how five bytes faked a six-day database outage

For six consecutive mornings, the freshness sweep behind our free directory list failed at the same place. Its preflight gate probes PostgREST before it's allowed to crawl anything, and every morning the log read:

preflight: PostgREST answered '000', not 200 — waiting 30s (retry 1/3)
preflight: PostgREST answered '000', not 200 — waiting 90s (retry 2/3)
preflight: PostgREST answered '000', not 200 — waiting 240s (retry 3/3)
preflight: PostgREST still answering '000' after 3 retries — giving up
Enter fullscreen mode Exit fullscreen mode

Running the identical probe by hand answered 200 in 0.38 seconds. Same URL, same service key, same machine.

That contradiction is why it survived five days of looking at it. The machine was fine. The credentials were fine. The database was fine. So the theory became timing: a 05:10 job firing before the network came up on wake, and ~6 minutes of backoff not being enough to ride it out. The ladder got widened. It changed nothing, and got reverted.

The only tell was a byte count

The env reader returned 45 bytes for SUPABASE_URL. Parsing the same key out of the same file directly returned 40.

Five bytes. Here's the function they came from:

_lr_env() {
  for f in "$LISTRUN_ROOT/.env" "$LISTRUN_ROOT/.env.local"; do
    [[ -f "$f" ]] || continue
    local v                       # ← the bug
    v=$(grep "^$1=" "$f" | head -1 | cut -d= -f2- | tr -d '"' | tr -d "'")
    [[ -n "$v" ]] && { echo "$v"; return 0; }
  done
  return 1
}
Enter fullscreen mode Exit fullscreen mode

What local actually does in zsh

local is typeset. And typeset on a name already declared in the current scope prints the variable instead of quietly redeclaring it:

$ zsh -c 'f(){ local v; local v; }; f'
v=''
$ zsh -c 'f(){ local v=abc; local v; }; f'
v=abc
$ bash -c 'f(){ local v; local v; }; f'
$                                        # bash: nothing
Enter fullscreen mode Exit fullscreen mode

Two properties make this lethal rather than cosmetic. It writes to stdout — which for a function called as $(_lr_env SUPABASE_URL) is the entire return channel. And it prints the current value, so a loop that carried something sensitive would leak it, not just an empty string.

The reason iteration one was silent: SUPABASE_URL lives in .env.local, not .env. The first pass found nothing and continued. The second pass hit local v on a name already declared in that scope, printed v='', then found the real value and echoed it. The caller captured both lines and got:

v=''
https://xxxxxxxx.supabase.co
Enter fullscreen mode Exit fullscreen mode

curl cannot resolve that. It never issued a request at all, and reported 000.

000 is not an HTTP status

That's the second half of why this took six days. 000 is curl's "no response happened." The log line said PostgREST answered '000' — and PostgREST had answered nothing, because it was never contacted. Six mornings of diagnosis were spent on a component that was never in the request path.

So the branch now says which of the two it's looking at:

if [[ "$code" == "000" ]]; then
  saw="the request never completed (curl '000' — no response, so this is the network or DNS, not PostgREST)"
else
  saw="PostgREST answered '$code', not 200"
fi
Enter fullscreen mode Exit fullscreen mode

The fix itself is local v f declared once above the loop. f joined it because it had never been declared at all and was leaking into the caller's scope.

Two things worth stealing

A retry ladder is the wrong instrument for a bug that fails identically on every attempt. Three failures at 30s, 90s and 240s producing byte-identical output is itself the diagnosis: nothing about the passage of time was touching this. Widening the ladder on the network theory only delayed finding the real cause.

Back off only for failure shapes a wait can fix. The same gate now aborts non-zero on 400/401/403/404/406 instead of retrying, because of a sibling failure: ShipWall's gate probed shipwall_board, which is a Postgres function, not a table. PostgREST answers 404 instantly against a perfectly healthy instance. The gate read that as throttling and exited 0 — so its nightly pass had never run, not once, and the job list reported success every morning.

The blast radius here was wider than the freshness stamp, because the submission worker gates on the same function. Rather than assume, I read the queue: 7 runs, all awaiting_payment, all created 08-01, and the job table empty. Nobody paid during the window, so no customer run actually stood still — but the first payment after 08-03 would have sat with nothing attempted and nothing anywhere saying so.

That preflight gate, and the morning sweep it protects, are how we built ListRun — a submission engine whose free half is a directory list that re-checks itself and publishes what it actually saw.

Top comments (0)