- Book: System Design Pocket Guide: Fundamentals
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
On October 8, 2026, Google Cloud Storage in us-central1 spent twelve and a half hours returning errors and timeouts on writes. Google's incident report puts the impact from 04:00 to 16:30 US/Pacific, which is 11:00 to 23:30 UTC. Every status update posted during the day carried the same line under Workaround: "None at this time."
Then it came back. The final report adds a second window on October 9, 04:05 to 09:46 Pacific, in the us-central1-b zone, for a total of 18 hours and 11 minutes.
If your service writes user uploads or exports straight to a bucket in that region, that was up to 18 hours in which any write could come back as a 503 or hang until your timeout fired, and the writes that did succeed were slow. Google's load balancer is not yours to fix; what your process does with the bytes when the bucket says no is.
The answer fits in plain Node. Failed writes land in a bounded spool on local disk; a drainer pushes them back with jittered retries once the bucket recovers; a gauge shows how deep the backlog is. It all runs against a fake bucket you can break on command, so you can watch the spool fill and empty.
What the report says, and why your client's retries run out
The report is specific. A "traffic sorting optimization configuration update" to the Cloud Storage load-balancing layer bypassed capacity-aware routing and pushed write traffic onto low-capacity backend nodes, which ran out of resources. A separate metadata database issue added errors on top. Engineers tried to add emergency capacity, but per the report "this was blocked by systemic regional capacity limits." The October 9 recurrence came from an incomplete rollback of an upload planner feature.
What clients saw: "Customers primarily observed HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts." Reads were hit less. Other regions were fine.
The official client's answer to a 503 is on the Cloud Storage retry strategy page, which lists 408, 429 and 5xx responses, plus socket timeouts, as retryable, and recommends exponential backoff with jitter. The Node.js defaults on that page: 3 retries, a 1-second initial wait, a multiplier of 2, a 64-second maximum wait, and a 600-second overall deadline. Uploads are only retried automatically when they are idempotent, meaning you passed an ifGenerationMatch precondition.
So the client gives up after three retries, or ten minutes at the very most. Against a twelve-hour outage, in-request retries are a rounding error. Past that point, the bytes have to live somewhere other than the request, and they have to survive a process restart, because over twelve hours you will deploy, scale, or crash at least once.
When the bucket stops accepting writes, the bytes need a place to wait that outlives the request.
A bounded spool on disk
The spool is a directory. One file per pending object. Each file holds a one-line JSON header with the object name, then the raw bytes. Three rules keep it honest:
-
Write to a temp file, then rename. A crash mid-write leaves a
.tmpfile thatinit()deletes, never a half-written entry the drainer would choke on. - Cap both file count and bytes. A spool without a cap turns a storage outage into a full-disk outage on your own host.
-
Reserve capacity before the first
await. Twenty concurrent failures must not all pass the cap check and then all write.
// spool.mjs
import {
mkdir, readdir, readFile, rename, rm, stat, writeFile,
} from 'node:fs/promises';
import { join } from 'node:path';
import { randomUUID } from 'node:crypto';
export class SpoolFullError extends Error {}
export class Spool {
constructor(dir, { maxBytes, maxFiles }) {
this.dir = dir;
this.maxBytes = maxBytes;
this.maxFiles = maxFiles;
this.bytes = 0;
this.files = 0;
}
async init() {
await mkdir(this.dir, { recursive: true });
for (const f of await readdir(this.dir)) {
const path = join(this.dir, f);
if (f.endsWith('.tmp')) {
await rm(path, { force: true });
} else if (f.endsWith('.spool')) {
this.bytes += (await stat(path)).size;
this.files += 1;
}
}
}
async put(name, data) {
const head = Buffer.from(JSON.stringify({ name }) + '\n');
const body = Buffer.concat([head, data]);
if (this.files + 1 > this.maxFiles ||
this.bytes + body.length > this.maxBytes) {
throw new SpoolFullError(`spool full: ${this.files}`);
}
// reserve first, so concurrent puts respect the cap
this.files += 1;
this.bytes += body.length;
const id = `${Date.now()}-${randomUUID()}`;
const final = join(this.dir, `${id}.spool`);
const tmp = `${final}.tmp`;
try {
await writeFile(tmp, body, { flush: true });
await rename(tmp, final);
} catch (err) {
this.files -= 1;
this.bytes -= body.length;
await rm(tmp, { force: true });
throw err;
}
}
async list() {
const names = await readdir(this.dir);
return names.filter((f) => f.endsWith('.spool')).sort();
}
async oldestAgeMs() {
const [first] = await this.list();
return first ? Date.now() - Number(first.split('-')[0]) : 0;
}
async oldest() {
const [first] = await this.list();
if (!first) return null;
const path = join(this.dir, first);
const buf = await readFile(path);
const nl = buf.indexOf(10); // the header ends at the first \n
const { name } = JSON.parse(buf.subarray(0, nl).toString());
const data = buf.subarray(nl + 1);
return { path, name, data, size: buf.length };
}
async remove(entry) {
await rm(entry.path);
this.files -= 1;
this.bytes -= entry.size;
}
}
flush: true makes writeFile call sync() on the handle before closing it, so the bytes are on disk before the rename publishes the entry. The option landed in Node v21.0.0 and v20.10.0 per the Node fs docs. File names start with Date.now(), so a lexical sort gives you oldest-first, and the age of the oldest entry falls out of the name without opening the file. JSON.stringify escapes newlines inside the object name, so the first raw \n in the file always ends the header.
The upload path is short. Try the bucket with a timeout. If the failure is one a retry could fix, spool it. If the spool is full, put throws SpoolFullError and your caller decides what to do, which is the backpressure: return a 503 to your own client, pause a consumer, stop reading from a queue. The one thing you must not do is drop the write silently.
// uploader.mjs
const WRITE_TIMEOUT_MS = 2_000;
const NET = ['ECONNRESET', 'ETIMEDOUT', 'EPIPE'];
export function isRetryable(err) {
if (err?.name === 'TimeoutError') return true;
if (NET.includes(err?.code)) return true;
const status = Number(err?.code);
return status === 408 || status === 429 || status >= 500;
}
export async function upload(bucket, spool, name, data) {
try {
await bucket.write(name, data, {
signal: AbortSignal.timeout(WRITE_TIMEOUT_MS),
});
return 'stored';
} catch (err) {
if (!isRetryable(err)) throw err;
await spool.put(name, data); // may throw SpoolFullError
return 'spooled';
}
}
The retryable set mirrors Google's list: 408, 429, 5xx, and timeouts. A 403 or 404 is a bug in your code or config, and spooling it would hide that bug for hours. AbortSignal.timeout() gives you a signal that aborts after the given milliseconds with a TimeoutError as its reason, which is what isRetryable checks for.
Drain with full jitter
The drainer is one loop. Take the oldest entry, try to write it, delete it on success. On a retryable failure, sleep a random amount between zero and an exponentially growing ceiling. That is the "full jitter" variant from Marc Brooker's Exponential Backoff and Jitter post on the AWS Architecture Blog. When a region comes back, every client that spooled is about to retry at once. Randomized sleeps spread that herd out instead of hitting the recovering backend in synchronized waves.
// drainer.mjs
import { isRetryable } from './uploader.mjs';
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
export async function drain(bucket, spool, opts = {}) {
const {
baseMs = 250, capMs = 30_000, idleMs = 1_000,
timeoutMs = 5_000, signal,
} = opts;
let failures = 0;
while (!signal?.aborted) {
const entry = await spool.oldest();
if (!entry) {
await sleep(idleMs);
continue;
}
try {
await bucket.write(entry.name, entry.data, {
signal: AbortSignal.timeout(timeoutMs),
});
await spool.remove(entry);
failures = 0;
} catch (err) {
if (!isRetryable(err)) throw err;
failures += 1;
const ceiling = Math.min(capMs, baseMs * 2 ** failures);
await sleep(Math.random() * ceiling); // full jitter
}
}
}
One drainer per process, writing one entry at a time, is deliberately slow. A recovering storage backend is the worst place to aim a burst. If your backlog is large enough that serial draining takes too long, run a small fixed number of these loops over disjoint entries, not a Promise.all over the whole directory.
Full jitter: each retry waits a random slice of a growing window, so recovering clients don't arrive in lockstep.
The real bucket adapter has one more task beyond calling the client: make the write safe to repeat. A write that timed out on your side may still have landed on Google's side. With ifGenerationMatch: 0, the request preconditions docs say the write only succeeds if no live object with that name exists, and fails with 412 Precondition Failed otherwise. For an object name that is unique per upload, a 412 on a retry means "your earlier attempt worked," so the adapter treats it as success. The preconditionOpts shape is the one Google's Node upload sample uses.
// gcs-bucket.mjs
import { Storage } from '@google-cloud/storage';
function abortable(promise, signal) {
return new Promise((resolve, reject) => {
if (signal.aborted) return reject(signal.reason);
signal.addEventListener(
'abort', () => reject(signal.reason), { once: true },
);
promise.then(resolve, reject);
});
}
export class GcsBucket {
constructor(bucketName) {
this.bucket = new Storage().bucket(bucketName);
}
async write(name, data, { signal }) {
const save = this.bucket.file(name).save(data, {
resumable: false,
preconditionOpts: { ifGenerationMatch: 0 },
});
try {
await abortable(save, signal);
} catch (err) {
if (err?.code === 412) return; // an earlier try landed
throw err;
}
}
}
abortable stops your code from waiting. It does not cancel the HTTP request underneath, which is why the adapter sends the precondition: the abandoned request can still succeed later, and the next attempt has to cope with that. The status-on-err.code check matches how the client's ApiError carries the HTTP status. Confirm it against the client version you pin, because a wrong check here turns every successful retry into a 412 that stops the drainer, with the entry still on disk.
Watch it fill and drain
The fake bucket fails on command. While failing is true, half its writes throw a 503 and the other half hang until the caller's timeout fires, which matches the two symptoms in Google's report.
// fake-bucket.mjs
export class FakeBucket {
objects = new Map();
failing = false;
async write(name, data, { signal }) {
if (this.failing) {
if (Math.random() < 0.5) {
const err = new Error('503 UNAVAILABLE');
err.code = 503;
throw err;
}
await new Promise((_, reject) => {
if (signal.aborted) return reject(signal.reason);
signal.addEventListener(
'abort', () => reject(signal.reason), { once: true },
);
});
}
this.objects.set(name, Buffer.from(data));
}
}
The metrics endpoint exposes the three numbers you will alert on, in the Prometheus text exposition format. No client library needed.
// metrics.mjs
import { createServer } from 'node:http';
export function serveMetrics(spool, port = 9464) {
return createServer(async (req, res) => {
if (req.url !== '/metrics') return res.writeHead(404).end();
const age = (await spool.oldestAgeMs()) / 1000;
res.writeHead(200, {
'content-type': 'text/plain; version=0.0.4',
});
res.end([
'# TYPE upload_spool_files gauge',
`upload_spool_files ${spool.files}`,
'# TYPE upload_spool_bytes gauge',
`upload_spool_bytes ${spool.bytes}`,
'# TYPE upload_spool_oldest_age_seconds gauge',
`upload_spool_oldest_age_seconds ${age}`,
'',
].join('\n'));
}).listen(port);
}
The demo produces 50 writes a second, breaks the bucket from second 2 to second 8, and stops producing at second 10. The spool is capped at 200 files, which a six-second outage at that rate overflows on purpose, so you also see backpressure kick in.
// demo.mjs (Node 20.10+; run: node demo.mjs)
import { FakeBucket } from './fake-bucket.mjs';
import { Spool, SpoolFullError } from './spool.mjs';
import { upload } from './uploader.mjs';
import { drain } from './drainer.mjs';
import { serveMetrics } from './metrics.mjs';
const bucket = new FakeBucket();
const spool = new Spool('./spool-data', {
maxBytes: 5 * 1024 * 1024,
maxFiles: 200,
});
await spool.init();
const stop = new AbortController();
const drainer = drain(bucket, spool, {
signal: stop.signal, capMs: 2_000,
});
const metrics = serveMetrics(spool);
let n = 0;
const counts = { stored: 0, spooled: 0, rejected: 0 };
const producer = setInterval(async () => {
const name = `events/${n++}.json`;
const data = Buffer.from(JSON.stringify({ n, at: Date.now() }));
try {
counts[await upload(bucket, spool, name, data)] += 1;
} catch (err) {
if (!(err instanceof SpoolFullError)) throw err;
counts.rejected += 1;
}
}, 20);
const report = setInterval(() => {
console.log(
`depth=${spool.files} in_bucket=${bucket.objects.size}`,
JSON.stringify(counts),
);
}, 500);
setTimeout(() => { bucket.failing = true; }, 2_000);
setTimeout(() => { bucket.failing = false; }, 8_000);
setTimeout(() => clearInterval(producer), 10_000);
setTimeout(async () => {
stop.abort();
clearInterval(report);
metrics.close();
await drainer;
console.log('final depth', spool.files);
}, 20_000);
What to look for in the log lines: depth sits at 0 for two seconds, climbs once the bucket breaks, hits the 200 cap, and rejected starts counting. After second 8 the drainer's next attempt succeeds, its failure counter resets, and depth falls back to 0. On a clean run, the number to check at the end is that in_bucket equals stored + spooled: everything the spool accepted made it to the bucket, and everything it refused was reported to the caller, not lost. While it runs, curl localhost:9464/metrics shows the gauges move. Kill the process mid-outage and start it again, and init() picks the backlog up from disk.
Alert on the age, not only the depth. A deep spool that is draining is fine. A shallow spool whose oldest entry is two hours old means the drainer is stuck.
- alert: UploadSpoolStuck
expr: upload_spool_oldest_age_seconds > 900
for: 5m
- alert: UploadSpoolNearlyFull
expr: upload_spool_bytes > 0.8 * 5242880
for: 2m
Where this stops helping
The spool needs a disk that survives the process. On Cloud Run, the container contract says the writable filesystem "is an in-memory file system, so writing to it uses the instance's memory," and that data written to it "doesn't persist when the instance stops." A spool there eats your memory limit and dies with the instance. Same story for a Kubernetes emptyDir when the pod is deleted. You want a persistent volume, or a VM disk.
Ordering is not preserved. While the spool drains, fresh writes go straight to the bucket and overtake older spooled ones. If a consumer depends on write order, route every write through the spool whenever it is non-empty.
list() reads the whole directory on every drain step. That is fine for thousands of entries and wasteful for millions. At that scale, keep an in-memory index built at init() or move to an append-only log with an offset file.
A poison entry blocks the line. If a spooled object keeps failing with a non-retryable error, the drainer throws and stops, and the age alert above is how you find out. Decide up front whether such entries get moved to a dead-letter directory or need a human.
And the spool lives in the same failure domain as your host. If the zone that holds your disk is the one having a bad day, the backlog waits with it. A second bucket in another region is the other half of this design, and it costs you reconciliation logic on read. The spool buys you hours of not losing data. It doesn't replace that decision.
If this was useful
The spool is a small instance of an old idea: when a dependency is down, absorb the load in a bounded buffer, push back on callers when the buffer fills, and recover gently with jittered retries. The System Design Pocket Guide: Fundamentals covers those building blocks, from queues and backpressure to retries, timeouts and availability trade-offs, so you can reach for the right one before the next status page says "None at this time."
Top comments (0)