Incident Report
A routine PR added an auto-resizing MutationObserver to the NOC telemetry table widget so column widths adjusted when live alerts streamed in. Every 10-minute QA smoke test passed with flat memory graphs.
Six hours into the Monday shift, dedicated wallboard monitors running the dispatch SPA began crashing with Chrome's Aw, Snap! (Out of Memory) error.
- T+00:00: Initial JS heap settles at 120 MB after dashboard boot.
- T+02:00: Operators switch between regional tabs; heap climbs at 280 MB/hour to 680 MB.
-
T+05:30: V8 Mark-Compact GC pauses hit 900 ms per frame; DevTools heap snapshots show 18,400
Detached HTMLDivElementnodes and 28 retainedFiberNodetrees. -
T+07:45: Renderer process crosses 2.4 GB and terminates with
Aw, Snap!.
The Red Herring
The first Slack reflex was to add telemetryRows = null and host = null inside the useEffect cleanup function. Local testing showed a brief drop in the DevTools memory timeline after manual GC, so the team shipped the hotfix.
Production crashes continued on the exact same 6-to-8-hour cadence. Nulling local bindings inside a cleanup closure does not sever the native C++ retainer edge held by an active MutationObserver or an uncancelled fetch stream.
Causal Chain
The leak compounds across five runtime layers. Uncollected allocations promote into V8 Old Space, native DOM observers act as immortal GC roots, stale hook closures pull entire component trees into those roots, un-disconnected MutationObserver instances persist across route unmounts, and absent AbortController wiring leaves no unified teardown path.
Failure Simulator
Layer Dissection
01 → 02: Surviving Allocations Promote to Immortal Retainers
Under JavaScript Generational Garbage Collection, short-lived objects die in V8's Young Generation during fast Scavenge cycles. Allocations that remain reachable across two scavenges promote into Old Space, where reclamation requires expensive Mark-Compact sweeps.
As shown in Memory Leaks, Retainers, and Detached DOM Trees, a single reachable JS reference to an unmounted <div class="chart-host"> pins the entire detached DOM subtree in Old Space.
02 → 03: Retainer Paths Originate in Stale Hook Closures
The retainer edge rarely comes from a global window.leak = node assignment. Instead, Stale Closures in Hooks & Callbacks capture the lexical environment of the render in which an effect was created:
-
Captured state: A 40 MB
telemetryRowsarray fromt = 0h 15mstays bound inside the callback scope. -
Captured ref:
widgetRef.currentpoints to the detached DOM container after the user navigates to another route.
03 → 04: Native MutationObserver Keeps the Closure Reachable
V8 fails to collect the stale closure after unmount because MutationObserver and Batched Microtask Delivery registers the callback inside Blink's C++ DOM mutation agent.
-
Native GC root: Until
observer.disconnect()runs, Blink holds a strong edge to the JS callback. - Retained graph: Every variable and DOM node inside that callback closure remains reachable across route swaps.
04 → 05: Absent AbortController Prevents Unified Teardown
In the incident codebase, the effect also opened a fetch stream and attached a window resize listener. Without AbortController and Cooperative Cancellation, early returns skipped observer.disconnect() whenever host changed mid-flight, leaving partial subscriptions alive.
Architectural Fix & Invariants
useEffect(() => {
const host = widgetRef.current;
if (!host) return;
const controller = new AbortController();
const { signal } = controller;
const observer = new MutationObserver(() => syncLayout(host));
observer.observe(host, { childList: true, subtree: true });
signal.addEventListener('abort', () => observer.disconnect(), { once: true });
window.addEventListener('resize', () => syncLayout(host), { signal });
fetch('/api/telemetry', { signal })
.then((r) => r.json())
.then((rows) => renderChart(host, rows))
.catch((err) => { if (err.name !== 'AbortError') reportError(err); });
return () => controller.abort();
}, []);
Architectural Invariants:
-
Single-token teardown: Every
useEffectthat allocates a DOM observer,windowlistener, or network stream creates oneAbortControllerand returns() => controller.abort(). -
Paired observer disconnect: Every
observer.observe()call registers{ once: true }onsignalto invokeobserver.disconnect()atomically. -
Thin callback scopes: Observers read only the DOM node passed to their callback or a mutable
ref, never large historical state arrays from outer renders. -
Soak-test verification: Navigation stress tests swap routes 50 times in Playwright and assert
Detachednode count returns to baseline aftergc().
Golden Rule: Nulling variables never collects memory held by native C++ DOM roots; bind every observer, listener, and fetch to one AbortSignal and abort on unmount.

Top comments (0)