Storage monitoring assumes decay. Wear counters creep, spare blocks drain, thermals climb, and every dashboard I've ever set up is built to catch that slow slide. At 00:37 Eastern on October 24 my drive skipped the slide entirely: it stopped answering reads, its self-report stayed spotless through everything that followed, and a cold power cycle was what brought it back. The hardware in question is a Samsung 980 PRO 500GB on firmware 1B2QGXA7.
The night, straight from the log
The sequence moved fast on the way down. Read timeouts first. Then controller resets that failed. Then the kernel disabled the device. Btrfs went into emergency shutdown, and the model engine exited with the mount effectively gone. Nothing in that trail warned me; the first read timeout is the first line of the incident, and there is no decay curve anywhere in it for a monitor to catch.
The log told the story because the kernel was watching from the outside: it saw a device stop answering and said so. SMART had nothing to say, and the reason is structural.
A perfect report from a silent drive
SMART had nothing bad to report at any point. One percent used. One hundred percent spare. Zero media errors. Zero error-log entries. 47C and no thermal warning time. That is a report card written by the component that had just refused to answer, and it graded itself clean.
The mechanic underneath is simple: every one of those counters lives inside the firmware that stopped responding. One percent used is the drive's estimate of endurance consumed, one hundred percent spare means every reserve block is still there, and the zero media errors line holds even against that night's read timeouts, because the counter only moves when the drive's own firmware records the failure. All of it clean, on the same drive that refused to answer. A drive that decays slowly can still report on itself accurately, and that's the case the tools are built around. A drive going silent cannot, because the reporter is the thing that failed. My dashboards watch the first failure mode and have no view of the second.
Cold power, nothing else
A cold power cycle brought it back. Remounted read-write, engine running on the same drive again.
I'm not calling the drive fixed. The cause sits somewhere between controller or firmware on one side and PCIe or power on the other, and nothing in the evidence separates them yet. I chose to keep running on it until it recurs.
What the launch path does now
The power cycle fixed the night. The launch guard fixes the next one.
The router that picks engines no longer starts one blind. It validates the pack is actually readable before launching, and it answers 503 for storage and SSH lifecycle failures, so the engine never starts on a mount it can't read. That check turns the next incident into a refusal at launch.
The rest of the guard stack from this rewrite: routing excludes max_context=0 wrappers; a narrow overlay allowance capped at 32 MiB, exact voxtype only, with the unknown-workload guards retained; cold start reuses the resident launcher when one exists; Strata can no longer evict a resident GLM lane; lifecycle calls go through sudo -n so they match the root-owned logs and markers.
The rescue copy, and its caveat
I also keep a ~101GiB rescue copy of the pack on a different path. The copy finished. Nothing has verified its contents: no checksum comparison was performed. It's insurance against the next recurrence, and its own integrity is an open item.
What shipped, measured
Four router tests pass. A live request through the router comes back MAYA_GLM_OK on glm-5.3-flash with the fallback empty. The health endpoint reports max_context 500000 and the engine command line points at the Samsung pack. A live Strata request gets rejected with the GLM priority message. The kernel storage log has stayed quiet since the reboot.
Scope: no full-context stress test has run, and vision was never tested. The two things I can't verify are the two things I haven't run.
What's still open
Whether it recurs, and which of the two candidate causes it was. If it happens again on a drive reporting zero problems, the pack check runs first and the launch gets refused.
Ever had a component grade its own homework and come back clean while it was failing? How did you catch it? I'm collecting these for the next guard I build, comments are open.
Top comments (0)