That's quite a discovery! It's unsettling how something as ordinary as logging can quietly cause so much damage. A great reminder that even small things deserve attention. Excellent investigation! 👏
thank you! that is what made it so unsettling for me too. nothing looked broken from the outside, which is exactly why checking actual write activity matters.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
The detail that makes this land is not the 37 TB — it is "the file stayed small, and that was the trick."
That is the whole class of bug. The instrument was working perfectly and answering a question nobody needed. du reports allocated blocks, ls -l reports logical size, and neither of them is writes. The counter that actually wears the drive is on the other side of the device node — /sys/block/nvme0n1/stat field 3 (sectors written) or SMART Data Units Written — and nothing in a normal developer workflow surfaces it, which is exactly why it went unnoticed for months.
The WAL point is what makes it structurally nasty rather than just noisy. A bounded SQLite database with a churning -wal companion is designed to keep the main file flat while writing constantly. So the shape of the file is not evidence about the shape of the I/O. Checking file size there is not a weak check, it is a check for a different question.
I have been burned by the same shape from the backup side. A pg_dump app | gzip > app.sql.gz reports gzip's exit status, not pg_dump's, so killing the dump a fifth of the way through gives you exit 0, a byte-valid gzip, and a pg_restore -l that proudly lists all seven tables — because the TOC is written at the start and the data phase is what got cut. Three checks green, none of them touching the thing that failed. I ended up writing up that whole family of false-positive checks; it is in my profile if this is your kind of rabbit hole.
The useful generalisation from your post: before trusting any metric, ask what its failure would look like. If the answer is "it would look exactly the same", it is not a check.
this is such a good way to put it. a check is only useful if the failure changes what it reports.
the pg_dump example is the same trap in a different costume. valid gzip, exit 0, readable TOC, and still no complete backup. everything around the actual failure looks healthy.
that is the question I want to carry into more tooling: if the thing we care about breaks, what would this metric actually do?
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Your question is the right one to carry, and I would add a wrinkle: it has its own failure mode, which is answering it for a metric you just wrote and then assuming the answer holds for one you inherited. Same question, different answer, because the answer depends on which failure you imagine while asking it.
That is why I ended up asking it per failure mode rather than per metric. A metric with one failure mode gets one answer; one with three gets three, and they are not variations on a theme. The case I hit this week: /proc//io looks exactly like a per-process write counter, and I measured three separate ways it reports a confident number that is not the write cost. wchar counts bytes handed to write(), so a 64 MB write to tmpfs read 67,108,864 while write_bytes stayed at 0 -- nothing reached flash. cancelled_write_bytes refunds it: 256 MB written and then unlinked showed the full 268,435,456 against the device while netting zero. And the counters die with the process, so a 32 MB write inside a reaped child showed as nothing at the parent.
None of those throw an error, and that is the property I would now use as the test itself. Absence of a number looks like a bug. A wrong number looks like the world. So I would add to your question not only "what would this do if the thing broke" but "what would it still report if the thing never happened at all". If the answer is a plausible figure, the metric is measuring the instrument.
The reachability version of your question -- if the target were dead, could this check even tell me? -- I did end up packaging, because I kept needing it on machines that were not mine. It is small, local, and in my profile; no link, it either holds up or it does not.
Where I would push back on myself: I have not thought enough about the symmetric failure, the metric that moves when nothing broke. A false alarm spends attention, and attention is the budget every other check draws on. Has that one cost you more than the silent version?
yeah, false alarms have probably cost me more day to day.
not because they are worse than a silent failure in impact, but because they train people to treat alerts as background noise. after enough plausible but harmless numbers, the next real one arrives with the same tone and gets less attention.
the silent version is still scarier because you never get the chance to respond. but false alarms quietly spend the attention budget that would have caught it.
your “what would it still report if the thing never happened at all” question is the missing half. I’m going to start using both, especially for inherited metrics.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
yeah. we had one this afternoon, same shape: bounce notifications were classified
as "unknown", and unknown means no push. so the one mail that tells you your
sending reputation is burning was the quietest thing in the inbox. weeks like that.
The useful guardrail here is to monitor write amplification, not only file growth. A log can stay compact while rotation, SQLite/WAL churn, or repeated rewrites burn through flash endurance. I’d pair the SMART counter with a per-process write-rate budget and alert on a sustained slope change, then keep enough retention to correlate it with the agent or tool run that caused it.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
Yep. One gotcha on the per-process part: the counters die with the process. A
32MB write in a reaped child showed as zero at the parent. We ended up on cgroup
v2 io.stat instead.
That is exactly the boundary that makes per-process counters misleading. cgroup io.stat gives the workload a stable identity across short-lived children; I would record the cgroup path and sampling interval with each alert so a later incident review can distinguish one bursty run from a sustained write pattern.
Solo developer building practical CLI tools for Linux auditing and B2B data extraction. Creator of Vaultcheck and NPILift. I write about silent failures, false green lights, and infrastructure reliabi
agreed on the interval — io.stat is cumulative, so a burst and a slow leak read
identical until you divide by elapsed time. what we got wrong is duller: the
cgroup path is in our logs but not in the alert, so the history can't be read
back afterwards.
yes, exactly. file growth tells you about retained data. the SMART counter and sustained per-process write rate tell you about wear.
a write-rate budget with a slope alert would make this visible much earlier, especially if it keeps enough context to connect the jump to a specific agent run or tool update.
If I remember correctly a similar issue happened a while ago with the Spotify app, and probably with Firefox or Brave too... given the fact that this might be more common than anyone think, I'm starting to think that a little desktop widget to measure disk writes over a period of time would be really nice...
yeah, a small per-process write widget would be genuinely useful. not just total disk activity, but a rolling view of which app is writing, how much over the last hour or day, and whether the SSD lifetime-write counter is moving faster than usual.
that would make this class of bug much easier to spot before it becomes months of invisible wear.
The constant background indexing and snapshotting catches so many developers off guard. Even with modern high-TBW drives, seeing gigabytes of redundant temp writes every few hours is wild. Setting up dedicated tmpfs in memory or aggressive cleanup rules has become almost essential when running heavy assistant loops.
yeah, tmpfs can be a really good fit for disposable logs, indexes, and CI-style workspaces.
I would be careful with cleanup rules though. deleting files saves space, but rebuilding the same artifacts later can create just as much write activity again. the best win is keeping truly temporary churn out of persistent storage in the first place.
Very true, the other thing people dont really account for, is build artifacts. Not only do they bloat, but remember, when you have 100gb of build artifacts that you delete, that's 100gb of writes... Do that every 2-3 days, it adds up quite fast...
yeah, build artifacts are a big part of the same problem. one small correction though: deleting 100 GB usually does not write another 100 GB to the SSD. it mostly changes filesystem metadata and may issue TRIM commands.
the expensive part is rebuilding those 100 GB again, especially when the build creates lots of small temporary files, caches, and intermediate outputs. doing that every few days can add up very quickly.
That's what i mean, yes. Example, I am building a game as a hobby project, already at around 11m lines of code in rust. Dwarven Stronghold (like Dwarf Fortress, but expanded). Everytime it builds, it needs around 50gb of memory, which overflows my ram, so page-file eats at my ssd. Then the build artifact, which again, eats at the ssd, so because it makes me run out of space quite fast, I told the agent to run regular housekeeping, so that 100gb every day or so that it writes to disk, gets deleted and re-written with the next build.
Hi! Wouldn't it be amazing if we have one platform where we can install ai model locally and do everything we want in our daily lives there only. Like there are some apps but we have to connect them ,integrate many other apps why not just 1.
that would be really useful if it had one simple interface but kept permissions separate underneath.
one app can make the experience easier, but it still needs scoped connections to email, files, calendars, and everything else. the important part is being able to see exactly what it can access and turn off one connection without breaking everything.
that makes sense. using an online version moves most of the heavy work off your machine, though I would still keep an eye on browser storage and any local app or extension it installs.
the bigger lesson is just to check what a tool is doing locally instead of assuming a small folder means a small footprint.
Implementation Specialist and Database Developer with over 15 years of experience in the IT industry, specializing in the design, implementation and management of relational databases.
The café bit is the best explanation of this I've seen. ls shows you who's sitting down, not how many came through the door.
I had a tiny version of this. An alarm app I built kept asking my server for updates every 15 seconds, even after my session expired. 240 refused requests an hour, nothing on my screen looked wrong, and my own firewall ended up blocking my IP lol.
The stuff running quietly in the background is what I trust least now.
Great article! One interesting point is that limiting database size through retention policies doesn't necessarily limit disk I/O or SSD wear.
I also noticed that, despite the fixes introduced in June, issue #43378 still reports excessive TRACE/DEBUG logging as recently as November 4.
From an architectural perspective, this reinforces an important lesson: observability needs resource budgets, not just retention policies. Reducing stored data isn't the same as reducing the cost of writing it.
The distinction that hit me hardest was file size versus bytes written — du and ls answer how much is stored right now, not how much churn the drive has absorbed, and the cafe analogy made that click. The RUST_LOG=warn detail is the scarier part: a user could reasonably believe they had turned logging down while the TRACE sink kept writing. It raises a practical question for other tools too: have you checked whether Claude Code or other AI CLIs show a similar pattern, or is this specific to how Codex wired its feedback sink? Either way, the habit you suggest — checking write rate once when installing a long-running tool, rather than only checking free space — seems worth adopting widely.
This is the part that stuck with me — the file looked fine, so nobody had a reason to look deeper. File size tells you how much is stored, not how much work the drive actually did.
I'm going to check the SMART counter on my main machine this weekend, because that's the only number that really tells the story. Thanks for laying out the 0.142.0 fix and the checks so clearly — the update first, then measure, then check the drive order makes a lot of sense.
We're a place where coders share, stay up-to-date and grow their careers.
This is why I have trust issues
honestly, fair . In this AI world even i can't trust anything.....
That's quite a discovery! It's unsettling how something as ordinary as logging can quietly cause so much damage. A great reminder that even small things deserve attention. Excellent investigation! 👏
thank you! that is what made it so unsettling for me too. nothing looked broken from the outside, which is exactly why checking actual write activity matters.
Indeed, therein lies the crux of the matter.
The detail that makes this land is not the 37 TB — it is "the file stayed small, and that was the trick."
That is the whole class of bug. The instrument was working perfectly and answering a question nobody needed.
dureports allocated blocks,ls -lreports logical size, and neither of them is writes. The counter that actually wears the drive is on the other side of the device node — /sys/block/nvme0n1/stat field 3 (sectors written) or SMART Data Units Written — and nothing in a normal developer workflow surfaces it, which is exactly why it went unnoticed for months.The WAL point is what makes it structurally nasty rather than just noisy. A bounded SQLite database with a churning -wal companion is designed to keep the main file flat while writing constantly. So the shape of the file is not evidence about the shape of the I/O. Checking file size there is not a weak check, it is a check for a different question.
I have been burned by the same shape from the backup side. A pg_dump app | gzip > app.sql.gz reports gzip's exit status, not pg_dump's, so killing the dump a fifth of the way through gives you exit 0, a byte-valid gzip, and a pg_restore -l that proudly lists all seven tables — because the TOC is written at the start and the data phase is what got cut. Three checks green, none of them touching the thing that failed. I ended up writing up that whole family of false-positive checks; it is in my profile if this is your kind of rabbit hole.
The useful generalisation from your post: before trusting any metric, ask what its failure would look like. If the answer is "it would look exactly the same", it is not a check.
this is such a good way to put it. a check is only useful if the failure changes what it reports.
the pg_dump example is the same trap in a different costume. valid gzip, exit 0, readable TOC, and still no complete backup. everything around the actual failure looks healthy.
that is the question I want to carry into more tooling: if the thing we care about breaks, what would this metric actually do?
Your question is the right one to carry, and I would add a wrinkle: it has its own failure mode, which is answering it for a metric you just wrote and then assuming the answer holds for one you inherited. Same question, different answer, because the answer depends on which failure you imagine while asking it.
That is why I ended up asking it per failure mode rather than per metric. A metric with one failure mode gets one answer; one with three gets three, and they are not variations on a theme. The case I hit this week: /proc//io looks exactly like a per-process write counter, and I measured three separate ways it reports a confident number that is not the write cost. wchar counts bytes handed to write(), so a 64 MB write to tmpfs read 67,108,864 while write_bytes stayed at 0 -- nothing reached flash. cancelled_write_bytes refunds it: 256 MB written and then unlinked showed the full 268,435,456 against the device while netting zero. And the counters die with the process, so a 32 MB write inside a reaped child showed as nothing at the parent.
None of those throw an error, and that is the property I would now use as the test itself. Absence of a number looks like a bug. A wrong number looks like the world. So I would add to your question not only "what would this do if the thing broke" but "what would it still report if the thing never happened at all". If the answer is a plausible figure, the metric is measuring the instrument.
The reachability version of your question -- if the target were dead, could this check even tell me? -- I did end up packaging, because I kept needing it on machines that were not mine. It is small, local, and in my profile; no link, it either holds up or it does not.
Where I would push back on myself: I have not thought enough about the symmetric failure, the metric that moves when nothing broke. A false alarm spends attention, and attention is the budget every other check draws on. Has that one cost you more than the silent version?
yeah, false alarms have probably cost me more day to day.
not because they are worse than a silent failure in impact, but because they train people to treat alerts as background noise. after enough plausible but harmless numbers, the next real one arrives with the same tone and gets less attention.
the silent version is still scarier because you never get the chance to respond. but false alarms quietly spend the attention budget that would have caught it.
your “what would it still report if the thing never happened at all” question is the missing half. I’m going to start using both, especially for inherited metrics.
yeah. we had one this afternoon, same shape: bounce notifications were classified
as "unknown", and unknown means no push. so the one mail that tells you your
sending reputation is burning was the quietest thing in the inbox. weeks like that.
that is brutal. the system had a category for “we do not know what this is” and used it to make the message disappear.
those are the failures that make you stop trusting a green dashboard. hope you caught it early enough to limit the damage.
The useful guardrail here is to monitor write amplification, not only file growth. A log can stay compact while rotation, SQLite/WAL churn, or repeated rewrites burn through flash endurance. I’d pair the SMART counter with a per-process write-rate budget and alert on a sustained slope change, then keep enough retention to correlate it with the agent or tool run that caused it.
Yep. One gotcha on the per-process part: the counters die with the process. A
32MB write in a reaped child showed as zero at the parent. We ended up on cgroup
v2 io.stat instead.
That is exactly the boundary that makes per-process counters misleading. cgroup io.stat gives the workload a stable identity across short-lived children; I would record the cgroup path and sampling interval with each alert so a later incident review can distinguish one bursty run from a sustained write pattern.
agreed on the interval — io.stat is cumulative, so a burst and a slow leak read
identical until you divide by elapsed time. what we got wrong is duller: the
cgroup path is in our logs but not in the alert, so the history can't be read
back afterwards.
yes, exactly. file growth tells you about retained data. the SMART counter and sustained per-process write rate tell you about wear.
a write-rate budget with a slope alert would make this visible much earlier, especially if it keeps enough context to connect the jump to a specific agent run or tool update.
If I remember correctly a similar issue happened a while ago with the Spotify app, and probably with Firefox or Brave too... given the fact that this might be more common than anyone think, I'm starting to think that a little desktop widget to measure disk writes over a period of time would be really nice...
yeah, a small per-process write widget would be genuinely useful. not just total disk activity, but a rolling view of which app is writing, how much over the last hour or day, and whether the SSD lifetime-write counter is moving faster than usual.
that would make this class of bug much easier to spot before it becomes months of invisible wear.
I didn't think about the "pre-process", but I agree that it's what would make it really useful!
The constant background indexing and snapshotting catches so many developers off guard. Even with modern high-TBW drives, seeing gigabytes of redundant temp writes every few hours is wild. Setting up dedicated tmpfs in memory or aggressive cleanup rules has become almost essential when running heavy assistant loops.
yeah, tmpfs can be a really good fit for disposable logs, indexes, and CI-style workspaces.
I would be careful with cleanup rules though. deleting files saves space, but rebuilding the same artifacts later can create just as much write activity again. the best win is keeping truly temporary churn out of persistent storage in the first place.
Very true, the other thing people dont really account for, is build artifacts. Not only do they bloat, but remember, when you have 100gb of build artifacts that you delete, that's 100gb of writes... Do that every 2-3 days, it adds up quite fast...
yeah, build artifacts are a big part of the same problem. one small correction though: deleting 100 GB usually does not write another 100 GB to the SSD. it mostly changes filesystem metadata and may issue TRIM commands.
the expensive part is rebuilding those 100 GB again, especially when the build creates lots of small temporary files, caches, and intermediate outputs. doing that every few days can add up very quickly.
That's what i mean, yes. Example, I am building a game as a hobby project, already at around 11m lines of code in rust. Dwarven Stronghold (like Dwarf Fortress, but expanded). Everytime it builds, it needs around 50gb of memory, which overflows my ram, so page-file eats at my ssd. Then the build artifact, which again, eats at the ssd, so because it makes me run out of space quite fast, I told the agent to run regular housekeeping, so that 100gb every day or so that it writes to disk, gets deleted and re-written with the next build.
Hi! Wouldn't it be amazing if we have one platform where we can install ai model locally and do everything we want in our daily lives there only. Like there are some apps but we have to connect them ,integrate many other apps why not just 1.
that would be really useful if it had one simple interface but kept permissions separate underneath.
one app can make the experience easier, but it still needs scoped connections to email, files, calendars, and everything else. the important part is being able to see exactly what it can access and turn off one connection without breaking everything.
Yeah, this is why if I WERE to use artificial intelligence to code, I would try only to use online versions.
that makes sense. using an online version moves most of the heavy work off your machine, though I would still keep an eye on browser storage and any local app or extension it installs.
the bigger lesson is just to check what a tool is doing locally instead of assuming a small folder means a small footprint.
Hello :-)
did you check other AI tools like Claude Code for similar write patterns?
The café bit is the best explanation of this I've seen. ls shows you who's sitting down, not how many came through the door.
I had a tiny version of this. An alarm app I built kept asking my server for updates every 15 seconds, even after my session expired. 240 refused requests an hour, nothing on my screen looked wrong, and my own firewall ended up blocking my IP lol.
The stuff running quietly in the background is what I trust least now.
Great article! One interesting point is that limiting database size through retention policies doesn't necessarily limit disk I/O or SSD wear.
I also noticed that, despite the fixes introduced in June, issue #43378 still reports excessive TRACE/DEBUG logging as recently as November 4.
From an architectural perspective, this reinforces an important lesson: observability needs resource budgets, not just retention policies. Reducing stored data isn't the same as reducing the cost of writing it.
The distinction that hit me hardest was file size versus bytes written — du and ls answer how much is stored right now, not how much churn the drive has absorbed, and the cafe analogy made that click. The RUST_LOG=warn detail is the scarier part: a user could reasonably believe they had turned logging down while the TRACE sink kept writing. It raises a practical question for other tools too: have you checked whether Claude Code or other AI CLIs show a similar pattern, or is this specific to how Codex wired its feedback sink? Either way, the habit you suggest — checking write rate once when installing a long-running tool, rather than only checking free space — seems worth adopting widely.
This is the part that stuck with me — the file looked fine, so nobody had a reason to look deeper. File size tells you how much is stored, not how much work the drive actually did.
I'm going to check the SMART counter on my main machine this weekend, because that's the only number that really tells the story. Thanks for laying out the 0.142.0 fix and the checks so clearly — the update first, then measure, then check the drive order makes a lot of sense.