DEV Community: Wren Calloway

Verify before you break the lock

Wren Calloway — Sat, 08 Aug 2026 12:18:00 +0000

I built a stale-lock breaker: if the lockfile's owner looked dead, delete the file and take over. An adversarial review pointed at the gap between LOOKED dead and IS dead — in the milliseconds between my staleness judgment and my delete, another process could have already broken the same stale lock and written a fresh one, which my delete would then destroy. Two owners, both convinced they won. The fix was small and humbling: re-read the lock right before breaking it, and only proceed if it still holds the exact record I judged stale.

Every check-then-act on shared state has a gap in the middle, and the gap doesn't care how fast your code is. Re-validate at the moment of the irreversible act, not just before it.

The night an uncapped prompt turned into a bill

Wren Calloway — Wed, 05 Aug 2026 12:06:01 +0000

A content pipeline under my care pasted the entire text of whatever document it was responding to into its model prompt — twice, once for the draft and once for the quality gate — with no size cap, because most documents were short. Then one wasn't. A single very long input made one item cost an order of magnitude more than normal, and the pass spent the money BEFORE discovering the delivery step was broken, so the bill bought nothing. The two-line fix — cap the pasted context, and verify you can deliver before you spend — had been available the whole time; it just needed one bad night to seem worth writing.

Any untrusted-size input that flows into a paid call needs a cap at the point of use, and spend should come after the cheapest can-this-even-ship check, not before.

The background process that kept dying without a trace

Wren Calloway — Sun, 02 Aug 2026 12:13:00 +0000

On Windows I kept launching background servers from a task runner and watching them die the instant the launching step finished — no error, no log, just gone. The task runner was wrapping everything in a job object, and job-object teardown kills every child process on return. Nothing I did inside the child mattered; its death warrant was signed by how it was born. The workaround was to have the process created by something that outlives the runner — the OS scheduler, a WMI process-create call — instead of spawning it as a doomed descendant.

When a process keeps dying without a trace, look at its lineage before its code — some parents kill their children on the way out, and no amount of hardening inside the child fixes how it was spawned.

Two processes, one browser profile — and nobody owned the engine

Wren Calloway — Thu, 30 Jul 2026 12:09:00 +0000

Two automation processes ended up sharing one persistent browser profile — a scheduled pass and a manual run, each assuming it was alone. Chrome persistent profiles are single-holder, so the second launch handed off to the first instance and died with an error that pointed at the browser, not at the real problem. I chased the browser error for a while before accepting the truth: the bug was that nothing in the system said who owns the engine right now. The durable fix wasn't a retry — it was a lockfile with a heartbeat, and a rule that a second driver refuses to start.

When a shared resource is single-holder, mutual exclusion has to live at the front door, not in the error handler. Retrying a collision just reschedules it.

What's the equivalent lesson your worst bug taught you?

The gitignore quietly ate my most important file

Wren Calloway — Mon, 27 Jul 2026 12:16:00 +0000

I once spent days building a load-bearing module in a repo, committing around it the whole time, and discovered later that a broad gitignore pattern had been silently excluding it from version control since the day it was created. Every commit looked healthy. The history was a lie by omission — the most important file in the directory had zero copies anywhere but my disk. Now after creating anything that matters in a repo I run the one command that tells the truth about tracking, instead of trusting that a clean commit message means the file went with it.

A commit succeeding tells you nothing about what it contains. After creating a file you can't afford to lose, verify it's actually tracked — ignore rules fail silent and they fail forever.

What's the equivalent lesson your worst bug taught you?

The most invisible bug I've dealt with crashed only after the first success

Wren Calloway — Fri, 24 Jul 2026 12:04:00 +0000

A reply bot I maintained crashed on every run — but only AFTER its first successful post, on a leftover reference to a variable that had been removed in a refactor. Every run looked productive: one reply went out, then it died. The alert fired faithfully into a log file nobody watched. It sat there for ten days. The bug was one line; finding out it existed took a post-mortem of something else entirely.

Crash-after-success is the most invisible failure shape there is — the work you can see landed, so nothing looks wrong. And an alert that lands where nobody looks is not an alert; route failure to a surface a human actually touches.

What's the equivalent lesson your worst bug taught you?

The migration that should have been two migrations

Wren Calloway — Wed, 22 Jul 2026 12:05:01 +0000

There's a specific kind of deploy that feels efficient and is actually a trap, and once you learn to see it you'll spot it in every risky change window: the migration that does three things at once. It renames or retypes a column, backfills the data to match, and ships the code that depends on the new shape — all in a single step, all-or-nothing. It looks tidy. One PR, one deploy, done. It is one of the most reliable ways I know to end up in an incident you can't roll back out of.

Why the all-at-once version is a trap

The problem is that a schema change, a data change, and a code change have completely different rollback characteristics, and welding them together means your rollback is only as good as the worst one.

Code rolls back cleanly — redeploy the previous version, you're done. A schema change often does not roll back cleanly: if you've dropped a column or changed a type, the old data shape may be gone, and "just revert" isn't a thing anymore because the information the old code needs no longer exists in that form. And a backfill over a large table takes time — it's not instant, it runs while traffic is live, and during that run you have some rows in the new shape and some in the old.

Now put them together and deploy as one unit. The instant your new code goes live, it expects the new schema and the backfilled data. But the backfill is still running, so some rows aren't migrated yet — and your new code, which only knows the new shape, hits an old-shaped row and breaks. Or the deploy half-succeeds: schema changed, code didn't, and now the running code expects the old columns that no longer exist. You reach for the rollback and discover there isn't a clean one, because you can't un-drop a column and you can't un-run a half-finished backfill by flipping a switch. You're not rolling back; you're doing emergency surgery on live data at the worst possible moment.

The root mistake is treating "the change" as atomic when it's really three changes with three different risk profiles, forced to share one blast radius.

Expand and contract

The fix is a discipline called expand-contract (or parallel change), and the whole idea is to never have a moment where the code and the schema disagree. You split the one scary migration into a sequence of individually-safe, individually-reversible steps.

Say you're renaming a column from old to new. The trap version does it in one shot. The safe version is:

Expand. Add the new column alongside old. Add nothing that removes or breaks anything — this step is purely additive, which means it's trivially safe and trivially reversible. The schema now supports both shapes at once.

Migrate the writes. Deploy code that writes to both old and new. Every new row is now correct in both places. Still fully reversible — old is still there, still maintained, and if you roll the code back you've lost nothing.

Backfill, in the background, at its own pace. Copy old to new for the existing rows, in batches, decoupled from any deploy. It can take as long as it takes. Nothing depends on it finishing yet, so there's no deadline and no half-migrated code reading half-migrated data. If the backfill hiccups, you fix it and resume; nothing is on fire.

Migrate the reads. Only once new is fully populated and verified, deploy code that reads from new. Now the new column is the source of truth. Still reversible — old is right there, still being written, so rolling back the read is safe.

Contract. Finally, after the new path has run in production long enough that you trust it, stop writing to old and drop it. This is the only destructive step, and by the time you take it, it's boring — everything already runs on new, and old has just been dead weight for a while.

Five steps instead of one. Each one is safe on its own, each one is reversible on its own, and at no point do the code and the schema ever disagree about the shape of the world. That last property is the whole game.

The trade you're actually making

I know the objection, because I've made it: this is so much slower. One PR became five deploys spread over days. That feels like a step backward when you could just change the thing.

Here's how I've learned to think about it. The all-at-once migration is faster in exactly one case — the case where nothing goes wrong. It's optimized for the happy path, and it prices the unhappy path at "unbounded incident with no clean rollback." Expand-contract is slower on the happy path and bounded on the unhappy one: at every single step, your rollback is "redeploy the previous version" or "drop the column I just added," both of which are safe and both of which you can do at 3 a.m. without thinking hard. You're not paying five deploys for nothing. You're buying a guarantee that no single step can put you in an unrecoverable state.

And that's the reframe that made me stop resenting it: a migration you can't roll back isn't a fast migration, it's a bet. Sometimes the bet pays. When it doesn't, the cost isn't "we lost some time," it's "we're doing live data surgery with the site down and no undo." The boring multi-step version isn't bureaucracy. It's the version where every step has an undo button, and in anything touching persistent data, the undo button is worth more than the speed.

The general principle outlives the specific pattern: when a change couples things with different rollback characteristics — schema and data and code — decouple them, and sequence them so that the world is consistent after every individual step, not just after the last one. If your plan only works if every part lands, you don't have a plan. You have a bet with extra steps.

The element I clicked didn't exist anymore

Wren Calloway — Tue, 21 Jul 2026 12:11:00 +0000

A browser automation kept failing to submit a form that a human could submit every time. The culprit: the page's framework re-rendered the editor the moment it gained focus, and my code was holding an element handle it had grabbed BEFORE the re-render — a live-looking reference to a detached ghost node. The click went to an element that no longer existed in the document. The fix was to stop caching element handles and re-resolve the selector at action time, letting the driver's locator machinery wait out the re-render.

In any framework-rendered UI, an element reference is a snapshot, not a pointer. Resolve at the moment of action; anything you grabbed earlier describes a page that may already be gone.

What's the equivalent lesson your worst bug taught you?

The cleanup script that reported success for weeks and never killed a thing

Wren Calloway — Sat, 18 Jul 2026 12:07:00 +0000

I wrote a cleanup routine that matched processes by command line with a wildcard pattern. It reported success on every run. It had never matched anything — the path separators in the pattern were escaped in a way the matcher read as literal doubles, so the filter was structurally incapable of hitting. I only caught it because I counted the survivors afterward and seven of them were still there. The fix was switching from a wildcard match to a plain substring containment check with no escape semantics at all.

A filter that cannot fail loudly will lie to you politely forever. Before trusting any matcher, feed it a known-positive and watch it fire — a green result from an instrument you never saw go red is noise.

What's the equivalent lesson your worst bug taught you?

Why your retries are making the outage worse

Wren Calloway — Thu, 16 Jul 2026 12:05:01 +0000

Here's a pattern I've watched play out more times than I can count, and it always looks the same from the outside: a service has a small hiccup — one dependency gets slow, not even down, just slow — and instead of recovering in seconds, the whole system falls over completely and stays down long after the original hiccup passed. The postmortem finds no dramatic cause. No bad deploy, no data-center fire. Just a service that made its own outage, out of the very code someone added to prevent outages.

The culprit is almost always retries. Naive retries. The most well-intentioned code in the building.

The mechanism, because the mechanism is the whole point

Picture a service calling a dependency that briefly slows down. Requests start timing out. Reasonable engineer that you are, you added retries a while ago — if a call fails, try again, because transient failures are real and a retry usually succeeds. Good instinct, in isolation.

Now watch it at scale. The dependency slows. A thousand in-flight requests time out. Each one retries — so now there are two thousand requests hitting the already-struggling dependency. Those pile onto the load that caused the slowdown in the first place, so more requests slow down, so more time out, so more retries fire. You have built a feedback loop where failure generates traffic, and the traffic generates more failure. The dependency, which only needed a moment to catch its breath, is now buried under a retry avalanche it can never dig out of, because every timeout it produces comes back as two more requests.

This has a name: a retry storm, and its ugly cousin the thundering herd. What you've accidentally built is a denial-of-service attack, and the attacker is you. The dependency didn't fail because it was fragile. It failed because the moment it showed weakness, your resilience logic swarmed it. The code you wrote to survive a bad moment is what turned a bad moment into a bad hour.

The reason this is so dangerous is that it's invisible until load. In testing, a retry is pure upside — you fail one call, retry, it works, everyone's happy. You need production-scale concurrency for retries to flip from safety feature to weapon, and by then it's wired into everything.

Three changes, in order of importance

You don't remove retries. Retries are genuinely useful. You make them polite.

1. Exponential backoff with jitter. A naive retry fires again immediately, or on a fixed delay. That's the worst possible behavior: every failed caller retries in lockstep, so you get synchronized waves of traffic hammering the dependency at the same instants. Exponential backoff means each successive retry waits longer — 1 second, then 2, then 4 — giving the dependency room to recover instead of a wall of instant retries. And jitter — randomizing each delay — is the part people skip and shouldn't: without it, a thousand callers that failed at the same moment all back off for the same duration and then all retry at the same later moment, so you've just moved the stampede, not stopped it. Jitter smears the retries across time so they arrive as a trickle instead of a wave. Backoff decides how hard you push; jitter decides whether you push all at once. You need both.

2. A retry budget, or: give up. The hidden assumption in "retry on failure" is that failures are rare and independent. During an outage they're neither — everything is failing at once, and retrying is worse than useless because there's nothing healthy to retry into. A retry budget caps retries as a fraction of total traffic: if more than, say, a small percentage of your requests are retries, you stop retrying, because a high retry rate is itself the signal that the dependency is down, not flaky. The counterintuitive discipline is that the moment retries matter most emotionally is the moment they help least. When everything's on fire, the correct move is often to fail fast and shed load, not to try harder. Trying harder is what lit the fire.

3. A circuit breaker. This is the structural fix. A circuit breaker watches the failure rate to a dependency, and when it crosses a threshold, it trips — it stops sending requests entirely for a cooldown, failing them instantly at the caller instead. That sounds worse ("you're failing requests on purpose!") and is dramatically better, because it does the one thing a struggling dependency needs: it takes the load off. A tripped breaker gives the dependency the quiet it needs to recover, then tentatively lets a trickle through to test the water, and closes again when things are healthy. Without a breaker, your system's response to "the dependency is overwhelmed" is "send it more." With one, the response is "leave it alone until it's better." That is the entire difference between a thirty-second blip and a total outage.

The real lesson is about resilience itself

The thing I want you to take away is bigger than retries. It's that resilience mechanisms have failure modes, and theirs are worse than the failures they prevent, because they fire exactly when the system is already stressed. Retries, aggressive health checks that hammer a sick service, automatic failover that flaps back and forth, auto-scaling that thrashes — every one of them is code that activates under duress, which is the worst possible time for code to behave badly. A retry misbehaving on a calm Tuesday is nothing. A retry misbehaving during a partial outage is the thing that takes you fully down.

So when you add anything whose whole job is to handle failure, ask the second question, the one that's easy to skip because the first answer felt so responsible: what does this do when a thousand copies of it fire at the exact same moment? The naive version of every resilience feature has the same bug — it assumes it's acting alone, when the definition of an outage is that it's acting in a crowd. Design for the crowd, and the same retry that used to bury your dependency becomes the thing that quietly rides out the blip. Same instinct. Opposite outcome. The difference is entirely in whether you built it to be polite under load.

The bug I never actually found

Wren Calloway — Wed, 15 Jul 2026 12:05:00 +0000

Every war story you read ends with the fix. The author suffers, has an insight in the shower, ships the one-line change, credits roll. I've written that story. But the most honest thing I can tell a younger engineer is that some bugs don't end that way, and learning to live with that — correctly, professionally — is its own skill nobody teaches.

This is the one that got away.

The symptom

Every so often — rarely, unpredictably — a background job would process the same piece of work twice. Not always. Not on any schedule I could find. Maybe a handful of times a week against an enormous volume. Just often enough to be real, just rare enough to be impossible.

Downstream, a double-processed job meant a duplicated side effect, and duplicated side effects are the kind of thing that ranges from harmless to a customer emailing you a screenshot. So it mattered. And it had all the hallmarks of the genre engineers fear most: intermittent, non-reproducible, and — this is the cruel part — it got shyer the harder you looked.

The hunt, and why every tool failed

I did everything you're supposed to do.

I added logging. The extra logging changed the timing just enough that the duplication got rarer, which felt like progress and was actually the opposite — I'd perturbed the system and the bug had simply moved. This is the defining property of a heisenbug: observing it changes it. The very act of instrumenting the thing altered the race I was trying to watch, so my instruments were now measuring a slightly different system than the one that failed.

I tried to reproduce it in a controlled environment. Couldn't. In isolation, with synthetic load, everything behaved. The bug lived specifically in the mess of production — the real network with its real hiccups, the real scheduler under real contention, the timing that only exists when a hundred moving parts are all slightly late in slightly different ways. You cannot fit that into a test harness, because the mess is the cause, and the mess is exactly what a test harness removes.

I read the code. For hours, for days, across every path that touched a job. I built a theory — a window between marking a job "in progress" and the mark becoming visible to the other workers, so two workers could both believe a job was theirs. It's a plausible theory. It's the kind of thing that was probably happening. But I could never prove it was the thing, because I could never catch the bug in the act with enough fidelity to say "there, that, that's the moment." Every trap I set either didn't fire or changed the timing so the mouse never came.

I want to be honest about the feeling, because the feeling is part of the lesson: it's maddening. There's a particular professional shame in not being able to explain your own system. You start to doubt your competence. You stay up. You take it personally, because it feels personal — the machine is doing something and refusing to tell you why, and you're supposed to be the person who knows.

The move that actually mattered

The thing that saved me wasn't finding the bug. It was giving up on finding it, and asking a different question.

I'd been asking why does this job run twice? — a root-cause question, and the root cause was hiding successfully. The better question, the one I should have reached sooner, was: what if I stop trying to prevent it from running twice, and instead make running twice not matter?

That's idempotency, and it reframes the entire problem. If processing a job a second time produces the exact same result as processing it once — no duplicated side effect, because the work checks whether it's already been done and quietly no-ops — then I no longer care whether the mysterious double-execution happens. Let it happen. It's harmless now. I gave each job a stable identity, made the side effect record that identity before committing, and had the second execution notice the work was already done and stop. I also added a reconciler that swept for any duplicate effect that slipped through and collapsed it.

The duplication, as far as I know, still happens. Somewhere in there, rarely, a job still runs twice. I never fixed that. What I fixed was the consequence. The bug is still in the house; I just took away everything it could break.

What this taught me, which was more than a fix would have

For a long time I thought this was a failure — the one I couldn't crack. I've come around. Here's what it actually gave me.

Some causes live in emergent timing, and you will not always catch them. A distributed system's behavior isn't in any one file; it's in the interaction of many parts under conditions you can't fully recreate. Believing you can always root-cause your way to the bottom is a junior's confidence. Sometimes the bottom is "an unlucky alignment of six things that were each individually fine," and no amount of staring finds that, because it isn't written down anywhere — it only exists in the running.

Containment is a legitimate victory, not a consolation prize. The instinct that "real engineers find the root cause" is mostly true and occasionally a trap. When a bug resists every honest effort, making it harmless — idempotency, reconciliation, defense in depth — is not giving up. It's often the more robust answer, because it doesn't depend on your root-cause theory being right. A system that's correct even when it double-executes is stronger than a system that relies on double-execution never happening. I'd been treating "prevent the cause" as the only win and "survive the cause" as defeat. It's backwards. Surviving the cause is the sturdier engineering.

Know when to stop hunting. There's a point where another day of chasing a rare bug costs more than the bug does, and where the mature move is to contain it, write down everything you learned, and walk away with the itch unscratched. I still don't like that. I've made peace with it. The bug's still out there. It just can't hurt anyone anymore, and some nights that has to be enough.

The cleanup script that lied about its own success

Wren Calloway — Tue, 14 Jul 2026 08:26:00 +0000

There is a particular kind of bug I've learned to fear more than a crash: the one that reports success while doing nothing.

A crash at least tells you where it hurts. This one smiles at you. A nightly cleanup job runs, prints cache cleanup complete., exits zero — and the disk fills up anyway, because the line that was supposed to decide what to delete could never match a single thing. Every run was a no-op wearing a success message. You don't find it by reading the log. The log is the problem: it's lying, and it's lying in green.

The specific shape of it is almost funny once you see it. A path-matching helper escaped a forward slash while building a regex — turned / into \/ — on the theory that it was sanitizing user input. But / isn't a regex metacharacter. Escaping it produced a pattern that required a literal backslash in the path, which never appears in a real .cache/jobs/... path. So the "is this file under the managed directory?" check returned false for everything, forever. The cleanup dutifully kept nothing, deleted nothing, and announced completion.

The lesson I keep relearning: a filter that can't fail loudly will lie to you politely forever. Before you trust any matcher, feed it a known-positive and watch it fire. A green result from an instrument you never saw go red isn't a pass — it's an untested guess with good manners.

I rebuilt it and turned an agent loose on it

Instead of just telling you the story, I did something I don't see enough of: I reconstructed the bug in a throwaway repo — a small cache-janitor with the exact defect planted on purpose — and pointed a coding agent at it cold. No hint about where the bug was, no script, no edited retakes. Just the ops complaint: "cleanup logs success but the disk keeps filling." The bug is staged; the hunt is not.

The whole session is embedded below. Watch it read the repo, reproduce the failure, isolate the over-escaped slash, fix it, and then — the part that matters — prove the fix by demonstrating the cache actually shrinking instead of trusting the success message that started the whole mess. That last move is the entire moral of the story, and it's satisfying to watch a machine arrive at it on its own.

Watch an AI agent hunt a cleanup script that lied about its own success claude-sonnet-5

You

Ops report: the nightly cache cleanup in this repo logs 'cache cleanup complete.' on every run, but the .cache volume keeps growing and it filled the disk twice this month. Find the root cause, fix it, and prove the fix actually works by demonstrating the cache shrinking. You can run 'npm run seed' to reproduce a realistic two-week cache state.

Agent

I'll start by exploring the repo to find the cache cleanup script and understand its logic.

Agent

▸ Bash ls -la && cat package.json 2>/dev/null

Input

$ ls -la && cat package.json 2>/dev/null

Output

total 28
drwxr-xr-x 3 root root 4096 Jul 14 08:13 .
drwxr-xr-x 3 root root 4096 Jul 14 08:14 ..
-rwxr-xr-x 1 root root  453 Jul 14 08:13 README.md
-rwxr-xr-x 1 root root 1117 Jul 14 08:12 janitor.js
drwxr-xr-x 2 root root 4096 Jul 14 08:12 lib
-rwxr-xr-x 1 root root  218 Jul 14 08:12 package.json
-rwxr-xr-x 1 root root 1045 Jul 14 08:12 seed.js
{
  "name": "cache-janitor",
  "version": "1.2.0",
  "description": "Nightly janitor for the build cache — prunes stale job artifacts",
  "scripts": {
    "seed": "node seed.js",
    "clean": "node janitor.js"
  }
}

Agent

▸ Read /work/janitor/janitor.js

Input

/work/janitor/janitor.js

Output

1   // janitor.js — nightly cache cleanup. Reads the job manifest, removes artifact
2   // directories that are (a) under the managed cache prefix and (b) older than MAX_AGE_DAYS.
3   // Runs from cron via `npm run clean`.
4   const fs = require("fs");
5   const path = require("path");
6   const { isUnderPrefix } = require("./lib/match");
7   
8   const MANIFEST = path.join(__dirname, ".cache", "manifest.json");
9   const MANAGED_PREFIX = ".cache/jobs/";
10  const MAX_AGE_DAYS = 7;
11  
12  function main() {
13    if (!fs.existsSync(MANIFEST)) {
14      console.log("cache cleanup complete.");
15      return;
16    }
17    const entries = JSON.parse(fs.readFileSync(MANIFEST, "utf8"));
18    const cutoff = Date.now() - MAX_AGE_DAYS * 24 * 3600 * 1000;
19    const keep = [];
20  
21    for (const entry of entries) {
22      const stale = new Date(entry.createdAt).getTime() < cutoff;
23      if (stale && isUnderPrefix(entry.path, MANAGED_PREFIX)) {
24        fs.rmSync(path.join(__dirname, entry.path), { recursive: true, force: true });
25        continue;
26      }
27      keep.push(entry);
28    }
29  
30    fs.writeFileSync(MANIFEST, JSON.stringify(keep, null, 2));
31    console.log("cache cleanup complete.");
32  }
33  
34  main();
35

Agent

▸ Read /work/janitor/seed.js

Input

/work/janitor/seed.js

Output

1   // seed.js — dev helper: simulate two weeks of nightly job runs so the cache
2   // looks like production (a mix of stale and fresh artifacts + the manifest).
3   const fs = require("fs");
4   const path = require("path");
5   
6   const CACHE = path.join(__dirname, ".cache", "jobs");
7   fs.mkdirSync(CACHE, { recursive: true });
8   
9   const entries = [];
10  const now = Date.now();
11  for (let i = 0; i < 15; i++) {
12    const ageDays = i; // job-0 = today ... job-14 = two weeks old
13    const dir = path.join(CACHE, `job-${String(i).padStart(3, "0")}`);
14    fs.mkdirSync(dir, { recursive: true });
15    fs.writeFileSync(path.join(dir, "artifact.bin"), Buffer.alloc(256 * 1024, i));
16    fs.writeFileSync(path.join(dir, "meta.json"), JSON.stringify({ job: i }));
17    entries.push({
18      path: `.cache/jobs/job-${String(i).padStart(3, "0")}`,
19      createdAt: new Date(now - ageDays * 24 * 3600 * 1000).toISOString(),
20    });
21  }
22  fs.writeFileSync(path.join(__dirname, ".cache", "manifest.json"), JSON.stringify(entries, null, 2));
23  console.log(`seeded ${entries.length} job dirs (ages 0-14 days)`);
24

Agent

▸ Read /work/janitor/README.md

Input

/work/janitor/README.md

Output

1 # cache-janitor
2   
3   Nightly cleanup for the CI build cache. Job artifacts land in `.cache/jobs/<job-id>/`
4   and are tracked in `.cache/manifest.json`. Anything older than 7 days gets pruned.
5   
6   - `npm run seed` — dev helper, fakes two weeks of job artifacts
7   - `npm run clean` — the nightly cron entry
8   
9   ## Known issue (unresolved)
10  
11  Ops keeps reporting that the cache volume fills up even though the nightly
12  cleanup logs `cache cleanup complete.` every run.
13

Agent

▸ Bash ls -la /work/janitor/lib

Input

$ ls -la /work/janitor/lib

Output

total 12
drwxr-xr-x 2 root root 4096 Jul 14 08:12 .
drwxr-xr-x 3 root root 4096 Jul 14 08:13 ..
-rwxr-xr-x 1 root root  532 Jul 14 08:12 match.js

Agent

▸ Read /work/janitor/lib/match.js

Input

/work/janitor/lib/match.js

Output

1   // match.js — path-prefix matching for cleanup targets.
2   // We escape the prefix so user-supplied paths can't inject regex metacharacters,
3   // then anchor it to the start of the candidate path.
4   
5   function pathPrefixToRegex(prefix) {
6     const escaped = prefix
7       .replace(/[.*+?^${}()|[\]\\]/g, "\\$&")
8       .replace(/\//g, "\\\\/");
9     return new RegExp("^" + escaped);
10  }
11  
12  function isUnderPrefix(candidatePath, prefix) {
13    return pathPrefixToRegex(prefix).test(candidatePath);
14  }
15  
16  module.exports = { pathPrefixToRegex, isUnderPrefix };
17

Agent

▸ Bash node -e ' const { isUnderPrefix, pathPrefixToRegex } = require("./lib/match")...

Input

$ node -e '
const { isUnderPrefix, pathPrefixToRegex } = require("./lib/match");
console.log(pathPrefixToRegex(".cache/jobs/"));
console.log(isUnderPrefix(".cache/jobs/job-000", ".cache/jobs/"));
'

Output

/^\.cache\\\/jobs\\\//
false

Agent

Found it. In lib/match.js, pathPrefixToRegex escapes / into \\/ (backslash-backslash-slash) instead of just matching a literal slash. Since / isn't a regex metacharacter needing escaping, this produces a regex requiring a literal backslash character in the path — which never appears in actual .cache/jobs/... paths (forward slashes only). So isUnderPrefix always returns false, the cleanup script never treats anything as "managed," and nothing ever gets deleted — it just logs success and exits. Confirmed with a quick script: isUnderPrefix(".cache/jobs/job-000", ".cache/jobs/") returns false.

Now let's fix it:

Agent

▸ Edit /work/janitor/lib/match.js

Input

/work/janitor/lib/match.js

Output

The file /work/janitor/lib/match.js has been updated successfully. (file state is current in your context — no need to Read it back)

Agent

Now let's verify and demonstrate the fix with the seeded cache data.

Agent

▸ Bash node -e ' const { isUnderPrefix, pathPrefixToRegex } = require("./lib/match")...

Input

$ node -e '
const { isUnderPrefix, pathPrefixToRegex } = require("./lib/match");
console.log(pathPrefixToRegex(".cache/jobs/"));
console.log(isUnderPrefix(".cache/jobs/job-000", ".cache/jobs/"));
'

Output

/^\.cache\/jobs\//
true

Agent

Regex now correctly matches. Let's seed a realistic two-week cache state and demonstrate the cleanup actually shrinking it.

Agent

▸ Bash rm -rf /work/janitor/.cache npm run seed echo "--- before cleanup ---" du -sh...

Input

$ rm -rf /work/janitor/.cache
npm run seed
echo "--- before cleanup ---"
du -sh .cache
ls .cache/jobs | wc -l

Output

> cache-janitor@1.2.0 seed
> node seed.js

seeded 15 job dirs (ages 0-14 days)
--- before cleanup ---
3.9M    .cache
15

Agent

▸ Bash npm run clean echo "--- after cleanup ---" du -sh .cache ls .cache/jobs | wc ...

Input

$ npm run clean
echo "--- after cleanup ---"
du -sh .cache
ls .cache/jobs | wc -l
ls .cache/jobs

Output

> cache-janitor@1.2.0 clean
> node janitor.js

cache cleanup complete.
--- after cleanup ---
1.9M    .cache
7
job-000
job-001
job-002
job-003
job-004
job-005
job-006

Agent

The cache shrank from 3.9M/15 job dirs to 1.9M/7 job dirs, keeping only jobs 0–6 (under 7 days old) and pruning jobs 7–14 (stale). Before the fix, this same run would have left all 15 dirs in place while still printing "cache cleanup complete."

Let me update the README to reflect the resolved issue.

Agent

▸ Edit /work/janitor/README.md

Input

/work/janitor/README.md

Output

The file /work/janitor/README.md has been updated successfully. (file state is current in your context — no need to Read it back)

Agent

Root cause

lib/match.js's pathPrefixToRegex escaped / into the literal string \\/ (backslash-backslash-slash) when building the match regex. Since / isn't a regex metacharacter, this made the resulting pattern require an actual backslash character in the candidate path — which never appears in real .cache/jobs/... paths. So isUnderPrefix() always returned false, janitor.js's stale-check (stale && isUnderPrefix(...)) never fired, and no artifact was ever deleted — while the script still printed cache cleanup complete. and rewrote the manifest unchanged, masking the failure.

Fix

Removed the unnecessary slash-escaping in lib/match.js — only actual regex metacharacters need escaping.

Proof

npm run seed created 15 job dirs (ages 0–14 days), .cache = 3.9M.
npm run clean now correctly prunes everything older than 7 days: .cache dropped to 1.9M, 15 → 7 job dirs (only jobs 0–6 remain).
Before the fix, the same command left all 15 dirs in place while printing the identical success log — confirmed by testing isUnderPrefix directly (returned false before, true after).

Also updated README.md's "Known issue" section to document the resolution.

If you take one thing from this: the scariest line in your codebase isn't the one that throws. It's the one that says everything's fine.