Dev Log: 25 August 2026 — Silent Failures, Double Counting, and Constraints You Can't Defend

Four repos moved today. One is public and got its own post; the rest were private work, so what follows is the reasoning and the patterns, with the specifics filed off. The through-line across all of them turned out to be the same: a system that reports success it hasn't earned.

---

A job that died is not a job in progress

Half a day went into one category of bug on a queue-heavy control plane, and the shape repeats often enough to be worth naming.

You dispatch a job. The job writes a "running" row, does its work, writes a terminal row. Clean. Except a job doesn't only end by finishing — it can throw, it can time out, it can be released and exhaust its tries, or the worker can be killed mid-flight and never come back. In every one of those, the record you wrote at the start still says running, forever.

The UI then does what it's told: shows a spinner. The operator waits on something that has been dead for six hours.

class ProvisionResourceJob implements ShouldQueue
{
    public function handle(): void
    {
        // ... the happy path writes a terminal state
    }

    public function failed(?Throwable $e): void
    {
        $this->run->markFailed($e?->getMessage() ?? 'Job failed without a message.');
    }
}

Three things I'd now treat as non-negotiable for any job that owns a status record:

1. failed() on every job, not on the one that bit you. I found this on one job and fixed one job. Then found eight more with exactly the same gap. If your codebase has a job that owns state, grep for the ones that don't implement failed() — that list is your backlog.
2. A sweeper for the deaths failed() never sees. failed() runs when the queue knows the job failed. A SIGKILLed worker doesn't get that courtesy. A scheduled command that reclassifies "started more than N minutes ago, still not terminal" is the only thing that catches those, and N should come from your actual job timeout, not a round number that felt nice.
3. Don't let the sweeper offer actions the state can't support. A subtle one: the sweeper marked abandoned runs failed and the UI offered Retry. But some of those runs had actually completed their side effect before dying — retrying re-ran a one-time setup. Withholding the action when the target is already in the end state it was reaching for is the fix; "failed" and "safe to retry" are two different questions.

An architectural note: a failed() handler that has to be remembered on each job is a convention, and conventions rot. This is what a trait plus a base class earns its keep on — InteractsWithRunRecord implementing failed() once, and an arch test asserting every job in App\Jobs uses it. Pest can enforce that:

arch('every job records its own failure')
    ->expect('App\Jobs')
    ->toUseTrait(InteractsWithRunRecord::class);

Four placements of one deployment are one application

A different flavour of the same disease: a dashboard confidently reporting numbers that were wrong.

An overview panel was counting things twice, and elsewhere counting four things where there was one. Both bugs came from aggregating on the wrong identity. If a logical application is deployed to four nodes, count() over placements gives you four applications. If two apps share two services, counting the join rows gives you four services.

The fix isn't a smarter query — it's deciding what the noun means before you count it, and then grouping on the key that expresses it. "How many applications do I have" and "how many running processes do I have" are both legitimate questions with different answers, and a panel that doesn't say which one it's answering will eventually be wrong for someone.

Same day, a related one: a value was being read from a place where a different kind of value was being written — a shell error string had ended up stored in a column meant to hold a version number. Every consumer of that column then rendered garbage. A cast, or an enum, or just a value object with a constructor that validates, turns that into an exception at write time instead of nonsense at read time. Wide-open string columns are where these bugs live.

Deploys that cost you a build's worth of downtime

A redeploy was taking the site down for the duration of the build, and logging everyone out on the way through.

Two separate causes worth separating:

• The 502s are a sequencing problem. If the new release is built in place, there's a window where the app is neither the old thing nor the new thing. Build into a fresh release directory, run migrations and warm caches there, then flip the symlink and reload — the flip is atomic and the window disappears.
• The logout is a state-location problem. Anything living inside the release directory — file sessions, file cache — is gone the moment you flip. Sessions belong in Redis or the database, shared across releases. And if you're prefixing cache keys with a release identifier, be deliberate about it: it's a correct way to invalidate stale caches, and a very effective way to log everyone out.

The general rule: a deploy should replace code, and nothing else. Any user-visible state that a deploy resets is state stored in the wrong place.

Unattended automation needs a stop rule

On the membership platform side, the big piece was an engine that runs renewal campaigns without anyone watching. The design constraint I kept returning to: an unattended loop that sends things to real people needs an explicit condition under which it stops on its own.

Not a rate limit — a stop rule. "Halt the campaign if X" where X is something you'd want a human to look at: a bounce rate crossing a threshold, a repeated failure, an unexpected recipient count. Rate limiting makes a runaway slower; a stop rule makes it finite. Without one, the failure mode of automation isn't "it does the wrong thing", it's "it does the wrong thing several thousand times before anyone notices."

The importer that landed alongside it follows the same instinct from the other end: dry-run by default. The destructive mode should be the one you have to ask for. --dry-run as an opt-in flag means every accidental invocation is a real import; making the flag --commit instead means every accident is a report.

public function handle(ImportRecordsAction $import): int
{
    $result = $import($this->file, dryRun: ! $this->option('commit'));

    $this->table(['Row', 'Outcome', 'Reason'], $result->rows());

    return $result->hasErrors() ? self::FAILURE : self::SUCCESS;
}

There's a broader point about modelling rules that come from a written constitution — quorum, notice periods, cooldowns between petitions, who's eligible to vote at what scope. That work is much closer to compiling a specification than to CRUD, and it deserves its own post rather than a paragraph here.

Two small ones that keep recurring

Generate URLs, never concatenate them. A multi-tenant bug came from string-building a tenant link instead of going through the URL generator. Concatenation ignores the scheme, the port, the path prefix, the domain strategy, and every future change to any of them. Route helpers exist so there's exactly one place that knows how a link is shaped.

A quality gate that doesn't fail the build isn't a gate. Static analysis was configured at a level the CI badge advertised, but the command exited non-zero and the workflow had been arranged so nobody noticed. Meanwhile the test suite had one permanently-red test that everyone had learned to read past. Both are the same failure: a signal that always says the same thing carries no information. Either the gate blocks a merge or it's decoration — and a permanently-red test is worse than no test, because it trains the team to ignore red.

---

The thread

Every one of today's bugs was a system reporting something it hadn't verified. A run marked in-progress that was dead. A count that was confidently wrong. A CI badge claiming a level the command didn't reach. And in the public package, a green test suite that was green because Blade never recompiled.

The lesson I'm taking is not "check more carefully". It's that the absence of a failure signal is not a success signal, and the two are easy to confuse because they look identical on a dashboard. Somewhere in your system there's a thing that only ever reports good news. That's the one to go look at.