DEV Community: Nasrul Hazim

Dev Log: 3 September 2026 — Everything I Fixed Today Was Already Written

Nasrul Hazim — Fri, 04 Sep 2026 02:40:37 +0000

Six commits across two codebases today — a deployment platform and an identity system. Different stacks, different problems. Reading them back in one sitting, the same shape six times:

The feature was written. Someone just couldn't reach it.

Not one of these was a missing implementation. The role management existed. The setup-command runner existed. The delivery tracking existed. The "already verified" message existed. All of it shipped, tested, working — and unreachable, because of a default, a branch, a missing entry point, or a token that had already been spent.

That's a genuinely different category of bug from "we haven't built it yet", and it's much harder to see, because every code review of the feature passes. The feature is fine. The path to it isn't.

The badge in the `@else`

Start with the smallest one, because it's the clearest.

A team page shows members and lets an owner change their roles. The component had addMember, changeRole and removeMember all along. But the page read as though role management didn't exist.

The badge naming a member's current role was inside the @else branch of @can('manageMembers').

Sit with what that means. A plain member saw everyone's role and could change nothing. An owner could change any role and never saw what it currently was — and the dropdown offered "Set as Lead", "Set as Member", "Set as Viewer" identically, with no marker on the one already true. The page never named a role to the one person allowed to act on it.

{{-- Before: the person with the power was the one kept in the dark --}}
@can('manageMembers', $team)
    <x-dropdown>...</x-dropdown>
@else
    <x-badge>{{ ucfirst($member->pivot->role) }}</x-badge>
@endcan

{{-- After: the badge is information, not a permission --}}
<x-badge>{{ $roleLabels[$member->pivot->role] }}</x-badge>
@can('manageMembers', $team)
    <x-dropdown>...</x-dropdown>
@endcan

Two follow-ons worth naming:

The label came from ucfirst() on a raw pivot value. Untranslated, and it disagreed with the labels in the dropdown sitting right next to it, which came from the enum's label(). If you've got an enum with label()/color(), that enum is the only place a role should ever be turned into words. I pass a [value => label] map in from render() rather than resolving the enum in the Blade file — an inline @php() holding a nested static call is its own landmine in a component view.

Three of four confirmations were arriving a page late. The toast component lives in the layout and pulls session toast.* keys as it renders. So a session()->flash() from a Livewire action that doesn't redirect surfaces on the next full page load — which, for addMember, removeMember and changeRole, means never at the moment it meant anything. Those dispatch a browser event now. delete() still flashes, because it redirects, and that's what the flash mechanism is actually for.

If you're mixing Livewire actions and a layout-level toast: flash is for redirects, dispatch is for stays. Getting that backwards produces a bug where the user's action appears to do nothing and then a stale confirmation appears somewhere unrelated.

And a third: removeMember() refuses to remove the organisation owner, and reported that refusal via addError('memberEmail') — a field on the add member form. So clicking Remove put a validation error on an input the operator wasn't touching, and said nothing about what actually happened. It's a dispatched error toast now, and the menu item is disabled for the owner instead of offered and then refused. The server-side guard stays, obviously: a disabled control is not an authorisation check.

The runner with no door

Same shape, one level up.

There's an action and an MCP tool for running an application's declared post-install commands. Both have existed for weeks. What didn't exist was any way to reach them from the application's page in the UI.

So the workaround for "I need to run a seeder on this host" had become: push an empty commit to trigger a deploy. A workaround with a commit in it. That's the tell that something's unreachable rather than unbuilt — the workaround is absurd and everyone's fine with it.

The panel I added is mostly wiring, but two decisions in it are worth writing down.

The list of commands is read from the same config the deploy reads. Not a copy. Two lists drift, and the drift doesn't arrive as a diff — it arrives as "it worked when I ran it by hand and failed on deploy", six weeks later, on someone else's shift.

The refusal order matters, and it's the opposite of the obvious one. My deployment policy already refuses while an application is provisioning or being destroyed. So authorizing first handed an operator who is mid-deploy a bare 403 — for something that is not a permission problem at all. The status check runs first now, and says what's actually true:

public function __invoke(RunSetupRequest $request, Application $application): RedirectResponse
{
    // Status BEFORE authorization: a mid-deploy operator is not
    // unauthorized, and telling them so sends them to the wrong person.
    if (! $application->status->canRunSetup()) {
        return back()->with('error', $application->status->setupBlockedReason());
    }

    $this->authorize('runSetup', $application);

    // ...
}

The usual objection to ordering it this way is information disclosure — you've told an unauthenticated-ish caller something about the resource before checking their rights. Worth checking every time, and here it's fine: the page is already tenancy-scoped, so anyone who can read that status could read it from the deployment list beside it. If that weren't true, authorize first and accept the worse error message. That trade-off is real, and it goes the other way often enough that it's worth naming rather than assuming.

Two smaller things:

The output is kept and shown. The reason an operator reached for this is that the app was refusing to start. "Setup completed" with no output leaves them exactly as unable to distinguish a seeded database from a silently skipped seeder as they were before.
The force checkbox carries a sentence, not the word "force". These commands are first-install by nature and mostly not idempotent — a seeder calling User::create() fails the second time, and re-provisioning a tenant database is worse than failing. The label says that.

It runs what the application already declared, in order, once. It is not a shell and not a step toward one. An arbitrary-command surface on a customer's production node is a different feature with a different threat model, and I'd want to design it as one.

The tracking that was complete, and off

Over in the identity system, every message on the mail-history detail screen read "No delivery events recorded" — including the account-verification mail whose delivery we're asked to prove more often than any other.

The package was complete. The defaults weren't. The config declared tracking as env('MAIL_HISTORY_TRACK_OPENS', false), and production's .env carried neither key. So the tracking routes were never registered and no pixel was ever injected. I probed it live before touching anything: both the open and click endpoints answered 404.

// config/mailhistory.php
'track_opens'  => env('MAIL_HISTORY_TRACK_OPENS', true),
'track_clicks' => env('MAIL_HISTORY_TRACK_CLICKS', true),

Flipping a default is a one-line diff and a real decision. There is deliberately no UI switch for it — whether an activation email was opened is not an operator preference, it's an audit question. And defaulting to true means no environment needs an .env edit to be correct, which matters when the environment you most need it in is the one you touch least.

The consequence you have to think about: click rewriting genuinely starts running in production for the first time. So Laravel's signed email-verification URL joins the exclusion list, because its ?expires=...&signature=... is exactly the escaped-ampersand round trip that breaks under rewriting — and a broken signature locks an account out of verifying itself. Turning on a feature that has never run in production is a deploy, not a config change.

Two counting rules that went into the statistics service, both of which are the kind of thing that silently produces a plausible wrong number:

Engagement is counted from the events table, never from status. status holds only the latest state, so a click hides the open that preceded it. Count events, not states.
Count distinct messages, not event rows. One recipient opening four times must not push an open rate past 100%. If a percentage can exceed 100, the denominator is wrong.

Then a set of surfaces on top: summary cards computed inside the Livewire component so they follow the active filters instead of silently contradicting them; an empty state that distinguishes "nobody opened it" from "nothing was ever measured" (they used to render identically, which is how the original bug hid for so long); a report page with period, status and origin breakdowns; and three MCP tools behind the same permission as the screens.

The MCP tools never return a message body or a clicked URL — only the destination host. Verification and reset links live in there, and a read-only reporting tool is not a place to hand them out.

Every surface now reports the tracking state next to the figures. A number whose collection was off for an unknown period is not a number.

The token that was already spent

The best one, and the one with real user pain attached.

A personal-email verification link works once. The token is nulled the moment it's spent — correctly, that's the point. But the lookup is by that token. So a second click on a link that had worked perfectly fell into the Invalid or expired verification link branch. The already_verified branch sitting right below it was unreachable by construction.

$subject = Subject::where('verification_token', $token)->first();

if (! $subject) {
    return $this->invalidLink();   // ← every second click lands here
}

if ($subject->personal_email_verified_at) {
    return $this->alreadyVerified(); // ← unreachable: token is gone by now
}

Users whose accounts had verified flawlessly were being told their account was broken. That's a support ticket, then six more.

Nulling the token stays. What changes is the response:

Post/Redirect/Get. On success it redirects to a landing route carrying the subject id in the session, so a refresh no longer replays a spent token URL. Verification is a state change; it should not be sitting behind a re-runnable GET in someone's history.
An unresolvable token redirects to that same landing page, which reports "Email Already Verified" rather than a red failure. And here's the honest part: once the token is gone, the two cases genuinely cannot be told apart. A never-issued token and a spent one look identical. So the page says what's true for the overwhelming majority, and keeps a quiet "never verified?" hint for the rest. When you can't distinguish two cases, pick the message that's right most of the time and leave a door for the exception — don't show a failure page to be safe.
The mail no longer claims the link "will expire in 24 hours". There's no expiry column, and the verification never checked the token's age. The claim was fiction — and it's precisely what made the old failure page read as plausible. A user clicks their link twice, sees "invalid or expired", remembers the email said 24 hours, and concludes the system is right and they're late. A false reassurance in an email is what turns a confusing error into a believed one.
The two dead failure views are deleted, along with the branch.

The one that wasn't already written

One bug today was mine, fresh, and shipped an hour earlier: a baseline of security headers I'd added to every generated nginx vhost was appending rather than deferring, so applications that set their own X-Frame-Options: DENY were answering with DENY and SAMEORIGIN, which browsers read as neither.

That one got its own post, because the nginx mechanics deserve the space — "nginx add_header appends. It doesn't override." (companion post, link at review time). Short version: add_header can't ask whether the upstream already sent a header, a map on $upstream_http_* can answer before you ask, and a security header added carelessly is a downgrade rather than an addition.

Worth noting how it was found, though, because it fits today's theme from the other side: I went and checked the live response headers right after claiming they were proven. The test suite was green. The suite verified the config generator, which was working exactly as written. Nothing in CI could have caught it, and I'd have believed CI for weeks.

Takeaway

Six bugs, one shape. If I had to write the search query for tomorrow:

Anything inside an @else. Ask who's in the other branch and what they lose.
Any branch that can't be reached given how the state above it is mutated. The already_verified check was dead code that looked like a safety net.
Any env(..., false) default in a config for a feature you assume is on. Then go probe the endpoint in production instead of assuming.
Any action reachable only from an MCP tool, a console command, or a deploy. If the workaround involves an empty commit, the door is missing.
Any two states that render identically. "No events recorded" meaning both nobody engaged and nothing was measured is how an off switch hides for months.

None of these show up as failing tests, because the code under test is correct. They show up when you use the thing as the person it was built for — which is the cheapest verification step available and the one easiest to skip.

My Agent Skills All Said How to Start. None Said When to Stop.

Nasrul Hazim — Fri, 04 Sep 2026 02:37:55 +0000

TL;DR — I read someone else's skill library, went back to my own 32 skills, and found every one of them was a tutorial. They all explained how to begin a task. Not one explained how to know you were finished, or what a wrong turn looks like on the way. Claude Toolkit 2.5.0 adds three sections — Common Rationalizations, Red Flags, Verification — to the nine skills where being wrong is expensive.

The gap

A skill file for a coding agent is a strange document. It's not documentation, because nobody reads it for reference. It's not a prompt, because it's loaded conditionally. It's closest to a playbook you hand a competent contractor on their first day: here's how we do this, here's what we care about, here's what will get you a phone call.

Mine were only ever the first third of that. Take the deploy skill before this release. It knew the deploy command, the SSH flow, the order of operations, how to run migrations. Everything about starting.

It said nothing about the moment that actually matters, which is when the agent — or a tired human — decides the deploy is done. bin/deploy exited 0. Is that success? The skill had no opinion. So the answer became whatever the model felt like, which on a good day is thorough and on a bad day is "the command exited 0, deploy complete."

That's not a model problem. That's a missing section.

Three sections, and why each exists

I added the same three headings to nine skills. Each one targets a different way work goes wrong, and the order matters — they run at different moments.

`## Common Rationalizations` — the arguments you'll make to yourself

A table. Left column: the excuse. Right column: what's actually true.

This is the one I expected to feel patronising and it's turned out to be the most useful. Bad engineering decisions are rarely made in ignorance. They're made with a reason — a sentence that sounds fine when you say it in your head at 6pm. Writing the sentence down next to its rebuttal takes away the "well, in this case…" move, because the case is right there in the table.

From the code-quality skill:

Rationalization	Reality
"Just baseline it, it's legacy code"	A baseline entry is a permanent, invisible ignore. Baseline to unblock CI, then file the issue — a baseline that only ever grows is a broken quality gate.
"Level 5 is good enough for now"	Levels only ever ratchet up when someone forces them. "For now" has been the level for two years in most codebases.
"Pint fixed it, so the code is reviewed"	Pint fixes spacing and ordering. It has no opinion on the god class it just reformatted.
"Adding `@var` fixed the error"	It silenced the error by lying to the analyser. If the annotation is wrong, you have made the codebase less safe, not more.

That first row cost me a real bug this week, in a different repo. Static analysis had flagged two classes as used zero times. Someone baselined the warning rather than deleting the files — and one of those "unused" classes was supposed to be doing PII redaction. The baseline hid it for months.

The rebuttal has to be specific and mechanical, not moral. "Don't be lazy" changes nothing. "A baseline that only ever grows is a broken quality gate" is an argument.

`## Red Flags` — signals visible during the work

A flat list of observable things. Not principles — sightings. If you can see it in a diff, a terminal, or a PR, it belongs here.

From the debugging skill:

sleep() or --retry added to make a test pass
The same bug fixed in three different callers in one PR
A ->skip() or ->markTestIncomplete() added during a bug fix
dd(), dump(), ray() or Log::debug('here') left in the diff
A diff that touches files unrelated to the reported failure
Blaming the framework, the database or the browser before reading your own stack trace

And from deploy-app, where the stakes are higher:

Deploying without a fresh, size-checked backup
migrate:fresh, db:wipe, or --seed typed against a production host
New .env keys added to the repo but never set on the host
A deploy that also contains a refactor
APP_KEY in the diff

Notice the second entry in the debugging list. "The same bug fixed in three different callers" is not a style complaint — it's near-proof that the fix is at the wrong layer. That's the value of the format: a red flag encodes a diagnosis, compressed into something you can spot rather than something you have to reason about.

`## Verification` — the exit gate

A checkbox list run before the work is called done. Not a summary of the skill; a set of conditions that can each be false.

## Verification

Before calling a bug fixed:

- [ ] The original failure reproduces on demand (or its impossibility is documented)
- [ ] The root cause is stated in one sentence, and it is a cause, not a symptom
- [ ] Every caller of the changed function was checked, not just the reported path
- [ ] A regression test exists, and it fails when the fix is reverted
- [ ] The full test suite passes, not just the new test
- [ ] No debug output (`dd`, `dump`, `ray`, `Log::debug`) remains in the diff
- [ ] The diff contains the fix and its test — nothing else
- [ ] If the bug reached production, the blast radius is stated: who was affected, what data, over what window

"A regression test exists, and it fails when the fix is reverted." If I could keep one line out of this entire release, it'd be that one. A test that passes both with and against your fix is a test of something else. Almost nobody checks, because checking means deliberately breaking working code, and it feels like a waste of ninety seconds. It's the only thing that distinguishes a guard from a decoration.

Same energy in the last box. "The blast radius is stated" is not a coding task at all — it's the part of incident work that gets skipped because the code is already fixed and the adrenaline is gone.

Why the end of the file, not the top

The obvious objection: an agent reads the whole skill, so why does placement matter?

It matters because the failure isn't at the entry, it's at the exit. Instructions at the top of the file compete with the user's actual request and lose. A checklist under a ## Verification heading is a thing you can be asked to run — "run the verification section" — and the answer is auditable, box by box. It's a different mode: not guidance, but a gate.

That's the reframe I'd offer anyone writing skills for a coding agent:

A skill is a policy document, not a tutorial. The tutorial part is the cheap part — the model mostly knows the framework. What it doesn't know is your organisation's definition of done, and which shortcuts you've been burned by.

The happy path is where the model is already strong. The failure modes are where it needs you.

Coverage, hygiene, and the boring half

Three other things landed in 2.5.0, and two of them are only interesting as process lessons.

Two agents had no backing skill. The role personas — code reviewer, QA engineer, DevOps engineer and so on — load skills as their playbook. Two were pointing at nothing. So the release adds:

debugging — the five-step triage (reproduce, localise, reduce, fix at the root, guard), a table of Laravel-specific traps (N+1 masked as a timeout, queue jobs failing silently on the wrong connection, config cache serving stale values, Livewire state desync), flaky-test diagnosis, and two reference files.
security-hardening — defensive OWASP-in-Laravel, authorization coverage auditing, uploads, secrets, production config, CI/CD hardening. It's the proactive counterpart to the reactive soc-analyst and log-monitor pair I already had.

Step 1 of that triage is blunter than I'd usually write:

You have not started debugging until the failure happens on demand. … If you cannot reproduce it, say so plainly and stop. A fix for a bug you never saw fail is a guess wearing a diff.

My own convention file was being violated by my own skills. The repo's CLAUDE.md says every skill ships a references/ directory and a Reference Files table. Two skills had neither, and both were orphaned — no agent loaded them. Fixed, and both are now wired to an agent.

That's the unglamorous lesson and it applies well beyond agent skills: a convention nobody checks is a preference. Mine had drifted in the exact way today's other repo drifted, where a hand-maintained SECURITY.md mirroring composer.json had gone wrong in every single entry.

Companions, installed not vendored. Two external tools are now referenced from the skills that benefit — a codebase knowledge-graph tool for research and architecture work, and a minimalism-ladder plugin for the code-review and refactoring skills. Both are installed from upstream, never copied into the repo. Both are optional: the installer reports a skip when prerequisites are missing, and every referencing skill states its fallback, so a missing companion degrades the skill rather than breaking it. New --no-companions flag for anyone who'd rather not.

Copying someone else's tool into your repo is how you end up maintaining a fork of it by accident.

And a smaller thing that changed every response

The default output format is now TLDR + tables, written as a delimited managed block into ~/.claude/CLAUDE.md so it applies to every prompt. Replaced — never duplicated — on reinstall, and the user's own instructions are untouched.

A new /output command switches mode per session: tldr, table, verbose, bullets, narrative, json, raw. Three rules keep it from becoming a footgun:

A one-off instruction in the prompt always beats the session mode. "explain in detail" wins for that reply, then the mode returns.
Never drop required content to satisfy a format. A safety caveat, a failing test, an unmet requirement or a stated assumption gets reported in every mode — tldr and json included.
tldr is not permission to omit; verbose is not permission to pad.

That middle rule is the same idea as the verification section, honestly. A format is allowed to shape the prose. It is not allowed to quietly delete the bad news.

Takeaway

If you're writing skills, playbooks, or CLAUDE.md files for a coding agent, the highest-leverage paragraph you can add isn't another example of the happy path. It's the answer to two questions:

What excuse will be made for cutting this corner, and what's the rebuttal?
What has to be true before this is done — as boxes that can each be false?

The model already knows the framework better than your docs do. What it can't know is which shortcut burned you in March.

34 skills, 20 agents, 12 commands. MIT, and it's all at github.com/nasrulhazim/claude.

Next on the list: making the nine verification sections runnable as an actual gate rather than a prompt the agent is trusted to honour. A checklist you can skip is, by today's standard, a decoration.

nginx add_header Appends. It Doesn't Override — And That Can Remove Protection.

Nasrul Hazim — Fri, 04 Sep 2026 02:35:37 +0000

TL;DR — I shipped a baseline of security headers into every generated nginx vhost, meaning it as a fallback for applications that send none of their own. Within the hour, production was answering with two X-Frame-Options values: DENY from the app and SAMEORIGIN from my baseline. Browsers treat a contradictory set as no directive at all, so a header added to increase protection had removed it. The fix is a map on $upstream_http_*, because add_header has no way to ask a question.

The setup

I maintain a system that generates nginx server blocks for applications it deploys. The applications aren't mine. Some are well-built Laravel apps that set their own security headers in middleware; some are ten-year-old PHP that sets nothing at all and is being moved off a managed hosting panel that used to set headers on its behalf.

So the requirement is easy to state: give every vhost a sane baseline, without stepping on an application that already knows what it wants.

The first cut looked like this, rendered into every server block:

server_tokens off;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

Tests green. Config valid. Deployed.

What production actually answered

I went to check the thing I had just claimed was proven — a habit that has earned its keep — and curl gave me this:

x-content-type-options: nosniff
x-content-type-options: nosniff
x-frame-options: DENY
x-frame-options: SAMEORIGIN
referrer-policy: strict-origin-when-cross-origin
referrer-policy: strict-origin-when-cross-origin

Every header, twice. The application's own middleware was setting X-Frame-Options: DENY. My baseline had added SAMEORIGIN beside it.

Here's the thing I want to be blunt about, because I think it's the part people assume their way past:

Two contradictory X-Frame-Options values is not "the stricter one wins." There is no merge rule. Browsers that receive an unparseable set of directives for that header treat the header as absent. So DENY — the strongest possible value, which the application had correctly set — became nothing, because I added a weaker value next to it.

On that particular host, Content-Security-Policy: frame-ancestors 'none' still covered the gap; CSP takes precedence over X-Frame-Options in every modern browser. But the applications this baseline was written for are exactly the ones relying on X-Frame-Options alone. For all of them, running my "hardening" left them worse off than before it ran.

Three nginx facts worth internalising

The bug wasn't a typo. It was three separate properties of add_header, none of which I had actually sat with.

1. `add_header` appends. Always.

The name is honest and I read it as if it said set_header. It adds a field to the response. It does not replace an existing one, and there is no add_header ... if_not_present flag. Think of it as $response->headers->set() versus ->add() — nginx only gives you ->add().

2. `add_header` cannot ask a question.

This is the real constraint. Inside a location, there is no expression available to add_header that means "unless the upstream already sent this." if won't help you — if inside location is famously not what you want, and it can't inspect response headers anyway, because at request-evaluation time the response doesn't exist yet.

3. `add_header` does not merge across levels — it replaces the whole set.

One add_header in a nested block silently discards every add_header inherited from the parent. Not the matching one. All of them.

This one bit me in a place I would never have looked. My generated config has a fallback location that returns a friendly 503 while an application is starting up:

location @app_unavailable {
    add_header Retry-After 5 always;
    return 503 '<!doctype html>...';
}

That single Retry-After wiped the entire inherited baseline. Which means the response an operator is most likely to see with their own eyes — the "just a moment, starting up" page during a restart — was the one response on the whole node with no protection headers on it. The fix is to repeat the set inside the location, with a comment explaining that the repetition is load-bearing:

location @app_unavailable {
    # Repeated, not inherited: nginx replaces the whole add_header set at the
    # deepest level that defines one, so Retry-After above would otherwise
    # strip the baseline.
    add_header Retry-After 5 always;
    add_header X-Content-Type-Options $fallback_xcto always;
    add_header X-Frame-Options $fallback_xfo always;
    add_header Referrer-Policy $fallback_rp always;
    return 503 '<!doctype html>...';
}

And while we're here: always matters. Without it nginx omits the header from 4xx and 5xx responses — protecting only the responses that were never the risk.

The fix: `map` on `$upstream_http_*`

add_header can't ask a question, but a map can answer one before the question is asked. Two behaviours combine into exactly what I needed:

$upstream_http_<header> holds the value the upstream sent for that header, or the empty string if it sent nothing.
nginx omits an add_header whose value evaluates to an empty string.

Put those together and the fallback writes itself. In http context:

# The baseline security headers are a FALLBACK: each value is empty when the
# application already sent that header, and nginx skips an add_header with an
# empty value.
map $upstream_http_x_content_type_options $fallback_xcto {
    ''      "nosniff";
    default "";
}

map $upstream_http_x_frame_options $fallback_xfo {
    ''      "SAMEORIGIN";
    default "";
}

map $upstream_http_referrer_policy $fallback_rp {
    ''      "strict-origin-when-cross-origin";
    default "";
}

Then the vhost uses the variables instead of literals:

server_tokens off;
add_header X-Content-Type-Options $fallback_xcto always;
add_header X-Frame-Options $fallback_xfo always;
add_header Referrer-Policy $fallback_rp always;

An application that sets its own headers keeps exactly what it set — no duplicate, no contradiction. An application that sets none gets the baseline. That's the behaviour the requirement asked for, and it's what the first cut didn't do.

Two properties of $upstream_http_* that made me comfortable shipping this:

It's populated for fastcgi and uwsgi upstreams too, not just proxy_pass. So a PHP-FPM vhost behaves the same as a reverse-proxied one.
It's empty when nginx serves the file itself, which is the correct answer for a static vhost: nothing upstream spoke, so apply the fallback.

The deployment trap that nearly cost more than the bug

Here's the operational edge case, and it's the one I'd want a reviewer to catch if I'd missed it.

A vhost that references $fallback_xfo when no map defines it is a config nginx refuses outright. Not a warning. Not that vhost failing. nginx -t fails, the reload is rejected, and if a deploy pushes the vhost without the map, you have taken down every site on the node — including the ones you weren't touching.

So the maps and the vhosts have to land together, in all three of the config-writing paths in my system (a systemd driver writing to disk, a container-oriented writer, and a Kubernetes ConfigMap). All three render vhosts from the same shared trait, so all three needed the maps added — the shared trait is exactly what makes it easy to forget, because the change looks like one place.

If you're adding this to an existing setup: deploy the map block first, on its own, and reload. Then deploy the vhosts. A map nothing references is harmless. A reference with no map is a node-wide outage.

What I didn't copy

The panel these applications are migrating from also sends X-XSS-Protection: 1; mode=block. I deliberately left it out. Every current browser has removed the XSS auditor that header names, and on the browsers that kept it, the header introduced a cross-site information leak of its own. Matching the old platform's protection level is the goal; matching its config file is not. Worth writing that distinction into a comment, because "the old system sent it" is a very persuasive argument at review time.

Testing this honestly

The config generator is unit-testable, and the test is worth writing because the shape of the output is the whole contract:

it('renders baseline headers as a fallback, not a second opinion', function () {
    $config = $this->proxy->renderVhost($application);

    expect($config)
        ->toContain('server_tokens off;')
        ->toContain('add_header X-Frame-Options $fallback_xfo always;')
        // The literal is the bug: a hardcoded value cannot defer to the app.
        ->not->toContain('add_header X-Frame-Options "SAMEORIGIN"')
        // Not copied on purpose — the auditor it names no longer exists.
        ->not->toContain('X-XSS-Protection');
});

it('repeats the baseline inside the unavailable location', function () {
    // One add_header at a deeper level discards the entire inherited set,
    // so the 503 page needs its own copy or it ships bare.
    expect($this->proxy->renderVhost($application))
        ->toContain('add_header Retry-After 5 always;')
        ->toContain('add_header X-Frame-Options $fallback_xfo always;');
});

But be clear-eyed about what that proves. It asserts a directive was emitted. It cannot assert what a browser does with the response. The original bug passed a green suite. A test over generated config verifies the generator, not the behaviour — and the difference between "CI-proven" and "live-proven" is exactly the gap this bug lived in for an hour.

The only thing that caught it was hitting the live host and reading the response headers, right after shipping.

Takeaways

add_header appends, can't be conditional, and replaces the entire inherited set at the deepest level that defines one. All three of those will surprise you at least once.
A map on $upstream_http_* plus nginx's empty-value rule is the idiomatic way to express "only if the upstream didn't". No if, no Lua, no duplicated header.
A security header added carelessly can be a downgrade, not an addition. "More headers" isn't a safety property.
Ship the map before the reference. A dangling variable in a vhost fails the whole config, not just that site.
Verify the claim you just made, against the live thing, in the hour you made it. A green suite is evidence about the generator. It is not evidence about the response.

Dev Log: 2 September 2026 — Every Bug Today Was a Claim Nobody Checked

Nasrul Hazim — Fri, 04 Sep 2026 02:34:42 +0000

Twenty-two commits across four repos today. Different stacks, different problems, and — reading them back in one sitting — the same bug wearing four costumes.

Something asserted a state. Nothing verified it. The assertion was wrong, and it had been wrong for a while.

A compliance control was green on evidence that didn't exist. A kitchen screen said there was nothing to cook while paid orders were sitting there. A dashboard reported a 97.8% success rate directly above thirteen failed rows. A service worker claimed to be serving today's CSS and was serving the first CSS it ever saw.

None of these threw. That's the shared trait. A stack trace is a gift — it tells you exactly where to look. A confident wrong number tells you nothing, and it keeps telling you nothing until someone counts by hand.

The control that was green on nothing

Start with the worst one, because it's the cleanest illustration.

A compliance reporter generates control evidence automatically — for each control, a line saying why the system considers it satisfied. One data-minimisation control cited two mechanisms: encrypted casts on the sensitive columns, and a redaction trait applied to the audited models.

The first half was true. The second half was not. The trait existed. It was applied to no model at all. The control had been permanently green on a citation that pointed at nothing.

And it wasn't simply forgotten. As written it could not be applied to the main user model — the audit package it needed to cooperate with declares a method of the same name, and two traits declaring the same method in one class is a fatal composition error in PHP:

class User extends Authenticatable
{
    use Auditable, RedactsPiiInAudit {
        RedactsPiiInAudit::transformAudit insteadof Auditable;
    }
}

That insteadof is doing real work. Without it the class doesn't compile. Descendants of a base class that had one of the two were fine — a trait on the child beats a method inherited through the parent — so the conflict only appeared on the one model that mattered.

Two things I'd generalise from this:

Generated evidence needs a test that the evidence is reachable. A reporter that emits evidence_source: "...trait X" should be able to answer "which classes actually use X?" and fail when the answer is none. Otherwise the report is a string, and strings don't know if they're true.

Configuration on a trait belongs in a method, not a property. A trait and its consumer cannot both declare the same property with different defaults — also fatal. So:

trait RedactsPiiInAudit
{
    // Overridable. A property here would be a fatal clash the first time
    // a consumer wants its own list — and a consumer that forgets to
    // declare one would have no default at all.
    protected function redactedAuditAttributes(): array
    {
        return ['ic_number', 'phone', 'address'];
    }
}

The bonus find, which is the part I'd actually lose sleep over: password hashes were being written into the audit trail. An audit table is immutable by design and retained for a year. A hash written there can't be withdrawn. Masking is the right treatment for PII that has to stay recognisable in an audit — a credential is never in that category, so credentials and 2FA secrets are now excluded outright, not masked.

The same commit deleted three classes referenced from nowhere. Static analysis had flagged two of them as used zero times, and someone had baselined the warning instead of deleting the files — which is its own version of today's theme. A baseline entry is a permanent, invisible ignore.

And a SECURITY.md that hand-mirrored composer.json had drifted in every single entry: it named a framework version two majors behind, and listed four dependencies the project doesn't have. A hand-maintained copy of a lockfile does not stay true. It now points at composer.json and names the audit commands instead. A stale security document is worse than no security document, because it looks like diligence.

Two screens that contradicted themselves

Same shape, two different products.

An audit dashboard rendered its summary cards from the controller using an unfiltered count, while the table below them came from the datatable's filtered query. Filter to "failed, last four days" and you got 1,949 attempts and a 97.8% success rate printed directly above thirteen failed rows. Both halves of one screen, disagreeing — and the reassuring half was the wrong one.

The fix is architectural, not arithmetic: the cards moved inside the Livewire component and are computed from the same baseQuery() that produces the rows.

// Whatever narrows the rows must narrow the summary. One source, two renders.
public function summary(): array
{
    $base = $this->baseQuery();

    return [
        'total'  => (clone $base)->count(),
        'failed' => (clone $base)->where('status', Status::Failed)->count(),
    ];
}

If a number and a list on the same screen come from two different builders, they will eventually disagree. Not might — will.

The rebuilt report also added a period selector, because all-time was the only view and a large intake of successful records pinned the headline near 98% no matter what broke today. A metric with no time window is a metric that can't report a bad day.

Best change in the lot: a "why they failed" breakdown, grouping the recorded error messages and collapsing multi-line database stack traces to their error code. That data had been persisted from day one and never read once. The step tells you where something stopped; only the message tells you why.

A kitchen display screen in a point-of-sale app was the same bug in food form. The display queried for orders whose status was pending or confirmed. But the till takes payment first and cooks after — checkout creates the order, then processes payment, which sets the order to PAID while every line item is still PENDING.

So no order ever taken at the counter reached the kitchen. The screen had only ever shown "No pending orders", which reads exactly like a quiet afternoon.

The insight is a modelling one worth stealing:

Order status tracks money. Item status tracks cooking. A cook is looking at food, so the query has to key on items.

The query now excludes only orders that were killed — cancelled, voided — and keys everything else on item state. A ticket drops off the board once every item is READY.

Two statuses that both say "pending" are not the same status. Naming them for what they measure (PaymentStatus vs PreparationStatus) would have made the bug hard to write in the first place — which is most of the argument for enums over strings.

Caches that were confidently out of date

Two variations, both ending in "works after a hard refresh", which is the single most misleading symptom in web development because it makes the developer the only person who can't reproduce it.

A service worker was matching static assets by pathname alone. In development that meant it also intercepted the dev server's own CSS and JS on a different origin and cached them cache-first — permanently. Cmd+Shift+R bypasses the service worker entirely, which is precisely why only that showed the new styling.

Two fixes, and the second is the general one:

Ignore every cross-origin request. Nothing from another origin belongs in an offline shell.
Cache-first is only safe for content-hashed URLs. Anything at a stable path — /js/app.js, /icons/* — freezes at whatever shipped first, forever. Those moved to stale-while-revalidate: instant from cache, refreshed in the background. Bump the cache name so the frozen entries get evicted on activation.

The other variant: a generated report was styled by a bundle built before the report existed. Same family — a build artifact asserting it was current when it predated the thing it was styling.

While I was in there, a third one, and this is a nasty little Livewire trap:

Uncaught TypeError: $wire.view is not a function

A list/grid toggle introduced a $view property on a component that already had a view() method. On the $wire proxy, a property shadows a method of the same name. PHP is happy. Blade is happy. Static analysis is happy. The only symptom is a click that silently stops working.

The guard is a test, because nothing else in the stack can see it:

it('has no property shadowing a method on any Livewire component', function () {
    foreach (livewire_components() as $class) {
        $properties = collect((new ReflectionClass($class))->getProperties())
            ->map->getName();

        $methods = collect((new ReflectionClass($class))->getMethods())
            ->map->getName();

        expect($properties->intersect($methods))
            ->toBeEmpty("{$class}: property shadows a method of the same name on \$wire");
    }
});

Same day, same repo, a related guard: a missing icon component throws at render time, so only a test that opens that exact page catches it — and a page nobody covers stays broken until a user clicks it. The new test walks every Blade file for icon references and asserts the file exists. That's the right level. Don't test that page 47 renders; test that no page can reference an icon that isn't there.

The one where the bug was blamed outward

My favourite, because I've made this exact mistake and so has everyone reading this.

Four directory attributes came back null on live objects. This was written up as a missing read permission on the service account — a plausible story, an external cause, a ticket for someone else.

It wasn't. The profile screens in the same application display one of those very attributes, from the same directory, on the same bind, with no special grant. The evidence that the story was wrong was already on screen.

The actual defect: the helper normalises the needle to lowercase, but the attribute bag is keyed exactly as the directory returned it. A mixed-case name like pwdLastSet silently missed and returned null. The profile views never hit it because they iterate the raw bag and lowercase both sides of the comparison.

// Lowercasing one side of a comparison is not case-insensitive matching.
$value = collect($object->getAttributes())
    ->first(fn ($v, $k) => strtolower($k) === strtolower($needle));

An earlier "fix" had added an explicit attribute selection — which only narrowed what came back, since an unrestricted query was already returning those attributes. That's the tell for a wrong diagnosis: the fix makes the system do less and the symptom stays.

The part that took the longest wasn't the code. It was going back to correct the audit finding that said the evidence was unobtainable without a permission change. It was obtainable the whole time. If the write-up survives and the diagnosis doesn't, the write-up is the thing people act on next year.

And the same idea, one layer up

The public repo today got a version of this on purpose. My Claude Toolkit shipped 2.5.0, and the substance of the release is that none of its 32 skills said when to stop — they were all "how to start".

Nine high-stakes skills now carry ## Common Rationalizations, ## Red Flags and ## Verification. The verification section is a checklist you run before claiming the work is done, and it's aimed squarely at today's failure mode:

[ ] The root cause is stated in one sentence, and it is a cause, not a symptom
[ ] Every caller of the changed function was checked, not just the reported path
[ ] A regression test exists, and it fails when the fix is reverted

That third box is the whole discipline in one line. Full write-up in the companion post.

Takeaway

The bugs that cost the most today weren't the ones that threw. They were the ones that answered confidently.

A stack trace is a system admitting it doesn't know. A green control, a 97.8%, an empty kitchen queue and a cached stylesheet are systems claiming they do. Only one of those two categories can be found by waiting for it to break.

So the pattern I'd take out of today:

Whenever code emits a claim — a compliance status, a summary count, a "no results", a cached response — ask what would fail if the claim were false. If the answer is "nothing", you don't have a feature. You have a decoration that people will make decisions from.

Every fix today shipped with a guard: a test that fails if the trait goes unused, a test that fails on a property/method collision, a test that fails on a missing icon, a query that can't disagree with the list beneath it. Not because the tests are impressive — most of them are ten lines — but because a claim that nothing can falsify will drift back the moment nobody's watching.

What's next: I want the compliance reporter to refuse to emit an evidence string it can't resolve to a live class. Right now it will happily cite a ghost.

Dev Log: 1 September 2026 — Every Bug Today Was a Name Problem

Nasrul Hazim — Fri, 04 Sep 2026 02:29:29 +0000

Fifty-three commits across eight repos today. Reading them back in one sitting, they sorted themselves into a single shape: something used a name as if it were an identity, and the name wasn't unique.

Two people sharing a username. Two accounts sharing an email. Sixteen environments sharing the word "Production". A class whose name got rewritten under a caller that looks it up by name. Nineteen URLs sharing one document. Different stacks, same bug.

Case-sensitive database, case-insensitive directory

The one that took the longest to explain, and it's four words: = is case-sensitive.

PostgreSQL and Oracle both compare strings case-sensitively. Active Directory, by specification, does not. So a uniqueness gate written as where('username', $input) will happily tell you AliceW is available while alicew exists — and when the downstream sync writes to the directory, both names resolve to the same object.

The gate says yes. The directory says "oh, that one". Two different answers to what everyone assumed was one question.

The fix is a small support class rather than a scope, because it's needed on both Eloquent and query builders in several places:

final class UsernameQuery
{
    public static function whereMatches(Builder|QueryBuilder $query, string $column, ?string $username)
    {
        return $query->whereRaw(self::expression($query, $column), [self::normalize($username)]);
    }

    /**
     * Two usernames collide when they differ only by case or surrounding
     * whitespace — the comparison the directory makes, and the one the
     * gates must make too.
     */
    public static function matches(?string $a, ?string $b): bool
    {
        return self::normalize($a) === self::normalize($b) && self::normalize($a) !== '';
    }

    public static function normalize(?string $username): string
    {
        return mb_strtolower(trim((string) $username));
    }
}

Three decisions in there are worth more than the code:

LOWER(col) = ?, not a case-insensitive LIKE. This is an exact-match gate. Usernames legitimately contain _, and LIKE treats _ as a single-character wildcard — you'd get false collisions on every underscore.

Assert on the generated SQL, not on the returned rows. SQLite folds ASCII case for =. A test that only checks "did I get the row back" passes before and after the fix when it runs on SQLite. The test that actually pins the behaviour asserts the query contains LOWER(.

it('lowercases both sides of the comparison', function () {
    $sql = User::query()->tap(fn ($q) => UsernameQuery::whereMatches($q, 'username', 'AliceW'))->toSql();

    expect($sql)->toContain('LOWER(');
});

Stopping new collisions surfaces none of the old ones. So there's now a sweep that answers the inverse question. Every other lookup in that system is subject-scoped — "who owns X?", and you need X up front. This one is an aggregate over the entire name space, which no existing tool could express.

Two details from writing it that generalise:

The parts are combined with UNION, not UNION ALL, because one person can hold the same name in two different columns. Without the dedup, every ordinary user is reported as colliding with themselves.
Any column with a stored prefix (ext_ and friends) has to have the prefix stripped for grouping, or that whole population silently contributes nothing to the sweep. Strip it with SUBSTR, not LIKE 'ext_%' — _ is a wildcard, remember.

And then the finding I nearly shipped wrong: the first sweep flagged dozens of collisions as live takeovers. Almost none were. The risk label assumed a single directory namespace, and there isn't one — different identity types derive their directory object from different attributes entirely. Two owners can share a name without sharing an object.

Grouping now aggregates in two steps — per (name, type) first — so the outer query can distinguish "two of the same type share this" from "two different types do". The ordering keys on that signal too, because sorting on the raw count buried the one case worth acting on beneath thirty that weren't collisions at all.

A risk score that ranks noise above signal is worse than no score. You'll read the top of the list and stop.

A picker that collected a choice nobody consumed

Different system, same species.

A personal email address is not unique in most institutional databases — a person who progresses through two programmes keeps the same email on two ids. So the password reset flow shows a picker: here are your accounts, choose one.

The picker was correct. It rendered the accounts, it validated the selection, it carried it through the form. And then the resolver ran:

$row = IdmUser::whereEmailAlt($email)->first();   // by email. unordered.

The selection was collected and then ignored. Resets reported success, wrote to a stale id, and the live account was never touched — so the directory's pwdLastSet never moved and the client read it as "your reset isn't doing anything".

What makes this one worth writing down isn't the bug, it's the history. This was logged and closed a year ago. The fix that closed it shipped the picker — the UI half — and was marked Resolved. No test covered the multi-account path, so the incomplete fix shipped green and the gap survived twelve months behind a ticket marked done.

Four changes went in, and only the first is the fix:

// 1. Scope on the selected username AND the email, not the email alone.
UserCategory::whereUsernameAndEmailAlt($username, $email);

// 2. Backstop: abort and log before any backend write if the resolved
//    identity doesn't match the one that was selected.
abort_unless(UsernameQuery::matches($resolved->username, $selected), 409);

The picker now labels each account with its directory status, so two bare ids are actually distinguishable to the human choosing between them.
The success screen names the account that was written. A bare "Success" is what hid this for a year — from the user and from support. If an operation picks one of several candidates, the confirmation has to say which one it picked. That's not UX polish; it's the only observability that costs nothing.

There's a diagnostics footnote too. The first report masked emails with one asterisk per character, so two unrelated people whose local parts shared their opening letters and length printed identically — the report read as one group where there were two. The counts were never wrong; the rendering was ambiguous. It now carries a stable group ref (first 8 hex of a sha256 over the lowercased address) and a fixed-width mask, because per-character masking leaks the local part's length for no benefit at all.

An identifier in a report needs to be unambiguous and stable across runs. A mask is neither.

"Confirm by typing the name" only works if the name is unique

Shipped a set of destructive MCP tools today — delete a project, an environment, a provider; retire a machine — and the guard pattern is the usual one: the caller must retype an identifier to confirm.

Except sixteen of the seventeen environments on the live control plane are called Production.

Retyping a name that sixteen records share confirms nothing an agent couldn't have guessed. So the confirmation is on the slug, and for a machine it's on the SSH host address rather than the hostname, which can be blank or shared.

The pattern, as a trait:

trait RequiresTypedConfirmation
{
    protected function confirmOrFail(string $argument, string $expected, string $noun): void
    {
        // The argument key is passed in so a refusal is audited identically
        // whichever tool refused.
    }
}

Two other things went in the response text of every one of those tools, both because they're ways an agent reports a teardown wrongly:

Deleting a provider removes the credential, not the host. An adopted machine keeps running and keeps costing money until its own hosting provider destroys it.
The deletion is soft. "Deleted" in a tool response and "deleted" in a database are not the same claim, and an agent will relay the first as if it were the second.

If you're building MCP tools, that second one deserves a rule of its own: say what the tool did not do. An LLM will fill any silence with the most obvious inference, and for a delete tool the obvious inference is "the thing is gone".

There was also a dependency rule to move. The "can this be deleted?" check lived as a private method on a Livewire component, with a comment from its author warning it would be forgotten at the fifth call site. This was the fifth. It's now a RecordDeletionGuard service asked by both the UI and the tool — and extracting it exposed that the rule was also wrong: it counted every deployment row including destroyed ones, so the final step of a teardown was refused forever by the very deployments the teardown had just destroyed.

Related, same day: a "has history" rule that was blocking deletes. History is not a dependency. Every provider anyone ever bootstrapped accumulates job rows, so the rule read, in practice, "you may delete a provider only if you never used it". The rows are kept on purpose, the parent is soft-deleted anyway, and they still resolve through withTrashed(). Dropped.

A relation cached before the rows existed

This one is pure Laravel, and it's the sharpest edge I hit today.

A 24-step provisioning pipeline carries one Deployment model instance through every step in a context object. Step 11 reads $deployment->routingRules — as a property. That loads the relation and caches it on the instance. At step 11 it's empty, because the rules are written at step 16.

Steps 20 and 21 then read the same cached, empty collection, and took the "an empty set is an honest success" branch. So DNS and SSL both reported success on a deployment that finished 24/24 Active with no DNS record and no certificate. A second run worked, because it got a fresh instance.

// Cached at first access. Every later read on this instance sees step 11's answer.
$rules = $deployment->routingRules;

// Re-queries. In a long-lived pipeline this is the one you want.
$rules = $deployment->routingRules()->get();

Everyone knows relation access is cached. The part that bites is that it's cached on the instance, and in a request/response app instances don't live long enough for it to matter. Move that same model into a pipeline, a queued job chain, or a long-running worker and the lifetime changes underneath you without the code changing at all.

The general rule I'm taking from it: if a model outlives the thing that loaded its relations, treat property access as a snapshot, not a query. And be suspicious of any branch that reads "empty set → success". Empty is ambiguous. It means both "nothing to do" and "I looked at the wrong moment", and only one of those is a success.

A coverage test that could only see what today's fixtures happened to draw

Spent a chunk of the day on localisation — making English the key language, with the second language selectable rather than forced by an org-level setting.

I'd added a translation-coverage test the day before. It reported clean. Dozens of public-facing keys were untranslated.

The test was render-driven: render the pages, ask Laravel which keys were looked up and not found. Reasonable design. But the local seed had no priced membership tiers, so the tier cards never rendered, their keys were never looked up, and the test was satisfied.

A render-driven test is only as good as your fixtures. It can only fail on the paths your seed data happens to reach.

The fix is not to replace it but to put a second pass beside it:

Source-driven: every __() key found in the public view tree must have a translation, whether or not anything renders it today.
Render-driven: kept, because it still catches strings emitted from outside that tree — a component library's own labels, for instance.

Neither alone is sufficient, and the two failure modes are exactly complementary.

Then a third one that neither pass can catch. Page titles were being set as literal view attributes:

<x-layouts.auth.card title="Apply for membership">

Never a key at all. The source scan looks for __(; the render pass needs a key to be looked up. A string that was never a key is invisible to both. Found it by opening the page.

Two lessons in one afternoon: automated coverage has a shape, and things outside that shape are not merely uncovered — they're reported as covered. And the most important page in a funnel is the one most worth loading manually before you trust a green test.

The build tool renamed a class the native layer looks up by name

The mobile side of the day. An app was dying mid-match — no Dart exception, no error screen, the process just vanished.

R8 (Android's shrinker/obfuscator) had renamed most of an ML runtime's Java classes down to two-letter names, the way it's supposed to. That runtime's native layer resolves those classes by their original names through JNI, so the first inference hit a null class in GetMethodID and the Android runtime aborted the process.

The rule R8 can't know: a class referenced only from native code has no Java-side reference to keep it, so as far as the shrinker is concerned nothing is using that name. Keeping it takes an explicit -keep — the same reason reflection needs one.

A crash with no stack trace in your language is almost always a layer boundary. JNI, FFI, a native plugin — something on the other side of a boundary that your exception handler doesn't cross.

And it's the name problem again, in the most literal form of the day: a tool rewrote a name, and something else was still holding the old one.

Sibling fix in the same release: two macOS crash reports were SIGSEGVs inside the bundled SQLite framework, and one of them showed the framework mapped twice — same UUID, two base addresses. Two independent copies of SQLite live in one process, each with its own global config, handles crossing between them. The database was being hosted on a spawned background isolate with handles sent across the boundary; opening it on one isolate removes the surface entirely.

Same name, two live things. Again.

And nineteen URLs sharing one document

The public one, and the reason it's the whole other post: devhub.my was serving the identical index.html for every path, at HTTP 200, because of the standard SPA catch-all rule. An SEO crawl reported five pages; the sitemap listed nineteen. Both were correct.

That's now a prerender step that writes a real HTML document per route and fails the build on three things no validator catches — a page that never declared its head, a JSON-LD graph with a dangling @id, and a canonical pointing somewhere the page isn't. Full write-up in the companion post; repo is developers-hub-my/website if you want to read the scripts.

Also, briefly

A framework upgrade plus a design-system pass on one product: Laravel 13 / PHP 8.5, a starter kit dragged forward across thirty-odd minor versions, unified design tokens, and a shared list pattern rolled out screen by screen with one reference screen built first. Building the reference screen properly and then porting is slower on screen one and much faster on screens two through twelve.
Stale published assets after a package update. If your framework has a "publish these vendor assets" step, it has a "these are now out of date and nobody noticed" state. Wire the republish into the update path rather than the install path.
Migration planning as documentation, not as a ticket: what the current platform actually does, observed rather than assumed, before deciding what to replace it with. The probe that produced the observations got committed alongside the notes, which is the part I'd have skipped a year ago.

Takeaway

If today had a single rule in it, it's this: a name is not an identity. It's a label that happens to be unique in the sample you looked at.

Every bug above is what happens when code treats a label as a key — and every fix is the same move in a different dialect: normalise before comparing, confirm on something actually unique, name the thing you acted on in the response, and check the assumption at the layer that will be surprised, not the one that's convenient.

The cheap tell: any time you're about to compare, group, or confirm on a human-readable string, ask who else could hold that string. If the answer isn't "nobody, by construction", you have a bug waiting for a coincidence.

Dev Log: 31 August 2026 — Loaded Is Not Serving

Nasrul Hazim — Fri, 04 Sep 2026 02:05:05 +0000

Thirty-eight commits across five repos today. Reading them back, most were the same species of bug: two different things that a system had been treating as one word. Loaded and serving. Both-S3 and copyable. Present and correct. Every one of them was cheap to fix and expensive to find.

Loaded is not serving

Found in production, and my favourite bug of the day.

A site was answering 200 to every request. The dashboard showed the workload as failed, with its last release green and its deployment active. The journal was clean — php-fpm up, migrations run, no errors anywhere. A red workload sitting next to a perfectly healthy site, which is exactly the thing that sends somebody to go and fix a deployment that is fine.

The blue-green runtime picks the "active" colour like this:

foreach ([SLOT_BLUE, SLOT_GREEN] as $slot) {
    if ($this->unitLoaded($slot)) {
        return $slot;
    }
}

Here's the thing: systemd keeps a unit loaded after it dies. systemctl show on a stopped-but-known unit still reports LoadState=loaded. So a retired blue outranked the green that was actually serving every request, the status probe went and asked blue how it was doing, blue said ActiveState=failed, and the UI rendered red.

The quieter half of the same bug: standbySlot() is the inverse of that answer. With blue permanently "active", every release went to green. Two consecutive production releases landed on the same colour — blue-green alternating had silently stopped, which defeats the entire arrangement.

The fix reads both properties in one round trip and lets an active unit win outright, even though blue is examined first:

$state = $this->unitState($slot);   // LoadState + ActiveState, one call

if ($state->isRunning()) {           // active or activating
    return $slot;
}

$loadedFallback ??= $slot;           // only used if NEITHER is running

The fallback is deliberate — it's what keeps a genuinely stopped workload reporting Stopped rather than Pending. And the test that scripted the old single-property probe was updated rather than deleted, because it now expresses serving instead of merely loaded, which is the distinction the whole bug turned on.

Both endpoints being S3 is not "can copy server-side"

While wiring bring-your-own object storage, I hit a variant of the same mistake in my own contract design.

CopyObject in the S3 API is served by the endpoint you send it to. That endpoint cannot reach into a different service's buckets. So "source is S3 and destination is S3" does not imply a server-side copy is available — same endpoint does.

The awkward part is that the implementation is resolved per driver, not per row. An S3 destination asked "can you server-side copy from this?" genuinely cannot tell which of its own configured endpoints it's being asked about. So the method grew a parameter:

public function supportsServerSideCopyFrom(
    BackupDestination $destination,
    BackupArtefact $source,
): bool;

PHPStan caught the arity break across one call site and two test doubles, which is the cheapest possible way to find out you changed a contract.

The consequence is worth stating out loud: cross-endpoint S3-to-S3 has to stream, which makes the streaming path the common case rather than the rare one. That promotes the transfer size ceiling from a defensive guard to a load-bearing part of the design — and it's why the byte counter counts bytes through the control plane, so a genuine server-side copy correctly records zero.

A version that is a fact beats a version that is a guess

A one-click Drupal install on an AlmaLinux 9 node bootstrapped, built, deployed, passed the health gate — and then died at step 18 with:

The database server version 13.23 is less than the minimum required version 16.

EL9's AppStream defaults the postgresql module to 13. Drupal 11 pins a minimum of 16. Those two can never agree, which means that recipe could not have installed on any RHEL-family node, ever — and the cost of learning it was a full composer build on a 1 vCPU box, reported on a step whose name points nowhere near a package stream.

Two fixes, because either one alone leaves the failure sitting there.

Install a server modern apps accept. A per-OS hook runs before a service's packages go on. Empty on Debian, whose postgresql metapackage tracks the release's own server. On RHEL it switches the module stream, guarded three ways, each load-bearing: only when 16 actually exists in that node's AppStream, so an older minor keeps working; module reset first, because enabling a second stream over an enabled one is an error rather than a switch; and only from the "no server installed yet" branch, because a stream switch does not upgrade a cluster — it would leave the data on 13 and the packages on 16, which is worse than either.

Refuse early when it still can't be satisfied. A recipe declares its minimum engine versions and a guard checks them at the end of the provisioning step — not in preflight, and this is the part worth arguing about.

Preflight has no database yet. The only thing available there is a guess from a table of distribution defaults: a table that goes stale silently and is wrong the moment somebody installs a version by hand. Six steps later, the version is a fact — the provisioner has read SHOW server_version off the running daemon. Still ten steps earlier than the failure it replaces, and it cannot be wrong.

Two details from the same guard:

An engine that isn't in the deployment at all is unconstrained, not forbidden. The app might use SQLite, or a database this platform never provisioned.
version_compare, not string comparison. '9.6' sorts after '16' as text, which is how a naive check waves through an engine seven majors too old.

A hostname nobody typed

An application with no domain isn't merely unrouted — on a plain VM it's usually undeployable. With no domain the runtime drops off fpm serving and onto the recipe's php -S 0.0.0.0:${PORT} fallback, on the port the catalogue declared for the container image, and on any node already serving a domain, nginx is holding 80.

Cloning is what makes this the common case rather than an edge case. Cloning strips every domain, correctly — a hostname is served by exactly one deployment — so the copy of a working site arrives in precisely the state that can't run, and the operator is asked to invent a hostname before they can see whether the copy even works.

So a preview hostname gets issued rather than typed:

{workload-slug}-{deployment-suffix}.{preview-base}

The deployment's own random suffix carries the uniqueness, and that's the entire reason it's in the label. A scheme that can collide needs a retry loop, a "name taken" error, and a human to resolve it — for a name nobody asked for in the first place. The workload slug is in there because one deployment may attach more than one HTTP application.

The nice part: it's almost all wiring that already existed. One new step writes the name onto the routing rule; the reverse-proxy, DNS and SSL steps then do exactly what they already do for a hostname somebody typed, and none of them needs to know where it came from.

Who owns the decision — a menu refactor that's really an access refactor

On a multi-tenant product I work on, the Administration section had quietly become where anything unclassifiable landed: 22 items, six sub-groups, three levels deep, mixing a tenant's own settings with the vendor's diagnostics. A committee secretary was being shown Telescope, Horizon, an Artisan runner and an MCP console — tools they will never touch and shouldn't see.

The dividing line I took wasn't what kind of screen is this. It was who owns the decision. Vendor-owned tooling moved into its own section, invisible to a tenant entirely. Tenant-owned administration stayed and flattened — single destinations moved to the top level instead of each sitting inside a sub-group that costs a click to reveal one item. Net effect on a tenant's sidebar: 22 items to 9, three levels to two.

One implementation note that generalises. The new section gates on a landlord-level flag rather than a permission, for the same reason the platform routes do: these screens run with no tenant current, and a permission check then has no table to consult. It throws rather than returning false.

And the test that had checked the moved item is now stronger than before — it reads the new section and short-circuits on that section's own authorisation, so the item can't leak through a per-item gate even if somebody adds one.

The rest, briefly

Re-pricing moved out of a Livewire component and into an action. Two callers now share one definition of what re-pricing means — a form and an MCP tool. If the tool had copied the "retire the active version, cut the next one" logic, there'd be two definitions free to drift, and the thing that drifts is what people get charged. The write tool also carries forward anything the caller didn't name rather than defaulting to zero, so an agent asked to raise one number can't silently wipe the parameters sitting beside it.

A statistics dashboard that couldn't reach its own rows. A mail-history package renders eight status counts and a banner reading "N messages stuck in Sending for over 1 hour" — with no way to reach those N. It can't have a list, because the table has no recipient or subject column; both live inside a JSON headers blob, which is why status is the only thing it can aggregate. Fix: add the two columns, fill them on saving in a model that extends the package's, and backfill via a deploy operation that's idempotent and deliberately does not bump updated_at — the stuck-message window measures from it, so a backfill mustn't make a stuck message look fresh.

Gateway credentials got a UI, and a rule. A stored secret is never sent back to the browser; the form shows only whether one is set, and a blank field on save means "leave it alone" — so saving the page to change a URL can't silently blank a credential. Switching to a gateway with empty credentials is refused outright, because allowing it means every checkout throws and the operator hears about it from a customer instead of from that screen.

A Livewire trap worth writing down: wire:model="settings.payment.stripe.secret" doesn't bind one value — Livewire reads each dot as nesting and builds a three-level array. It fails as "Array to string conversion", which names nothing useful. Flat field names mapped to dotted storage keys through an explicit fieldMap().

A sandbox route with no guard. A pay-by-button-press endpoint that marks an invoice paid with no money involved — registered unconditionally by a glob-based route loader, checking nothing but that the invoice exists. That's fine in a sandbox and very much not fine anywhere else.

And one genuinely small thing: a mobile app version bump for optional player accounts and Google Sign-In, plus its changelog entry and two new BSD-3 licence notices. Third-party notices are one of those chores that's trivial the day you do it and awful the day you're asked for them at once.

The takeaway

Every bug in the first half of today was a word doing two jobs. loaded meant both "systemd knows about this" and "this is serving traffic". s3 meant both "speaks the protocol" and "can copy from that". A distribution default meant both "probably installed" and "definitely installed".

They're hard to spot because the collapsed version reads fine — it's usually the shorter, more natural sentence. The tell is when a screen and reality disagree and nobody can immediately say which one is wrong. That's not a display bug. That's two concepts sharing a name, and the name picked the wrong one.

The Most Dangerous Backup Is the One That Reports Success

Nasrul Hazim — Fri, 04 Sep 2026 02:05:00 +0000

TL;DR — I spent today building out a backup subsystem, and almost every design decision that mattered came down to the same question: can this thing fail in a way that still renders as a green tick? A backup that fails loudly is an annoyance. A backup that fails quietly is the reason the restore doesn't exist.

The failure mode nobody designs for

Backups have an unusual property among features: the moment you find out whether yours works is the worst possible moment to find out. Everything in between is inference. And inference is exactly where a system gets to lie to you politely.

So the rule I kept coming back to today: a component may not report a success it did not earn. Not "should not" — may not, structurally, because the code path that would produce the false green isn't there.

Here's what that looked like in practice.

Sync is not backup

The first one is the one people get wrong most confidently, because the tool that does it wrong is excellent at its job.

A sync tool mirrors a source into a destination. If a key disappears from the source, it disappears from the destination. That is correct behaviour for a mirror and catastrophic behaviour for a backup, because the thing you are backing up against is somebody deleting the wrong folder at 2am — and a mirror propagates that deletion to the only copy that could have undone it, usually within the hour, and then reports success.

So the bucket driver I wrote today records deletions and never replays them:

/**
 * A key that vanished at the source is marked deleted in the manifest.
 * Its object is LEFT WHERE IT IS until retention expires it.
 * Nothing in this driver deletes an object at a destination.
 */

Two tests pin the pair of consequences, and the second one matters as much as the first:

an object deleted at the source is still restorable from the last run that had it;
restoring a later run does not resurrect it.

That second test is the one people forget. A restore reproduces the bucket as of that run. An operator recovering one folder did not also ask to undo every intentional deletion since — and if your restore does that, you've handed them a second incident while they were cleaning up the first.

"Move" is copy, verify, then delete — in that order, always

Moving a backup between destinations is the most obviously dangerous operation in the whole subsystem, because there's a window where the artefact exists in exactly one place and something is actively trying to remove it.

The ordering rule is boring. What's interesting is making the rule checkable:

// source_deleted_at stays null until the destination copy is Verified,
// and is written in its own statement, after verified_at.

Two separate writes, deliberately. If they went out together you could argue forever about whether a bad row was a race or a bug. Kept apart, a query can find the violation:

test('a transfer never deletes the source before the destination verifies', function () {
    expect(BackupTransfer::deletedSourceBeforeVerification())->toBeEmpty();
});

The other half: the destination copy gets its own artefact row, with its own key, size, checksum and verification state — and the driver is asked to verify that row. A move that verified the source's checksum would have proved nothing about the transfer it existed to prove. It would have been a green tick describing the wrong object.

Verification that admits what it didn't check

This is my favourite one, because the honest version is uglier than the dishonest version and that's the whole point.

Verifying a bucket copy properly means re-hashing every object. To do that, the control plane has to read every object back through itself — during the verification of a transfer whose entire purpose was to keep that data out of the control plane. You can't have both.

The dishonest option is to check what's cheap and call it verified. What shipped instead: check object count, total size, and a spread sample of per-object sizes — and then say so.

'verification' => [
    'objects_checked' => 1_284,
    'bytes_expected'  => 4_118_233_712,
    'etags_rehashed'  => false,
],

etags_rehashed: false sits in the record permanently. A passing verification cannot be read, later, by somebody who wasn't there, as proving something it never checked. The same instinct produced a test that scans every operator-visible string in the feature for the words "point-in-time", "versioning" and "snapshot" — because the driver doesn't do those, and prose that merely declines to claim something is not the same as prose that states the opposite. Somebody reading the screen in two years will assume whatever the words allow.

Integrity is not restorability

Every run checks that the artefact is present, the right size, the right checksum, and that it opens. All cheap, all worth having, and none of them prove it restores.

An archive that passes pg_restore --list and then dies part-way through a real restore — a missing extension, an incompatible server version, an encoding mismatch — is indistinguishable from a good one right up until 3am.

So: restore drills. A policy can opt into verify_level = restorability, on its own cadence and its own cursor (drill_cron, next_drill_at — a nightly backup does not want a nightly drill; a drill costs a real target and a real restore).

Two design calls in there I'd defend anywhere:

The drill uses the ordinary restore path. Not a drill-specific one. A drill with its own code path proves the drill works, which is the one thing it was never in doubt about. It builds a restore record in new-target mode and hands it to the same restore() an operator would trigger.

A failed drill flags the policy; it never condemns the artefact. Tempting to auto-demote a backup whose drill failed. That is precisely how your last good copy disappears on the strength of a drill-environment problem. The artefact keeps its verified status. The policy goes red. There's a test pinning that, because it's the kind of rule a future refactor would "fix".

And a bonus the drill gave me for free: restore time is a number nothing else in the system ever measures. An operator planning around a backup needs to know if they're committing to ten minutes or six hours. There's no honest way to find out except doing it. So restore time is recorded from the drill's own clock, or from a real restore's — and where neither exists, the screen says "never measured". Never an estimate. An estimate here is a fabricated green tick wearing a number.

Teardown, incidentally, goes in a finally and is best-effort silent. A teardown error must not replace the real reason a drill failed. And a drill that leaves its target behind fills the server with drill databases — a backup feature causing the outage is a bad look.

The quiet ones: pagination and fakes

Two smaller things, same family.

ListObjectsV2 returns 1000 keys per call. Back up the alphabetical first thousand objects of every bucket, report success, and you have built the worst possible version of this feature — the one that is indistinguishable from the correct one until somebody needs a file starting with "z". Paginate, and pin it with a test that seeds more than a thousand.

A fake is handed out only with a reason. The resolver that picks an object-store client can fall back to a no-op in environments that have no daemon. It never does that silently:

$store = $resolver->for($destination);

if ($store->isFake()) {
    return BackupOutcome::refused($store->reason());
}

The driver turns that reason into a refusal. A silent fake records a beautifully successful backup of nothing — which is, again, the same bug: a green tick with no work behind it.

The incremental baseline has to be verified, not just complete

Objects are compared on (key, etag, size) against the last run that both completed and has a verified copy.

Seeding from a merely-completed run inherits its gaps silently and forever — every future incremental agrees with a baseline that was already wrong, and the gap only surfaces during the restore that needed it.

lastModified is deliberately excluded from the identity triple. A re-uploaded object gets a new timestamp with identical bytes, and comparing on a clock turns an incremental quietly back into a full — the opposite failure, but still the system being wrong about its own work.

The takeaway

Every one of these is the same shape. Somewhere there's a path where the system can report a success it didn't earn, and the fix is never "add a check" — it's usually to make the dishonest path impossible to express:

delete-after-verify, in separate writes, so the violation is queryable rather than debatable;
verification that records what it didn't do;
a drill that shares the production restore path;
a fake that must carry a reason;
a baseline that must be verified, not merely finished.

If you're building anything in this space, the question I'd start with isn't "does it work?" It's "if this quietly stopped working, how long until anyone noticed, and what would the screen say in the meantime?"

If the answer to the second half is "success", that's the bug. Everything else is detail.

Next up on my side: the restore-source selector — when several verified copies exist, picking the cheapest one to restore from is its own small pile of trade-offs.

A Symlink Swap Is Not a Zero-Downtime Deploy

Nasrul Hazim — Mon, 31 Aug 2026 01:14:00 +0000

TL;DR — I had Capistrano-style releases on a plain VM: build into releases/<ref>, swap a symlink, restart the service. Atomic, I thought. Then I put curl in a loop and watched an actual deploy: 502 on every single one, for the length of the app's boot. The files were atomic. The process never was. And the nginx directive everyone reaches for to paper over that does absolutely nothing in the shape most people write it.

The half that was already right

The release layout is the well-trodden one, and there's nothing wrong with it:

/srv/app/
  releases/20260831-101500-a1b2c3d/
  releases/20260830-093000-9f8e7d6/
  current -> releases/20260831-101500-a1b2c3d

Build into a fresh directory, then move the pointer. One detail worth getting right, because two of the three obvious ways to write it are broken:

# WRONG — `-sf` follows an existing symlink-to-a-directory and
# quietly creates /srv/app/current/20260831-101500-a1b2c3d
ln -sf "$RELEASE" /srv/app/current

# STILL WRONG — `-n` fixes that, but this is unlink() then symlink().
# There is a window where the path resolves to nothing.
ln -sfn "$RELEASE" /srv/app/current

# RIGHT — create beside, then rename(2) over the top. Atomic.
ln -sfn "$RELEASE" /srv/app/current.tmp
mv -Tf /srv/app/current.tmp /srv/app/current

mv needs -T for the same reason ln needed -n: without it, moving onto an existing symlink-to-a-directory moves into it.

Get that right and your files switch instantly. A request that started on the old release finishes against the old tree; the next one gets the new one. Genuinely atomic, genuinely nice.

Which is exactly why the next line goes unexamined for months.

The half nobody looks at

systemctl restart app

restart is stop, then start. Between those two, nothing is listening on your app's port.

t+0.00  mv -Tf current            files switch, atomic ✓
t+0.01  systemctl stop app        port closes
t+0.01  GET /  →  502
t+0.4   GET /  →  502
t+1.2   GET /  →  502
t+2.8   app finished booting, binds port
t+2.9   GET /  →  200

The deploy is atomic right up to step 3. Everything red after it is the part the symlink swap never covered.

Three seconds of hard 502s, on every deploy, recovering by itself. That last part is why it survives so long: nobody files a bug for something that fixed itself before they could screenshot it. It gets called "flaky."

It isn't flaky. It's exactly what you told the machine to do.

Two shapes, two different 502s

If you serve a process on a port, it's the obvious one:

location / {
    proxy_pass http://127.0.0.1:3000;
}

Process dies → port closes → connect() refused → nginx has nothing to say but 502.

The second shape catches people out, and it caught me. Conventional PHP on a VM: nginx owns the docroot and only .php reaches the app.

root /srv/app/current/public;

location ~ \.php$ {
    include fastcgi_params;
    fastcgi_pass unix:/run/app/php-fpm.sock;
    fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
}

The reasoning goes: nginx serves the files itself, so a restart can't matter much. It matters completely — if the fpm master runs under the same systemd unit as the app. Which is often exactly how you want it, because then logs, resource limits and lifecycle all work per-app instead of being shared through one system-wide pool.

The cost of that choice: restarting the unit kills the master, the socket file is unlinked, and fastcgi_pass gets ENOENT until it comes back. Same 502, different mechanism, and the static assets keep serving perfectly the whole time — which makes it look like an application bug rather than a deploy bug.

The directive that does nothing

Everyone's first fix, mine included:

location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_next_upstream error timeout;   # ← inert
}

This changes nothing. proxy_next_upstream retries the next peer in the upstream group. A proxy_pass at a literal host:port is a group of one. There is no next peer, so there is nothing to retry to, and you get the same 502 you got before — now with a config line that makes you think you handled it.

The version that works looks like a typo:

upstream app {
    server 127.0.0.1:3000 max_fails=0;
    server 127.0.0.1:3000 max_fails=0;   # yes, the same address twice
}

location / {
    proxy_pass http://app;
    proxy_next_upstream error timeout invalid_header non_idempotent;
    proxy_next_upstream_tries 3;
    proxy_connect_timeout 2s;
}

Listing the address twice is the entire mechanism. Now there is a next peer, so a refused connect gets retried instead of reported.

max_fails=0 is not decoration either. The default is max_fails=1 fail_timeout=10s, and these two "servers" are one process — so a single refused connect marks both peers down and nginx serves 502 for the next ten seconds. You'd have converted a 200 ms gap into a ten-second outage while believing you'd added resilience.

And while you're in there, stop showing people the nginx default page:

error_page 502 504 = @unavailable;

location @unavailable {
    default_type text/html;
    add_header Retry-After 5 always;
    return 503 '<!doctype html><title>Starting up</title><p>This service is starting. Refresh in a few seconds.';
}

503 with Retry-After, not 502. A process that is starting is not a broken upstream, and the status code is the only part of that a client, a CDN or a health checker can act on.

But be clear-eyed: all of this buys you a retry, not zero downtime. It covers a sub-second gap. It does not cover a three-second boot. For that you need to stop having a gap at all.

What actually fixes it: two of everything

Blue-green, but with systemd doing the work. A template unit gets you both colours from one file:

# /etc/systemd/system/app@.service
[Unit]
Description=app (%i)
After=network-online.target

[Service]
Type=simple
User=app
WorkingDirectory=/srv/app/slots/%i
EnvironmentFile=-/srv/app/slots/%i.env
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

systemctl start app@blue and app@green are now two independent services. The layout grows a slots directory:

/srv/app/
  releases/<ref>/
  current      -> releases/<ref>     the version that is SERVING
  slots/blue   -> releases/<ref>     one colour's release
  slots/green  -> releases/<ref>
  slots/blue.env                     PORT=41000
  slots/green.env                    PORT=41001

And the deploy becomes a sequence rather than a restart:

active=$(systemctl is-active --quiet app@blue && echo blue || echo green)
standby=$([ "$active" = blue ] && echo green || echo blue)

# 1. Build the release. Nothing serving is touched.
build_into "$RELEASE"
printf 'PORT=%s\n' "$STANDBY_PORT" > "/srv/app/slots/$standby.env"
ln -sfn "$RELEASE" "/srv/app/slots/$standby.tmp"
mv -Tf "/srv/app/slots/$standby.tmp" "/srv/app/slots/$standby"

# 2. Start the standby beside the one that is serving.
systemctl restart "app@$standby"

# 3. Prove it answers, on its OWN port — not through the proxy,
#    which is still pointing at the version we are replacing.
curl -fsS --retry 20 --retry-delay 1 "http://127.0.0.1:$STANDBY_PORT/up" >/dev/null

# 4. The switch. A reload does not drop connections in flight.
write_upstream "$STANDBY_PORT"
nginx -s reload

# 5. Only now is the old one expendable.
systemctl disable --now "app@$active"

Same deploy, two colours. Blue keeps answering until step 4, and step 4 is a reload, not a restart.

Step 3 is the one people skip and shouldn't. Probe the standby's own port, never the public URL — the public URL is still answered by the version you're trying to replace, so a probe through it always passes and proves nothing.

Step 4 is the moment traffic moves, and it is the only moment anything visible changes. An nginx -s reload starts new workers for the new config and lets the old workers finish what they're holding. Nobody gets cut off mid-response.

The thing container people never have to think about

If you've only done blue-green with containers, this is the part that doesn't transfer.

Each container gets its own network namespace, so both colours can bind port 3000 and only the network alias tells them apart. The name moves; the number stays.

On a plain host there is one port space. Two processes cannot both bind 3000. So the number is what moves, and the hostname stays 127.0.0.1. Everything downstream follows from that one fact:

your proxy repoint has to rewrite the port, not the upstream host;
you need somewhere to keep "which port is this colour on" — I write it to slots/<colour>.env and read it back at switch time;
pick your slot ports above the registered range and below ip_local_port_range (32768–60999 on most boxes), or you'll eventually collide with an ephemeral port the kernel handed out for an outbound connection. 41000–41999 is a quiet neighbourhood.

And one consequence worth refusing rather than shipping: if your app serves two domains on two ports, a colour only rebinds the primary one. The second domain keeps answering from the colour you are about to kill. That's a half-switched release, which is worse than an honest restart — so detect it and refuse, loudly, with a message that says why.

Two rules I'd argue for in any implementation

A colour's WorkingDirectory points at its own release, never at current. This sounds like a detail and it's the whole thing. Two versions can only run at once if each has its own working directory. Point both at current and you've rebuilt the restart with extra ceremony.

current moves at promote, not at deploy. It should keep meaning "the version that is serving" right up until the switch. That matters because your companion units follow it:

# worker unit — deliberately on `current`, not on a colour
[Service]
WorkingDirectory=/srv/app/current
ExecStart=/usr/bin/php artisan queue:work

Restarting that at promote — and only at promote — is also exactly right for a queue worker. A worker holds the code it started with, and the moment the new version becomes canonical is the moment it should pick it up. Not while it's still a candidate.

Gate on what you can actually reach

Back to the php-fpm shape, because it's where I had to be honest with myself.

A standby fpm master listens on a unix socket and nothing else. There's no port to curl. And the only nginx on the box is still pointing at the colour that's serving — so an HTTP probe "at the standby" through nginx is answered by the version you're replacing. It passes every time. It's not a gate, it's a gate-shaped thing that can only ever say yes.

Real options: stand up a second loopback listener just for the gate, or speak FastCGI to the socket directly. Both are actual work.

What I did instead was record that switch as unverified rather than dressing it up as passed, and keep the checks that genuinely are reachable — did the unit stay up through its first few seconds, and did the pre-start migration exit non-zero. That's less than an HTTP probe and it is not nothing.

An honest "we could not verify this one" is worth more than a green tick you invented. A check that cannot fail is indistinguishable from a check that never ran, and six months later nobody can tell you which one it was.

Prove it with curl, not with your eyes

Whatever you do, the test is two lines and you should run it before you believe anybody, including me:

while :; do
  printf '%s ' "$(curl -s -o /dev/null -w '%{http_code}' http://localhost/)"
  sleep 0.1
done

Leave it running. Deploy. Watch.

before:  200 200 200 502 502 502 502 502 502 502 200 200
after:   200 200 200 200 200 200 200 200 200 200 200 200

That's the entire acceptance criterion, and it's the one your users are actually running.

The takeaway

Atomic file swaps are the easy half, and they're the half that gets all the attention because they're satisfying to get right. Nobody's curl loop can see them.

Ask the boring question instead: between the old process exiting and the new one accepting connections, what answers the phone? If the answer is "nginx, with a 502", you don't have a zero-downtime deploy — you have an atomic file swap and an outage you've agreed not to measure.

And if you're about to fix it with proxy_next_upstream on a single-target proxy_pass: that line does nothing. List the peer twice.

Dev Log: 30 August 2026 — Absence Is Not the Same as Zero

Nasrul Hazim — Mon, 31 Aug 2026 00:26:48 +0000

Twenty-three commits across four repos today. Looking back at them, most had the same shape underneath: something that was empty, and the question of what empty was supposed to mean.

Empty, none, and unknown are three different answers

I built a card on an integrations screen that shows which permission scopes a stored API credential actually needs. Straightforward — until you hit a provider where you don't know.

The naive model has two states: a list of scopes, or an empty list. That's wrong, because an empty list on screen reads as "this credential needs no special permissions" — which is a confident, specific, and possibly false claim.

There are three states:

documented — we know the exact scopes, because we know which API calls we make. Show them.
none — the provider has no scope model at all. It's a plain key that works or doesn't. Saying that is useful.
undocumented — nobody has written them down yet. Show nothing at all, not an empty list.

final readonly class ScopeGuidance
{
    public static function of(array $scopes, ?string $note = null): self  { /* documented */ }
    public static function none(string $note): self                        { /* no scope model */ }
    public static function undocumented(): self                            { /* not written down */ }
}

Same rule a vulnerability scanner has to follow: an image that couldn't be scanned records nothing, and unscanned must never render like scanned-and-clean. The absence of a finding is not a finding of absence, and if your data model can't tell those apart, your UI will confidently lie.

While I was in there I also learned that a cloud provider token needs tag:create and not just tag:read if you intend to tag the resources you create — the kind of thing that fails at exactly the wrong moment. That note now lives on the card, above the list, because a scope nobody would guess deserves its reason next to it rather than in a comment nobody reads.

A count that was always zero

Same theme, uglier version: a dashboard tile showing open incidents that read 0 regardless of reality. A zero that's wrong is worse than a tile that's missing, because a wrong zero is reassuring.

The neighbouring tile was rendering a raw enum case name straight into the page — awaiting_promotion where a human wanted "Awaiting promotion". Which is the argument for enums carrying label() and color() rather than the Blade template doing Str::headline() on the way out. Put the presentation on the enum and every surface gets it right; leave it to templates and every surface gets it right separately, until one doesn't.

`/storage` is already taken

My favourite bug of the day. I added a new section to a Laravel app under the URL prefix /storage. Every test passed. In production, every page under it returned 403.

public/storage is Laravel's own storage:link symlink and it exists on essentially every Laravel deployment. A web server configured with try_files $uri $uri/ matches /storage as a real directory and answers before the request ever reaches PHP. The router never sees it.

The test suite never catches this, because the test suite doesn't go through a web server at all. That's not a gap in the tests, it's a category of bug tests of that kind structurally cannot see.

The fix was a different prefix. The actual fix was a test that walks every registered route prefix and fails if one collides with something that exists in public/:

it('does not register a route prefix that public/ shadows', function () {
    $shadowed = collect(File::directories(public_path()))
        ->map(fn (string $path) => basename($path));

    $prefixes = collect(Route::getRoutes())
        ->map(fn ($route) => Str::before(ltrim($route->uri(), '/'), '/'))
        ->unique()
        ->filter();

    expect($prefixes->intersect($shadowed))->toBeEmpty();
});

Route names can stay whatever reads well in the sidebar. It's the URL that has to dodge the framework's own conventions.

Refusing loudly beats hiding the button

I shipped a "clone this application" feature — take a running app's code and shape and stand up a copy somewhere else. Then, in the follow-up commit, I gated it: only an application that has deployed successfully at least once may be copied.

An app that has never deployed isn't worth copying. Its environment was never proved to boot anything, its build never produced an artifact, and a copy inherits all of that while looking like a copy of something that works.

The interesting part is how the gate surfaces. Two options:

Hide the Clone button when the app can't be cloned.
Show it, and render the reason when it's refused.

I went with 2, and I'd argue it's almost always right. A hidden button is indistinguishable from a feature that doesn't exist, and the user's next move is to ask you why it's missing. A refusal that says "this application has not deployed successfully yet — deploy it once, then it can be copied" is a hidden button that answers its own support ticket.

The shape that makes it work:

/**
 * Why this cannot be cloned, or null when it can.
 * For rendering, never for deciding.
 */
public function refusal(Application $app): ?string;

refusal() renders. The action enforces the same rule itself, independently. A gate that only lives in a Livewire component is not a gate — it's a suggestion with a nice tooltip.

Naming things for whose thing it is

Two smaller ones from a multi-tenant membership platform, both the same mistake in different clothes.

The first: its MCP server was announcing itself with the name of the toolkit it was built on — the same name for every organisation running it. If an assistant is holding connections to several of these, "which one is this?" is not a question it can answer from a shared name. Now the server names itself after the organisation it belongs to.

The second: the public homepage of each organisation was written in the voice of a software vendor — features, capabilities, the case for the platform. But the person landing there is a member, not someone shopping for member-management software. They want to know how to renew, what's coming up, and who to contact. Rewriting that copy changed no logic at all and was probably the highest-value change of the day.

Both are the same bug: content written from the builder's seat instead of the reader's.

Where the ceiling goes

Last one, from a chess app for kids. I added optional accounts, and with them a question: what does a signed-out player not get?

The easy answer is "a trial" — cap everything, nag for signup. I went the other way. Puzzles, lessons, the journal, cosmetics — all uncapped for a guest. Guest play is the primary path, not a funnel. The ceiling sits on exactly two things: the upper rungs of the engine ladder, and anything that connects you to another human.

Then the part I actually want to note, which is where the rule lives:

abstract final class AccountAccess {
  static const int guestLevelCeiling = 3;

  static bool allowsLevel(EngineLevel level, {required bool signedIn}) =>
      signedIn || level.index <= guestLevelCeiling;

  static bool allowsConnectedPlay({required bool signedIn}) => signedIn;
}

One place, unit-testable, consumed by both the level picker and the play controller. A ceiling nobody can unit-test is a ceiling that quietly moves — someone adds a third surface, reimplements the check slightly differently, and now your product rule has two versions of itself.

And locked levels stay visible, greyed with "Make an account to play this one". A child should be able to see the ladder they're climbing, not wonder where it ends.

The through-line

Every one of these was about a value that was empty or absent, and what the system chose to say about it. Empty scope list. Zero incidents. Missing button. Missing account.

The failure mode is always the same: absence gets rendered as a confident negative. No scopes needed. No incidents open. Feature doesn't exist. Nothing above level three.

None of those were true. They were just what empty looked like when nobody decided what empty should mean.

A Credential Rotation Without an Overlap Is Just an Outage You Scheduled

Nasrul Hazim — Mon, 31 Aug 2026 00:26:35 +0000

TL;DR — I shipped managed S3-compatible storage servers today, and the most interesting thing in the whole feature wasn't the provisioning pipeline. It was discovering that the storage daemon has no "rotate key" command — and that this absence is a better design than the setPassword() I'd already written for databases.

The rotation I got wrong first

My database provisioning has had credential rotation for a while. The contract looks about how you'd expect:

interface DatabaseAdminContract
{
    public function setPassword(string $username, string $password): void;
}

One call. The old password stops working, the new one starts working, done. Clean API, satisfying to write.

Here's the thing: between the moment that call returns and the moment every application holding the old password has been redeployed with the new one, every one of those applications is broken. Not degraded — broken, authentication-failure broken. The rotation didn't secure anything during that window. It just caused an outage, and the outage's length is however long your slowest deploy takes.

We call that a rotation. It's a cutover wearing a rotation's clothes.

The absence that turned out to be the answer

When I wired up the S3-compatible daemon, I went looking for its rotate command. There isn't one. You can create an access key, you can delete an access key, and that's the vocabulary.

My first reaction was "fine, I'll emulate it — delete then create." My second reaction, about ten seconds later, was that emulating it would reproduce exactly the window I just described, on purpose, in a system that didn't have it.

So rotation stopped being a call and became a workflow:

Issue a replacement key on the same server.
Re-attach it to the same buckets with the same privileges.
Let the application cut over on its own schedule.
Delete the outgoing key.

Both keys work for the whole span between 1 and 4. There is no moment where nothing authenticates. The cost is that a one-liner became four acts, two of which belong to a human.

Modelling "both keys are valid" as normal

This is the part that bit me in review. If two live credentials for the same bucket look like a fault in your UI, the next operator who sees it will "fix" it — by deleting one, probably the wrong one, probably during a deploy.

So the overlap has to be a first-class state, not an anomaly. In Laravel terms that's an enum carrying its own presentation:

enum AccessKeyStatus: string
{
    case Pending  = 'pending';
    case Active   = 'active';
    case Retiring = 'retiring';   // rotating out — still valid, on purpose
    case Deleted  = 'deleted';

    public function label(): string
    {
        return match ($this) {
            self::Retiring => __('Retiring — still valid until the overlap ends'),
            // …
        };
    }

    public function color(): string
    {
        return match ($this) {
            self::Retiring => 'amber',   // attention, not alarm
            self::Active   => 'emerald',
            // …
        };
    }
}

Amber, not red. The distinction matters more than it sounds: red means something went wrong, amber means something is in progress and will need you later. A retiring key is the second thing.

Two actions, because the second one is a decision

Starting a rotation is safe and can be one click. Finishing one is not, so it isn't the same action:

final class RotateAccessKeyAction
{
    public const DEFAULT_OVERLAP_DAYS = 7;

    public function execute(User $user, AccessKey $outgoing, int $overlapDays = self::DEFAULT_OVERLAP_DAYS): AccessKey
    {
        if ($overlapDays < 1) {
            throw ValidationException::withMessages([
                'overlapDays' => __('The overlap must be at least a day — a rotation with no window is a cutover, and this exists to avoid the outage a cutover causes.'),
            ]);
        }

        // Resume, don't start a second one. Two live replacements is worse
        // than none: the recorded window would cover the wrong pair.
        $existing = $outgoing->replacements()
            ->whereIn('status', [AccessKeyStatus::Pending, AccessKeyStatus::Active])
            ->first();

        if ($existing instanceof AccessKey) {
            return $existing;
        }

        // …issue the replacement, mirror the bucket permissions,
        //   mark the outgoing key Retiring with an expiry.
    }
}

Two details in there I'd argue for in any rotation implementation.

The idempotency check isn't defensive coding, it's correctness. Double-clicking "Rotate" shouldn't produce two replacement keys. Not because two keys is untidy, but because the compliance record says "credential X is being replaced by credential Y, and X stays valid until Z" — and that sentence has no meaning with two Ys in it.

The validation message explains the rule, not the constraint. The overlap must be at least a day is a form error. The sentence after the dash is the reason someone can act on. Error copy is where your architectural decisions actually reach the people using the thing.

Completing the rotation is separate, and it refuses by default:

final class CompleteAccessKeyRotationAction
{
    public function execute(User $user, AccessKey $outgoing, bool $force = false): void
    {
        $replacement = $outgoing->replacements()
            ->where('status', AccessKeyStatus::Active)
            ->first();

        if (! $replacement instanceof AccessKey) {
            throw ValidationException::withMessages([
                'key' => __('The replacement has not been created on the daemon yet. Withdrawing this one now would leave nothing able to authenticate.'),
            ]);
        }

        if (! $force && ! $outgoing->hasExpired()) {
            throw ValidationException::withMessages([
                'key' => __('The overlap runs until :until. Finish early only if you are certain nothing is still using this key.', [
                    'until' => $outgoing->expires_at?->toDayDateTimeString(),
                ]),
            ]);
        }

        // …audit, close the rotation record, then reuse the ordinary delete.
    }
}

Why not just delete it on a timer?

This was the tempting shortcut and I want to be honest about why I didn't take it.

The platform cannot see whether an application still holds the old secret. Nothing reports that. There's no callback, no heartbeat, no "I have cut over" signal. So a scheduled job that deletes the outgoing key when the timer expires is a platform arranging an outage for itself, at 3am, with nobody watching.

The expiry on the key is the deadline. The completion action is the decision. Keeping those two things separate is the whole point — the deadline creates the pressure, a human confirms the fact.

The trade-off is real and I'll name it: keys can now linger past their window if nobody clicks the button. That's a reporting problem (surface the overdue ones loudly), and I'd rather have a reporting problem than a scheduled outage.

The `force` flag is not a backdoor

Note that force doesn't skip the "does a replacement exist" check — only the "has the window elapsed" check. That's deliberate. One of those guards protects against impatience; the other protects against leaving a bucket with zero working credentials. Impatience is a judgement call. Zero working credentials never is.

If you take one thing from this post, take that shape: when you add an escape hatch, work out which of your guards it's allowed to open. A force that opens all of them isn't an escape hatch, it's an unguarded second code path.

Testing the window, not the calls

Rotation is exactly the kind of feature where mocking the daemon and asserting "createKey was called" tests nothing worth testing. What you actually care about is that both credentials work at the same time, and that the record says so:

it('keeps the outgoing key valid for the whole overlap', function () {
    $key = AccessKey::factory()->active()->create();

    $replacement = app(RotateAccessKeyAction::class)
        ->execute($this->operator, $key, overlapDays: 7);

    expect($key->fresh()->status)->toBe(AccessKeyStatus::Retiring)
        ->and($key->fresh()->hasExpired())->toBeFalse()
        ->and($replacement->status)->toBe(AccessKeyStatus::Active)
        ->and($replacement->id)->not->toBe($key->id);
});

it('refuses to start a second rotation for the same key', function () {
    $key = AccessKey::factory()->active()->create();

    $first  = app(RotateAccessKeyAction::class)->execute($this->operator, $key);
    $second = app(RotateAccessKeyAction::class)->execute($this->operator, $key->fresh());

    expect($second->id)->toBe($first->id);
});

it('refuses to complete while the window is still open', function () {
    // …rotate, then immediately try to complete
    expect(fn () => app(CompleteAccessKeyRotationAction::class)->execute($this->operator, $key))
        ->toThrow(ValidationException::class);
});

Three tests, three rules, none of them touching a real daemon. A fake admin driver behind the same contract makes that possible — which is the other half of the story and probably its own post.

The takeaway

I went in thinking the missing rotate command was a gap to paper over. It was a constraint that made me build the right thing.

If your rotation is a single call that swaps a value in place, ask what's holding the old value and how it finds out. If the answer is "we redeploy everything quickly," you don't have rotation — you have a cutover you've agreed not to talk about.

Overlap first. Deadline second. Human decides when the old one goes.

Next up: I still need to go back and give the database side the same treatment. Its setPassword() works fine, right up until the day it doesn't.

Dev Log: 29 August 2026 — Holes in the Ops Surface, and Docs That Lie

Nasrul Hazim — Sun, 30 Aug 2026 08:38:04 +0000

Twenty-five commits across eight repos today. The thread running through most of them: finishing things that were 90% automated and still needed a human in the middle.

The one-step-in-the-middle problem

I've been building a deployment platform with an MCP surface — an agent can provision a node, deploy an app, diagnose it, redeploy it. Today I deployed the first real tenant through it end to end and it went green, answered on its own domain with TLS, and returned 500 on every request.

Migrations had run. The seeders hadn't. And nothing in the product could run them.

Then the second hole, right next to it: the app was missing thirteen environment variables its seeders read, and no tool on the ops surface could set one. So a pipeline that could do everything still needed somebody in a web form for the one step in the middle. That's not a boundary. That's a hole.

Two tools closed it, and the interesting part of both is what they refuse to do.

Setting env vars: values go inbound only. Every read tool on that surface reports env keys and never values, and that had to stay true. So the write tool's response names each key and what happened to it — added, changed, unchanged, removed, generated — and never echoes a value back, not even one the caller just supplied. An agent that can read a password out of a deployment is a categorically worse tool than one that can set it. The audit trail records key names for the same reason: recording values would rebuild the plaintext credential store the encrypted column exists to avoid.

There's also a generate mode, so the values nobody ever needs to read never appear in a transcript or a shell history at all. APP_KEY is the case that matters — strong, secret, stable, never typed by a human. Deliberately not offered for anything that has to match something else, like a mail password, where a generated value is just silently wrong.

And it merges, never replaces. That rule is scar tissue: a wholesale restore once discarded the APP_KEY the runtime had minted, so every release got a fresh one — signing out every user and making every encrypted column unreadable. A write that replaced the array would be the same defect from the other side.

Running first-install commands: declared, not dictated. Some work can't live in the deploy path. A preStartCommand runs on every deploy, so it can only hold work that's safe to repeat. First-install work is the opposite shape — a seeder that creates a superadmin with User::create() is a duplicate-key error on its second run.

The tool decides when first-install work runs and never what it is. The commands live on the workload definition, and a small validator refuses anything that isn't php artisan … free of shell syntax:

final class SetupCommands
{
    public static function assertAllowed(string $command): void
    {
        // Allow-list of SHAPE, not a deny-list of metacharacters.
        if (! preg_match('/^php artisan [a-z0-9:_\-\. ]+$/', $command)) {
            throw new InvalidArgumentException(
                'Setup commands must be a plain `php artisan` command.'
            );
        }
    }
}

An allow-list of shape, not a deny-list of dangerous characters. A deny-list is a list somebody has to keep complete forever, and the first omission is remote code execution as the application's own account. An MCP tool that accepted a command string outright would be RCE on a customer's node for anyone holding an ops token.

The rest is unglamorous and load-bearing: run as the application's own account via runuser and never as root (a root-created file in the release tree breaks the app permanently), load the unit's environment so a seeder reading SUPERADMIN_EMAIL sees what the service sees, stop at the first failure because "seed, then provision" has an order, and refuse a second run unless forced — recorded in a new setup_completed_at column.

Both tools got Pest coverage for the input systemd fails silently on: an unparseable key is dropped outright, and a newline in a value makes the remainder read as further variables. Silent input mangling is exactly what you want a test for, because nothing else will tell you.

Trusting a rebuilt host's key

Third one from the same session. You rebuild a machine, the OS regenerates its SSH host key, and every subsequent connection fails the way it should — because from the client's side, "the host key changed" and "someone is on the wire" are the same event.

The thing you must not build is a tool that shrugs and accepts whatever key answers. So the shape is: a probe that fetches and fingerprints the key that's currently answering, a result object that carries it, and a distinct exception for mismatch versus probe failed. The operator sees the new fingerprint and confirms the re-trust. The system never decides on its own that a changed key is fine.

HostKeyMismatch and HostKeyProbeFailed being separate types matters more than it looks. One means "the machine you rebuilt is answering with a new identity, as expected". The other means "I could not reach it at all". Collapse those into one exception and the UI has to guess, and it will guess wrong on the day it counts.

CSP, twice, in opposite directions

The day's other theme got its own post, because one story wasn't enough: a Laravel app that enabled CSP by accident through a null default that means "auto-enable in production", and a static site whose correctly-strict script-src 'none' broke every mailto: link on it because Cloudflare's email obfuscation needs a script to undo itself.

Both failures were silent. Both happened at a seam where two systems each behaved correctly on their own. Full write-up in the companion post.

Shipping a desktop build

A Flutter app I've been building got a Windows platform target, a release workflow, and a public releases repository — the app repo stays private, the built artifacts and their download page go somewhere public.

That split is worth stealing. Your CI needs somewhere to put binaries that end users can reach, and "make the whole repo public" is not the only answer. A separate public repo holding nothing but tagged releases gives you a URL you can link from a marketing site, without exposing source or issue history. It got an ADR, because six months from now the reason will be less obvious than the arrangement.

The marketing site now renders its download buttons from that repo's latest release — fetched at build time only. Two reasons, both worth stating plainly: the site's CSP is connect-src 'none', so a client-side fetch would be blocked anyway; and GitHub's unauthenticated rate limit of 60 requests per hour per IP does not survive real traffic. Every failure path returns null and the section simply doesn't render — a 404 is the normal state before the first release exists, and a build shouldn't fail over the expected case.

Google Sign-In, behind a contract

The API gained a native Google Sign-In endpoint plus SSO buttons on the dashboard auth screens. Structure:

interface GoogleIdTokenVerifier
{
    public function verify(string $idToken): GoogleIdentity;
}

One implementation verifies against Google's JWKS. The action depends on the contract, so tests bind a fake and never touch the network — and if the verification strategy ever changes, the action doesn't. This is the default shape for anything that talks to a third party: a contract, a DTO for the verified result, and a driver behind it.

Docs that were quietly lying

Five commits on a documentation repo, and most of them were corrections, not additions:

A claim that dependency-allowlist enforcement was in place. It wasn't.
An endpoint documented as "planned" that had shipped.
A behavioural rule — guest play capped at a lower engine level, an account unlocks the rest — that existed in code and nowhere in the spec.

Plus two new ADRs, one of which forced a partial SRS rewrite. That's the correct outcome, by the way: if an architecture decision doesn't ripple into the requirements doc, one of the two documents isn't being read.

Documentation drift is the same category as the silent CSP failure. Nothing errors. The build is green. The document is just wrong, and it stays wrong until somebody reads it closely enough to notice — usually while relying on it.

Also today

Legal pages, with every placeholder replaced by an actual commitment, plus a Bahasa Malaysia privacy notice alongside the English one. Not an optional translation — PDPA s.7(3) requires the notice in both languages, so the bilingual pair is the deliverable, with lang and alternate props wiring up the hreflang link.
A UI bug worth naming: a failed in-app encounter was rendering the "you've collected everything" state instead of an error. A failure path falling through to the success path is a bug I keep meeting in different costumes. Test it as "the request failed" rather than "the request returned nothing", and it stops happening.
A mail history dashboard and a sidebar regression from the baseline patch — collapsible menu groups had stopped collapsing, and a logo had gone missing from two sidebars. The unglamorous half of any framework upgrade.

Takeaway

Today's pattern, said once: the last 10% of an automation is where all the security decisions live. Every tool I added had a version that would have been faster to write and wrong — echo the value back, accept any command string, trust whatever host key answers. The design work wasn't making the tools do things. It was deciding, precisely, what they must refuse to do.

CSP Broke Two of My Apps Today, in Opposite Directions

Nasrul Hazim — Sun, 30 Aug 2026 08:37:59 +0000

TL;DR — Content-Security-Policy took down parts of two different apps in one day. One app enabled it by accident and blocked its own fonts and analytics. The other had it locked down correctly, and that correctness broke a legally-required contact link. Same header, opposite failure modes, and neither one threw an error.

Failure one: a `null` that means "on"

I patched an app onto a newer baseline of our internal Laravel starter kit. The baseline shipped a SecurityHeaders middleware and a config/security.php that looked like this:

'headers' => [
    'csp' => env('SECURITY_CSP', null), // null = auto (production only)
    'csp_policy' => env(
        'SECURITY_CSP_POLICY',
        "default-src 'self'; ".
        "script-src 'self' 'unsafe-inline' 'unsafe-eval'; ".
        "style-src 'self' 'unsafe-inline'; ".
        "font-src 'self' data:; ".
        "connect-src 'self'; ".
        "frame-ancestors 'none'; ".
        "form-action 'self'"
    ),
],

Read that default carefully. null doesn't mean "off". It means auto, and auto means on in production. The middleware branches on it:

private function cspEnabled(): bool
{
    $flag = config('security.headers.csp');

    return $flag === null ? app()->isProduction() : (bool) $flag;
}

I did notice the flag during the patch. I set SECURITY_CSP=false in .env.example and moved on, which felt like enough.

It wasn't. .env.example is documentation, not configuration. The production box has its own .env written months ago, and that file has no SECURITY_CSP key at all. So env() returned the default, the default was null, null meant auto, and the app started enforcing default-src 'self' on a page that loads a webfont CDN, Google Tag Manager, the Meta Pixel and a Cloudflare analytics beacon.

Every one of those got blocked. The pages still rendered. Nothing 500'd, no exception hit the logs, no test failed — the fonts just fell back to system stacks and analytics quietly stopped recording. The only evidence lived in a browser console nobody had open.

The fix is boring, which is the point:

// Opt-in, not auto. An explicit null still restores the old behaviour.
'csp' => env('SECURITY_CSP', false),

And then widen the default policy so it's actually usable when someone does turn it on — the font host under style-src and font-src, the tag manager and pixel under script-src, their beacon endpoints under connect-src.

The lesson isn't about CSP. It's about defaults that change behaviour based on environment. A null that resolves differently in production than in local is a config value you cannot test locally by definition. If a security control needs to be on, make it explicitly on, per-app, after somebody looked at the policy. "Secure by default" is a good instinct, but a policy that was never verified against the app's actual asset list isn't security — it's an outage with good intentions.

There's a second-order rule underneath it: a change to .env.example changes nothing that is already running. Any new key you introduce has exactly two safe forms — a safe default in config/, or a deploy step that writes the key. .env.example is neither.

Failure two: a CSP that was right, and broke something anyway

Different project, same afternoon. A small static marketing site, ships an intentionally brutal policy: script-src 'none', connect-src 'none'. It's a site for a children's app, and there's no JavaScript on it worth the attack surface.

It's proxied through Cloudflare, with Scrape Shield's Email Address Obfuscation on — a feature that rewrites every mailto: in the HTML into /cdn-cgi/l/email-protection#<hex> and injects a small script to decode it back in the browser.

You can see where that lands. The rewrite happens at the edge, the decoder is a script, and script-src 'none' blocks the decoder. So all ten contact links rendered as the literal string:

[email protected]

...pointed at a dead /cdn-cgi/ URL. That address happens to be the PDPA access-and-deletion contact and the child-safety contact a store review requires to be reachable, so "it's just a mailto" wasn't available to me.

Three options:

Turn off Email Obfuscation in the Cloudflare dashboard. Works. It's also invisible state living in a web console that no repo remembers, and it dies the day the site moves to another DNS provider or someone spins up a fresh zone.
Loosen the CSP to admit the decoder script. Weakening script-src on a children's site to fix an email link is not a trade I'd defend in review.
Use Cloudflare's own opt-out markers in the markup.

Option three. Cloudflare skips anything wrapped in  / , so the whole fix is one tiny component:

---
/**
 * The contact address as a working mailto link.
 * Never write a bare mailto: anywhere else — route every one through this.
 */
import { CONTACT_EMAIL } from '../config';
---

<!--email_off--><a href={`mailto:${CONTACT_EMAIL}`}>{CONTACT_EMAIL}</a><!--email_on-->

Ten mailto: links became ten <MailLink />. The behaviour now lives in version control, survives a DNS move, and the docblock tells the next person why the component exists at all — which is most of its value. A one-line component with no explanation is the kind of thing a tidy-up PR deletes six months later.

What the two have in common

Both failures were silent, and silent in the way that costs you the most: the page rendered, the deploy went green, and the thing that broke was something you only notice by going and looking.

Both also happened at a boundary where two systems each behaved correctly on their own. The middleware honoured its documented default. Cloudflare performed the feature it was asked to perform. Nobody's code was wrong; the combination was.

That's the shape I'd watch for. When you add a policy that says "deny everything not on this list", you have quietly signed up to maintain that list against every other system in the request path — your CDN, your edge proxy, your analytics vendor, your font host — none of which will tell you when they start needing something new.

Two practical habits I'm taking from today:

Enable restrictive headers explicitly, per app, after checking the real asset list. Not by inheriting a default from a shared baseline.
Encode edge behaviour in the repo, not in a dashboard. If a fix can be expressed in markup or config that ships with the code, prefer that over a toggle in someone's control panel.

And if you run a strict CSP behind Cloudflare: go check your mailto: links right now. I'll wait.

DEV Community: Nasrul Hazim

Dev Log: 3 September 2026 — Everything I Fixed Today Was Already Written

The badge in the @else

The runner with no door

The tracking that was complete, and off

The token that was already spent

The one that wasn't already written

Takeaway

My Agent Skills All Said How to Start. None Said When to Stop.

The gap

Three sections, and why each exists

## Common Rationalizations — the arguments you'll make to yourself

## Red Flags — signals visible during the work

## Verification — the exit gate

Why the end of the file, not the top

Coverage, hygiene, and the boring half

And a smaller thing that changed every response

Takeaway

nginx add_header Appends. It Doesn't Override — And That Can Remove Protection.

The setup

What production actually answered

Three nginx facts worth internalising

1. add_header appends. Always.

2. add_header cannot ask a question.

3. add_header does not merge across levels — it replaces the whole set.

The fix: map on $upstream_http_*

The deployment trap that nearly cost more than the bug

What I didn't copy

Testing this honestly

Takeaways

Dev Log: 2 September 2026 — Every Bug Today Was a Claim Nobody Checked

The control that was green on nothing

Two screens that contradicted themselves

Caches that were confidently out of date

The one where the bug was blamed outward

And the same idea, one layer up

Takeaway

Dev Log: 1 September 2026 — Every Bug Today Was a Name Problem

Case-sensitive database, case-insensitive directory

A picker that collected a choice nobody consumed

"Confirm by typing the name" only works if the name is unique

A relation cached before the rows existed

A coverage test that could only see what today's fixtures happened to draw

The build tool renamed a class the native layer looks up by name

And nineteen URLs sharing one document

Also, briefly

Takeaway

Dev Log: 31 August 2026 — Loaded Is Not Serving

Loaded is not serving

Both endpoints being S3 is not "can copy server-side"

A version that is a fact beats a version that is a guess

A hostname nobody typed

Who owns the decision — a menu refactor that's really an access refactor

The rest, briefly

The takeaway

The Most Dangerous Backup Is the One That Reports Success

The failure mode nobody designs for

Sync is not backup

"Move" is copy, verify, then delete — in that order, always

Verification that admits what it didn't check

Integrity is not restorability

The quiet ones: pagination and fakes

The incremental baseline has to be verified, not just complete

The takeaway

A Symlink Swap Is Not a Zero-Downtime Deploy

The half that was already right

The half nobody looks at

Two shapes, two different 502s

The directive that does nothing

What actually fixes it: two of everything

The thing container people never have to think about

Two rules I'd argue for in any implementation

Gate on what you can actually reach

Prove it with curl, not with your eyes

The takeaway

Dev Log: 30 August 2026 — Absence Is Not the Same as Zero

Empty, none, and unknown are three different answers

A count that was always zero

/storage is already taken

Refusing loudly beats hiding the button

The badge in the `@else`

`## Common Rationalizations` — the arguments you'll make to yourself

`## Red Flags` — signals visible during the work

`## Verification` — the exit gate

1. `add_header` appends. Always.

2. `add_header` cannot ask a question.

3. `add_header` does not merge across levels — it replaces the whole set.

The fix: `map` on `$upstream_http_*`

`/storage` is already taken

The `force` flag is not a backdoor

Failure one: a `null` that means "on"