DEV Community: Nasrul Hazim

I turned my Claude Code sessions into a 3D office my team can watch

Nasrul Hazim — Mon, 21 Sep 2026 05:02:38 +0000

I run several Claude Code sessions at once, and each one spawns subagents in the background. From a terminal I could not tell who was doing what, who was stuck waiting on me, or what it all cost. Now I open one page and watch a 3D office instead.

TL;DR — seven Claude Code hooks → one Python script → Agent Bus for routing → Hermes3D for the office. Every session is a named staff member with a role, a live activity line, a kanban card and a token count. No LLM calls anywhere in the pipeline, so it costs nothing to run.

What you get

The same office in Hermes3D's 2D pixel mode, lighter on a laptop:

In the office	Where it comes from
One character per session and per subagent	Claude Code hooks
Label `Danial · Laravel Dev · Standardise listings`	A staff roster of 30 names, the subagent's role, and the task it was given
The lead is me: `Nasrul · Product Owner`	The main session, titled with Claude's own session title
Live activity text: `Run PHPStan`, `Edit app/Actions/...`	Each tool call
`⏳ Tunggu kelulusan` ("waiting for approval") when a permission prompt is open	The `Notification` hook
Kanban: working → needs attention → done	One card per delegated task
Tokens and model per staff member	Read from each transcript

The labels are in Malay because that is how my team talks; swap the strings and nothing else changes.

Nothing here calls an LLM. It is routing and bookkeeping on top of events Claude Code already emits, so it adds no token cost.

How it fits together

Three pieces, all on localhost: Claude Code fires hooks, Agent Bus turns them into agents, and Hermes3D draws the office.

%%{init: {"theme":"neutral","themeVariables":{"fontSize":"18px"},"flowchart":{"nodeSpacing":30,"rankSpacing":45,"padding":10}}}%%
flowchart TD
  CC["Claude Code sessions + subagents"] -->|"7 hooks"| H["office_hook.py"]
  H -->|"events"| HUB["Agent Bus hub :4000"]
  HUB --> GW["Agent Bus gateway :18789"]
  GW -->|"gateway protocol"| ST["Hermes3D Studio :3000"]
  H -->|"task cards"| ST
  ST --> RO["Read-only proxy :3100"]
  RO -->|"ngrok, on demand"| TEAM["Team"]

The hook writes to two places: agent events go to the hub, kanban cards go straight to Studio's task store. Hub, gateway and Studio all bind to 127.0.0.1; only the read-only proxy is ever tunnelled, and only while I am sharing.

Step 1: Hermes3D and Agent Bus, on demand

Clone both, install, and point Hermes3D at the Agent Bus gateway. I do not want daemons running all day, so one script starts everything when I want to look and Ctrl-C stops it.

git clone https://github.com/iamlukethedev/Hermes3D.git ~/Projects/hermes
git clone https://github.com/emiliovos/agent-bus.git ~/Projects/agent-bus
(cd ~/Projects/hermes && npm install && cp .env.example .env)
(cd ~/Projects/agent-bus && npm install)

In Hermes3D's .env:

HERMES3D_GATEWAY_URL=ws://localhost:18789
HERMES3D_GATEWAY_ADAPTER_TYPE=hermes
UPSTREAM_ALLOWLIST=localhost,127.0.0.1

The launcher, trimmed to what matters:

#!/usr/bin/env bash
pids=(); trap 'kill "${pids[@]}" 2>/dev/null' EXIT INT TERM
cd ~/Projects/agent-bus
npx tsx watch src/index.ts & pids+=($!)                          # hub :4000
AGENT_PRUNE_HOURS=1 npx tsx watch src/gateway/index.ts & pids+=($!) # gateway :18789
cd ~/Projects/hermes
[ -f .next/BUILD_ID ] || npm run build
npm start & pids+=($!)                                           # Studio :3000
wait

Two choices in there are deliberate:

Start the hub and gateway separately, not with npm run dev:all. That also starts Agent Bus's own UI on :3000, which collides with Studio.
Run Studio with npm start, not npm run dev (see Gotchas).

I aliased it: alias office="$HOME/.agent-bus/office.sh".

Step 2: the hook that hires the staff

One Python script (stdlib only, so it runs under whatever PATH Claude Code gives hooks) handles seven hook events. Register it in ~/.claude/settings.json:

"hooks": {
  "SessionStart":     [{ "hooks": [{ "type": "command", "command": "bash ~/.agent-bus/hermes3d-hook.sh" }] }],
  "UserPromptSubmit": [{ "hooks": [{ "type": "command", "command": "bash ~/.agent-bus/hermes3d-hook.sh" }] }],
  "PostToolUse":      [{ "matcher": "*", "hooks": [{ "type": "command", "command": "bash ~/.agent-bus/hermes3d-hook.sh" }] }],
  "Notification":     [{ "hooks": [{ "type": "command", "command": "bash ~/.agent-bus/hermes3d-hook.sh" }] }],
  "Stop":             [{ "hooks": [{ "type": "command", "command": "bash ~/.agent-bus/hermes3d-hook.sh" }] }],
  "SubagentStop":     [{ "hooks": [{ "type": "command", "command": "bash ~/.agent-bus/hermes3d-hook.sh" }] }],
  "SessionEnd":       [{ "hooks": [{ "type": "command", "command": "bash ~/.agent-bus/hermes3d-hook.sh" }] }]
}

The .sh just execs the Python file, so I never touch settings.json again when the logic changes. The hook exits immediately unless the session's cwd is under a folder listed in ~/.agent-bus/roots, so my other Claude sessions stay out of the office. It never blocks Claude: 0.5s timeouts, every error swallowed, always exit 0. It takes about 70ms.

What each hook does in the office:

Hook	Character	Kanban card
`SessionStart`	Lead appears	—
`UserPromptSubmit`	Lead: `Kerja baru: <prompt>`	New lead card → working
`PostToolUse`	Activity text, model, tokens	Subagent card → working
`Notification`	Permission prompt → `⏳ Tunggu kelulusan`	Card → needs attention
`Stop`	Lead: `✅ Siap — tunggu Nasrul`, or keeps supervising if staff are still busy	Lead card → done
`SubagentStop`	Subagent leaves, its name is freed	Card → done
`SessionEnd`	Lead leaves	—

Who is who

The hook reads everything from the JSON Claude Code sends on stdin. Subagent tool calls carry agent_type (laravel-developer, tech-writer) and agent_id, which is all it needs:

Character	Id	Name	Task
Main session	`lead-<session6>`	`lead` in `staff.json`	Latest `aiTitle` in the session transcript
Subagent	`<role>-<agent_id6>`	First free name in that role's list	`description` in `<session>/subagents/agent-<id>.meta.json`

The roster is a small JSON file; I gave 30 staff names across 17 roles, more names for roles that run in parallel:

{
  "lead":              { "short": "Product Owner", "names": ["Nasrul"] },
  "laravel-developer": { "short": "Laravel Dev", "names": ["Danial", "Farhan", "Iqbal", "Syafiq", "Haziq", "Aiman"] },
  "tech-writer":       { "short": "Writer", "names": ["Mei Ling", "Hana"] }
}

A name is held until that subagent's SubagentStop, so three Laravel devs in parallel get three different names. Edits to the roster apply on the next event.

The lead is marked busy while its team works: at most once a minute it posts 👀 Menyelia: Danial, Farhan, Iqbal ("supervising"). When the last subagent stops it says ✅ Pasukan siap — tunggu Nasrul ("team done, waiting on Nasrul").

Step 3: kanban, tokens and the model

Hermes3D already has a task board whose columns map straight onto a Claude session: inbox, scheduled, working, needs_attention, done. The hook upserts cards through Studio's own endpoint, only when a card's status actually changes:

curl -X PUT http://127.0.0.1:3000/api/task-store \
  -H 'Content-Type: application/json' \
  -d '{"task":{"id":"cc-sub-<agent_id>","title":"Standardise listings","status":"working","source":"hermes_event","channel":"claude-code"}}'

The board is under OPEN HQ → Kanban on the right edge. The "Kanban board" desk in the 3D scene opens an install prompt instead — that is upstream behaviour, not your cards missing.

Tokens. Every assistant message in a transcript carries its usage. Claude Code writes one line per content block, each repeating the same usage, so the hook de-duplicates by message id before summing input, output, cache read and cache write. A patched Agent Bus serves the totals through sessions.usage, which Studio's Analytics panel reads. Cost stays at zero on purpose: there is no pricing table in the pipeline, and a wrong cost is worse than none.

Model. The newest "model":"..." in the tail of each transcript, sent on every event. Studio reads it from the agent's main session in sessions.list, not from config.get, because it caches config.get for the whole connection — agents that appear later would never get one.

Office opens populated. Everything the hook sends is also appended to a spool file. When the launcher starts, it waits for hub, gateway and Studio, then replays the last 30 minutes. Without that, the office opens empty and fills only as each session next uses a tool.

Daily report. The same spool feeds office-summary: staff, active time, tool calls, approval waits, output tokens and every task with its final status, as a markdown table.

Gotchas I hit

Most of the work was here. Each looked like "it doesn't work" and had one specific cause.

Symptom	Cause	Fix
Every session collapsed into one character	Agent Bus's sample hooks read the agent name from a shell env var, shared by every session	Take identity from the `session_id` / `agent_id` in the hook's stdin JSON
15 characters for 4 sessions, some named after vendor packages	I named characters after the `cwd` folder; subagents roam into worktrees, `vendor/`, docs	Identity is the session or subagent, never the folder
Sample hooks sent `Using unknown`	They read `CLAUDE_TOOL_NAME`, which current Claude Code does not set	Read `tool_name` from stdin
A closed session stood at its desk for a day	Agent Bus marks `session_end` as idle; pruning runs hourly with a 24h TTL	Patch: remove the agent on `session_end`; `AGENT_PRUNE_HOURS=1` as the backstop
Working agents showed as idle	Studio re-infers "running" on every roster reload from `sessions.preview` — only if the last item is from the user. Agent Bus returned a different shape, all assistant messages	Return `previews[].items[]`; append the task as a trailing user item while the agent is active
The lead was always idle	It delegates to background subagents and ends its turn; subagent activity only sent heartbeats	Lead posts `👀 Menyelia: …` while its team works
"Waiting for approval" when nothing needed approving	`Notification` also fires for the 60-second idle nudge ("waiting for your input")	Only messages about permission become approvals
Kanban cards titled `<task-notification>`	A finishing background subagent is injected into the session as a user prompt	Ignore prompts that start with `<task-notification`, `<system-reminder`, `<command-`
"Connecting to your runtime…" forever in a fresh tab	In dev mode the Next.js hot-reload socket hangs, and Chrome queues the gateway socket behind it	Run Studio with `npm start`
`studio.gateway_url_blocked` after switching to production	Production refuses every upstream gateway not listed	`UPSTREAM_ALLOWLIST=localhost,127.0.0.1`
Hub and gateway reachable from the LAN	Agent Bus listens on `0.0.0.0`	Patch both to bind `127.0.0.1`
A jq filter silently dropped every event without a file path	`(.x // empty) \	length > 0 `—` empty` short-circuits the whole object

The Agent Bus changes live as commits on a local local-patches branch, so a git pull shows exactly what I changed instead of silently undoing it.

Sharing a read-only view for a few minutes

Studio is not a viewer. Anyone who reaches it can change settings, move cards, and hit endpoints that read local files (/api/gateway/media), list your home directory (/api/path-suggestions) or call GitHub with your credentials (/api/office/github). So I never tunnel Studio itself. A 120-line Node proxy on 127.0.0.1:3100 sits in front of it:

Request	Proxy
Pages and static assets	Pass through
`GET /api/*`	Only an allowlist: `studio`, `task-store`, `health`, `office`, `office/layout`, `office/presence`
`PUT /api/studio` (the UI saves view state constantly)	Answered with the current settings, nothing written
Any other write	403
Gateway RPC `connect`, `health`, `status`, `.list`, `.get`, `.preview`, `.history`, `*.usage`	Forwarded
Any other RPC (`chat.send`, `config.set`, `agents.delete`, …)	Answered with a `read_only` error

An allowlist, not a blocklist: when a new Studio endpoint appears, it is closed until I decide otherwise.

Then one command shares it and Ctrl-C ends it:

#!/usr/bin/env bash
curl -sf http://127.0.0.1:3100/api/health >/dev/null || { echo "start the office first"; exit 1; }
pass=$(cat ~/.agent-bus/share-password)   # chmod 600
echo "user: team  password: $pass"
exec ngrok http 127.0.0.1:3100 --basic-auth "team:$pass"

The link changes every time, and nothing is exposed once I stop it. For named people instead of a shared password, ngrok's Google login works: ngrok http 127.0.0.1:3100 --oauth google --oauth-allow-email ali@example.com.

One thing to say to yourself before sharing: viewers see your prompts, task titles and file paths.

Try it, and what's next

The whole thing is two cloned repos, one hook script, a roster file and a launcher. My day now looks like this:

office            # office + read-only view, Ctrl-C stops all
office-share      # only while showing the team
office-summary    # end of day: who did what

Still open:

Cost. Tokens are there; dollars need a pricing table I trust.
Labels. The 3D label shows only the first name. Role and task are in the panels and on the board.
One floor. Hermes3D removed multi-floor offices upstream, so several products share one office with [product] in the label.

Credit where it is due: Hermes3D by LukeTheDev draws the office, and Agent Bus by emiliovos does the routing. I only wired Claude Code into them.

If you build something similar, I would like to see how you name your staff.

Dev Log: 6 September 2026 — A Guarantee You Only Wrote Down Isn't One

Nasrul Hazim — Wed, 09 Sep 2026 10:40:04 +0000

92 commits today across six repositories, two products from scratch — a learning platform in Laravel and a puzzle game in Flutter. Very different stacks. Reading the day back in one sitting, almost every decision I made was the same decision:

Where do I put a fact so that it cannot become wrong?

Not "where is it convenient." Not "where will I remember to update it." Where can it not drift. Every good call today was some version of derive it, or enforce it — and the one call I got wrong was a guarantee I enforced on people who never asked for it.

1. The event stream that refuses to be updated

The learning platform records every analytic — funnel, item analysis, verification counts — into one append-only event table. The obvious way to build that is a table with a comment above it saying append only, please don't update.

I don't trust that comment. Neither should you. So the model has no updated_at, and it actively refuses:

public const UPDATED_AT = null;

/**
 * Refuse any mutation of recorded history.
 *
 * Enforced on the model rather than only by convention: an append-only
 * stream that can be quietly rewritten is just a table.
 */
protected static function booted(): void
{
    static::updating(function (): never {
        throw new LogicException('The event stream is append-only; a recorded event cannot be updated.');
    });

    static::deleting(function (): never {
        throw new LogicException('The event stream is append-only; a recorded event cannot be deleted.');
    });
}

The : never return type is doing real work there — static analysis knows the closure can't fall through, so nothing downstream has to pretend an update might succeed.

An append-only stream that can be quietly rewritten is just a table. The word "guarantee" is doing no work unless something throws.

One nicety in the writer action: an unmapped event name records as unknown rather than throwing. Losing an event because someone typo'd the name is worse than an untidy label in a report. Strict about the shape, forgiving about the vocabulary.

2. The count that is never stored

Same system, immediately downstream. There's a public verification page, and I need to know how many times a credential has been verified.

Tempting: a verification_count column, incremented on each check. Fast reads, one integer.

It's also a number that can disagree with reality — a failed increment, a replayed request, a backfill that forgot, and now you have a count that nobody can reconcile against anything. So the count is a query over the event stream, always. The test that matters isn't "does the counter go up"; it's a replay test: rebuild the derived figure from the stream and assert it matches. That's the property that makes a single source of truth worth the join cost. A denormalised counter is a cache, and if you're going to keep a cache you have to be able to prove it's right — which means you needed the stream anyway.

3. A level is a name for a band, not a second number

Over on the puzzle game, the whole day's cleanest change. The game has a per-genre Glicko-2 rating — one number for "how good you are at this." Players don't read 1,732 as skill, so I wanted levels.

The lazy version: add a level field, bump it when the rating crosses a threshold. Now you have two numbers competing to mean the same thing, and they will disagree, and when they disagree neither one is trustworthy.

So nothing is stored:

/// The rating at which each level begins. Index 0 is level 1.
const List<int> kLevelThresholds = [0, 1100, 1350, 1600, 1850, 2100];

/// The level a rating sits in, counting from 1.
int levelFor(num rating) {
  var level = 1;
  for (var i = 1; i < kLevelThresholds.length; i++) {
    if (rating >= kLevelThresholds[i]) level = i + 1;
  }
  return level;
}

Forty lines, no migration, no sync bug possible. Level and rating cannot disagree, because there is only one of them.

The genuinely useful part fell out sideways, and this is why I keep pushing on derived state. Each of the three puzzle genres had its own difficulty formula — its own magic constants, none of them agreeing on where a band started. Once "level" existed as a single derived idea, all three difficulty curves could be expressed in terms of it. "Level 4" went from being three formulas that happened to look similar, to being one idea. And now crossing a level is exactly the moment the puzzles get bigger, in every genre, which is a promise the UI can make honestly. A test pins that all three step at the same ratings.

I didn't set out to unify the difficulty curves. Naming the shared concept made the duplication visible.

4. The solve check that doesn't look at the answer

Second genre landed today — a nonogram, the grid puzzle where numbers along the edges tell you the runs of filled cells. Generating one is easy: make a pattern, derive the clues from it.

The trap is checking the solve. The obvious implementation compares the player's grid to the generated pattern. That's wrong, and it's wrong in a way that only shows up as an angry player: a clue set doesn't always describe a unique picture. Sometimes two different grids satisfy the same numbers. Rejecting a player's valid alternative because it isn't the one you happened to generate is a bug you'd never reproduce yourself.

So solveState compares clues to clues — derive the clues from what the player built, compare to the clues on screen, never look at the stored pattern. The test enumerates every grid the clues allow across 40 seeds and asserts all of them are accepted.

Same theme, third time: the pattern was a stored answer. The clues are the actual contract. Check against the contract.

Worth noting what didn't change: adding this second genre required zero edits to the engine core. The puzzle contract, the registry, the seeded PRNG, the rating maths, the attempt rules — all untouched. Shared code changed in two places: five lines in the composition root to register the engine, and a genre dropdown in the session screen that any second genre would need. That's the checkpoint I set the abstraction up for, and it's the only honest way to find out whether a one-implementation abstraction was real.

5. A parsed question, when it stopped being an array

Small refactor, unusually instructive. The learning platform imports question banks from Markdown — parse line by line, build up each question as you go.

The first version built each question as an associative array and mutated it as more lines arrived. It worked, then it started attaching options to the wrong question, silently, because the version before it juggled array references. Nothing in the type system was watching, and nothing in the reader's head was either — an array grown line by line has no shape you can hold.

Replacing it with a small object fixed the class of bug rather than the instance:

$questions[] = new ParsedQuestion(trim($m[1]), $outcome);
// ...
$questions[$last]->addOption(trim($m[2]), correct: isset($m[3]));

Validation moved onto the object that owns it, so the "more than one correct option means multi-select, not a mistake" rule lives with the thing it describes instead of in a loop 40 lines below. And the static analyser can finally see the shape, which is the part that would have caught the original bug for free.

6. The one I got wrong

I required two-factor authentication for staff roles. It wasn't asked for. I did it because it seemed obviously correct, and it was the wrong call — you don't get to impose a security policy on somebody else's operational reality because it's good hygiene. It's now opt-in behind a config flag, off by default, and it switches on when the platform actually holds credentials worth protecting.

Turning it on for a day did earn its keep, though, because it exposed two dormant defects on the one page it herded people to:

The middleware redirected staff to the two-factor setup page — which the auth scaffolding guards behind password confirmation — and the middleware then bounced the confirmation screen straight back. ERR_TOO_MANY_REDIRECTS. My test checked only the first hop. First hop was correct. The chain wasn't. The test now follows redirects to the end and insists it terminates, and I verified it reproduced the loop before fixing it.
The scaffold shipped views calling an OTP component it never provided. Dormant for as long as nobody was forced onto that page. Requiring 2FA turned a missing component into a lockout with nothing else reachable.

Two lessons, and the redirect one is the sharper: asserting on the first response of a redirect chain proves almost nothing. A loop is exactly a sequence of individually correct hops.

The second is about blast radius. An error on a page nobody visits is a bug. The same error on the only page a blocked user can reach is a lockout. So enrolment is now tested end to end — enable, QR, confirm with a real TOTP — and there's a console command to enrol or reset an account from the CLI. That last one isn't a dev shortcut: an admin who loses their authenticator is locked out of every page including the one that would fix it, and on a one-person team there's no second admin to unlock them.

7. And the boundaries

I also drew eleven bounded contexts in the learning platform and made them executable the same afternoon — which got its own post — "Eleven Empty Folders and a Test" (companion post, link at review time) — because Pest's arch() helpers can't help you when the modules don't have any classes in them yet, and the workaround (read use statements out of the source text) has trade-offs worth spelling out.

Short version: an architectural rule that isn't a failing test is a preference.

Takeaway

One question, asked seven times today:

Can this fact be derived instead of stored? (level, verification count) → then store nothing.
Can this rule throw instead of being documented? (append-only, module boundaries) → then make it throw.
Am I checking against the answer or against the contract? (nonogram clues) → check the contract.
Does this data have a shape the analyser can see? (parsed question) → give it a class.
Did I test the first hop or the whole chain? (2FA loop) → the chain.
Is this guarantee mine to impose? (staff 2FA) → probably not; make it a flag.

The through-line: a fact stored in two places is a bug with a delay on it. Every one of today's good decisions was refusing to accept that delay, and the bad one was refusing somebody else a choice. Different mistake, same root — deciding on behalf of a future you can't see.

Eleven Empty Folders and a Test: Enforcing Module Boundaries Before the First Class Exists

Nasrul Hazim — Wed, 09 Sep 2026 10:39:18 +0000

TL;DR — I started a modular monolith today and drew eleven bounded contexts before writing a single class in any of them. Then I tried to enforce the boundaries with Pest's arch() helpers and couldn't, because arch() needs classes to exist and mine didn't yet. So the boundary test reads use statements out of the source text instead. It's cruder than reflection and it catches the violation on the day the boundary is drawn, which is the only day that matters.

The failure mode I was trying to avoid

Every modular monolith I've seen go bad went bad the same way. Somebody draws the modules in a wiki page. Everybody agrees. Six weeks later a controller in Billing does Catalogue\Models\Course::find($id) because it was two in the morning and the alternative was writing an action. Nobody notices, because nothing was watching.

The wiki page is still there. It's just not true any more.

So the rule I actually care about isn't "draw good boundaries." It's a boundary that isn't executable is a preference, not a boundary. Same class of thing as a comment that says // don't call this directly.

The structure I settled on is ordinary — eleven modules under app/Platform/, each shaped the same way:

app/Platform/<Module>/
├── Actions/     public entry points — the only way in from another module
├── Contracts/   interfaces this module owns
├── Events/      what it publishes
├── Models/      private to this module
└── README.md    what it owns and publishes

Three rules, and I wanted all three green from the first commit:

A module may not reference another module's Models.
Cross-module access goes through the owning module's Actions, Contracts or Events.
No domain action may import a vendor SDK — it calls a contract.

Where `arch()` couldn't help

Pest's architecture testing is the obvious tool here. You write something like:

arch('billing does not touch catalogue models')
    ->expect('App\Platform\Billing')
    ->not->toUse('App\Platform\Catalogue\Models');

Clean. Reads like the rule. And on day one it does nothing at all, because arch() works off the class map: it resolves classes, walks their dependencies, and reasons about real symbols. Point it at a namespace with no classes in it and there's nothing to reason about. The expectation passes on emptiness.

That's fine when you're retrofitting boundaries onto a codebase that already has 400 classes. It's exactly wrong when you're declaring boundaries on a codebase that has none, which is the moment the boundary is cheapest to hold and the moment the first violation is most likely — the very first class someone drops into Billing/Models is the one that decides what "normal" looks like in that folder.

I wanted the test to be red on a violation the same afternoon I drew the lines. So I gave up on reflection and went to the text.

Reading `use` statements like a linter

The whole enforcement is a Pest feature test that globs the filesystem and regexes imports:

/** @return array<int, string> Module names, discovered from the filesystem. */
function modules(): array
{
    $names = array_map('basename', glob(app_path('Platform/*'), GLOB_ONLYDIR) ?: []);
    sort($names);

    return $names;
}

/** @return array<int, string> The fully-qualified imports in a file. */
function importsIn(string $path): array
{
    preg_match_all('/^use\s+(?:function\s+)?([^\s;]+)/m', (string) file_get_contents($path), $m);

    return $m[1] ?? [];
}

Note modules() discovers from the filesystem rather than a hardcoded list. That matters for the rule itself — a new module folder is automatically subject to every rule below, so nobody can opt out by creating a twelfth context quietly.

The Models rule then falls out in about fifteen lines:

it('never references another module\'s models', function () {
    $violations = [];

    foreach (modules() as $module) {
        foreach (moduleFiles($module) as $file) {
            foreach (importsIn($file) as $import) {
                if (! str_starts_with($import, 'App\\Platform\\')) {
                    continue;
                }

                [$owner, $layer] = [explode('\\', $import)[2] ?? '', explode('\\', $import)[3] ?? ''];

                // Reaching into another module is allowed only via its public
                // Actions, Contracts or Events — never its Models.
                if ($owner !== $module && ! in_array($layer, ['Actions', 'Contracts', 'Events'], true)) {
                    $violations[] = sprintf(
                        '%s imports %s (reach %s through its Actions, Contracts or Events)',
                        str_replace(app_path().'/', 'app/', $file),
                        $import,
                        $owner,
                    );
                }
            }
        }
    }

    expect($violations)->toBe([], "cross-module boundary violations:\n".implode("\n", $violations));
});

Two deliberate choices in there worth calling out.

Collect, then assert once. The naive version asserts inside the loop and dies on the first violation. If you've just merged a branch that broke the rule in nine places, you want all nine in one failure message, not nine round trips through the suite. The message is written as an instruction — reach Catalogue through its Actions — because the person reading it at 2am is the person the rule exists for.

The allowed layers are a whitelist, not a blacklist. I'm not banning Models; I'm permitting Actions, Contracts, Events and nothing else. When someone adds a Support/ or Queries/ folder to a module later, it is private by default and the test says so. A blacklist would have silently let it through.

The vendor-SDK rule is the same shape with a prefix allowlist:

$allowed = ['App\\', 'Illuminate\\', 'Carbon\\', 'Spatie\\LaravelData\\'];

Anything else imported inside an Actions/ directory fails. That's what keeps the payment provider, the storage provider and the video provider behind contracts, which in turn is what lets the whole suite run offline with fake drivers — and makes swapping a provider a driver change rather than a migration project.

The test that asserts documentation exists

This one felt slightly ridiculous when I wrote it and I've come around on it:

it('gives every module a README stating what it owns and publishes', function () {
    foreach (modules() as $module) {
        $readme = app_path('Platform/'.$module.'/README.md');

        expect(is_file($readme))->toBeTrue("module [{$module}] has no README");
        expect((string) file_get_contents($readme))->toContain('**Owns:**', '**Publishes:**');
    }
});

It's a very dumb check — the README could say **Owns:** stuff. It isn't checking quality. What it's actually enforcing is that creating a module is a decision you have to write down, and the two headings force the only two sentences that matter: what data is yours, and what other modules are allowed to react to.

I've written a lot of architecture docs that rotted. This one can't rot to absent, which turns out to be most of the rot.

What this buys, and what it doesn't

Be clear-eyed about the trade. Text-level checking is weaker than reflection in specific, knowable ways:

Inline FQCNs slip through. \App\Platform\Catalogue\Models\Course::find(1) has no use statement, so the regex never sees it. In practice this is rare in a codebase with a formatter and an IDE that auto-imports, but it is a real hole.
Grouped and aliased imports need care. use App\Platform\Catalogue\{Models\Course, Actions\Publish}; isn't parsed by that pattern.
It knows nothing about runtime. A container binding, a string class name, an event listener resolved by name — all invisible.

If you want those closed properly, the answer is a real parser (nikic/php-parser) or deptrac, and at some point I'll probably graduate to one. But the honest cost/benefit today: the fifteen-line regex version was working before lunch, it fails loudly, it has zero dependencies, and it covers the way violations actually get written. A perfect checker I hadn't finished yet would have caught nothing.

There's also a deliberate hole I want: the shared kernel. App\Models, App\Enums, App\Contracts, App\Support are usable from anywhere. App\Models\User in particular is shared across all eleven modules, because every persona in the system is a user and duplicating identity per module would be worse than sharing it. Purists will wince. I'd rather have one honest shared kernel that everyone knows about than eleven private User copies pretending to be independent.

The bit I keep coming back to

Directories appear as the first class lands in them. An empty module is a declared boundary, not dead scaffolding.

Eleven folders with nothing in them but a README looks like over-engineering right up until you notice the test is already guarding them. The boundaries are drawn where a service split would go, so that option stays open — and unexercised, which is the point. I'm not running eleven services. I'm keeping the seam visible so that if one of these ever needs to leave, it can leave without a six-month untangling.

The general version, and the reason this is worth 150 lines of test:

Every architectural rule you can state in a sentence should be a failing test before it is a paragraph in a README. If you can't make it fail, you haven't decided anything — you've expressed a hope.

What's next

Rule four is written down but not yet enforced: every entitlement decision must route through one module. It needs classes to exist before it means anything, so that one genuinely does wait. When it lands it'll probably be an arch() test, and that'll be the right tool by then — which is the whole shape of this post. Use reflection when there's something to reflect on. Until then, read the text.

Four Env Vars and No Foreign Key: What It Means to Actually Attach a Resource

Nasrul Hazim — Sun, 06 Sep 2026 02:20:42 +0000

TL;DR — One commit today, and it's a Markdown file. No migration, no action class, no tests. I spent the day writing down the shape of a feature instead of building it, and the thing I wrote down is embarrassingly small: an application that uses object storage should have a foreign key to that storage, not four env vars somebody typed in. Everything else in this post is downstream of that one sentence.

The gap that hides behind "it works"

Here's the setup. A deployment platform can do two things with object storage:

It can run it — provision a storage server, create buckets, mint access keys.
It can record somebody else's — an S3-compatible connection to whatever the operator already pays for.

Both work. Both have UIs. Both have tests. And neither one is connected to an application in any way.

So the actual integration procedure is: provision a bucket, open the workload's environment editor, and hand-copy four values across — endpoint, region, key id, secret. It works on the first deploy. Everybody moves on.

What that copy-paste can't do is the whole point:

It can't be gated. Nothing stops a deployment starting against a bucket that hasn't finished provisioning.
It can't be rotated. Rotate the access key and the running app breaks, with no way to answer "which apps did I just break?"
It can't be audited. There is no query that returns "which workloads use this bucket."
It can't be detached. Nobody knows the link exists, so nobody can cleanly remove it.

A copied string is not a relationship. It's a coincidence that currently holds.

This is the same shape as the bug family I wrote about yesterday, one level up: the system took something it did know — this app uses that bucket — and stored it somewhere it couldn't reason about. Not a wrong fact. A fact with no home.

Copy the shape you already solved

The useful thing about this gap is that it isn't new. Managed databases went through it months ago and came out the other side with a shape: one action, nullable foreign keys on the deployment workload, environment variables written per language preset, and a gate that refuses to deploy against a database that isn't ready.

So the storage plan mirrors it. Deliberately, and almost literally.

That's a decision worth stating out loud, because "just do it the same way" is not automatically right. You inherit whatever's slightly wrong with the original, and you make the two things harder to evolve independently later. I took that trade knowingly. The alternative — a second vocabulary for the same concept, sitting one tab away from the first — is how a platform starts to feel arbitrary to the people operating it. When two features answer the same question ("how do I connect this app to that thing?") they should answer it with the same words.

Source	Credentials	What the platform creates
Managed	minted per deployment	bucket + access key + permission
External	operator's, already stored encrypted	nothing — it names an existing bucket

Two sources, one attach. Both end in the same place: env vars on the workload, and a foreign key so the link is a fact rather than a string.

Nullable columns, and where the rule lives

The schema part is dull, which is a good sign:

Schema::table('deployment_workloads', function (Blueprint $table) {
    $table->foreignId('storage_bucket_id')->nullable()->constrained();
    $table->foreignId('storage_access_key_id')->nullable()->constrained();
    $table->foreignId('object_storage_connection_id')->nullable()->constrained();
    $table->string('storage_prefix')->nullable();
});

Managed and external are mutually exclusive — you attach a bucket the platform manages, or you name one it doesn't, never both. The interesting question is where that rule lives.

I'm putting it in the action, not in a CHECK constraint:

if ($bucket && $connection) {
    throw StorageAttachmentInvalid::bothSourcesGiven();
}

The honest counter-argument first, because it's a real one: a database constraint is the only rule that survives a seeder, a console command, or somebody in tinker at 2am. Application-level invariants are invariants right up until code writes around them.

I still went with the action, for two reasons. The rule is about intent, not data integrity — two populated columns isn't corrupt data, it's an ambiguous request, and it deserves an error message that says so rather than a constraint violation. And the platform targets several database engines, so a CHECK here means either a portability problem or the same rule written five ways. When the constraint would be lying about why it exists, I'd rather it live where the reason is readable.

If you disagree, add the constraint too. Belt and braces is fine. What isn't fine is having the rule in neither place because you assumed the form would enforce it.

The secret you can read exactly once

This one shapes the whole flow, and it's easy to miss until you're halfway through building.

A well-behaved object storage engine hands you the secret key at creation and never again. It doesn't keep the plaintext. There's no "show secret" endpoint to fall back on, because there's nothing to show.

Which means the moment of creation is the only moment you can persist it. Practically:

final class ProvisionDeploymentStorageAction
{
    public function __invoke(Deployment $deployment, Workload $workload): array
    {
        $bucket = ($this->createBucket)($deployment, $this->bucketName($deployment, $workload));
        $key    = ($this->createAccessKey)($bucket);   // ← plaintext exists only here

        ($this->attachPermission)($bucket, $key, Permission::ReadWrite);

        return [
            'bucket' => $bucket,
            'key'    => $key,
            'env'    => $this->env($bucket, $key),     // written into encrypted env now
        ];
    }
}

Two consequences fall straight out of that:

You can't design an "attach later, fetch the credentials then" flow. There is no "then". If your UI implies one, you'll discover it the first time an operator tries to re-attach a bucket whose key was minted last week.

No daemon calls in the action. Rows first; queued jobs make them real. The action's job is to record the intent atomically — provisioning a bucket over the network inside a request is how you get a half-created attachment when the connection blips.

And the bucket name is derived, not free text, through a small identifier value object. Object storage naming is stricter than people expect — lowercase, digits, hyphens, dots; no uppercase, no underscores. Deriving it from the deployment and workload slugs means the name is reproducible, which matters more than it sounds like, for reasons in a moment.

Real values, not template tokens

For services that live inside the deployment, env vars can be {{ tokens }} resolved against a service registry at deploy time — the database host isn't known until the thing is placed.

Object storage isn't like that. It lives outside the deployment. There's no registry entry to resolve against, so the values go in real, at attach time. Same reasoning the database attach already uses.

Per language preset, because the ecosystem conventions differ:

laravel  → AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION,
           AWS_BUCKET, AWS_ENDPOINT, AWS_USE_PATH_STYLE_ENDPOINT=true
node     → S3_ENDPOINT, S3_REGION, S3_BUCKET, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY
python   → AWS_* (boto3 reads the same names)

Two traps worth writing down before you hit them:

Path style vs virtual-host style. Self-hosted engines generally want use_path_style = true (https://host/bucket/key). Hosted providers usually want virtual-host style (https://bucket.host/key). Get it wrong and you get a 403 or a DNS failure that reads like a credentials problem. For an external connection this has to be whatever the operator recorded, not whatever your managed default is.

Build the endpoint through a helper, never string concatenation. An IPv6 literal needs brackets in a URL:

// "http://fd00::1:3900" — silently wrong, the port isn't a port
$endpoint = "http://{$host}:{$port}";

// "http://[fd00::1]:3900"
$endpoint = $publishedHost->authority();

That's the kind of bug that only shows up on one operator's network, six weeks after you shipped.

The gate: don't start something that can't work yet

The deployment gets a check beside the existing managed-database one, with the same three answers:

The row is gone — the bucket was deleted out from under the attachment.
The row failed — provisioning errored; surface the recorded error, don't just say "not ready".
The row is pending — it's coming, but it isn't here yet.

public function managedStorageGate(): ?string
{
    $bucket = $this->storageBucket;

    return match (true) {
        $this->storage_bucket_id && ! $bucket => 'The attached bucket no longer exists.',
        $bucket?->status === StorageResourceStatus::Failed
            => "Storage provisioning failed: {$bucket->last_error}",
        $bucket?->status !== StorageResourceStatus::Ready
            => 'Storage is still being provisioned.',
        default => null,
    };
}

The value of a gate is entirely about where the failure surfaces. Without it, the deployment starts, the app runs happily for however long, and then the first user upload gets a 403 from an endpoint that doesn't have that bucket. The symptom lands in application logs, hours later, nowhere near the cause. With the gate, the deploy refuses at the point where the missing thing is named.

it('refuses to deploy when the attached bucket is not ready', function () {
    $workload = DeploymentWorkload::factory()
        ->withManagedBucket(StorageResourceStatus::Pending)
        ->create();

    expect($workload->managedStorageGate())->toContain('still being provisioned');
});

it('reports the recorded error when provisioning failed', function () {
    $workload = DeploymentWorkload::factory()
        ->withManagedBucket(StorageResourceStatus::Failed, error: 'no space on device')
        ->create();

    expect($workload->managedStorageGate())->toContain('no space on device');
});

Detach must never delete

The hardest rule in the whole plan, and the shortest:

Data outlives deployments.

Detaching a bucket removes the link and the env vars. It does not touch a single object. Someone tearing down a staging deployment at 6pm on a Friday must not be able to delete a year of uploads by clicking a button labelled Detach.

Which is why the derived bucket name matters: re-attaching finds a live bucket under the name it would have generated and reuses it rather than refusing. Detach, re-attach, and you're back where you were, with your files. If the operator genuinely wants the data gone, that's a separate, louder, differently-worded action.

The same idempotency thinking as treating a vendor 404 as success on delete: a repeated operation has to be able to finish.

Rotation is the part that bites

Key rotation already exists here: mint a successor, keep the old key valid for an overlap window so nothing dies mid-request.

The overlap is the safety mechanism, and it's also the thing that makes the bug easy to miss. Rotate, everything keeps working, ship it. Then the overlap expires and apps start failing — at a moment with no deploy, no code change, and nothing in the timeline to point at.

So completing a rotation has to walk the attachments pointing at the old key and update them. Otherwise the platform is scheduling its own outage, on a delay, for a reason its own logs won't explain.

One more bit of honesty owed to the operator: updating a stored env var doesn't change the environment of a process that's already running. The new value reaches the app on the next deploy. The toast has to say "redeploy to apply" rather than implying it's live — a UI that overstates what just happened is how you get someone confidently debugging the wrong layer.

The bit I got wrong first: knowing what not to model

This plan exists because of a wrong turn.

The new-application wizard briefly offered "uploads on their own storage server" as a component you could compose into an app's blueprint, alongside the runtime and the database. It looked consistent. It was wrong twice over.

The shallow reason: the specific engine I reached for went into maintenance mode upstream, so the platform's own unsupported-components check rejected it. Fair enough — that's a dependency risk, and the validator did exactly its job.

The reason that actually matters: object storage isn't a per-application component at all. It reads its own config file and owns its own cluster layout. A per-app layer can't own that — you'd be asking one application's blueprint to hold configuration that belongs to a service several applications share.

That's a useful test for any composable system:

If a thing owns its own config file and its own cluster identity, it's a peer service you attach to, not a component you compose in.

Getting that wrong doesn't produce an error. It produces an abstraction that works for the first case, then quietly fails the second — the moment two applications want the same bucket, or the storage needs a setting that isn't any single app's business.

The fix wasn't to make the component work harder. It was to delete it and model the relationship instead.

What I left out on purpose

Two things, both tempting:

Migrating existing objects into a new bucket. There's no safe general answer — prefixes differ, sizes differ, and a half-finished copy loses files. A feature that sometimes loses uploads is worse than no feature.

Multiple buckets per workload. One bucket, one key, one prefix. The second bucket arrives with the first real request for one, not on the strength of me imagining it at design time.

The takeaway

A plan is the cheapest place to be wrong. Today's entire output is one Markdown file, and it already caught one bad abstraction, one dependency risk, and one silent-outage path in the rotation flow. Those are much more expensive to find in a migration.

The question I'd take to any integration in your own system: what happens when the credential rotates?

If the answer involves a human remembering which apps to update, you don't have an integration. You have four env vars and no foreign key.

Unknown Is Not Failed: Four Bugs Where My Control Plane Claimed to Know Something It Didn't

Nasrul Hazim — Sat, 05 Sep 2026 01:47:26 +0000

TL;DR — Nineteen commits into a deployment control plane today. Reading them back, four of them are the same bug wearing different clothes: the system took "I could not find out" and stored it as "I found out, and the answer is bad." A timed-out health probe became a failed workload. An IP address appearing became a machine being reachable. A cached object became a class that would still resolve later. A record being retired became a machine that had stopped billing. Different subsystems, same missing distinction — absence of evidence recorded as evidence of absence.

The setup

I work on a control plane: software whose entire job is to hold an opinion about the state of machines and workloads it doesn't run in-process. Everything it knows, it knows by asking something else over a network — SSH, a vendor API, a container runtime — and every one of those questions can fail to produce an answer.

Which means a control plane has three possible outcomes per question, not two:

It's healthy.
It's broken.
I don't know.

Every bug below is what happens when option three gets folded into option two — or, worse, into option one.

1. A timed-out probe is not a failed workload

The one that made me sit up. The control plane's own application was showing Failed on a page it was, at that moment, successfully serving to me. Its service unit was active running continuously. A root shell on the box couldn't reproduce a single failing read. And yet roughly one health sample in three was recording a 100% error rate.

Here's the shape. Each runtime driver — the systemd one, the Docker one — implements a WorkloadRuntime contract and probes the workload's state. Each one wrapped the probe like this:

public function state(Workload $workload): WorkloadState
{
    try {
        return $this->probe($workload);
    } catch (Throwable) {
        return WorkloadState::Pending;   // ← the lie starts here
    }
}

And the only consumer — a scheduled sampler — did this:

$state = $runtime->state($workload);

$this->record($workload, $state === WorkloadState::Running
    ? Outcome::Healthy
    : Outcome::Failed);          // ← and gets laundered into truth here

Read those two together. An SSH connection that took one second too long returns Pending. Pending is not Running. Not Running maps to Failed. So a slow minute on the network becomes a red badge next to a site answering 200, plus a row in the health log claiming a 100% error rate.

That's already bad as a dashboard bug. It got genuinely dangerous one layer up, because automatic rollback reads those same rows. A healthy release could be rolled back on the strength of a connection that timed out. The safety mechanism was being fed fiction.

The fix is a third channel, not a third enum case:

final class WorkloadStateUnreadable extends RuntimeException
{
    public static function for(Workload $workload, Throwable $previous): self
    {
        return new self(
            "Could not read state for workload {$workload->uuid}.",
            previous: $previous,
        );
    }
}

The driver now throws it where the reading failed, and never where the runtime answered:

public function state(Workload $workload): WorkloadState
{
    try {
        $unit = $this->ssh->run("systemctl show {$workload->unit} --property=ActiveState,LoadState");
    } catch (Throwable $e) {
        throw WorkloadStateUnreadable::for($workload, $e);   // I don't know
    }

    return match ($this->activeState($unit)) {
        'active', 'reloading' => WorkloadState::Running,
        'failed'              => WorkloadState::Failed,
        ''                    => throw WorkloadStateUnreadable::for($workload, ...),
        default               => WorkloadState::Pending,
    };
}

Two details in that match are worth more than the exception itself:

reloading is Running. A unit taking new config is serving traffic the whole time. It was falling through to the default arm and turning into a false red.
An empty ActiveState is unreadable, not a state. Empty string is output that never arrived. Treating it as "some state I don't recognise" is exactly the category error the whole commit is about.

And the flip side, which is the part people get wrong when they first reach for this pattern: LoadState=not-found stays Pending. A missing systemd unit, a missing Docker container, a missing Kubernetes deployment — those are answers. The runtime replied. The reply was "it isn't here." That's knowledge, and it belongs in the state enum, not in the exception.

The sampler needed no new branch, because it already had the right instinct in a catch that was never being reached:

try {
    $state = $runtime->state($workload);
    $this->record($workload, $state);
} catch (WorkloadStateUnreadable) {
    // Record nothing. Not enough evidence.
}

A gap in the sample history is honest. A row saying "100% errors" is not.

it('records nothing when the runtime cannot be read', function () {
    $runtime = Mockery::mock(WorkloadRuntime::class);
    $runtime->shouldReceive('state')
        ->andThrow(WorkloadStateUnreadable::for($this->workload, new RuntimeException('timeout')));

    app(DeploymentHealthSampler::class)->sample($this->workload);

    expect(DeploymentHealthLog::count())->toBe(0);
});

it('treats a missing unit as pending, not unreadable', function () {
    $this->ssh->respondsWith('ActiveState=inactive'.PHP_EOL.'LoadState=not-found');

    expect($this->runtime->state($this->workload))->toBe(WorkloadState::Pending);
});

One deliberate non-fix: I left the Kubernetes driver alone. Its probe surfaces a missing deployment through the same failure path as a broken kubectl, and telling those two apart means matching on stderr strings I can't verify against a live cluster right now. Fixing it by guessing would replace a known-wrong classifier with an unknown-wrong one. Leaving a documented gap beats shipping confidence I don't have — which, you'll notice, is the same principle as the bug.

2. An IP address is not a reachable machine

Every first-time setup of a freshly provisioned machine failed on Connection refused, and only a manual Retry ever got past it.

The wait job's completion condition was: the cloud provider has reported an address for this machine. Reasonable-sounding. Wrong. The address is published tens of seconds before sshd actually binds to the port. The provider is telling you the truth about the allocation; you're reading it as a claim about the daemon.

So the wait now waits for the thing it actually depends on:

final class SshPortProbe
{
    public function accepts(string $host, int $port = 22, float $timeout = 3.0): bool
    {
        $socket = @fsockopen($host, $port, $errno, $errstr, $timeout);

        if ($socket === false) {
            return false;
        }

        fclose($socket);

        return true;
    }
}

Released every 10 seconds against the same overall deadline the job already had, so this tightens the condition without inventing a new timeout budget.

The important line in that class is @fsockopen, not for the suppression but for what it doesn't do: it does not authenticate. That was a real fork in the road. It's tempting to make the readiness check a full SSH handshake — surely a better signal? No. If the credential is wrong, that's not a race, it's a misconfiguration, and it must fail loudly at the bootstrap step in five seconds rather than get retried politely for ten minutes. A readiness probe that also validates config will happily spend your entire deadline on a problem that will never resolve.

Readiness probes answer "can I start yet?". They should never be answering "will it work?".

3. A cached object is not a class that will still resolve

This one cost me the most time, because the stack trace pointed three frames away from the cause.

Picking a region on the New Server page threw:

Argument #1 ($s) must be of type CloudSize, __PHP_Incomplete_Class given

__PHP_Incomplete_Class is what unserialize() hands back when the class named in the payload cannot be resolved at the moment the payload is read. It doesn't throw. It doesn't warn. It returns an object that satisfies no type declaration anywhere in your codebase, and lets that object travel until something eventually type-hints against it. So the explosion happens in a fn (CloudSize $s) => ... callback, and the cache write that caused it is nowhere in the trace.

Why couldn't the class resolve? Because on this platform it genuinely might not, and I'd never thought about it:

Every release installs into its own directory tree.
composer install --optimize-autoloader writes a classmap of absolute paths into that tree.
Old releases get pruned.
A long-lived PHP-FPM worker can be holding a classmap pointing into a directory that no longer exists — while the shared cache still holds a payload naming a class in it.

So this isn't a Laravel bug or a cache-driver bug. It's a structural consequence of atomic-release deploys: the process that writes a cache entry and the process that reads it are not guaranteed to agree on what classes exist.

I could have fixed the symptom — flush on deploy, version the cache key, guard with instanceof. I went structural instead, because the rule is easy to hold in your head and impossible to half-apply:

Caches hold arrays of scalars. Never objects.

Which, with promoted constructor properties, is about six lines per DTO:

final class CloudSize
{
    public function __construct(
        public string $slug,
        public int $vcpus,
        public int $memoryMb,
        public CloudFamily $family,
    ) {}

    public function toCache(): array
    {
        return [...get_object_vars($this), 'family' => $this->family->value];
    }

    public static function fromCache(array $data): self
    {
        return new self(...[
            ...$data,
            'family' => CloudFamily::tryFrom($data['family']) ?? CloudFamily::Unknown,
        ]);
    }
}

Two things I'd point at in a review:

get_object_vars() works here only because every constructor parameter is promoted and public. Spreading a string-keyed array back into the constructor passes them as named arguments, so the round trip is symmetric for free. Add one non-promoted property and this silently drops it. That's a real fragility — it's bought deliberately, in exchange for not hand-writing four mapping methods per DTO, and it's the kind of thing that deserves a comment rather than a shrug.

tryFrom, not from. A backed enum is an object too, so it has to travel as its scalar value. But the vendor can add a new machine family tomorrow, and from() would turn "a distribution I haven't heard of" into a fatal on a cached read — which would be this exact bug class again, one level down. An unknown value is not an invalid one.

4. A retired record is not a stopped bill

The last one isn't a state-reading bug, but it's the same family, and it's the one with an invoice attached.

Two machines were retired from the fleet and went on running — and on billing — because the cloud provider contract shipped with no destroy path at all. That absence was deliberate, and I want to be fair to the reasoning, because it was mine: destroy was sequenced last in the plan, and "last" was enforced by the interface simply not having the method, rather than by anyone remembering.

That's usually a good technique. Make the unsafe thing unrepresentable and you can't do it by accident. But here it produced a platform that can spend a customer's money and cannot stop spending it — and it hid that, because "retire" removing a record looks like the operation finished. The record moving is not the machine stopping.

So destroy landed, and the safety a missing method used to buy got re-bought explicitly:

Provenance decides. Only a machine the platform itself created at a connected account is eligible for deletion. A machine someone adopted by typing in an address is refused — the platform is not what made it exist, and that address might be somebody's employer's VM. The option is absent on the page for an adopted machine, not disabled. Disabled invites a support ticket asking how to enable it.

final class RetireManagedNodeAction
{
    public function __invoke(ManagedNode $node, bool $destroyMachine = false): NodeRetirement
    {
        if ($destroyMachine && ! $node->provider->isCloudCreated()) {
            throw NodeNotDestroyable::notCreatedHere($node);
        }

        if ($destroyMachine) {
            $this->cloud->destroy($node);   // vendor call FIRST
        }

        return $this->retireRecords($node);
    }
}

Opt-in everywhere, defaulting to the old behaviour. The UI requires the machine's address retyped. The equivalent MCP tool takes destroy_machine, defaulting to false, on top of the scope and confirmation it already required.

The vendor call runs before any record moves. This ordering is the whole design. If the API token is missing the delete scope, a 403 retires nothing, and the operator retries against records that still describe reality. Move the records first and a failed vendor call leaves you with a machine nobody is tracking and everybody is paying for.

A vendor 404 is success. Already-gone is the desired end state. A retry has to be able to finish, not die on the absence of the thing it was trying to remove. Idempotency here isn't a nicety; without it the only way out of a partial failure is manual database surgery.

it('refuses to destroy a machine it did not create', function () {
    $node = ManagedNode::factory()->adopted()->create();

    expect(fn () => app(RetireManagedNodeAction::class)($node, destroyMachine: true))
        ->toThrow(NodeNotDestroyable::class);

    expect($node->fresh()->retired_at)->toBeNull();   // nothing moved
});

it('treats an already-deleted machine as destroyed', function () {
    $this->cloud->respondsWith(404);

    $result = app(RetireManagedNodeAction::class)($this->node, destroyMachine: true);

    expect($result->destroyed)->toBeTrue();
});

The takeaway

If your system's job is to hold an opinion about something it can only reach over a network, then "I don't know" is a first-class outcome and needs somewhere to live. Not a nullable field you forget to check. Not a catch that returns a plausible-looking default. Somewhere the type system makes you handle.

A useful audit, and it took me about twenty minutes across this codebase: grep for catch blocks that return instead of throw. Every one is a place where a failure to find out is being converted into a finding. Some of those are correct and considered. Some of them are quietly feeding your alerting, your dashboards, and — in my case — your automatic rollback.

The general shape, stated once:

What actually happened	What the system recorded	Cost
The probe timed out	The workload failed	Rollback of a healthy release
An address was allocated	The machine was reachable	Every first setup failed
An object was cached	The class will resolve on read	Intermittent 500s, no trace
A record was retired	The machine stopped	An invoice that never stops

Three of those four were reported to me as flaky. None of them were flaky. They were all deterministic consequences of a missing third case.

What's next: the Kubernetes probe still can't tell "no such deployment" from "kubectl is broken", and I'd rather fix that against a real cluster than against my assumptions about stderr formatting. Same principle — don't ship certainty you haven't earned.

Dev Log: 3 September 2026 — Everything I Fixed Today Was Already Written

Nasrul Hazim — Fri, 04 Sep 2026 02:40:37 +0000

Six commits across two codebases today — a deployment platform and an identity system. Different stacks, different problems. Reading them back in one sitting, the same shape six times:

The feature was written. Someone just couldn't reach it.

Not one of these was a missing implementation. The role management existed. The setup-command runner existed. The delivery tracking existed. The "already verified" message existed. All of it shipped, tested, working — and unreachable, because of a default, a branch, a missing entry point, or a token that had already been spent.

That's a genuinely different category of bug from "we haven't built it yet", and it's much harder to see, because every code review of the feature passes. The feature is fine. The path to it isn't.

The badge in the `@else`

Start with the smallest one, because it's the clearest.

A team page shows members and lets an owner change their roles. The component had addMember, changeRole and removeMember all along. But the page read as though role management didn't exist.

The badge naming a member's current role was inside the @else branch of @can('manageMembers').

Sit with what that means. A plain member saw everyone's role and could change nothing. An owner could change any role and never saw what it currently was — and the dropdown offered "Set as Lead", "Set as Member", "Set as Viewer" identically, with no marker on the one already true. The page never named a role to the one person allowed to act on it.

{{-- Before: the person with the power was the one kept in the dark --}}
@can('manageMembers', $team)
    <x-dropdown>...</x-dropdown>
@else
    <x-badge>{{ ucfirst($member->pivot->role) }}</x-badge>
@endcan

{{-- After: the badge is information, not a permission --}}
<x-badge>{{ $roleLabels[$member->pivot->role] }}</x-badge>
@can('manageMembers', $team)
    <x-dropdown>...</x-dropdown>
@endcan

Two follow-ons worth naming:

The label came from ucfirst() on a raw pivot value. Untranslated, and it disagreed with the labels in the dropdown sitting right next to it, which came from the enum's label(). If you've got an enum with label()/color(), that enum is the only place a role should ever be turned into words. I pass a [value => label] map in from render() rather than resolving the enum in the Blade file — an inline @php() holding a nested static call is its own landmine in a component view.

Three of four confirmations were arriving a page late. The toast component lives in the layout and pulls session toast.* keys as it renders. So a session()->flash() from a Livewire action that doesn't redirect surfaces on the next full page load — which, for addMember, removeMember and changeRole, means never at the moment it meant anything. Those dispatch a browser event now. delete() still flashes, because it redirects, and that's what the flash mechanism is actually for.

If you're mixing Livewire actions and a layout-level toast: flash is for redirects, dispatch is for stays. Getting that backwards produces a bug where the user's action appears to do nothing and then a stale confirmation appears somewhere unrelated.

And a third: removeMember() refuses to remove the organisation owner, and reported that refusal via addError('memberEmail') — a field on the add member form. So clicking Remove put a validation error on an input the operator wasn't touching, and said nothing about what actually happened. It's a dispatched error toast now, and the menu item is disabled for the owner instead of offered and then refused. The server-side guard stays, obviously: a disabled control is not an authorisation check.

The runner with no door

Same shape, one level up.

There's an action and an MCP tool for running an application's declared post-install commands. Both have existed for weeks. What didn't exist was any way to reach them from the application's page in the UI.

So the workaround for "I need to run a seeder on this host" had become: push an empty commit to trigger a deploy. A workaround with a commit in it. That's the tell that something's unreachable rather than unbuilt — the workaround is absurd and everyone's fine with it.

The panel I added is mostly wiring, but two decisions in it are worth writing down.

The list of commands is read from the same config the deploy reads. Not a copy. Two lists drift, and the drift doesn't arrive as a diff — it arrives as "it worked when I ran it by hand and failed on deploy", six weeks later, on someone else's shift.

The refusal order matters, and it's the opposite of the obvious one. My deployment policy already refuses while an application is provisioning or being destroyed. So authorizing first handed an operator who is mid-deploy a bare 403 — for something that is not a permission problem at all. The status check runs first now, and says what's actually true:

public function __invoke(RunSetupRequest $request, Application $application): RedirectResponse
{
    // Status BEFORE authorization: a mid-deploy operator is not
    // unauthorized, and telling them so sends them to the wrong person.
    if (! $application->status->canRunSetup()) {
        return back()->with('error', $application->status->setupBlockedReason());
    }

    $this->authorize('runSetup', $application);

    // ...
}

The usual objection to ordering it this way is information disclosure — you've told an unauthenticated-ish caller something about the resource before checking their rights. Worth checking every time, and here it's fine: the page is already tenancy-scoped, so anyone who can read that status could read it from the deployment list beside it. If that weren't true, authorize first and accept the worse error message. That trade-off is real, and it goes the other way often enough that it's worth naming rather than assuming.

Two smaller things:

The output is kept and shown. The reason an operator reached for this is that the app was refusing to start. "Setup completed" with no output leaves them exactly as unable to distinguish a seeded database from a silently skipped seeder as they were before.
The force checkbox carries a sentence, not the word "force". These commands are first-install by nature and mostly not idempotent — a seeder calling User::create() fails the second time, and re-provisioning a tenant database is worse than failing. The label says that.

It runs what the application already declared, in order, once. It is not a shell and not a step toward one. An arbitrary-command surface on a customer's production node is a different feature with a different threat model, and I'd want to design it as one.

The tracking that was complete, and off

Over in the identity system, every message on the mail-history detail screen read "No delivery events recorded" — including the account-verification mail whose delivery we're asked to prove more often than any other.

The package was complete. The defaults weren't. The config declared tracking as env('MAIL_HISTORY_TRACK_OPENS', false), and production's .env carried neither key. So the tracking routes were never registered and no pixel was ever injected. I probed it live before touching anything: both the open and click endpoints answered 404.

// config/mailhistory.php
'track_opens'  => env('MAIL_HISTORY_TRACK_OPENS', true),
'track_clicks' => env('MAIL_HISTORY_TRACK_CLICKS', true),

Flipping a default is a one-line diff and a real decision. There is deliberately no UI switch for it — whether an activation email was opened is not an operator preference, it's an audit question. And defaulting to true means no environment needs an .env edit to be correct, which matters when the environment you most need it in is the one you touch least.

The consequence you have to think about: click rewriting genuinely starts running in production for the first time. So Laravel's signed email-verification URL joins the exclusion list, because its ?expires=...&signature=... is exactly the escaped-ampersand round trip that breaks under rewriting — and a broken signature locks an account out of verifying itself. Turning on a feature that has never run in production is a deploy, not a config change.

Two counting rules that went into the statistics service, both of which are the kind of thing that silently produces a plausible wrong number:

Engagement is counted from the events table, never from status. status holds only the latest state, so a click hides the open that preceded it. Count events, not states.
Count distinct messages, not event rows. One recipient opening four times must not push an open rate past 100%. If a percentage can exceed 100, the denominator is wrong.

Then a set of surfaces on top: summary cards computed inside the Livewire component so they follow the active filters instead of silently contradicting them; an empty state that distinguishes "nobody opened it" from "nothing was ever measured" (they used to render identically, which is how the original bug hid for so long); a report page with period, status and origin breakdowns; and three MCP tools behind the same permission as the screens.

The MCP tools never return a message body or a clicked URL — only the destination host. Verification and reset links live in there, and a read-only reporting tool is not a place to hand them out.

Every surface now reports the tracking state next to the figures. A number whose collection was off for an unknown period is not a number.

The token that was already spent

The best one, and the one with real user pain attached.

A personal-email verification link works once. The token is nulled the moment it's spent — correctly, that's the point. But the lookup is by that token. So a second click on a link that had worked perfectly fell into the Invalid or expired verification link branch. The already_verified branch sitting right below it was unreachable by construction.

$subject = Subject::where('verification_token', $token)->first();

if (! $subject) {
    return $this->invalidLink();   // ← every second click lands here
}

if ($subject->personal_email_verified_at) {
    return $this->alreadyVerified(); // ← unreachable: token is gone by now
}

Users whose accounts had verified flawlessly were being told their account was broken. That's a support ticket, then six more.

Nulling the token stays. What changes is the response:

Post/Redirect/Get. On success it redirects to a landing route carrying the subject id in the session, so a refresh no longer replays a spent token URL. Verification is a state change; it should not be sitting behind a re-runnable GET in someone's history.
An unresolvable token redirects to that same landing page, which reports "Email Already Verified" rather than a red failure. And here's the honest part: once the token is gone, the two cases genuinely cannot be told apart. A never-issued token and a spent one look identical. So the page says what's true for the overwhelming majority, and keeps a quiet "never verified?" hint for the rest. When you can't distinguish two cases, pick the message that's right most of the time and leave a door for the exception — don't show a failure page to be safe.
The mail no longer claims the link "will expire in 24 hours". There's no expiry column, and the verification never checked the token's age. The claim was fiction — and it's precisely what made the old failure page read as plausible. A user clicks their link twice, sees "invalid or expired", remembers the email said 24 hours, and concludes the system is right and they're late. A false reassurance in an email is what turns a confusing error into a believed one.
The two dead failure views are deleted, along with the branch.

The one that wasn't already written

One bug today was mine, fresh, and shipped an hour earlier: a baseline of security headers I'd added to every generated nginx vhost was appending rather than deferring, so applications that set their own X-Frame-Options: DENY were answering with DENY and SAMEORIGIN, which browsers read as neither.

That one got its own post, because the nginx mechanics deserve the space — "nginx add_header appends. It doesn't override." (companion post, link at review time). Short version: add_header can't ask whether the upstream already sent a header, a map on $upstream_http_* can answer before you ask, and a security header added carelessly is a downgrade rather than an addition.

Worth noting how it was found, though, because it fits today's theme from the other side: I went and checked the live response headers right after claiming they were proven. The test suite was green. The suite verified the config generator, which was working exactly as written. Nothing in CI could have caught it, and I'd have believed CI for weeks.

Takeaway

Six bugs, one shape. If I had to write the search query for tomorrow:

Anything inside an @else. Ask who's in the other branch and what they lose.
Any branch that can't be reached given how the state above it is mutated. The already_verified check was dead code that looked like a safety net.
Any env(..., false) default in a config for a feature you assume is on. Then go probe the endpoint in production instead of assuming.
Any action reachable only from an MCP tool, a console command, or a deploy. If the workaround involves an empty commit, the door is missing.
Any two states that render identically. "No events recorded" meaning both nobody engaged and nothing was measured is how an off switch hides for months.

None of these show up as failing tests, because the code under test is correct. They show up when you use the thing as the person it was built for — which is the cheapest verification step available and the one easiest to skip.

My Agent Skills All Said How to Start. None Said When to Stop.

Nasrul Hazim — Fri, 04 Sep 2026 02:37:55 +0000

TL;DR — I read someone else's skill library, went back to my own 32 skills, and found every one of them was a tutorial. They all explained how to begin a task. Not one explained how to know you were finished, or what a wrong turn looks like on the way. Claude Toolkit 2.5.0 adds three sections — Common Rationalizations, Red Flags, Verification — to the nine skills where being wrong is expensive.

The gap

A skill file for a coding agent is a strange document. It's not documentation, because nobody reads it for reference. It's not a prompt, because it's loaded conditionally. It's closest to a playbook you hand a competent contractor on their first day: here's how we do this, here's what we care about, here's what will get you a phone call.

Mine were only ever the first third of that. Take the deploy skill before this release. It knew the deploy command, the SSH flow, the order of operations, how to run migrations. Everything about starting.

It said nothing about the moment that actually matters, which is when the agent — or a tired human — decides the deploy is done. bin/deploy exited 0. Is that success? The skill had no opinion. So the answer became whatever the model felt like, which on a good day is thorough and on a bad day is "the command exited 0, deploy complete."

That's not a model problem. That's a missing section.

Three sections, and why each exists

I added the same three headings to nine skills. Each one targets a different way work goes wrong, and the order matters — they run at different moments.

`## Common Rationalizations` — the arguments you'll make to yourself

A table. Left column: the excuse. Right column: what's actually true.

This is the one I expected to feel patronising and it's turned out to be the most useful. Bad engineering decisions are rarely made in ignorance. They're made with a reason — a sentence that sounds fine when you say it in your head at 6pm. Writing the sentence down next to its rebuttal takes away the "well, in this case…" move, because the case is right there in the table.

From the code-quality skill:

Rationalization	Reality
"Just baseline it, it's legacy code"	A baseline entry is a permanent, invisible ignore. Baseline to unblock CI, then file the issue — a baseline that only ever grows is a broken quality gate.
"Level 5 is good enough for now"	Levels only ever ratchet up when someone forces them. "For now" has been the level for two years in most codebases.
"Pint fixed it, so the code is reviewed"	Pint fixes spacing and ordering. It has no opinion on the god class it just reformatted.
"Adding `@var` fixed the error"	It silenced the error by lying to the analyser. If the annotation is wrong, you have made the codebase less safe, not more.

That first row cost me a real bug this week, in a different repo. Static analysis had flagged two classes as used zero times. Someone baselined the warning rather than deleting the files — and one of those "unused" classes was supposed to be doing PII redaction. The baseline hid it for months.

The rebuttal has to be specific and mechanical, not moral. "Don't be lazy" changes nothing. "A baseline that only ever grows is a broken quality gate" is an argument.

`## Red Flags` — signals visible during the work

A flat list of observable things. Not principles — sightings. If you can see it in a diff, a terminal, or a PR, it belongs here.

From the debugging skill:

sleep() or --retry added to make a test pass
The same bug fixed in three different callers in one PR
A ->skip() or ->markTestIncomplete() added during a bug fix
dd(), dump(), ray() or Log::debug('here') left in the diff
A diff that touches files unrelated to the reported failure
Blaming the framework, the database or the browser before reading your own stack trace

And from deploy-app, where the stakes are higher:

Deploying without a fresh, size-checked backup
migrate:fresh, db:wipe, or --seed typed against a production host
New .env keys added to the repo but never set on the host
A deploy that also contains a refactor
APP_KEY in the diff

Notice the second entry in the debugging list. "The same bug fixed in three different callers" is not a style complaint — it's near-proof that the fix is at the wrong layer. That's the value of the format: a red flag encodes a diagnosis, compressed into something you can spot rather than something you have to reason about.

`## Verification` — the exit gate

A checkbox list run before the work is called done. Not a summary of the skill; a set of conditions that can each be false.

## Verification

Before calling a bug fixed:

- [ ] The original failure reproduces on demand (or its impossibility is documented)
- [ ] The root cause is stated in one sentence, and it is a cause, not a symptom
- [ ] Every caller of the changed function was checked, not just the reported path
- [ ] A regression test exists, and it fails when the fix is reverted
- [ ] The full test suite passes, not just the new test
- [ ] No debug output (`dd`, `dump`, `ray`, `Log::debug`) remains in the diff
- [ ] The diff contains the fix and its test — nothing else
- [ ] If the bug reached production, the blast radius is stated: who was affected, what data, over what window

"A regression test exists, and it fails when the fix is reverted." If I could keep one line out of this entire release, it'd be that one. A test that passes both with and against your fix is a test of something else. Almost nobody checks, because checking means deliberately breaking working code, and it feels like a waste of ninety seconds. It's the only thing that distinguishes a guard from a decoration.

Same energy in the last box. "The blast radius is stated" is not a coding task at all — it's the part of incident work that gets skipped because the code is already fixed and the adrenaline is gone.

Why the end of the file, not the top

The obvious objection: an agent reads the whole skill, so why does placement matter?

It matters because the failure isn't at the entry, it's at the exit. Instructions at the top of the file compete with the user's actual request and lose. A checklist under a ## Verification heading is a thing you can be asked to run — "run the verification section" — and the answer is auditable, box by box. It's a different mode: not guidance, but a gate.

That's the reframe I'd offer anyone writing skills for a coding agent:

A skill is a policy document, not a tutorial. The tutorial part is the cheap part — the model mostly knows the framework. What it doesn't know is your organisation's definition of done, and which shortcuts you've been burned by.

The happy path is where the model is already strong. The failure modes are where it needs you.

Coverage, hygiene, and the boring half

Three other things landed in 2.5.0, and two of them are only interesting as process lessons.

Two agents had no backing skill. The role personas — code reviewer, QA engineer, DevOps engineer and so on — load skills as their playbook. Two were pointing at nothing. So the release adds:

debugging — the five-step triage (reproduce, localise, reduce, fix at the root, guard), a table of Laravel-specific traps (N+1 masked as a timeout, queue jobs failing silently on the wrong connection, config cache serving stale values, Livewire state desync), flaky-test diagnosis, and two reference files.
security-hardening — defensive OWASP-in-Laravel, authorization coverage auditing, uploads, secrets, production config, CI/CD hardening. It's the proactive counterpart to the reactive soc-analyst and log-monitor pair I already had.

Step 1 of that triage is blunter than I'd usually write:

You have not started debugging until the failure happens on demand. … If you cannot reproduce it, say so plainly and stop. A fix for a bug you never saw fail is a guess wearing a diff.

My own convention file was being violated by my own skills. The repo's CLAUDE.md says every skill ships a references/ directory and a Reference Files table. Two skills had neither, and both were orphaned — no agent loaded them. Fixed, and both are now wired to an agent.

That's the unglamorous lesson and it applies well beyond agent skills: a convention nobody checks is a preference. Mine had drifted in the exact way today's other repo drifted, where a hand-maintained SECURITY.md mirroring composer.json had gone wrong in every single entry.

Companions, installed not vendored. Two external tools are now referenced from the skills that benefit — a codebase knowledge-graph tool for research and architecture work, and a minimalism-ladder plugin for the code-review and refactoring skills. Both are installed from upstream, never copied into the repo. Both are optional: the installer reports a skip when prerequisites are missing, and every referencing skill states its fallback, so a missing companion degrades the skill rather than breaking it. New --no-companions flag for anyone who'd rather not.

Copying someone else's tool into your repo is how you end up maintaining a fork of it by accident.

And a smaller thing that changed every response

The default output format is now TLDR + tables, written as a delimited managed block into ~/.claude/CLAUDE.md so it applies to every prompt. Replaced — never duplicated — on reinstall, and the user's own instructions are untouched.

A new /output command switches mode per session: tldr, table, verbose, bullets, narrative, json, raw. Three rules keep it from becoming a footgun:

A one-off instruction in the prompt always beats the session mode. "explain in detail" wins for that reply, then the mode returns.
Never drop required content to satisfy a format. A safety caveat, a failing test, an unmet requirement or a stated assumption gets reported in every mode — tldr and json included.
tldr is not permission to omit; verbose is not permission to pad.

That middle rule is the same idea as the verification section, honestly. A format is allowed to shape the prose. It is not allowed to quietly delete the bad news.

Takeaway

If you're writing skills, playbooks, or CLAUDE.md files for a coding agent, the highest-leverage paragraph you can add isn't another example of the happy path. It's the answer to two questions:

What excuse will be made for cutting this corner, and what's the rebuttal?
What has to be true before this is done — as boxes that can each be false?

The model already knows the framework better than your docs do. What it can't know is which shortcut burned you in March.

34 skills, 20 agents, 12 commands. MIT, and it's all at github.com/nasrulhazim/claude.

Next on the list: making the nine verification sections runnable as an actual gate rather than a prompt the agent is trusted to honour. A checklist you can skip is, by today's standard, a decoration.

nginx add_header Appends. It Doesn't Override — And That Can Remove Protection.

Nasrul Hazim — Fri, 04 Sep 2026 02:35:37 +0000

TL;DR — I shipped a baseline of security headers into every generated nginx vhost, meaning it as a fallback for applications that send none of their own. Within the hour, production was answering with two X-Frame-Options values: DENY from the app and SAMEORIGIN from my baseline. Browsers treat a contradictory set as no directive at all, so a header added to increase protection had removed it. The fix is a map on $upstream_http_*, because add_header has no way to ask a question.

The setup

I maintain a system that generates nginx server blocks for applications it deploys. The applications aren't mine. Some are well-built Laravel apps that set their own security headers in middleware; some are ten-year-old PHP that sets nothing at all and is being moved off a managed hosting panel that used to set headers on its behalf.

So the requirement is easy to state: give every vhost a sane baseline, without stepping on an application that already knows what it wants.

The first cut looked like this, rendered into every server block:

server_tokens off;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

Tests green. Config valid. Deployed.

What production actually answered

I went to check the thing I had just claimed was proven — a habit that has earned its keep — and curl gave me this:

x-content-type-options: nosniff
x-content-type-options: nosniff
x-frame-options: DENY
x-frame-options: SAMEORIGIN
referrer-policy: strict-origin-when-cross-origin
referrer-policy: strict-origin-when-cross-origin

Every header, twice. The application's own middleware was setting X-Frame-Options: DENY. My baseline had added SAMEORIGIN beside it.

Here's the thing I want to be blunt about, because I think it's the part people assume their way past:

Two contradictory X-Frame-Options values is not "the stricter one wins." There is no merge rule. Browsers that receive an unparseable set of directives for that header treat the header as absent. So DENY — the strongest possible value, which the application had correctly set — became nothing, because I added a weaker value next to it.

On that particular host, Content-Security-Policy: frame-ancestors 'none' still covered the gap; CSP takes precedence over X-Frame-Options in every modern browser. But the applications this baseline was written for are exactly the ones relying on X-Frame-Options alone. For all of them, running my "hardening" left them worse off than before it ran.

Three nginx facts worth internalising

The bug wasn't a typo. It was three separate properties of add_header, none of which I had actually sat with.

1. `add_header` appends. Always.

The name is honest and I read it as if it said set_header. It adds a field to the response. It does not replace an existing one, and there is no add_header ... if_not_present flag. Think of it as $response->headers->set() versus ->add() — nginx only gives you ->add().

2. `add_header` cannot ask a question.

This is the real constraint. Inside a location, there is no expression available to add_header that means "unless the upstream already sent this." if won't help you — if inside location is famously not what you want, and it can't inspect response headers anyway, because at request-evaluation time the response doesn't exist yet.

3. `add_header` does not merge across levels — it replaces the whole set.

One add_header in a nested block silently discards every add_header inherited from the parent. Not the matching one. All of them.

This one bit me in a place I would never have looked. My generated config has a fallback location that returns a friendly 503 while an application is starting up:

location @app_unavailable {
    add_header Retry-After 5 always;
    return 503 '<!doctype html>...';
}

That single Retry-After wiped the entire inherited baseline. Which means the response an operator is most likely to see with their own eyes — the "just a moment, starting up" page during a restart — was the one response on the whole node with no protection headers on it. The fix is to repeat the set inside the location, with a comment explaining that the repetition is load-bearing:

location @app_unavailable {
    # Repeated, not inherited: nginx replaces the whole add_header set at the
    # deepest level that defines one, so Retry-After above would otherwise
    # strip the baseline.
    add_header Retry-After 5 always;
    add_header X-Content-Type-Options $fallback_xcto always;
    add_header X-Frame-Options $fallback_xfo always;
    add_header Referrer-Policy $fallback_rp always;
    return 503 '<!doctype html>...';
}

And while we're here: always matters. Without it nginx omits the header from 4xx and 5xx responses — protecting only the responses that were never the risk.

The fix: `map` on `$upstream_http_*`

add_header can't ask a question, but a map can answer one before the question is asked. Two behaviours combine into exactly what I needed:

$upstream_http_<header> holds the value the upstream sent for that header, or the empty string if it sent nothing.
nginx omits an add_header whose value evaluates to an empty string.

Put those together and the fallback writes itself. In http context:

# The baseline security headers are a FALLBACK: each value is empty when the
# application already sent that header, and nginx skips an add_header with an
# empty value.
map $upstream_http_x_content_type_options $fallback_xcto {
    ''      "nosniff";
    default "";
}

map $upstream_http_x_frame_options $fallback_xfo {
    ''      "SAMEORIGIN";
    default "";
}

map $upstream_http_referrer_policy $fallback_rp {
    ''      "strict-origin-when-cross-origin";
    default "";
}

Then the vhost uses the variables instead of literals:

server_tokens off;
add_header X-Content-Type-Options $fallback_xcto always;
add_header X-Frame-Options $fallback_xfo always;
add_header Referrer-Policy $fallback_rp always;

An application that sets its own headers keeps exactly what it set — no duplicate, no contradiction. An application that sets none gets the baseline. That's the behaviour the requirement asked for, and it's what the first cut didn't do.

Two properties of $upstream_http_* that made me comfortable shipping this:

It's populated for fastcgi and uwsgi upstreams too, not just proxy_pass. So a PHP-FPM vhost behaves the same as a reverse-proxied one.
It's empty when nginx serves the file itself, which is the correct answer for a static vhost: nothing upstream spoke, so apply the fallback.

The deployment trap that nearly cost more than the bug

Here's the operational edge case, and it's the one I'd want a reviewer to catch if I'd missed it.

A vhost that references $fallback_xfo when no map defines it is a config nginx refuses outright. Not a warning. Not that vhost failing. nginx -t fails, the reload is rejected, and if a deploy pushes the vhost without the map, you have taken down every site on the node — including the ones you weren't touching.

So the maps and the vhosts have to land together, in all three of the config-writing paths in my system (a systemd driver writing to disk, a container-oriented writer, and a Kubernetes ConfigMap). All three render vhosts from the same shared trait, so all three needed the maps added — the shared trait is exactly what makes it easy to forget, because the change looks like one place.

If you're adding this to an existing setup: deploy the map block first, on its own, and reload. Then deploy the vhosts. A map nothing references is harmless. A reference with no map is a node-wide outage.

What I didn't copy

The panel these applications are migrating from also sends X-XSS-Protection: 1; mode=block. I deliberately left it out. Every current browser has removed the XSS auditor that header names, and on the browsers that kept it, the header introduced a cross-site information leak of its own. Matching the old platform's protection level is the goal; matching its config file is not. Worth writing that distinction into a comment, because "the old system sent it" is a very persuasive argument at review time.

Testing this honestly

The config generator is unit-testable, and the test is worth writing because the shape of the output is the whole contract:

it('renders baseline headers as a fallback, not a second opinion', function () {
    $config = $this->proxy->renderVhost($application);

    expect($config)
        ->toContain('server_tokens off;')
        ->toContain('add_header X-Frame-Options $fallback_xfo always;')
        // The literal is the bug: a hardcoded value cannot defer to the app.
        ->not->toContain('add_header X-Frame-Options "SAMEORIGIN"')
        // Not copied on purpose — the auditor it names no longer exists.
        ->not->toContain('X-XSS-Protection');
});

it('repeats the baseline inside the unavailable location', function () {
    // One add_header at a deeper level discards the entire inherited set,
    // so the 503 page needs its own copy or it ships bare.
    expect($this->proxy->renderVhost($application))
        ->toContain('add_header Retry-After 5 always;')
        ->toContain('add_header X-Frame-Options $fallback_xfo always;');
});

But be clear-eyed about what that proves. It asserts a directive was emitted. It cannot assert what a browser does with the response. The original bug passed a green suite. A test over generated config verifies the generator, not the behaviour — and the difference between "CI-proven" and "live-proven" is exactly the gap this bug lived in for an hour.

The only thing that caught it was hitting the live host and reading the response headers, right after shipping.

Takeaways

add_header appends, can't be conditional, and replaces the entire inherited set at the deepest level that defines one. All three of those will surprise you at least once.
A map on $upstream_http_* plus nginx's empty-value rule is the idiomatic way to express "only if the upstream didn't". No if, no Lua, no duplicated header.
A security header added carelessly can be a downgrade, not an addition. "More headers" isn't a safety property.
Ship the map before the reference. A dangling variable in a vhost fails the whole config, not just that site.
Verify the claim you just made, against the live thing, in the hour you made it. A green suite is evidence about the generator. It is not evidence about the response.

Dev Log: 2 September 2026 — Every Bug Today Was a Claim Nobody Checked

Nasrul Hazim — Fri, 04 Sep 2026 02:34:42 +0000

Twenty-two commits across four repos today. Different stacks, different problems, and — reading them back in one sitting — the same bug wearing four costumes.

Something asserted a state. Nothing verified it. The assertion was wrong, and it had been wrong for a while.

A compliance control was green on evidence that didn't exist. A kitchen screen said there was nothing to cook while paid orders were sitting there. A dashboard reported a 97.8% success rate directly above thirteen failed rows. A service worker claimed to be serving today's CSS and was serving the first CSS it ever saw.

None of these threw. That's the shared trait. A stack trace is a gift — it tells you exactly where to look. A confident wrong number tells you nothing, and it keeps telling you nothing until someone counts by hand.

The control that was green on nothing

Start with the worst one, because it's the cleanest illustration.

A compliance reporter generates control evidence automatically — for each control, a line saying why the system considers it satisfied. One data-minimisation control cited two mechanisms: encrypted casts on the sensitive columns, and a redaction trait applied to the audited models.

The first half was true. The second half was not. The trait existed. It was applied to no model at all. The control had been permanently green on a citation that pointed at nothing.

And it wasn't simply forgotten. As written it could not be applied to the main user model — the audit package it needed to cooperate with declares a method of the same name, and two traits declaring the same method in one class is a fatal composition error in PHP:

class User extends Authenticatable
{
    use Auditable, RedactsPiiInAudit {
        RedactsPiiInAudit::transformAudit insteadof Auditable;
    }
}

That insteadof is doing real work. Without it the class doesn't compile. Descendants of a base class that had one of the two were fine — a trait on the child beats a method inherited through the parent — so the conflict only appeared on the one model that mattered.

Two things I'd generalise from this:

Generated evidence needs a test that the evidence is reachable. A reporter that emits evidence_source: "...trait X" should be able to answer "which classes actually use X?" and fail when the answer is none. Otherwise the report is a string, and strings don't know if they're true.

Configuration on a trait belongs in a method, not a property. A trait and its consumer cannot both declare the same property with different defaults — also fatal. So:

trait RedactsPiiInAudit
{
    // Overridable. A property here would be a fatal clash the first time
    // a consumer wants its own list — and a consumer that forgets to
    // declare one would have no default at all.
    protected function redactedAuditAttributes(): array
    {
        return ['ic_number', 'phone', 'address'];
    }
}

The bonus find, which is the part I'd actually lose sleep over: password hashes were being written into the audit trail. An audit table is immutable by design and retained for a year. A hash written there can't be withdrawn. Masking is the right treatment for PII that has to stay recognisable in an audit — a credential is never in that category, so credentials and 2FA secrets are now excluded outright, not masked.

The same commit deleted three classes referenced from nowhere. Static analysis had flagged two of them as used zero times, and someone had baselined the warning instead of deleting the files — which is its own version of today's theme. A baseline entry is a permanent, invisible ignore.

And a SECURITY.md that hand-mirrored composer.json had drifted in every single entry: it named a framework version two majors behind, and listed four dependencies the project doesn't have. A hand-maintained copy of a lockfile does not stay true. It now points at composer.json and names the audit commands instead. A stale security document is worse than no security document, because it looks like diligence.

Two screens that contradicted themselves

Same shape, two different products.

An audit dashboard rendered its summary cards from the controller using an unfiltered count, while the table below them came from the datatable's filtered query. Filter to "failed, last four days" and you got 1,949 attempts and a 97.8% success rate printed directly above thirteen failed rows. Both halves of one screen, disagreeing — and the reassuring half was the wrong one.

The fix is architectural, not arithmetic: the cards moved inside the Livewire component and are computed from the same baseQuery() that produces the rows.

// Whatever narrows the rows must narrow the summary. One source, two renders.
public function summary(): array
{
    $base = $this->baseQuery();

    return [
        'total'  => (clone $base)->count(),
        'failed' => (clone $base)->where('status', Status::Failed)->count(),
    ];
}

If a number and a list on the same screen come from two different builders, they will eventually disagree. Not might — will.

The rebuilt report also added a period selector, because all-time was the only view and a large intake of successful records pinned the headline near 98% no matter what broke today. A metric with no time window is a metric that can't report a bad day.

Best change in the lot: a "why they failed" breakdown, grouping the recorded error messages and collapsing multi-line database stack traces to their error code. That data had been persisted from day one and never read once. The step tells you where something stopped; only the message tells you why.

A kitchen display screen in a point-of-sale app was the same bug in food form. The display queried for orders whose status was pending or confirmed. But the till takes payment first and cooks after — checkout creates the order, then processes payment, which sets the order to PAID while every line item is still PENDING.

So no order ever taken at the counter reached the kitchen. The screen had only ever shown "No pending orders", which reads exactly like a quiet afternoon.

The insight is a modelling one worth stealing:

Order status tracks money. Item status tracks cooking. A cook is looking at food, so the query has to key on items.

The query now excludes only orders that were killed — cancelled, voided — and keys everything else on item state. A ticket drops off the board once every item is READY.

Two statuses that both say "pending" are not the same status. Naming them for what they measure (PaymentStatus vs PreparationStatus) would have made the bug hard to write in the first place — which is most of the argument for enums over strings.

Caches that were confidently out of date

Two variations, both ending in "works after a hard refresh", which is the single most misleading symptom in web development because it makes the developer the only person who can't reproduce it.

A service worker was matching static assets by pathname alone. In development that meant it also intercepted the dev server's own CSS and JS on a different origin and cached them cache-first — permanently. Cmd+Shift+R bypasses the service worker entirely, which is precisely why only that showed the new styling.

Two fixes, and the second is the general one:

Ignore every cross-origin request. Nothing from another origin belongs in an offline shell.
Cache-first is only safe for content-hashed URLs. Anything at a stable path — /js/app.js, /icons/* — freezes at whatever shipped first, forever. Those moved to stale-while-revalidate: instant from cache, refreshed in the background. Bump the cache name so the frozen entries get evicted on activation.

The other variant: a generated report was styled by a bundle built before the report existed. Same family — a build artifact asserting it was current when it predated the thing it was styling.

While I was in there, a third one, and this is a nasty little Livewire trap:

Uncaught TypeError: $wire.view is not a function

A list/grid toggle introduced a $view property on a component that already had a view() method. On the $wire proxy, a property shadows a method of the same name. PHP is happy. Blade is happy. Static analysis is happy. The only symptom is a click that silently stops working.

The guard is a test, because nothing else in the stack can see it:

it('has no property shadowing a method on any Livewire component', function () {
    foreach (livewire_components() as $class) {
        $properties = collect((new ReflectionClass($class))->getProperties())
            ->map->getName();

        $methods = collect((new ReflectionClass($class))->getMethods())
            ->map->getName();

        expect($properties->intersect($methods))
            ->toBeEmpty("{$class}: property shadows a method of the same name on \$wire");
    }
});

Same day, same repo, a related guard: a missing icon component throws at render time, so only a test that opens that exact page catches it — and a page nobody covers stays broken until a user clicks it. The new test walks every Blade file for icon references and asserts the file exists. That's the right level. Don't test that page 47 renders; test that no page can reference an icon that isn't there.

The one where the bug was blamed outward

My favourite, because I've made this exact mistake and so has everyone reading this.

Four directory attributes came back null on live objects. This was written up as a missing read permission on the service account — a plausible story, an external cause, a ticket for someone else.

It wasn't. The profile screens in the same application display one of those very attributes, from the same directory, on the same bind, with no special grant. The evidence that the story was wrong was already on screen.

The actual defect: the helper normalises the needle to lowercase, but the attribute bag is keyed exactly as the directory returned it. A mixed-case name like pwdLastSet silently missed and returned null. The profile views never hit it because they iterate the raw bag and lowercase both sides of the comparison.

// Lowercasing one side of a comparison is not case-insensitive matching.
$value = collect($object->getAttributes())
    ->first(fn ($v, $k) => strtolower($k) === strtolower($needle));

An earlier "fix" had added an explicit attribute selection — which only narrowed what came back, since an unrestricted query was already returning those attributes. That's the tell for a wrong diagnosis: the fix makes the system do less and the symptom stays.

The part that took the longest wasn't the code. It was going back to correct the audit finding that said the evidence was unobtainable without a permission change. It was obtainable the whole time. If the write-up survives and the diagnosis doesn't, the write-up is the thing people act on next year.

And the same idea, one layer up

The public repo today got a version of this on purpose. My Claude Toolkit shipped 2.5.0, and the substance of the release is that none of its 32 skills said when to stop — they were all "how to start".

Nine high-stakes skills now carry ## Common Rationalizations, ## Red Flags and ## Verification. The verification section is a checklist you run before claiming the work is done, and it's aimed squarely at today's failure mode:

[ ] The root cause is stated in one sentence, and it is a cause, not a symptom
[ ] Every caller of the changed function was checked, not just the reported path
[ ] A regression test exists, and it fails when the fix is reverted

That third box is the whole discipline in one line. Full write-up in the companion post.

Takeaway

The bugs that cost the most today weren't the ones that threw. They were the ones that answered confidently.

A stack trace is a system admitting it doesn't know. A green control, a 97.8%, an empty kitchen queue and a cached stylesheet are systems claiming they do. Only one of those two categories can be found by waiting for it to break.

So the pattern I'd take out of today:

Whenever code emits a claim — a compliance status, a summary count, a "no results", a cached response — ask what would fail if the claim were false. If the answer is "nothing", you don't have a feature. You have a decoration that people will make decisions from.

Every fix today shipped with a guard: a test that fails if the trait goes unused, a test that fails on a property/method collision, a test that fails on a missing icon, a query that can't disagree with the list beneath it. Not because the tests are impressive — most of them are ten lines — but because a claim that nothing can falsify will drift back the moment nobody's watching.

What's next: I want the compliance reporter to refuse to emit an evidence string it can't resolve to a live class. Right now it will happily cite a ghost.

Dev Log: 1 September 2026 — Every Bug Today Was a Name Problem

Nasrul Hazim — Fri, 04 Sep 2026 02:29:29 +0000

Fifty-three commits across eight repos today. Reading them back in one sitting, they sorted themselves into a single shape: something used a name as if it were an identity, and the name wasn't unique.

Two people sharing a username. Two accounts sharing an email. Sixteen environments sharing the word "Production". A class whose name got rewritten under a caller that looks it up by name. Nineteen URLs sharing one document. Different stacks, same bug.

Case-sensitive database, case-insensitive directory

The one that took the longest to explain, and it's four words: = is case-sensitive.

PostgreSQL and Oracle both compare strings case-sensitively. Active Directory, by specification, does not. So a uniqueness gate written as where('username', $input) will happily tell you AliceW is available while alicew exists — and when the downstream sync writes to the directory, both names resolve to the same object.

The gate says yes. The directory says "oh, that one". Two different answers to what everyone assumed was one question.

The fix is a small support class rather than a scope, because it's needed on both Eloquent and query builders in several places:

final class UsernameQuery
{
    public static function whereMatches(Builder|QueryBuilder $query, string $column, ?string $username)
    {
        return $query->whereRaw(self::expression($query, $column), [self::normalize($username)]);
    }

    /**
     * Two usernames collide when they differ only by case or surrounding
     * whitespace — the comparison the directory makes, and the one the
     * gates must make too.
     */
    public static function matches(?string $a, ?string $b): bool
    {
        return self::normalize($a) === self::normalize($b) && self::normalize($a) !== '';
    }

    public static function normalize(?string $username): string
    {
        return mb_strtolower(trim((string) $username));
    }
}

Three decisions in there are worth more than the code:

LOWER(col) = ?, not a case-insensitive LIKE. This is an exact-match gate. Usernames legitimately contain _, and LIKE treats _ as a single-character wildcard — you'd get false collisions on every underscore.

Assert on the generated SQL, not on the returned rows. SQLite folds ASCII case for =. A test that only checks "did I get the row back" passes before and after the fix when it runs on SQLite. The test that actually pins the behaviour asserts the query contains LOWER(.

it('lowercases both sides of the comparison', function () {
    $sql = User::query()->tap(fn ($q) => UsernameQuery::whereMatches($q, 'username', 'AliceW'))->toSql();

    expect($sql)->toContain('LOWER(');
});

Stopping new collisions surfaces none of the old ones. So there's now a sweep that answers the inverse question. Every other lookup in that system is subject-scoped — "who owns X?", and you need X up front. This one is an aggregate over the entire name space, which no existing tool could express.

Two details from writing it that generalise:

The parts are combined with UNION, not UNION ALL, because one person can hold the same name in two different columns. Without the dedup, every ordinary user is reported as colliding with themselves.
Any column with a stored prefix (ext_ and friends) has to have the prefix stripped for grouping, or that whole population silently contributes nothing to the sweep. Strip it with SUBSTR, not LIKE 'ext_%' — _ is a wildcard, remember.

And then the finding I nearly shipped wrong: the first sweep flagged dozens of collisions as live takeovers. Almost none were. The risk label assumed a single directory namespace, and there isn't one — different identity types derive their directory object from different attributes entirely. Two owners can share a name without sharing an object.

Grouping now aggregates in two steps — per (name, type) first — so the outer query can distinguish "two of the same type share this" from "two different types do". The ordering keys on that signal too, because sorting on the raw count buried the one case worth acting on beneath thirty that weren't collisions at all.

A risk score that ranks noise above signal is worse than no score. You'll read the top of the list and stop.

A picker that collected a choice nobody consumed

Different system, same species.

A personal email address is not unique in most institutional databases — a person who progresses through two programmes keeps the same email on two ids. So the password reset flow shows a picker: here are your accounts, choose one.

The picker was correct. It rendered the accounts, it validated the selection, it carried it through the form. And then the resolver ran:

$row = IdmUser::whereEmailAlt($email)->first();   // by email. unordered.

The selection was collected and then ignored. Resets reported success, wrote to a stale id, and the live account was never touched — so the directory's pwdLastSet never moved and the client read it as "your reset isn't doing anything".

What makes this one worth writing down isn't the bug, it's the history. This was logged and closed a year ago. The fix that closed it shipped the picker — the UI half — and was marked Resolved. No test covered the multi-account path, so the incomplete fix shipped green and the gap survived twelve months behind a ticket marked done.

Four changes went in, and only the first is the fix:

// 1. Scope on the selected username AND the email, not the email alone.
UserCategory::whereUsernameAndEmailAlt($username, $email);

// 2. Backstop: abort and log before any backend write if the resolved
//    identity doesn't match the one that was selected.
abort_unless(UsernameQuery::matches($resolved->username, $selected), 409);

The picker now labels each account with its directory status, so two bare ids are actually distinguishable to the human choosing between them.
The success screen names the account that was written. A bare "Success" is what hid this for a year — from the user and from support. If an operation picks one of several candidates, the confirmation has to say which one it picked. That's not UX polish; it's the only observability that costs nothing.

There's a diagnostics footnote too. The first report masked emails with one asterisk per character, so two unrelated people whose local parts shared their opening letters and length printed identically — the report read as one group where there were two. The counts were never wrong; the rendering was ambiguous. It now carries a stable group ref (first 8 hex of a sha256 over the lowercased address) and a fixed-width mask, because per-character masking leaks the local part's length for no benefit at all.

An identifier in a report needs to be unambiguous and stable across runs. A mask is neither.

"Confirm by typing the name" only works if the name is unique

Shipped a set of destructive MCP tools today — delete a project, an environment, a provider; retire a machine — and the guard pattern is the usual one: the caller must retype an identifier to confirm.

Except sixteen of the seventeen environments on the live control plane are called Production.

Retyping a name that sixteen records share confirms nothing an agent couldn't have guessed. So the confirmation is on the slug, and for a machine it's on the SSH host address rather than the hostname, which can be blank or shared.

The pattern, as a trait:

trait RequiresTypedConfirmation
{
    protected function confirmOrFail(string $argument, string $expected, string $noun): void
    {
        // The argument key is passed in so a refusal is audited identically
        // whichever tool refused.
    }
}

Two other things went in the response text of every one of those tools, both because they're ways an agent reports a teardown wrongly:

Deleting a provider removes the credential, not the host. An adopted machine keeps running and keeps costing money until its own hosting provider destroys it.
The deletion is soft. "Deleted" in a tool response and "deleted" in a database are not the same claim, and an agent will relay the first as if it were the second.

If you're building MCP tools, that second one deserves a rule of its own: say what the tool did not do. An LLM will fill any silence with the most obvious inference, and for a delete tool the obvious inference is "the thing is gone".

There was also a dependency rule to move. The "can this be deleted?" check lived as a private method on a Livewire component, with a comment from its author warning it would be forgotten at the fifth call site. This was the fifth. It's now a RecordDeletionGuard service asked by both the UI and the tool — and extracting it exposed that the rule was also wrong: it counted every deployment row including destroyed ones, so the final step of a teardown was refused forever by the very deployments the teardown had just destroyed.

Related, same day: a "has history" rule that was blocking deletes. History is not a dependency. Every provider anyone ever bootstrapped accumulates job rows, so the rule read, in practice, "you may delete a provider only if you never used it". The rows are kept on purpose, the parent is soft-deleted anyway, and they still resolve through withTrashed(). Dropped.

A relation cached before the rows existed

This one is pure Laravel, and it's the sharpest edge I hit today.

A 24-step provisioning pipeline carries one Deployment model instance through every step in a context object. Step 11 reads $deployment->routingRules — as a property. That loads the relation and caches it on the instance. At step 11 it's empty, because the rules are written at step 16.

Steps 20 and 21 then read the same cached, empty collection, and took the "an empty set is an honest success" branch. So DNS and SSL both reported success on a deployment that finished 24/24 Active with no DNS record and no certificate. A second run worked, because it got a fresh instance.

// Cached at first access. Every later read on this instance sees step 11's answer.
$rules = $deployment->routingRules;

// Re-queries. In a long-lived pipeline this is the one you want.
$rules = $deployment->routingRules()->get();

Everyone knows relation access is cached. The part that bites is that it's cached on the instance, and in a request/response app instances don't live long enough for it to matter. Move that same model into a pipeline, a queued job chain, or a long-running worker and the lifetime changes underneath you without the code changing at all.

The general rule I'm taking from it: if a model outlives the thing that loaded its relations, treat property access as a snapshot, not a query. And be suspicious of any branch that reads "empty set → success". Empty is ambiguous. It means both "nothing to do" and "I looked at the wrong moment", and only one of those is a success.

A coverage test that could only see what today's fixtures happened to draw

Spent a chunk of the day on localisation — making English the key language, with the second language selectable rather than forced by an org-level setting.

I'd added a translation-coverage test the day before. It reported clean. Dozens of public-facing keys were untranslated.

The test was render-driven: render the pages, ask Laravel which keys were looked up and not found. Reasonable design. But the local seed had no priced membership tiers, so the tier cards never rendered, their keys were never looked up, and the test was satisfied.

A render-driven test is only as good as your fixtures. It can only fail on the paths your seed data happens to reach.

The fix is not to replace it but to put a second pass beside it:

Source-driven: every __() key found in the public view tree must have a translation, whether or not anything renders it today.
Render-driven: kept, because it still catches strings emitted from outside that tree — a component library's own labels, for instance.

Neither alone is sufficient, and the two failure modes are exactly complementary.

Then a third one that neither pass can catch. Page titles were being set as literal view attributes:

<x-layouts.auth.card title="Apply for membership">

Never a key at all. The source scan looks for __(; the render pass needs a key to be looked up. A string that was never a key is invisible to both. Found it by opening the page.

Two lessons in one afternoon: automated coverage has a shape, and things outside that shape are not merely uncovered — they're reported as covered. And the most important page in a funnel is the one most worth loading manually before you trust a green test.

The build tool renamed a class the native layer looks up by name

The mobile side of the day. An app was dying mid-match — no Dart exception, no error screen, the process just vanished.

R8 (Android's shrinker/obfuscator) had renamed most of an ML runtime's Java classes down to two-letter names, the way it's supposed to. That runtime's native layer resolves those classes by their original names through JNI, so the first inference hit a null class in GetMethodID and the Android runtime aborted the process.

The rule R8 can't know: a class referenced only from native code has no Java-side reference to keep it, so as far as the shrinker is concerned nothing is using that name. Keeping it takes an explicit -keep — the same reason reflection needs one.

A crash with no stack trace in your language is almost always a layer boundary. JNI, FFI, a native plugin — something on the other side of a boundary that your exception handler doesn't cross.

And it's the name problem again, in the most literal form of the day: a tool rewrote a name, and something else was still holding the old one.

Sibling fix in the same release: two macOS crash reports were SIGSEGVs inside the bundled SQLite framework, and one of them showed the framework mapped twice — same UUID, two base addresses. Two independent copies of SQLite live in one process, each with its own global config, handles crossing between them. The database was being hosted on a spawned background isolate with handles sent across the boundary; opening it on one isolate removes the surface entirely.

Same name, two live things. Again.

And nineteen URLs sharing one document

The public one, and the reason it's the whole other post: devhub.my was serving the identical index.html for every path, at HTTP 200, because of the standard SPA catch-all rule. An SEO crawl reported five pages; the sitemap listed nineteen. Both were correct.

That's now a prerender step that writes a real HTML document per route and fails the build on three things no validator catches — a page that never declared its head, a JSON-LD graph with a dangling @id, and a canonical pointing somewhere the page isn't. Full write-up in the companion post; repo is developers-hub-my/website if you want to read the scripts.

Also, briefly

A framework upgrade plus a design-system pass on one product: Laravel 13 / PHP 8.5, a starter kit dragged forward across thirty-odd minor versions, unified design tokens, and a shared list pattern rolled out screen by screen with one reference screen built first. Building the reference screen properly and then porting is slower on screen one and much faster on screens two through twelve.
Stale published assets after a package update. If your framework has a "publish these vendor assets" step, it has a "these are now out of date and nobody noticed" state. Wire the republish into the update path rather than the install path.
Migration planning as documentation, not as a ticket: what the current platform actually does, observed rather than assumed, before deciding what to replace it with. The probe that produced the observations got committed alongside the notes, which is the part I'd have skipped a year ago.

Takeaway

If today had a single rule in it, it's this: a name is not an identity. It's a label that happens to be unique in the sample you looked at.

Every bug above is what happens when code treats a label as a key — and every fix is the same move in a different dialect: normalise before comparing, confirm on something actually unique, name the thing you acted on in the response, and check the assumption at the layer that will be surprised, not the one that's convenient.

The cheap tell: any time you're about to compare, group, or confirm on a human-readable string, ask who else could hold that string. If the answer isn't "nobody, by construction", you have a bug waiting for a coincidence.

Dev Log: 31 August 2026 — Loaded Is Not Serving

Nasrul Hazim — Fri, 04 Sep 2026 02:05:05 +0000

Thirty-eight commits across five repos today. Reading them back, most were the same species of bug: two different things that a system had been treating as one word. Loaded and serving. Both-S3 and copyable. Present and correct. Every one of them was cheap to fix and expensive to find.

Loaded is not serving

Found in production, and my favourite bug of the day.

A site was answering 200 to every request. The dashboard showed the workload as failed, with its last release green and its deployment active. The journal was clean — php-fpm up, migrations run, no errors anywhere. A red workload sitting next to a perfectly healthy site, which is exactly the thing that sends somebody to go and fix a deployment that is fine.

The blue-green runtime picks the "active" colour like this:

foreach ([SLOT_BLUE, SLOT_GREEN] as $slot) {
    if ($this->unitLoaded($slot)) {
        return $slot;
    }
}

Here's the thing: systemd keeps a unit loaded after it dies. systemctl show on a stopped-but-known unit still reports LoadState=loaded. So a retired blue outranked the green that was actually serving every request, the status probe went and asked blue how it was doing, blue said ActiveState=failed, and the UI rendered red.

The quieter half of the same bug: standbySlot() is the inverse of that answer. With blue permanently "active", every release went to green. Two consecutive production releases landed on the same colour — blue-green alternating had silently stopped, which defeats the entire arrangement.

The fix reads both properties in one round trip and lets an active unit win outright, even though blue is examined first:

$state = $this->unitState($slot);   // LoadState + ActiveState, one call

if ($state->isRunning()) {           // active or activating
    return $slot;
}

$loadedFallback ??= $slot;           // only used if NEITHER is running

The fallback is deliberate — it's what keeps a genuinely stopped workload reporting Stopped rather than Pending. And the test that scripted the old single-property probe was updated rather than deleted, because it now expresses serving instead of merely loaded, which is the distinction the whole bug turned on.

Both endpoints being S3 is not "can copy server-side"

While wiring bring-your-own object storage, I hit a variant of the same mistake in my own contract design.

CopyObject in the S3 API is served by the endpoint you send it to. That endpoint cannot reach into a different service's buckets. So "source is S3 and destination is S3" does not imply a server-side copy is available — same endpoint does.

The awkward part is that the implementation is resolved per driver, not per row. An S3 destination asked "can you server-side copy from this?" genuinely cannot tell which of its own configured endpoints it's being asked about. So the method grew a parameter:

public function supportsServerSideCopyFrom(
    BackupDestination $destination,
    BackupArtefact $source,
): bool;

PHPStan caught the arity break across one call site and two test doubles, which is the cheapest possible way to find out you changed a contract.

The consequence is worth stating out loud: cross-endpoint S3-to-S3 has to stream, which makes the streaming path the common case rather than the rare one. That promotes the transfer size ceiling from a defensive guard to a load-bearing part of the design — and it's why the byte counter counts bytes through the control plane, so a genuine server-side copy correctly records zero.

A version that is a fact beats a version that is a guess

A one-click Drupal install on an AlmaLinux 9 node bootstrapped, built, deployed, passed the health gate — and then died at step 18 with:

The database server version 13.23 is less than the minimum required version 16.

EL9's AppStream defaults the postgresql module to 13. Drupal 11 pins a minimum of 16. Those two can never agree, which means that recipe could not have installed on any RHEL-family node, ever — and the cost of learning it was a full composer build on a 1 vCPU box, reported on a step whose name points nowhere near a package stream.

Two fixes, because either one alone leaves the failure sitting there.

Install a server modern apps accept. A per-OS hook runs before a service's packages go on. Empty on Debian, whose postgresql metapackage tracks the release's own server. On RHEL it switches the module stream, guarded three ways, each load-bearing: only when 16 actually exists in that node's AppStream, so an older minor keeps working; module reset first, because enabling a second stream over an enabled one is an error rather than a switch; and only from the "no server installed yet" branch, because a stream switch does not upgrade a cluster — it would leave the data on 13 and the packages on 16, which is worse than either.

Refuse early when it still can't be satisfied. A recipe declares its minimum engine versions and a guard checks them at the end of the provisioning step — not in preflight, and this is the part worth arguing about.

Preflight has no database yet. The only thing available there is a guess from a table of distribution defaults: a table that goes stale silently and is wrong the moment somebody installs a version by hand. Six steps later, the version is a fact — the provisioner has read SHOW server_version off the running daemon. Still ten steps earlier than the failure it replaces, and it cannot be wrong.

Two details from the same guard:

An engine that isn't in the deployment at all is unconstrained, not forbidden. The app might use SQLite, or a database this platform never provisioned.
version_compare, not string comparison. '9.6' sorts after '16' as text, which is how a naive check waves through an engine seven majors too old.

A hostname nobody typed

An application with no domain isn't merely unrouted — on a plain VM it's usually undeployable. With no domain the runtime drops off fpm serving and onto the recipe's php -S 0.0.0.0:${PORT} fallback, on the port the catalogue declared for the container image, and on any node already serving a domain, nginx is holding 80.

Cloning is what makes this the common case rather than an edge case. Cloning strips every domain, correctly — a hostname is served by exactly one deployment — so the copy of a working site arrives in precisely the state that can't run, and the operator is asked to invent a hostname before they can see whether the copy even works.

So a preview hostname gets issued rather than typed:

{workload-slug}-{deployment-suffix}.{preview-base}

The deployment's own random suffix carries the uniqueness, and that's the entire reason it's in the label. A scheme that can collide needs a retry loop, a "name taken" error, and a human to resolve it — for a name nobody asked for in the first place. The workload slug is in there because one deployment may attach more than one HTTP application.

The nice part: it's almost all wiring that already existed. One new step writes the name onto the routing rule; the reverse-proxy, DNS and SSL steps then do exactly what they already do for a hostname somebody typed, and none of them needs to know where it came from.

Who owns the decision — a menu refactor that's really an access refactor

On a multi-tenant product I work on, the Administration section had quietly become where anything unclassifiable landed: 22 items, six sub-groups, three levels deep, mixing a tenant's own settings with the vendor's diagnostics. A committee secretary was being shown Telescope, Horizon, an Artisan runner and an MCP console — tools they will never touch and shouldn't see.

The dividing line I took wasn't what kind of screen is this. It was who owns the decision. Vendor-owned tooling moved into its own section, invisible to a tenant entirely. Tenant-owned administration stayed and flattened — single destinations moved to the top level instead of each sitting inside a sub-group that costs a click to reveal one item. Net effect on a tenant's sidebar: 22 items to 9, three levels to two.

One implementation note that generalises. The new section gates on a landlord-level flag rather than a permission, for the same reason the platform routes do: these screens run with no tenant current, and a permission check then has no table to consult. It throws rather than returning false.

And the test that had checked the moved item is now stronger than before — it reads the new section and short-circuits on that section's own authorisation, so the item can't leak through a per-item gate even if somebody adds one.

The rest, briefly

Re-pricing moved out of a Livewire component and into an action. Two callers now share one definition of what re-pricing means — a form and an MCP tool. If the tool had copied the "retire the active version, cut the next one" logic, there'd be two definitions free to drift, and the thing that drifts is what people get charged. The write tool also carries forward anything the caller didn't name rather than defaulting to zero, so an agent asked to raise one number can't silently wipe the parameters sitting beside it.

A statistics dashboard that couldn't reach its own rows. A mail-history package renders eight status counts and a banner reading "N messages stuck in Sending for over 1 hour" — with no way to reach those N. It can't have a list, because the table has no recipient or subject column; both live inside a JSON headers blob, which is why status is the only thing it can aggregate. Fix: add the two columns, fill them on saving in a model that extends the package's, and backfill via a deploy operation that's idempotent and deliberately does not bump updated_at — the stuck-message window measures from it, so a backfill mustn't make a stuck message look fresh.

Gateway credentials got a UI, and a rule. A stored secret is never sent back to the browser; the form shows only whether one is set, and a blank field on save means "leave it alone" — so saving the page to change a URL can't silently blank a credential. Switching to a gateway with empty credentials is refused outright, because allowing it means every checkout throws and the operator hears about it from a customer instead of from that screen.

A Livewire trap worth writing down: wire:model="settings.payment.stripe.secret" doesn't bind one value — Livewire reads each dot as nesting and builds a three-level array. It fails as "Array to string conversion", which names nothing useful. Flat field names mapped to dotted storage keys through an explicit fieldMap().

A sandbox route with no guard. A pay-by-button-press endpoint that marks an invoice paid with no money involved — registered unconditionally by a glob-based route loader, checking nothing but that the invoice exists. That's fine in a sandbox and very much not fine anywhere else.

And one genuinely small thing: a mobile app version bump for optional player accounts and Google Sign-In, plus its changelog entry and two new BSD-3 licence notices. Third-party notices are one of those chores that's trivial the day you do it and awful the day you're asked for them at once.

The takeaway

Every bug in the first half of today was a word doing two jobs. loaded meant both "systemd knows about this" and "this is serving traffic". s3 meant both "speaks the protocol" and "can copy from that". A distribution default meant both "probably installed" and "definitely installed".

They're hard to spot because the collapsed version reads fine — it's usually the shorter, more natural sentence. The tell is when a screen and reality disagree and nobody can immediately say which one is wrong. That's not a display bug. That's two concepts sharing a name, and the name picked the wrong one.

The Most Dangerous Backup Is the One That Reports Success

Nasrul Hazim — Fri, 04 Sep 2026 02:05:00 +0000

TL;DR — I spent today building out a backup subsystem, and almost every design decision that mattered came down to the same question: can this thing fail in a way that still renders as a green tick? A backup that fails loudly is an annoyance. A backup that fails quietly is the reason the restore doesn't exist.

The failure mode nobody designs for

Backups have an unusual property among features: the moment you find out whether yours works is the worst possible moment to find out. Everything in between is inference. And inference is exactly where a system gets to lie to you politely.

So the rule I kept coming back to today: a component may not report a success it did not earn. Not "should not" — may not, structurally, because the code path that would produce the false green isn't there.

Here's what that looked like in practice.

Sync is not backup

The first one is the one people get wrong most confidently, because the tool that does it wrong is excellent at its job.

A sync tool mirrors a source into a destination. If a key disappears from the source, it disappears from the destination. That is correct behaviour for a mirror and catastrophic behaviour for a backup, because the thing you are backing up against is somebody deleting the wrong folder at 2am — and a mirror propagates that deletion to the only copy that could have undone it, usually within the hour, and then reports success.

So the bucket driver I wrote today records deletions and never replays them:

/**
 * A key that vanished at the source is marked deleted in the manifest.
 * Its object is LEFT WHERE IT IS until retention expires it.
 * Nothing in this driver deletes an object at a destination.
 */

Two tests pin the pair of consequences, and the second one matters as much as the first:

an object deleted at the source is still restorable from the last run that had it;
restoring a later run does not resurrect it.

That second test is the one people forget. A restore reproduces the bucket as of that run. An operator recovering one folder did not also ask to undo every intentional deletion since — and if your restore does that, you've handed them a second incident while they were cleaning up the first.

"Move" is copy, verify, then delete — in that order, always

Moving a backup between destinations is the most obviously dangerous operation in the whole subsystem, because there's a window where the artefact exists in exactly one place and something is actively trying to remove it.

The ordering rule is boring. What's interesting is making the rule checkable:

// source_deleted_at stays null until the destination copy is Verified,
// and is written in its own statement, after verified_at.

Two separate writes, deliberately. If they went out together you could argue forever about whether a bad row was a race or a bug. Kept apart, a query can find the violation:

test('a transfer never deletes the source before the destination verifies', function () {
    expect(BackupTransfer::deletedSourceBeforeVerification())->toBeEmpty();
});

The other half: the destination copy gets its own artefact row, with its own key, size, checksum and verification state — and the driver is asked to verify that row. A move that verified the source's checksum would have proved nothing about the transfer it existed to prove. It would have been a green tick describing the wrong object.

Verification that admits what it didn't check

This is my favourite one, because the honest version is uglier than the dishonest version and that's the whole point.

Verifying a bucket copy properly means re-hashing every object. To do that, the control plane has to read every object back through itself — during the verification of a transfer whose entire purpose was to keep that data out of the control plane. You can't have both.

The dishonest option is to check what's cheap and call it verified. What shipped instead: check object count, total size, and a spread sample of per-object sizes — and then say so.

'verification' => [
    'objects_checked' => 1_284,
    'bytes_expected'  => 4_118_233_712,
    'etags_rehashed'  => false,
],

etags_rehashed: false sits in the record permanently. A passing verification cannot be read, later, by somebody who wasn't there, as proving something it never checked. The same instinct produced a test that scans every operator-visible string in the feature for the words "point-in-time", "versioning" and "snapshot" — because the driver doesn't do those, and prose that merely declines to claim something is not the same as prose that states the opposite. Somebody reading the screen in two years will assume whatever the words allow.

Integrity is not restorability

Every run checks that the artefact is present, the right size, the right checksum, and that it opens. All cheap, all worth having, and none of them prove it restores.

An archive that passes pg_restore --list and then dies part-way through a real restore — a missing extension, an incompatible server version, an encoding mismatch — is indistinguishable from a good one right up until 3am.

So: restore drills. A policy can opt into verify_level = restorability, on its own cadence and its own cursor (drill_cron, next_drill_at — a nightly backup does not want a nightly drill; a drill costs a real target and a real restore).

Two design calls in there I'd defend anywhere:

The drill uses the ordinary restore path. Not a drill-specific one. A drill with its own code path proves the drill works, which is the one thing it was never in doubt about. It builds a restore record in new-target mode and hands it to the same restore() an operator would trigger.

A failed drill flags the policy; it never condemns the artefact. Tempting to auto-demote a backup whose drill failed. That is precisely how your last good copy disappears on the strength of a drill-environment problem. The artefact keeps its verified status. The policy goes red. There's a test pinning that, because it's the kind of rule a future refactor would "fix".

And a bonus the drill gave me for free: restore time is a number nothing else in the system ever measures. An operator planning around a backup needs to know if they're committing to ten minutes or six hours. There's no honest way to find out except doing it. So restore time is recorded from the drill's own clock, or from a real restore's — and where neither exists, the screen says "never measured". Never an estimate. An estimate here is a fabricated green tick wearing a number.

Teardown, incidentally, goes in a finally and is best-effort silent. A teardown error must not replace the real reason a drill failed. And a drill that leaves its target behind fills the server with drill databases — a backup feature causing the outage is a bad look.

The quiet ones: pagination and fakes

Two smaller things, same family.

ListObjectsV2 returns 1000 keys per call. Back up the alphabetical first thousand objects of every bucket, report success, and you have built the worst possible version of this feature — the one that is indistinguishable from the correct one until somebody needs a file starting with "z". Paginate, and pin it with a test that seeds more than a thousand.

A fake is handed out only with a reason. The resolver that picks an object-store client can fall back to a no-op in environments that have no daemon. It never does that silently:

$store = $resolver->for($destination);

if ($store->isFake()) {
    return BackupOutcome::refused($store->reason());
}

The driver turns that reason into a refusal. A silent fake records a beautifully successful backup of nothing — which is, again, the same bug: a green tick with no work behind it.

The incremental baseline has to be verified, not just complete

Objects are compared on (key, etag, size) against the last run that both completed and has a verified copy.

Seeding from a merely-completed run inherits its gaps silently and forever — every future incremental agrees with a baseline that was already wrong, and the gap only surfaces during the restore that needed it.

lastModified is deliberately excluded from the identity triple. A re-uploaded object gets a new timestamp with identical bytes, and comparing on a clock turns an incremental quietly back into a full — the opposite failure, but still the system being wrong about its own work.

The takeaway

Every one of these is the same shape. Somewhere there's a path where the system can report a success it didn't earn, and the fix is never "add a check" — it's usually to make the dishonest path impossible to express:

delete-after-verify, in separate writes, so the violation is queryable rather than debatable;
verification that records what it didn't do;
a drill that shares the production restore path;
a fake that must carry a reason;
a baseline that must be verified, not merely finished.

If you're building anything in this space, the question I'd start with isn't "does it work?" It's "if this quietly stopped working, how long until anyone noticed, and what would the screen say in the meantime?"

If the answer to the second half is "success", that's the bug. Everything else is detail.

Next up on my side: the restore-source selector — when several verified copies exist, picking the cheapest one to restore from is its own small pile of trade-offs.

DEV Community: Nasrul Hazim

I turned my Claude Code sessions into a 3D office my team can watch

What you get

How it fits together

Step 1: Hermes3D and Agent Bus, on demand

Step 2: the hook that hires the staff

Who is who

Step 3: kanban, tokens and the model

Gotchas I hit

Sharing a read-only view for a few minutes

Try it, and what's next

Dev Log: 6 September 2026 — A Guarantee You Only Wrote Down Isn't One

1. The event stream that refuses to be updated

2. The count that is never stored

3. A level is a name for a band, not a second number

4. The solve check that doesn't look at the answer

5. A parsed question, when it stopped being an array

6. The one I got wrong

7. And the boundaries

Takeaway

Eleven Empty Folders and a Test: Enforcing Module Boundaries Before the First Class Exists

The failure mode I was trying to avoid

Where arch() couldn't help

Reading use statements like a linter

The test that asserts documentation exists

What this buys, and what it doesn't

The bit I keep coming back to

What's next

Four Env Vars and No Foreign Key: What It Means to Actually Attach a Resource

The gap that hides behind "it works"

Copy the shape you already solved

Nullable columns, and where the rule lives

The secret you can read exactly once

Real values, not template tokens

The gate: don't start something that can't work yet

Detach must never delete

Rotation is the part that bites

The bit I got wrong first: knowing what not to model

What I left out on purpose

The takeaway

Unknown Is Not Failed: Four Bugs Where My Control Plane Claimed to Know Something It Didn't

The setup

1. A timed-out probe is not a failed workload

2. An IP address is not a reachable machine

3. A cached object is not a class that will still resolve

4. A retired record is not a stopped bill

The takeaway

Dev Log: 3 September 2026 — Everything I Fixed Today Was Already Written

The badge in the @else

The runner with no door

The tracking that was complete, and off

The token that was already spent

The one that wasn't already written

Takeaway

My Agent Skills All Said How to Start. None Said When to Stop.

The gap

Three sections, and why each exists

## Common Rationalizations — the arguments you'll make to yourself

## Red Flags — signals visible during the work

## Verification — the exit gate

Why the end of the file, not the top

Coverage, hygiene, and the boring half

And a smaller thing that changed every response

Takeaway

nginx add_header Appends. It Doesn't Override — And That Can Remove Protection.

The setup

What production actually answered

Three nginx facts worth internalising

1. add_header appends. Always.

2. add_header cannot ask a question.

3. add_header does not merge across levels — it replaces the whole set.

The fix: map on $upstream_http_*

The deployment trap that nearly cost more than the bug

What I didn't copy

Testing this honestly

Takeaways

Dev Log: 2 September 2026 — Every Bug Today Was a Claim Nobody Checked

The control that was green on nothing

Two screens that contradicted themselves

Caches that were confidently out of date

Where `arch()` couldn't help

Reading `use` statements like a linter

The badge in the `@else`

`## Common Rationalizations` — the arguments you'll make to yourself

`## Red Flags` — signals visible during the work

`## Verification` — the exit gate

1. `add_header` appends. Always.

2. `add_header` cannot ask a question.

3. `add_header` does not merge across levels — it replaces the whole set.

The fix: `map` on `$upstream_http_*`