Dev Log: 2026-08-24 — Everything That Only Breaks in Production

TL;DR — Forty-one commits across six repositories and almost all of them share one shape: code that is correct on every developer's machine and wrong the moment it ships. Different mechanisms — a security header that only exists in production, a database engine the test suite never runs, a command that only runs at deploy — same root. Here's the log.

---

1. CSP is a production-only feature, so CSP bugs are production-only bugs

Three separate repositories got bitten by the same thing today, which is usually the sign that it's not a bug, it's a category.

Content Security Policy is typically enabled in production only — locally a strict policy breaks the Vite dev server, so nobody runs it. Which means every remote asset in your codebase works on 100% of developer machines and is refused on the one that matters. The only trace is a console log, on a page that's already broken.

The concrete case: a package UI pulled https://cdn.tailwindcss.com at render time. Under script-src 'self' the browser refused it, so the page arrived with zero styles — and the header logo, sized only by h-9 w-9, rendered at its natural SVG size and filled the viewport. The inline tailwind.config = {...} right after the blocked script then threw ReferenceError: tailwind is not defined, taking the rest of that block with it.

Fixed in laravel-artisan-runner by compiling the stylesheet into the package and self-hosting the webfonts, so the page fetches nothing cross-origin at all. Written up properly in a separate post today — including the test that fails if any view reintroduces a remote asset.

Two neighbours of the same problem in a private app:

• Some third-party assets you can't remove, only permit. A CDN in front of the app injects an analytics beacon into proxied responses. The application can't opt out of the tag — it can only decide whether the browser is allowed to load it. Refusing it logged a violation on every single page view and told us nothing useful. script-src and connect-src now name it.
• A support widget from a third-party host needed four directives at once — script, connect, frame (no frame-src means default-src applies, so the chat iframe was refused too) and image. The interesting part was how to widen a policy safely: compose it rather than reading it whole. A directive the base policy doesn't name is seeded from default-src first, so adding an origin can never accidentally loosen something that was locked down, and 'none' gets replaced rather than appended to.

And the thing behind all three: nothing ever verified the configuration. Saving a typo'd base URL reported success. So a Test connection button now exists, and it says plainly that it proves the base URL and not the credentials — a green check that overstates what it checked is worse than no check.

2. Two signatures that expire under the user

Small one, big lesson. A signed identity payload for that widget stamped exp = time() + 300 at page render. A user who worked for ten minutes and then opened the support widget presented a five-minute-old signature.

Read that failure mode again: it misbehaved exactly for the users who had been working long enough to hit a problem worth reporting. The people most likely to need support were the only ones who couldn't reach it.

Now it refreshes a minute before it lapses, and when a hidden tab comes back to the front. Short expiry with an active refresh, not long expiry — the window stays five minutes, the client just stops presenting a stale one.

(Related, and a classic: both script tags now carry data-navigate-once, because the layout navigates with wire:navigate, which re-executed them and stacked widget instances.)

3. The database you test on is not the database you run on

Two independent bugs, same root, and one is in a public package.

mailhistory's report built raw SQL with MySQL backticks for column aliases and DATE_FORMAT() for period buckets. On PostgreSQL the first is a hard syntax error — SQLSTATE[42601] — and the mail dashboard 500'd on render. It survived because the suite runs on SQLite, which accepts backticks as an identifier quote for MySQL compatibility and never reaches the DATE_FORMAT branch. Fixed by asking DB::connection()->getQueryGrammar()->wrap() instead of typing a vendor's quote by hand. Separate post on that one too, including how to test dialect portability without installing a single database.

The second is nastier. A tool resolver accepted "uuid or slug" and, on a native PostgreSQL uuid column, compared it against whatever string arrived. On MySQL that just doesn't match. On PostgreSQL the database throws — invalid input syntax for type uuid — and the QueryException surfaced as a bare 500. So every slug, on every tool, returned "An internal server error occurred."

The guard is one conditional: only compare the uuid column when the identifier could plausibly be one. But guarding introduced a trap worth naming out loud, because I'd have shipped it:

With the comparison made conditional, a non-uuid identifier against a model that has no slug column leaves a where() holding no predicate. Which matches every row. So first() returns an arbitrary record of the tenant, as though the caller had named it.

An empty where() isn't "no filter", it's "all of them". That case throws not-found instead, and it's tested.

Both fixes assert the query shape — a non-uuid identifier never reaches the uuid column — rather than the result, because shape holds on every engine and the suite runs where the defect can't reproduce. Running the suite against PostgreSQL in CI remains the missing half. There's currently no engine matrix at all, in either repo. That's the honest state of it.

4. Things that are only wrong at deploy time

A cluster of these today, and they all hide in the same blind spot: development never runs the deploy chain.

• route:cache refuses duplicate route names. Four names collided — two convenience redirects that took the same names as the packages' own routes, and two Livewire routes re-registered under a tenant prefix while the originals stayed in the collection, relying on "the later registration wins". That's true when serving a request and false when caching. So route:cache had been failing since that provider shipped, bootstrap/cache/routes-v7.php was never written, and view:cache and queue:restart never ran either. Requests resolve fine in dev. Nothing tells you.
• Config isn't rows. New permissions reached config/access-control.php alongside a new screen, but config only becomes database rows when the seeder runs — and that runs at provisioning. Every organisation already live had no such permission, so the gate denied and the menu hid itself. Even for a superadmin. Shipping a screen without a migration path for existing tenants is the omission, not the gate.

The tempting fix — re-run the seeder — would have been shorter and destructive: it calls syncPermissions(), which replaces a role's entire permission set with whatever config says, and operators can edit role permissions in the UI. A re-seed silently discards every customisation a tenant made. The operation is additive on purpose: create three permissions, grant them per role, touch nothing else.

• $PATH depends on which credential reached the machine. Build steps exported PATH=<workload bin>:"$PATH". As root, the inherited PATH carries /usr/local/bin. Through a restricted management user, every command goes through sudo -n sh -c, and sudo's secure_path on the RHEL family is /sbin:/bin:/usr/sbin:/usr/bin — no /usr/local/bin, which is exactly where composer lands. Same build, same code, dies with composer: command not found on one provider and not the other. The systemd unit had always listed the directories explicitly; build and exec now use that same resolved list.
• Published vendor assets go stale silently. public/vendor/livewire was pinned at May while composer moved on — the browser running one release's client against another's server. Nothing in the deploy chain refreshed them. Now it does, on every deploy.

5. Errors that lie about themselves

Two today, and I keep finding these.

Dropping a database role that owns a database reported "the PostgreSQL service on this server is not running" — about a server that had just answered. The daemon returned an ownership refusal; the exec helper treated any non-zero exit as "wrong port", retried the default port where nothing listens, and returned that error. So the operator was told to start a running service instead of being told which database the role owns. The fallback now only advances when the client couldn't reach a server at all: a daemon that answered is the daemon, whatever it answered.

And a permission check that threw instead of returning false. Platform tools served from the root have no tenant resolved, but their gates rested on permission tables that live in each tenant's own database — so the check didn't deny, it raised "No database selected". The Gate swallowed the exception and showed "This action is unauthorized" to an operator who genuinely held the permission. Now the check answers false when no tenant is current, which is the correct answer anyway: you hold no roles outside an organisation.

Both are the same failure: an error message that is confidently about the wrong thing costs more than no error message, because it sends someone to debug a system that was never broken.

6. The features, briefly

• Self-managed tenant databases, bounded by a namespace. Some apps provision a database per tenant themselves, and the least-privilege user the platform mints can't create anything — so the capability got granted by hand, on the live box, with a manual revoke after. Now it's a per-user permission with a namespace prefix bounding it. On MySQL the daemon itself enforces the bound via a pattern grant (and yes: _ is a single-character wildcard, so the underscore in the prefix has to be escaped or app_tenant_% also matches appXtenantY). On PostgreSQL it genuinely cannot be enforced — CREATEDB is a cluster-wide role attribute and no event trigger fires for CREATE DATABASE — so the UI says that plainly rather than implying a boundary that isn't there. What is enforceable there is reading, so CONNECT is revoked from PUBLIC on every database the role doesn't own. Off by default, audited on grant, on revoke, and on refusal.
• The application log is its own source, not an append to the journal. Reading a framework's own log files is a different question from reading the service journal, and merging them made both harder to read.
• Branded, self-contained error pages for a multi-tenant app — self-contained meaning they render when the thing that broke is the thing that styles them.
• Membership categories as tenant data with a fee UI, instead of a hard-coded enum. The usual trade: an enum with label()/color() is lovely until two tenants disagree about the list.
• A full dependency sweep, majors included — Laravel 13.26, Livewire 4.4, Pest 4 → 5 with PHPUnit 13. The architecture tests, which are the most version-sensitive part of any suite, passed unchanged. That's the whole reason to have them.

7. And on the public site: a blog with a git-backed CMS

developers-hub-my/website got /blog and an authoring UI at /admin, which is a nice palate cleanser after all of the above.

Posts are markdown in content/blog and the filename is the URL. A prebuild step validates front matter against a Zod schema with .strict(), renders with marked + highlight.js, and writes a generated JSON plus rss.xml. Strict is the point: a typo'd or invented key fails the build with a readable error naming the field, instead of rendering undefined into a page. Markdown is parsed at build time only, so no parser reaches the bundle.

The CMS is Sveltia — git-backed, CDN-served, no backend and no database. Publishing is a commit; the host rebuilds. Two things I'd flag from it:

Generated files must not be tracked. rss.xml and sitemap.xml were in git and written by prebuild. Posts published through the CMS commit markdown only — the CMS never runs a build — so the tracked copies went stale the instant anyone published, and an untracked rss.xml had already blocked a git pull outright. Untracked now.

Don't style a third-party UI by its button variants. A CSS rule hid .primary on the sign-in screen to present a single door. But Sveltia's variants are contextual, not semantic: on localhost the primary button is "Work with Local Repository", while in production the GitHub sign-in is the primary. So the rule hid the wrong button on the live site and offered only the one path that needs a token nobody can create. The lesson generalises past this library — if a selector encodes a visual role rather than an identity, it will eventually mean something else somewhere else.

---

The through-line

Every one of these was invisible where it was written and obvious where it ran. CSP off locally. SQLite in the suite, PostgreSQL in production. Route caching never invoked in dev. Seeders that only run at provisioning. Root in one place, sudo -n in another.

You can't eliminate the gap. But you can stop letting each one be a surprise: encode the rule as a test that doesn't need the environment. No view declares a remote asset. No two routes share a name. No identifier reaches a uuid column unless it looks like a uuid. None of those need production to run — they just need someone to have been burned once and written it down.

Today was mostly writing them down.