DEV Community: Eric Mollenthiel

Cloudflare was 403-ing ChatGPT, Perplexity and Claude on my site, and my logs never knew

Eric Mollenthiel — Sun, 09 Aug 2026 05:50:43 +0000

For three weeks I wrote content aimed squarely at answer engines. An llms.txt,
FAQPage JSON-LD on two pages, a comparison page built to be quotable, a "how it
works" page with HowTo markup. The site is a small dating app I run on my own
server, loviam.com, Symfony and PostgreSQL on a single
box behind Cloudflare.

Referrals stayed at zero. The only external referrer my analytics table had ever
recorded, over its whole history, was chatgpt.com: three visits, one day in
July, never seen again.

I assumed the content was not good enough. The robots had never read it.

The five minute probe that should have been the first thing I did

Nothing in my usual instruments could see the problem, so I stopped looking at
them and asked the site directly, once per user agent, at its real public
address:

for ua in \
  "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +claudebot@anthropic.com)" ; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://www.loviam.com/)
  echo "$code  $ua"
done

200  Googlebot
403  OAI-SearchBot
403  PerplexityBot
403  Claude-SearchBot

Same result for ChatGPT-User, Claude-User and MistralAI-User. Meanwhile
bingbot, DuckDuckBot, Applebot, YandexBot and an ordinary browser all
got a 200.

The 403 body was eight bytes long, blocked., with server: cloudflare in the
headers, and there was no matching line in the Apache access log. The refusal
happened at the edge. My origin never heard about it.

Why every instrument I had was blind to this

This is the part worth stealing, because the failure mode generalises.

Application logs cannot record a request that never arrives. Obvious once
said, easy to forget when you are grepping the access log for PerplexityBot
and concluding "it never came".

First party analytics is worse than blind, it is reassuring. My beacon fires
from the page. A robot that gets a 403 never gets the page, so it never appears,
so the dashboard looks exactly like "nobody is interested".

My deployment smoke tests talked to the wrong server. They run with
curl --resolve www.loviam.com:443:127.0.0.1, which is the right call for a
deployment gate: you want to test the code you just shipped, not the CDN cache.
But it means the whole test suite speaks to Apache directly and Cloudflare is
invisible to it by construction. Thirty-eight green checks, every deploy, on a
path no visitor uses.

And the managed robots.txt actively pointed the wrong way. Cloudflare
injects its own block above yours. Mine named GPTBot, ClaudeBot, CCBot,
Bytespider, Amazonbot, meta-externalagent, Google-Extended and
Applebot-Extended, all of them training crawlers. Blocking those is a
defensible editorial decision and I stand by it.

But the HTTP filter was blocking something else: the search and on-demand
reading robots, which were not named in that file at all. So a perfectly
obedient crawler read "you are allowed", requested the page, and got a 403. That
mismatch between the two settings is the tell. It is not a policy, it is a
misconfiguration, and reading either file alone will never show it to you.

The fix is one toggle, and it is not the one named after crawlers

Cloudflare dashboard, zone level: Security → Settings → Block AI bots.

Not "AI Crawl Control", which is where I looked first and where the interesting
per-crawler table lives. The switch that returns the 403 is the plain one in
Security Settings, and it treats "AI bot" as a single category: the crawler that
builds ChatGPT's search index and the crawler that scrapes you for training data
are the same thing to it, even though for a publisher they are close to
opposites. One sends you traffic. The other does not.

Also worth checking afterwards: the managed robots.txt block, so it does not
still name a robot you have just decided to let through.

The guard I put in, and why it is a timer and not a test

What broke here is not my code. It is an edge configuration that can change
without a deploy, from a dashboard, possibly by a provider default I never
chose. A test in CI would only prove it was fine at build time.

So it is a daily command instead, app:seo:check-crawlers, which:

probes the real public URL, through Cloudflare, never --resolve;
walks sixteen user agents in three families: search, answer engines, training;
checks two independent things per robot, the actual HTTP status and whether the served robots.txt names it in a Disallow, because those two disagreeing is the exact signature I missed;
keeps an ordinary browser user agent as a control, so that "the robots are blocked" is never confused with "the site is down";
fails for search and answer engines, and never for training crawlers, which stay blocked on purpose.

A systemd timer runs it every morning and mails me only on state change.
An alert that fires every day is an alert nobody reads.

The second thing Cloudflare was doing, which was worse

While I was in there, I checked what else the proxy changed about requests, and
found the site had been taking itself down for weeks.

Every request reached Apache carrying the IP of a Cloudflare relay.
mod_remoteip was loaded but never configured, so nothing restored the real
client address:

172.71.135.63 - - [25/Jul/2026:12:21:03 +0000] "GET /fr/ HTTP/2.0" 200 10399

fail2ban reads those logs. So fail2ban was banning Cloudflare's own relays.
Cloudflare could then no longer reach my origin, and served HTTP 521 to every
visitor routed through that relay. The correlation was exact, to the second:

18/07 14:50:39  fail2ban  [apache-auth] Ban 141.101.98.192   (a Cloudflare range)
18/07 14:50:39  uptime probe: DOWN (HTTP 521)
18/07 15:49     unban cascade
18/07 15:54:02  uptime probe: RECOVERED

Five episodes in one week, the longest 42 minutes. My apache-auth jail triggers
at maxretry = 2, and my staging host sits behind htpasswd and therefore
manufactures 401s: two fat-fingered logins behind a shared relay were enough to
take production down for everyone else behind it.

Two quieter casualties of the same root cause: every rate limiter keyed on
$request->getClientIp() was bucketing the entire planet into a handful of
relays, and the visitor hash behind my "unique visitors" number was hashing the
relay, not the visitor. The weekly figure I had been reading to judge growth was
not counting what I thought it counted.

The fix is to trust CF-Connecting-IP, but only when the request comes from a
published Cloudflare range. Without that restriction anyone could forge the
header and walk straight past your rate limiting and your bans. Second layer, as
a net: put those same ranges in fail2ban's ignoreip, so that if the first layer
ever stops working you ban nobody rather than banning your own CDN. Both files
are generated by a monthly script, because the ranges do move, and a frozen list
would quietly reopen the trap.

What I would tell myself in June

If something sits between your users and your server, test through it, with
the user agent of the thing you care about. Not from your laptop, not with
--resolve, not from the logs. The whole class of bug here is that the failure
happens in a place none of your instruments can observe, and every instrument
you own will report a calm, plausible, completely wrong "nobody came".

Splitting €10 three ways: the largest remainder method, and the tiebreak everyone forgets

Eric Mollenthiel — Fri, 07 Aug 2026 23:44:05 +0000

Split €10 three ways and you get €3.33, €3.33, €3.33. That is €9.99. One cent has gone missing, and you now have to decide, in code, who pays it.

It sounds like a rounding detail. It is not: it is the difference between an app whose numbers close and an app whose numbers almost close. I hit it while building an expense-sharing app, and the fix turned out to be a voting-theory algorithm from the 1790s.

The three wrong answers

Round each share and hope. round(1000 / 3) = 333 per person, 999 total. You are one cent short of the expense. Every balance downstream inherits that error, and it compounds: fifty three-way expenses and the group's books are off by fifty cents with no line item to point at.

Give the remainder to the largest share. This is the common fix, and it is fine at €10. It is not fine when the split is uneven. If someone entered exact amounts that do not add up to the total, "absorb the difference on the biggest share" silently decides that one person pays €15 more than they typed. Three cents of rounding go unnoticed. Fifteen euros go unnoticed too, right up until someone checks.

Use floats. 0.1 + 0.2 != 0.3. You know this. The whole domain is integer cents or nothing.

The right answer is a 1792 apportionment method

The problem (divide a whole number of indivisible units proportionally to weights) is the same problem as allocating seats in a parliament to parties by vote share. Alexander Hamilton proposed a solution for the US House of Representatives in 1792. It is called the largest remainder method, and it is three steps:

Compute each participant's exact (fractional) share.
Give everyone the floor of it.
Hand the leftover units, one each, to whoever has the largest fractional remainder.

For €10 among three people: exact share is 333.33 cents each, floor is 333, allocated is 999, one cent left over. All three remainders are 0.33, so one of them gets the extra cent: 334 / 333 / 333.

The sum is exactly 1000. Always. Not approximately.

Here is the core of it, in PHP, working entirely in integer cents:

/**
 * @param array<int, float> $weights  participant id => weight
 * @return array<int, int>            participant id => cents
 */
private function prorate(int $totalCents, array $weights): array
{
    $sum = array_sum($weights);
    if ($sum <= 0) {
        throw new \InvalidArgumentException('No shares to split.');
    }

    $amounts = [];
    $remainders = [];
    $allocated = 0;

    foreach ($weights as $id => $weight) {
        $exact = $totalCents * $weight / $sum;
        $floor = (int) floor($exact);

        $amounts[$id] = $floor;
        $remainders[$id] = $exact - $floor;
        $allocated += $floor;
    }

    $left = $totalCents - $allocated;

    if ($left > 0) {
        $order = array_keys($remainders);
        usort($order, static function ($a, $b) use ($remainders, $weights) {
            $cmp = $remainders[$b] <=> $remainders[$a];
            if ($cmp !== 0) {
                return $cmp;
            }
            $cmp = $weights[$b] <=> $weights[$a];

            return $cmp !== 0 ? $cmp : ($a <=> $b);
        });

        foreach (array_slice($order, 0, $left) as $id) {
            ++$amounts[$id];
        }
    }

    return $amounts;
}

$left is bounded by the number of participants minus one, so this is never more than a handful of increments.

The tiebreak is the part people skip

Look at the usort comparator. It does not stop at comparing remainders. When two remainders are equal, which is exactly what happens in the €10-among-three case and is the case you will hit most often, it falls through to the weight, and then to the participant id.

Without that fallback you have a non-deterministic split. PHP's usort is not stable across all inputs, and even a stable sort leaves you at the mercy of insertion order. The same expense, recalculated after an edit, can hand the cent to someone else. Balances shift by a cent for no visible reason. Someone notices, does not trust the app any more, and they are right not to.

So the rule is: the tiebreak chain must terminate in something total and immutable. The id works. "Whoever was added to the group first" works. "Whatever order the hash table gave me" does not.

What this buys you: an invariant you can assert

Once every split sums exactly to its expense, a much stronger property falls out of the model for free. Each participant's balance is:

balance = what they paid - what they owe + what they sent - what they received

Sum that across every member of a group and every term cancels: every euro paid is owed by someone, every transfer sent is received. The balances of a group always sum to exactly zero.

That is not a nice-to-have, it is a test oracle. It turns "did I get the money maths right" into a single assertion you can run after every operation:

public function testBalancesAlwaysSumToZero(): void
{
    $balances = $this->calculator->forGroup($group);

    self::assertSame(0, array_sum(array_map(
        static fn (Balance $b) => $b->cents(),
        $balances,
    )));
}

Any bug that loses or invents a cent anywhere trips this: a bad split, a mishandled refund, a currency conversion, a deleted participant. It is the cheapest high-value test in the codebase, and it only exists because the splitter is exact.

Three edge cases worth stealing

Negative totals. Refunds and corrections are negative expenses. floor(-333.33) is -334, not -333, so the remainder logic inverts and you over-allocate. Take the absolute value, split that, negate at the end. Two lines, and it stops a whole category of sign bugs.

Exact amounts should refuse, not repair. If a mode lets people type each share by hand and the total does not match, do not silently fix it. Reject the input and name the gap: "you entered €85.00, the expense is €100.00, €15.00 missing". Someone who does not want to do the arithmetic has the other modes. Someone who does want to do it deserves to be told they got it wrong rather than have it quietly rewritten.

Zero-decimal currencies. Store everything in hundredths regardless. ¥1,500 is 150000. Then one integer travels the entire calculation without ever needing to know what currency it is, and formatting stays a presentation concern where it belongs.

Where this came from

I ran into all of this building Kotisso, a shared-expense tracker for flatshares, group holidays and separated parents. The zero-sum invariant is the whole design: everything else in the app is arranged so that it cannot be violated.

The largest remainder method is old, well-studied, and takes about thirty lines. If you are dividing indivisible units by proportion anywhere (money, seats, inventory, rate limits) it is probably the algorithm you want, and the tiebreak is probably the part you are about to forget.

Encrypting personal data at rest in Symfony: a 60-line Doctrine type, and the four things it breaks

Eric Mollenthiel — Wed, 05 Aug 2026 11:27:59 +0000

A review form asked me a yes/no question I could not answer with yes: do you encrypt personal data at rest and in transit?

In transit, sure, TLS everywhere, that part is free in 2026. At rest, no. The app runs on a plain VPS with no full-disk encryption, and the database held customer names, addresses and email addresses in a json column, in clear, exactly as PostgreSQL received them.

There are two ways out of that question. One is to click yes and move on, because everybody knows the reviewer will not ssh into your box. The other is to make the answer true. This post is the second one, taken from InvoicePilot, a Shopify app that issues invoices and therefore stores, by definition, the buyer's identity.

The encryption itself is the easy part, and it is about sixty lines. What follows it is the interesting part: the moment a column becomes an opaque blob, four things you took for granted stop working, and two of them will not fail loudly.

Why application-level encryption at all

The usual objection is fair: if an attacker gets your database, they probably have your application server too, and the key lives there. So what is the point?

The point is that "the attacker has your whole box" is one threat among several, and the cheaper ones are more common:

a database dump copied to a laptop, a CI job, or a support ticket;
a backup file on object storage with the wrong ACL;
a SELECT run by a future you at 2am, against production, with the output scrolling into a terminal buffer that gets pasted somewhere;
a restored snapshot handed to a contractor for debugging.

Application-level encryption does nothing against a full host compromise and everything against those four. That is a real gain, and it is the gain the compliance question is actually asking about. Nothing here protects you from yourself if you keep the key in the same dump as the data, which is the mistake at the end of this post.

Full-disk encryption solves a different problem entirely: a stolen physical drive. On a rented VM, where the disk is a network volume you do not control and the machine boots unattended, it protects roughly nothing that matters. So on a VPS, column-level encryption is not the cheap approximation of disk encryption. It is the one that fits the threat.

The cipher: libsodium, no bundle

PHP has had libsodium in core since 7.2. A secretbox is authenticated encryption (XSalsa20 for confidentiality, Poly1305 for integrity), it takes a 32-byte key and a nonce, and there is no way to hold it wrong except reusing a nonce, which random_bytes() makes a non-issue.

final readonly class TokenCipher
{
    private string $key;

    public function __construct(string $base64Key)
    {
        $key = base64_decode($base64Key, true);
        if (false === $key || \SODIUM_CRYPTO_SECRETBOX_KEYBYTES !== \strlen($key)) {
            throw new \InvalidArgumentException(
                'The encryption key must be 32 random bytes, base64-encoded.'
            );
        }
        $this->key = $key;
    }

    public function encrypt(string $plaintext): string
    {
        $nonce = random_bytes(\SODIUM_CRYPTO_SECRETBOX_NONCEBYTES);

        return base64_encode($nonce.sodium_crypto_secretbox($plaintext, $nonce, $this->key));
    }

    public function decrypt(string $encoded): string
    {
        $raw = base64_decode($encoded, true);
        if (false === $raw || \strlen($raw) <= \SODIUM_CRYPTO_SECRETBOX_NONCEBYTES) {
            throw new \RuntimeException('Invalid ciphertext.');
        }

        $nonce = substr($raw, 0, \SODIUM_CRYPTO_SECRETBOX_NONCEBYTES);
        $plaintext = sodium_crypto_secretbox_open(
            substr($raw, \SODIUM_CRYPTO_SECRETBOX_NONCEBYTES),
            $nonce,
            $this->key,
        );

        if (false === $plaintext) {
            throw new \RuntimeException('Decryption failed (wrong key or corrupted data).');
        }

        return $plaintext;
    }
}

Two details worth stating out loud, because they are what people get wrong when they roll this by hand:

The nonce travels with the ciphertext. It is not a secret, it must just never repeat under the same key. Prepending 24 random bytes and slicing them off on the way back means you never have to store or manage a nonce column.

sodium_crypto_secretbox_open() returning false is not a decoding error, it is a failed authentication. The value was truncated, corrupted or tampered with. Throwing there is the whole point of using an AEAD construction instead of raw openssl_encrypt with a mode you picked from a Stack Overflow answer.

I generate the key with one line and put it in the environment, next to the database URL:

php -r "echo base64_encode(random_bytes(32)), PHP_EOL;"

The Doctrine type

Now the part that makes it disappear from the rest of the codebase. Doctrine's custom types sit exactly at the boundary you want: one hook on the way to the database, one on the way back.

final class EncryptedJsonType extends Type
{
    public const string NAME = 'encrypted_json';

    private static ?TokenCipher $cipher = null;

    /** Test seam: lets a test drive the type without touching the environment. */
    public static function setCipher(?TokenCipher $cipher): void
    {
        self::$cipher = $cipher;
    }

    public function getName(): string
    {
        return self::NAME;
    }

    public function getSQLDeclaration(array $column, AbstractPlatform $platform): string
    {
        return $platform->getClobTypeDeclarationSQL($column);
    }

    public function convertToDatabaseValue($value, AbstractPlatform $platform): ?string
    {
        if (null === $value) {
            return null;
        }

        return $this->cipher()->encrypt(json_encode($value, \JSON_THROW_ON_ERROR));
    }

    public function convertToPHPValue($value, AbstractPlatform $platform): ?array
    {
        if (!\is_string($value) || '' === $value) {
            return null;
        }

        // Rows written before the switch are still plain JSON.
        $json = ('{' === $value[0] || '[' === $value[0])
            ? $value
            : $this->cipher()->decrypt($value);

        $decoded = json_decode($json, true, 512, \JSON_THROW_ON_ERROR);
        if (!\is_array($decoded)) {
            throw new \RuntimeException('An encrypted JSON column must hold an object.');
        }

        return $decoded;
    }

    private function cipher(): TokenCipher
    {
        // Doctrine builds its types statically, outside the container: read the
        // key the way Dotenv exposes it to every process (bin/console, FPM,
        // the messenger worker and migrations alike).
        $key = $_SERVER['ENCRYPTION_KEY'] ?? $_ENV['ENCRYPTION_KEY'] ?? '';

        return self::$cipher ??= new TokenCipher(\is_string($key) ? $key : '');
    }
}

Registered once:

doctrine:
    dbal:
        types:
            encrypted_json: App\Doctrine\EncryptedJsonType

And used like any other type, which is the whole payoff. Everything above the entity, the repositories, the controllers, the PDF renderer, the tests, keeps handling a plain PHP array:

/** @var array<string, mixed> */
#[ORM\Column(type: EncryptedJsonType::NAME)]
private array $buyer;

Three decisions in there deserve a sentence each.

The type reads $_SERVER directly, and that is not laziness. Doctrine instantiates types through a static registry, before and outside the service container. There is no constructor injection available, and there is no container to ask. Reading the same superglobal that Symfony's Dotenv component populates is what makes the type behave identically under FPM, bin/console, a Messenger worker and a migration, which is exactly the set of processes that will touch this column.

The static setter is a test seam, and it earns its keep. A unit test sets a fixed key, exercises the round trip, and asserts on the stored value:

public function testTheStoredValueLeaksNothing(): void
{
    $stored = (string) $this->type->convertToDatabaseValue(self::BUYER, $this->platform);

    self::assertStringNotContainsString('Claire', $stored);
    self::assertStringNotContainsString('example.com', $stored);
}

That second assertion is the one I would keep if I could only keep one. It is the test that fails the day someone "simplifies" the type back to a plain JSON column, and it is worth more than any amount of documentation saying please do not.

Reading tolerates plaintext, writing never produces it. A JSON document always starts with { or [; a base64 ciphertext, in practice, does not. That single-byte check buys a deploy where old rows and new rows coexist, and it is the difference between a migration you can run calmly and one that has to be atomic with the code deploy.

Update, 9 August 2026. A reader pushed back on that paragraph, and the objection holds: detecting plaintext by its first byte leaves the storage format ambiguous forever. A row written through another path, or edited by anyone holding a database write, opts out of decryption merely by looking like JSON. That is a permission check spelled as a guess. The fix is to record the format instead of inferring it: every value now reads enc:v1:<key id>:<base64>, anything else is refused, and the key id turns the "prefix a key id now" advice further down this post into an actual rotation path. The section The envelope, and why the first byte was a mistake, near the end, has the code and what it costs.

Migrating the rows you already have

Changing the column type is one line. Rewriting the existing rows is the migration, and Doctrine Migrations has the right hook for it: postUp() runs after the schema change, with the new type available.

public function up(Schema $schema): void
{
    $this->addSql('ALTER TABLE invoice ALTER buyer TYPE TEXT');
}

public function postUp(Schema $schema): void
{
    $type = Type::getType(EncryptedJsonType::NAME);
    $platform = $this->connection->getDatabasePlatform();

    foreach ($this->connection->iterateAssociative('SELECT id, buyer FROM invoice') as $row) {
        $buyer = $row['buyer'];
        if (!\is_string($buyer) || '' === $buyer || ('{' !== $buyer[0] && '[' !== $buyer[0])) {
            continue; // already encrypted
        }

        $this->connection->update(
            'invoice',
            ['buyer' => $type->convertToDatabaseValue(json_decode($buyer, true), $platform)],
            ['id' => $row['id']],
        );
    }
}

Note iterateAssociative() rather than fetchAllAssociative(): the migration streams, so the memory it uses does not depend on how many rows you have. And the same loop lives in a console command (app:encrypt-buyers), because the migration is not the only way a plaintext row can appear. A rollback to a previous release, or a dump restored from before the switch, both produce rows the migration will never see again. Having a command means the answer to "prove the database holds no readable personal data" is something you run, not something you believe.

Write the down() too, and make it decrypt. A migration you cannot reverse is a deploy you cannot abort.

What breaks, part 1: you cannot query it any more

This is the obvious one, and it is still the one that bites.

An encrypted column has no equality, no LIKE, no index, no ORDER BY, no GROUP BY. Two rows holding the same email produce two different ciphertexts, because the nonce differs. Every predicate you had on that data is gone, and the compiler will not tell you.

In my case exactly one query touched it, and it came from an obligation I could not drop: the GDPR data-request webhook, which must answer "what do you hold about this customer?" given an email address. It used to be a JSON predicate in SQL. It became a stream in PHP:

/**
 * The buyer block is encrypted at rest, so no SQL predicate can match on
 * it: the rows are streamed and compared in PHP (case-insensitively, as
 * an email is). This runs on the GDPR customer webhooks only, never on a
 * merchant-facing page.
 *
 * @return list<string>
 */
public function findNumbersByShopAndBuyerEmail(Shop $shop, string $email): array
{
    $connection = $this->getEntityManager()->getConnection();
    $type = Type::getType(EncryptedJsonType::NAME);
    $platform = $connection->getDatabasePlatform();
    $needle = mb_strtolower(trim($email));

    $numbers = [];
    $rows = $connection->iterateAssociative(
        'SELECT number, buyer FROM invoice WHERE shop_id = :shop ORDER BY number',
        ['shop' => $shop->getId()],
    );

    foreach ($rows as $row) {
        $buyer = $type->convertToPHPValue($row['buyer'], $platform);
        $buyerEmail = \is_array($buyer) ? ($buyer['email'] ?? null) : null;
        $number = $row['number'];

        if (\is_string($buyerEmail) && \is_string($number)
            && mb_strtolower(trim($buyerEmail)) === $needle) {
            $numbers[] = $number;
        }
    }

    return $numbers;
}

Yes, that is a full scan per tenant, decrypting as it goes. I am fine with it, and the docblock says why: it runs on a webhook that fires a handful of times a year, never on a page a merchant waits for. The scope is narrowed by shop_id, which is indexed, so the scan is over one tenant's rows, not the table.

The decision this really forces is upstream. Before you encrypt a column, list every query that reads it, and sort them into "rare and allowed to be slow" and "on a hot path". If anything lands in the second pile, you have a genuine design choice to make, and none of the options are free:

Deterministic encryption for that one field (same input, same ciphertext, no nonce): equality queries and unique indexes come back, and you leak which rows share a value, plus you become vulnerable to frequency analysis on low-cardinality fields.
A blind index: store hash(hmac_key, normalize(value)) in a second, indexed column, query on that, keep the real value encrypted. Equality only, but it is exact and fast. This is what I would build the day the lookup moves onto a hot path.
Do not encrypt that field. Encrypting the fields that carry identity and leaving a non-identifying one queryable is a defensible line, as long as you draw it deliberately and write down why.

What is not defensible is discovering the constraint after the migration, in production, because a page that used to filter now returns nothing.

What breaks, part 2: your logs become the plaintext copy

This one is quiet, and it is the one I would go looking for first in someone else's codebase.

The same compliance form that asks about encryption also asks for access logs: prove you know who read personal data and when. The obvious implementation logs the record. Do that and you have carefully encrypted a column while writing its contents, in clear, into a file that is rotated, shipped to a log aggregator, and backed up by a completely different policy.

So the audit trail names the actor and the object, never the data:

/**
 * The trail names the shop, the document and the channel, never the
 * personal data itself: a log file must not become a second,
 * unencrypted copy of what the database encrypts.
 */
public function documentRead(Invoice $invoice, string $channel, string $format): void
{
    $this->logger->info('Document read.', [
        'shop' => $invoice->getShop()->getShopDomain(),
        'document' => $invoice->getNumber(),
        'channel' => $channel, // 'admin' (the merchant) or 'buyer-link'
        'format' => $format,
    ]);
}

While you are there, check the three other places that leak the same way: exception context (an entity dumped into a stack trace), your error tracker's breadcrumbs, and the Symfony profiler in any environment where it is enabled against real data.

What breaks, part 3: the backup that carries the key

The failure I want to spell out, because it turns the whole exercise into theatre.

Encrypting a column protects a dump only if the dump does not also contain the key. Mine did not, the key lives in an environment file, not in PostgreSQL. But the backup itself was a plain pg_dump sitting on the same machine, and half of the app's personal data lives in columns that are not encrypted anyway, because they are the merchant's own data, not their customers'.

So the dumps are now encrypted too, streaming, with the key readable only by root and the postgres user:

pg_dump -Fc "$DB" \
  | zstd -T0 -q \
  | openssl enc -aes-256-ctr -pbkdf2 -pass file:/etc/pg-backup.key \
  > "$DEST/$DB.dump.zst.enc"

Two things to get right, and both are about the day you need it rather than the day you set it up. Test the restore path before you need it, including the decryption, because an encrypted backup you cannot open is worse than no backup: it looks like a backup in the monitoring. And write down where the key is and who can read it, somewhere that is not the encrypted backup, which sounds obvious right up to the moment you watch someone reason in a circle about it.

What breaks, part 4: key rotation, which you will not do

Be honest about this one from day one. With the design above, rotating the key means decrypting every row with the old key and re-encrypting with the new one, which means both keys have to exist at once, which means the cipher needs a key-id prefix in the ciphertext, which nothing above has.

I did not build that, and I decided so explicitly: one product, one key, rotation is a scripted maintenance window and I would rather do that once in five years than carry a key hierarchy from day one. If you are in a context where rotation is a scheduled control rather than an incident response, prefix a key id to the ciphertext now, when the column is empty. It is one byte of format and it costs nothing today.

The compensating control is smaller than key rotation and worth more than it looks: the key is in the environment, so a leaked dump never contains it, and a leaked repository never contains it either. What would force a rotation is a compromised host, and on a compromised host, rotating the key is the last item on a long list.

The envelope, and why the first byte was a mistake

Added 9 August 2026, after a comment on this post.

The version above recognised a legacy plaintext row by its first byte. The objection, in one sentence: that makes the storage format ambiguous forever, and the ambiguity is exploitable, because a row that merely looks like JSON bypasses decryption whoever wrote it.

The failure is not the encryption, it is that convertToPHPValue() was asking the payload what it was. A payload is not a trustworthy narrator. The format and the key version belong next to the data, as data:

public const string PREFIX = 'enc:v1:';

public function convertToDatabaseValue($value, AbstractPlatform $platform): ?string
{
    if (null === $value) {
        return null;
    }

    $keyring = self::keyring();

    return self::PREFIX.$keyring->activeId().':'
        .$keyring->active()->encrypt(json_encode($value, \JSON_THROW_ON_ERROR));
}

public function convertToPHPValue($value, AbstractPlatform $platform): ?array
{
    if (!\is_string($value) || '' === $value) {
        return null;
    }

    if (!str_starts_with($value, self::PREFIX)) {
        // Fail closed: a readable row is a bug or an intrusion, never a fallback.
        throw new \RuntimeException('This value is not in the "'.self::PREFIX.'" envelope.');
    }

    [$keyId, $ciphertext] = self::split($value);

    return $this->decode(self::keyring()->get($keyId)->decrypt($ciphertext));
}

Three things change, and only the first is obvious.

Reading fails closed. There is no longer a branch that returns plaintext. The cost is that a dump restored from before the switch stops the application instead of quietly serving personal data in clear, which is the trade I want on a column that exists because a compliance form asked about it.

The key id makes rotation real. get($keyId) reads from a keyring: one active key that writes, any number of retired keys that only read. Rotating is now three steps that never need a flag day (promote the new key, backfill, drop the old one from the keyring), and the day a key leaves, the failure is a message naming the missing key id rather than a MAC error you have to guess at.

The backfill stops guessing too. The conversion command tries decryption first, with every key in the keyring, and only then treats the value as plain JSON. Base64 decoding of {"name":...} fails outright, so plaintext never decrypts and a ciphertext is never mistaken for plaintext. The heuristic is gone from the read path and from the migration path.

And because the type now refuses anything unenveloped, the command doubles as the check that says whether the database is fully converted:

bin/console app:encrypt-buyers --dry-run

exits non-zero as long as one row is outside the envelope, which is a cron-shaped alert rather than a paragraph in a runbook.

The whole change is about eighty lines including tests, and it was possible because the table held five rows. That is the real lesson, and it is the one this post got wrong by one step: storage formats are not refactorable. Everything else in this codebase can be changed later. A format that is already in a million rows cannot, and the version prefix that would make it changeable costs seven bytes.

The part that is not code

Two hours of work and about a hundred lines. The reason to write it up is not that any of it is clever. It is that the whole thing was triggered by a yes/no question in a form, and the interesting decision was upstream of all the code: treating the form as an audit of what was true rather than a box to tick.

That reframing found three false answers, not one. Encryption at rest was the first. Encrypted backups was the second, and it was a server-wide fix that now protects every other project on the same box. Access logging was the third, and it did not exist at all.

If you sell software to businesses, you will meet that form. Shopify, Stripe, any enterprise procurement questionnaire, the ISO 27001 checklist your first serious customer sends you. The answers are cheaper to make true while the codebase is small, and the questions are a surprisingly good roadmap for a solo developer who cannot afford a security review.

The app this comes from is InvoicePilot, compliant invoices for Shopify merchants, built on ShipAnvil, which ships the auth, billing, admin and deploy pipeline these apps stand on. If you are wiring the Shopify side of things in PHP, the previous post covers the whole embedded-app authentication path in Symfony, and the foundations are in deploying Symfony to a production VPS.

Building an embedded Shopify app in Symfony: session tokens, token exchange, and the parts nobody writes in PHP

Eric Mollenthiel — Sun, 02 Aug 2026 23:01:29 +0000

Shopify's app documentation has exactly one first-class path: Node, the
official CLI, and a Remix template that wires authentication for you.
Pick any other language and you leave the paved road at the first turn,
because everything interesting happens before your framework's router
sees the request: the app runs in an iframe inside the Shopify admin, the
browser will not give you a cookie there, and the token you need to call
the Admin API has to be traded for on the fly.

None of that is hard. It is just undocumented outside JavaScript. This
post is the complete authentication and security path for an embedded
Shopify app written in Symfony 7.4 on PHP 8.5, taken from
StockPilot, an app that passed
Shopify's App Store review and is live today. Every snippet below is
production code, not a sketch.

The one architectural decision everything else follows from

An embedded app is a page inside admin.shopify.com, in a cross-origin
iframe. Third-party cookies are dead in that context, so a PHP session
is not available to you and never will be. Shopify's answer is a
short-lived JWT called a session token: App Bridge (their JS shim,
served from Shopify's CDN) mints one per request, valid for one minute,
and attaches it to every same-origin fetch() your page makes.

Which forces a split most PHP developers do not make by default:

The HTML shell is public and carries no shop data. It is a layout, a nav, and empty containers. Serving it requires no authentication, sets no cookie, and reveals nothing.
Every byte of merchant data is behind /api, authenticated by the session token on each call.

That split is worth stating out loud because it inverts the usual
Symfony instinct (secure the controller, render the data server-side).
Here, rendering data into the shell would mean authenticating a page
load that has no credentials to authenticate with. The payoff is that
the shell is trivially cacheable and the security surface is one
firewall over one path prefix.

# config/packages/security.yaml
firewalls:
    # Webhooks authenticate via HMAC signature in the controller, not a firewall.
    webhooks:
        pattern: ^/webhooks
        security: false
    # Embedded admin API: App Bridge session token (JWT) on every request.
    api:
        pattern: ^/api
        stateless: true
        provider: users_in_memory
        custom_authenticators:
            - App\Security\SessionTokenAuthenticator
    main:
        lazy: true
        provider: users_in_memory

access_control:
    - { path: ^/api, roles: ROLE_SHOP }

stateless: true is not decoration. It tells Symfony not to try to
store the token in a session it does not have.

Verifying a session token: five checks, not one

The session token is a JWT signed HS256 with your app's client secret.
Verifying the signature is the part everyone does. The four claim checks
after it are the part that gets skipped, and each one closes a real hole:
without aud you accept tokens minted for a different app that happens
to share nothing but the algorithm; without the iss/dest match you
accept a token that claims one shop in one place and another shop
elsewhere.

// src/Shopify/SessionTokenVerifier.php
$expected = hash_hmac('sha256', $encodedHeader.'.'.$encodedPayload, $this->apiSecret, true);
if (null === $signature || !hash_equals($expected, $signature)) {
    throw new InvalidSessionTokenException('Invalid JWT signature.');
}

$now = $this->clock->now()->getTimestamp();

// exp / nbf, with a small leeway for clock skew
if (!\is_int($exp) || $now - self::LEEWAY_SECONDS >= $exp) {
    throw new InvalidSessionTokenException('Session token has expired.');
}

// aud must be *our* client id
$audiences = \is_array($aud) ? $aud : [$aud];
if (!\in_array($this->apiKey, $audiences, true)) {
    throw new InvalidSessionTokenException('Session token audience mismatch.');
}

// dest is the shop; iss must live on the same host
$shopDomain = parse_url($dest, \PHP_URL_HOST);
if (!\is_string($shopDomain) || !ShopDomain::isValid($shopDomain)) {
    throw new InvalidSessionTokenException('dest claim is not a valid shop domain.');
}
if (parse_url($iss, \PHP_URL_HOST) !== $shopDomain) {
    throw new InvalidSessionTokenException('iss and dest claims do not match.');
}

Two details that cost time if you get them wrong. Use hash_equals, not
===, on the signature: this is the textbook timing-attack surface and
it costs one function name. And take the clock from
Symfony\Component\Clock\ClockInterface rather than calling time(),
so your test suite can produce an expired token without sleeping.

A five-second leeway on exp and nbf is deliberate. The token lives
sixty seconds; a server clock a couple of seconds behind Shopify's
would otherwise reject perfectly good tokens at a rate that looks like a
random, unreproducible bug.

Token exchange, and the `expiring=1` that returns 403 without it

A session token proves who is asking. It does not let you call the
Admin API. For that you trade it for an access token, using OAuth 2.0
token exchange (RFC 8693) rather than the old authorization-code dance
with redirects. With managed installation, this is the entire install
flow: no /auth route, no redirect, no callback. The first authenticated
request from a new shop simply performs the exchange.

// src/Shopify/TokenExchanger.php
private const GRANT_TYPE = 'urn:ietf:params:oauth:grant-type:token-exchange';
private const SUBJECT_TOKEN_TYPE = 'urn:ietf:params:oauth:token-type:id_token';
private const OFFLINE_TOKEN_TYPE = 'urn:shopify:params:oauth:token-type:offline-access-token';

public function exchangeForOfflineToken(string $shopDomain, string $sessionToken): AccessToken
{
    return $this->requestToken($shopDomain, [
        'grant_type' => self::GRANT_TYPE,
        'subject_token' => $sessionToken,
        'subject_token_type' => self::SUBJECT_TOKEN_TYPE,
        'requested_token_type' => self::OFFLINE_TOKEN_TYPE,
        'expiring' => 1,
    ]);
}

That last line is the one to copy. Ask for a non-expiring offline
token and the Admin API answers 403 (observed live on 2 July 2026),
with an error that does not say so. An expiring token lives about an
hour and is renewed with a refresh token, which Shopify rotates on
every call: whatever persists your tokens has to write the new refresh
token back, or your background jobs work for an hour and then stop.

Offline (not online) is the right request for anything that runs without
a merchant present: webhook handlers, nightly digests, sync jobs.

The authenticator, and the header that saves your error rate

Symfony's custom authenticator maps onto this cleanly. The passport is
self-validating because the JWT signature is the credential check.

// src/Security/SessionTokenAuthenticator.php
public function supports(Request $request): bool
{
    return str_starts_with($request->headers->get('Authorization', ''), 'Bearer ');
}

public function authenticate(Request $request): Passport
{
    $jwt = substr($request->headers->get('Authorization', ''), \strlen('Bearer '));

    try {
        $sessionToken = $this->sessionTokenVerifier->verify($jwt);
        $shop = $this->shopInstaller->ensureInstalled($sessionToken);
    } catch (InvalidSessionTokenException|TokenExchangeFailedException $e) {
        throw new CustomUserMessageAuthenticationException($e->getMessage(), [], 0, $e);
    }

    return new SelfValidatingPassport(new UserBadge(
        $sessionToken->shopDomain,
        static fn (): ShopUser => new ShopUser($shop, $sessionToken->subject),
    ));
}

private function unauthorized(): JsonResponse
{
    return new JsonResponse(
        ['error' => 'Unauthorized.'],
        Response::HTTP_UNAUTHORIZED,
        ['X-Shopify-Retry-Invalid-Session-Request' => '1'],
    );
}

X-Shopify-Retry-Invalid-Session-Request is worth a paragraph of its
own. Tokens last sixty seconds, so a page left open on a merchant's
second monitor will eventually fire a request with a token that expired
while they were in another tab. Return a bare 401 and the merchant sees
an error. Return 401 with that header and App Bridge silently fetches
a fresh token and retries once. One header turns a class of user-visible
failures into nothing at all.

Note also where installation happens: ensureInstalled() inside
authenticate(). A shop that appears with a valid token and no row in
your database is a new install, and the token exchange happens right
there. There is no separate install endpoint to secure.

Webhooks: raw body, constant time, and the query HMAC that is not `http_build_query`

Webhooks carry no session token. They are signed: HMAC-SHA256 over the
raw request body, base64-encoded in X-Shopify-Hmac-Sha256. Raw
means raw, before any JSON decoding, before any middleware touches it.
Shopify's automated App Store review sends a deliberately mis-signed
webhook and requires a 401.

public function verifyWebhook(string $rawBody, string $hmacHeader): bool
{
    if ('' === $hmacHeader) {
        return false;
    }

    $expected = base64_encode(hash_hmac('sha256', $rawBody, $this->apiSecret, true));

    return hash_equals($expected, $hmacHeader);
}

The second signature type is the sharp one. Links from the Shopify admin
carry an hmac query parameter computed over the sorted query string,
and the message Shopify signs is not a URL-encoded query string. Only
&, % and = are escaped, and only in the places shown below.
Reaching for http_build_query() here produces a signature that is
wrong for any value containing a space or a slash, which is exactly the
kind of bug that passes every test you thought to write:

$pairs = [];
foreach ($query as $key => $value) {
    $pairs[] = strtr($key, ['&' => '%26', '%' => '%25', '=' => '%3D'])
        .'='
        .strtr($value, ['&' => '%26', '%' => '%25']);
}

$expected = hash_hmac('sha256', implode('&', $pairs), $this->apiSecret);

Two things the App Store checks that are pure infrastructure

Access tokens encrypted at rest. A shop's access token is a
credential for someone else's business. Storing it in plaintext means a
read-only SQL injection anywhere in your app hands over every merchant's
store. Libsodium makes this eight lines, and PHP ships it:

// src/Shopify/TokenCipher.php — XSalsa20-Poly1305 secretbox
public function encrypt(string $plaintext): string
{
    $nonce = random_bytes(\SODIUM_CRYPTO_SECRETBOX_NONCEBYTES);

    return base64_encode($nonce.sodium_crypto_secretbox($plaintext, $nonce, $this->key));
}

The key is 32 random bytes, base64-encoded, in an environment variable,
and the constructor rejects anything else at boot rather than at the
first decrypt.

A frame-ancestors CSP scoped to the current shop. Your app must be
frameable by the merchant's admin and by nobody else, which means the
header is computed per request from the shop parameter, not set once
in the vhost:

// src/EventListener/EmbeddedAppHeadersListener.php
$frameAncestors = ShopDomain::isValid($shop)
    ? \sprintf('frame-ancestors https://%s https://admin.shopify.com;', $shop)
    : "frame-ancestors 'none';";

$event->getResponse()->headers->set('Content-Security-Policy', $frameAncestors);

Deny-all on the fallback is the correct default: a request without a
valid shop is not an admin iframe, and it should not be framed at all.
That request is also your public landing page: the app URL registered in
the Partner Dashboard is the address a crawler or a curious merchant
reaches from a link outside the admin, and serving them the App Bridge
shell shows an inert blank page. Branch on the shop parameter and
render marketing HTML instead.

One more App Store requirement that is easy to miss in a Twig layout:
App Bridge must be the first script in <head>, loaded synchronously
from Shopify's CDN, never bundled, never deferred, on every embedded
page.

The Symfony bug that this app found, which has nothing to do with Shopify

Worth knowing whoever you build for. This email subject line lost its
first two words in production, silently:

digest.attachment: 'Attached: your restock list (%count% items).'

It rendered as "your restock list (12 items)." because
%count% is numeric, which routes the string through Symfony's
pluralization logic. There, in
symfony/translation-contracts/TranslatorTrait.php, each part is tested
against '/^\w+\:\s*(.*?)$/' — an explicit-interval syntax for keyed
plural rules — and a message that innocently begins with a word followed
by a colon matches. The prefix is consumed as if it were a rule name. No
exception, no log line, just a shorter sentence.

Two lessons, one specific and one general. Specific: with %count% in a
message, never start it with Word:. General, and the more expensive
one, is why the test suite missed it. The assertion checked
assertStringContainsString('restock list', $subject) — it started
matching in the middle of the sentence, so it could only ever have
verified the part that never breaks. Assert a string from its first
character. The fix now also walks both translation catalogues and fails
CI on any %count% message matching that pattern.

What this adds up to

An embedded Shopify app in Symfony is roughly 400 lines of security code
you cannot copy from the docs: a JWT verifier, a token exchanger with
refresh-token rotation, a stateless authenticator, an HMAC verifier with
two algorithms, an encryption wrapper, and a response listener. After
that, it is a Symfony app like any other, with Doctrine, Messenger for
the background sync, and the same testing tools you already use. The
platform-specific surface is small and it stays where you put it.

The app that produced this code is StockPilot,
low-stock alerts for Shopify merchants, live in the App
Store, built on
ShipAnvil with its auth, billing, admin and deploy pipeline
already in place. If you are weighing an inventory tool because of the
Stocky shutdown on 31 August 2026,
that page is an honest map of which replacement covers which part,
including the parts StockPilot does not do.

For the Symfony foundations under all of this, start with the
production VPS deploy and
Symfony 7.4 LTS support math.

https://shipanvil.com/blog/shopify-embedded-app-symfony

A league season is not a long tournament: four product bugs my test suite could never catch

Eric Mollenthiel — Fri, 31 Jul 2026 00:56:22 +0000

I built a football prediction game for the 2026 World Cup. A month long, 104 matches,
one winner at the end. It worked, people played, nothing caught fire.

Then I pointed the same app at a domestic league: Ligue 1, 306 matches, 34 matchdays,
nine months. Same data model, same scoring, same templates. Nothing crashed. Nothing
threw. Every test stayed green.

And almost every product decision inside the app was suddenly wrong.

Here are the four that mattered, because none of them were technical, and none of them
would have shown up in a test suite.

1. A cumulative leaderboard is decided by October

In a tournament, the overall standings are the whole game. You play for four weeks and
the table at the end is the story.

Over 34 matchdays, that table stops being a game around week eight. The player who
started well is 60 points ahead, the player who joined in November is mathematically out,
and everyone else is reading a scoreboard they cannot change. The product still worked.
It just had no stakes left.

The fix was not a better algorithm, it was a second unit of time: a per-matchday
leaderboard, so each weekend has its own winner, plus a season honours table counting
how many matchdays each player has won. Same points, same scoring, sliced differently.
A player who is 14th overall can still win this weekend, and that is the thing that
makes them come back on Friday.

Worth noting what I did not do: no reset, no handicap, no catch-up bonus. Anything
retroactive on a scoring system that people are currently playing destroys trust in the
standings, and the standings are the entire asset.

2. A sliding 24 hour reminder becomes 100 emails

The reminder job was built for a tournament: "if a player has unpredicted matches
kicking off in the next 24 hours, email them." Matches trickle in daily, so that reads
as one email a day, and it is fine.

A Ligue 1 matchday runs from Friday 20:45 to Sunday 20:45. The same job, unchanged,
would have sent three emails per weekend to the same person: one on Friday for one
match, one on Saturday for three, one on Sunday for five. Around a hundred emails per
season, per player. That is not a reminder, that is a spam complaint with extra steps.

It would also have arrived on the morning of the first match, when the natural gesture in
a league is the opposite: you fill all nine games in one sitting, once, whenever you
think of it.

So the job now has two disjoint code paths in the same command. Tournaments keep the
sliding window. Leagues get one email per matchday, fired when the first kickoff of
that matchday is 24 to 48 hours out, listing every still open match of the matchday.

The part I like: there is no reminder_sent table. Idempotency comes from the window
itself. "The first kickoff is between 24 and 48 hours away" is true on exactly one
calendar day, and the cron runs once a day. The one case that legitimately produces a
second email is a postponement that drags the first kickoff back into the window, and a
calendar that moved is exactly when you want to remind people again.

The tradeoff is written in the crontab in plain words: doubling the cron frequency would
double the emails. A comment is cheaper than a table, as long as the comment is where the
mistake would be made.

3. "Upcoming matches" is not "all future matches"

The dashboard listed every future match. In a tournament that is at most a few dozen
cards, and the progress badge reads "12/18 predicted", which feels achievable.

In a league it is 306 cards and "12/306", which feels like homework.

Now the list is bounded to the next two matchdays. The subtle part is how you pick
them. My first version took MIN(round_number) over unplayed matches. That is wrong in
any real league, because postponements are routine: one match of matchday 3 replayed in
November would have pinned the dashboard to matchdays 3 and 4 and hidden the actual
weekend. The window follows nearest kickoff times, not round numbers, and the
postponed match reappears by itself when its new slot comes around.

4. In a tournament there is always something to do

Four weeks of a World Cup is four weeks of permanent attention. Nine months is not. Most
of a league season, an engagement product is competing with the user forgetting it
exists.

The two moments I built for are both moments the app already knew about and was throwing
away: the player who just won a matchday (one per week, peak pride) and the player
who just finished predicting the coming matchday (everyone, 34 times a season, peak
engagement and then several days of nothing). Both now offer a share, native share sheet
on mobile with a WhatsApp fallback, and both link to the public competition page rather
than the private league.

That last detail took a minute of thought and is worth the minute: sharing a league
invite code from a mobile share sheet means it can land in a public post, and a private
standings table with strangers in it is not a feature.

The actual lesson

The data model was right. Event, Game, Prediction, a round_number column that was
already there. Not one migration was needed for any of this.

What was wrong was every assumption about cadence: how often the user shows up, how
long a unit of competition lasts, how far ahead they can see, how long they wait between
two moments of interest. Those assumptions are almost never in your schema. They are
spread across cron expressions, query limits, email conditions, and empty state copy,
which is exactly where nobody looks when they say "we just need to support a new
competition format".

If you are about to reuse a working product on a longer or shorter timescale, grep your
codebase for time: every 24 hours, every setMaxResults, every "next" and "current"
and "upcoming". That is your real diff.

The app is a free prediction game for friends and coworkers, no betting and no money
involved, built with Symfony, Turbo Streams over Mercure for live standings, and 16
locales. It is at pronoarena.com if you want to see the result,
and the Ligue 1 season starts on August 21.

Building a production-grade SaaS foundation in Symfony in 2026 — lessons from building ShipAnvil

Eric Mollenthiel — Mon, 20 Jul 2026 12:51:17 +0000

I've been writing Symfony professionally for 15 years. Over those years I started enough SaaS projects to notice an embarrassing pattern: the first two or three weeks were never about the product. They were email verification, password reset, 2FA, billing webhooks, tenant scoping, transactional emails, and a deploy checklist I kept half-remembering.

This year I built that foundation one final time — properly, with tests — and turned it into a product (ShipAnvil). This article is the technical write-up: the architecture decisions I'd defend in a code review, the ones that surprised me, and the parts that are harder than they look. Everything here stands on its own — you can take these patterns and build your own foundation with them.

1. The stack: boring on purpose, and zero Node.js

Symfony 7.4 LTS + PHP 8.4/8.5, PostgreSQL.
The LAST stack: AssetMapper, Stimulus, Turbo, Live Components.
Tailwind CSS v4 via symfonycasts/tailwind-bundle.

The controversial choice is the absence of Node.js. AssetMapper serves ES modules with an importmap; the Tailwind bundle wraps a standalone binary. The practical consequence is bigger than it sounds: a new machine goes from git clone to a running app with PHP and PostgreSQL alone. No lockfile drift, no build pipeline to babysit in CI, one less runtime in production.

In 2024 I would have hedged on this. In 2026, after shipping real features with Live Components and Turbo, I wouldn't go back for a standard SaaS UI.

2. Billing: one interface, two providers, and a fake one

The single highest-leverage decision in the codebase. Stripe is the default answer, but Merchant-of-Record providers (Lemon Squeezy, Paddle) solve VAT for solo developers — a very real pain if you sell from the EU. I wanted both, swappable by env var:

#[AutoconfigureTag('billing.payment_provider')]
interface PaymentProviderInterface
{
    public function name(): BillingProvider;

    /** Creates a hosted checkout session for the plan and returns its URL. */
    public function createCheckoutUrl(
        Organization $organization,
        Plan $plan,
        User $user,
        string $successUrl,
        string $cancelUrl,
    ): string;

    /** Provider-hosted customer portal (payment methods, invoices, cancellation…). */
    public function createPortalUrl(Subscription $subscription, string $returnUrl): string;

    public function cancelAtPeriodEnd(Subscription $subscription): void;

    public function resume(Subscription $subscription): void;

    // … webhook verification, see below
}

Three implementations: StripeProvider, LemonSqueezyProvider, and — the one I underestimated — FakeProvider. The fake one runs the entire subscription lifecycle (checkout, upgrade, dunning, cancellation) locally with no account, no API key, no network. It exists for tests, but it turned out to be the best onboarding feature: you can develop your whole billing UX before deciding which provider you'll even use.

Two non-obvious rules this interface encodes:

Hosted checkout only. Embedding payment forms means PCI scope and provider-specific JS. Redirecting to Stripe Checkout / LS hosted checkout keeps the integration surface to "create a URL, handle a webhook".
The organization id must travel with the checkout (metadata/custom data), because the webhook is the source of truth that attaches the subscription — the redirect back to your app is not guaranteed to happen.

3. Webhooks: the part that will page you at 3am

Every billing tutorial shows the happy path: event arrives, update the database. Production has other plans — duplicate deliveries, out-of-order events, retries during your deploy window. The pattern that survives all of that is verify → store → ACK → process async:

#[Route('/webhooks/{providerSlug}', name: 'billing_webhook', methods: ['POST'])]
public function __invoke(string $providerSlug, Request $request): JsonResponse
{
    $provider = $this->providers->fromSlug($providerSlug);
    // …

    try {
        $identity = $provider->verifyWebhook($request); // signature check on the RAW body
    } catch (WebhookVerificationFailed $exception) {
        return new JsonResponse(['error' => 'Invalid webhook signature or payload.'], 400);
    }

    $existing = $this->webhookEvents->findOneByProviderEventId($provider->name(), $identity->eventId);

    if (null !== $existing) {
        if ($existing->isProcessed()) {
            return new JsonResponse(['status' => 'already processed'], 200);
        }
        // Stored but not (successfully) processed yet: requeue it.
        $this->bus->dispatch(new ProcessWebhookEvent((string) $existing->getId()));

        return new JsonResponse(['status' => 'requeued'], 202);
    }

    $event = new WebhookEvent($provider->name(), $identity->eventId, $identity->type, $identity->payload);
    // persist + dispatch…
}

The details that matter:

Verify the signature on the raw request body. Any framework normalization (decoding, re-encoding) breaks HMAC comparison. This is the most common webhook bug I've seen in the wild.
Idempotency lives in the database, as a unique constraint on (provider, event_id) — not in application logic. Two PHP-FPM workers receiving the same delivery concurrently will both pass an if check; only one will survive the constraint.
ACK before processing. The webhook HTTP response should take milliseconds. The actual state change happens in a Messenger handler with retries. Symfony's Doctrine transport means this needs no Redis — a database table is a perfectly good queue at this scale.
Test with replayed fixtures. The suite replays full billing lifecycles from signed webhook fixtures — including duplicates and out-of-order deliveries. Those are test cases, not incidents.

4. Multi-tenancy: a Doctrine filter, and the tests that earn the trust

For a team-based SaaS, the pragmatic model is single-database with an organization_id column. The risk is the day someone writes a query and forgets the WHERE. Doctrine's SQLFilter closes that hole globally:

final class OrganizationFilter extends SQLFilter
{
    public function addFilterConstraint(ClassMetadata $targetEntity, string $targetTableAlias): string
    {
        if (!$targetEntity->getReflectionClass()->implementsInterface(OrganizationOwnedInterface::class)) {
            return '';
        }

        return \sprintf('%s.organization_id = %s', $targetTableAlias, $this->getParameter('organization_id'));
    }
}

A request subscriber enables the filter as soon as an authenticated user with a current organization is detected. Every query against an entity implementing OrganizationOwnedInterface gets scoped automatically — ORM queries, lazy-loads, joins.

Two honest caveats, because this pattern is often oversold:

Console commands and Messenger workers run unfiltered. There is no "current request" there. You must scope explicitly in async code — and document that loudly, which is exactly the kind of footgun that belongs in a comment on the filter class itself.
A filter you haven't tested is a hope, not a guarantee. The test suite creates two organizations and asserts that org A's queries can never see org B's rows — including through relations. If you take one thing from this article: write the isolation tests before you trust the filter.

Plan gating then becomes declarative — an attribute on the controller, and a listener that redirects lower plans to the billing page:

#[RequiresPlan('pro')]
public function advancedReports(): Response { … }

5. The AI module: HttpClient, not an SDK

Every SaaS I talk to is adding an AI feature, so the foundation ships a Claude integration. Deliberate choice: no vendor SDK — a thin client on Symfony's HttpClient with retries, timeouts and SSE streaming. An LLM API is two endpoints; an SDK is a dependency tree you don't control.

The pattern I use most is structured extraction via forced tool use. You give the model exactly one tool whose input schema is your target structure, and you force it to call that tool. The answer is the structure — never prose around it, never "Here's your JSON:":

$result = $this->aiClient->complete($request); // tool_choice forces the tool

$input = $result->toolInput
    ?? throw new AiRequestFailed(\sprintf('The model did not call the "%s" tool.', $tool->name));

foreach ($tool->requiredProperties() as $property) {
    if (!\array_key_exists($property, $input)) {
        throw new AiRequestFailed(\sprintf('The extraction is missing the required "%s" property.', $property));
    }
}

return new ExtractionResult($input, $result);

Note the validation after the call: a model can return a tool call that omits required properties. Trust, but verify — then your callers get a typed array, every time.

Token usage is metered per organization in a plain table. That's the seed of usage-based billing later, and it answers "which customer is costing me money" on day one. The whole module hides behind an interface with a fake implementation, so dev and CI run without an API key — same philosophy as billing.

One more 2026 reality: developers buy code they'll modify with an agent next to them. The repo ships a CLAUDE.md describing the architecture, conventions and test commands. It costs an afternoon and it changes what Claude Code or Cursor can do with the codebase on day one.

6. Quality as a feature, not a slogan

Numbers from the codebase as of this week: 379 tests, 1,667 assertions, PHPStan at level max with zero errors, CS-Fixer, and a CI pipeline (lint + static analysis + tests against a real PostgreSQL service).

The non-obvious lesson: the test suite changed what I dared to build. Replayed webhook lifecycles made the double-provider billing tractable. Isolation tests made the Doctrine filter trustworthy. On a foundation meant to outlive its author's attention, tests aren't insurance — they're the enabling technology.

The other quality feature is rehearsed installation. make init checks your PHP version, installs dependencies, creates the database, builds the CSS and runs the full test suite. The deploy documentation is a step-by-step VPS recipe (PHP-FPM pool, Apache vhost, certbot, systemd worker for Messenger, OPcache with validate_timestamps=0 and what that implies for deploys) — because "works on my machine" is where most boilerplates quietly end.

What I'd tell you to steal

An interface in front of your payment provider, with a fake implementation — even if you only ever use Stripe.
Verify → store idempotently → ACK → process async for every webhook, with the unique constraint in the database.
A Doctrine SQLFilter for tenancy, plus the isolation tests that make it trustworthy — and explicit scoping in CLI/workers.
Forced tool use for any LLM feature that feeds your domain logic.
A CLAUDE.md in any codebase another human (or agent) will inherit.

Full disclosure: this article comes out of building ShipAnvil, a commercial Symfony SaaS boilerplate I just launched (the site itself runs on it, and there's a public demo account on the landing page). The patterns above are reproducible from this write-up alone — but if you'd rather start from the tested implementation, that's what it's for. Criticism and questions welcome in the comments; I answer everything.