DEV Community: NessFlow

Googlebot Renders Your JavaScript. ChatGPT Doesn't.

NessFlow — Tue, 15 Sep 2026 17:17:25 +0000

A few weeks ago I posted a milestone on LinkedIn: NessFlow had crossed 50 million server log lines analyzed in private beta. Most of the replies weren't about the number. They were about one line:

Sites that traditional crawlers fail to audit: WAFs, firewalls, adaptive rate limiting, 10k-SKU e-commerce catalogs, media.

Several people asked the same thing, in different words: which sites, and what exactly fails?

Here is one run, and the finding at the end of it, which had nothing to do with crawling.

The site

One of Carrefour's regional storefronts. Roughly 30,000 crawlable pages, on a PWA front end: the initial HTML response is a shell, and product, category and facet content is assembled client-side.

An SEO consultant had already tried to audit it with Screaming Frog. The crawl never produced a usable dataset.

Could Screaming Frog have crawled it?

I want to answer this honestly. The tempting version of this story, "their tool failed, ours didn't", is false, and any competent reader would take it apart in the comments.

Yes. Screaming Frog could have crawled this site. Let's be precise about why it didn't.

Scale was never the problem. 30,000 URLs is small. Screaming Frog's memory storage mode is recommended up to 500k URLs, handling a couple hundred thousand on 8GB of RAM; database storage mode goes into the millions. Anyone claiming 30k pages broke the tool is describing a misconfiguration, not a limit.

Rendering wasn't the problem either. The SEO Spider has Chromium-based JS rendering. It's expensive: you must keep resource crawling enabled, and Screaming Frog's own guidance notes most servers don't want to be crawled faster than about 5 URLs/second. So 30k rendered URLs is a multi-hour job on a laptop. Painful, not impossible.

The URL space was a real problem, but a solvable one. A grocery catalog with facets doesn't have 30,000 URLs, it has a combinatorial space. Screaming Frog handles this with exclude regex, if you already know the facet patterns. On an unfamiliar site you don't, so the first crawl diverges, and you find out at hour six.

Access was the actual blocker. One desktop, one IP, one user agent, sustained concurrency, against adaptive rate limiting and WAF edge rules. And here's the part that matters: the fixes are mostly not client-side. You can slow down and change your UA, but the reliable fix is having the client allowlist your IP at the edge, which is a permissions and relationship problem, not a software feature.

So the honest framing isn't "we crawl harder." It's: we start from the server side, where you don't need to be let in, because you're already inside.

The question a crawl doesn't answer

Here's where technical audits usually stop and where the commercial work starts.

Suppose you do get a clean crawl of 30,000 pages. You now have 30,000 rows of status codes, titles, word counts and canonicals. Almost none of that is a business finding, because in grocery e-commerce those 30,000 pages are not equal in any way that matters.

Search demand in grocery is brutally concentrated. Nobody types "plain yogurt 4×125g, brand X, reference 384920." They type "grocery delivery [city]", "buy basmati rice online", "organic products cheap". A 30,000-SKU catalog rarely carries more than a few hundred URLs with meaningful non-branded demand, and they're almost all category, subcategory and brand pages, not product detail pages.

That single fact reorganizes the audit:

Crawl budget spent on PDPs is spent on pages that will never rank and rarely convert from organic. They still need to be indexable for long-tail and structured data, but they aren't where the money is.
Facets are not uniformly bad. Most combinations are index bloat and should be closed. A small subset (organic, gluten-free, halal, brand-within-category) carries genuine standalone demand and deserves promotion to real landing pages with its own copy. Deciding which is an SEO judgement call, not a crawl setting.
SKU churn is the silent killer. Grocery rotates constantly: seasonal lines, delisted references, permanent out-of-stock. Thousands of PDPs die every month. Whether they 404, 410, redirect to their category, or stay live with alternatives changes both your index footprint and your revenue, and most storefronts have never made that decision deliberately.
Expired promotion pages accumulate. Weekly flyers and campaign pages go stale, become orphans and soft 404s, and they're often the pages that earned the site's few real backlinks.
For a chain, store pages are frequently the strongest organic asset and the most neglected: thin, templated, duplicated across hundreds of locations, competing with each other.

So the first job isn't "find broken pages." It's segment by template, then ask which templates carry demand. Everything else is triage.

Your platform picked your no-JS ratio before you did

Once you segment by template, a pattern shows up across projects that has nothing to do with any individual site. The commerce platform and theme you chose already determined how much of your catalog exists in the initial HTML.

Every major platform now offers two paths:

Platform	Server-rendered path	Headless / React path
Magento	Hyvä: PHP templates, Tailwind, Alpine.js	PWA Studio: React, Apollo, GraphQL
PrestaShop	Warehouse: Smarty templates	custom headless front end
WooCommerce	Astra: PHP/WordPress	headless WP + Next.js
Shopify	Online Store 2.0: Liquid + JSON templates, Dawn	Hydrogen: React
BigCommerce	Stencil: Handlebars	Catalyst: Next.js, React, GraphQL

The left column ships fully-formed HTML. Hyvä is the clearest case: PHP composes the page on the server, Tailwind styles it inline, Alpine.js adds reactivity to specific components. Crawlers get the content and the structured data immediately, with no hydration gap. Warehouse, Astra, Liquid and Stencil work the same way for the same reason: they never stopped being server-rendered.

The right column is not "bad for SEO," and I'd push back on anyone who says so. Hydrogen and Catalyst both server-render. The leak is narrower and far more specific:

The shell renders. Then facets, pagination, variant selection, stock status and price are fetched client-side after hydration.

Which means the page a non-rendering client sees is a category page with no products in it. The template is fine. The catalog isn't there. And because the header, nav and footer render normally, a naive word count says the page has content, which is exactly why a raw word count is a useless test and a ratio isn't.

You can check any site in one command:

# words in the raw HTML
curl -sA "Mozilla/5.0 (compatible; Googlebot/2.1)" "$URL" \
  | sed -e 's/<[^>]*>/ /g' | wc -w

Compare that to the word count of the rendered DOM. If the first number is a small fraction of the second on your category templates, the rest of this article is about you.

Two measurements

The control probe

Before crawling anything, NessFlow sends reachability probes to a handful of URLs with different identities, and one of them is deliberately fake:

$probes = [
    'googlebot' => 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
    'gptbot'    => 'Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)',
    'browser'   => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 …',

    // the control: a user agent that exists nowhere,
    // matches no rule, and is on nobody's allowlist
    'control'   => 'NessFlowReachabilityProbe/1.0',
];

The control is the whole point. On its own, "Googlebot got challenged" tells you nothing, because maybe the edge challenges everyone. Compared against a user agent no rule could possibly target, it becomes a diagnosis:

Googlebot UA	Control UA	What it means
passes	challenged	Verified-bot allowlist. No third-party crawler will ever see what Google sees.
challenged	passes	UA-based blocking of known bots. Crude, and usually accidental.
challenged	challenged	Global edge rule. The site is closed to every automated client, Google included.
passes	passes	Open. Thin pages are a rendering problem, not an access problem.

Note what the probe does not do: work around the answer. It reports the wall, it doesn't climb it. When the wall is real, the audit moves server-side.

The no-JS ratio

For every page we keep both views, raw HTML and rendered DOM, and compare word counts:

ratio = words(raw_html) / words(rendered_dom)

flag JavaScript-Only Content when:
    ratio < 0.3
  AND words(rendered_dom) >= 200

The 200-word floor matters more than the threshold. Without it, every legitimately short page (a thin category, a sold-out product, a redirect stub) trips the rule and the report becomes noise. With it, the flag means something specific: this page has real content, and a client that doesn't execute JavaScript sees almost none of it.

Run it per template, not per page. The flag almost never lights up uniformly. In practice it concentrates exactly where client-side filtering lives, in category and facet templates, which is also, per the section above, where the demand is. That coincidence is the finding. A JS-only PDP is a long-tail annoyance. A JS-only category template is the head of your demand curve.

The part that isn't technical

Googlebot renders JavaScript. It has for years, on a delay, and a well-built PWA can rank perfectly well. That fact has been used for a decade to close the conversation: "Google handles it, move on."

The crawlers that feed AI answers don't. Per Vercel's crawler analysis, GPTBot fetched JavaScript files in a minority of requests and never executed them; ClaudeBot downloaded JS and never executed it either. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot parse the HTML they first receive, take what's there, and move on. Bingbot renders partially and unreliably at scale.

Put those two facts side by side and a PWA storefront lands somewhere very specific:

It ranks in Google, and it does not exist in ChatGPT, Claude or Perplexity.

Not "ranks poorly." Does not exist. When someone asks an assistant "where can I buy organic rice online in [country]", the retrieval layer reads a shell with a nav bar and no products. The competitor on a server-rendered theme gets cited instead, and the gap is invisible in every rank tracker and every Search Console report, because neither measures retrieval by AI crawlers.

That reframes the no-JS ratio entirely. It isn't technical hygiene. It's the answer to "are we in the answer set?", and for a grocery chain the queries at stake are the high-intent ones: where to buy, delivery, availability, price.

One caveat I'd insist on: don't take Vercel's numbers, or mine, as your own. Crawler behavior changes quietly and varies per site. That's the argument for having logs at all. You can measure it directly on your own domain: which AI user agents arrived, what they requested, whether they ever fetched the JS bundles, whether they came back. We track 19 AI crawlers with their declared purpose (training / search / user-request) and whether they respect robots.txt, because several don't, by design.

On log analysis, and giving credit

Screaming Frog ships a Log File Analyser, and it's a good tool. It verifies bots against spoofed user agents, handles large files, and has done so for years. Claiming log analysis as a novelty would be silly.

The differences that made it worth building our own are narrow and specific:

It's a desktop application, project-based, disk-bound: you import a file and look at it. We needed a continuous server-side pipeline, because crawl behavior is a time series, not a snapshot.
No Search Console join. Ours joins crawl, GSC and logs on the same canonical URL hash, the registry from Part 4, which is what turns three separate reports into one answer about where the budget actually goes.
No AI-crawler taxonomy. In 2026 that's the half of the log file that decides whether you're in the answer set.

Our parser handles 13 log formats at roughly 52,000 lines/second on a single node, parsing, normalizing and joining to canonical URLs, with no IP ever persisted. IPs are used in memory for reverse-DNS bot verification and discarded; they never reach an analysis table, and never reach a model prompt.

Three things I'd keep

When a crawler stalls, name the wall: access, rendering, or URL space. They fail identically from the outside and the fix for each makes at least one of the others worse. The control probe is four HTTP requests and it tells you which one you're facing.
A crawl of 30,000 pages is not an audit. Segment by template, find the few hundred URLs that carry demand, and spend the analysis there. In grocery that means categories, brands and store pages, not the catalog.
"Google renders JavaScript" stopped being a sufficient answer. Check the ratio on your category templates, then check your logs for who actually came and what they got.

NessFlow is in private beta: crawl, Search Console and server logs joined on one canonical URL, with AI-crawler visibility built in. If you run a storefront that breaks crawlers, that's the profile I'm looking for. Reach out on LinkedIn.

Laravel SEO: one registry for every URL a crawler sees

NessFlow — Thu, 27 Aug 2026 14:03:13 +0000

Laravel 13, Blade, two locales, no SEO package.

A bilingual public site usually carries four lists of the same URLs, and each one is an SEO surface. The routes file holds one. The sitemap builder holds a second. The layout that prints canonical and hreflang tags holds a third. The language switcher in the footer holds a fourth.

Nothing makes those four agree, and they come apart in a specific way. Someone adds a page, the route lands, and the other three get updated by whoever remembers. When they do not, nothing happens. No exception, no failing test, and the page answers 200 to every visitor who has its address. The consequence surfaces weeks later in Search Console, as a page nobody ever crawled or a hreflang tag pointing at a 404. That delay is what makes URL drift an SEO problem rather than a bug: by the time it is visible, the commit that caused it is months old.

We built our public site so those four lists cannot exist. There is one table, keyed by page, and everything a crawler sees is a projection of it. The site itself runs inside the same Laravel application as the product, in Blade, which is what makes a single table plausible in the first place.

31 page keys, two locales, 136 sitemap URLs on the day this was written. One expression decides every SEO surface of the site: the routes, the canonicals, the hreflang pairs, the x-default, the sitemap, the Open Graph card, the language switcher and the navigation. Fifty files in the application read it, and forty nine test files hold it in place.

A page is a key, not a URL

The unit is a stable key. solutions.agencies is a page. Its French and English addresses are values.

'solutions.agencies' => [
    'fr-FR' => 'solutions/agences-seo',
    'en-US' => 'solutions/seo-agencies',
],
'hub.engineering' => [
    'fr-FR' => 'ingenierie',
    'en-US' => 'engineering',
],

Naming the page rather than the URL is what lets a language switcher point at the same page in the other language instead of falling back to the homepage. That fallback is the most common defect on multilingual sites, and it costs the visitor precisely the page they were reading.

The homepage slug is the empty string, which forces one decision to be explicit.

public static function slug(string $key, string $locale): ?string
{
    return self::map()[$key][$locale] ?? null;
}

It returns null, never an empty string as a fallback. An empty string is the legitimate slug of the homepage, so using it as a failure value would silently point every missing page at /.

Routes are a projection of the table

The routes file contains no literal slug at all.

foreach (MarketingLocale::all() as $localeCode => $localeAttributes) {
    Route::prefix($localeAttributes['prefix'])
        ->middleware([SetMarketingLocale::class.':'.$localeCode, CacheMarketingResponse::class])
        ->name('marketing.'.$localeAttributes['prefix'].'.')
        ->group(function () use ($localeCode) {
            Route::get(SlugRegistry::slug('pricing', $localeCode), PricingController::class)
                ->name('pricing');

One group per locale, carrying that locale's literal prefix. A single {locale} group would have been shorter, and it would have answered 200 to every slug under every prefix: /en/notre-approche and /en/our-approach both serving the same page, two URLs per page per language. That is the duplicate content our own SEO product teaches its users to hunt, so the shape of the route file is pinned by a test rather than by a convention.

it('refuses one locale slug under another locale prefix', function () {
    $this->get('/en/notre-approche')->assertNotFound();
    $this->get('/fr/our-approach')->assertNotFound();
});

The routes do get names, marketing.fr.pricing and marketing.en.pricing. No view uses them. Views resolve addresses through the registry, because the registry is the thing that guarantees navigation, hreflang and sitemap agree.

Canonical, hreflang and x-default: three SEO tags, one read

The layout resolves two values, once, at the top.

$canonical  = $canonicalOverride  ?? SlugRegistry::url($pageKey, $marketingLocale);
$alternates = $alternatesOverride ?? SlugRegistry::alternates($pageKey);

Those two values are everything the head prints on the subject.

<link rel="canonical" href="{{ $canonical }}">

@foreach ($alternates as $hreflang => $url)
    <link rel="alternate" hreflang="{{ $hreflang }}" href="{{ $url }}">
@endforeach

alternates() walks the configured locales and returns only the pairs that exist.

public static function alternates(string $key): array
{
    $alternates = [];

    foreach (MarketingLocale::all() as $code => $attributes) {
        $url = self::url($key, $code);

        if ($url !== null) {
            $alternates[$attributes['hreflang']] = $url;
        }
    }

    return $alternates;
}

Declaring a hreflang toward a page that does not exist is a crawl error, not a courtesy, so a missing pair is skipped rather than guessed. That single array then feeds three surfaces in the same request: the <link rel="alternate"> tags, the og:locale:alternate metas, and the footer language switcher. One read, so the robot and the visitor cannot be looking at different sites.

x-default is derived rather than declared. It points at whichever locale the root path serves when the visitor's Accept-Language names nothing we know, which makes it a property of the fallback rule instead of a fifth list to maintain.

The SEO sitemap is a loop, not a list

foreach (SlugRegistry::keys() as $key) {
    foreach (MarketingLocale::codes() as $locale) {
        $url = SlugRegistry::url($key, $locale);

        if ($url === null) {
            continue;
        }

        $entries[] = [
            'url' => $url,
            'alternates' => SlugRegistry::alternates($key),
            'lastmod' => $this->pageLastmod($key, $locale),
        ];
    }
}

Item pages are appended after that loop: article pages and the AI bot directory entries, each deriving its path from its index page rather than from a path built by hand. On the day this was measured that came to 136 URLs, 62 straight from the registry and 74 derived from it.

There is no priority and no changefreq anywhere in the output. Both have been ignored for years, and they survive in SEO folklore rather than in any engine's behaviour. Emitting them mostly suggests we are steering something.

lastmod is the field worth being strict about. The rule is not "pages that have a date in the database". It is "pages that already declare a dateModified to a robot", read from the same expression that renders the page. Two surfaces qualify today, the article page and the changelog. Static pages get nothing, because a lastmod recomputed on each request is wrong on each request, and that is exactly the kind of signal a search engine learns to discount. Emitting it is a net loss, not a neutral gain.

That rule is an enumeration, and an enumeration drifts, so it is held by parity in both directions.

$declared = declaredDateModified(
    (string) $this->get(SlugRegistry::path($key, $locale))->assertOk()->getContent(),
);

$sitemap = $lastmods[$url] ?? null;

// The fact must coincide, not its precision of writing.
$left  = $declared === null ? null : substr($declared, 0, 10);
$right = $sitemap  === null ? null : substr($sitemap, 0, 10);

The test fetches every registry page in both locales, parses the JSON-LD the page actually serves, and requires equivalence. A page that starts declaring a date without gaining a sitemap entry fails the suite, and so does the reverse.

It also carries a witness, which is the part that took a second pass to get right.

expect($dated)->toBe(1, 'No registry page declares a date: the test proves nothing.');

"No divergence" is equally true of a sitemap with no lastmod at all and pages that declare none. The fixture seeds exactly one dated page, so the test insists on finding exactly one before it is allowed to conclude.

The defect a single source does not prevent

Item pages cannot live in the registry. There are as many of them as there are rows in a table. They derive their path from their index instead.

public static function childPath(string $indexKey, string $locale, string $childSlug): ?string
{
    $parent = self::path($indexKey, $locale);

    return $parent === null ? null : rtrim($parent, '/').'/'.$childSlug;
}

Which means an item page has no key of its own, and for a while the layout resolved its canonical from the index key it was handed. Every article on the site declared its section listing as its canonical URL, and de-indexed itself in favour of a list. It is the most expensive SEO defect we have shipped, and it shipped green.

Nothing about that was visible. The page rendered, the content was right, the tags were well formed, and every piece we published was asking search engines to ignore it.

The fix is two override props, and the distinction between them is the part worth keeping.

'canonicalOverride' => null,
'alternatesOverride' => null,

null means no override. An empty array means no twin exists. A piece written natively in one language has no twin, and there is no correct hreflang for it. Collapsing the two values into one would make the empty case inherit the index alternates and send English readers to a French page that does not contain the article.

That distinction is observable from outside. Of the 136 sitemap URLs measured for this piece, 132 carried two xhtml:link entries and four carried one. Those four are the engineering notes, which are written in English and not translated. This one makes five.

One predicate for the page and for its absence

Our data processing agreement is written and seeded, and it is not published. The text is done; what is missing is the set of clauses that legally commit the company. It enters the table only when its gate is open.

if (TrustPack::dpaIsPublished()) {
    $map['trust.dpa'] = [
        'fr-FR' => 'accord-de-traitement-des-donnees',
        'en-US' => 'data-processing-agreement',
    ];
}

And the route file reads the result rather than the gate.

$dpaSlug = SlugRegistry::slug('trust.dpa', $localeCode);

if ($dpaSlug !== null) {
    Route::get($dpaSlug, [TrustController::class, 'dpa'])->name('trust.dpa');
}

No slug means no route, which means a 404. And since the navigation, the sitemap, the hreflang tags and the language switcher all read the same table, none of them can cite it. Every SEO surface goes dark with the page, in the same gesture. A dead link and a phantom sitemap URL are structurally impossible, because the two faces switch on and off together.

Note what is not there: an abort(404) in the controller. Two guards saying the same thing mask each other under mutation testing. Each one is sufficient, so each survives alone, and neither is genuinely tested. The absence of the route is the 404.

Eight of our nine module pillars carry a landing page on the same mechanism. Closing a pillar's gate closes its page, its sitemap entry, its hreflang, its Open Graph card and its megamenu link in one gesture.

Both directions, or it is not a source of truth

One direction sweeps the registry and serves every page it names.

it('serves every registry page in every locale', function () {
    $keys = SlugRegistry::keys();

    expect($keys)->not->toBeEmpty('The slug registry is empty.');

    foreach ($keys as $key) {
        foreach (MarketingLocale::codes() as $locale) {
            $path = SlugRegistry::path($key, $locale);

            expect($path)->not->toBeNull("Page [{$key}] has no slug in [{$locale}].");

            $this->get($path)->assertOk("Page [{$key}] does not respond in [{$locale}].");
        }
    }
});

That is 62 HTTP requests in one test. The emptiness witness matters more than it looks, because a gate can empty the registry, and an empty sweep otherwise reads as a complete one.

The other direction walks the router.

foreach (app('router')->getRoutes() as $route) {
    $name = $route->getName();

    if ($name === null || ! str_starts_with($name, 'marketing.')) {
        continue;
    }

    // marketing.fr.solutions.agencies becomes solutions.agencies
    $logical = preg_replace('/^marketing\.[a-z]{2}\./', '', $name) ?? '';

    expect($logical)->toBeIn($keys, "Route [{$name}] has no slug registry key.");
}

This is the one that catches the mistake people actually make, which is adding a route and forgetting the table. A page served without a registry key has no hreflang, no twin in the language switcher and no sitemap entry. It has no SEO surface at all, and it looks perfect in a browser.

The trap that cost us our datasets

SlugRegistry::keys() was a static array for months, and Pest datasets consumed it, one case per page, with per page reporting in the output.

Then a gated pillar received a landing page. The registry now calls PillarCatalog::isPublished(), which reads config('marketing.pillars.*'). A Pest dataset is resolved when the test case is constructed, before the application exists, container included. Wrapping it in a closure does not help, because the closure is resolved at that same moment.

The symptom has nothing to do with the cause. First expects 1 argument, but no dataset was provided, then Target class [config] does not exist, raised about 150 lines away from the change that caused it.

Keys are now iterated inside the test body, with the subject name in every failure message so the granular reporting is not lost. The corollary is repository wide: no dataset here can read configuration, and a registry that becomes gate dependent silently breaks every with() that consumes it.

What it costs

The suite grows with the site. One registry sweep is 62 requests. The routing and SEO files together are 35 tests, 995 assertions and 16.0 seconds. The whole public site feature suite is 850 tests and 153,711 assertions in 4 minutes 3 seconds. Adding a page adds work to roughly forty nine files, and that is the deal: the SEO guarantee is paid for in wall clock.
Datasets are unavailable, permanently. Loops with named failure messages recover the reporting, and they are more verbose than the with() they replaced.
One central file everyone edits. The registry is 537 lines, mostly comment, and every new page touches it. It is a reliable source of merge conflicts, and we consider a conflict there a feature, because it is a conversation about a URL.
It is rebuilt, not cached. map() costs 3.83 microseconds and alternates() 14.03, measured over 10,000 calls. A page render calls them a handful of times, so the total stays under a tenth of a millisecond, and the response cache in front of it hides even that. It is still a rebuild on every call, with nothing memoising it. We would rather pay microseconds than own a cache invalidation.
Item pages sit outside the guarantee. Everything derived from the registry is proven in both directions. Everything derived from childPath() is only as good as the overrides passed with it, which is exactly where the canonical defect lived, and exactly where the next one will.

What transfers

Not the registry. The registry is small, specific to a two locale Blade site, and would be the wrong shape for a site with regional variants or a translation workflow. It also only works because the public site is not rendered by the single page application that serves the product.

What transfers is the question that produced it. For any SEO fact a crawler can read from your site, ask how many places would have to be edited, in step, for that fact to stay true. If the answer is more than one, you do not have a source of truth. You have copies that agree today.

The follow up is just as cheap. Once there is one place, write the test that walks it in both directions, and give it a witness so an empty sweep cannot pass for a complete one. Ours has been red exactly when it should be, which is the only evidence that matters.

Measurements taken 26 August 2026. Registry size, key list and micro benchmarks from the application on PHP 8.4.23, Apple M5 Max, 10,000 iterations per benchmark. Sitemap composition parsed from the live production sitemap on the same day, 136 URLs, 268 xhtml:link elements, 38 lastmod elements. Test counts and durations from a local Pest 5 run, single process, SQLite in memory.

Originally published at nessflow.com.
I write about how NessFlow is built at nessflow.com/en/engineering.

Millions of log lines in PHP, at constant memory

NessFlow — Thu, 20 Aug 2026 16:09:01 +0000

Laravel 13, Horizon, PostgreSQL 18.

Our product needs to answer a question about a customer's site: which crawler fetched which URL, on which day, and what status code did it get. Not "how many hits yesterday". The cross product.

Why not a log analyzer

We started with GoAccess, which is an excellent tool, and abandoned it. The reason is worth stating precisely because it is the decision the rest of this article follows from.

GoAccess produces a report.json containing panels: top URLs, top user agents, status code distribution, hits per day. Each panel is already aggregated, and no panel is crossed with any other. So it can tell you that Googlebot fetched 40,000 pages, and separately that 3,000 requests returned 404. It cannot tell you whether Googlebot got any of those 404s.

An aggregate you cannot cross is not data. It is a picture of data.

What we needed was a cube: date, hour, bot, URL, status code, with hit counts and bytes. Once that is the requirement, no report-producing tool helps, because the aggregation has to happen on our axes. So the parser is ours.

What one line costs

Thirteen input formats are supported. Seven of them are variations on the common log format and share one engine: GoAccess style format strings compiled into a regex once, then applied per line.

// app/Services/Logs/Parsers/FormatStringLineParser.php
public static function combined(): self
{
    return new self('%h %^[%d:%t %z] "%r" %s %b "%R" "%u"', '%d/%b/%Y', '%T', null);
}

public static function amazonS3(): self
{
    return new self('%^ %^ [%d:%t %z] %h %^ %^ %^ %^ "%r" %s %^ %b %^ %^ %^ "%R" "%u"', ...);
}

public static function squidNative(): self
{
    return new self('%x %^ %h %^/%s %b %m %U %^ %^ %^', null, null, self::EPOCH);
}

%^ means "a field is here, skip it", which is what makes a 17 column S3 line expressible in one string. %z is ours, not GoAccess's: it captures the UTC offset so that every timestamp is normalized to UTC at parse time rather than at query time. Getting that wrong is how a daily report ends up with 25 hours in it twice a year.

The other formats do not fit a format string and get dedicated parsers: Cloudflare JSON, Caddy JSON, Google Cloud Storage CSV, and a W3C parser that is stateful because IIS declares its columns in a #Fields: header partway through the file. That same W3C parser serves CloudFront, whose lines are URL encoded and whose spaces arrive as +.

The measurement

Benchmarked on this machine on 19 August 2026, running the real three stages: parse the line, classify the user agent, normalize the URL. The corpus is synthetic but deliberately hostile to memoization, with 5,000 distinct paths and 240 distinct agent strings.

Lines	File	Time	Throughput	Peak memory
500,000	86.6 MB	5.91 s	84,550 /s	44.5 MB

Two honest caveats. Real production logs measured slower, around 52,000 lines per second, because real user agent strings are longer and messier than generated ones. And a synthetic corpus flatters any parser.

The number that matters is not the throughput. It is that peak memory was 44.5 MB in this run and 44.5 MB in the previous run with a fraction of the cardinality. It does not move, because nothing accumulates. A 2 GB file uses the same 44.5 MB as an 86 MB one, and the job's memory limit is therefore a constant you can reason about instead of a function of what a customer uploads.

The cube, and the upsert that makes partial work safe

Aggregates land in one table whose unique key is the cube itself.

$table->unique([
    'project_id', 'date', 'hour', 'bot_token', 'url_hash', 'status_code',
]);

$table->string('bot_token')->default('');   // sentinel, never null
$table->string('url_hash', 64);             // sha-256 of the normalized URL

bot_token defaults to an empty string and is never null, because a null inside a unique key means the key stops deduplicating: two rows with a null bot are distinct as far as the index is concerned. PostgreSQL has NULLS NOT DISTINCT to fix that. SQLite, which our tests run on, does not. So the sentinel is the portable answer, and every column in a uniqueness key is NOT NULL.

The write is an additive upsert.

insert into project_log_aggregates (…, hit_count, bytes_transferred, …)
values (…)
on conflict (project_id, date, hour, bot_token, url_hash, status_code) do update set
    hit_count         = project_log_aggregates.hit_count + excluded.hit_count,
    bytes_transferred = project_log_aggregates.bytes_transferred + excluded.bytes_transferred,
    updated_at        = excluded.updated_at

Adding rather than replacing is what makes the whole pipeline restartable. A flush that wrote half a buffer, whose keys then reappear in a later flush, is still correct. Two log files from two servers covering the same hour merge instead of overwriting each other. The excluded pseudo-table is portable across PostgreSQL and SQLite, which matters because the tests run on one and production on the other.

Two constraints come with that choice, and both are load bearing.

Intra-batch duplicates have to be impossible, not merely unlikely. PostgreSQL refuses an ON CONFLICT statement that contains the same conflict key twice in one batch. Our buffer is keyed by the cube, so a duplicate cannot exist by construction; deduplication is structural rather than a step someone could forget. The chunk size is 500 rows, which bounds the number of bindings per statement.

Replaying the same file would double the volumes. Addition has no idempotence of its own, so idempotence is enforced upstream, by a SHA-256 of the uploaded file computed server side while the chunks are assembled. The check is in application code rather than a SQL unique index, deliberately: a run that failed must remain re-uploadable.

The bug that ate its own error handler

This one cost us a production incident and it is the most transferable thing in this article.

Real access logs contain bytes that are not valid UTF-8, and sometimes null bytes, because scanners send binary payloads at your server and your server logs the request line. PostgreSQL rejects both with SQLSTATE 22021. Fine: the ingest catches the exception and writes it to the run's error_message column so the customer sees a failure instead of a spinner.

Except that a Laravel QueryException message contains the SQL with its bindings interpolated. So the message about the invalid byte contains the invalid byte. Writing it fails with the identical 22021. And because that write also happens in failed(), the retry fails the same way.

The run stayed in processing forever. The uploaded file had already been purged, correctly, so there was nothing to retry. The interface showed a job in progress that no longer existed.

private function sanitizeErrorMessage(string $message): string
{
    $message = str_replace("\0", '', $message);

    if (! mb_check_encoding($message, 'UTF-8')) {
        $message = mb_convert_encoding($message, 'UTF-8', 'UTF-8');
    }

    return Str::limit($message, 500);
}

The same scrubbing applies at the other end of the pipeline, in the URL normalizer, which also strips tracking parameters and caps length at 2,048 bytes using mb_strcut so a multibyte character is never sliced in half.

The part worth underlining: SQLite accepts every one of those bytes happily. No test we could have written on the test database would have found this. It is a whole class of defect that is green in CI and fatal in production, and the only defence is knowing it exists.

Queue discipline for a job that cannot be retried

public $tries = 1;
public $timeout = 1800;
// dedicated queue: logs-processing

One attempt, on purpose. A file that was half ingested must never be replayed, because the upsert adds. The dedicated queue exists so that a customer uploading a 2 GB archive cannot starve every other job in the system for half an hour, and it needs its own Horizon supervisor, which in turn means Horizon has to be restarted on deploy or the new code never runs.

The upload itself is chunked and hand written rather than a single multipart POST: 4 MB binary chunks, the SHA-256 accumulated server side as they are assembled, and a 422 on completion that returns the list of missing chunk indexes so the client re-pushes only those. Resumable uploads are not a feature we wanted to build, they are what a 2 GB file on a hotel connection requires.

Retention that keeps the answer and drops the resolution

A cube grows. Ours drops resolution before it drops data.

public const DAILY_AFTER_DAYS = 90;
public const PURGE_AFTER_DAYS = 400;

Past 90 days, the 24 hourly rows of a given day, bot, URL and status collapse into a single row at hour = 0, using the same additive upsert as the ingest, inside one transaction, and the hourly rows are then deleted. Past 400 days, rows go.

This is only safe because no analysis query filters on hour: the column exists for ingest fidelity, not for reporting. Checking that before writing the compaction is the entire difference between a retention job and a data loss incident.

No IP address is stored, and the nuance matters

The cube has no IP column. Nothing in the log services persists a client address, and nothing sends one to a model.

But the parser does read it, and pretending otherwise would be a lie: the common log format is positional, so %h has to be captured for the fields after it to line up. The address exists in memory for the lifetime of one line and is then discarded with it.

That distinction is the honest version of the claim, and it is also the useful one. "We do not process IP addresses" would be false. "No client address is written to disk or leaves the machine" is true, verifiable by looking at the schema, and it is what a customer handing over their server logs actually needs to know.

What we would do differently

Reverse DNS verification is still missing. We classify crawlers by user agent, which is trivially spoofable. Verifying that a self declared Googlebot actually comes from Google requires a reverse lookup followed by a forward confirmation, and we have not built it. Until then, our bot attribution is a declared identity, not a verified one, and the interface should say so.
The throughput number is machine dependent and we treat it as such. The figure in our own engineering journal was 52,000 lines per second. Re-measuring it for this article gave 84,550 on different hardware with a friendlier corpus. Both are true, neither is "the" number, and a written measurement is an observation with a date rather than a property of the system.
Streaming was not the hard part. Reading a file line by line in PHP is a while loop. The hard parts were the byte level hostility of real logs, the portability of one SQL statement across two engines, and deciding what a partially completed ingest is allowed to mean.

Benchmarked 19 August 2026 on a synthetic 500,000 line COMBINED corpus, 86.6 MB, 5,000 distinct paths and 240 distinct user agent strings, running FormatStringLineParser, LogBotRegistry and LogUrlNormalizer in sequence. Peak memory via memory_get_peak_usage. Production figure of 52,000 lines per second measured 21 July 2026 on real access logs.

Originally published at nessflow.com.
I write about how NessFlow is built at nessflow.com/en/engineering.

Blade, Inertia and Filament, on purpose

NessFlow — Thu, 20 Aug 2026 16:06:41 +0000

Laravel 13, Inertia 3, React 19, Filament 5.

Part one covered the public site: 26 Blade views, no CDN, no front-end framework, 4,323 bytes of JavaScript. It lives beside a product of 50 Inertia pages and an admin console we did not write.

Why each world

Each of the three is here because it is the best available answer to one specific problem. The clearest way to show that is one example each.

Blade: markup and structured data cannot disagree

Server rendered by default, no hydration, no serialization bridge. The strength shows up somewhere unglamorous. Our breadcrumb component emits the visible navigation and the JSON-LD from the same array, in the same render.

<nav aria-label="{{ __('marketing.nav.breadcrumb') }}">
    @foreach ($items as $item)
        <span aria-current="page">{{ $item['name'] }}</span>
    @endforeach
</nav>

<script type="application/ld+json">
    {!! json_encode($jsonLd, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE) !!}
</script>

One $items, two consumers: the human and the crawler. They cannot drift, because there is nothing between them to drift through. The same page built with client rendering has two code paths to the same truth, and a rich result that silently stops matching the page is a genuinely miserable bug to find.

Inertia: typed props, and no API to maintain

An audit tool is dense: sortable tables, charts, filter panels, virtualized lists. React has mature accessible answers for all of it, and Inertia removes the expensive part of a single page application, which is the API. A controller returns a readonly DTO and the page receives it.

// Controller
return Inertia::render('issues/index', [
    'crawl'  => CrawlSummary::from($crawl),
    'groups' => $groups,
]);

// app/Data/Seo/CrawlSummary.php, aligned with resources/js/types/seo.ts
readonly class CrawlSummary
{
    /** @param array{discovered: int, crawled: int, depth: int}  $stats */
    public function __construct(
        public int $id,
        public string $status,
        public int $progress,
        public array $stats,
        public bool $isTruncated,
    ) {}
}

No client store to keep in sync, no serializer layer, no second set of routes to version, no endpoint that has to be deprecated on its own schedule. Static analysis checks the PHP side, TypeScript checks the React side, and the shape is written once.

Filament: the least differentiated code in the product, for free

A back office is CRUD, and CRUD is the least differentiated code anyone writes. Two chained calls produce a searchable, sortable, paginated column.

TextColumn::make('slug')
    ->searchable()
    ->sortable(),

That is the whole argument. Its Pest integration means the panel is tested like the rest of the application rather than quietly exempted from testing, the ecosystem is unusually rich for a Laravel package, and we already know Livewire and Alpine for the rare resource that needs something specific.

One precision. Livewire is not a direct dependency. The composer.json requires filament/filament, there is no app/Livewire directory, and we have never written a Livewire component by hand. Livewire arrives with Filament, which is a different decision with different consequences.

What one origin buys

The three worlds are served by one application, from one origin, over one session, against one database. Both build entries import the same token sheet, so the product and the public site cannot drift apart.

// app.css and marketing.css, two entries, one source of truth
@import './theme.css';

Cross world state, in this arrangement, is a cookie and a stylesheet. On separate origins it is a specification. That is the whole argument, and everything below is the price of it.

Seam one: a preference that never reaches a request

Light and dark is stored client side. Two of our worlds used two different storage keys, which did not produce two independent settings: it produced one setting that was forgotten on every crossing. A visitor who switched to dark on a public page found the product in light, and the reverse.

Both worlds now read one contract, declared twice and mirrored explicitly.

// app/Support/Appearance.php
final class Appearance
{
    public const STORAGE_KEY = 'nessflow-appearance';
    public const COOKIE      = 'nessflow-appearance';
}

// resources/js/lib/appearance.ts
/** Mirror of App\Support\Appearance::STORAGE_KEY. */
export const APPEARANCE_STORAGE_KEY = 'nessflow-appearance';

Three details are worth more than the key itself.

There is a third copy of this predicate and it is irreducible. An inline Blade script has to resolve the theme before the first paint, which means before any bundle exists. Its parity with the TypeScript module is held by a test, so changing one without the other fails the suite.

"Auto" is the absence of a key, never the string system. Three surfaces resolve the theme before first render. A third stored value would have to be known by all three, and forgetting it in one place renders a light page on a dark machine: a theme flash nobody reproduces in development.

The cookie is not the source of truth. It is the server side mirror of local storage, and it exists for exactly one purpose: letting the Inertia shell put class="dark" on <html> before any script runs. On divergence, local storage wins, because it is the one that survives a cookie purge. Reading it also has to be wrapped, since in private browsing the read itself throws and would take the theme render down with it.

The test that had to be written against the machine

No feature test can see any of this, because nothing is wrong with either HTTP response. The guard is a browser test, and it has one property worth copying.

it('carries the theme chosen on the public site into the product', function () {
    $page = visit(SlugRegistry::path('home', 'fr-FR'))->inLightMode();

    $page->assertNoJavaScriptErrors()
        ->click('footer [data-appearance-choice="dark"]');

    $page->navigate('/login');

    expect(documentIsDark($page))->toBeTrue(
        'The product forgot the theme chosen on the public site.'
    );
});

The visitor picks dark on a machine set to light, and in the mirrored test light on a machine set to dark. Without running against the system preference, both worlds could simply follow the operating system on their own and the test would pass on precisely the broken state it exists to catch.

Seam two: two definitions of the current language

The server derived the locale from our cookie and fell back to Accept-Language. The front end read the same cookie and fell back to French. On the first request of a session, before the cookie exists, a browser configured in English produced English validation messages, an English <html lang> and English exports, inside an interface that was entirely French.

The fix was a deletion. The header branch could produce no benefit, because the front end never reads that header, so the interface stayed French regardless. It could only manufacture divergence.

// app/Http/Middleware/SetLocale.php
public const CODES    = ['fr', 'en'];
/** The fallback is FRENCH, the front-end runtime's own, and NOT
    config('app.locale'), which is 'en' and would reopen the divergence. */
public const FALLBACK = 'fr';

public static function resolve(mixed $declared): string
{
    return is_string($declared) && in_array($declared, self::CODES, true)
        ? $declared
        : self::FALLBACK;
}

The predicate is isolated in a static method so it can be argued with. Nothing else takes part: no header, no request body. Removing one branch corrected ten call sites at once instead of propagating a workaround into each of them.

A locale predicate is set on both sides of a boundary or on neither. Outside HTTP, in a queued job or a console command, there is no request and no cookie, so the language of a generated artifact travels as a parameter rather than being inferred.

Stated plainly, because it is a real trade: the product no longer detects browser language. It never did on screen. If detection becomes a requirement it belongs to the front end, which can write the cookie, the only place where the answer is true on both sides.

Seam three: leaving the single page application

Signing out is an XHR request. Fortify answered 302 to /, the client followed the redirect, received the complete HTML of the marketing home, found no X-Inertia header in it, and opened its <dialog id="inertia-error-dialog"> with the public site inside an iframe.

No error. No log. The user sits on /dashboard with the marketing site overlaid on top, address bar unchanged, already signed out.

// app/Http/Responses/LogoutResponse.php
public function toResponse($request): Response
{
    return $request->wantsJson()
        ? new JsonResponse('', 204)
        : Inertia::location(Fortify::redirects('logout', '/'));
}

Inertia::location() answers 409 with an X-Inertia-Location header, which the client translates into a full page load. Outside an Inertia request, the same call degrades to an ordinary redirect, so the contract of the route does not change for a bare form, an HTTP test or an API client.

Four exits cross worlds this way: signing out, deleting an account, leaving impersonation, and heading to checkout. The guard is unusual in that no content assertion can express it.

// Only the response CODE says it. Nothing in the body moves.
$response->assertStatus(409)
    ->assertHeader('X-Inertia-Location', '/');

$this->assertGuest();

Seam four: the panel is authenticated, so it sees the tenant scope

56 of our 77 models carry a trait whose global scope keys on Auth::user()?->current_team_id. That scope is the backbone of tenant isolation. The Filament panel is authenticated, so every Eloquent read of a business model inside it is silently restricted to the administrator's team rather than to the team on the row being looked at.

The symptom is treacherous because nothing raises. A withCount('projects') on a list of teams returns the administrator's project count on their own row, and zero on every other row. A back office showing "0 projects" across an entire fleet looks like an empty fleet, not like a bug. Sorting on that column then ranks teams in an order that means nothing.

We shipped that in two places: the Projects column of the teams list, and the ranking of a "Top Active Teams" widget.

// app/Filament/Support/CrossTenant.php
public static function unscoped(): Closure
{
    return fn (Builder $query): Builder => $query->withoutGlobalScope('team');
}

// Every panel read declares what it wants to read.
TextColumn::make('projects_count')
    ->counts(['projects' => CrossTenant::unscoped()])

Two decisions inside that small class matter more than the class.

The escape is per read, never global. Neutralizing the scope for the whole panel is tempting and wrong: the same human also browses the product, and a scope conditioned on the current URL is exactly the kind of rule that eventually applies at the wrong moment.

The relation constraint is not removed, only the ambient filter. projects.team_id = teams.id still holds, so the count remains the count of the right team, which is the team on the row.

The pattern

A stale storage key. A fallback chain disagreeing with another fallback chain. A redirect rendering in the wrong container. A protective scope doing its job in the wrong context.

None of them threw. All of them produced output a reviewer would approve. And no feature test could have caught a single one, because in every case the server answered correctly.

So the seams get a small, specific category of test that the middle of the application never needs: browser tests that genuinely cross from one world into another, response code assertions at the crossing points, and parity tests between surfaces that must agree. It is a handful of tests, and they are the only ones we have that can fail for reasons nothing else can express.

What it costs, and when we would not do this

Three mental models. Free for a team that knows all three. Not free for a team learning one of them.
Three styling contexts. The panel does not load the product stylesheet, and its theming goes through a palette rather than CSS, which is a different skill from writing Tailwind.
The seams are permanent. Cheap once understood, but they never disappear, and every new crossing point is a new chance at a silent failure.
It rewards one repository. Everything good here follows from one origin, one session and one database. Split those and most of the argument goes with them.

Same product, same team, we would make the same three choices tomorrow. It is not a compromise between frameworks. It is three tools doing what each is best at, sharing everything worth sharing.

Next. Part three: processing millions of server log lines with Laravel, Horizon and a hand written streaming parser.

Verified in the repository on 19 August 2026: 26 public Blade views, 50 Inertia page components, 56 of 77 models carrying the team scope, 4 cross world exits, 10 locale call sites corrected by removing one fallback branch.

Originally published at nessflow.com.
I write about how NessFlow is built at nessflow.com/en/engineering.

A modern marketing site without a modern front-end stack

NessFlow — Thu, 20 Aug 2026 16:04:05 +0000

Laravel 13, Blade, Tailwind 4, Vite 8.

Astro, Hugo, Eleventy and Next are all good at building fast marketing sites. This is not a comparison, and there is no version of this article where one of them loses. Every one of those tools carries real advantages and real costs, and choosing between them is a question of context.

This is our context, and these are our numbers.

4,323 bytes. Gzipped JavaScript on a public page, including navigation, the theme switcher and audience measurement. Four modules, hand written, no framework.

The entry number, and where it comes from

A PageSpeed run on 17 August 2026 at 19:51, mobile, returns 99 for performance and 100 for accessibility, best practices and SEO. Those four numbers come from a lab pass on a single URL. The field data section reads "no data", because the site is too young and too lightly trafficked to appear in the user experience report.

Lab scores are cheap. They are also the only thing most teams publish, which is why the rest of this article is about the decisions underneath them, including one that consists of refusing to trust one of those four numbers.

What we did not install

Four absences, all verifiable from outside in a few seconds.

No CDN. The apex domain and its www subdomain resolve straight to an OVH server. No intermediate proxy, no edge network, no distributed cache rules to reason about when a page looks stale.

No asset domain. There is no ASSET_URL. Stylesheets, scripts and fonts come from the same origin as the HTML, through the same web server, under the same cache policy and the same certificate.

No front-end framework for the site. The 26 public views are Blade views. The 28 public routes go through no single page application layer. There is no second repository, no second pipeline, no second deployment model, and no second place where content can drift.

No flat-file CMS and no headless CMS. This is the absence people find most surprising, so it gets its own section.

Two build entries, and why that is the load-bearing decision

One application holds a React product and a Blade public site. Nothing prevents the public homepage from shipping the application bundle. That is in fact what happens by default the moment a shared entry point looks convenient.

So the Vite config declares two separate entries, and the comment next to them states the reason rather than the rule.

// A marketing page must ship neither React, nor Inertia,
// nor the product bundle. Putting them in app.css/app.tsx
// would make the public homepage pay for the whole app.
input: [
    'resources/css/app.css',
    'resources/js/app.tsx',
    'resources/css/marketing.css',
    'resources/js/marketing.ts',
],

Walking the production manifest, following the imports of the public entry and compressing each file, gives the payload of a page.

Resource	Files	Raw	Gzipped
JavaScript	4	10,833 B	4,323 B
CSS	1	73,931 B	13,110 B
Total	5	84,764 B	17,433 B

A complete public page, navigation and interactions included, is 17.4 KB of compressed script and style. The JavaScript is compiled TypeScript with no framework, written for what those pages actually do.

The separation is structural, not a convention someone has to remember. That distinction is the whole point: a rule written in a document decays, a rule written in the build does not.

Content lives in Postgres, not in Markdown

The usual path for a static site is Markdown files in the repository, rebuilt and redeployed on every edit, or a headless CMS: another service, another bill, another API to keep in sync, another authentication story.

Our content hub is a set of Eloquent models in PostgreSQL, administered through a Filament panel. A non developer edits an article and it is live. No rebuild, no deploy, no content API, no webhook to invalidate anything. Laravel already has the model layer, the admin panel, the validation, the authorization and the localization; using them for content costs close to nothing.

Putting content in the database rather than in files also has a consequence that took us a while to appreciate, and it is the most strategically interesting part of this architecture.

The public site can read the session

The marketing site runs inside the same Laravel application as the product. Same database, same session, same authentication, same authorization. Which means a public Blade page can know, server side, before a single byte of HTML is rendered: whether this visitor is signed in, which team they belong to, which plan they are on, whether they have ever run an audit, whether they abandoned onboarding halfway through.

We do not do this yet. Our public views currently read no session state at all. But the capability is sitting there at zero additional infrastructure cost, and it is worth naming what category it belongs to.

Personalizing a marketing site against product state is normally the territory of Adobe Experience Manager, Optimizely, and the personalization tier of a headless CMS. Those products largely exist to reconnect a static marketing site to state that lives somewhere else.

When the site and the product are the same application, that reconnection is a function call. A static build plus a headless CMS cannot do it without standing up an API, shipping a client side fetch, and accepting a flash of the generic version before the personalized one arrives. We get the server rendered, no flicker, no extra request version for free, and we have not spent it yet.

One design system, two entries, no drift

The two Vite entries are separate, but they are not independent. Both import the same token sheet.

// app.css
@import './theme.css';

// marketing.css: tokens come from theme.css, shared with app.css
@import './theme.css';

Change a color, a radius or a spacing step, and the product and the public site both move, in the same build, in the same commit, verified by the same tests. There is no design system package to version and publish, no synchronization step between two repositories, no pair of Tailwind configs slowly drifting apart, and no marketing site that quietly starts looking like last year's product.

This is not a claim that a separate front-end stack does this badly. It is a claim that it cannot do it this way. Two codebases means a publish step between them, and a publish step means a version, a changelog and a lag. One codebase means the lag is zero because there is nowhere for it to accumulate.

Tailwind with tokens

The public stylesheet is 73,931 bytes raw and 13,110 compressed, covering all 26 views. That is simply what Tailwind gives when you do not work against it.

In practice that means design tokens instead of arbitrary values. The palette lives in CSS variables, declared once, and the utilities consume them. A color hard coded into a view is a color that escapes the dark theme, escapes contrast auditing and escapes every future adjustment. Once the same surface has to exist in light and dark, the difference between the two approaches stops being aesthetic and becomes structural.

It is also what makes the next section possible.

The score that lied

Our first real Lighthouse campaign returned 100 for accessibility. It was wrong.

The token used for every piece of secondary text on the site produced a contrast ratio of 4.47 to 1 against the page canvas. The AA minimum is 4.5. We were three hundredths short, across every subtitle and every piece of metadata, on pages that discuss compliance, inside a product whose accessibility module grades our customers' sites.

Three hundred and twenty feature tests could not see it, for a reason worth stating as it is.

A CSS file has no type checking. The TypeScript compiler does not read it, the static analyser does not read it, the formatter does not read it. Its tests are its only verification.

The fix was not the color. Fixing a color does nothing about the next one. The fix was a test that reads the shipped stylesheet from disk, follows variable aliases, composites semi transparent colors over their real background, then redoes the WCAG math.

$muted  = themeToken('--sp-ink-500');
$canvas = themeToken('--sp-canvas');

expect(contrastRatio($muted, $canvas))->toBeGreaterThanOrEqual(4.5)
    ->and(contrastRatio($muted, '#ffffff'))->toBeGreaterThanOrEqual(4.5);

Two properties of that test matter more than the test itself.

It reads the shipped baseline, never an injected value. A test that hands itself the color it is about to measure only tests arithmetic. This one opens the file the browser will receive.

It composites before measuring. A semi transparent color has no contrast of its own; it only has one once it sits on a background. Measuring the nominal value would certify white at 20 percent opacity. That is the class of mistake that produces a green test and an unreadable page.

The current ratio is 4.753. The threshold is a pinned floor rather than a target, so if someone lightens the canvas in six months, the test fails before production does.

The same math settled a design argument

On our deep blue surface, the green check mark that marks features included in a plan measures 2.921 to 1. On the dark theme accent it measures 4.29. Below the threshold on both.

So the check mark is not green. It takes the text color, and a test pins that refusal, so nobody restores the green believing they are improving the screen. A brand decision settled by a calculation instead of an opinion.

What CI asserts, and what it refuses to assert

The Lighthouse CI budget for our public pages is three lines.

"categories:accessibility": ["error", { "minScore": 1 }],
"categories:seo":           ["error", { "minScore": 1 }],
"categories:performance":   "off"

Accessibility and SEO are blocking assertions, required at 1.00. Performance is explicitly off.

The budget runs a single pass, and performance is non deterministic: it depends on the load of the CI machine, on the network, on luck. A threshold on it produces red builds with no cause and green builds with no merit, and within three weeks the team learns to ignore the color. A guard that gets ignored is worse than no guard, because it leaves people believing they are covered.

Accessibility and SEO are deterministic. The same files yield the same verdict, so they can be demanded at perfection, and they are.

The number that reframes the whole exercise

Here is a performance trace of the site taken with a 20 times CPU slowdown and Slow 4G throttling, which is a deliberately brutal profile. The interesting column is main thread time by party.

Party	Main thread
Wappalyzer (browser extension)	690.2 ms
AdBlock (browser extension)	629.9 ms
Fake Filler (browser extension)	364.3 ms
React DevTools (browser extension)	213.4 ms
The site itself, first party	206.7 ms
Cookieless analytics, our only third party	23.5 ms

Every single browser extension in that profile costs more main thread time than the entire website. Wappalyzer alone costs 3.3 times more. Our one third party script, cookieless analytics, costs 23.5 milliseconds.

We find this genuinely useful rather than flattering. Once first party work is down in the low hundreds of milliseconds, the dominant term in a real visitor's experience is no longer your framework choice. It is their extensions, their device and their network. The engineering value of going from 4 KB to 2 KB is close to zero. The engineering value of the accessibility guard, which fixes something a visitor actually experiences, is real.

The trade-offs we accepted

Page transitions are full loads. No instant navigation, no hover prefetching unless we write it. Correct for a content site, wrong for an application, which is exactly why our product does not run this way.
Every interactive behaviour is hand written. Menu, theme switcher, scroll reveals: TypeScript nobody maintains for us. A 4 KB budget is a constraint as much as a result.
Without a CDN, geography is real. A visitor far from the origin pays the distance. Our audience is concentrated, so we take that trade, and it is a trade rather than a superiority.
The two worlds have to stay watertight, and that boundary has a cost we paid in production. It is the subject of the next article.

How we decide

Three questions, asked of every piece we consider adding.

Does this solve a problem we have, or a problem we might have?
How many surfaces will we maintain because of it, and who maintains them?
If it disappears during a version upgrade, is that a degradation or an outage?

The answers gave us a public site in the same application as the product, with no intermediate layer and 4,323 bytes of JavaScript. Different answers give different architectures, and that is fine. What transfers is not the stack. It is measuring before deciding, then writing a guard so the decision outlives the people who made it.

Next. Three rendering worlds share this application: Blade for the public site, Inertia and React for the product, Filament for the admin console. We did not have to choose, and that was the right call. But the boundaries between those worlds cost something, and we shipped a defect to production that no feature test could ever have caught. That is the next article.

Measurements: production manifest, 18 August 2026, gzip level 6 applied per file. PageSpeed mobile, 17 August 2026 19:51, lab, single URL, no field data. Chrome performance trace, 20x CPU throttling, Slow 4G. Contrast ratios computed with the WCAG 2.2 formula.

Originally published at nessflow.com.
I write about how NessFlow is built at nessflow.com/en/engineering.