DEV Community: Gabriel Richards

Is Your Hydrogen Product Page Readable by an AI Agent?

Gabriel Richards — Thu, 29 Oct 2026 17:11:25 +0000

Originally published on the Endertech blog by Gabriel Richards, Founder & CEO. Republished here with a canonical link to the original.

AI agents and crawlers read the first server response, not the finished page. Here are the three ways product pages hide content from them, how to check your own, and what we found on a production Hydrogen store.

TLDR

Judge a product page by its first server response. Many agents don't run JavaScript or wait for streamed content.
Pages hide content three ways: browser-fetched blocks, streamed placeholders and bot-detection gates.
On the Hydrogen store we checked, price, specs and Product JSON-LD were in the initial HTML, but review text and delivery terms weren't for every reader.
Render anything that affects a purchase on the server, for every user agent.

A product page can look complete in a browser and still be half empty to the software reading it. AI shopping agents, search crawlers and link previews often read the first HTML response, and many don't run JavaScript or wait for content that arrives later. The general lesson: judge a product page by what's in the first server response, not by what a person sees after everything has loaded.

Modern storefronts make this easy to get wrong because they're good at loading things later. Headless stacks like Shopify Hydrogen stream the page, defer slow data behind placeholders and fetch some blocks in the browser. All of that helps shoppers. Some of it hides content from machines. This post covers the three ways that happens, how to check your own pages in an afternoon, and what we found when we ran the check on a production Hydrogen store in October 2026.

Crawl signals like robots.txt, sitemaps and canonical tags are covered in Headless Storefront SEO: Own the Crawl Signals Your Theme Used to Send. How structured data fits in is covered in What Structured Data Actually Does for SEO and AI Search. This post is about the product page itself.

Why the page still matters when Shopify sends a feed

As of October 2026, Shopify sends eligible product data to AI channels through Shopify Catalog, so you might assume the page doesn't matter. Two things say otherwise.

First, Shopify's own agentic storefronts documentation says AI channels can also reach products through the open web, by crawling and indexing. Catalog isn't the only reader.

Second, the shoppers those agents send land on product pages. Shopify's GEO Playbook reports that in Q1 2026, more than half of AI-referred sessions started on a product page, compared with about 20% for organic search. The page is the first thing an agent quotes and the first thing a referred shopper sees.

Three ways a product page hides content from machines

Content fetched in the browser. Reviews, delivery estimates and recommendation widgets often load from a third-party service after the page renders. A reader that doesn't run JavaScript never sees them. In a searchVIU test from October 2025, ChatGPT and Claude didn't pick up a price inserted by JavaScript when fetching the page live, while ChatGPT did read prices in the visible HTML.
Content streamed behind a placeholder. Streaming frameworks send a "Loading..." fallback first and the real content later in the same response. A browser swaps it in. A simple parser may stop at the placeholder.
Content gated by bot detection. Some storefronts wait for all deferred data only when the user agent looks like a known bot. Known crawlers get the full page. A new AI agent the detection library doesn't recognize gets the placeholder version.

My opinion: anything that affects a buying decision shouldn't depend on bot detection or JavaScript. Price, availability, specifications, delivery promises, warranty terms and some review text belong in the initial render for everyone. Only truly slow, nonessential blocks should be deferred.

How to check your own product pages

Fetch the page without JavaScript. Request a product URL with a plain HTTP client, once with a browser user agent, once with a known crawler user agent and once with a made-up agent string, and save all three responses.
Search the raw HTML for the facts that sell the product. Price, availability, dimensions, materials, key specifications, review rating and count, review text, delivery and warranty terms.
Trace anything missing. For each missing fact, find whether it's deferred on the server, fetched in the browser or never rendered.

What we found on one Hydrogen store

We ran that check on a Hydrogen storefront we build and maintain for a furniture retailer. The pattern is common, so it's a useful example.

Present for everyone: title, price, description and specifications rendered as plain HTML from metafields, plus server-generated Product JSON-LD with one offer per variant and a rating only when real reviews exist.
Dependent on who was asking: delivery, warranty and brand blocks were streamed behind a placeholder, and the server waited for them only for recognized bots.
Missing for everyone: review text, which loaded in the browser from a reviews service. The rating and count were in the HTML. The reviews themselves weren't.

The fixes follow directly: render purchase-critical blocks for every user agent, and render at least a few recent reviews on the server if your reviews provider offers a server-side API. Shopify's Agentic admin lists reviews among its listing quality signals, and reviews are often the most persuasive content on the page.

A checklist for any product page

Compare raw HTML for a browser, a known crawler and an unknown user agent.
List every deferred or browser-fetched block and decide whether it affects a purchase. If it does, render it on the server.
Keep specifications in the HTML, even when they're inside a collapsed section.
Render some review text on the server, not only the rating.
Generate Product JSON-LD from the same data the page renders, with one offer per variant and ratings only when they exist.
Check headings. One H1, sensible levels and no hidden headings that also hide from assistive technology.
Test after every release. Deferred loading is easy to add in a refactor and hard to notice.

If you're planning a new build or a rebuild, our Shopify Hydrogen work starts from these checks.

How we know

We reviewed the product route of a production Hydrogen storefront, including its server entry, product loader and structured data module, on October 22, 2026. Shopify claims come from Shopify's Help Center and GEO Playbook, and the JavaScript fetch behavior from searchVIU's published test, all accessed the same day.

Limits

This is one storefront's route, and your stack may defer different blocks or detect bots differently. AI agents don't publish whether they run JavaScript or how long they wait, and the searchVIU result comes from a single test page, so we treat the first server response as the safe baseline rather than predicting each agent's behavior.

Last reviewed: October 22, 2026.

Sources

The GEO Playbook: How (and Why) to Optimize for AI Discovery, Shopify (accessed October 22, 2026).
Shopify Catalog and product discovery for agentic storefronts, Shopify Help Center (accessed October 22, 2026).
Data fetching in Hydrogen, Shopify developer documentation (accessed October 22, 2026).
Schema Markup and AI in 2025: What ChatGPT, Claude, Perplexity and Gemini Really See, searchVIU (accessed October 22, 2026).

Make your product pages readable to every reader

This is part of our series on Shopify's agentic storefronts. If you want a readability audit of your product pages, talk with Endertech.

Isolate Shared State Before Adding Parallel Test Workers

Gabriel Richards — Wed, 28 Oct 2026 23:37:59 +0000

Parallel test runs break on shared state, not CPU count. Here is how we isolated databases, temp files, locks, and reports in a Symfony coverage run, locked only the truly shared fixtures, and merged coverage without hiding failures.

TLDR
Parallel test failures usually come from shared state, not too few CPUs.
Give each process its own database suffix, and each queued test its own TMPDIR, lock store, JUnit file, and coverage file.
Lock only the fixtures that must stay shared, and keep those locks outside the per-test TMPDIR.
Warm one-time installs first, build the queue from phpunit --list-tests, and merge coverage with phpcov after the run.
Let the test exit code win, and treat any test that passes only on rerun as a bug report.
When a mature Symfony test suite gets slow, adding workers looks like the obvious fix. It usually isn't the first fix. Parallel test runs break on shared state, not on CPU count: two processes that both think they own the same database, temp file, lock, or report will produce failures that have nothing to do with the code under test.

The rule we follow is simple. Give every worker, and ideally every queued test, its own database, temporary directory, lock store, report file, and coverage file. Serialize the few resources that truly must stay shared with a narrow, explicit lock. Warm one-time installs before you fan out. Then merge the outputs and keep the test exit code in charge.

In October 2026 we applied this to the coverage run of a retail ecommerce integration middleware: a Symfony and Doctrine application whose suite exercises CSV imports, queues, and order and inventory flows. The client isn't named here, and the code below is trimmed from the real script and config. The lesson carries over to most PHP suites.

Start with a shared-state inventory
Before touching the test command, list every resource the suite creates, reads, overwrites, or reports into. For each one, decide whether it can be unique per process or has to stay shared.

Resource What we did Why it matters
Doctrine test database Appended the runner's process (channel) number to the test database name Fixtures and test data can't cross process boundaries
Temporary files Set TMPDIR per queued test (coverage/tmp/{n}) Generated and cached files don't collide
Symfony Lock store Pointed LOCK_DSN at a flock directory per queued test The lock backend stops being accidental shared infrastructure
JUnit reports One XML file per queued test No process overwrites another's results
Coverage data One .cov file per queued test, merged at the end Each process records coverage independently
CSV import fixtures in a shared directory An exclusive cross-process flock per fixture file Tests that must modify and restore the same file take turns
An inventory like this is worth doing even if you never parallelize. Flaky tests usually point at unclear ownership of files and data. Finding them early is a normal part of our custom software development work.

Give every process its own environment
The runner here is liuggio/fastest, which reads a queue of tests from standard input and runs them across N processes. Its command template has two placeholders: {p} is the process (channel) number and {n} is a unique number for each queued test.

For the database, the Doctrine test config appends both the ParaTest convention and the fastest channel to the database name:

when@test:
doctrine:
dbal:
# "TEST_TOKEN" is typically set by ParaTest
# "ENV_TEST_CHANNEL" is typically set by liuggio/fastest ({p})
dbname_suffix: '_test%env(default::TEST_TOKEN)%%env(default::ENV_TEST_CHANNEL)%'
For files, locks, and reports, the script keys everything on {n}, so each queued test gets its own directory and output files:

FASTEST_ISOLATION_ENV="TMPDIR=\"$COVERAGE_BASE_DIR/coverage/tmp/{n}\" LOCK_DSN=\"flock://$COVERAGE_BASE_DIR/coverage/tmp/{n}/locks\"";
FASTEST_COMMAND="$COVERAGE_COMMAND_PREFIX ./bin/phpunit -c \"$PHPUNIT_CONFIG\" {} --colors=never --coverage-php $COVERAGE_BASE_DIR/coverage/php/{n}.cov --log-junit $COVERAGE_BASE_DIR/coverage/junit/{n}.xml";
FASTEST_EXECUTE="mkdir -p \"$COVERAGE_BASE_DIR/coverage/tmp/{n}/locks\" && $FASTEST_ISOLATION_ENV $FASTEST_COMMAND";
Lock only what has to stay shared
Some state can't be split without changing what a test verifies. Import tests write CSV fixtures into the application's var/import directory. Two processes writing the same fixture file at once would corrupt each other's input.

The fix was a narrow lock per fixture file, kept outside TMPDIR so every process can see it:

// Shared across workers (do not use TMPDIR — coverage runner sets a per-method TMPDIR).
$lockDir = '/tmp/phpunit-import-fixture-locks';
if (!is_dir($lockDir)) {
mkdir($lockDir, 0777, true);
}
$lockPath = $lockDir.'/'.preg_replace('/[^A-Za-z0-9.-]/', '', $filename).'.lock';
$handle = fopen($lockPath, 'c+');
if (!flock($handle, LOCK_EX)) {
fclose($handle);
throw new \RuntimeException(sprintf('Unable to acquire import fixture lock "%s"', $lockPath));
}
$this->importCommandFixtureLocks[$filename] = $handle;
Isolate by default, then lock the smallest shared thing that can't be isolated, and name the lock after it.

Warm one-time setup before you fan out

Warm the Symfony PHPUnit bridge install once so parallel workers do not race on first boot.

$PHP_BINARY ./bin/phpunit -c "$PHPUNIT_CONFIG" --version >/dev/null;
Let PHPUnit build the queue
Build the queue from bin/phpunit --list-tests, not find tests/ -name "*Test.php". That skips abstract base classes. Process count comes from PARALLEL_PROCESSES, then nproc / sysctl, then a fallback of 4.

Treat reruns as bug reports
--rerun-failed can keep one unlucky collision from failing a pipeline — and hide a race inside a green build. A test that passes only on rerun is a bug report, not a pass.

Merge coverage without letting it mask failures

./vendor/bin/fastest -vvv --ansi --process "$PROCESS_COUNT" --preserve-order --rerun-failed --no-errors-summary "$FASTEST_EXECUTE" < "$QUEUE_FILE";
FASTEST_EXIT=$?;
$PHP_BINARY "$PHPCOV_PHAR" merge \
--html "$COVERAGE_BASE_DIR/coverage/html" \
--cobertura "$COVERAGE_BASE_DIR/coverage/cobertura.xml" \
--text "$COVERAGE_BASE_DIR/coverage/coverage.txt" \
"$COVERAGE_BASE_DIR/coverage/php";
PHPCOV_EXIT=$?;
if [ "$FASTEST_EXIT" -ne 0 ]; then
exit "$FASTEST_EXIT";
fi
exit "$PHPCOV_EXIT";
Coverage still merges after a failed run, but the test result wins.

A practical sequence
Inventory every database, directory, lock backend, fixture, report, and coverage file the suite touches.
Make each one per process or per test.
Put narrow cross-process locks on what must stay shared.
Run one-time installs before starting workers.
Build the queue from phpunit --list-tests.
Merge coverage after the run, and exit with the test runner's status first.
Log every rerun and fix the test behind it.
If CI is slow and parallel runs feel risky, start with the shared-state inventory. For more on how we approach this kind of engineering work, see our custom software practice and engineering strategy pages — or read the full original with sources.

Written by Willem Luijt at Endertech. Originally published at https://endertech.com/blog/isolate-shared-state-before-adding-parallel-test-workers.