DEV Community: Jonas

Don't Count Failed AI Visibility Checks as Brand Misses

Jonas — Wed, 21 Oct 2026 11:59:03 +0000

Cover: AI-generated editorial illustration.

An AI-answer tracker schedules eight checks. Five finish, two fail, and one is still pending. Two of the completed answers mention the configured brand. What percentage should the dashboard display?

This is a synthetic example. It exposes a reporting bug that is easy to introduce: dividing mentions by every row, including rows that never produced a usable answer.

Two out of eight is 25%. But that number treats three missing results as if they were completed answers that omitted the brand. Two out of five is 40% of the completed, classified answers. That still needs a coverage label: five of eight checks completed.

Separate collection state from answer classification

A request failing tells you something about the collection process. It does not tell you whether the brand would have appeared. Keep that state separate from the result of inspecting a successful answer.

There is another unknown to preserve. A request may complete while classification is unavailable or inconclusive. That answer should not silently become a negative result either.

For this example, completed means an answer was collected successfully. The mentioned field is true, false, or unclassified. Mentioned means only that the configured brand was identified in that answer; it does not mean recommended or cited.

Aggregate only the records that support the metric

The function below expects validated input with one final record per logical check. Resolve retries and duplicate records before calling it. Its recorded count is the number of rows supplied, not an independently verified schedule size.

export function summarize(rows) {
  const completed = rows.filter(r => r.status === 'completed');
  const classified = completed.filter(r =>
    typeof r.mentioned === 'boolean'
  );
  const mentions = classified.filter(r => r.mentioned).length;
  return {
    recorded: rows.length,
    completed: completed.length,
    classified: classified.length,
    mentions,
    misses: classified.length - mentions,
    unclassified: completed.length - classified.length,
    failed: rows.filter(r => r.status === 'failed').length,
    pending: rows.filter(r => r.status === 'pending').length,
    mentionRate: classified.length
      ? mentions / classified.length
      : null
  };
}

The boolean guard is intentional. A missing field, null, or the string false cannot supply a trustworthy negative classification. With no classified answers, the rate is unavailable rather than zero.

Show the denominator beside the percentage

For the eight-record example, the summary is: 8 recorded, 5 completed, 5 classified, 2 mentions, 3 misses, 2 failed, 1 pending, and a mention rate of 0.4.

A useful label is “2 of 5 classified answers mentioned the brand.” Put “5 of 8 checks completed” beside it. If some completed answers are unclassified, show that count too.

For a separately configured schedule, compare the collected records with the expected logical check IDs. A record missing from the database is not automatically a pending check, and this function does not discover missing scheduled jobs.

Test the cases that can make a chart lie

The included function was run against the eight-record example and three boundary cases: an empty collection returns an unavailable rate; a failed row carrying mentioned: true does not increase mentions; and a completed row carrying mentioned: null remains unclassified with an unavailable rate.

Also verify retry handling in the system that produces these records. Counting three attempts at the same question as three separate scheduled observations changes the denominator before the reporting function ever runs.

Keep interpretation separate from arithmetic

A correct percentage does not describe every person's AI experience. It describes the answers in the selected collection. Keep the question set, provider, dates and known request conditions available so a reader can judge what was measured.

If a later run has lower completion or a different question set, display that context before presenting the change as a visibility trend. Fixing an aggregation bug improves the report; it does not change the underlying answers.

AI disclosure: This article was drafted with an AI assistant. The examples use synthetic data and were checked by running the included JavaScript. They are not production measurements or customer results.

JavaScript Links and AI Crawlers: Make Navigation Discoverable

Jonas — Wed, 07 Oct 2026 10:28:23 +0000

Cover: AI-generated editorial illustration.

A pricing card can work perfectly when you click it and still expose no link to the pricing page.

That is the navigation problem I would check before changing how the whole app renders. The useful question is simple: where is the destination represented?

My earlier raw HTML versus rendered HTML article covered the different documents a crawler might receive. This article stays with one part of that problem: whether important pages are connected by usable links.

Inspect the HTML behind the component

A component called ServiceCard might produce this:

<button onClick={() => navigate('/services/maintenance')}>
  Website maintenance
</button>

The button can run JavaScript and change the route. The markup does not expose that destination as a link.

For navigation, the relevant output is:

<a href="/services/maintenance">Website maintenance</a>

You can use your router's link component to preserve client-side navigation. Inspect the HTML it actually produces. The component name alone doesn't prove that an anchor and a usable href are present.

Keep buttons for actions such as opening a dialog or applying a filter. Use a link when the intent is to take someone to another page.

Google's crawlable-link guidance specifically recommends anchors with resolvable href values. It also explains that JavaScript-inserted links can be crawlable when they produce that markup. This is documented Google behavior, not a guarantee about every AI crawler.

Check the destination before testing the click

An anchor is only useful if its href identifies the intended destination.

Look for empty values, a bare #, javascript: URLs or handlers that ignore the href and send the visitor somewhere else. A card can look like navigation while its only useful routing information lives inside an event handler.

For the maintenance example, I would expect to find /services/maintenance in the anchor. Then I would copy that address and open it directly in a fresh tab.

If the route works after entering through the homepage but fails when opened directly, investigate the production host and route handling. A successful in-app transition has hidden a separate delivery problem.

Check the returned page as well. A 200 response containing the homepage is not evidence that the maintenance route works. The destination needs its own relevant heading and content.

Find out when the link exists

There are three useful checkpoints for the page containing the link:

The initial HTML response.
The rendered DOM before any interaction.
The DOM after opening a menu or clicking a control.

Search for the actual anchor and its href at each stage. A destination appearing inside a script string or JSON object is not the same as a navigation link.

If the anchor is already in the initial response, you have established that it is available without running the app. If rendering adds it, its availability depends on the reader's rendering path. If it only appears after a click, discovery depends on that additional interaction too.

This is a useful distinction for menus. A collapsed menu may already contain anchors in the document. Another menu may fetch and create them only when opened. They can look identical in the interface while presenting different navigation paths to an automated reader.

Do not treat your browser test as proof of a particular AI crawler's behavior. It establishes what your page delivered in that test. Check the relevant service's documentation and available crawl evidence before making a crawler-specific claim.

Audit a path through the site

Checking the homepage alone leaves too much untested. Follow a real path a visitor needs, such as product page to integration guide to setup instructions.

For each step, record the source page, the anchor's href and whether the destination loads directly. Also record whether the anchor appears before rendering, after rendering or only after interaction.

That gives you a useful bug report: "The product card reaches the integration guide in-app, but it is a button with no anchor. The destination itself loads correctly." The repair can stay in the card component.

A different finding needs a different fix: "The guide has a real link, but a direct request to the setup route returns the homepage." Rewriting the anchor text won't repair that route.

The link graph matters too. A detailed guide can exist without a useful link from the page that explains the related product. Creating another guide would leave that missing connection in place. Add the relevant path and verify it.

Verify the shared component in more than one place

After a repair, check the header, cards and related-article navigation separately. They may look similar while using different components. Read the link text without the design around it. It should help someone understand where they are going.

Vite itself does not decide whether your navigation becomes anchors, buttons or a rendered server document. Diagnose the deployed output and the route behavior.

I would call this audit complete when the important paths expose real destinations, those routes load independently, and each destination contains the intended page. A rendering layer may help with a measured delivery gap, but it cannot infer a missing navigation relationship that the application never creates.

This article was adapted from my earlier technical writing with AI assistance.

Raw HTML vs Rendered HTML: What AI Crawlers Actually See

Jonas — Tue, 15 Sep 2026 18:48:51 +0000

Cover: AI-generated editorial illustration.

A web page can look complete in your browser and still send almost no useful content in its initial response.

That sounds contradictory until you separate two things developers often treat as interchangeable: the HTML returned by the server and the document produced after a browser executes the page's JavaScript.

For a person using a modern browser, the distinction is easy to miss. The browser runs the application, fetches data, updates the DOM, and presents the finished page.

A crawler does not necessarily follow the same path. Some crawlers render JavaScript. Some render it later. Some use limited execution environments. Some primarily inspect the response they receive. A crawler may also stop when a script fails, a request is blocked, or rendering exceeds its resource budget.

The useful question is therefore not just, “Does this page load?” It is, “Which version of this page is available to the system trying to read it?”

Three versions of the same page

It helps to distinguish three representations.

Initial or raw HTML is the response body returned by the server before client-side JavaScript runs. You can inspect it with an HTTP client or your browser's “View Source” feature.

Browser-rendered DOM is the document after the browser has parsed the response and executed the relevant JavaScript. This is what you normally inspect in the Elements panel of developer tools.

Crawler response is what the server returns for a particular crawler request. It may match the initial HTML sent to a normal browser, but it does not have to. A site may route known crawlers to server-rendered or prerendered HTML while sending its client application to normal visitors.

Consider a simplified client-rendered application:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Acme Analytics</title>
    <script type="module" src="/assets/app.js"></script>
  </head>
  <body>
    <div id="root"></div>
  </body>
</html>

After JavaScript runs, the DOM might contain a product heading, several paragraphs, pricing links, documentation navigation, structured data, and customer questions. None of that information is present in the response above.

The browser has enough information to build the page. A system reading only the response does not.

CSR and hydration are related, but not identical

Client-side rendering (CSR) usually means that JavaScript creates most of the meaningful document in the browser. The server sends a shell and the client fetches or computes the content.

Hydration starts from HTML that the server has already rendered. JavaScript attaches behavior to that existing markup. A hydrated page can therefore expose substantial content before JavaScript runs—provided the server response really contains that content.

Framework choice alone does not tell you which result a crawler receives. React, Vue, Svelte, Angular, and similar tools can participate in client rendering, server rendering, static generation, or combinations of them. What matters is the deployed response for a specific URL.

This is why statements such as “React sites are invisible to crawlers” are not useful. Some React sites return complete documents. Others return a thin shell. Many fall somewhere in between.

What can disappear from the raw response?

When the initial HTML is thin, the missing material is not limited to body copy. Common gaps include:

The primary heading and descriptive text
Product or article content loaded through an API
Internal navigation and contextual links
Canonical, description, or social metadata inserted by client code
Structured data generated after mount
Pagination or related-content links
Text inside tabs, accordions, or client-side routes
Error and empty states that replace expected content when a request fails

Each gap has a different consequence. Missing body text reduces what a non-rendering reader can understand. Missing internal links affects discovery. Missing structured data removes an explicit machine-readable description, even when visible copy remains available.

A character-count difference alone cannot tell you whether a page has a serious problem. A cookie banner can add thousands of unimportant characters, while a smaller difference can contain the only product description or links to deeper pages. The comparison needs both numbers and inspection.

Search crawlers and AI crawlers are not one audience

It is tempting to divide the world into “Googlebot” and “AI bots,” but both groups contain systems with different purposes and capabilities.

A search engine may fetch a document, schedule rendering separately, revisit it with another service, and combine the result with other signals. An AI company may operate different crawlers for search indexing, model training, user-requested retrieval, and link previews. An assistant answering a live question may also use a retrieval system that behaves differently from the company's general web crawler.

Do not assume every crawler executes the same JavaScript—or that one successful render proves availability to all the others. Make the important public meaning available in useful HTML, then verify crawler-specific behavior with evidence.

This matters beyond traditional search because AI visibility has at least two separate layers:

Can a system access and understand the page?
Does it choose to retrieve, cite, mention, or recommend that information for a particular question?

Improving the first layer does not guarantee the second. But a page that exposes no meaningful content creates an avoidable technical obstacle.

How to compare the versions

Start with the exact public URL, not a development route or a component in isolation.

1. Fetch the initial response

Use an HTTP client and follow redirects deliberately:

curl -L https://example.com/product

Save the body and inspect it as a document. Look for the page's main heading, a distinctive sentence, important links, canonical metadata, and structured data.

Also record the final URL, status code, content type, redirect chain, and response size. A 200 response only proves that the server returned something successfully; it does not prove that the response contains the page.

2. Inspect View Source

“View Source” is a convenient representation of the server response. Do not confuse it with the Elements panel, which shows the live DOM after browser processing.

Search the source for a sentence that visibly appears on the page. If it is absent, determine whether the browser adds it during rendering.

3. Disable JavaScript

Loading the page without JavaScript is a useful diagnostic, although it is not a perfect crawler simulation. It quickly reveals whether meaningful HTML exists before the application starts.

The expected result depends on the product. A complex editor may reasonably require JavaScript to function. Its public landing page should still be understandable without waiting for an application API.

4. Render in a controlled browser

Use Playwright, Puppeteer, or another browser automation tool to capture the resulting DOM. Define what “finished” means: load may be too early, while network idle may never arrive on a page with analytics or live connections.

Compare meaningful signals rather than performing a byte-for-byte diff:

Normalized visible text
Headings
Links and destinations
Title, canonical, and description
Robots directives
Structured-data blocks
Final URL and HTTP behavior

5. Repeat with relevant user agents

User-agent testing can reveal routing differences, but it must be interpreted carefully. A different response does not automatically mean that a crawler will execute or accept it. Conversely, an identical response does not show what the crawler later renders.

Check what your own infrastructure does. Avoid claiming knowledge of a crawler's internal rendering process based only on a user-agent request.

When prerendering helps

Prerendering can be useful when a public, content-oriented route sends a thin application shell and changing the application's rendering architecture is not immediately practical.

A rendering service can execute the page, capture useful HTML, cache it, and serve that representation to eligible crawlers. This can be especially practical for an established SPA where a framework migration would be disproportionate to the problem.

The output should remain current, preserve correct statuses and redirects, and represent the same public content a visitor receives. Stale or materially different content creates a new problem.

When prerendering is unnecessary

Do not add a rendering layer merely because the site uses JavaScript.

You probably do not need it when:

Server rendering or static generation already returns the important content
The raw and rendered versions differ only in interactive behavior
The route is private or application-only and is not intended for discovery
The missing content can be fixed directly with a smaller server-side change
The page's public metadata, text, links, and structured data are already present

The best outcome is not “prerender everything.” It is “make each public page's important content reliably available with the least unnecessary complexity.”

Test the response, not the framework label

This raw-versus-rendered gap was one of the problems that led me to build Prerender Buddy. Its free checker compares the initial response, crawler-facing HTML, and browser-rendered page so the diagnosis is based on the URL's actual behavior rather than assumptions about its stack.

You do not need a product to perform the basic investigation. curl, browser source, developer tools, and a controlled browser script will take you a long way.

The important habit is to stop treating “it loads in my browser” as the end of the test. A modern page is a sequence of representations. If discovery matters, inspect the representation that arrives before the interface becomes visible.