DEV Community: Matthias Ramahi

Image SEO in Next.js: next/image, alt text and the LCP hero it can't fix for you

Matthias Ramahi — Thu, 29 Oct 2026 15:00:00 +0000

next/image takes care of a lot of image SEO and performance work by default. It generates a srcset, serves WebP, lazy-loads, and reserves space so the layout does not jump. Those defaults are good enough that many teams stop looking at their images at all.

The problems that remain come from four decisions the component cannot make for you: which image is the LCP element, how wide each image actually renders, what the alt text says, and whether the image is in the HTML at all. Here is how to get each one right, plus a console snippet that checks a page in a few seconds.

1. The hero image is lazy-loaded

loading="lazy" is the default, and for most images that is correct. For the large image at the top of a product or article page it is usually wrong. That image is often the Largest Contentful Paint element, and lazy loading means the browser waits for layout before it even requests the file.

Next.js 16 deprecated the old priority prop in favor of preload. The docs are clear that in most cases you should use loading="eager" or fetchPriority="high" instead of preload:

// app/products/[slug]/page.tsx
import Image from "next/image";
import { getProduct } from "@/lib/catalog";

export default async function ProductPage({
  params,
}: {
  params: Promise&lt;{ slug: string }&gt;;
}) {
  const { slug } = await params;
  const product = await getProduct(slug);

  return (
    &lt;article&gt;
      &lt;Image
        src={product.hero.url}
        alt={product.hero.alt}
        width={1600}
        height={900}
        sizes="(min-width: 1024px) 66vw, 100vw"
        loading="eager"
        fetchPriority="high"
      /&gt;
      &lt;h1&gt;{product.name}&lt;/h1&gt;
    &lt;/article&gt;
  );
}

Reserve preload for an image that would otherwise be discovered late, and skip it when different images become the LCP element at different viewport sizes. Give fetchPriority="high" to one image per page. Marking six images as high priority is the same as marking none.

2. `sizes` is missing

Without sizes, the browser assumes the image will be as wide as the viewport, and Next.js generates only a small 1x/2x srcset. A card that renders at a third of the screen on desktop then downloads a file sized for the whole screen.

With sizes, Next.js generates the full width-based srcset and the browser can pick a file that fits. The value has to describe your layout, so it belongs next to the grid that defines it:

&lt;Image
  src={item.image.url}
  alt={item.image.alt}
  fill
  sizes="(min-width: 1024px) 33vw, (min-width: 640px) 50vw, 100vw"
/&gt;

If you change the grid from three columns to four and forget sizes, nothing breaks visibly. You just ship heavier images, which is why this one is worth checking automatically.

3. Alt text is an afterthought

alt is a required prop, so TypeScript forces you to pass something. That is exactly how alt="" and alt="image" end up on product photos. The Next.js docs describe the rule well: alt text should be able to replace the image without changing the meaning of the page, and purely decorative images get an empty string.

Search engines use alt text, together with the surrounding text and the file name, to understand what an image shows. The fix is not a better prop at render time. It is making alt text part of the content model, so an editor writes it when the image is added:

export type ContentImage = {
  url: string;
  width: number;
  height: number;
  alt: string;
  decorative?: boolean;
};

export function altFor(image: ContentImage): string {
  if (image.decorative) return "";
  if (!image.alt.trim()) {
    throw new Error(`Missing alt text for ${image.url}`);
  }
  return image.alt;
}

Throwing works well for statically generated pages, because the build fails before a bad page ships. For pages rendered at request time, log the missing value instead and report on it.

4. The image is not in the HTML

Google does not index images set through CSS background-image. And an image that only appears after a client-side carousel has hydrated is invisible to crawlers that do not run JavaScript. If an image matters for search, such as a product photo or a diagram that explains the article, render it as a real <img> in the server response.

Checking a page in the console

Paste this into DevTools on any page. It prints the LCP element and flags images with no alt attribute, lazy loading above the fold, files much larger than their rendered size, and missing intrinsic dimensions:

new PerformanceObserver((list) =&gt; {
  const lcp = list.getEntries().at(-1);
  console.log("LCP:", lcp.element, lcp.url, Math.round(lcp.startTime), "ms");
}).observe({ type: "largest-contentful-paint", buffered: true });

for (const img of document.images) {
  const box = img.getBoundingClientRect();
  const needed = box.width * devicePixelRatio;
  const issues = [];
  if (!img.hasAttribute("alt")) issues.push("no alt attribute");
  if (img.loading === "lazy" &amp;&amp; box.top &lt; innerHeight) issues.push("lazy above the fold");
  if (needed &gt; 0 &amp;&amp; img.naturalWidth &gt; needed * 1.5)
    issues.push(`${img.naturalWidth}px file for ${Math.round(needed)}px`);
  if (!img.getAttribute("width") &amp;&amp; img.dataset.nimg !== "fill")
    issues.push("no width/height");
  if (issues.length) console.log(img.currentSrc, issues);
}

data-nimg="fill" is the attribute next/image sets on fill images, which are sized by their container and do not need width and height. Run the snippet at a mobile and a desktop viewport, because the LCP element and the oversized files often differ between the two.

From one page to every page

A console snippet covers one URL. Image problems usually come from templates, so the next question is which templates are affected and how much it costs. Crawl Foundry Site Audit runs the official Lighthouse runner on crawled pages, and the Lighthouse measurement itself costs nothing extra on top of the audit, so slow LCP elements show up per URL next to the rest of the technical findings. See the Site Audit feature page.

For the broader picture of formats, compression and file names, read the glossary entry on image optimization, and the entry on Core Web Vitals for the LCP thresholds.

robots.txt for AI crawlers: block training, keep AI search, and test that it works

Matthias Ramahi — Thu, 22 Oct 2026 15:00:00 +0000

At some point someone on your team will ask you to "block the AI bots". Before you add a blanket rule, it is worth knowing that the major AI providers run different crawlers for different purposes, and that blocking the wrong one removes your site from AI search answers you probably want to appear in.

This post is the developer version of a longer guide on the Crawl Foundry blog: what each bot does, how to express the common policies in robots.txt (including in Next.js), one grouping rule that trips people up, and how to verify the result.

Three purposes, three kinds of bots

AI companies fetch pages for three reasons:

Training: collecting content to train future models
Search: building the index behind the assistant's search feature, which decides whether you can be cited as a source
User-initiated fetches: an assistant opening a specific page because a user asked it to

The main user agents, per the providers' own documentation:

OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user-initiated; OpenAI says robots.txt may not apply to these fetches)
Anthropic: ClaudeBot (model development), Claude-SearchBot (search), Claude-User (user-initiated)
Perplexity: PerplexityBot (its search index), Perplexity-User (user-initiated)
Google: Google-Extended is not a crawler but a robots.txt token. It controls whether content Google already crawls may be used for Gemini training and grounding. It does not affect Google Search, and AI Overviews and AI Mode run on the regular Googlebot.
Common Crawl: CCBot, whose datasets many model providers use

The list changes. Check the providers' documentation before a larger change.

The most common policy: block training, allow AI search

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /api/

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Several User-agent lines in a row form one group, so this is shorter than repeating the rules per bot.

The grouping gotcha

Notice that the Disallow: /admin/ and Disallow: /api/ lines appear twice. That is deliberate.

Under the robots.txt standard (RFC 9309), a crawler picks the group that matches its user agent most specifically and follows only that group. Rules are not merged with the * group. So the moment you give OAI-SearchBot its own group with Allow: /, it ignores everything you wrote under User-agent: *. If your private paths are only disallowed in the wildcard group, the bots you named explicitly are now allowed into them.

Any path restriction that should apply to every crawler has to be repeated in every named group.

The same policy in Next.js

If you use the App Router, app/robots.ts generates the file, and userAgent accepts an array:

import type { MetadataRoute } from "next";

const privatePaths = ["/admin/", "/api/"];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: ["GPTBot", "ClaudeBot", "Google-Extended", "CCBot"],
        disallow: "/",
      },
      {
        userAgent: ["OAI-SearchBot", "Claude-SearchBot", "PerplexityBot"],
        allow: "/",
        disallow: privatePaths,
      },
      {
        userAgent: "*",
        allow: "/",
        disallow: privatePaths,
      },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

Keeping privatePaths in one constant makes the grouping rule hard to forget.

What robots.txt can't do

robots.txt is a request. Reputable crawlers follow it; nothing forces the rest to. If content must not be fetched, protect it on the server or behind a login.

Changes also only apply going forward and not instantly. Content that was already collected doesn't disappear, and OpenAI notes that it can take about 24 hours for OAI-SearchBot to pick up a change.

For Google Search specifically, robots.txt is the wrong tool for controlling what appears in snippets or AI Overviews. Page-level directives such as nosnippet or max-snippet do that. Blocking Googlebot removes the page from search entirely.

Verifying the result

1. Check the rules per bot. Parse your live robots.txt and ask, for each user agent you care about, whether a sample URL is allowed. Crawl Foundry has a free robots.txt checker for AI crawlers that shows which AI and search crawlers are allowed or blocked and which rule applies.

2. Check your edge. Many CDNs and WAFs ship their own "block AI bots" switches that act before robots.txt is ever read. A quick test with a spoofed user agent tells you whether the edge returns something other than your page:

for ua in "GPTBot" "OAI-SearchBot" "ClaudeBot" "Claude-SearchBot" "PerplexityBot"; do
  printf "%-18s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/
done

A 403 or a challenge page for a bot your robots.txt allows means the CDN setting wins, whatever the file says. Spoofed requests don't come from the providers' IP ranges, so some bot-management products treat them differently from the real crawler; use this as a smoke test, not proof.

3. Check your logs. Server or CDN logs show which bots actually visit and what status they get. OpenAI and Anthropic publish the IP ranges of their crawlers, which lets you separate real visits from spoofed user agents.

One more thing: can they read the page?

Allowing a crawler doesn't help if it receives an empty shell. Many AI crawlers don't execute JavaScript, so content that only exists after client-side rendering may be invisible to them. Check the raw HTML response of your important pages, not the DOM in your browser.

The full guide, including setups for shops, software vendors, publishers and paywalled content, plus a list of common mistakes, is on the Crawl Foundry blog: How to Block or Allow AI Crawlers: GPTBot, ClaudeBot and Others in robots.txt. For the basics of the file itself, see the glossary entry on robots.txt.

Finding striking-distance keywords with the Search Console API and a short Node script

Matthias Ramahi — Wed, 21 Oct 2026 17:19:23 +0000

Striking-distance keywords are the queries where your site already shows up, just not quite high enough: average positions somewhere between 8 and 20, with real impressions behind them. They are often the cheapest SEO wins you have, because Google has already decided the page is relevant. The Search Console performance report can show them, but it caps exports at 1,000 rows and makes it awkward to ask the one question that matters: which URL is ranking for each query, and is it the only one?

The Search Console API answers that question in a few dozen lines of Node. Below is a small script for it, and the reasoning behind its thresholds.

Setup: a read-only service account

In Google Cloud, create a project, enable the Google Search Console API and create a service account with a JSON key.
In Search Console, open the property settings and add the service account's email address as a user. Restricted permission is enough for reading performance data.
Install the client library and point it at the key:

npm install googleapis
export GOOGLE_APPLICATION_CREDENTIALS=./gsc-key.json
export GSC_SITE="sc-domain:example.com"

Use sc-domain:example.com for a domain property or the full URL with trailing slash (https://www.example.com/) for a URL-prefix property. The only scope the script needs is webmasters.readonly. When I built the Search Console connection for Crawl Foundry, the SEO tool I work on in my spare time, it requests exactly that scope and nothing else. A reporting script has no business holding write access to your property.

Pulling query and page rows

The key decision is the dimensions. Requesting query alone gives you one average position per query across all of your URLs. Requesting query and page gives you one row per URL that appeared for that query, which is what you need to see which page Google picked and whether several pages compete.

// striking-distance.mjs
import { google } from "googleapis";

const auth = new google.auth.GoogleAuth({
  scopes: ["https://www.googleapis.com/auth/webmasters.readonly"],
});
const gsc = google.searchconsole({ version: "v1", auth });

function daysAgo(n) {
  const d = new Date();
  d.setUTCDate(d.getUTCDate() - n);
  return d.toISOString().slice(0, 10);
}

async function fetchRows(startDate, endDate) {
  const rows = [];
  for (let startRow = 0; ; startRow += 25000) {
    const res = await gsc.searchanalytics.query({
      siteUrl: process.env.GSC_SITE,
      requestBody: {
        startDate,
        endDate,
        type: "web",
        dimensions: ["query", "page"],
        rowLimit: 25000,
        startRow,
      },
    });
    const batch = res.data.rows ?? [];
    rows.push(...batch);
    if (batch.length &lt; 25000) return rows;
  }
}

rowLimit tops out at 25,000 per request, so the loop pages with startRow until a short page comes back. The window ends three days ago because the most recent days are still incomplete, and a four-week window evens out weekday swings.

Filtering for striking distance

const rows = await fetchRows(daysAgo(31), daysAgo(3));

const byQuery = new Map();
for (const row of rows) {
  const [query, page] = row.keys;
  const pages = byQuery.get(query) ?? [];
  pages.push({ page, impressions: row.impressions, clicks: row.clicks, position: row.position });
  byQuery.set(query, pages);
}

const candidates = [];
for (const [query, pages] of byQuery) {
  pages.sort((a, b) =&gt; b.impressions - a.impressions);
  const lead = pages[0];
  const impressions = pages.reduce((sum, p) =&gt; sum + p.impressions, 0);
  if (impressions &lt; 100) continue;
  if (lead.position &lt; 8 || lead.position &gt; 20) continue;
  candidates.push({
    query,
    page: lead.page.replace(/^https?:\/\/[^/]+/, ""),
    position: lead.position.toFixed(1),
    impressions,
    clicks: pages.reduce((sum, p) =&gt; sum + p.clicks, 0),
    pages: pages.length,
  });
}

candidates.sort((a, b) =&gt; b.impressions - a.impressions);
console.table(candidates.slice(0, 30));

Run it with node striking-distance.mjs and you get a table sorted by impressions, which is a rough proxy for how much traffic page one could bring.

Why the thresholds look like this

The lead page, not the query average. For each query the script takes the URL with the most impressions and uses its position. That is the page you would actually improve.

A minimum of 100 impressions. Average position over a handful of impressions is noise. The number is a starting floor for a four-week window on a small site; raise it for larger sites until the list becomes something you can work through.

The pages column. When it is greater than 1, several of your URLs appeared for the same query. Sometimes that is harmless, a product page and its category page. Sometimes it is keyword cannibalization, and improving either page won't help until you decide which one should own the query.

What the numbers do not tell you

A few caveats before acting on the list:

Average position is a blend. It averages the topmost position of your result across all impressions. An 11.4 can mean steady 11s, or 8 on desktop and 15 on mobile. Add device to the dimensions if a candidate looks odd.
Rare queries are missing. Search Console leaves out anonymized queries, so query-level rows will not add up to the property totals. That is fine for finding candidates and wrong for estimating total traffic.
CTR depends on the result page. Moving from 11 to 7 does not guarantee clicks if the results are full of ads and SERP features. The glossary entry on organic click-through rate covers what drives it.

Turning candidates into work

Each row deserves one of three decisions: improve the lead page for that query, consolidate competing pages, or drop it because the intent does not match what you offer. Write the decision next to the query, for example in a CSV export of the table, so next month's run shows whether it worked.

For that follow-up, Search Console's blended average is a weak instrument. A rank tracker that checks one keyword from a fixed location and device every day shows movement much more clearly. If you would rather not maintain the script yourself, the Search Console integration connects a property read-only to the Site Audit of the same domain. The Google Search Console glossary entry explains the reports this data comes from.

Connecting Claude Code to an SEO MCP server, and what a safe first session looks like

Matthias Ramahi — Thu, 08 Oct 2026 15:00:00 +0000

Ask a coding agent which keywords your docs should target and it will happily give you a list with search volumes. The volumes are invented. The model has no idea how often a term is searched, where your pages rank, or which of your URLs return a 404. It fills the gap with plausible numbers, and plausible numbers are worse than none because they look like data.

The Model Context Protocol (MCP) is the standard way to fix that: the agent calls tools on a server that has the real data. This post walks through connecting Claude Code to the Crawl Foundry SEO MCP server and, more importantly, through how to run the first session so nothing gets bought or changed by accident.

What the server exposes

Crawl Foundry is an SEO web app with keyword research, rank tracking, site audits and content tools. Its MCP server gives an agent access to the same functions inside one workspace: researching and organizing keywords, reading rankings and competitors, starting and reviewing site audits, checking backlinks and AI visibility, building SEO plans, and writing, scoring and optimizing content.

The part worth understanding before you connect anything is the cost model, because some of those tools fetch fresh third-party data and that costs money. Every tool falls into one of four classes:

Free: reading data already saved in your workspace
Pay per use: fresh data or generation, charged to your EUR balance
Price first: larger actions that need your approval or must fit inside the agent budget
Approval only: publishing, regardless of how high the budget is

Adding the server

The endpoint is https://app.crawlfoundry.com/api/mcp and it speaks MCP over Streamable HTTP. For Claude Code, one command adds it at user scope:

claude mcp add --transport http --scope user crawl_foundry "https://app.crawlfoundry.com/api/mcp"

Codex has an equivalent:

codex mcp add crawl_foundry --url "https://app.crawlfoundry.com/api/mcp"

Cursor and VS Code have one-click install links on the feature page, and any other client that supports Streamable HTTP works by pasting the address into its MCP settings.

There is no API key in this setup. The first time the agent calls the server, your client opens the Crawl Foundry sign-in in the browser. You log in, see which workspace and permissions are requested and allow access. That is plain OAuth, which means no long-lived secret ends up in a dotfile, a shell history or a chat transcript.

One workspace per session

If you manage several sites, this is the detail that matters most. Each MCP session works in exactly one workspace, and the agent has to confirm which one before it starts. Data from two clients never mixes in a single conversation, and grants and budgets apply per workspace.

In practice the agent will ask, or you tell it up front: "Use the workspace for docs.example.com." If your organization has more than 25 workspaces, you use a workspace-specific address instead of the organization-wide one.

The first prompt should buy nothing

Start with a question that reads saved data and changes nothing. Something like:

Show me the saved SEO overview for this workspace.
Don't buy new data and don't change anything.

This does three things. It confirms the connection and the workspace. It shows you what is already stored, which on a fresh workspace may be very little. And it shows you how the agent reports gaps: the server returns explicit data gaps and coverage states instead of empty arrays the model might paper over, and a well-behaved agent will tell you "there is no stored ranking data yet" rather than guess.

Only after that is it worth moving to a paid action, with the limit named in the prompt:

Research keywords around "self-hosted analytics".
Show me the expected cost before you run anything.

Budgets with hard caps

Out of the box, the agent budget is EUR 5 per action, EUR 10 per workspace per month and EUR 50 per organization per month. Owners and admins can change those numbers or switch the agent budget off, and only they manage connections and grants.

Some actions sit outside any budget. Publishing, and changes to deployments, DNS, users or billing, always require explicit approval. An agent that runs in a loop can spend at most what you allowed; it cannot publish a draft to your CMS because it decided the draft was ready.

When to use the REST API instead

MCP is designed for agents: the model chooses the tools. If you are writing your own automation, say a nightly job that pulls audit findings into a dashboard, the same functions are available through a versioned REST API with an OpenAPI description. API keys there are created and revoked by organization admins, each client is bound to one workspace, and the same EUR balance and cost classes apply. The REST side gets its own post later in this series.

Things to check after the first session

Run claude mcp list and confirm the server shows as connected.
In Crawl Foundry, look at the activity and cost log for the workspace. Every paid run shows up there with its cost.
Revoke and reconnect once, so you know how to cut access quickly if you ever need to.

The full setup for every client, the cost classes and the guardrails are documented on the Crawl Foundry SEO MCP server page. If you want to see what multi-step agent workflows look like on top of it, the AI agents overview is the next stop. Crawl Foundry is pay-per-use without a subscription obligation, and the MCP server is included on every plan, including the free one.