<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Search Engine Basics</title>
    <description>The latest articles on DEV Community by Search Engine Basics (@searchenginebasics).</description>
    <link>https://dev.to/searchenginebasics</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4151833%2F7d79328e-040a-4314-86e9-66f302addc2d.png</url>
      <title>DEV Community: Search Engine Basics</title>
      <link>https://dev.to/searchenginebasics</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/searchenginebasics"/>
    <language>en</language>
    <item>
      <title>Core Web Vitals in Practice: Fix LCP, INP, and CLS Without Guessing</title>
      <dc:creator>Search Engine Basics</dc:creator>
      <pubDate>Tue, 20 Oct 2026 11:30:54 +0000</pubDate>
      <link>https://dev.to/searchenginebasics/core-web-vitals-in-practice-fix-lcp-inp-and-cls-without-guessing-1b76</link>
      <guid>https://dev.to/searchenginebasics/core-web-vitals-in-practice-fix-lcp-inp-and-cls-without-guessing-1b76</guid>
      <description>&lt;p&gt;Core Web Vitals is one of the most searched topics in technical SEO, and one of the most misunderstood. Teams run a Lighthouse test, see a red score, and start optimizing whatever the report lists first. That is backwards. The scores that matter are not the ones from your laptop. They are the ones from your real visitors, collected over 28 days, judged at the 75th percentile.&lt;/p&gt;

&lt;p&gt;I have debugged these metrics on content sites, dashboards, and e-commerce builds. This is the order of work that actually moves them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three metrics and what "good" means
&lt;/h2&gt;

&lt;p&gt;Google keeps the set deliberately small:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;LCP (Largest Contentful Paint)&lt;/strong&gt;: how fast the main content appears. Good is &lt;strong&gt;2.5 seconds or less&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;INP (Interaction to Next Paint)&lt;/strong&gt;: how fast the page responds when a user clicks, taps, or types. Good is &lt;strong&gt;200 milliseconds or less&lt;/strong&gt;. INP replaced FID in March 2024 because it measures the full interaction, not just the first delay.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CLS (Cumulative Layout Shift)&lt;/strong&gt;: how much the layout jumps around while loading. Good is &lt;strong&gt;0.1 or less&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To pass, at least 75% of real visits must land in the "good" range for all three. One failing metric fails the assessment. If you want the full reference in one place, this &lt;a href="https://www.searchenginebasics.dev/technical/core-web-vitals/" rel="noopener noreferrer"&gt;Core Web Vitals&lt;/a&gt; guide covers each metric's definition and measurement in depth. The rest of this article is about fixing them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Field data beats lab data, every time
&lt;/h2&gt;

&lt;p&gt;Lighthouse and PageSpeed's top score are lab data: one simulated run, on one device, on one connection. Useful for debugging, useless as a verdict. Field data (the Chrome User Experience Report, or your own analytics) is what Google actually uses for the page experience signal.&lt;/p&gt;

&lt;p&gt;The gap between the two surprises people. A page can score 95 in the lab and fail in the field because real visitors arrive on mid-range Android phones, on slow networks, with caches in unknown states. Always start from field data, find the templates that fail, then reproduce them in the lab to debug. Optimizing in the other direction burns weeks.&lt;/p&gt;

&lt;h2&gt;
  
  
  LCP: find the element before you fix anything
&lt;/h2&gt;

&lt;p&gt;LCP fails more than the other two combined, and the first step is embarrassingly simple: identify which element is the LCP element on your failing templates. It is usually the hero image or the headline block, and the fix depends entirely on which one it is.&lt;/p&gt;

&lt;p&gt;The usual culprits, in the order I check them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Slow server response.&lt;/strong&gt; If TTFB is poor, nothing downstream can save LCP. Cache the page, fix the backend, or put a CDN in front of it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://dev.to/jenchen/optimize-core-web-vitals-fcp-and-lcp-render-blocking-resources-34ng"&gt;Render-blocking resources&lt;/a&gt;.&lt;/strong&gt; CSS and synchronous scripts in the head delay first paint, and LCP waits behind them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The hero image itself.&lt;/strong&gt; Uncompressed, wrongly sized, lazy-loaded by mistake, or discovered late because it is set as a CSS background. Preload the actual LCP image and never lazy-load it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Web fonts.&lt;/strong&gt; A headline waiting on a font swap paints late. Preload the critical font or use font-display properly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Performance work runs on budgets, and the budget mindset matters more than any single trick. Serving teams in AI products work the same way, where a strict &lt;a href="https://www.softbrixai.com/insights/sub-250ms-inference-latency-budget/" rel="noopener noreferrer"&gt;inference latency budget&lt;/a&gt; decides whether a feature feels instant or broken. Your LCP budget deserves the same discipline: a number, an owner, and a check that runs on every release.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F59w1zme880krs1m9bcki.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F59w1zme880krs1m9bcki.png" alt="Fix LCP in This Order" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  INP: the metric your JavaScript is failing
&lt;/h2&gt;

&lt;p&gt;INP is where modern front-ends struggle. Every interaction is measured in three parts: input delay (waiting for the main thread), processing time (your event handlers), and presentation delay (rendering the result). Long tasks block all three.&lt;/p&gt;

&lt;p&gt;Practical fixes that consistently help:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Break up long tasks with yielding (scheduler.yield or setTimeout splits) so input can interleave.&lt;/li&gt;
&lt;li&gt;Audit third-party scripts. Tag managers and chat widgets are frequent INP killers, and nobody owns them.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://dev.to/alex_aslam/optimizing-dom-updates-in-javascript-for-better-performance-90k"&gt;Keep the DOM smaller on interactive pages&lt;/a&gt;. Style and layout costs scale with node count.&lt;/li&gt;
&lt;li&gt;Debounce input handlers, and move heavy computation off the critical interaction path.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Measure INP per interaction type in the field. The search box, the menu, and the checkout button will tell you different stories.&lt;/p&gt;

&lt;h2&gt;
  
  
  CLS: stop the page from moving
&lt;/h2&gt;

&lt;p&gt;Layout shift is the most fixable metric because its causes are mechanical:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Images and videos without width and height attributes (or an aspect-ratio box) reserve no space.&lt;/li&gt;
&lt;li&gt;Ads, embeds, and iframes injected into space that was never reserved for them.&lt;/li&gt;
&lt;li&gt;Content pushed in late: cookie banners, promo bars, and client-side rendered blocks appearing above existing content.&lt;/li&gt;
&lt;li&gt;Web fonts swapping and reflowing text.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Reserve space for everything that loads late. If an element cannot have a reserved box, load it below the fold or behind a user action. Most pages can reach near-zero CLS with a day of disciplined fixes, and it stays fixed because the causes do not drift the way performance does.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg1vmtecrud72y3hxfhig.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg1vmtecrud72y3hxfhig.png" alt="4 CLS Traps (All Fixable)" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The workflow that keeps scores green
&lt;/h2&gt;

&lt;p&gt;Passing once is not the goal. Scores decay as pages gain features, images, and third-party weight. The sites that stay green treat Core Web Vitals as a release gate: field data reviewed monthly, a budget per template, and a rule that a regression gets a ticket like any other bug.&lt;/p&gt;

&lt;p&gt;Start with the field data, fix the failing template, and verify at the 75th percentile over the next 28-day window. That loop, repeated, is the entire discipline.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>performance</category>
      <category>javascript</category>
    </item>
    <item>
      <title>AI Crawlers Are Hitting Your Site Right Now. Should You Block Them?</title>
      <dc:creator>Search Engine Basics</dc:creator>
      <pubDate>Wed, 30 Sep 2026 09:09:52 +0000</pubDate>
      <link>https://dev.to/searchenginebasics/ai-crawlers-are-hitting-your-site-right-now-should-you-block-them-2e2e</link>
      <guid>https://dev.to/searchenginebasics/ai-crawlers-are-hitting-your-site-right-now-should-you-block-them-2e2e</guid>
      <description>&lt;p&gt;Go check your server logs. I'll wait.&lt;/p&gt;

&lt;p&gt;See those user agents you don't recognize? &lt;code&gt;GPTBot&lt;/code&gt;, &lt;code&gt;ClaudeBot&lt;/code&gt;, &lt;code&gt;CCBot&lt;/code&gt;, &lt;code&gt;PerplexityBot&lt;/code&gt;, &lt;code&gt;Bytespider&lt;/code&gt;, &lt;code&gt;Amazonbot&lt;/code&gt; — that's not Googlebot doing its usual rounds. That's a second, newer wave of crawlers: bots collecting your content to train AI models and feed retrieval systems that answer questions &lt;em&gt;without ever sending you a visitor&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;As developers, we've spent a decade optimizing for search crawlers. Now we have to make a call our SEO colleagues can't make for us: &lt;strong&gt;do we let these bots in, or shut the door?&lt;/strong&gt; Here's how to think about it like an engineer.&lt;/p&gt;

&lt;h2&gt;
  
  
  What AI crawlers actually do
&lt;/h2&gt;

&lt;p&gt;Search crawlers (Googlebot, Bingbot) fetch your pages to build an index that sends you traffic. The deal is obvious: they crawl, you get visitors.&lt;/p&gt;

&lt;p&gt;AI crawlers have a murkier value exchange. They generally do one of two things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Training data collection&lt;/strong&gt; — scraping the web to train the next generation of language models. Your content becomes weights in a model. You get nothing directly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrieval for AI answers&lt;/strong&gt; — fetching pages to ground responses in tools like AI Overviews, ChatGPT, or Perplexity. Here there &lt;em&gt;is&lt;/em&gt; a potential upside: being cited in an AI answer is the new front page.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The tricky part? The same bot often does both, and the operators are not always transparent about which. That's why a blanket "block everything" or "allow everything" is the wrong move.&lt;/p&gt;

&lt;h2&gt;
  
  
  The costs developers actually feel
&lt;/h2&gt;

&lt;p&gt;This isn't abstract. AI crawlers are aggressive — some ignore &lt;code&gt;crawl-delay&lt;/code&gt;, hit your site thousands of times a day, and request heavy pages (image galleries, faceted search URLs, API endpoints). I've seen them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Spike hosting bills&lt;/strong&gt; on metered infrastructure — one misbehaving bot can cost real money on serverless.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pollute analytics&lt;/strong&gt; — bot traffic mixed into your conversion data unless you filter it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trigger rate limits and WAF rules&lt;/strong&gt; — ironically, your own defenses can start blocking legitimate users when a bot hammers shared IPs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scrape behind-login or paywalled content&lt;/strong&gt; via aggressive URL guessing on sites with predictable routes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your site is small, a single aggressive crawler can be a noticeable percentage of your total traffic. Check before you decide anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case for letting them in
&lt;/h2&gt;

&lt;p&gt;Blocking feels good until you realize what you're giving up. AI-powered answer engines are where a growing share of discovery happens. If your content is never crawled by these systems, it can never be cited by them. For documentation sites, blogs, and open knowledge bases, that's a genuine distribution channel you're closing.&lt;/p&gt;

&lt;p&gt;There's also a middle path most teams miss: &lt;strong&gt;you don't have to treat every AI crawler the same.&lt;/strong&gt; Retrieval bots that cite sources (and can send traffic) are very different from pure training scrapers. A selective policy beats a binary one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make the decision like an engineer
&lt;/h2&gt;

&lt;p&gt;Here's the framework I use:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Measure first.&lt;/strong&gt; Pull 7 days of logs. Which AI user agents hit you, how often, which URLs, how much bandwidth? No data, no decision.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Classify your content.&lt;/strong&gt; Public marketing content and docs? Probably fine to allow retrieval. User-generated content, paywalled material, or expensive-to-render pages? Lean toward blocking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Separate training from retrieval.&lt;/strong&gt; Allow bots that attribute and cite; block pure scrapers that give nothing back.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Start permissive, tighten deliberately.&lt;/strong&gt; It's easier to block a specific bad actor you can name than to undo months of invisibility in AI answers.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For a deeper breakdown of the trade-offs — including which specific bots do what — this guide on &lt;a href="https://www.searchenginebasics.dev/crawling/should-you-block-ai-crawlers/" rel="noopener noreferrer"&gt;whether you should block AI crawlers&lt;/a&gt; walks through each major crawler's behavior and the robots.txt directives that control them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Controlling them properly (not just vibes)
&lt;/h2&gt;

&lt;p&gt;If you decide to block or throttle, do it at the right layer:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;robots.txt&lt;/strong&gt; — the standard mechanism. But write it carefully: a sloppy &lt;code&gt;Disallow&lt;/code&gt; can nuke your search traffic too. Group AI bots explicitly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp28g7bo77tybc8rd3qxa.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp28g7bo77tybc8rd3qxa.webp" alt="Infographic showing how to block AI crawlers with robots.txt rules" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Keep Googlebot and other search crawlers on their own rules, and test the file — one misplaced wildcard has taken down more sites than any algorithm update.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Server-level controls&lt;/strong&gt; — for bots that ignore robots.txt (and some do), enforce at the edge: rate-limit by user agent, return 403s, or challenge with your WAF. Robots.txt is a request, not a lock.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;llms.txt&lt;/strong&gt; — the emerging convention: a markdown file at &lt;code&gt;/llms.txt&lt;/code&gt; that gives AI systems a clean, structured summary of your site. Think of it as a sitemap for language models. Early, but worth watching.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this matters beyond SEO: the RAG connection
&lt;/h2&gt;

&lt;p&gt;Here's the part most web developers miss: AI crawlers aren't just an SEO problem. The content they collect feeds &lt;a href="https://www.esaholic.com/services/enterprise-rag-systems/" rel="noopener noreferrer"&gt;enterprise RAG systems&lt;/a&gt; — retrieval-augmented generation pipelines that companies build on top of web-scale data. Your documentation, your tutorials, your forum answers: they end up chunked, embedded, and retrieved inside someone else's AI product.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnvj9xy60rj1gflyx5v5j.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnvj9xy60rj1gflyx5v5j.webp" alt="Diagram of an enterprise RAG pipeline showing how crawled web content is chunked, embedded, and retrieved" width="800" height="267"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That cuts both ways. If you &lt;em&gt;build&lt;/em&gt; AI products, you should understand exactly how this pipeline consumes web content — because the same crawling, chunking, and embedding mechanics apply to your own knowledge bases. The crawler problem and the RAG problem are the same problem viewed from opposite sides.&lt;/p&gt;

&lt;h2&gt;
  
  
  And then there are the agents
&lt;/h2&gt;

&lt;p&gt;The next wave is already here: &lt;a href="https://www.softbrixai.com/services/ai-agent-development/" rel="noopener noreferrer"&gt;AI agents that browse the web like users&lt;/a&gt; — clicking, scrolling, filling forms. They don't respect robots.txt the way crawlers do because they often drive real browsers. If you thought bot management was solved, agents are about to reopen the whole discussion: how do you distinguish a helpful agent completing a task for a user from a scraper wearing a browser costume?&lt;/p&gt;

&lt;p&gt;For now: get your crawler policy right, instrument your traffic so you can see what's actually hitting you, and keep the policy reviewable. The bot landscape is changing quarterly.&lt;/p&gt;

&lt;h2&gt;
  
  
  The checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Audit 7 days of logs for AI user agents and their bandwidth cost&lt;/li&gt;
&lt;li&gt;[ ] Classify content: public / attributable vs. private / expensive&lt;/li&gt;
&lt;li&gt;[ ] Write explicit robots.txt rules per bot family — don't blanket-block&lt;/li&gt;
&lt;li&gt;[ ] Enforce at the edge for bots that ignore robots.txt&lt;/li&gt;
&lt;li&gt;[ ] Revisit quarterly — new bots appear constantly&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The bots aren't going away. The developers who measure, decide deliberately, and control at the right layer will be fine. The ones running on vibes and blanket blocks will either pay the bandwidth bill or vanish from AI answers entirely. Pick your trade-off on purpose.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;More on crawlers, robots.txt, and technical SEO: browse the &lt;a href="https://dev.to/t/seo"&gt;SEO&lt;/a&gt; and &lt;a href="https://dev.to/t/ai"&gt;AI&lt;/a&gt; tags here on DEV.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>seo</category>
      <category>webdev</category>
      <category>programming</category>
    </item>
  </channel>
</rss>
