<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ellie</title>
    <description>The latest articles on DEV Community by Ellie (@ellieui).</description>
    <link>https://dev.to/ellieui</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4166029%2F90039acc-074e-4b68-a9bb-ccc89ef52e6a.gif</url>
      <title>DEV Community: Ellie</title>
      <link>https://dev.to/ellieui</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ellieui"/>
    <language>en</language>
    <item>
      <title>What Is an AI Web Crawler? Crawler vs Scraper, Explained Simply</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Thu, 12 Nov 2026 10:37:46 +0000</pubDate>
      <link>https://dev.to/spicrawl/what-is-an-ai-web-crawler-crawler-vs-scraper-explained-simply-467</link>
      <guid>https://dev.to/spicrawl/what-is-an-ai-web-crawler-crawler-vs-scraper-explained-simply-467</guid>
      <description>&lt;p&gt;People say "AI web crawler" and mean two different things. Mixing them up leads to picking the wrong tool. Here is the short version.&lt;/p&gt;

&lt;p&gt;Me, the first time someone said "just use an AI crawler":&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpp9ddi8qmowhf862bodm.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpp9ddi8qmowhf862bodm.gif" alt="A confused woman with math formulas floating around her" width="504" height="322"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Meaning 1: a bot that an AI company runs
&lt;/h2&gt;

&lt;p&gt;OpenAI, Anthropic, Google and Perplexity send bots to read public web pages. If you own a website, these are the bots that visit you. Each company publishes the names:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bot&lt;/th&gt;
&lt;th&gt;Company&lt;/th&gt;
&lt;th&gt;What it is for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;GPTBot&lt;/td&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;Content that may be used to train models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OAI-SearchBot&lt;/td&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;Search results in ChatGPT&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ChatGPT-User&lt;/td&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;A person asked ChatGPT to open a page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ClaudeBot&lt;/td&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;Content that could be used to train models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude-SearchBot&lt;/td&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;Better search results&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude-User&lt;/td&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;A person asked Claude a question&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PerplexityBot&lt;/td&gt;
&lt;td&gt;Perplexity&lt;/td&gt;
&lt;td&gt;Search results in Perplexity&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So the bots do three jobs: &lt;strong&gt;training&lt;/strong&gt;, &lt;strong&gt;search&lt;/strong&gt;, and &lt;strong&gt;one person's request&lt;/strong&gt;. You can allow one and refuse another in &lt;code&gt;robots.txt&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Think before you block everything. OpenAI says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Meaning 2: a tool that you run
&lt;/h2&gt;

&lt;p&gt;You are building an agent, a chatbot or a RAG pipeline. Your model needs web pages as input. A raw page is full of menus and scripts, so you use a tool that fetches the page and returns clean text, usually Markdown.&lt;/p&gt;

&lt;p&gt;These tools do one or both of two jobs. This is the part people mix up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Crawler vs scraper
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;A crawler finds pages.&lt;/strong&gt; You give it one starting address. It follows links and discovers more pages.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv610gnzh4ofsb2quofvd.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv610gnzh4ofsb2quofvd.gif" alt="A small spider going down a thread" width="480" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A scraper reads pages.&lt;/strong&gt; You give it an address. It returns what is on that page and goes nowhere else.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgf9d9qnphcuizpynep4b.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgf9d9qnphcuizpynep4b.gif" alt="A woman reading a book very fast" width="480" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Crawler&lt;/th&gt;
&lt;th&gt;Scraper&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;You give it&lt;/td&gt;
&lt;td&gt;One starting address&lt;/td&gt;
&lt;td&gt;The exact addresses you want&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;It does&lt;/td&gt;
&lt;td&gt;Follows links to find pages&lt;/td&gt;
&lt;td&gt;Reads each page you named&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;Hard to know before you start&lt;/td&gt;
&lt;td&gt;One request for each page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Best for&lt;/td&gt;
&lt;td&gt;"Get everything on this site"&lt;/td&gt;
&lt;td&gt;"Get these pages"&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Which one do you need?
&lt;/h2&gt;

&lt;p&gt;Ask one question: &lt;strong&gt;do you already know the URLs?&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Yes&lt;/strong&gt; (from a search result, a sitemap, your own list, or a link a user pasted): a scraper is enough.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No&lt;/strong&gt; (you only know the site and must find every page): you need a crawler first, then a scraper to read the pages.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you use a crawler, set a page limit and a depth limit. One site may have 50 pages. The next may have 50,000.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tools for each job
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Crawling:&lt;/strong&gt; &lt;a href="https://docs.firecrawl.dev/features/crawl" rel="noopener noreferrer"&gt;Firecrawl&lt;/a&gt; (hosted), &lt;a href="https://docs.crawl4ai.com/core/deep-crawling/" rel="noopener noreferrer"&gt;Crawl4AI&lt;/a&gt; (open source), and Apify's &lt;a href="https://apify.com/apify/website-content-crawler" rel="noopener noreferrer"&gt;Website Content Crawler&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scraping:&lt;/strong&gt; &lt;a href="https://spicrawl.com" rel="noopener noreferrer"&gt;Spicrawl&lt;/a&gt;, which is what we build. You send a URL, or up to 10,000 in one batch, and get Markdown, JSON, HTML, text or PDF back. It does &lt;strong&gt;not&lt;/strong&gt; crawl. It reads the pages you name.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Reading one page looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl https://api.spicrawl.com/v1/scrape \
  -H &lt;span class="s2"&gt;"Authorization: Bearer $SPICRAWL_API_KEY"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{"url": "https://example.com", "response_format": "markdown"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Be a good visitor
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Read &lt;code&gt;robots.txt&lt;/code&gt; and follow it.&lt;/li&gt;
&lt;li&gt;Go slowly on small sites.&lt;/li&gt;
&lt;li&gt;Keep a copy, and do not fetch the same page again and again.&lt;/li&gt;
&lt;li&gt;A 403 or a block page is a message. Stop and check.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What a 403 is really saying:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcs7berdvvsx4l3ooapm4.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcs7berdvvsx4l3ooapm4.gif" alt="Gandalf saying you shall not pass" width="250" height="163"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  In one line
&lt;/h2&gt;

&lt;p&gt;A crawler finds pages. A scraper reads them. Know which one you mean, and the right tool is easy to pick.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;This is a shorter version of a guide on the Spicrawl blog. The full post has every bot, more examples and all the sources: &lt;a href="https://spicrawl.com/blog/ai-web-crawler" rel="noopener noreferrer"&gt;What is an AI web crawler?&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>webscraping</category>
      <category>beginners</category>
    </item>
    <item>
      <title>Your API Returned 200 OK, But Your Dataset Is Still Incomplete 👀</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Tue, 03 Nov 2026 12:47:08 +0000</pubDate>
      <link>https://dev.to/ellieui/your-api-returned-200-ok-but-your-dataset-is-still-incomplete-5h2a</link>
      <guid>https://dev.to/ellieui/your-api-returned-200-ok-but-your-dataset-is-still-incomplete-5h2a</guid>
      <description>&lt;p&gt;&lt;strong&gt;Your API Says 200 OK. But Where Are the Other 1,900 Records from 2000?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You make an API request.&lt;/p&gt;

&lt;p&gt;Get &lt;code&gt;200 OK&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Everything looks fine. No errors. No warnings.&lt;/p&gt;

&lt;p&gt;But you expected 2,000 records and received only 100.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1waoj03rxim949dkhlzn.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1waoj03rxim949dkhlzn.gif" alt="got angry for the result" width="498" height="276"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Wait... where did the other 1,900 go? 😅&lt;/p&gt;

&lt;p&gt;This is usually where &lt;strong&gt;API pagination&lt;/strong&gt; comes into the picture.&lt;/p&gt;

&lt;p&gt;Many APIs don't return all records at once. They split the results into smaller pages.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Total records: 2,000&lt;/li&gt;
&lt;li&gt;Records per page: 100&lt;/li&gt;
&lt;li&gt;First response: 100&lt;/li&gt;
&lt;li&gt;Remaining: 1,900&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Your request succeeded, but your data collection isn't complete.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8vp2e1wj7c1viogqy171.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8vp2e1wj7c1viogqy171.gif" alt="disappointed" width="480" height="270"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A few things worth checking:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Look for &lt;code&gt;next&lt;/code&gt;, &lt;code&gt;next_cursor&lt;/code&gt; or &lt;code&gt;total&lt;/code&gt; in the response.&lt;/li&gt;
&lt;li&gt;Check whether the API uses page numbers or cursor-based pagination.&lt;/li&gt;
&lt;li&gt;Keep fetching until there are no more pages.&lt;/li&gt;
&lt;li&gt;Verify the final record count.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A &lt;code&gt;200 OK&lt;/code&gt; tells you the request worked. It doesn't guarantee you collected everything.&lt;/p&gt;

&lt;p&gt;And that's a pretty easy detail to miss when building an integration 👀&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Have you ever spent time debugging missing data, only to discover pagination was the problem?&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>api</category>
      <category>webdev</category>
      <category>programming</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Google Was Autocorrecting Our Brand Name, here's What I Did</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Sat, 31 Oct 2026 10:03:18 +0000</pubDate>
      <link>https://dev.to/spicrawl/google-was-autocorrecting-our-brand-name-heres-what-i-did-o7k</link>
      <guid>https://dev.to/spicrawl/google-was-autocorrecting-our-brand-name-heres-what-i-did-o7k</guid>
      <description>&lt;p&gt;&lt;strong&gt;How I Helped Google Recognise Our Brand Name: A Month of Learning SEO for Spicrawl&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When we launched &lt;strong&gt;Spicrawl on September 25, 2026&lt;/strong&gt;, I noticed something that felt a little strange.&lt;/p&gt;

&lt;p&gt;If I searched for our brand name, &lt;em&gt;&lt;a href="https://spicrawl.com/?utm_source=devto&amp;amp;utm_medium=blog&amp;amp;utm_campaign=&lt;post-name" rel="noopener noreferrer"&gt;Spicrawl&lt;/a&gt;&lt;/em&gt;, Google kept correcting it to &lt;em&gt;Sprawl&lt;/em&gt;. Our website wasn't showing up properly, even when I searched for the exact brand name.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F58ae4fjvus5fihewczq5.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F58ae4fjvus5fihewczq5.jpg" alt="screenshot prove" width="800" height="1262"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;At first, I thought this was just normal for a new website. And in a way, it was. Google didn't really know our brand yet.&lt;/p&gt;

&lt;p&gt;I honestly didn't know much about SEO at that point. I wasn't an SEO expert (and I'm still learning), and I had no proper roadmap. I started researching, reading, trying things, making mistakes, and slowly understanding how search engines discover and recognise a new website.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fttgdr8ni9hcbxlbdd9ze.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fttgdr8ni9hcbxlbdd9ze.gif" alt="avatar sad thinking" width="480" height="270"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;After around a month of work, Google finally started recognising our brand name correctly.&lt;/p&gt;

&lt;p&gt;Is this a huge SEO achievement? Probably not. There is still a lot of work left. But for me, it was a small milestone worth sharing.&lt;/p&gt;

&lt;p&gt;Here are some things I worked on and learned along the way.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Started with the technical SEO basics
&lt;/h2&gt;

&lt;p&gt;I first focused on making sure search engines could properly discover and understand our website.&lt;/p&gt;

&lt;p&gt;Some of the things I worked on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Page titles and meta descriptions.&lt;/li&gt;
&lt;li&gt;Sitemap and robots.txt.&lt;/li&gt;
&lt;li&gt;Canonical URLs and duplicate-page issues.&lt;/li&gt;
&lt;li&gt;Making important pages discoverable.&lt;/li&gt;
&lt;li&gt;Fixing indexing issues in Google Search Console.&lt;/li&gt;
&lt;li&gt;Making the website accessible to AI crawlers and tools through resources like &lt;code&gt;llms.txt&lt;/code&gt; and machine-readable discovery files.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The goal was simple: make it easier for search engines and AI systems to understand what Spicrawl is.&lt;/p&gt;

&lt;p&gt;I also learned that submitting a page for indexing doesn't guarantee Google will index it. Sometimes Google discovers a page but chooses not to index it.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Listed Spicrawl on MCP registries
&lt;/h2&gt;

&lt;p&gt;Since Spicrawl has an MCP server, getting it listed in relevant MCP directories was another important part of the work.&lt;/p&gt;

&lt;p&gt;These are the platforms I worked on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Official MCP Registry — &lt;strong&gt;my first priority and the most important listing.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Glama&lt;/li&gt;
&lt;li&gt;Smithery&lt;/li&gt;
&lt;li&gt;LobeHub MCP Marketplace&lt;/li&gt;
&lt;li&gt;cursor.directory&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Some submissions are still pending.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One thing I learned the hard way:&lt;/strong&gt; Submit to the Official MCP Registry first, then wait at least a week or two before manually submitting to other directories.&lt;/p&gt;

&lt;p&gt;Many MCP directories automatically fetch listings from the official registry. Check the others after a week or two; you may find that most listings are already there.&lt;/p&gt;

&lt;p&gt;I made the mistake of submitting manually first and adding the official listing later. This resulted in duplicate listings on some platforms.&lt;/p&gt;

&lt;p&gt;So, save yourself that little headache: &lt;strong&gt;official registry first, then check the others.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Made our brand information consistent
&lt;/h2&gt;

&lt;p&gt;I realised that Google needs consistent information to understand that different profiles and websites belong to the same brand.&lt;/p&gt;

&lt;p&gt;I worked on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Adding social media links to our website.&lt;/li&gt;
&lt;li&gt;Updating our About page.&lt;/li&gt;
&lt;li&gt;Adding our YouTube channel to structured brand information.&lt;/li&gt;
&lt;li&gt;Keeping our brand name and description consistent across platforms.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Small details, but they help build a clearer picture of a brand online.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Worked on product and startup directories
&lt;/h2&gt;

&lt;p&gt;I also explored product, startup, and software directories to build Spicrawl's presence across the web.&lt;/p&gt;

&lt;p&gt;Some platforms include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;G2&lt;/li&gt;
&lt;li&gt;SaaS Hub&lt;/li&gt;
&lt;li&gt;SourceForge&lt;/li&gt;
&lt;li&gt;AlternativeTo&lt;/li&gt;
&lt;li&gt;Make.rs&lt;/li&gt;
&lt;li&gt;StackShare&lt;/li&gt;
&lt;li&gt;GoodFirms&lt;/li&gt;
&lt;li&gt;Siteefy&lt;/li&gt;
&lt;li&gt;PeerPush&lt;/li&gt;
&lt;li&gt;Website Hunt&lt;/li&gt;
&lt;li&gt;Startup Inspire&lt;/li&gt;
&lt;li&gt;OpenHunts&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most of the marketplace listings were handled by my colleague, while I also explored and worked on several directories.&lt;/p&gt;

&lt;p&gt;There are many more to explore. My suggestion is to focus on platforms relevant to your product and audience rather than submitting everywhere just for backlinks.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Created social media profiles and started posting
&lt;/h2&gt;

&lt;p&gt;I also worked on creating and maintaining Spicrawl's social media presence.&lt;/p&gt;

&lt;p&gt;Social profiles can appear in Google search results quite quickly. Having consistent brand names, profile URLs, descriptions, and regular posts gives people more ways to discover your brand.&lt;/p&gt;

&lt;p&gt;You don't need to post every day. Just start building your presence.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Researched keywords and planned content
&lt;/h2&gt;

&lt;p&gt;I started exploring what people were actually searching for around web scraping and AI agents.&lt;/p&gt;

&lt;p&gt;Some topics I explored:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Web scraping for RAG and LLMs.&lt;/li&gt;
&lt;li&gt;Scraping dynamic websites.&lt;/li&gt;
&lt;li&gt;Web scraping 403 errors.&lt;/li&gt;
&lt;li&gt;Scraping websites with login.&lt;/li&gt;
&lt;li&gt;HTML vs Markdown for LLMs.&lt;/li&gt;
&lt;li&gt;Web scraping vs APIs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This helped me understand that writing content isn't just about publishing more blogs. It's about answering questions people are already searching for.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Monitored everything
&lt;/h2&gt;

&lt;p&gt;I used Google Search Console and Bing Webmaster Tools to understand what was happening with our website.&lt;/p&gt;

&lt;p&gt;I checked:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which pages were indexed or not indexed.&lt;/li&gt;
&lt;li&gt;Search impressions and clicks.&lt;/li&gt;
&lt;li&gt;Indexing and canonical issues.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I also explored IndexNow to notify search engines about updated URLs.&lt;/p&gt;

&lt;p&gt;These tools helped me understand what was happening instead of simply guessing.&lt;/p&gt;




&lt;h2&gt;
  
  
  What I learned from this experience
&lt;/h2&gt;

&lt;p&gt;A few things I'd suggest to anyone launching a new brand:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Set up Google Search Console and Bing Webmaster Tools early.&lt;/li&gt;
&lt;li&gt;Get your technical SEO basics right.&lt;/li&gt;
&lt;li&gt;Keep your brand information consistent across platforms.&lt;/li&gt;
&lt;li&gt;Submit to the Official MCP Registry first if you have an MCP server.&lt;/li&gt;
&lt;li&gt;Explore relevant product and startup directories.&lt;/li&gt;
&lt;li&gt;Create social profiles and keep them active.&lt;/li&gt;
&lt;li&gt;Research search queries before planning blog content.&lt;/li&gt;
&lt;li&gt;Be patient. Indexing and brand recognition take time.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A small milestone, not the finish line
&lt;/h2&gt;

&lt;p&gt;Today, when I search for &lt;a href="https://spicrawl.com/?utm_source=devto&amp;amp;utm_medium=blog&amp;amp;utm_campaign=&lt;post-name" rel="noopener noreferrer"&gt;Spicrawl&lt;/a&gt;, Google still shows a “Did you mean…” suggestion, but it no longer autocorrects the brand name. Instead, it shows information about Spicrawl, including our website, GitHub, and listings across other platforms.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmfhcruqp2ciaz0kkdho4.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmfhcruqp2ciaz0kkdho4.jpg" alt="prove of my task" width="576" height="1280"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That's a small change, but it means a lot to me because I know how much research and effort went into getting here.&lt;/p&gt;

&lt;p&gt;I don't think one particular SEO task fixed everything. It was a combination of small improvements over time.&lt;/p&gt;

&lt;p&gt;And honestly, this is just the beginning. There is still so much more to learn and work on.&lt;/p&gt;

&lt;p&gt;I'm sharing this because I know there are other people launching their first product or website who might be facing the same issue.&lt;/p&gt;

&lt;p&gt;If you're building a new brand, don't get discouraged if Google doesn't recognise your name immediately. Start with the basics, keep building your presence, and give it time.&lt;/p&gt;

&lt;p&gt;I'm still learning too. :) &lt;/p&gt;

&lt;p&gt;If you've worked on SEO for a new brand or have any suggestions to improve our visibility further, I'd love to hear your thoughts. I'm still learning, and every suggestion helps!&lt;/p&gt;

</description>
      <category>seo</category>
      <category>beginners</category>
      <category>startup</category>
      <category>webdev</category>
    </item>
    <item>
      <title>We built our 20-second launch video entirely in code (React, CSS and no 3D library)</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Thu, 22 Oct 2026 11:47:16 +0000</pubDate>
      <link>https://dev.to/spicrawl/we-built-our-20-second-launch-video-entirely-in-code-react-css-and-no-3d-library-16b1</link>
      <guid>https://dev.to/spicrawl/we-built-our-20-second-launch-video-entirely-in-code-react-css-and-no-3d-library-16b1</guid>
      <description>&lt;p&gt;&lt;strong&gt;We built our 20-second launch video entirely in code (React, CSS and no 3D library)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;We needed a launch video for Spicrawl, our web scraping API for AI agents. We didn't have a motion designer or After Effects, so we built the whole thing in code. It's React, CSS and a bit of SVG, rendered to MP4 with Remotion. Even the music is generated in JavaScript.&lt;/p&gt;

&lt;p&gt;Here's the result 👇&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/j6Bbpsy6ZkM" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;Below is how each part works, in case you want to try it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The setup: Remotion&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Remotion lets you write a video as React components. Each frame is a render of your components at a given frame number. You animate by reading the current frame and computing styles from it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;frame&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;useCurrentFrame&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;opacity&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;interpolate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;frame&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;extrapolateRight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;clamp&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;h1&lt;/span&gt; &lt;span class="na"&gt;style&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;opacity&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;Scrape.&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;h1&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you're building apps or AI agents that need live web data, check out &lt;a href="https://spicrawl.com/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=launch_video" rel="noopener noreferrer"&gt;Spicrawl&lt;/a&gt;. It provides a web scraping API and MCP to help you extract data from websites and use it in your projects.&lt;/p&gt;

&lt;p&gt;Spicrawl is currently free to try during beta.&lt;/p&gt;

</description>
      <category>react</category>
      <category>webdev</category>
      <category>javascript</category>
      <category>showdev</category>
    </item>
    <item>
      <title>Do We Actually Need AI Agents for Everything? 🤔</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Tue, 13 Oct 2026 12:43:42 +0000</pubDate>
      <link>https://dev.to/ellieui/do-we-actually-need-ai-agents-for-everything-3mj0</link>
      <guid>https://dev.to/ellieui/do-we-actually-need-ai-agents-for-everything-3mj0</guid>
      <description>&lt;p&gt;&lt;strong&gt;Do We Actually Need AI Agents for Everything?&lt;/strong&gt; 🤔&lt;/p&gt;

&lt;p&gt;AI agents are getting better at doing things on their own. They can write code, browse websites, use tools, and automate entire workflows.&lt;/p&gt;

&lt;p&gt;But sometimes I wonder… &lt;strong&gt;are we making simple tasks more complicated by turning everything into an AI agent?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiijrmweednchcropt3ah.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiijrmweednchcropt3ah.gif" alt="I'm thinking" width="480" height="270"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Think about it!!&lt;/p&gt;

&lt;p&gt;If you need to process a CSV file, a simple script might be enough. You don't necessarily need an AI agent making decisions at every step.&lt;/p&gt;

&lt;p&gt;But what if your project needs to collect data from hundreds of URLs, handle JavaScript-heavy websites, extract specific information, and keep that data updated?&lt;/p&gt;

&lt;p&gt;Now things get more interesting.&lt;/p&gt;

&lt;p&gt;You might need an AI agent to decide &lt;em&gt;what&lt;/em&gt; data to collect, but a dedicated tool or API to handle the actual data extraction reliably.&lt;/p&gt;

&lt;p&gt;I've been thinking about this while working on &lt;a href="https://spicrawl.com/?utm_source=devto&amp;amp;utm_medium=blog&amp;amp;utm_campaign=&lt;post-name" rel="noopener noreferrer"&gt;Spicrawl&lt;/a&gt;, especially when it comes to giving AI agents access to web data. Sometimes, letting an agent handle everything isn't the best approach. Giving it the right tools might be much more effective.&lt;/p&gt;

&lt;p&gt;So I'm curious:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;When do you think a simple script is enough, and when does a task actually need an AI agent?&lt;/li&gt;
&lt;li&gt;Would you let an agent handle every step, or give it dedicated tools for specific jobs?&lt;/li&gt;
&lt;li&gt;Are we making automation smarter, or just adding more complexity?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I'd love to hear how you're approaching this in your projects. &lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>programming</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Are We Becoming Better Developers, or Just Better at Prompting? 🤔</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Sat, 10 Oct 2026 12:55:30 +0000</pubDate>
      <link>https://dev.to/ellieui/are-we-becoming-better-developers-or-just-better-at-prompting-58i6</link>
      <guid>https://dev.to/ellieui/are-we-becoming-better-developers-or-just-better-at-prompting-58i6</guid>
      <description>&lt;p&gt;&lt;strong&gt;Are We Becoming Better Developers, or Just Better at Prompting?&lt;/strong&gt; 🤔&lt;/p&gt;

&lt;p&gt;Something I've been thinking about lately.&lt;/p&gt;

&lt;p&gt;AI tools can write code, fix bugs, explain errors, and even build entire features in minutes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2loa57yyf6exyysrb317.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2loa57yyf6exyysrb317.gif" alt="thinking" width="480" height="270"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Things that used to take hours can now take much less time.&lt;/p&gt;

&lt;p&gt;And that's amazing.&lt;/p&gt;

&lt;p&gt;But I sometimes wonder…&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are we actually becoming better developers, or are we just getting better at asking AI to do the work for us?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Think about it.&lt;/p&gt;

&lt;p&gt;When AI fixes a bug, do we take the time to understand &lt;em&gt;why&lt;/em&gt; it happened? When it generates a feature, do we understand how the code works, or are we just happy that it runs?&lt;/p&gt;

&lt;p&gt;I'm not saying using AI is a bad thing. I use AI tools too, and they've helped me a lot.&lt;/p&gt;

&lt;p&gt;But I think there's a difference between using AI to learn something and using AI to skip learning it.&lt;/p&gt;

&lt;p&gt;And maybe the real skill now isn't just writing code. It's knowing what to ask, understanding the answer, and recognizing when the answer is wrong.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I'm curious about your experience:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Have AI tools helped you understand coding better?&lt;/li&gt;
&lt;li&gt;Do you still try to solve problems yourself before asking AI?&lt;/li&gt;
&lt;li&gt;If AI disappeared tomorrow, how confident would you feel building something on your own? 👀&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No right or wrong answers. I'd genuinely love to hear different perspectives.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>discuss</category>
      <category>webdev</category>
      <category>programming</category>
    </item>
    <item>
      <title>AI Can Write the Code. But Can You Debug It? 😅</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Fri, 09 Oct 2026 11:24:47 +0000</pubDate>
      <link>https://dev.to/ellieui/ai-can-write-the-code-but-can-you-debug-it-539a</link>
      <guid>https://dev.to/ellieui/ai-can-write-the-code-but-can-you-debug-it-539a</guid>
      <description>&lt;p&gt;&lt;strong&gt;AI Can Write the Code. But Can You Debug It?&lt;/strong&gt; 😅&lt;/p&gt;

&lt;p&gt;AI makes writing code so much faster.&lt;/p&gt;

&lt;p&gt;You describe what you want, get some code, run it, and sometimes it works on the first try. Feels amazing, right?&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fak7028x9wnnhzc5cas75.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fak7028x9wnnhzc5cas75.gif" alt="avatar giving savage smile" width="560" height="315"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;But then you need to change one small thing.&lt;/p&gt;

&lt;p&gt;Suddenly, you're reading code you barely understand, fixing one bug creates another, and you're asking AI to fix the code it wrote in the first place. 😂&lt;/p&gt;

&lt;p&gt;I've been thinking about this a lot lately.&lt;/p&gt;

&lt;p&gt;AI is a great assistant, but I think there's still a big difference between &lt;strong&gt;getting code that works and understanding why it works.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;And I'm curious about something:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Do you always understand AI-generated code before using it?&lt;/li&gt;
&lt;li&gt;Have you ever fixed an AI-generated bug only to create two more?&lt;/li&gt;
&lt;li&gt;Or has AI actually made debugging easier for you?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No judgment here. I'm genuinely curious about your experience. 👀&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What's your most memorable experience with AI-generated code?&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>discuss</category>
      <category>ai</category>
      <category>programming</category>
      <category>webdev</category>
    </item>
    <item>
      <title>If Claude and Codex Can Scrape the Web, Why Do We Still Need Web Scraping Tools?</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Thu, 08 Oct 2026 16:26:45 +0000</pubDate>
      <link>https://dev.to/ellieui/if-claude-and-codex-can-scrape-the-web-why-do-we-still-need-web-scraping-tools-1i2l</link>
      <guid>https://dev.to/ellieui/if-claude-and-codex-can-scrape-the-web-why-do-we-still-need-web-scraping-tools-1i2l</guid>
      <description>&lt;p&gt;&lt;strong&gt;If Claude and Codex Can Scrape the Web, Why Do We Still Need Web Scraping Tools?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This question has been on my mind lately.&lt;/p&gt;

&lt;p&gt;Tools like &lt;strong&gt;Claude and Codex can already browse websites, read pages, extract information, and work with web content.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So if AI agents can do all of that...&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why would we still need a separate web scraping tool? 🤔&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3umwd6ocfqiegp9e9ovm.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3umwd6ocfqiegp9e9ovm.gif" alt="Avatar thinking" width="480" height="270"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Maybe the difference starts showing when you need more than a few pages.&lt;/p&gt;

&lt;p&gt;Hundreds of URLs. Structured output. JavaScript-heavy pages. Sessions. Repeated jobs. Reliable data that another part of your application can actually use.&lt;/p&gt;

&lt;p&gt;I’m currently working in this space on &lt;a href="https://spicrawl.com/?utm_source=devto&amp;amp;utm_medium=blog&amp;amp;utm_campaign=&lt;post-name" rel="noopener noreferrer"&gt;Spicrawl&lt;/a&gt;, so I’m genuinely curious how other people see it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Would you let your AI agent handle the whole web-data process itself, or would you still give it a dedicated scraping tool/API?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Would love to hear how you’re handling this in your projects. &lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>webscraping</category>
      <category>discuss</category>
    </item>
    <item>
      <title>What’s the Hardest Part of Building for the Web Right Now?</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Wed, 07 Oct 2026 07:43:59 +0000</pubDate>
      <link>https://dev.to/ellieui/whats-the-hardest-part-of-building-for-the-web-right-now-5di</link>
      <guid>https://dev.to/ellieui/whats-the-hardest-part-of-building-for-the-web-right-now-5di</guid>
      <description>&lt;p&gt;&lt;strong&gt;I’m curious.. what’s the hardest part of building for the web right now?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For me, it’s making AI and web data work reliably together.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdhieglqsimm7hblix8r5.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdhieglqsimm7hblix8r5.gif" alt="My avatar typing on a laptop" width="480" height="270"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I’m currently working on a web scraping project and exploring how to make web data more accessible and useful for AI agents, especially when dealing with dynamic pages, blocked requests, and messy data 😅&lt;/p&gt;

&lt;p&gt;What’s something you’ve been struggling with lately while building?&lt;br&gt;
Would love to hear from other developers.&lt;/p&gt;

</description>
      <category>discuss</category>
      <category>ai</category>
      <category>webdev</category>
      <category>webscraping</category>
    </item>
    <item>
      <title>Cloudflare Is Changing How AI Agents Access the Web in 2026</title>
      <dc:creator>Ellie</dc:creator>
      <pubDate>Tue, 06 Oct 2026 10:24:06 +0000</pubDate>
      <link>https://dev.to/ellieui/cloudflare-is-changing-how-ai-agents-access-the-web-in-2026-11ib</link>
      <guid>https://dev.to/ellieui/cloudflare-is-changing-how-ai-agents-access-the-web-in-2026-11ib</guid>
      <description>&lt;p&gt;AI agents are getting better at using the web.&lt;/p&gt;

&lt;p&gt;They can search for information, read documentation, compare products, research topics, and collect data for users.&lt;/p&gt;

&lt;p&gt;But there is a problem:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;More websites are deciding which kinds of AI traffic they want to allow.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Cloudflare is one of the biggest pieces of infrastructure between websites and their visitors. So when Cloudflare changes how AI traffic is handled, it matters to anyone building software that fetches web pages.&lt;/p&gt;

&lt;p&gt;In July 2026, Cloudflare introduced separate controls for three types of AI-related traffic:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Search&lt;/li&gt;
&lt;li&gt;Agent&lt;/li&gt;
&lt;li&gt;Training&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then, in September 2026, Cloudflare changed the default settings for new domains.&lt;br&gt;
Here is what developers need to know.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What changed?&lt;/strong&gt;&lt;br&gt;
Cloudflare's AI traffic controls were introduced in stages.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;July 1, 2026&lt;/strong&gt;&lt;br&gt;
Cloudflare introduced three separate AI traffic categories:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Search&lt;/strong&gt;&lt;br&gt;
Crawlers that collect or index content so it can be used to answer questions later.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Agent&lt;/strong&gt;&lt;br&gt;
Automated software acting on behalf of a person, usually in real time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Training&lt;/strong&gt;&lt;br&gt;
Crawlers collecting content to train or fine-tune AI models.&lt;/p&gt;

&lt;p&gt;Website owners can decide how each type of traffic should be handled.&lt;/p&gt;

&lt;p&gt;They can allow it, block it, or block it only on pages that display ads.&lt;br&gt;
Existing Cloudflare customers keep their existing settings.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;September 15, 2026&lt;/strong&gt;&lt;br&gt;
Cloudflare changed the default settings for new domains.&lt;br&gt;
For new domains:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Traffic type&lt;/strong&gt;     &lt;strong&gt;New default&lt;/strong&gt;&lt;br&gt;
Search                   Allowed&lt;br&gt;
Agent                    Blocked on pages with ads&lt;br&gt;
Training             Blocked on pages with ads&lt;/p&gt;

&lt;p&gt;Website owners can change these settings.&lt;br&gt;
The important part for AI developers is Agent traffic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why does this matter for AI agents?&lt;/strong&gt;&lt;br&gt;
Imagine a user asks an AI agent:&lt;br&gt;
"Find the latest information about this product"&lt;/p&gt;

&lt;p&gt;The agent may need to open several web pages, read them, and return the relevant information.&lt;br&gt;
That is different from a traditional search crawler.&lt;br&gt;
Cloudflare's definition of Agent is focused on automated software acting for a person in real time.&lt;br&gt;
That is very close to how many AI agents work today.&lt;br&gt;
So if more website owners use Cloudflare's new defaults, some AI agents may encounter more blocked pages.&lt;br&gt;
But this won't happen to every website.&lt;br&gt;
The new defaults apply to new domains onboarding to Cloudflare. Existing sites keep their current configuration unless the owner changes it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not every Cloudflare site will block AI agents&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is important.&lt;br&gt;
Cloudflare isn't simply saying:&lt;br&gt;
"AI crawlers are blocked"&lt;/p&gt;

&lt;p&gt;The website owner decides what happens.&lt;br&gt;
A site can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Allow Search traffic&lt;/li&gt;
&lt;li&gt;Allow Agent traffic&lt;/li&gt;
&lt;li&gt;Allow Training traffic&lt;/li&gt;
&lt;li&gt;Block one of them&lt;/li&gt;
&lt;li&gt;Block traffic only on pages with ads&lt;/li&gt;
&lt;li&gt;Block traffic everywhere&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So a Cloudflare-protected website does not automatically mean an AI agent cannot access it.&lt;br&gt;
You need to look at what the site actually allows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What about Googlebot and other mixed crawlers?&lt;/strong&gt;&lt;br&gt;
Some crawlers do more than one job.&lt;br&gt;
Cloudflare's documentation describes these as mixed-purpose crawlers.&lt;br&gt;
For example, a crawler may be used for search indexing while also being associated with other AI-related activities.&lt;/p&gt;

&lt;p&gt;Cloudflare has also added a Disallow AI Training option.&lt;br&gt;
This allows a website owner to express a preference against AI training while still allowing accountable mixed-use crawlers to index the website for search.&lt;br&gt;
So the exact setting chosen by the website owner matters.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What happens when your agent gets blocked?&lt;/strong&gt;&lt;br&gt;
A blocked request doesn't always mean the same thing.&lt;br&gt;
You might receive:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;403 Forbidden&lt;/li&gt;
&lt;li&gt;A Cloudflare challenge&lt;/li&gt;
&lt;li&gt;A rate-limit response&lt;/li&gt;
&lt;li&gt;Another error from the website
And you cannot always tell from the response whether the website intentionally blocked AI traffic or whether something else went wrong.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, in Spicrawl, an upstream bot challenge can appear as:&lt;br&gt;
ERR::UPSTREAM::CHALLENGE&lt;br&gt;
The target website's HTTP status can also be returned through the X-Target-Status header.&lt;br&gt;
This distinction matters.&lt;br&gt;
Your scraper's HTTP response is not always the same as the target site's actual response.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How should developers handle this?&lt;/strong&gt;&lt;br&gt;
The answer isn't to keep trying until the block disappears.&lt;br&gt;
If a website has clearly decided not to allow your automated traffic, treat that as a decision.&lt;br&gt;
Here are some better practices.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Check the site's rules&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Look at:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;robots.txt&lt;/li&gt;
&lt;li&gt;Terms of service&lt;/li&gt;
&lt;li&gt;API documentation&lt;/li&gt;
&lt;li&gt;Other access policies
These can tell you what the website allows.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;2. Use an official source when available&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If the website provides:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An API&lt;/li&gt;
&lt;li&gt;A feed&lt;/li&gt;
&lt;li&gt;A sitemap&lt;/li&gt;
&lt;li&gt;An official data source
use it instead of repeatedly fetching the website.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;3. Treat a deliberate block as a "no"&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If a website intentionally blocks automated access, don't try to work around the decision.&lt;br&gt;
Ask for permission or use another source.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Fetch only what you need&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Don't download an entire website when you only need one piece of information.&lt;br&gt;
For example, request the specific content you need and avoid unnecessary requests.&lt;br&gt;
This is better for both your application and the website you're accessing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Avoid repeated requests&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Caching can help prevent fetching the same page again and again.&lt;br&gt;
For example, Spicrawl's cache is enabled by default and can keep results for up to 48 hours.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Slow down when necessary&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If a website starts returning errors or rate limits, don't immediately send more requests.&lt;br&gt;
Reduce the request rate and back off.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Check the real target status&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you're using a scraping API, don't assume its HTTP status tells the whole story.&lt;br&gt;
Check whether the response contains the target site's actual status.&lt;br&gt;
What this means for AI developers&lt;br&gt;
The web is becoming a more controlled environment for AI agents.&lt;br&gt;
Search crawlers, AI training crawlers, and real-time AI agents are increasingly being treated as different types of traffic.&lt;br&gt;
That means an AI agent can't simply assume:&lt;br&gt;
"If a browser can open the page, my agent can open it too."&lt;/p&gt;

&lt;p&gt;Access can depend on the website's policies, infrastructure, bot controls, rate limits, and Cloudflare configuration.&lt;br&gt;
For developers, this means your web-access layer needs to handle failures properly and respect the decisions made by website owners.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What website owners can do&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you run a website behind Cloudflare, you now have more control over AI traffic.&lt;br&gt;
You can separately manage:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Search&lt;/li&gt;
&lt;li&gt;Agent&lt;/li&gt;
&lt;li&gt;Training
You can also use Cloudflare's Disallow AI Training option to express that preference while allowing certain mixed-use crawlers to continue indexing for search.
Cloudflare has also introduced Pay Per Use, currently in beta, for publishers who want to receive payment when AI products use their content.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;The bigger picture&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;AI agents need access to the web to be useful.&lt;br&gt;
At the same time, websites need control over how their content is accessed and used.&lt;br&gt;
Cloudflare's new AI traffic controls are part of that shift.&lt;br&gt;
For developers, the practical lesson is simple:&lt;br&gt;
Don't treat every blocked page as a technical problem to bypass. First understand why access was denied and whether you're allowed to access the content.&lt;br&gt;
As more websites define separate rules for search, AI agents, and AI training, responsible web access will become an important part of building reliable AI systems.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Originally published on Spicrawl&lt;/strong&gt;&lt;br&gt;
This article is adapted from the original: &lt;a href="https://spicrawl.com/blog/cloudflare-blocking-ai-crawlers" rel="noopener noreferrer"&gt;Cloudflare AI crawler blocking in 2026&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;About Spicrawl&lt;/strong&gt;&lt;br&gt;
Spicrawl is a web data platform for developers and AI agents, providing APIs for fetching web pages and returning clean data such as Markdown, JSON, HTML, and text.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>cloudflare</category>
      <category>webscraping</category>
    </item>
  </channel>
</rss>
