<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: PIT Toolkit</title>
    <description>The latest articles on DEV Community by PIT Toolkit (@pittoolkit).</description>
    <link>https://dev.to/pittoolkit</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4250459%2F7d6d1c73-3d29-48ce-a433-7249ff0e705f.png</url>
      <title>DEV Community: PIT Toolkit</title>
      <link>https://dev.to/pittoolkit</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/pittoolkit"/>
    <language>en</language>
    <item>
      <title>How to write an llms.txt for your side project</title>
      <dc:creator>PIT Toolkit</dc:creator>
      <pubDate>Tue, 17 Nov 2026 19:21:44 +0000</pubDate>
      <link>https://dev.to/pittoolkit/how-to-write-an-llmstxt-for-your-side-project-4j4b</link>
      <guid>https://dev.to/pittoolkit/how-to-write-an-llmstxt-for-your-side-project-4j4b</guid>
      <description>&lt;p&gt;An &lt;code&gt;llms.txt&lt;/code&gt; file is a short Markdown map of your site, published at &lt;code&gt;/llms.txt&lt;/code&gt; (or under a path such as &lt;code&gt;/docs/llms.txt&lt;/code&gt;). The idea, set out in the &lt;a href="https://llmstxt.org/" rel="noopener noreferrer"&gt;community proposal&lt;/a&gt;, is to give coding agents and chat assistants a concise place to start, instead of asking them to parse a whole HTML site through navigation, ads, and JavaScript.&lt;/p&gt;

&lt;p&gt;It is not an official web standard. Putting a file at &lt;code&gt;/llms.txt&lt;/code&gt; does not guarantee that any model, crawler, or answer engine will read it, cite you, or mention your project. Treat it as a courtesy file for agents that already choose to look, the same way &lt;code&gt;robots.txt&lt;/code&gt; is a courtesy file for crawlers that choose to honour it.&lt;/p&gt;

&lt;p&gt;That is still enough reason to write a careful one for a side project, especially if you have docs or an API that an agent might need mid-task.&lt;/p&gt;

&lt;h2&gt;
  
  
  The format, in plain terms
&lt;/h2&gt;

&lt;p&gt;The proposal uses Markdown on purpose, so a person and a parser can both read it. The shape is strict enough to be useful:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;An &lt;code&gt;H1&lt;/code&gt; with the project name. This is the only required section.&lt;/li&gt;
&lt;li&gt;A blockquote with a short summary — the one or two facts someone needs before the rest of the file makes sense.&lt;/li&gt;
&lt;li&gt;Optional notes as ordinary paragraphs or lists. No extra headings in this block.&lt;/li&gt;
&lt;li&gt;One or more &lt;code&gt;H2&lt;/code&gt; sections, each containing a list of links in the form &lt;code&gt;[name](url)&lt;/code&gt;, optionally followed by &lt;code&gt;: a short note&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;An &lt;code&gt;## Optional&lt;/code&gt; section, by convention, for links an agent can skip when context is tight.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A good file stays small. The detail lives behind the links. Where you can, those links should point at clean, text-friendly pages — Markdown versions of docs are ideal — not at a marketing page wrapped in a cookie banner.&lt;/p&gt;

&lt;h2&gt;
  
  
  A worked example
&lt;/h2&gt;

&lt;p&gt;Imagine a small tool called ShipLog: a CLI that turns pull requests into a changelog. A useful &lt;code&gt;/llms.txt&lt;/code&gt; is curated, not a dump of every URL in the sitemap:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;# ShipLog
&lt;/span&gt;
&lt;span class="ge"&gt;&amp;gt; ShipLog is a CLI and GitHub Action that turns merged pull requests into a human-readable changelog. It is not a commit-message linter and it does not publish releases for you.
&lt;/span&gt;
ShipLog reads PR titles, labels and breaking-change notes. Configuration lives in &lt;span class="sb"&gt;`.shiplog.yml`&lt;/span&gt; at the repo root.

&lt;span class="gu"&gt;## Docs
&lt;/span&gt;
&lt;span class="k"&gt;- &lt;/span&gt;[Quick start](https://shiplog.example.com/docs/quick-start.md): Install the CLI, generate a changelog from the last 20 merged PRs, and add the GitHub Action.
&lt;span class="k"&gt;- &lt;/span&gt;[Configuration](https://shiplog.example.com/docs/config.md): All &lt;span class="sb"&gt;`.shiplog.yml`&lt;/span&gt; keys, including label maps and ignore rules.
&lt;span class="k"&gt;- &lt;/span&gt;[GitHub Action](https://shiplog.example.com/docs/github-action.md): Workflow inputs, permissions, and a minimal YAML example.

&lt;span class="gu"&gt;## Optional
&lt;/span&gt;
&lt;span class="k"&gt;- &lt;/span&gt;[Changelog style guide](https://shiplog.example.com/docs/style.md): Voice and heading conventions if you want the output to match your docs.
&lt;span class="k"&gt;- &lt;/span&gt;[Pricing](https://shiplog.example.com/pricing): The CLI is free for public repos; the Action minutes pack is optional.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few editorial choices are doing the work here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The summary says what the tool is &lt;em&gt;and what it is not&lt;/em&gt;. That saves an agent a wrong assumption.&lt;/li&gt;
&lt;li&gt;Docs come first, in the order a new user would need them.&lt;/li&gt;
&lt;li&gt;Each link has a one-line note, so the agent can pick a page without fetching all of them.&lt;/li&gt;
&lt;li&gt;Pricing sits under &lt;strong&gt;Optional&lt;/strong&gt;. It matters, but it is not what you want loaded first when someone asks "how do I configure the label map?"&lt;/li&gt;
&lt;li&gt;The docs links point at &lt;code&gt;.md&lt;/code&gt; versions. If you do not yet serve Markdown, link the HTML pages, then add &lt;code&gt;.md&lt;/code&gt; mirrors later. The proposal suggests &lt;code&gt;page.md&lt;/code&gt; or &lt;code&gt;page.html.md&lt;/code&gt; next to the HTML, plus a &lt;code&gt;rel="alternate"&lt;/code&gt; link if you can.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;After you publish the file, do the test the proposal itself recommends: give an agent only your &lt;code&gt;llms.txt&lt;/code&gt; and ask it questions about the project. If it cannot find the install steps, or it invents a flag you do not have, the file needs a clearer link or a better note — not a longer sales paragraph.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start from your sitemap, then edit
&lt;/h2&gt;

&lt;p&gt;You do not have to write the first draft from a blank page. If you already have a &lt;code&gt;sitemap.xml&lt;/code&gt;, you can pull titles and meta descriptions into a Markdown list and then cut ruthlessly.&lt;/p&gt;

&lt;p&gt;A free generator that does exactly that first step is the &lt;a href="https://pittoolkit.com/app/llms-txt?src=devto" rel="noopener noreferrer"&gt;llms.txt Generator&lt;/a&gt;. Enter your site. It reads &lt;code&gt;/sitemap.xml&lt;/code&gt; when it can (or homepage links if there is no sitemap) and builds a draft from each page's title and meta description. The free run covers the homepage plus 8 pages. A one-time €9 Pro unlock covers the homepage plus 40 pages. Pages without a description are listed by title only, which is a useful hint: write those descriptions before you publish the file.&lt;/p&gt;

&lt;p&gt;What it will not do — and what no generator should claim to do — is curate the file for you, invent Markdown mirrors you have not published, or get your project mentioned by an AI assistant. You still download or copy the draft, edit it, and upload it as &lt;code&gt;/llms.txt&lt;/code&gt; on your own site.&lt;/p&gt;

&lt;p&gt;A raw draft often looks like a flat &lt;code&gt;## Pages&lt;/code&gt; list. That is fine as a starting point. Your edit pass should:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Drop URLs an agent does not need (tag archives, legal mirrors, thin marketing variants).&lt;/li&gt;
&lt;li&gt;Group the rest under headings such as &lt;strong&gt;Docs&lt;/strong&gt;, &lt;strong&gt;API&lt;/strong&gt;, and &lt;strong&gt;Optional&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Rewrite the summary in your own words, including the gotchas.&lt;/li&gt;
&lt;li&gt;Add the missing meta descriptions, or write the notes by hand.&lt;/li&gt;
&lt;li&gt;Point at &lt;code&gt;.md&lt;/code&gt; versions where they exist.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Put the file where an agent can actually fetch it
&lt;/h2&gt;

&lt;p&gt;Save the edited file at the site root, or at the path it describes (&lt;code&gt;/docs/llms.txt&lt;/code&gt; for documentation only). On a static host or in Next.js, that is usually a file in &lt;code&gt;public/llms.txt&lt;/code&gt;. Then confirm you are not accidentally serving an HTML 404 or a soft-404 "page not found" document with HTTP 200:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sI &lt;span class="s2"&gt;"https://yourproject.com/llms.txt"&lt;/span&gt;
curl -s &lt;span class="s2"&gt;"https://yourproject.com/llms.txt"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; head
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You want HTTP 200 and a &lt;code&gt;Content-Type&lt;/code&gt; of &lt;code&gt;text/plain&lt;/code&gt; or &lt;code&gt;text/markdown&lt;/code&gt;, not &lt;code&gt;text/html&lt;/code&gt;. If your host defaults unknown files to HTML, set the type explicitly. In a common Nginx snippet:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="na"&gt;location&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;llms&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;txt&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;default_type&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;plain&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;charset&lt;/span&gt; &lt;span class="n"&gt;utf&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or in a Cloudflare Worker or any &lt;code&gt;Response&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;llmsTxt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;"content-type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"text/plain; charset=utf-8"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you already have &lt;code&gt;robots.txt&lt;/code&gt; rules for AI crawlers, keep the two files' jobs separate. &lt;code&gt;robots.txt&lt;/code&gt; says who may crawl. &lt;code&gt;llms.txt&lt;/code&gt; says what is worth reading once someone is allowed in. A welcoming &lt;code&gt;llms.txt&lt;/code&gt; does nothing if a &lt;code&gt;Disallow&lt;/code&gt; blocks the agents you hoped would use it, and the reverse is also true: allowing a crawler is not the same as giving it a clear map.&lt;/p&gt;

&lt;p&gt;Write the file as if a careful human will read it in one sitting. That is the version most likely to help a model, too.&lt;/p&gt;

&lt;p&gt;I build PIT Toolkit.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>tutorial</category>
      <category>tooling</category>
    </item>
    <item>
      <title>"Discovered, currently not indexed": a developer's checklist</title>
      <dc:creator>PIT Toolkit</dc:creator>
      <pubDate>Tue, 17 Nov 2026 19:20:33 +0000</pubDate>
      <link>https://dev.to/pittoolkit/discovered-currently-not-indexed-a-developers-checklist-3mk0</link>
      <guid>https://dev.to/pittoolkit/discovered-currently-not-indexed-a-developers-checklist-3mk0</guid>
      <description>&lt;p&gt;You shipped a docs page, a changelog, or a new landing URL. A week later, Google Search Console still lists it under &lt;strong&gt;Discovered – currently not indexed&lt;/strong&gt;. The instinct is to open URL Inspection and click &lt;strong&gt;Request indexing&lt;/strong&gt; again. Do the technical work first: this status is usually about crawl priority, and it is not the same problem as &lt;strong&gt;Crawled – currently not indexed&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the status actually means
&lt;/h2&gt;

&lt;p&gt;Google has found the URL — from a link, a sitemap, or both — but has not crawled it yet. In URL Inspection the last crawl date is empty, because there is no crawl to date.&lt;/p&gt;

&lt;p&gt;Search Console's own explanation is narrower than most posts make it sound: Google typically wanted to crawl the URL, expected that doing so then would overload the site, and rescheduled the crawl. You will also see it on URLs that look low priority: thin or near-duplicate paths, pages with almost no internal links, or a site whose crawl time is spent on filter and archive URLs.&lt;/p&gt;

&lt;p&gt;This is not a penalty, and it is not evidence of a broken &lt;code&gt;noindex&lt;/code&gt; tag. Google does not promise to crawl or index every URL it knows about. A few discovered-but-not-crawled URLs on a small site can clear on their own. A long list of &lt;em&gt;important&lt;/em&gt; URLs is worth a systematic pass.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Crawled – currently not indexed&lt;/strong&gt; is a different verdict. Google already fetched the page and chose not to keep it, at least for now. That is usually about the page: thin content, a near-copy of something already indexed, or a page that is simply less useful than what already ranks. Requesting a recrawl of an unchanged page does not fix that.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Decide whether the URL should be in Google at all
&lt;/h2&gt;

&lt;p&gt;Before you debug anything, answer two questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Should this exact URL appear in search results?&lt;/li&gt;
&lt;li&gt;Is there a better URL for the same content?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Thank-you pages, login screens, staging leftovers, faceted filter URLs, and print or tracking-parameter copies should usually stay out. If the content already has a canonical URL, point the sitemap, the internal links, and the canonical tag there, and stop spending time on the duplicates.&lt;/p&gt;

&lt;p&gt;Google's &lt;a href="https://developers.google.com/crawling/docs/crawl-budget" rel="noopener noreferrer"&gt;crawl budget guidance&lt;/a&gt; is aimed at large or rapidly changing sites, and at sites where many URLs sit in this bucket. The useful idea still applies on a side project: do not offer Google a large inventory of URLs you do not want crawled.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Rule out hard technical blockers
&lt;/h2&gt;

&lt;p&gt;A leftover &lt;code&gt;noindex&lt;/code&gt;, a &lt;code&gt;robots.txt&lt;/code&gt; &lt;code&gt;Disallow&lt;/code&gt;, a 404, a 5xx, or a canonical that points elsewhere will beat any indexing request. Check the &lt;em&gt;live&lt;/em&gt; URL, including &lt;code&gt;https&lt;/code&gt; and the trailing slash your site uses.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Follow redirects and print response headers&lt;/span&gt;
curl -sI -L &lt;span class="s2"&gt;"https://example.com/docs/install"&lt;/span&gt;

&lt;span class="c"&gt;# robots.txt at the site root — a leftover "Disallow: /" from staging is a classic&lt;/span&gt;
curl -s &lt;span class="s2"&gt;"https://example.com/robots.txt"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What to look for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;HTTP status.&lt;/strong&gt; Google cannot treat a 4xx or 5xx as useful content. If there is a redirect, the &lt;em&gt;final&lt;/em&gt; URL is the one that should be in your sitemap and in internal links.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;X-Robots-Tag&lt;/code&gt; and &lt;code&gt;&amp;lt;meta name="robots"&amp;gt;&lt;/code&gt;.&lt;/strong&gt; A &lt;code&gt;noindex&lt;/code&gt; (or &lt;code&gt;none&lt;/code&gt;) is a hard stop for crawlers that honour it. Common sources: a WordPress "Discourage search engines" setting, an SEO plugin's per-page flag, or staging headers copied to production.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;robots.txt&lt;/code&gt;.&lt;/strong&gt; A &lt;code&gt;Disallow&lt;/code&gt; that matches this path stops Googlebot from crawling it. Remember the awkward case: to &lt;em&gt;remove&lt;/em&gt; a URL from the index you usually need to allow crawling and use &lt;code&gt;noindex&lt;/code&gt;, because Google has to fetch the page to see the tag.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Canonical.&lt;/strong&gt; If the tag points at a different URL, Google may index that URL instead of this one. Make the canonical, the internal links, and the sitemap agree.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A short Node script for the HTML signals:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// check-index-signals.mjs&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Usage: node check-index-signals.mjs https://example.com/page"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;redirect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"follow"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;pick&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;requested&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;finalUrl&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;xRobots&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"x-robots-tag"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;^&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;*&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;\&lt;span class="n"&gt;s&lt;/span&gt;\&lt;span class="no"&gt;S&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;*?&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;\&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;/&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;\&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;+/&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;robotsMeta&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;pick&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;^&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"']robots["&lt;/span&gt;&lt;span class="s1"&gt;'][^&amp;gt;]*&amp;gt;/i),
  canonical: pick(/&amp;lt;link[^&amp;gt;]+rel=["'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="n"&gt;canonical&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"'][^&amp;gt;]*&amp;gt;/i),
});
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If this pass is clean, the status is probably not a broken tag.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Make the sitemap tell one story
&lt;/h2&gt;

&lt;p&gt;A sitemap helps discovery; it does not guarantee indexing. For a URL that is already "Discovered", Google already knows it exists. The sitemap still matters because a messy one wastes crawl time and sends mixed signals. Keep it to canonical, indexable URLs that return HTTP 200. If the live page redirects, list the final URL. If you use a sitemap index, make sure the child file that lists this URL is reachable and submitted.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Quick look in a single sitemap file (follow child files yourself if this is a sitemap index)&lt;/span&gt;
curl -s &lt;span class="s2"&gt;"https://example.com/sitemap.xml"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -F &lt;span class="s2"&gt;"https://example.com/docs/install"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Also check that &lt;code&gt;robots.txt&lt;/code&gt; points at the sitemap you maintain (&lt;code&gt;Sitemap: https://example.com/sitemap.xml&lt;/code&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Raise the priority of the pages that matter
&lt;/h2&gt;

&lt;p&gt;If the important URL is technically fine and still sits in "Discovered", treat it as a priority problem:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Add a descriptive internal link from a page Google already crawls — the homepage, a docs index, or a related article — not only a footer or a sitemap entry.&lt;/li&gt;
&lt;li&gt;Cut or &lt;code&gt;noindex&lt;/code&gt; large sets of thin URLs (empty tags, filters, calendars) so they stop competing. For URLs you never want crawled at all, &lt;code&gt;robots.txt&lt;/code&gt; is the cheaper signal; &lt;code&gt;noindex&lt;/code&gt; still requires a fetch.&lt;/li&gt;
&lt;li&gt;Check Search Console crawl stats for server errors and slow responses. Unhealthy hosts get crawled less.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then request indexing &lt;strong&gt;once&lt;/strong&gt; for a few key URLs, after a real change. Google's recrawl documentation is blunt: there is a quota, &lt;a href="https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl" rel="noopener noreferrer"&gt;requesting the same URL multiple times does not get it crawled any faster&lt;/a&gt;, crawling can take days to weeks, and a request does not guarantee inclusion in results.&lt;/p&gt;

&lt;h2&gt;
  
  
  A free technical pass, and the rest of the triage
&lt;/h2&gt;

&lt;p&gt;To run those header and HTML checks in one go, the free &lt;a href="https://pittoolkit.com/app/index-check?src=devto" rel="noopener noreferrer"&gt;Not Indexed Checker&lt;/a&gt; reads the live page, &lt;code&gt;robots.txt&lt;/code&gt;, and sitemap the way a public crawler would. It looks for a non-success status, &lt;code&gt;noindex&lt;/code&gt;, a &lt;code&gt;robots.txt&lt;/code&gt; rule that blocks Googlebot, a canonical that points elsewhere, or a missing sitemap entry (a large, split sitemap may not be fully opened).&lt;/p&gt;

&lt;p&gt;It cannot log into Search Console, and it will not tell you whether Google has already indexed the page. Only Search Console can. A clean result is not a promise that the page will appear in Google. "Discovered – currently not indexed" and "Crawled – currently not indexed" often will not show up there at all, because they are judgements, not broken tags.&lt;/p&gt;

&lt;p&gt;If you want the common Search Console reasons — including this one — explained in order, with a short triage and a weekly check, the &lt;a href="https://pittoolkit.com/p/not-indexed-guide?src=devto" rel="noopener noreferrer"&gt;Not Indexed Fix Guide&lt;/a&gt; is a four-page PDF. The introductory price is €4.50 until 1 December 2026 (normally €9), one-time, instant download. It does not replace Search Console, and it does not get a page indexed for you.&lt;/p&gt;

&lt;p&gt;I build PIT Toolkit.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
      <category>tooling</category>
    </item>
  </channel>
</rss>
