DEV Community: Anosh

Structured Data: How Schema Markup Actually Helps Search Engines Understand Pages

Anosh — Tue, 06 Oct 2026 19:11:00 +0000

"Add schema to get rich snippets" is the most common way structured data gets explained, and it's backwards. Rich results are a possible side effect. The real job of schema is simpler: it helps a machine understand what a page is about.

Search engines are good at reading text, but text is ambiguous. Is "Apple" a fruit or a company? Is "Jordan" a country or a person? Is "$49" a price, a fee or a random number? Schema removes that guesswork.

The mental model

Think of it as a pipeline:

Page content
    ↓
Structured entities (things)
    ↓
Machine-readable relationships
    ↓
Search engine understanding

Page content: the words, images and numbers a human reads
Entities: the distinct things the page is about, like a company, a person, a product or an event
Relationships: how those things connect (this article was written by this person, who works for this organization)
Understanding: the search engine can place your page in its knowledge of the world, instead of guessing from keywords

Schema.org and JSON-LD

These two get mixed up constantly, but they're different things.

Schema.org is the shared vocabulary. It defines types (Product, Event) and properties (price, startDate). Think of it as the dictionary.
JSON-LD is the format you write it in. Think of it as the sentence.

Google recommends JSON-LD because it lives in a separate <script> block, so it doesn't tangle with your visible HTML:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Co",
  "url": "https://example.com",
  "logo": "https://example.com/logo.png",
  "sameAs": [
    "https://www.linkedin.com/company/example"
  ]
}
</script>

A few core ideas:

@context says which vocabulary you're using
@type says what kind of thing this is
Properties describe it
@id lets you give an entity a stable identifier, so other blocks can refer to it

The types you'll actually use

Organization

Describes the business or brand behind the site
Key properties: name, url, logo, sameAs (links to official profiles)
It tells search engines which entity your site belongs to, and connects your profiles together

Person

Describes an individual, such as an author or founder
Key properties: name, url, jobTitle, sameAs
Helpful for tying content to a real author

WebSite

Describes the site as a whole
Key properties: name, url
Helps clarify your site name. Google retired the old sitelinks search box feature, so don't add it expecting that result.

Article

Describes a blog post or news piece
Key properties: headline, author, datePublished, dateModified, image
Tells the engine who wrote it and when, and when it was last updated

Product

Describes something for sale
Key properties: name, image, description, offers (price, currency, availability), aggregateRating
Turns a product page into clean, comparable data: what it is, what it costs, whether it's in stock

BreadcrumbList

Describes where a page sits in your site's hierarchy
Key properties: a list of items with position, name and item (URL)
Shows how your content is organized, which helps both understanding and navigation

LocalBusiness

Describes a business with a physical presence
Key properties: name, address, telephone, openingHours, geo
Connects the page to a real-world location, which supports local search

FAQPage

Describes a page of questions and answers
Key properties: mainEntity with Question and acceptedAnswer
Marks up the question-and-answer structure clearly. Note that Google now shows FAQ rich results only for a small set of well-known authoritative sites, so the value for most sites is clarity, not appearance.

Review

Describes a review of something, including the rating and reviewer
Key properties: reviewRating, author, itemReviewed
Make sure the review is about something real and visible on the page. Self-serving reviews of your own business, placed on your own site, aren't eligible for review stars.

Event

Describes something happening at a specific time and place
Key properties: name, startDate, location, eventAttendanceMode, offers
Gives the engine exact dates, venues and ticket info, which is hard to pull reliably from prose

Where relationships come in

Single blocks are useful, but the real power is connecting entities. Here, an article points to its author and publisher using @id:

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://example.com/#org",
      "name": "Example Co",
      "url": "https://example.com"
    },
    {
      "@type": "Person",
      "@id": "https://example.com/#author",
      "name": "Jane Doe",
      "worksFor": { "@id": "https://example.com/#org" }
    },
    {
      "@type": "Article",
      "headline": "How Schema Works",
      "author": { "@id": "https://example.com/#author" },
      "publisher": { "@id": "https://example.com/#org" },
      "datePublished": "2026-10-01"
    }
  ]
}

Now the engine sees a small graph: an article, written by a person, who works for an organization. That's understanding, not just labelling.

Valid schema does not equal a guaranteed rich result

This is where expectations go wrong. Valid markup means your code follows the rules. It doesn't mean Google will do anything visible with it.

Reasons valid schema may produce no rich result:

Eligibility. Not every type has a rich result, and some have been restricted or retired.
Quality and trust. Google decides whether your page and site deserve the enhancement.
Content mismatch. Markup must reflect what users can see on the page. Marking up hidden or invented content breaks guidelines and can lead to a manual action.
Missing required properties. The markup might validate as schema but lack what a specific rich result needs.
Google's discretion. Even eligible pages don't always get the feature on every search.

So treat the two things separately:

Valid: passes syntax and vocabulary checks
Eligible: meets Google's documented requirements for a given feature
Displayed: Google chooses to show it, sometimes

Your control stops at the first two.

Common mistakes

Marking up content that isn't visible on the page
Copying a template without changing the values
Duplicate or conflicting blocks from both a plugin and the theme
Adding aggregateRating with invented numbers
Using the wrong type because it "gets stars"
Treating schema as a ranking factor. It helps understanding. It isn't a shortcut to higher rankings.
Setting it up once and never updating prices, dates or availability

How to test it

Run the page through the Rich Results Test to see which features it's eligible for
Use the Schema Markup Validator to check general schema.org syntax
Watch the Enhancements reports in Search Console for errors and warnings
Compare the markup against the visible page and ask whether they say the same thing

Final thoughts

Schema is a translation layer between how humans read a page and how machines parse it. Done well, it makes your pages clearer, your entities more consistent and your content easier to connect to the wider web of information. Sometimes that earns a rich result, and sometimes it simply means a search engine understood you correctly.

Start with the basics, which are Organization, WebSite, Article or Product depending on the page, and BreadcrumbList. Keep it accurate and keep it honest.

What's the schema type that gave you the most trouble? Tell me in the comments.

Suggested tags: seo, webdev, json, beginners

Visit my website: anoshbb.com

The Complete Guide to HTTP Status Codes for SEO

Anosh — Tue, 06 Oct 2026 19:06:32 +0000

Every time Googlebot requests a URL, your server answers with a three-digit number before anything else happens. That number decides whether the page gets indexed, replaced, dropped or crawled less often.

This is a reference you can come back to. For each code, you get what it means, the SEO implication, when to use it, and the mistake people commonly make.

2xx: Success

200 OK

What it means: The request worked and the content was returned.

SEO implication: This is the only status that makes a page eligible for indexing. Google still decides whether it's worth keeping.

Use it when: The page exists and should be seen by users and crawlers.

Common mistake: Returning 200 for pages that are really errors, like empty search results or "product not found" pages. Google calls these soft 404s and treats them as missing anyway.

3xx: Redirects

301 Moved Permanently

What it means: The URL has changed for good. Go to the new one.

SEO implication: Google treats the destination as the new canonical home. Ranking signals pass through, and the old URL eventually drops out of the index.

Use it when:

A URL changes
You migrate to a new domain
You move from HTTP to HTTPS
You restructure your URL paths
You merge duplicate pages

Common mistake: Redirect chains (A → B → C → D) and redirecting a batch of old URLs to the homepage. Redirect each page to its closest equivalent, in a single hop.

302 Found

What it means: The URL has moved temporarily. Keep using the original.

SEO implication: Google generally keeps the original URL indexed. If a 302 stays in place for a long time, Google may start treating it like a permanent move anyway.

Use it when:

A/B testing
Short-term promotions or seasonal pages
Sending users to a regional or device-specific version for a while

Common mistake: Using 302 for a permanent change because it's the default in many frameworks and plugins. Check what your redirect actually returns.

303 See Other

What it means: The response is somewhere else, and the client should fetch it with a GET request.

SEO implication: Rarely relevant for crawled pages. Google treats it as a temporary redirect.

Use it when: A form submission or POST request should send the user to a confirmation page, so refreshing doesn't resubmit the form.

Common mistake: Using 303 for regular page moves. It's meant for the post-submission flow, not for content that changed address.

307 Temporary Redirect

What it means: A temporary redirect that preserves the HTTP method. A POST stays a POST.

SEO implication: Treated like a 302 for search. Google keeps the original URL.

Use it when: The move is temporary and the request method matters, such as in APIs.

Common mistake: Assuming a 307 means "permanent but safer." You'll often see 307s when a site has HSTS, where the browser itself redirects HTTP to HTTPS. That's a browser behavior, not what your server sent.

308 Permanent Redirect

What it means: A permanent redirect that preserves the HTTP method.

SEO implication: Treated like a 301. Signals consolidate on the new URL.

Use it when: The move is permanent and you want the method preserved.

Common mistake: Thinking 308 is a different SEO outcome from 301. For regular page navigation they behave the same.

301 vs 302 vs 307 (and 308)

301: permanent, new URL gets indexed, old one fades out
302: temporary, original URL usually stays indexed
307: temporary like 302, but the HTTP method is preserved
308: permanent like 301, but the HTTP method is preserved

The practical rule is simple. Ask yourself whether the old URL is coming back. If no, use 301 (or 308). If yes, use 302 (or 307). Both kinds pass signals, so the old line that "302s don't pass PageRank" doesn't hold up. What differs is which URL Google chooses to keep in the index.

4xx: Client errors

404 Not Found

What it means: Nothing is here, and the server doesn't say whether it will return.

SEO implication: The URL drops out of the index over time. A handful of 404s is normal and doesn't hurt your site overall.

Use it when: A page doesn't exist, or was removed with no replacement.

Common mistake: Panicking over every 404 in Search Console. Fix the ones that have backlinks, internal links or real traffic, and ignore the junk.

410 Gone

What it means: The page existed, was removed on purpose, and isn't coming back.

SEO implication: A slightly stronger signal than 404. Google may drop the URL a little faster, though the difference is small.

Use it when: You deliberately delete content permanently, like expired listings or retired campaigns.

Common mistake: Expecting 410 to give a big SEO advantage. It's a clean way to say "gone," not a ranking trick.

429 Too Many Requests

What it means: The client is sending requests too quickly. Slow down.

SEO implication: Google treats 429 like a server error and reduces its crawl rate. If it persists, indexing can suffer.

Use it when: You rate-limit abusive bots or scrapers.

Common mistake: Rate-limiting so aggressively that Googlebot gets caught, usually through a security plugin or firewall. Check server logs if crawl stats suddenly drop.

5xx: Server errors

500 Internal Server Error

What it means: Something broke on the server, and it can't say what.

SEO implication: Google retries and slows its crawling. If the error continues for days, URLs can start falling out of the index.

Use it when: You shouldn't be using it on purpose. It means a bug.

Common mistake: Ignoring intermittent 500s because "the site loads for me." Check Search Console's crawl stats and your logs.

502 Bad Gateway

What it means: A server acting as a gateway or proxy got an invalid response from the server behind it.

SEO implication: Same as other 5xx errors: crawl slowdown, and de-indexing if prolonged.

Use it when: It's generated by infrastructure, not chosen.

Common mistake: Blaming the application when the issue is the proxy, load balancer or CDN configuration.

503 Service Unavailable

What it means: The server is temporarily unable to handle the request.

SEO implication: This is the right code for planned downtime. Google understands it's temporary and will come back, especially when a Retry-After header is present.

Use it when:

Maintenance windows
Planned deployments
Short-term overload

Common mistake: Showing a "we'll be back soon" page with a 200 status. Google may index the maintenance page. Return 503 instead. And don't leave it up for days.

504 Gateway Timeout

What it means: The gateway waited too long for the upstream server.

SEO implication: Treated as a server error. Slow backends lead to repeated timeouts and reduced crawling.

Use it when: Again, infrastructure produces it, not you.

Common mistake: Treating it as a one-off. Repeated 504s usually point to slow database queries or overloaded servers that also hurt your Core Web Vitals.

Quick reference

200: indexable, but confirm it isn't a soft 404
301 / 308: permanent move, signals consolidate on the new URL
302 / 307: temporary move, original URL usually stays
303: post-form redirect, rarely an SEO concern
404: not found, drops out over time
410: deliberately gone, drops out slightly faster
429: slow down, Google reduces crawling
500 / 502 / 504: server trouble, fix fast
503: temporary downtime, the correct code for maintenance

How to check your own status codes

curl -I https://example.com/old-page

Look at the first line for the status and the location header for redirect targets. To follow a whole chain:

curl -IL https://example.com/old-page

Also use Search Console's Page indexing and Crawl stats reports, and a crawler like Screaming Frog to catch chains, loops and soft 404s across the whole site.

Final thoughts

Status codes are the cheapest part of technical SEO to get right and one of the costliest to get wrong. A wrong redirect type, a maintenance page returning 200, or a firewall quietly serving 429s to Googlebot can undo months of content work.

Pick the code that matches what's really happening, and keep your signals consistent: the redirect, the canonical, the sitemap and your internal links should all agree.

Which status code has caused you the most confusion? Tell me in the comments.

Also do visit my website : anoshbb.com

Suggested tags: seo, webdev, http, beginners

Core Web Vitals: What They Actually Mean Technically

Anosh — Tue, 06 Oct 2026 19:02:02 +0000

"Core Web Vitals are important for SEO" is the kind of sentence that's true and useless at the same time. It tells you to care, but not what to open in your editor on Monday morning.

So let's skip the pep talk and look at what each metric measures, what causes it to go bad, and what a developer can actually change.

The three metrics, quickly

Core Web Vitals are three field metrics, measured on real users and judged at the 75th percentile of page loads:

LCP (Largest Contentful Paint): how long until the biggest visible element renders. Good is 2.5 seconds or less.
INP (Interaction to Next Paint): how quickly the page responds visually after a user interaction. Good is 200 milliseconds or less. It replaced FID in 2024.
CLS (Cumulative Layout Shift): how much visible content jumps around unexpectedly. Good is 0.1 or less.

One thing worth saying early: these are field numbers. Lighthouse in your own browser is a lab test on your fast laptop. Real users on mid-range phones and patchy connections are the ones Google measures, so use PageSpeed Insights (the CrUX data at the top) or Search Console to see what they experience.

LCP: why the biggest thing loads late

LCP is usually a hero image, a large heading, or a big text block. The element is whatever the browser decides is largest in the viewport.

The useful mental model is that LCP is a chain of four phases:

Time to First Byte (TTFB): how long the server takes to respond
Resource load delay: how long before the browser starts fetching the LCP resource
Resource load duration: how long the download takes
Element render delay: how long between "downloaded" and "painted"

When LCP is bad, one of those phases is almost always the culprit. Find which one before changing anything.

What causes slow LCP

Slow server response. If TTFB is already 1.5 seconds, you've spent most of your budget before the browser sees any HTML. Common causes are no caching, slow database queries, distant servers and no CDN.
Hero images that are too heavy. A 2 MB unoptimized JPEG as your banner is the classic offender.
Hero images discovered late. If the image is a CSS background or gets injected by JavaScript, the browser can't find it until much later.
Lazy-loading the LCP image. loading="lazy" on your hero tells the browser to deprioritize the exact thing it should hurry up.
Render-blocking CSS. The browser won't paint until it has the stylesheets in the <head>. A giant CSS bundle delays everything.
Render-blocking JavaScript. Synchronous scripts in the head pause HTML parsing.
Web fonts. If your LCP is a headline, it may sit invisible while the font file downloads.
Client-side rendering. If the page is an empty <div id="root"> until a JS bundle runs, the LCP element doesn't exist until all that work finishes.

What you can change

Fix the server first, if TTFB is the problem:

Cache HTML where you can (page caching, edge caching)
Put a CDN in front of the site
Optimize slow queries and heavy backend work
Avoid redirect chains before the page even loads

Make the hero image fast and discoverable:

<img
  src="/hero.avif"
  width="1200"
  height="600"
  fetchpriority="high"
  alt="Product hero image"
/>

Serve modern formats like WebP or AVIF
Use responsive images (srcset and sizes) so phones don't download desktop-sized files
Add fetchpriority="high" to the LCP image
Never lazy-load it
Keep it in the HTML as a real <img>, not a CSS background

If it must be discovered early and can't be in plain HTML, preload it:

<link rel="preload" as="image" href="/hero.avif" fetchpriority="high">

Deal with render-blocking resources:

Inline the small amount of critical CSS needed for above-the-fold content
Load the rest of the CSS without blocking
Add defer (or async, where order doesn't matter) to scripts
Remove unused CSS and JS instead of just minifying them

Handle fonts sensibly:

Use font-display: swap so text shows immediately in a fallback
Preload your one or two critical font files
Self-host fonts instead of waiting on a third-party domain
Subset fonts so you're not shipping characters you never use

Consider rendering strategy. If you're on a JS framework, server-side rendering or static generation gets real HTML to the browser faster than shipping an empty shell.

INP: why the page feels sluggish

INP measures the delay between a click, tap or keypress and the next frame the browser paints in response. It looks at interactions across the whole visit and reports roughly the worst one, which is why one bad interaction can hurt you.

Each interaction has three parts:

Input delay: the wait before the handler can even start (the main thread is busy)
Processing time: how long your event handlers run
Presentation delay: the time to recalculate layout and paint the result

What causes poor INP

Long tasks. Any task that holds the main thread for more than 50 ms can block the browser from responding. Several in a row make a page feel frozen.
Main-thread blocking in general. The main thread handles JavaScript, style calculation, layout and painting. If JS hogs it, nothing else happens.
Heavy event handlers. A click handler that filters 5,000 items, updates a huge state tree and re-renders half the page will feel slow.
Third-party scripts. Chat widgets, tag managers, analytics, A/B testing tools and ad scripts all compete for the same main thread, and you don't control their code.
Large DOM size. Updating a page with tens of thousands of nodes costs more at every step.
Expensive rendering work. Forcing layout repeatedly (layout thrashing) or re-rendering too much UI after each interaction.

What you can change

Break up long tasks. Yield back to the main thread so the browser can respond to the user between chunks of work:

async function processItems(items) {
  for (const item of items) {
    heavyWork(item);
    // Let the browser handle pending input and paint
    await new Promise(resolve => setTimeout(resolve, 0));
  }
}

Newer APIs like scheduler.yield() do this more cleanly where supported, but the idea is the same: don't hold the thread for 300 ms in one go.

Keep event handlers light:

Do the minimum needed to show visual feedback first, then defer the heavy part
Debounce or throttle handlers for input, scroll and resize
Avoid synchronous work you don't need before the next paint

Move heavy work off the main thread. Web Workers are good for parsing, calculations and data crunching that doesn't touch the DOM.

Cut down the JavaScript you ship:

Code-split so each page loads only what it needs
Remove unused libraries (that date library you imported for one function)
Lazy-load features that appear only after interaction

Tame third-party scripts:

Audit every one and delete what nobody uses
Load non-essential scripts after the page is interactive, or on user interaction
Prefer lighter alternatives where they exist
Measure the impact of each one, because the cost is often a surprise

Reduce rendering cost:

Keep the DOM small and avoid deeply nested structures
Virtualize very long lists
Batch DOM reads and writes to avoid layout thrashing
In React-style frameworks, avoid unnecessary re-renders with sensible state placement and memoization

CLS: why things jump around

CLS adds up unexpected layout shifts. A shift happens when a visible element moves between frames without the user causing it. The score combines how much of the viewport moved and how far it moved.

The painful version is familiar: you go to tap a button, an ad loads above it, everything slides down, and you tap the wrong thing.

What causes layout shifts

Images and videos without dimensions. The browser doesn't know how much space to reserve, so content below gets pushed when the media loads.
Ads and embeds. Ad slots that resize after loading, or iframes (maps, videos, social posts) with no reserved space.
Dynamic content. Banners, cookie notices, "related items" blocks or personalized content inserted above existing content.
Web fonts. When the fallback font swaps to the real one, text can reflow if the two have different metrics.
Injected elements. Anything added to the DOM above content the user is already looking at, like a promo bar sliding in after load.
Late-loading CSS. Styles that arrive after first paint can change the layout.

Note that shifts caused directly by user input (like clicking to expand an accordion) are excluded. It's the unexpected ones that count.

What you can change

Always give media dimensions:

<img src="/photo.webp" width="800" height="450" alt="..." />

The browser uses the width and height to calculate the aspect ratio and reserve space before the file arrives. You can also set aspect-ratio in CSS:

.video-wrapper {
  aspect-ratio: 16 / 9;
}

Reserve space for ads and embeds:

Give ad slots a min-height that matches the most common ad size
Put placeholders around iframes and embedded widgets
Avoid inserting ads above content that's already in view

Be careful with dynamic content:

Insert new content below the viewport or in space you've already reserved
For banners and cookie notices, use fixed or overlay positioning, or reserve their height from the start
Show skeleton placeholders that match the final size of the content

Reduce font-swap shifts:

Preload critical fonts
Use font-display: optional or swap depending on how much a brief flash matters to you
Match the fallback font's metrics to the web font with size-adjust, ascent-override and similar descriptors, so the swap barely changes the layout

Animate with transforms. Animations that change top, height or margin trigger layout and can cause shifts. transform and opacity don't.

How to find what's actually wrong

Fixing things blindly is how teams waste a sprint. A better workflow:

Start with field data. Check Search Console's Core Web Vitals report and the CrUX section in PageSpeed Insights to see which metric fails and on which pages.
Reproduce in the lab. Use Lighthouse and the Chrome DevTools Performance panel, ideally with CPU and network throttling to mimic a mid-range phone.
Find the specific cause.
- For LCP, look at which element it is and which of the four phases is longest
- For INP, record an interaction in the Performance panel and look for long tasks
- For CLS, use the Layout Shift regions in DevTools to see what moved
Fix one thing, then measure again.
Wait for field data. CrUX uses a rolling 28-day window, so improvements take a few weeks to show in your real-user numbers.

Quick reference

LCP too high: slow server, heavy or late-discovered hero image, render-blocking CSS and JS, fonts, client-side rendering. Fix with caching, image optimization, fetchpriority, preload, deferred scripts and SSR.
INP too high: long tasks, heavy handlers, third-party scripts, big DOM. Fix by splitting work, using workers, trimming JS and auditing third parties.
CLS too high: media without dimensions, ads, injected content, font swaps. Fix with explicit sizes, reserved space, stable fonts and transform-based animation.

Final thoughts

Core Web Vitals look intimidating until you notice that each one is really a simple complaint from a real user:

LCP: "this is taking forever to show up"
INP: "I tapped it and nothing happened"
CLS: "everything moved right as I tried to tap"

Every technical fix above comes from taking one of those complaints seriously. And it's worth remembering that these metrics are one signal among many. A fast page won't rescue weak content, but a slow, jumpy one makes everything else harder.

If you've got a Core Web Vitals problem that refuses to go away, share it in the comments. The stubborn ones are usually the most educational.

Also check out my website : anoshbb.com

Suggested tags: webdev, performance, seo, javascript

Robots.txt Deep Dive: What It Can and Cannot Do

Anosh — Tue, 06 Oct 2026 18:53:46 +0000

Robots.txt is one of the smallest files on your site and one of the most misunderstood. It's a plain text file, and a single wrong line can hide your whole site from search engines. Even more often, people expect it to do something it was never built for.

Let's sort out what it actually does.

What robots.txt controls

Robots.txt controls crawling. That's all. It tells compliant bots which URLs they may or may not request.

It lives at the root of a host: https://example.com/robots.txt
Each host, protocol and port needs its own file (blog.example.com doesn't inherit from example.com)
It's a polite request, not a lock. Reputable crawlers like Googlebot and Bingbot follow it. Malicious bots ignore it
It's public. Anyone can read it, so never use it to hide private pages

It does not control indexing, and it does not provide security.

The basic syntax

A robots.txt file is made of groups. Each group starts with a user-agent and is followed by rules.

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

User-agent

This names the bot a group applies to.

* means all bots
Googlebot, Bingbot and other names target specific crawlers
A bot follows only the most specific group that matches it. It does not combine that group with the * group

That last point trips people up. If you write a Googlebot group, Googlebot ignores the * group completely, so repeat any shared rules inside it.

Disallow

Blocks a path from being crawled.

Disallow: /private/ blocks everything under that folder
Disallow: / blocks the entire site
Disallow: (empty) blocks nothing

Allow

Makes an exception inside a blocked area.

Useful for opening one subfolder inside a blocked directory
When rules conflict, Google follows the most specific (longest) match. If it's a tie, the less restrictive rule wins

Sitemap

Points crawlers to your XML sitemap.

Use a full absolute URL
It isn't tied to any user-agent group, so it can sit anywhere in the file
You can list more than one

Wildcards

Google and Bing support two special characters:

* matches any sequence of characters
$ marks the end of a URL

User-agent: *
Disallow: /*.pdf$
Disallow: /search*

The first rule blocks URLs ending in .pdf. Without the $, it would also block something like /file.pdf?download=1.

Directory blocking

Be careful with trailing slashes. They change the meaning.

Disallow: /blog/ blocks the folder and everything inside it
Disallow: /blog blocks anything that starts with /blog, including /blog-news and /blogger-tools

Paths are also case-sensitive. /Admin/ and /admin/ are different things.

Query parameters

Parameters can create thousands of near-identical URLs: sorting, filters, session IDs, tracking tags. Robots.txt is a common way to stop crawlers from wasting time on them.

User-agent: *
Disallow: /*?sort=
Disallow: /*?*sessionid=

A few cautions:

Block only parameters that truly produce duplicate or useless pages
If a parameter changes the actual content (like ?page=2 on a real listing), blocking it can hide products or articles from crawlers
For duplicates you want consolidated rather than ignored, a canonical tag is often the better tool

Bot-specific rules

You can give different bots different instructions.

User-agent: Googlebot
Disallow: /internal-search/

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

This is how people opt out of certain AI crawlers while staying open to search engines
Remember that each bot reads only its own group
Crawl-delay is ignored by Google. Bing supports it. Don't rely on it for Googlebot

Why robots.txt is not an indexing directive

This is the big misunderstanding.

Disallow stops Google from fetching a page
It doesn't stop Google from knowing the URL exists
If other pages link to a blocked URL, Google can still index the bare address and show it in results with no description

So a blocked page can still appear in search. It just appears without content, because Google was never allowed to read it.

Google also no longer supports a noindex line inside robots.txt. Any guide telling you to write Noindex: /page/ in the file is out of date.

Blocking crawling vs noindex

These two tools solve different problems.

Disallow in robots.txt: "Don't fetch this URL"
noindex (meta tag or X-Robots-Tag header): "You may fetch this, but don't keep it in the index"

<meta name="robots" content="noindex">

The trap is combining them. If you block a page in robots.txt and add a noindex tag, Google never fetches the page, so it never sees the noindex. The URL can stay indexed for a long time.

To remove a page from search properly:

Let Google crawl it (no Disallow)
Serve a noindex tag or header
Wait for Google to recrawl and drop it
Only then consider blocking it, if you still want to

For pages that are truly private, use authentication. Neither robots.txt nor noindex is real protection.

Common syntax mistakes

Leaving Disallow: / live after launch. The classic staging-site disaster. Check it after every migration
Blocking CSS and JavaScript. Google needs these to render your pages properly
Wrong case or missing slash. /Blog/ won't match /blog/
Putting rules before any User-agent line. Rules must belong to a group
Relative sitemap URLs. Use the full https:// address
Using robots.txt to hide sensitive content. The file is public, so it can actually point people to what you want hidden
Assuming * rules apply to every group. They don't, as covered above
Wrong location or format. It must sit at the root, be plain text, and return a 200 status. If it returns a server error for a long time, Google may treat the site cautiously

A sensible starter file

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sessionid=

Sitemap: https://example.com/sitemap.xml

Short, clear and easy to audit. Add rules only when you have a reason, and write a comment (# reason) so future you remembers why.

How to test it

Use the robots.txt report in Google Search Console to see how Google reads your file
Run the URL Inspection tool on important pages and confirm they aren't blocked
Open /robots.txt in your browser after every deployment

Final thoughts

Think of robots.txt as a traffic sign for crawlers, not a vault door and not a deletion button. It manages where bots spend their time. If you want something kept out of search results, use noindex. If you want something truly private, use a login.

Have you ever found a rogue Disallow: / in production? Tell me about it in the comments.

Also , if you're interested, do visit my website - anoshbb.com

Crawlability vs Indexability: A Decision Tree for Figuring Out Why a Page Isn't in Google

Anosh — Tue, 06 Oct 2026 18:48:16 +0000

A page missing from Google is one of the most common problems in SEO. It's also one of the most misdiagnosed, because most people treat "not indexed" as a single problem. It's at least five different problems that look identical from the outside.

The mix-up usually starts with two words that get used interchangeably: crawlability and indexability. They are not the same thing, and fixing the wrong one is how people lose a whole afternoon.

Two different questions
Crawlability: Can Googlebot access this URL and fetch its content?
Indexability: Once Google has the content, is it allowed (and willing) to store the page in its index and show it in results?

An analogy: crawling is a librarian walking into your shop and reading your book. Indexing is deciding whether the book goes on the shelf. The librarian can read it and still leave it off the shelf.

That gives you four possible states:

Crawlable and indexable: the goal.
Crawlable but not indexable: Google reads the page but is told, or decides, not to keep it. noindex, canonicals pointing elsewhere and quality problems live here.
Not crawlable but still indexed: yes, this happens. A URL blocked in robots.txt can appear in results with no description if other pages link to it.
Not crawlable and not indexed: the usual "my page doesn't exist to Google" case.

Most beginners assume robots.txt is the tool for keeping pages out of Google. It isn't. It controls crawling, not indexing. That one misunderstanding explains a surprising number of broken sites.

The decision tree

Here's the flow I use whenever someone says "my page isn't showing up":

Page doesn't appear in Google
↓

Can Googlebot crawl it? (status code, server, redirects, login walls) ↓
Is crawling blocked? (robots.txt) ↓
Is indexing blocked? (noindex meta tag / X-Robots-Tag header) ↓
Is there a canonical pointing elsewhere? ↓
Is Google choosing not to index it? (discovered vs crawled, duplicates, soft 404) ↓
Is the page actually valuable enough?

Work through it in order. The first three steps are technical and yes/no. The last three are judgment calls, and they're harder to fix. Don't skip ahead to "content quality" when a stray noindex is sitting in your template.

Step 1: Can Googlebot even reach the page?

Start with the boring stuff. Check what your server actually returns.

HTTP status codes

200: the page loaded. This is what you want, but it's not proof the page is fine (see soft 404s below).
301 / 308: permanent redirect. Google follows it and generally treats the destination as the page that matters.
302 / 307: temporary redirect. Google may keep the original URL as the one it shows.
404 / 410: not found / gone. These URLs drop out of the index over time. A 410 is a slightly stronger signal that it's gone for good.
401 / 403: access denied. Googlebot can't get in, so it can't index the content.
429 / 5xx: the server is overloaded or erroring. If this keeps happening, Google slows its crawl rate, and pages can get dropped.

A quick way to see this yourself:

bash
curl -I https://example.com/your-page

Look at the first line for the status code, and look for x-robots-tag and location headers while you're there. Both come up again below.

Redirects

Redirects deserve their own attention because they fail quietly:

Chains: A → B → C → D. Google follows a limited number of hops (around 10), but every extra hop wastes crawl effort and slows things down. Keep it to one.
Loops: A → B → A. Googlebot gives up.
Redirecting everything to the homepage: Google often treats this like a soft 404, because the destination has nothing to do with the original page.
JavaScript or meta-refresh redirects: these can work, but they're slower and less reliable than a server-side redirect.