title: "SEOCode Update 1.6.4: Better Checks for Apple's Web Crawler"
date: 2026-09-16
version: 1.6.4
slug: seocod-update-1-6-4-applebot-crawl-eligibility
description: "Version 1.6.4 adds a full check for how Apple's Applebot crawler reads your site, covering Siri, Spotlight, Safari search, and Apple Intelligence."
tags: [applebot, apple intelligence, robots.txt, crawling, technical seo]
SEOCode Update 1.6.4: Better Checks for Apple's Web Crawler
This update adds a proper check for how Apple's web crawler reads your site, which matters because Apple now uses your pages to power Siri answers, Spotlight search, and its AI features.
New Checks
Applebot Crawl Eligibility and Directive Configuration
What is Applebot?
Apple has a web crawler called Applebot. A web crawler is a program that visits websites and reads their pages. Google has one too, called Googlebot. Applebot is what Apple uses to power Spotlight (the search on your iPhone and Mac), Siri, Safari's search suggestions, and Apple Intelligence (Apple's AI features).
This new check looks at whether your site is set up correctly for Applebot. Here is what it checks, and why each part matters.
Your robots.txt file and Applebot
A robots.txt file is a small text file on your website that tells crawlers which pages they are allowed to visit. Think of it as a sign on your door that says "come in" or "stay out."
Applebot reads this file and follows instructions written for "Applebot" specifically. If your robots.txt file does not mention Applebot at all but does mention Googlebot, Applebot will follow the Googlebot instructions instead.
What you should do: if you want Applebot to behave differently from Googlebot, make sure you write separate instructions for it in your robots.txt file.
Meta tags that control what Apple does with your pages
A meta tag is a small piece of code in the invisible header of a web page. It gives instructions to crawlers without showing anything to your visitors.
Applebot reads these meta tags:
- noindex: tells Applebot not to show this page in Spotlight or Siri results.
- nosnippet: tells Applebot not to pull a description or answer from this page. It also stops Apple from using the page's content as background information for its AI-generated answers.
- nofollow: tells Applebot not to follow the links on this page.
- none: a shortcut that means both noindex and nofollow.
- index: tells Applebot it is allowed to index the page (this is the default, so you usually do not need to add it).
What you should do: if there are pages you do not want showing up in Siri or Spotlight, or pages you do not want Apple's AI using as source material, check that you have the right meta tags in place.
HTTP response headers for non-HTML files
Some files on your site are not web pages. Think of PDFs, images, or videos. You cannot put a meta tag inside a PDF. For those files, you can use something called an X-Robots-Tag, which is an instruction sent alongside the file when someone (or a crawler) downloads it.
Applebot reads these headers. The same instructions apply: noindex, nosnippet, and so on.
What you should do: if you have PDFs or other non-HTML files you want to keep out of Apple's results, make sure the right X-Robots-Tag is set on them.
JavaScript and CSS must not be blocked
Applebot can load and read pages the same way a browser does, including running JavaScript and loading CSS (the code that controls how a page looks). If your robots.txt file blocks Applebot from loading JavaScript or CSS, it may not be able to read your page properly. That means it might miss content or misunderstand your page.
What you should do: make sure your robots.txt file does not block Applebot from accessing your JavaScript or CSS files.
Paywalled content
If part of your site is behind a paywall (meaning visitors have to pay or log in to read it), you can tell Applebot about this using a piece of structured data called JSON-LD. Structured data is extra information you add to your page's code to help crawlers understand what the page is about.
Setting isAccessibleForFree: false in your JSON-LD tells Apple that the content is paywalled. Your page can still show up in Spotlight and Siri results. Apple just will not use that locked content as background material for its AI answers.
One thing to note: Applebot only supports this setting at the page level. It does not support section-level markup (a more detailed way of marking up which specific parts of a page are paywalled).
What you should do: if you have paywalled content, add the correct JSON-LD markup so Apple handles it properly.
Applebot-Extended: a separate crawler for AI training
Apple has a second crawler called Applebot-Extended. This one does not visit your website on its own. It only looks at data that the main Applebot has already collected.
Applebot-Extended is used to decide whether your content can be used to train Apple's AI models (called foundation models). If you block Applebot-Extended in your robots.txt file, you are opting out of that AI training. But blocking it has no effect on whether your site shows up in Spotlight, Siri, or Safari search. Those are controlled by the main Applebot.
What you should do: decide whether you want your content used to train Apple's AI. If not, block Applebot-Extended in your robots.txt. If you are happy with it, you do not need to do anything.
Crawl-delay does not work with Applebot
Some robots.txt files include a "crawl-delay" instruction. This tells a crawler to wait a set number of seconds between each page visit, to avoid putting too much load on your server. Applebot ignores this instruction entirely.
What you should do: if you are worried about Applebot putting load on your server, you will need to use a different approach, such as blocking it temporarily in robots.txt.
Apple Podcasts and robots.txt
Apple Podcasts uses its own crawler called iTMS. This crawler does not follow robots.txt at all. So if you run a podcast and you have instructions in your robots.txt file, the Apple Podcasts crawler will not read them.
Updated Checks
Applebot-Extended Rules Do Not Affect Google Search Ranking
This check already existed. It reminds you that whatever you put in your robots.txt file about Applebot-Extended has no effect on how your site ranks on Google. Apple itself says this clearly.
We updated this check to reflect the latest guidance from Apple and to line it up with the new Applebot check above.
Sources
Catch SEO issues before they ship
SEOCode reviews every pull request against the latest Google SEO rules, automatically. No manual audits. No surprises after deploy.
It checks for missing metadata, broken structured data, hreflang errors, robots.txt issues, and more, right inside your existing GitHub workflow.
Want to see how your site is doing right now? Scan your website for SEO problems, free. No sign up.
Top comments (0)