DEV Community: ELYRS

How to Get Indexed and Crawled by AI Bots: A Practical Guide for Developers

ELYRS — Tue, 27 Oct 2026 07:31:00 +0000

AI assistants and AI search products now answer a growing share of technical questions, and what they cite depends on what their crawlers can find, read, and understand. This guide walks through the practical steps that make a site easy for those systems to access and use.
TL;DR: To be crawled and cited by AI systems, your pages need to be reachable by their bots, readable in the initial HTML, indexed by the search engines AI products rely on, and structured so a model can extract a clear answer. Allow search-focused AI crawlers in robots.txt, server-render your content, submit sitemaps to Google and Bing, write answer-first sections with schema markup, and confirm in your server logs that the bots are actually visiting. An llms.txt file is optional; current data suggests most AI crawlers do not request it.
Why AI crawlability matters
Search used to mean ten blue links. Today, a developer can ask ChatGPT, Claude, or Perplexity how to solve a problem and receive one synthesized answer with a handful of cited sources. If your documentation, blog, or product pages are not part of what those systems can retrieve, you are absent from that answer.
The good news is that most of the work overlaps with sound web fundamentals: accessible pages, clean HTML, clear structure, and trustworthy content. What is new is knowing which bots exist, what each one does, and how to confirm they are visiting.
Step 0: Know the three types of AI bots
Not every AI crawler serves the same purpose. Most major providers separate their bots into three groups.
Type
What it does
Examples
Training crawlers
Collect content for model training
GPTBot, ClaudeBot, CCBot
Search and index crawlers
Build the index behind AI search results and citations
OAI-SearchBot, Claude-SearchBot, PerplexityBot
User-triggered fetchers
Retrieve a specific page when a user asks for it
ChatGPT-User, Claude-User, Perplexity-User
This distinction matters for one reason: blocking a training crawler does not remove you from AI answers, but blocking a search crawler can. Many site owners add a blanket rule against anything labeled “AI” and then wonder why they never appear in AI search results.
User-agent names change over time, so confirm them against the official documentation from OpenAI, Anthropic, and Perplexity before you deploy any rules.
Step 1: Let the right bots in with robots.txt
Your robots.txt file at the root of each hostname is the first thing a well-behaved crawler checks. Two setups cover most sites.
Option A: Maximum visibility. Allow search and retrieval bots, and leave training crawlers open as well.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Option B: Appear in AI answers, opt out of training.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml
A few things to check:
• robots.txt applies per hostname. A subdomain such as docs.example.com needs its own file.
• A permissive robots.txt does not help if your CDN, WAF, or bot-protection layer returns 403 errors or challenge pages to unfamiliar user agents. Review the bot-management settings in tools like Cloudflare, Fastly, or Akamai.
• robots.txt is a request, not a security control. Compliant crawlers honor it; scrapers that ignore it need rate limits or WAF rules. Keep private content behind authentication.
• User-triggered fetchers may be handled differently from crawlers, so read each vendor’s documentation for how they treat your rules.
Step 2: Put your content in the initial HTML
Many AI crawlers fetch a page and read the HTML response without running JavaScript. If your article text, documentation, or product details only appear after client-side rendering, those systems may see an empty shell.
A quick test:
curl -s https://example.com/your-page | grep -i "a distinctive sentence from the page"
If the sentence is missing from the raw response, a crawler that does not render JavaScript will miss it too. The fix is to render on the server or at build time. Next.js, Nuxt, Astro, SvelteKit, and Remix all support server-side rendering or static generation.
Also confirm that:
• Important content is not hidden behind tabs, accordions, or “load more” buttons that need a click to inject text.
• Pages return accurate status codes: 200 for content, 301 for moved pages, 404 for missing pages, with no soft 404s.
• Server response times are reasonable. Slow or error-prone pages tend to be crawled less.
Step 3: Get indexed by the search engines AI products rely on
AI search products do not work in isolation. Many retrieve results from conventional search indexes, and Google’s guidance indicates its AI features in Search build on the same indexing and quality foundations as regular Search. Bing’s index is also widely reported to support parts of ChatGPT search and Microsoft Copilot. In practice, strong classic indexing is a prerequisite for AI visibility.
A practical checklist:
• Publish an XML sitemap with accurate values and reference it in robots.txt.
• Add a self-referencing rel="canonical" tag to each page.
• Link important pages from your navigation and from related content so crawlers can discover them.
• Verify your site in Google Search Console and Bing Webmaster Tools, then submit your sitemap to both.
• Use IndexNow to notify participating search engines when content is published or changed.
An IndexNow ping is a single request. Host your key file at https://example.com/YOUR_KEY.txt, then call:
curl "https://api.indexnow.org/indexnow?url=https://example.com/new-post&key=YOUR_KEY"
Step 4: Write for retrieval, not just for ranking
Language models extract answers from passages, so pages that are easy to quote are easier to cite. These habits help:
• Lead with the answer. Open each section with a direct one- or two-sentence response, then add context.
• Use descriptive headings. “How to configure rate limiting in Express” is more useful than “Getting started.” Phrase headings the way a person would ask the question.
• Keep sections self-contained. A passage should make sense when read without the rest of the page. Name the tool or concept instead of writing “it” or “this.”
• Prefer structure to long prose. Use lists for steps, tables for comparisons, and code blocks for commands.
• Show provenance. Include an author, a publication date, an updated date, and links to primary sources.
• Stay consistent with names. Use one name per product, feature, or concept across your site.
Here is a small example. Instead of:
Our platform offers a variety of options for teams who want to manage their deployments more effectively.
Write:
Acme Deploy is a CI/CD tool that automates container deployments to Kubernetes. It supports blue-green and canary releases and integrates with GitHub Actions.
The second version states what the product is, what it does, and how it fits into a workflow. That is exactly the kind of sentence a model can lift into an answer.
Step 5: Add structured data
Schema markup gives machines an explicit description of what a page is. Use JSON-LD, keep it accurate, and make sure it matches the visible content.

{
"@context": "https://schema.org",
"@type": "Article",
"headline": "How to Get Indexed and Crawled by AI Bots",
"author": { "@type": "Person", "name": "Your Name" },
"datePublished": "2026-10-27",
"dateModified": "2026-10-27",
"publisher": { "@type": "Organization", "name": "Example Co" }
}

CHECK OUT THE SCHEMA ON MY WEBPAGE " MERINO WOOL SCRUBS "
Useful types include Article, FAQPage, Organization, SoftwareApplication, Product, and BreadcrumbList. Structured data is not a magic switch, but it removes ambiguity for any system that reads it.
Step 6: Treat llms.txt as optional
llms.txt is a proposal introduced by Jeremy Howard in 2024: a Markdown file at your domain root that lists your most important pages with short descriptions. It is easy to publish, and many CMS plugins now generate it automatically.
The evidence that it affects AI search visibility is weak. Limy analyzed more than 500 million AI bot visits over 90 days and found only 408 requests for the file. Ahrefs examined server logs from 137,210 domains and reported that 97% of llms.txt files received no requests in May 2026. Google’s guidance on AI features in Search also lists the file among the things you do not need.
That does not make it useless. Documentation sites, SDKs, and developer tools may still benefit, because coding assistants and agent workflows can use it as a quick map of canonical pages. If you publish one, keep it short and current:

Example Project

Example is an open-source TypeScript library for validating API schemas.

Docs

Quickstart: Install the library and run a first validation
API reference: Methods, options, and return types

Optional

Changelog: Release history Think of it as a low-cost extra, not a substitute for the fundamentals above. Step 7: Verify that the bots are actually visiting Do not assume; check. Your server logs show who is requesting what. This command counts hits from common AI user agents: grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | sort | uniq -c | sort -rn Then look at what the search bots request and what status codes they receive (this assumes the common or combined log format): grep -Ei "OAI-SearchBot|Claude-SearchBot|PerplexityBot" access.log | awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -20 Watch for these patterns: • Repeated 403, 429, or 5xx responses to AI bots usually point to a firewall rule, rate limit, or server problem. • A bot that requests only robots.txt and nothing else may be blocked by your rules or by a CDN challenge. • User-agent strings can be spoofed. Where a vendor publishes IP ranges or a reverse-DNS verification method, use it before trusting the traffic. Keep in mind that a crawl is not the same as a citation. A bot visit means your page was fetched; whether it appears in an answer depends on relevance, quality, and each product’s own ranking. To check, ask AI products questions your content answers and see whether your pages are cited. Step 8: Build corroborating signals AI systems tend to favor information that is clear, attributed, and supported elsewhere. Beyond your own site: • Keep your documentation, README, package registry listing, and product pages consistent with each other. • Publish useful technical content on platforms developers already read, such as DEV, and link back to your canonical resources. • Earn mentions from reputable third parties: integrations, conference talks, community answers, and directories. • Update important pages regularly and show the date of the last update. Common mistakes to avoid • Blocking every AI bot without separating search crawlers from training crawlers. • Relying on client-side rendering for key content. • Copying an outdated robots.txt template from a 2023 list. • Letting a CDN or WAF challenge legitimate bots while robots.txt says they are welcome. • Publishing llms.txt while neglecting sitemaps, canonical tags, and clean HTML. • Producing thin, generic content with no original examples, data, or expertise. Quick checklist [ ] robots.txt allows the search and retrieval bots you want [ ] Training-crawler policy is a deliberate decision, not an accident [ ] Key content appears in the raw HTML response [ ] Sitemap submitted to Google Search Console and Bing Webmaster Tools [ ] Canonical tags and accurate status codes in place [ ] Answer-first sections with descriptive headings [ ] JSON-LD structured data that matches visible content [ ] Server logs reviewed for AI bot activity and errors [ ] Optional: a concise llms.txt for docs or developer resources Frequently asked questions Does allowing GPTBot or ClaudeBot make my site appear in AI answers? Not directly. Those are training crawlers. Search and retrieval crawlers are the ones tied to AI search results and citations, so allowing them is what matters for visibility. Whether to allow training is a separate decision. How long does it take for AI bots to crawl a new page? There is no guaranteed timeline. Sitemaps, internal links, and IndexNow help with discovery, and your logs will show when bots arrive. Do I need an llms.txt file? No. Current studies show very few AI crawlers request it. It can still be a useful, low-effort addition for documentation and developer resources. Can I guarantee that an AI product will cite my content? No. You can make your content accessible, indexable, and easy to extract, but each product decides what to cite. Conclusion Getting crawled by AI bots is less about special tricks and more about doing the basics well: let the right bots in, serve readable HTML, keep your indexing healthy, and write content that is clear enough to quote. Review your robots.txt today, run the curl test on your most important pages, and check your logs next week. Small fixes in those three places often make the biggest difference. About the author: [Your Name] writes about [your topic]. Find more at [ELYRS].

Odoo Landed Costs: Handling Partial Vendor Returns

ELYRS — Mon, 05 Oct 2026 06:44:33 +0000

When you manage inventory in Odoo, the purchase price is not always the complete cost of getting a product into your warehouse.

Freight, customs duties, insurance, shipping fees, and other import-related charges can increase the real cost of inventory. Odoo handles these additional expenses through landed costs, allowing them to be included in inventory valuation rather than treating every charge as an ordinary operating expense.

The workflow becomes more interesting when part of the shipment is later returned to the vendor.

This article walks through the logic behind Odoo landed costs and partial vendor returns, with a focus on inventory valuation, FIFO, AVCO, and accounting.

What Are Landed Costs in Od****oo?

Landed costs are additional expenses associated with bringing purchased goods into your inventory.

Common examples include:

Freight and transportation
Customs duties
Import taxes
Insurance
Port or handling charges

In Odoo, landed costs can increase the valuation of products that have already been received. The system supports different allocation methods, including equal distribution, quantity, current cost, weight, and volume. You'd like to shop for medical Bamboo scrubs

For example, imagine purchasing:

500 units
Purchase cost: $20 per unit
Freight: $500
Customs: $300

The inventory does not really cost the business only $10,000. The additional $800 can be incorporated into inventory valuation through landed costs.

Why Vendor Returns Make Landed Costs More Complicated

A vendor return changes both the quantity and value of inventory.

Suppose your original receipt contains 500 units and you later return 50 units because they are damaged or incorrect.

The original landed costs may already have been allocated across the received products.

That creates an important accounting question:

How should the inventory value of the returned units be handled after landed costs have been applied?

This matters because an incorrect workflow can make inventory valuation, product cost, and accounting reports difficult to reconcile.

Odoo's inventory valuation is based on stock movements, and vendor returns reduce inventory quantity and value according to the configured costing method.

A Simple Example

Consider this scenario:

Purchase quantity: 500 units
Product cost: $20/unit
Freight: $500
Customs: $300
Total landed cost: $800
Vendor return: 50 units

The original merchandise value is:

500 × $20 = $10,000

After the additional costs are applied:

$10,000 + $800 = $10,800

The effective inventory cost is therefore higher than the supplier's original unit price.

Now imagine that 50 units are returned.

The important point is that the return should not be analyzed only as a quantity adjustment. It also affects the value carried by inventory.

FIFO vs. AVCO in Odoo

The costing method matters when analyzing this situation.

FIFO

With First In, First Out (FIFO), incoming inventory can retain separate valuation layers. When goods leave inventory, Odoo determines their value according to the FIFO costing logic.

This is particularly useful when you need visibility into different purchase batches or costs.

Odoo's documentation shows that remaining quantities in FIFO can retain their individual valuations rather than being reduced to one universal unit cost.

AVCO

With Average Cost (AVCO), Odoo calculates a weighted average cost as inventory changes.

When additional landed costs are applied, they can influence the average inventory cost. This means a later vendor return needs to be considered in the context of the current valuation rather than simply assuming that every unit has one permanently fixed historical cost.

Odoo supports both FIFO and AVCO for landed-cost workflows; standard-price products behave differently because their cost is manually controlled.

What Should You Check After a Vendor Return?

One of the most useful habits is to inspect the inventory valuation rather than relying only on the quantity shown on the receipt.

After processing the return, check:

The original receipt

Confirm the quantity received, purchase price, and the stock valuation associated with the receipt.

The landed cost record

Review which transfer received the landed cost and how the additional amount was allocated.

Odoo only allows validated transfers to be selected for a landed-cost record.

The vendor return

A vendor return creates an outbound stock movement and decreases inventory value. Odoo calculates the value according to the configured costing method.

Inventory Valuation

Compare the inventory value before and after the return.

Odoo provides an Inventory Valuation report so businesses can review the value of stock and identify differences between inventory and accounting valuation.

Accounting entries

For companies using automated or perpetual inventory valuation, it is also important to verify the corresponding accounting impact.

Odoo records landed-cost accounting entries when the landed cost is validated, and those entries can be reviewed from the Accounting application.

A Common Mistake: Treating Freight as a Normal Expense

A frequent implementation mistake is recording every shipping or customs charge directly as an operating expense.

That may be appropriate for some costs, but when a charge is part of getting purchased inventory ready for use or sale, it may need to be included in the inventory cost instead.

This is exactly where landed costs become useful.

Instead of thinking:

Purchase price + shipping expense

the inventory team can think in terms of:

Purchase price

freight
customs
other applicable landed costs = inventory valuation

The correct accounting treatment still depends on the business's accounting policy and local requirements, but Odoo provides the landed-cost mechanism specifically for increasing product valuation with eligible additional costs.

A Practical Odoo Workflow

A clean workflow usually starts with the purchase order and receipt.

First, receive and validate the products.

Next, record the relevant vendor bill or landed-cost product and create the landed-cost record.

Then select the validated transfer, calculate the valuation adjustments, review the allocation, and validate the landed cost. Odoo adds the additional value to the selected inventory valuation.

When a vendor return becomes necessary later, process the return from the original receipt and then review the resulting valuation.

The important part is not simply asking whether the returned quantity is correct. You should also verify whether the resulting inventory value makes sense.

How to Troubleshoot Valuation Differences

When the numbers do not look right, start with the stock moves instead of immediately creating a manual journal entry.

Check:

Receipt
↓
Original inventory value
↓
Landed cost allocation
↓
Updated inventory value
↓
Vendor return
↓
Remaining inventory value

This makes it easier to identify whether the problem is related to the receipt, costing method, landed-cost allocation, return, or accounting configuration.

Odoo's inventory system tracks value through stock movements, while differences between inventory valuation and accounting valuation can be reviewed as stock variation.

Final Takeaway

Odoo landed costs are useful when the supplier's invoice does not represent the complete cost of bringing inventory into the warehouse.

Once vendor returns enter the picture, the process becomes more dependent on the company's costing method and the valuation of the underlying stock movements.

The safest approach is to treat the workflow as a complete chain:

Purchase → Receipt → Landed Cost → Vendor Return → Inventory Valuation → Accounting

Rather than making manual adjustments too early, review each stock movement and valuation layer first. This gives you a much clearer picture of how Odoo arrived at the final inventory value.

For businesses using FIFO or AVCO, understanding this relationship is especially important because inventory valuation can change as goods are received, returned, and adjusted.

The goal is not simply to make the quantity correct. The goal is to make sure the quantity, inventory cost, valuation, and accounting records tell the same story.