DEV Community

Cover image for Is your robots.txt blocking AI search? 4 free browser tools we just shipped
313 Automations
313 Automations

Posted on AI-assisted

Is your robots.txt blocking AI search? 4 free browser tools we just shipped

We build AI automation and websites for small businesses, and the same questions keep coming up on client calls. So we turned four of them into free, single-file browser tools on Hugging Face. No signup, no backend, nothing stored.

1. AI Crawler robots.txt Checker

The problem: "AI crawlers" are not one thing. Some collect training data (GPTBot, ClaudeBot, CCBot), some build AI search indexes (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and some fetch a page because a user asked (ChatGPT-User, Perplexity-User). Blocking the wrong one can quietly remove a site from AI answers.

The checker parses a pasted robots.txt the way major crawlers do (RFC 9309: most specific user-agent group wins, longest matching rule wins, Allow wins ties, * and $ wildcards) and evaluates 27 crawlers. It then generates a policy for one of three stances: get cited but opt out of training, allow everything, or block AI while keeping Google and Bing.

👉 https://huggingface.co/spaces/313Automations/ai-crawler-robots-checker

The crawler list is also an open dataset with the operator's documentation linked on every row: https://huggingface.co/datasets/313Automations/ai-crawler-user-agents

2. llms.txt Generator

Fill in a name, a one-line summary, key facts and your important pages, and it outputs a valid llms.txt (H1, blockquote, details, H2 link sections, Optional) with checks for missing names and non-https links.

We were honest on the page: Google has said Search doesn't use llms.txt, and no major AI search engine has confirmed it as a ranking or citation signal. It is cheap to add and can help agents navigate, but it is not a substitute for crawlable, clear content.

👉 https://huggingface.co/spaces/313Automations/llms-txt-generator

3. Automate or Hire Calculator

Hours per week, loaded hourly cost, the share automation can handle, build and running cost. It returns two-year cost for both paths, monthly saving, payback and a five-question fit check, then recommends automate, hybrid or hire.

👉 https://huggingface.co/spaces/313Automations/automate-or-hire-calculator

4. AI Missed-Call Responder ROI Calculator

Calls per month, missed rate, share that are new enquiries, conversion, customer value and margin. It shows revenue at risk, what an instant text-back or AI receptionist could win back, and compares profit (not revenue) with the monthly cost.

👉 https://huggingface.co/spaces/313Automations/missed-call-roi-calculator

How they're built

Each tool is one self-contained index.html on a static Hugging Face Space: inline CSS with light and dark themes, vanilla JS, WebApplication JSON-LD, and no external requests. View the source under Files on any Space.

We also published three more datasets: a GEO prompt library (125 buyer questions across 25 industries), 30 SMB workflow automation templates and 45 FAQ/FAQPage examples for service businesses.

Everything lives here: https://huggingface.co/313Automations

What should we build next? Ideas welcome in the comments.

Top comments (0)