The latest articles on DEV Community by Benoît Eveillard (@benoiteveillard). https://dev.to/benoiteveillard https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1126312%2F07cfcf42-d204-4832-8594-bd20a3a7b917.png https://dev.to/benoiteveillard en Benoît Eveillard Mon, 19 Jan 2026 09:19:28 +0000 https://dev.to/benoiteveillard/stop-letting-ai-bots-crawl-your-site-blindly-build-an-llmstxt-in-seconds-3ega https://dev.to/benoiteveillard/stop-letting-ai-bots-crawl-your-site-blindly-build-an-llmstxt-in-seconds-3ega <p>The robots.txt file was the hero of the 2000s. It told Google and Bing where to go. But in 2025, we have a new challenge: LLMs and AI agents.</p> <p>AI tools like ChatGPT, Claude, and specialized coding agents are constantly trying to understand our websites. If they have to scrape every single HTML page to find information, they waste your bandwidth, "hallucinate" structure, and get lost in the noise of your footer and navigation tags.</p> <p>That’s why the llms.txt standard (proposed by the folks at Answer.ai) is becoming a must-have.</p> <p>I decided to build the fastest way to generate one: an Apify Actor that turns your sitemap into a clean, LLM-ready roadmap.</p> <p><strong>What I built</strong><br> I created an automated llms.txt Generator. Instead of manually writing your site map for AI, this tool does the heavy lifting:</p> <p>Sitemap Deep-Dive: It doesn't just read one file; it recursively follows sitemap indexes.</p> <p>Smart Metadata Extraction: It pulls </p> tags and to give the AI context for every link. <p>Glob Filtering: You can easily exclude /tags/<em>, /admin/</em>, or legal pages that just clutter the AI's context window.</p> <p>Polite &amp; Ethical: It respects robots.txt by default and allows you to set concurrency limits so you don't stress your server.</p> <p><strong>Why you need this ?</strong><br> If you have a documentation site, a blog, or a product landing page, an llms.txt file at your root (e.g., mysite.com/llms.txt) allows AI agents to:</p> <ul> <li>Understand your site structure in milliseconds</li> <li>Avoid scraping unnecessary pages</li> <li>Provide better answers to users asking questions about your content</li> </ul> <p><strong>Performance &amp; Cost</strong><br> Since I built this using Crawlee and Cheerio (no heavy headless browsers needed), it’s incredibly fast and cheap.</p> <p><strong>How to use it</strong></p> <ol> <li>Head over to Apify llms.txt Generator</li> <li>Paste your sitemap URL</li> <li>Run it</li> <li>Get your direct download link and host it on your site!</li> </ol> <p>The project is live on the Apify Store. I'd love to hear what you think!<br> Check out the tool here: <a href="https://apify.com/justa/llms-txt-file-generator" rel="noopener noreferrer">https://apify.com/justa/llms-txt-file-generator</a></p> webdev Benoît Eveillard Tue, 13 Jan 2026 13:25:31 +0000 https://dev.to/benoiteveillard/how-i-built-a-builtwith-alternative-with-7000-technology-signatures-9lg https://dev.to/benoiteveillard/how-i-built-a-builtwith-alternative-with-7000-technology-signatures-9lg <h2> The Problem </h2> <p>I needed to detect what technologies websites use. The existing options:</p> <ul> <li> <strong>BuiltWith</strong>: Comprehensive but $295+/month</li> <li> <strong>Wappalyzer</strong>: Good but limited API and signatures</li> <li> <strong>Manual inspection</strong>: Time-consuming and incomplete</li> </ul> <p>So I built my own.</p> <h2> The Solution: 8-Tier Deep Inspection </h2> <p>Most detection tools only look at HTML. I went deeper with 8 tiers:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Tier</th> <th>What It Checks</th> <th>Example Detections</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>HTTP Headers</td> <td>Cloudflare, nginx, security headers</td> </tr> <tr> <td>2</td> <td>DOM/HTML</td> <td>Meta tags, script sources, CSS classes</td> </tr> <tr> <td>3</td> <td>JavaScript</td> <td>Global variables (React, Vue, jQuery)</td> </tr> <tr> <td>4</td> <td>Network</td> <td>XHR/Fetch requests to analytics, CDNs</td> </tr> <tr> <td>5</td> <td>Storage</td> <td>localStorage keys (Redux, auth tokens)</td> </tr> <tr> <td>6</td> <td>PWA</td> <td>Service workers, manifest.json</td> </tr> <tr> <td>7</td> <td>WebSocket</td> <td>Real-time connections (Socket.io)</td> </tr> <tr> <td>8</td> <td>Cookies</td> <td>Cookie names (tracking, auth)</td> </tr> </tbody> </table></div> <h2> Tech Stack </h2> <ul> <li> <strong>Runtime</strong>: Node.js</li> <li> <strong>Browser</strong>: Playwright + Camoufox (stealth mode)</li> <li> <strong>Signatures</strong>: 7,000+ patterns (Wappalyzer-compatible + custom)</li> <li> <strong>Hosting</strong>: Apify platform</li> </ul> <h2> Results </h2> <p>Scanning a typical site detects 10-15 technologies:</p> <ul> <li>CMS/Builder (Webflow, WordPress)</li> <li>Analytics (GA4, Plausible)</li> <li>Marketing (GTM, HubSpot)</li> <li>Infrastructure (Cloudflare, AWS)</li> </ul> <h2> Try It </h2> <p>Live on Apify: <a href="https://apify.com/justa/technology-profiling-engine" rel="noopener noreferrer">https://apify.com/justa/technology-profiling-engine</a></p> <p>Pricing: $0.005 per URL (~$5 per 1,000 URLs)</p> <p>What technologies do you wish were easier to detect? Let me know in the comments!</p> javascript showdev tooling webdev