SiteReadyForSi: Building an AI Search Readiness Scanner with Open-Source AI (Side Hustle)

The Saqib Tahir — Sat, 17 Oct 2026 11:10:22 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

SiteReadyForSi is an automated AI and search readiness diagnostic platform designed for the Generative Engine Optimization (GEO) era.

As AI search engines like Perplexity, SearchGPT, Claude, and Gemini transform how users discover information, traditional SEO is no longer enough. Many modern websites built on Client-Side Rendered (CSR) Single-Page Applications serve an almost empty HTML shell (<div id="root"></div>) that AI crawlers fail to parse within their strict rendering budgets.

SiteReadyForSi audits websites to ensure they are machine-retrievable and technically ready for autonomous AI scrapers. It analyzes:

Crawler Permissions: Verifies robots.txt access rules for modern AI bots (GPTBot, PerplexityBot, ClaudeBot, etc.).
Initial HTML Inspection: Detects empty shells vs server-rendered content.
Structured Data (JSON-LD): Validates semantic context for products, articles, and entities.
llms.txt Generator: Creates machine-readable markdown indexes for LLMs.

Who Would Pay For It?

SaaS Founders & E-Commerce Owners: Who risk losing organic AI search visibility and product citations.
Digital Agencies: Managing dozens of client domains who need on-demand, repeatable readiness reports.
Business Model: Monetized via pay-as-you-go audit credit packs and tiered monthly subscriptions ($25/month Pro plan) integrated with automated billing.

Demo

Live Deployed Application: https://sitereadyfor.si
Features in Action: Anyone can enter a homepage URL and immediately get a comprehensive readiness score, breakdown of crawler directives, and prioritized fix instructions.

Architecture & Implementation

The production platform is engineered as a resilient, full-stack micro-SaaS:

Frontend & Server: Built with React 19 and React Router 7 over Express, providing blazing-fast SSR for SEO diagnostics.
Background Queue & Processing: Uses BullMQ backed by Redis to orchestrate asynchronous background crawls and audit queues for large websites (>15 pages) without degrading main app latency.
Database & Storage: Powered by MongoDB Atlas, storing crawl histories, parsed audit signals, user workspaces, and transactional billing events.
Intelligent Multi-Stage AI Reasoning Pipeline: Uses an advanced hybrid AI workflow to evaluate technical site health:
- Fast Signal Parsing: Rapidly processes raw crawler directives, DOM elements, and server headers.
- Deep Architectural Reasoning: Analyzes complex rendering patterns, structured data integrity, and synthesizes clear, prioritized technical remediation guides.

Why Does Open Innovation Matter?

Building on an open, modular AI architecture is what enables SiteReadyForSi to operate reliably at scale:

Architectural Autonomy & Customization: Open and flexible AI pipelines allow us to tailor our diagnostic criteria specifically for crawler behaviors, without being constrained by rigid, generic black-box APIs.
Client Data Privacy & Isolation: Technical audits evaluate internal site architecture, proprietary route structures, and draft content. An open-oriented pipeline ensures all parsed data remains strictly isolated and is never fed back into external third-party models for training.
Consistent & Deterministic Scoring: Proprietary closed-source APIs frequently change their underlying weights and behaviors, causing fluctuating evaluations. A dedicated, self-contained AI pipeline ensures consistent, repeatable readiness benchmarks that website owners can rely on over time.

Prize Categories

Best Use of MongoDB Atlas (utilizing MongoDB Atlas to power the multi-tenant audit stores, crawl data pipelines, and user workspaces)

The Technical GEO Readiness Checklist: Preparing Your Website for AI Search & LLMs

The Saqib Tahir — Sat, 17 Oct 2026 11:04:16 +0000

The Technical GEO Readiness Checklist: Preparing Your Website for AI Search & LLMs

As Generative Engine Optimization (GEO) replaces traditional SEO assumptions, the way search engines discover, retrieve, and parse website content is undergoing a fundamental shift.

Traditional search engines focus primarily on sitemaps, link graphs, and static HTML snippets. However, modern AI search agents and LLM crawlers (such as Perplexity, GPTBot, ClaudeBot, and Google Gemini) require a solid technical foundation to evaluate content.

Here is the essential checklist to verify that your website is technically ready for AI retrieval.

1. Verify Crawler Access

Check your robots.txt directives for relevant AI search bots (GPTBot, PerplexityBot, ClaudeBot, Google-Extended).

A simple Allow rule alone does not guarantee access:

Firewalls, Cloudflare bot protection, and aggressive rate limits can inadvertently block legitimate search crawlers.
Ensure public documentation, pricing, and product pages return a 200 OK status with substantive HTML.

2. Inspect the Initial HTML Response

Compare the raw HTML delivered by the server with the rendered page in a browser:

If your web app is a pure Client-Side Rendered (CSR) Single-Page Application (SPA) without SSR or prerendering, crawlers may only receive an empty <div id="root"></div> shell.
AI crawlers operate on strict rendering time budgets. Critical headings, meta descriptions, and core content must be present in the initial server response.

3. Validate Structured Data (JSON-LD)

AI agents thrive on structured, semantic data:

Ensure your Product, SoftwareApplication, Article, and Organization schemas accurately mirror the visible content.
Validate your JSON-LD with standard schema validators to eliminate syntax errors.

4. Provide a Concise `llms.txt`

The proposed llms.txt standard offers a clean, machine-readable markdown guide for AI crawlers:

Summarize your core product, features, and technical capabilities.
Provide clean, direct links to essential documentation and pricing pages.

5. Measure Technical Readiness

Measuring your readiness is the first step toward visibility in modern search. You can test your website's crawlability, initial HTML delivery, and crawler directives in seconds.

👉 Run an instant diagnosis of your website at SiteReady.

Originally published on SiteReady.

DEV Community: The Saqib Tahir

SiteReadyForSi: Building an AI Search Readiness Scanner with Open-Source AI (Side Hustle)

What I Built

Who Would Pay For It?

Demo

Architecture & Implementation

Why Does Open Innovation Matter?

Prize Categories

The Technical GEO Readiness Checklist: Preparing Your Website for AI Search & LLMs

The Technical GEO Readiness Checklist: Preparing Your Website for AI Search & LLMs

1. Verify Crawler Access

2. Inspect the Initial HTML Response

3. Validate Structured Data (JSON-LD)

4. Provide a Concise llms.txt

5. Measure Technical Readiness

4. Provide a Concise `llms.txt`