Building a RAG Crawler with Supabase and Gemini for SEO Audits
Most RAG examples start with PDFs, documentation, or a collection of text files.
Websites are a different problem.
A website is a connected system of pages containing HTML, metadata, headings, links, structured data, canonical URLs, and content that changes over time. If an AI SEO auditor only receives the HTML of the page being analyzed, it has very limited context.
While building AuditMe, I needed a crawler and retrieval layer that could turn a website into structured, searchable context for an AI audit.
The resulting pipeline is:
URL
↓
HTTP Fetch
↓
Cheerio Parse
↓
Chunking
↓
Gemini Embeddings
↓
Supabase pgvector
↓
Cosine Retrieval
↓
Grounded LLM Prompt
↓
Structured JSON Audit
The interesting part is not just the RAG component.
The crawler also has to deal with SSRF, redirects, robots.txt, throttling, JavaScript-rendered applications, duplicate URLs, incremental refreshes, and the difference between deterministic SEO checks and AI-generated analysis.
This article explains how the system works.
Why an SEO auditor needs RAG
A conventional SEO crawler can detect many problems deterministically:
- missing title;
- missing meta description;
- missing canonical;
- broken links;
- missing alt text;
- HTTP status problems;
- redirect chains;
- missing structured data;
- heading structure;
- hreflang configuration.
Those checks do not need an LLM.
But other questions are semantic:
- Does the page adequately cover its topic?
- Are several pages targeting the same intent?
- Is the content too thin or repetitive?
- Which sections are relevant to a particular recommendation?
- Does the page's content support the conclusions generated by the AI?
This is where retrieval becomes useful.
Instead of giving the LLM an arbitrary amount of HTML, the system retrieves the most relevant chunks from the page and injects them into the audit prompt.
The goal is simple:
Give the model the evidence it needs, rather than asking it to invent context.
1. The architecture
The current pipeline is deliberately straightforward:
+----------------+
| URL |
+-------+--------+
|
v
+----------------+
| HTTP Fetch |
+-------+--------+
|
v
+----------------+
| Cheerio Parse |
+-------+--------+
|
v
+----------------+
| Chunking |
| 1400 / 150 |
+-------+--------+
|
v
+----------------+
| Gemini |
| Embeddings |
| 768 dimensions |
+-------+--------+
|
v
+----------------+
| Supabase |
| pgvector |
| HNSW |
+-------+--------+
|
v
+----------------+
| Top-K Retrieval|
| in Node |
+-------+--------+
|
v
+----------------+
| Grounded LLM |
| Prompt |
+-------+--------+
|
v
+----------------+
| Zod Validation |
| Structured JSON |
+----------------+
There is an important implementation detail here:
Supabase stores the embeddings and provides an HNSW index, but the current audit retrieval path performs brute-force cosine similarity in Node for the page's own chunks.
That may sound redundant.
It is intentional.
The database layer is ready for vector retrieval, while the current audit path can cheaply retrieve from the relatively small number of chunks belonging to one page.
2. Crawling the page
The first stage is an HTTP fetch.
The fetched HTML is parsed with Cheerio.
The crawler extracts substantially more than visible text:
title
meta description
canonical
Open Graph tags
Twitter cards
H1-H3
images
internal links
external links
nofollow links
word count
JSON-LD types
hreflang
redirect chains
This produces two different kinds of information.
Structured information
Examples:
title
canonical
status code
links
JSON-LD
hreflang
redirects
These are stored and analyzed as structured data.
Unstructured information
Examples:
paragraphs
headings
article content
product descriptions
documentation
This content is suitable for chunking and embeddings.
That distinction is important.
I do not want to turn every SEO fact into a vector.
If the question is:
Does this page have a canonical URL?
the crawler already knows the answer.
There is no reason to ask an embedding model.
3. SSRF protection
A crawler accepts URLs from users.
That makes SSRF protection a first-class requirement.
The crawler uses:
isSafeURL()
resolveAndValidateIP()
and validates every redirect hop.
This matters because checking only the initial URL is insufficient.
Consider:
https://example.com
↓
302 redirect
↓
http://internal-host
If the crawler validates only the first URL, the redirect can bypass the protection.
The crawler therefore resolves and validates the destination at every redirect hop.
The general rule is:
Never trust a redirect simply because the original URL was considered safe.
A production crawler also needs to consider private, loopback, link-local, and other non-public address ranges.
4. robots.txt
The crawler respects robots.txt.
The implementation caches robots.txt for one hour per domain.
It also resolves rules using the longest-pattern-match approach rather than treating the first matching rule as authoritative.
Conceptually:
Request URL
|
v
robots.txt cache
|
v
matching rules
|
v
longest matching pattern
|
v
Allow / Disallow
The crawler also honors a site-declared Crawl-delay when present.
The default throttle is 100 ms.
This is deliberately conservative.
A crawler should not turn a small SEO audit into a traffic spike against the target server.
5. BFS crawling
The scheduler uses breadth-first search.
The current limits are:
concurrency: 5
maxPages: 200
maxDepth: 5
A simplified crawl looks like:
Depth 0
|
+-- homepage
|
+-- Depth 1
|
+-- page A
+-- page B
+-- page C
|
+-- Depth 2
BFS is useful for SEO crawling because it tends to discover pages close to the site's entry points before going deeper.
The scheduler also needs URL deduplication.
Without it, the same page can be reached through multiple internal links and repeatedly scheduled.
6. Handling SPAs
Traditional HTTP fetching is not enough for every modern website.
A single-page application may initially return:
<div id="root"></div>
<script src="/assets/app.js"></script>
while the actual content is produced by JavaScript.
The crawler detects common SPA signals such as:
#root
#app
JavaScript bundles
and can optionally fall back to Playwright rendering.
The important principle is not to render every page in a full browser.
Browser rendering is more expensive.
The preferred strategy is:
HTTP fetch
|
+--> sufficient HTML?
| |
| yes
| |
| v
| parse
|
+--> likely SPA?
|
v
optional render
Use the expensive path only when necessary.
7. Content extraction
After Cheerio parsing, the page content is normalized.
The crawler caps content at 10,000 characters before the RAG pipeline.
This is important for two reasons.
First, very large pages can dominate embedding and prompt costs.
Second, most SEO audit questions do not require every byte of a page.
The extraction stage tries to preserve meaningful text while avoiding common HTML noise.
The result is roughly:
title
meta description
h1
body content
plus the structured page metadata collected separately.
8. Chunking strategy
The chunker uses:
MAX_CHARS = 1400
OVERLAP = 150
MAX_CHUNKS = 10
It uses a sliding window, but prefers a sentence boundary when possible.
The boundary preference is based on:
". "
rather than blindly cutting at character 1400.
This is a small implementation detail with a large practical effect.
Consider:
...Google recommends descriptive page titles because
they help users understand the result...
A hard character boundary can split a sentence.
A sentence-aware boundary is usually a better semantic unit.
Synthetic chunk 0
The first chunk is special.
Chunk 0 contains a synthetic header:
TITLE: ...
| meta_description
| h1
This gives the embedding representation immediate access to the page's most important SEO metadata.
For example:
TITLE: Technical SEO Guide
| Learn how to audit technical SEO
| Technical SEO Checklist
This can improve retrieval for queries related to the page's main topic.
Practical chunk ceiling
The body is capped at 10,000 characters.
With a maximum of 10 chunks and the current chunk size, the practical number of chunks is usually lower.
In practice, the current implementation generally produces around eight real body chunks for a page that reaches the content cap, with chunk 0 reserved for the synthetic header.
The point is to keep the retrieval corpus small and predictable.
9. Gemini embeddings
The embedding model is:
gemini-embedding-001
The stored vectors use 768 dimensions.
The model supports Matryoshka-style dimensionality reduction, allowing the embedding representation to be truncated to the required dimensionality.
The system batches up to:
96 texts per request
Embedding requests can fail transiently, particularly under rate limits.
The current retry strategy is:
maximum attempts: 3
backoff: attempt * 2000 ms
So the retry delays are approximately:
2000 ms
4000 ms
depending on which attempt is being retried.
Authentication is performed with the:
x-goog-api-key
header.
The application exposes the Gemini key through:
GOOGLE_GEMINI_API_KEY
10. Supabase and pgvector
The embeddings are stored in Supabase PostgreSQL using pgvector.
The core table is:
create extension if not exists vector;
create table page_embeddings (
id bigint generated by default as identity primary key,
url text not null,
chunk_index integer not null,
chunk_text text not null,
embedding vector(768),
source text not null default 'gemini-embedding-001',
meta jsonb not null default '{}'::jsonb,
created_at timestamptz not null default now(),
updated_at timestamptz not null default now(),
unique (url, chunk_index)
);
One correction is worth calling out explicitly.
If you copy an older version of this schema, you may see:
source text not null default 'gemini:text-embedding-004'
That is stale metadata from an earlier embedding setup.
The current embedding model for this implementation is:
gemini-embedding-001
The database schema should describe the model actually producing the vectors.
11. HNSW index
The table uses an HNSW index for cosine similarity:
create index page_embeddings_hnsw_idx
on page_embeddings using hnsw (embedding vector_cosine_ops);
HNSW is useful when vector collections become large enough that scanning every vector for every query becomes expensive.
The important distinction in this implementation is that the HNSW index exists at the database layer, but the current audit path does not depend on it for its normal page-local retrieval.
Instead, the current flow is:
page URL
|
v
retrieve its chunks
|
v
calculate cosine similarity in Node
|
v
sort
|
v
take top 5
Because a single page is limited to a small number of chunks, brute-force similarity is cheap.
This is a good example of not optimizing for a problem you do not currently have.
12. Supabase RPC
The database also exposes a vector-search RPC:
create or replace function search_page_chunks(
p_url text,
p_embedding vector,
p_limit integer default 5
)
returns table(
url text,
chunk_index integer,
chunk_text text,
similarity double precision
)
language sql
stable
as $$
select
e.url,
e.chunk_index,
e.chunk_text,
1 - (e.embedding <=> p_embedding) as similarity
from page_embeddings e
where e.url = p_url
order by e.embedding <=> p_embedding
limit p_limit;
$$;
The <=> operator is pgvector's cosine distance operator.
Therefore:
similarity = 1 - cosine_distance
The RPC makes it possible to move retrieval into PostgreSQL when that becomes preferable.
13. The actual RAG flow
The audit pipeline is deliberately narrow.
For each page:
1. Fetch page
2. Parse with Cheerio
3. Cap content at 10K characters
4. Chunk content
5. Generate embeddings
6. Store chunks
7. Create query from page metadata
8. Retrieve top chunks
9. Inject chunks into LLM prompt
10. Generate structured audit
11. Validate JSON with Zod
The indexing call is conceptually:
indexPageChunks(
url,
{
title,
meta_description,
h1,
content_text
}
)
The page's own metadata becomes the retrieval query:
[title, meta_description, h1].join(". ")
This is an important design choice.
The query is not an arbitrary user question.
It represents the page's own declared topic.
14. Top-K retrieval
The current retrieval function:
topKChunks()
performs cosine similarity in Node.
It returns:
top 5 chunks
with a minimum similarity threshold of:
0.15
So the practical rule is:
similarity > 0.15
and then take the five highest-scoring chunks.
The threshold should not be interpreted as a universal semantic quality boundary.
Embedding scores are model- and dataset-dependent.
A threshold that works for one corpus may be terrible for another.
The correct approach is to evaluate retrieval against representative queries and tune it empirically.
15. Grounding the LLM
The retrieved chunks are injected into the prompt under:
RETRIEVED PAGE CONTENT
The model therefore receives evidence retrieved from the actual page.
Conceptually:
SYSTEM / AUDIT INSTRUCTIONS
PAGE METADATA
TECHNICAL SEO DATA
RETRIEVED PAGE CONTENT
----------------------
[chunk 1]
[chunk 2]
[chunk 3]
[chunk 4]
[chunk 5]
Return structured JSON.
This does not magically eliminate hallucinations.
But it changes the model's job.
Instead of generating an SEO assessment from an isolated prompt, it can reason over the page's actual retrieved content.
16. Structured JSON and Zod
The LLM output is not accepted blindly.
The generated audit is validated with Zod.
The conceptual flow is:
LLM
|
v
JSON
|
v
Zod schema
|
+--> valid --> audit result
|
+--> invalid --> fallback/error handling
This matters because an LLM can produce syntactically invalid JSON, missing fields, unexpected values, or the wrong data types.
A schema turns the LLM from an untrusted output generator into a component with a defined contract.
17. Deterministic fallback
The AI layer is not the only source of insights.
If the LLM fails, the audit pipeline can fall back to:
buildDeterministicInsights()
This rule engine handles deterministic findings without requiring an AI response.
That gives the system an important property:
An AI outage should not make the entire SEO audit useless.
For example, if the crawler already knows that a page has no canonical, that fact does not disappear because an LLM request failed.
18. SEO checks that should remain deterministic
Some checks are simply better implemented as code.
For example:
title exists?
meta description exists?
canonical exists?
H1 exists?
HTTP status is 200?
internal link is broken?
JSON-LD exists?
hreflang is valid?
An LLM is unnecessary here.
The strongest architecture is therefore:
Deterministic crawler/checks
+
RAG retrieval
+
LLM reasoning
not:
Everything → LLM
This reduces cost and makes the technical checks reproducible.
19. Content quality signals
The crawler also calculates content-quality signals such as:
Flesch reading ease
keyword stuffing
complex-word ratio
word count
The keyword-stuffing detector currently flags keyword density above:
3%
These metrics should be treated as signals, not absolute truth.
For example, a technical article can legitimately repeat a domain-specific term many times.
A simple density threshold cannot understand that context.
The useful approach is to use these metrics as evidence alongside the actual content.
20. Cannibalization detection
One of the more interesting uses of embeddings is content cannibalization.
The system can compare page embeddings using pairwise cosine similarity.
The current threshold is:
0.85
Conceptually:
Page A ────────┐
│ cosine similarity
Page B ────────┘
|
v
similarity >= 0.85
|
v
potential semantic overlap
The current comparison is O(n²).
That is fine for a relatively small set of pages.
It becomes expensive as the number of pages grows.
For example:
100 pages -> 4,950 pairs
1,000 pages -> 499,500 pairs
10,000 pages -> 49,995,000 pairs
So pairwise comparison is useful as an initial implementation, but a large-scale crawler would eventually need a more efficient nearest-neighbor approach.
And again, similarity does not prove cannibalization.
It identifies candidates for further analysis.
21. Incremental refresh
A crawler that only works when a user manually starts an audit is not enough for monitoring.
AuditMe also has an incremental refresh process.
The refresh job checks the:
50 most recent scans
using HTTP validators:
If-None-Match
If-Modified-Since
If the page has not changed, there is no reason to download and re-embed it.
If it has changed:
re-scan
↓
re-extract
↓
re-chunk
↓
re-embed
↓
update audit data
↓
update score history
This is significantly cheaper than reprocessing every page every time.
22. Crawl job monitoring
Long-running jobs need a failure mechanism.
The crawler marks stale jobs as failed after:
10 minutes
Sites that have not been crawled for more than:
7 days
can be scheduled for another crawl.
This creates a basic monitoring loop:
crawl
|
v
fresh data
|
v
7 days
|
v
refresh
The exact interval can be changed depending on how frequently the target sites change.
23. Database model beyond embeddings
The RAG table is only one part of the application.
The other important tables are:
scans
Used for a 24-hour cache.
It includes:
url
data JSONB
etag
last_modified
The URL is the primary key.
audit_results
Stores permanent audits.
Each audit has a UUID and includes:
rag_debug
This is particularly useful when debugging retrieval quality.
If an AI recommendation looks wrong, you want to know what context the model actually received.
score_history
Stores score changes over time.
This makes trend analysis possible instead of showing only the current score.
crawl_jobs
Tracks crawler execution and status.
Old jobs are cleaned up using a seven-day TTL policy.
24. Why rag_debug matters
RAG systems are notoriously difficult to debug if retrieval is invisible.
Suppose the LLM makes a bad recommendation.
There are several possible causes:
crawler extracted the wrong content
↓
chunking lost context
↓
embedding was poor
↓
retrieval selected wrong chunks
↓
prompt was unclear
↓
LLM reasoned incorrectly
Without retrieval diagnostics, these problems all look like:
"The AI gave a bad answer."
That is not actionable.
Keeping retrieval information in rag_debug makes the pipeline observable.
You can inspect:
query
retrieved chunks
similarity scores
source URL
chunk indexes
and determine where the failure occurred.
25. Environment configuration
The main environment variables are:
GOOGLE_GEMINI_API_KEY
SUPABASE_SERVICE_ROLE_KEY
PAGESPEED_API_KEY
CRUX_API_KEY
CRON_SECRET
Their responsibilities are separated:
GOOGLE_GEMINI_API_KEY
-> Gemini embeddings + LLM
SUPABASE_SERVICE_ROLE_KEY
-> server-side database administration
PAGESPEED_API_KEY
-> PageSpeed enrichment
CRUX_API_KEY
-> Chrome UX / Core Web Vitals enrichment
CRON_SECRET
-> protected scheduled jobs
The service-role key should never be exposed to browser-side code.
26. What I learned from building it
The main lesson is that RAG is only one part of the system.
A useful web intelligence pipeline needs several layers.
Layer 1: Fetching
Get the actual page safely.
HTTP
redirects
robots
throttling
SSRF protection
Layer 2: Parsing
Turn HTML into useful information.
metadata
content
links
schema
headings
Layer 3: Structured analysis
Run deterministic checks.
status
canonical
titles
links
schema
hreflang
Layer 4: Retrieval
Find semantically relevant content.
chunk
embed
search
rank
Layer 5: Reasoning
Let the LLM interpret the evidence.
retrieved content
+
technical facts
+
audit instructions
Layer 6: Validation
Never trust generated output blindly.
LLM
↓
Zod
↓
structured audit
That separation makes the system much easier to reason about.
27. What I would improve next
There are several obvious directions for future iterations.
Better retrieval
The current page-local brute-force retrieval is simple and fast, but a larger corpus could use database-side HNSW retrieval or a hybrid search strategy.
Better chunking
The current 1400-character sliding window works, but heading-aware and semantic chunking could preserve context more effectively.
Better reranking
Top-K vector similarity is not necessarily the final ranking.
A reranker could improve retrieval precision.
Better site-wide RAG
The current audit query is primarily page-focused.
A broader retrieval layer could answer questions across the entire site.
Faster cannibalization detection
O(n²) pairwise comparison will eventually become a bottleneck.
Nearest-neighbor search can reduce the number of comparisons.
More browser rendering
Some JavaScript-heavy sites require full rendering to accurately inspect their content.
That should remain an optional expensive path rather than the default for every URL.
28. The bigger picture
The crawler is not just an ingestion script.
Once the site is represented as:
pages
+
links
+
metadata
+
content
+
embeddings
+
crawl history
it becomes possible to build much more than a basic SEO checker.
For example:
semantic content clusters
↓
internal linking recommendations
similar pages
↓
cannibalization candidates
content changes
↓
historical SEO monitoring
retrieved page content
↓
grounded AI recommendations
This is the direction I am taking with AuditMe.
The objective is not to replace deterministic SEO tooling with an LLM.
It is to combine deterministic analysis with semantic retrieval and AI reasoning.
29. Final architecture
The complete system can be summarized as:
WEBSITE
|
v
+---------------+
| Safe Crawler |
+-------+-------+
|
+--------------+--------------+
| | |
v v v
Metadata Links Content
| | |
+--------------+--------------+
|
v
Chunking 1400/150
|
v
Gemini embedding-001
|
v
768-dim vectors
|
v
+-----------------------+
| Supabase PostgreSQL |
| pgvector + HNSW |
+-----------+-----------+
|
v
Page-local Top-K
cosine retrieval
|
v
RETRIEVED PAGE CONTENT
|
v
Grounded LLM
|
v
Zod validation
|
+--------------+--------------+
| |
v v
Structured Audit Deterministic Fallback
|
v
AuditMe Results
The key idea is simple:
The crawler collects evidence. PostgreSQL stores facts. pgvector stores semantic representations. Retrieval selects context. The LLM reasons over that context.
That division of responsibilities is what makes the system practical.
Conclusion
Building RAG for websites is fundamentally different from building RAG over static documents.
The difficult part is not generating an embedding.
It is building a reliable representation of the website before the embedding is ever created.
You need to handle:
- safe URL fetching;
- redirects;
- robots.txt;
- crawl limits;
- SPA rendering;
- HTML extraction;
- content normalization;
- chunking;
- embeddings;
- vector storage;
- retrieval;
- structured SEO checks;
- LLM validation;
- incremental refresh;
- historical data.
Once those pieces are separated, the architecture becomes much easier to extend.
And that is the real value of combining a crawler with RAG:
You are not simply giving an AI access to a webpage. You are giving it a structured, retrievable representation of the website it is supposed to analyze.
You can see the resulting SEO auditing product at AuditMe and try the Website SEO Checker.
The code examples in this article are simplified versions of the implementation. Production crawlers should additionally account for rate limiting, resource limits, security hardening, rendering costs, database lifecycle management, and workload isolation.
Top comments (0)