DEV Community

Cover image for I Built a Website Intelligence Engine Instead of Another SEO Checklist
Kamayega Bharat
Kamayega Bharat

Posted on

I Built a Website Intelligence Engine Instead of Another SEO Checklist

#seo #ai #webdev #javascript

What I learned while building AuditForge AI: why crawling, rendering, shared resource intelligence, evidence, and root-cause analysis matter more than a long list of SEO checks.
Most website auditors give you a checklist.

A title is missing.

A meta description is missing.

A page has no schema.

An image has no alt text.

A page is slow.

Then you get a long list of warnings and have to figure out what actually matters.

While building AuditForge AI, I started thinking about the problem differently:

What if a website auditor tried to understand the website before deciding what is wrong with it?

That question changed the architecture of the project.

AuditForge AI is the project I am building around that idea.

AuditForge AI codebase shown in Visual Studio Code

A website issue rarely exists by itself

Consider a JavaScript-heavy page.
A basic auditor might report several separate problems:
content is not present in the initial HTML
indexability may be uncertain
structured data may not match the rendered page
accessibility checks based only on static HTML may miss something
the page may also have a performance problem
Those warnings are not necessarily five independent problems.
They can be different symptoms of the same underlying condition.
That led me toward a different model:

Page architecture
       ↓
Rendering behavior
       ↓
Content availability
       ↓
Indexability
       ↓
Schema / semantics
       ↓
Accessibility
       ↓
Performance
Enter fullscreen mode Exit fullscreen mode

The goal is not to hide individual findings. It is to understand their relationships.
From individual checks to a shared pipeline
The architecture gradually became closer to:

URL
 ↓
Security validation
 ↓
Audit job
 ↓
Crawl
 ↓
Browser rendering when required
 ↓
Site resource discovery
 ↓
Page / resource graph
 ↓
Engine analysis
 ↓
Finding reconciliation
 ↓
Scoring
 ↓
Root-cause relationships
 ↓
Recommendations
 ↓
Verification
Enter fullscreen mode Exit fullscreen mode

The important part is that the engines are not supposed to behave like completely isolated tools.
They share evidence.
That matters because the crawler may know something that the schema engine needs, or the resource layer may know something that the indexability engine needs.
Static HTML is not always enough
One of the biggest decisions was separating static analysis from browser analysis.
Static HTML can tell us a lot.
But it cannot reliably answer every question about a JavaScript application.
So the system has separate rendering modes:

static
browser
auto
Enter fullscreen mode Exit fullscreen mode

And the audit needs to know what actually happened.
For example:

modeRequested: browser
modeUsed: browser
rendered: true
renderingRequired: true
Enter fullscreen mode Exit fullscreen mode

This kind of provenance is important.
Otherwise an audit can accidentally present a browser-derived conclusion as though it came from the original HTML response.
A website is more than its HTML
A serious audit may need to understand resources such as:

robots.txt
sitemaps
sitemap indexes
RSS / Atom
JSON-LD
canonical URLs
hreflang
HTTP headers
llms.txt
manifests
service workers
security.txt
OpenAPI
internal links
external links
images
scripts
stylesheets
Enter fullscreen mode Exit fullscreen mode

Instead of letting every engine rediscover the same resources independently, I wanted a shared site-resource intelligence layer.
The basic idea is:

                 Site Resource Intelligence
                            │
          ┌─────────────────┼─────────────────┐
          ↓                 ↓                 ↓
       Robots            Sitemaps          HTML
          │                 │                 │
          └─────────────────┼─────────────────┘
                            ↓
                     Resource Graph
                            ↓
              ┌─────────────┼─────────────┐
              ↓             ↓             ↓
         Indexability     Schema        AI visibility
Enter fullscreen mode Exit fullscreen mode

This makes reconciliation possible.
A resource should also have a useful state.
For example:

DISCOVERED
VALID
INVALID
PARTIALLY_VALID
CONFLICTING
UNAVAILABLE
NOT_APPLICABLE
UNVERIFIABLE
Enter fullscreen mode Exit fullscreen mode

That is more useful than a simple found: true.
Crawlability and indexability are different
This was another important lesson.
A page can be crawlable without being indexable.
For example:

robots.txt
    allows crawling

meta robots
    noindex
Enter fullscreen mode Exit fullscreen mode

Those are two different signals.
Likewise, a 200 response does not automatically mean a page should be considered a healthy indexable page.
The system therefore treats indexability as a state derived from several pieces of evidence.
The model uses states such as:

INDEXABLE
INDEXABLE_WITH_WARNINGS
AT_RISK
BLOCKED
UNVERIFIABLE
Enter fullscreen mode Exit fullscreen mode

The goal is to avoid reducing a complicated website state to a single boolean.
Context matters
One of the easiest ways for an auditing tool to create noise is to apply every check to every page.
A homepage, an article, a login page, a checkout flow, a utility page, and a legal page do not necessarily have the same expectations.
That means a finding should be able to distinguish between:

ACTIONABLE
OPTIONAL
NOT_APPLICABLE
UNVERIFIABLE
Enter fullscreen mode Exit fullscreen mode

This principle affects SEO, accessibility, content, structured data, and AI visibility.
A missing signal is not automatically a failure.
Findings should contain evidence
A useful finding should answer more than:

"Your page has a problem."
It should answer:

What happened?
Where did it happen?
Why does it matter?
What evidence supports it?
What should be changed?
How can the change be verified?
Enter fullscreen mode Exit fullscreen mode

That leads to a structure like:

Finding
   ↓
Affected URL
   ↓
Observed evidence
   ↓
Root cause
   ↓
Recommendation
   ↓
Verification condition
Enter fullscreen mode Exit fullscreen mode

This is also important for AI-assisted development.
An AI coding assistant has much more useful information when it receives:

missing accessible name
affected element
HTML evidence
affected URL
recommended correction
verification condition
Enter fullscreen mode Exit fullscreen mode

than when it receives:

"Improve accessibility."
Scoring should follow evidence
Another thing I learned is how easy it is for a scoring system to become misleading.
Suppose a browser-only metric cannot be measured.
That should not automatically mean:

-10 points
Enter fullscreen mode Exit fullscreen mode

Likewise, an optional improvement should not automatically behave like a hard failure.
I prefer score changes to require stronger evidence:

deterministic violation
+
strong evidence
+
meaningful impact
=
possible score impact
Enter fullscreen mode Exit fullscreen mode

Everything else should remain clearly distinguishable as advisory or informational.
A clean website should be able to reach a clean score without being penalized simply because an unavailable measurement was unavailable.
Performance needs measured evidence
Performance analysis creates a similar problem.
There is a big difference between:

MEASURED
Enter fullscreen mode Exit fullscreen mode

and:

ESTIMATED
Enter fullscreen mode Exit fullscreen mode

For example:

LCP = 6.2s
Enter fullscreen mode Exit fullscreen mode

is measured evidence.
But:

Removing this script will save 420ms
Enter fullscreen mode Exit fullscreen mode

is much weaker if that saving has not actually been measured.
So an audit system should be explicit about evidence quality:

VERIFIED
MEASURED
ESTIMATED
HEURISTIC
UNMEASURED
UNAVAILABLE
Enter fullscreen mode Exit fullscreen mode

An estimate should never look like a measurement.
Security has to be part of the architecture
A website auditor makes network requests and can execute websites in a browser.
That makes security part of the core architecture.
An auditor needs to think about things such as:

SSRF
private IP ranges
localhost
cloud metadata services
redirect chains
browser process limits
timeouts
concurrency
network policies
resource limits
Enter fullscreen mode Exit fullscreen mode

The crawler and browser should not have completely different definitions of what is safe.
The same security policy needs to remain consistent across the audit pipeline.
The interesting part is connecting the engines
At first, a system like this can look like:

SEO checker
+
Accessibility checker
+
Performance checker
+
Schema checker
+
AI visibility checker
Enter fullscreen mode Exit fullscreen mode

But the more interesting architecture looks like:

                 Site Intelligence
                        │
        ┌───────────────┼────────────────┐
        ↓               ↓                ↓
      Pages          Resources        Graph
        │               │                │
        └───────────────┼────────────────┘
                        ↓
                 Engine analysis
                        ↓
              Finding reconciliation
                        ↓
                 Root-cause graph
                        ↓
                   Action plan
Enter fullscreen mode Exit fullscreen mode

A collection of independent checkers produces a collection of warnings.
A shared intelligence layer can start producing explanations.
What I am building toward
The long-term direction is not simply:

Run an SEO scan.
It is:

DISCOVER
    ↓
UNDERSTAND
    ↓
DIAGNOSE
    ↓
EXPLAIN
    ↓
PRIORITIZE
    ↓
PLAN
    ↓
FIX
    ↓
VERIFY
    ↓
MONITOR
Enter fullscreen mode Exit fullscreen mode

The goal is to move from:

"Here are 57 problems."
toward:
"These are the root causes affecting these pages. Here is the evidence, the affected systems, the recommended order of fixes, and how to verify each change."
That is the direction I am taking AuditForge AI.
Still building.
Still learning.
Still a lot left to improve.
But one lesson is already clear:
A good website auditor should not just detect problems. It should understand why they exist.

Top comments (1)

Collapse
 
kamayegabharat profile image
Kamayega Bharat •

One thing I’m still exploring is how far a website auditing system can go beyond detecting individual issues and actually identify the root causes connecting them.

For developers building similar tools: what would you want an audit to explain that most existing website/SEO tools don't explain well today?