DEV Community

Cover image for Your Backlinks Are Indexed But Invisible to AI Search — I Built a Free Checker to Prove It
SEO by Subham
SEO by Subham

Posted on Originally published at dev.to

Your Backlinks Are Indexed But Invisible to AI Search — I Built a Free Checker to Prove It

The Backlink That Google Loved and ChatGPT Never Heard Of

A few weeks back I published a Google Sheets tool to check backlink indexation with SerpApi — a simple, manual-only script that tells you whether a backlink has actually been picked up by Google, instead of guessing.

It worked well. One backlink I was tracking got indexed by Google in under 72 hours. Good news, right?

Then, out of curiosity, I opened ChatGPT and asked it about the exact page that backlink pointed to.

It had no idea the page existed.

Not "I don't have real-time data." Not a vague guess. It simply didn't know the content was there — despite Google indexing it days earlier.

That gap is the entire premise of this post: being indexed by Google and being visible to AI search are two completely different things, and almost nobody is measuring the second one.

So I extended my original checker to test for both.

Why "Indexed" Doesn't Mean "Visible" Anymore

For over a decade, indexation was the finish line. If Google crawled your page and added it to the index, you were in the game — rankings were just a matter of time and optimization from there.

That assumption is breaking down.

A growing share of search now happens inside AI systems — ChatGPT, Perplexity, and Google's own AI Overviews — that don't work like the traditional index. They don't list ten blue links; they synthesize an answer from a handful of sources they trust enough to cite. A page can be perfectly indexed and still never make it into that shortlist.

For anyone doing SEO or link building, that means a new failure mode nobody's watching for: a backlink that "worked" by every traditional metric, but contributes zero visibility where a growing number of searches actually happen.

I wanted a way to see that gap directly, on my own sites, without relying on vibes.

What I Built: An Indexation + AI Visibility Checker

The original tool answered one question per backlink: is this indexed?

The updated version answers three:

  1. Is the backlink indexed by Google? (unchanged from the original tool — SerpApi site: query)
  2. Does the target page appear in Google's AI Overview for the keywords it's meant to rank for?
  3. Does a general-purpose LLM (ChatGPT/Perplexity via API) recognize the page or domain at all when asked directly?

Each backlink row in the Google Sheet ends up with three simple signals instead of one, plus a combined score so I can sort and prioritize at a glance.

The Technical Walkthrough

Same stack as before: Python, the SerpApi library, and Google Sheets as the interface — no dashboard, no database, nothing to maintain.

1. Indexation check (recap)

This part is unchanged from the original tool — a straightforward site: search through SerpApi to confirm the URL is indexed:

import requests

def check_indexation(url, serpapi_key):
    params = {
        "engine": "google",
        "q": f"site:{url}",
        "api_key": serpapi_key
    }
    response = requests.get("https://serpapi.com/search", params=params)
    results = response.json()
    return bool(results.get("organic_results"))
Enter fullscreen mode Exit fullscreen mode

2. Google AI Overview check

SerpApi's Google engine returns an ai_overview block when one is present for a query. I check whether my domain shows up anywhere inside its sources:

def check_ai_overview(keyword, target_domain, serpapi_key):
    params = {
        "engine": "google",
        "q": keyword,
        "api_key": serpapi_key
    }
    response = requests.get("https://serpapi.com/search", params=params)
    results = response.json()

    ai_overview = results.get("ai_overview", {})
    sources = ai_overview.get("references", [])

    cited = any(target_domain in src.get("link", "") for src in sources)
    return {
        "has_ai_overview": bool(ai_overview),
        "domain_cited": cited
    }
Enter fullscreen mode Exit fullscreen mode

If there's no AI Overview at all for that keyword yet, I record that too — it's a useful signal on its own.

3. Direct LLM recognition check

This one is more experimental, and I want to be upfront about that. I send a small, consistent prompt to the OpenAI API asking whether the model has knowledge of the page or domain, and log the raw response for manual review rather than trying to auto-parse a yes/no:

from openai import OpenAI

client = OpenAI(api_key="YOUR_OPENAI_KEY")

def check_llm_recognition(url, domain):
    prompt = (
        f"Are you aware of the website or page at {url} "
        f"(domain: {domain})? If so, briefly describe what it's about. "
        f"If you have no knowledge of it, say so directly."
    )
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=150
    )
    return response.choices[0].message.content
Enter fullscreen mode Exit fullscreen mode

I run the same prompt structure against Perplexity's API too, since its answers are grounded in live search rather than training data alone — it tells me something different: not "does the model remember this," but "would this page surface in a real-time AI search right now."

4. Combining it into one sheet

Each backlink gets logged as a row with:

Backlink URL Indexed? AI Overview exists? Domain cited in Overview? LLM recognizes page? Notes

I don't collapse this into a single fake "score out of 100" — SEO tools do that too often and it hides more than it reveals. Instead I flag any row where Indexed = Yes but both AI columns are No. Those are the interesting rows. That's the actual gap.

What I Found Running It

Out of the backlinks I re-checked from my last batch:

  • Every single one was indexed by Google, as expected.
  • Roughly a third appeared in an AI Overview for their target keyword.
  • Almost none were recognized directly by the LLM when asked cold — which makes sense, since a backlink built last month isn't in any model's training data yet.

The AI Overview number was the useful one. It's the closest thing to a real-time "does AI search actually see this" signal available right now, and it's a much smaller percentage than the indexation rate. That gap is exactly what I was trying to surface.

Why This Is Still Manual-Only

Same reasoning as the original tool, and it applies even more here:

  • API costs add up fast. LLM calls aren't free, and neither are SerpApi credits. Automating this into a daily cron job for hundreds of URLs would get expensive quickly for very little marginal insight.
  • AI answers aren't stable. Ask the same question twice and you can get different phrasing, or a different verdict entirely. Automating on top of a non-deterministic signal just produces noisy automation, not a reliable pipeline.
  • This is a diagnostic, not a dashboard. I run it in batches when I want a checkpoint, review the raw answers myself, and move on. That's a better use of the signal than pretending it's precise enough to monitor continuously.

Manual-only isn't a limitation here — it's the correct way to use a signal that's still this noisy.

Limitations, Honestly

A few things worth being upfront about, since overselling this would defeat the point of writing it:

  • LLM "recognition" is not a real index. There's no API that tells you definitively what a model does or doesn't know — asking it directly is a proxy, not ground truth.
  • AI Overviews don't appear for every query, and their presence/absence can shift between runs.
  • None of this replaces actual traffic and conversion data. It's a leading indicator, not proof of impact.

If you're building something similar, treat every AI-visibility signal as directional, not authoritative.

Try It Yourself

The full script (indexation + AI Overview + LLM recognition check) is a single Python file plus a Google Sheet template — no server, no signup, nothing to host. If you want to run it on your own backlinks, I'm happy to share the setup in the comments or a follow-up post.

If you've noticed the same indexed-but-invisible gap on your own sites, I'd genuinely like to hear what you found — drop it below.

I write and build tools like this as part of my link-building work at SEO by Subham — if you're curious about the manual, white-hat approach behind the original indexation checker too, it's all there.

Top comments (0)