DEV Community: Mehul Jain

Your insurance brand is missing from the AI answer. Here is how to measure why.

Mehul Jain — Tue, 07 Jul 2026 08:07:54 +0000

If you work on growth or SEO for an insurance brand that is not State Farm, you have probably typed your own category into ChatGPT and watched it recommend seven companies, none of them yours. The instinct after that is to go rewrite a landing page. That is almost always the wrong first move, because you have not measured anything yet. You do not know which questions you lose, which engines you lose them on, or which sources those engines are actually reading.

This is a measurement problem before it is a content problem. Here is how to instrument it.

What the data says you are up against

Geology ran the version of this test that most teams never get around to. In its June 2026 insurance study, 100 insurance buyer prompts went through ChatGPT, Perplexity, Gemini, and Google AI Overviews, tracking 15 insurers. Incumbents took 78 to 82 percent of insurer mentions on ChatGPT, Perplexity, and Gemini, and 66 percent on the more balanced Google AI Overviews.

Two details from that study are the ones you will build your measurement around:

The engines cite external surfaces, not carrier sites. On ChatGPT, only about 1 percent of the roughly 16 sources per answer were carrier-owned. The rest were aggregators and editorial (MoneyGeek, Forbes, NerdWallet). On Perplexity, Reddit was the single most-cited source, in 75 of 100 answers.
Challengers win by niche. Next Insurance took its share almost entirely in small business; Lemonade in renters and pet. General-purpose challengers barely registered.

You cannot act on either of those until you have your own version of the numbers for your own brand. So build it.

Step 1: build a prompt set that mirrors how people actually ask

Do not test one query. Insurance buyers ask in predictable shapes, and your visibility differs wildly across them. Build 40 to 100 prompts across four buckets:

Category: "best renters insurance", "cheapest small business insurance"
Problem: "how do I insure a side business", "renters insurance for a home studio"
Comparison: "Lemonade vs State Farm renters", "best alternatives to GEICO"
Niche: the specific situations where you think you have a right to win

Store them as data, not as a doc, so the run is repeatable:

[
  { "id": "cat-01", "bucket": "category", "prompt": "best renters insurance 2026" },
  { "id": "niche-04", "bucket": "niche", "prompt": "renters insurance for a freelance photographer with gear" }
]

Step 2: run each prompt across the engines and log two things

For every prompt on every engine, record two fields: were you named, and which sources were cited. That second field is the one most audits skip, and it is where the strategy actually comes from.

{
  "prompt_id": "cat-01",
  "engine": "perplexity",
  "brands_named": ["State Farm", "Lemonade", "Progressive"],
  "you_named": false,
  "sources_cited": ["reddit.com/r/insurance/...", "nerdwallet.com/...", "moneygeek.com/..."]
}

Run it manually for a first pass if you have to. If you want it repeatable, the Perplexity and Gemini APIs return citations directly, and for ChatGPT and AI Overviews you can script the browser. The point is a table you can rerun next month and diff.

Step 3: compute the two numbers that matter

From that log, derive:

Share of voice per bucket = the fraction of prompts in a bucket whose answer named you. Do this per engine. You will almost certainly find your share is near zero on category prompts and higher on a niche or two. That split is your whole strategy.
Cited-source mix = across the prompts you lose, which domains keep getting cited. Rank them. This is your target list. If MoneyGeek, NerdWallet, and one subreddit account for most of the citations in your niche, those three surfaces are where your visibility is decided, not your blog.

Step 4: act on the surfaces, not the homepage

Now the fixes are obvious because the data pointed at them.

Pick the one bucket where your share of voice is already non-zero and go all in. That is the niche the engines are willing to let a challenger own. Next Insurance and Lemonade did not win by breadth. They won one room.
Get accurate, specific facts about your niche onto the cited surfaces. For the aggregators, that means being includable: clear coverage details, real pricing ranges, honest eligibility rules, in the liftable format a writer needs. For community surfaces like Reddit, it means being genuinely useful in real threads, not planting anything, because that gets caught and it poisons the exact signal you want.
Re-run the whole test on a fixed cadence and watch share of voice per bucket move. That diff is the only proof your work did anything.

Geology's breakdown of how insurance brands win in AI answers covers the content side of this in more depth, and its insurance AI-visibility service is the same measure-then-act loop run end to end. But the sequence starts here, with measurement. You cannot fix a visibility gap you have not located, and for insurance the gap is almost never on the page you were about to go rewrite.

Mehul Jain writes about generative engine optimization at Geology, where the team studies how AI engines decide which brands to recommend. This piece draws on Geology's June 2026 insurance visibility study.

Build a GEO strategy in 2026: four layers, in the order that compounds

Mehul Jain — Wed, 24 Jun 2026 17:23:19 +0000

Most GEO programs stall in the same spot. A team reads that schema markup helps AI engines parse a page, so they bolt FAQ and HowTo markup onto a dozen thin articles and wait. Nothing moves. The mention rate in ChatGPT stays flat, Perplexity keeps citing a competitor, and the team quietly concludes that generative engine optimization is hype. The real problem is that they started at layer three.

Rachel Whitmore made the case for the fix in How to Build a GEO Strategy from Scratch in 2026 on the Geology blog. Her framework has four layers, and the point that most teams skip is that the layers are a dependency chain, not a menu you pick from. Schema gets ignored when the content underneath it has no depth. Monitoring tells you nothing if you never recorded a baseline. This piece takes her four layers and works through why the order is the whole game.

A quick note on terms first. Classic SEO ranks your pages on Google. GEO earns your brand a mention inside an AI answer from ChatGPT, Perplexity, or Gemini. The work overlaps more than it looks like it does, and if you want the clean split between the two, GEO vs SEO covers it. The four layers below are the GEO half.

Layer one: get a baseline before you touch anything

Before you write a word or add a single tag, measure where you stand. Run about 20 prompts across ChatGPT, Perplexity, and Gemini. Split them into three buckets: category prompts ("best payroll software for remote teams"), problem prompts ("how do I handle multi-state payroll tax"), and direct brand prompts ("is Acme good for payroll"). For each answer, write down whether you were mentioned, which competitors showed up, and which sources got cited.

Then pick one metric you will defend for 90 days. Mention rate across the category prompts is a good default. The number itself matters less than committing to it, because every later decision points back to whether it moved.

This is layer one for a reason. You cannot claim a win you never measured, and "we feel more visible" is not a result anyone can act on.

Layer two: build authority where you actually have depth

Pick three to five topics where you can go deeper than an aggregator listicle ever will. For each one, write a pillar page plus five to eight supporting articles, and publish on a steady weekly cadence rather than in occasional bursts.

The constraint that does the work here is depth. AI engines synthesize answers from sources that demonstrate they understand the question, not from pages that restate the obvious. Every piece in a cluster should add something a reader could not get from the first page of search results. If a draft only summarizes what is already common knowledge, it will not get cited, and it dilutes the cluster around it.

This is also why breadth is a trap. Forty shallow posts spread across forty topics give an engine no reason to treat you as the authority on any of them.

Layer three: add structured signals on top of real content

Now the schema work pays off, because there is something underneath it worth parsing. Three jobs sit in this layer.

Fix entity consistency first. Your brand name, description, and key facts should read the same on your site, your LinkedIn, your Crunchbase entry, and anywhere else an engine cross-references you. Inconsistent entities make a model less confident about who you are.

Add the schema that fits the content: Organization, Article, FAQ, and HowTo where they genuinely apply. Format pages for extraction with clear headings, tables, and numbered steps, so an engine can lift a clean answer straight out of the page. Then reinforce internal linking so the cluster reads as a connected body of work instead of scattered posts.

Done in this order, schema amplifies depth. Done at layer one, it decorates pages that have nothing to say.

Layer four: monitor, and tie every number to an action

Rerun the baseline prompts weekly. Track mention rate, the sentiment of those mentions, which sources the engines cite, and your share of voice against the competitors who keep appearing. The discipline that separates a real program from a dashboard is connecting each metric to a specific action. If Perplexity cites a competitor's comparison page, that is the brief for next week's article, not just a line on a chart.

What this looks like in practice

Whitmore's worked example is a company called Northwind Ledger, starting from a 20 percent mention rate. They built two clusters, one on multi-state payroll and one on expense categorization, fixed their brand consistency across platforms, and reviewed their numbers every week. The visibility gain came from the layers reinforcing each other, not from any single tactic firing on its own.

A 90-day version of the same plan: weeks one and two for the audit and baseline, weeks three through six to build the first clusters and fix entity consistency, weeks seven through ten to add structured data and tighten internal linking, and weeks eleven and twelve to review what moved and plan the next phase.

Each engine rewards something slightly different

The four layers hold across platforms, but the emphasis shifts. ChatGPT rewards entity consistency and long-term depth. Perplexity favors citation-ready formatting and a clear, liftable answer. Gemini cares about crawlability and what renders in the initial HTML. Google AI Overviews tends to synthesize from pages that already rank, so classic SEO and GEO meet most directly there.

Where teams go wrong

The failure patterns are consistent: skipping the baseline so no win is provable, chasing breadth instead of depth, adding schema to thin content, publishing in sporadic bursts, and ignoring entity consistency across platforms. Each one is a layer attempted out of order, or skipped.

That is the whole argument. The four layers are not four projects you can run in parallel and bolt together at the end. They are a sequence where each one only works because the one before it is already in place. Consistency over a quarter reads to an engine as ongoing authority, which is exactly what a single burst of effort cannot fake.

For the full framework, including the prompt buckets and the worked example in detail, read Rachel Whitmore's original: How to Build a GEO Strategy from Scratch in 2026.

Mehul Jain writes about how search is shifting from keywords to model-mediated discovery at Geology. This article builds on the GEO strategy framework by Rachel Whitmore.

Building an OG Previewer: Per-Platform Fallback Chains

Mehul Jain — Thu, 04 Jun 2026 16:42:52 +0000

The first version of an OG previewer that anyone sketches reads the og: tags off a page and renders them. That design survives until the first real page hits it, because two assumptions in it are wrong. The first is that the tags are present. A large share of pages in the wild are missing at least one of og:title, og:description, or og:image. The second is that there is one right way to render them. There is not. Facebook, X, LinkedIn, and Slack each resolve the same set of tags through their own fallback chain, so the card a user sees depends on which platform built it.

So a previewer that wants to tell the truth cannot just print the tags. It has to model what each platform does when a tag is missing, then show you the four different rectangles those four chains produce from one page. This is a write-up of how we built that.

Fallback chains are data, not code

Each platform's resolution is an ordered preference list per field: try this tag, then that one, then the page-level fallback. The page-level fallbacks are just keys too. We write them as page:title and page:description so they sit in the same list as the tag keys. The chains we model look like this.

Facebook and LinkedIn share a chain:

Title: og:title, then page:title (the page's <title> text).
Description: og:description, then page:description (the page's meta description).
Image: og:image, with no fallback. If it is missing, there is no image.

X resolves its own layer first, then falls through to Open Graph, then to the page:

Title: twitter:title, then og:title, then page:title.
Description: twitter:description, then og:description, then page:description.
Image: twitter:image, then og:image.
Card type: twitter:card, else default to summary.

Slack follows the Open Graph resolution, the same as Facebook and LinkedIn for title and description. It also surfaces og:site_name above the card, and shows no image when og:image is missing.

The thing to notice is that these are the same shape: an ordered list of keys, walked until one produces a value, with the page-level fallbacks living in the list as page: keys. That shape is the whole design. Instead of a render function per platform with the fallbacks baked into branches, each chain is a list of keys, and one resolver walks any list. When a key starts with page: the resolver reads from the page dict instead of the tags:

def resolve(field_chain, tags, page):
    # field_chain: ordered keys to try, e.g.
    #   ["twitter:title", "og:title", "page:title"]
    # tags: extracted meta tags; page: {"title": ..., "description": ...}
    for key in field_chain:
        if key.startswith("page:"):
            value = page.get(key.split(":", 1)[1])
        else:
            value = tags.get(key)
        if value:
            return value
    return None

Adding a platform, or fixing a chain when a platform changes its behavior, is editing a list of keys, not rewriting render logic. All four previews come out of the same resolver fed four different chains.

og:image present is not og:image working

The single most common reason a card looks broken is the image, and the failure is sneaky because the tag is right there in the markup. og:image points at a URL, the validator that only reads tags says "image: present", and the card still renders empty. The tag being present and the image being reachable are two different facts, and only the second one matters to the platform.

So the previewer does not trust the markup. It makes a server-side HEAD request to the og:image URL, with a five-second timeout, following redirects, and flags any image that does not come back with a 2xx. That check catches the failures that the tag-presence check misses:

Relative URLs. og:image set to /img/card.png. Loads fine on the origin page where the browser resolves it against the current host, and resolves to nothing when a platform crawler with no base context fetches it.
CDN hotlink blocks. The image exists, but the CDN refuses requests whose referer is not the origin site. Your browser passes, Facebook's fetcher gets a 403, the card is grey.
Redirects to a login or interstitial. Following the redirect lands on an HTML page, not an image. The HEAD request follows it and sees the wrong thing come back.

What the check deliberately does not do is fully validate the image. A HEAD request confirms the URL answers and roughly what it answers with; it does not download the bytes, decode the image, and measure its dimensions. Confirming an image is actually 1200 by 630 means fetching and decoding it, which is a heavier operation than a header probe and a different kind of check. So we recommend 1200 by 630 (the 1.91 to 1 ratio platforms crop to) as guidance, and we flag what a header check can actually catch: the URL that does not resolve at all. That honesty matters more than pretending the tool measures something it does not.

A severity model with two real levels

Not every missing tag is equally bad, and the report should say so. The split we settled on maps to a real distinction: will the card render wrong, or will it not render the way it should at all.

Errors: missing og:title, og:description, og:image, or og:url. These are the load-bearing tags. Miss one and the card is either broken or built entirely from a guess.
Warnings: missing twitter:card. X still renders, it just falls back to the default summary card, the small-thumbnail layout, when you probably wanted summary_large_image. The card works; it is smaller and less prominent than intended. The warning text says exactly that: X will use a default card type.

The line we drew is between "the card will look wrong" and "the card will render in a less ideal form." A missing og:image is the first kind, an empty rectangle, so it is an error. A missing twitter:card is the second kind, a working but downgraded card, so it is a warning. Keeping those two states distinct is what lets someone reading the report triage: fix the errors before a share goes out, schedule the warnings.

Parsing hostile HTML

The clean case, a well-formed head with tidy meta tags, is the case that never causes bugs. Real pages are messier, and a previewer that only handles the clean case is a demo, not a tool. The pages we have to survive include:

Meta tags placed in the body instead of the head, because a CMS or a tag manager injected them after the fact.
Pages with no <head> element at all, just tags floating in malformed markup.
The same property declared two or three times, an og:image from the theme and another from an SEO plugin, with no agreement on which wins.

The rule we follow is to parse leniently and resolve the way a platform would, not the way a spec says a document should be. We collect meta tags wherever they appear, not only inside the head, because the platform crawlers are lenient too and a tag in the body still gets used. For duplicated properties we take a defined position rather than throwing, so the preview is deterministic instead of dependent on parse order. The goal is never to grade the HTML. It is to render the card the platform would render from that HTML, warts and all, so what you see in the previewer is what your audience will see.

Try it

The OG Previewer runs everything above on a single URL: four platform cards from four fallback chains, the server-side image reachability check, and the error and warning report. The tag audit underneath it pairs naturally with the Meta Tag Analyzer, which scores the full tag set rather than just the share-card subset, if you want the complete head graded in one pass.

If there is one idea here worth carrying into something you build, it is modeling per-consumer behavior as ordered data instead of branching code. The four platforms looked like four rendering problems. They turned out to be one resolver and four lists. Any time you find yourself writing a function per consumer with the differences baked into if-statements, check whether the differences are really just data the consumers disagree on. Usually they are, and pulling them out into a table makes the next consumer a row instead of a rewrite.

Mehul Jain is an AI entrepreneur and product builder. He works on Geology, a GEO platform.

Validating JSON-LD Beyond Syntax: Required Properties per schema.org Type

Mehul Jain — Thu, 04 Jun 2026 16:31:05 +0000

json.loads returning without an exception tells you exactly one thing: the bytes are well-formed JSON. It tells you nothing about whether the block means anything. An Article block can parse perfectly and have no author. A Product can parse and carry a ratingValue of "4.5", a string where the consumer wants a number. The parser is happy. The AI engine reading the page is not, because the block it just parsed does not assert the fact it exists to assert. Syntax validation is the floor. We built a validator that has to clear it and then keep going.

So the validator runs three layers, and each one catches a class of failure the layer before it waves through:

Syntax. Does the block parse as JSON at all. Cheap, binary, first.
Type rules. Given the @type, are the properties we treat as required for that type actually present, are the dates real ISO 8601, do the URL fields hold URLs. The "required" set follows Google's rich-results requirements, not the schema.org vocabulary, because that is what governs whether an engine trusts the block.
Content alignment. Does the markup match the page it is on, or is it claiming a headline and author that appear nowhere in the rendered text.

This piece is mostly about layer two, because that is where "valid JSON" and "valid schema" stop being the same sentence.

Encoding type rules as data

The temptation with type rules is to write them as code: a function per type, a branch per property, a growing if @type == "Article" tree. We did not do that. The rules are data, a table keyed by type, each row listing required and recommended properties. Checking a block is a lookup and a set comparison, not a walk through a switch statement. Here is the shape, for three of the types:

Type	Required	Recommended
`Article`	`headline`, `author`	`datePublished`, `image`, `publisher`
`Product`	`name`	`image`, `offers`, `aggregateRating`
`LocalBusiness`	`name`, `address`	`telephone`, `openingHours`, `priceRange`

One thing the table is not: a transcription of the schema.org spec. schema.org itself rarely mandates anything, so our "required" column is an opinionated baseline that follows Google's rich-results requirements, since that is the bar that decides whether an engine trusts the block in practice. The headline and author we mark required on Article are required by Google, not by the vocabulary.

The validator covers 15 common types this way. Alongside the three above, the table holds BlogPosting and NewsArticle (both, like Article, requiring headline and author), FAQPage, BreadcrumbList, Event, Recipe, VideoObject, HowTo, Organization, WebSite, and WebPage. The payoff of rules-as-data is the maintenance story. Adding a new type means adding a row to the table. It does not mean writing a function, finding the right place to call it, and remembering to wire it into the scorer. The check logic is written once and runs against every row. New type, new row, done.

The sneaky failures

Missing properties are the obvious failures and the easy ones to report. The annoying class is the values that are present, look fine to a human, and are wrong to a parser. Three keep showing up:

ISO 8601 dates. The standard wants 2026-05-18, optionally with a time and offset. People write 05/18/2026, May 18, 2026, 18-05-2026, 2026/05/18. Every one of those is a string a human reads correctly and a date parser rejects. So the validator does not just check that datePublished exists; it checks that the value is parseable as ISO 8601, and flags it as a type error when it is not.
Relative URLs in URL fields. An image of /img/header.jpg or a url of ../about is a valid string and a broken URL field. An engine cannot resolve a fragment with no origin. The check is for an absolute http(s) URL, not just a non-empty string.
Numbers shipped as strings. "ratingValue": "4.5", "price": "29.99", "reviewCount": "212". JSON cannot tell you these are wrong, because they are valid JSON strings. The type rule knows the field expects a number and reports the mismatch.

The common thread: none of these is a JSON error. They are all schema errors that pass straight through a syntax check, which is the entire reason layer two exists.

Scoring per block, averaging per page

A pass/fail verdict is too blunt to be useful. A block missing one recommended property is not in the same state as a block that does not parse, and a single number should say so. So each block scores out of 100, split four ways:

Syntax: 40. Valid JSON or zero. The largest slice, because nothing else matters if the block does not parse.
Required-property coverage: 30, scaled. If a type requires four properties and three are present, the block earns 30 times 3/4, which is 22.5.
Recommended-property coverage: 20, scaled. Same coverage math, applied to the recommended set.
Content alignment: 10. Does the markup match the page.

The page score is the average across all blocks on the page. The deliberate choice is to score per block rather than mashing every block into one global tally. A page with three clean blocks and one gutted one should not read as uniformly mediocre, and it should not read as fine either. Collapse those four into a single page number and you get something around 75, which describes none of them: the three good blocks look worse than they are and the broken one disappears into the average. Keep them separate and the 22 sits next to the three 95s, so you know precisely which block to open.

LLM-assisted suggestions

Layers one through three are deterministic. On top of them, there is a suggestion layer that is not. Gemini reads the page text alongside the schemas already present and proposes markup the page could carry but does not. The canonical case: the page has a visible FAQ section, real questions with real answers, and no FAQPage block describing it. A rules engine cannot find that, because there is no block to validate; the opportunity lives in the gap between the content and the markup, and spotting it takes reading the prose.

The hard design rule on this layer is that it is best-effort and must never block the validation results. The model call can time out, refuse the request, or come back with something we reject. When it does, the validator still returns the full deterministic output: every block's score, every error, every warning. You lose the suggestions for that run and you keep everything the rules produced. The probabilistic layer is strictly additive. It can fail without taking the rest of the report down with it, and that property is wired in by design rather than hoped for.

Multiple blocks and @graph

Real pages do not ship one tidy block. A WordPress site with a couple of plugins active can emit four or five separate <script type="application/ld+json"> tags, one from the theme, one from the SEO plugin, one from a reviews widget. So in URL mode the validator extracts every such tag on the page and scores each independently.

The other shape is the @graph bundle: one script tag whose payload is an object with a @graph array holding several entities, an Organization, a WebSite, a WebPage, a BreadcrumbList, all in one block. This is common output from WordPress SEO plugins, which like to ship the whole site graph in a single tag.

Here is an honest limitation. The type rules key off a block's top-level @type. When a tag's top level is a JSON array, we split it and each element becomes its own block. But a @graph bundle is a single object whose top level has no @type of its own, so the validator treats the whole bundle as one block of type Unknown. Unknown types get the syntax and @context checks only; the per-type rules never reach the Organization, WebSite, and the rest sitting inside the @graph. The bundle is checked, but its inner entities are not individually walked.

The workaround is mechanical. Copy a single entity out of the @graph array, paste it into the validator on its own, and it parses as a block with a real top-level @type, so the type rules apply. Pull the Organization node out, run it through paste mode, and you get the required-property check that the bundle does not give you.

Try it

The Schema Validator runs all of this on a URL or on JSON-LD you paste straight in: syntax, type rules, alignment, score per block, and the best-effort suggestions on top. If a page comes back with no schema to validate at all, generate a typed first draft with the Schema Generator, fill the placeholders, and run the result back through the validator to confirm your edits hold.

The thing worth stealing from how this is built is the layering, and the discipline about which layer is allowed to fail. Syntax, rules, and alignment are deterministic and load-bearing, so they run first and always return. The model sits on top, adds what only reading prose can add, and is fenced off so its failure is a missing section rather than a broken report. Decide up front which checks must never be probabilistic, make those the foundation, and let the clever part be the part you can afford to lose.

Mehul Jain is an AI entrepreneur and product builder. He works on Geology, a GEO platform.

Auto-Generating JSON-LD: Page Signals, Type Heuristics, and a Careful Gemini Prompt

Mehul Jain — Thu, 04 Jun 2026 16:23:22 +0000

The naive version of this tool is one prompt: "Here is a URL, write the JSON-LD for it." We tried that mental model early and threw it out. An LLM handed a bare URL will produce schema that looks perfect and is quietly wrong. It guesses an author when the page has none. It invents a publication date. On a commerce page it will cheerfully write a price that appears nowhere in the markup. The output validates, parses, and ships, and then an AI engine reads a fabricated author name as a confirmed fact. For a tool you paste straight into production, that is the worst possible failure, because it is invisible until something downstream cites the lie.

So we built the pipeline backwards from that risk. The model never sees a raw URL and never decides what the facts are. By the time Gemini runs, the page has already been read, the facts have already been extracted, and the page type has already been decided by deterministic code. The model's job is narrow: take known facts and a known shape, and emit well-formed JSON-LD. Everything that could be hallucinated is settled before the model is allowed to write a word.

Signal extraction: read the page first

Step one is fetching the page and pulling structured signals out of the DOM. No model here, just parsing. We extract a fixed set of things:

title, meta description, canonical URL: the page's own identity claims.
headings, up to 30: the outline, capped so a pathological page cannot blow up the payload.
images, up to 10: src and alt for each, since alt text is the only image content a schema block can carry.
author: searched across three selector families, namely rel="author", the .author and .byline classes, and itemprop="author".
publication date: read from <time> elements and the article:published_time meta tag.
breadcrumbs: from a nav ol or any breadcrumb-named class.
price markers: price-related classes and properties, the signal that a page is selling something.
FAQ markers: faq and accordion classes, plus headings phrased as questions.

The output of this stage is a plain signal bundle. For a photography tutorial it might look like this:

{
  "title": "Shooting in Manual Mode: A Beginner's Walkthrough",
  "metaDescription": "Learn aperture, shutter, and ISO in three steps.",
  "canonical": "https://example.com/blog/manual-mode-walkthrough",
  "headings": ["Step 1: Set your aperture", "Step 2: Pick a shutter speed", "Step 3: Dial in ISO"],
  "images": [{ "src": "/img/aperture.jpg", "alt": "Aperture ring on a lens" }],
  "author": "Dana Okoye",
  "publishedTime": "2026-04-22",
  "breadcrumbs": ["Home", "Blog", "Photography"],
  "priceMarkers": false,
  "faqMarkers": false
}

Notice what this bundle is: facts, not interpretation. Either author is a string we found in the DOM or it is null. We never fill it. That null is what protects the downstream steps from inventing one.

Heuristics before the LLM

With the signals in hand, we decide the page type ourselves, in plain code, before any model call. It runs as an ordered ladder and stops at the first match:

price markers present → product
FAQ markers present → faq
author present and date present → blog_post
title contains "about" or "team" → organization
title contains "contact" → local_business
two or more "Step N" headings → how_to
otherwise → generic

Order matters because the conditions overlap. A product page can also have an author and a date; checking price first means it resolves to product rather than getting misclassified as a blog post three rungs down. The ladder reads top to bottom and the first hit wins.

Three reasons this happens before the model, not inside it:

Deterministic grounding. When the model is told "this is a how_to," it writes schema for a known shape. It is not guessing the shape and the content in the same breath. Splitting "what kind of page is this" from "fill in the fields" removes the highest-variance decision from the part of the system that can hallucinate.
Cheaper calls. A rule that reads a boolean is free. Spending an LLM call to classify a page you can classify with a string match is waste at scale.
A constrained LLM job. The smaller the question we hand the model, the more reliable its answer. "Produce a HowTo block from these steps" is a tight prompt. "Figure out what this page is and write schema for it" is an open-ended one, and open-ended is where models drift.

The Gemini step

Now the model runs. Gemini Flash receives the signal bundle plus the detected type, and returns typed JSON-LD: one block per applicable type, each with a confidence score and a short explanation of why that type fit. We run it in JSON mode, so the response is constrained to valid JSON and our parser never has to scrape a code fence out of prose or recover from a stray sentence the model added. Structured output goes in, structured output comes back.

The rule we press hardest in the prompt is on missing facts. The model is instructed that any field it was not handed a value for must be emitted as an explicit, clearly labeled placeholder. It is forbidden to substitute a plausible value of its own. If the signal bundle has author: null, the author field comes back as a placeholder token, never as a name the model decided sounded right. This is the whole safety property of the tool stated as a prompt constraint: a gap stays a visible gap, marked for a human to fill, instead of becoming a confident fabrication that reads as fact. We would rather hand someone a block with three placeholders to complete than one with three invented values to discover later.

Degrading gracefully

The model call can fail. The API times out, refuses the request, or returns something we reject. When that happens, the tool does not show an error page and send the user away empty-handed. It still returns the type detection from the heuristic ladder and template JSON-LD blocks for that type, with the same placeholder structure the model would have produced. You lose the model's per-field confidence scoring and its explanation, and you keep a correctly typed skeleton you can fill in by hand. An empty, well-shaped suggestion beats a 500. The deterministic half of the pipeline carries the result on its own when the probabilistic half is unavailable.

What comes out

For the photography tutorial above, with the type resolved to how_to and an author present, the generated block looks like this, abbreviated:

{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "Shooting in Manual Mode: A Beginner's Walkthrough",
  "step": [
    { "@type": "HowToStep", "name": "Set your aperture", "text": "PLACEHOLDER_STEP_DETAIL" },
    { "@type": "HowToStep", "name": "Pick a shutter speed", "text": "PLACEHOLDER_STEP_DETAIL" },
    { "@type": "HowToStep", "name": "Dial in ISO", "text": "PLACEHOLDER_STEP_DETAIL" }
  ],
  "author": { "@type": "Person", "name": "Dana Okoye" }
}

The step names came off the headings, the author came off the DOM, and the per-step detail text the page did not expose cleanly is left as a visible placeholder. Nothing in that block is a value the model wished into existence.

One caveat before you dismiss the HowTo type: Google retired HowTo rich results in 2023, so this markup earns no badge in the SERP anymore. We keep emitting it anyway, because the GEO use is different. The block still hands an AI engine an ordered procedure it can reproduce faithfully, step for step, without reconstructing the sequence from prose. That is the point here, not chasing a rich-result enhancement that no longer exists. The full output ships as a ready-to-paste <script type="application/ld+json"> tag, across the type range the pipeline supports: Article and BlogPosting, Product, FAQPage, LocalBusiness, HowTo, BreadcrumbList, Organization, and WebPage or WebSite.

Try it

The Schema Generator runs this whole pipeline on a URL you give it: extract, classify, generate, score. Then, after you fill the placeholders, run the result through the Schema Validator to confirm your edits did not break the block against schema.org rules.

If you take one idea from how this is built, take the ordering. The instinct with a capable model is to hand it the whole problem and admire what comes back. The better discipline is to figure out which decisions must never be probabilistic, classification and fact-finding here, settle those in code, and let the model do only the bounded part that remains. The pipeline is more reliable not because the model is weaker but because we gave it less room to be wrong.

Mehul Jain is an AI entrepreneur and product builder. He works on Geology, a GEO platform.

Building a Sitemap Health Checker: Discovery, Index Files, Parallel Sampling

Mehul Jain — Thu, 04 Jun 2026 16:15:56 +0000

We built the Sitemap Checker around a budget before we wrote a line of parsing code. The tool is free, it takes anonymous users with no signup, and someone pasting a domain expects an answer in under a minute, not a progress bar they walk away from. That single constraint decided most of the architecture, because it ruled out the obvious approach.

The obvious approach is to crawl. Fetch the sitemap, request every URL it lists, report what is broken. That works fine for a 40-page brochure site and falls apart everywhere else. A sitemap can hold up to 50,000 URLs per file, and an index can point at many such files. Requesting all of them, even fast, blows the time budget and hammers the target server with traffic it did not ask for. So full crawls were off the table by design, not as a limitation we apologize for. The interesting work was figuring out what you can learn about a sitemap's health without reading every page in it.

Finding the sitemap before you can check it

You cannot check a file you cannot locate, and there is no single guaranteed address for a sitemap. So discovery runs as an ordered chain, and the order encodes which source to trust.

First we read robots.txt and look for a Sitemap: directive. This is the authoritative answer. If the site owner declared where their sitemap lives, that declaration wins, because they know something a convention does not: where they actually put it. A Sitemap: line can point anywhere, including a path you would never guess.

If robots.txt names nothing, we fall back to the two conventional locations in turn: /sitemap.xml, then /sitemap_index.xml. These are guesses, but they are good guesses, because most generators emit one or the other by default.

The order matters because the alternative is wrong. If you check /sitemap.xml first and find a stale leftover, you would report on the wrong file while the real one sits declared in robots.txt, unread. Declared beats conventional, conventional beats nothing, and the chain stops at the first hit.

Two XML shapes, one of them recursive

Once you have a sitemap, it is one of two shapes, and you do not know which until you parse the root element.

A <urlset> is a flat sitemap: a list of <url> entries, each with a <loc> and optionally <lastmod>, <changefreq>, and <priority>. You read the entries and you are done.

A <sitemapindex> is an index: a list of <sitemap> entries, each pointing at a child sitemap that is itself a <urlset>. To get the actual URLs you have to fetch the children and parse each one. An index can list dozens of children, and fetching all of them serially blows the budget.

So we cap it: for an index, we fetch up to five child sitemaps and parse those. Five is enough to characterize the sitemap's health and freshness without turning a one-minute check into a five-minute one. A site whose first five child sitemaps are clean and a site whose first five are full of broken entries are telling you very different things, and you do not need all forty to know which you are looking at. From whichever shape we land on, we extract loc, lastmod, changefreq, and priority per URL, which is everything the later checks run on.

Sampling instead of crawling

The liveness check is where the budget constraint becomes a statistics decision.

We do not request every URL. We take a random sample of min(50, total) URLs and check those. Fifty for any sitemap large enough to have fifty; the whole thing if it is smaller. Each sampled URL gets a HEAD request, run through a pool of ten workers in parallel so the fifty checks finish in roughly the time of the slowest five, not the sum of all fifty.

HEAD rather than GET because we only want the status code, not the body. We sort each response into one of three buckets: healthy for a 2xx, redirected for a 3xx, broken for a 4xx or 5xx. The broken ones come back with their status codes attached, because "broken" alone is not actionable and "404 on /old-page" is.

The framing to be honest about: a random fifty-URL sample bounds the broken-rate estimate well enough to assign a health grade. If 4 of 50 sampled URLs are broken, the file is in a different state than one where 0 of 50 are, and that difference is what a grade should capture. What the sample does not do is enumerate every broken URL in a 50,000-entry file. It is a measurement of the rate, not a complete inventory of the failures, and we say so rather than implying the broken list is exhaustive.

Servers that lie to HEAD requests

Here is the mess the clean version of this story leaves out: not every server answers a HEAD request honestly.

A HEAD is supposed to return exactly what a GET would, minus the body, so the status code should be identical. Plenty of servers do not implement it that way. Some reject HEAD outright with a 405, some return a 403, some behave differently than they would for the GET that a real visitor sends. So a URL that serves a perfectly good page to a browser can come back as broken in our sample purely because the server mishandles the method we used to probe it. That is a false negative, and we cannot fully eliminate it without sending full GETs, which puts us back over the time budget.

What saves the grade from this is the scoring shape. We never let a single failed check zero anything, because the score is built on the broken ratio, not a broken count. The liveness contribution scales as one minus the broken ratio. A couple of false negatives in a fifty-URL sample nudge the ratio slightly; they do not collapse the score. A sitemap that is genuinely fine but lives behind a HEAD-hostile server loses a few points, not the whole grade. The math degrades gracefully on purpose, because the failure mode we most wanted to avoid was confidently telling someone their healthy site is broken.

The rubric, in full

The score is additive to 100, and every component maps to something the checks above actually measured:

Sitemap found: +20. The file exists and was located. Discoverability is the precondition for everything else, so it earns real points on its own.
lastmod coverage × 20. The fraction of URLs carrying a real last-modified date, scaled to 20. Full coverage is the full 20; half coverage is 10.
Has date info: +15. A flat bonus when the file carries date metadata at all, separate from how complete that coverage is.
(1 minus broken ratio) × 20. The liveness term. A clean sample earns the full 20; the score scales down with the broken rate rather than dropping off a cliff.
No duplicates: +10. No URL listed more than once.
Within the 50,000 size limit: +15. No single file exceeds the protocol's per-file ceiling.

That sums to 100: 20 + 20 + 15 + 20 + 10 + 15. The choice worth defending is the +20 for simply being found. It looks generous for a check that did no real work yet. But discovery is the hardest gate to clear: a sitemap a crawler cannot find is worth zero regardless of how pristine its contents are. Awarding points for "found at all" reflects that the rest of the rubric is meaningless until this passes, so it deserves weight, not a footnote.

Try it

The Sitemap Checker is free and takes a root domain, no signup. You get the totals, the lastmod coverage, the freshest and stalest entries, the sampled broken list with status codes, and the score broken down by the components above. The companion on the permission side is the AI Crawler Checker, which reads robots.txt to tell you which AI bots are allowed in at all.

If you build sitemap tooling yourself, the one decision we would press on is the sampling cap. It is tempting to treat fifty as a placeholder and crank it to "be thorough." Resist that. The reason fifty holds up is that you are estimating a rate, and the precision of a rate estimate improves with the square root of the sample, so going from 50 to 500 buys you a factor of about three in precision for ten times the requests and ten times the load on someone else's server. The honest version of thorough is a sample sized to the question you are answering, not the largest number your timeout allows. We picked fifty and a one-minute budget on purpose, and the rubric was designed to mean something inside those limits rather than pretend they do not exist.

Mehul Jain is an AI entrepreneur and product builder. He works on Geology, a GEO platform.

Detecting Q&A Patterns and Heading Trees in Raw HTML

Mehul Jain — Thu, 04 Jun 2026 16:02:17 +0000

When we started building the Content Structure Analyzer, the first thing we did was throw out the question everyone expects a content tool to answer. We did not want to score whether a page was good content. Good is subjective, it needs a human, and there are a hundred tools that already pretend to measure it. The question we actually cared about was narrower and mechanical: can a model lift an answer out of this page. That one has a yes-or-no shape, which means you can write code that estimates it.

So the design problem became: what observable properties of raw HTML predict whether an extractive model can isolate a clean answer span. We settled on six measurable proxies, each weighted by how much we believe it moves that outcome:

Heading structure, 25 percent. The nested heading tree and its defects.
Content depth, 20 percent. Body word count from paragraph text.
Q&A patterns, 20 percent. Question-form headings, definition lists, FAQ sections.
Semantic HTML, 15 percent. Landmark element coverage.
Lists, 10 percent. Ordered and unordered list count.
Media, 10 percent. Alt text coverage across images.

None of these is a direct measurement of extractability, because you cannot measure that without running every model against every query. They are proxies. The rest of this post is about how each one is computed and where each one is honestly wrong.

Building the heading tree

Headings come out of the parser as a flat list: an H1, then an H2, then another H2, then an H3, in document order. But a model does not read them as a flat list. It reads them as an outline, where an H3 belongs under the H2 above it. So the first job is turning the flat sequence back into a tree of {level, text, children} nodes.

A stack handles this cleanly. You keep a running stack of open ancestors and, for each heading, pop until the top of the stack is a valid parent:

def build_tree(headings):  # headings: [(level, text), ...] in document order
    root = {"level": 0, "text": None, "children": []}
    stack = [root]
    for level, text in headings:
        node = {"level": level, "text": text, "children": []}
        # pop until the stack top is a shallower heading (the real parent)
        while stack[-1]["level"] >= level:
            stack.pop()
        stack[-1]["children"].append(node)
        stack.append(node)
    return root

The tree is what makes defect detection easy. Counting H1s is a pass over the top-level children. Skip detection falls out of comparing each node's level to its parent's: if a node is more than one level deeper than its parent, a level was skipped. An H2 with an H4 child means H3 never appeared, and that gap is exactly where a model loses track of what nests under what.

The penalty model is graded, applied to the heading dimension before weighting:

Missing H1: minus 40. The page has no top-level claim at all.
Multiple H1s: minus 20. The claim is split across competing tops.
Each skipped level: minus 15. Every jump is one more broken parent link in the outline.

A missing H1 costs more than multiple H1s on purpose. No H1 means the outline has no root; two H1s means it has two roots, which is confusing but recoverable. Per-skip stacking matters too: a page that jumps H2 to H4 twice has two broken spots, and the score should feel both.

Question detection heuristics

A heading counts as a question if it ends in a question mark, or if it starts with one of the interrogatives: who, what, why, how, when, which. That is deliberately loose. "How to reset your password" has no question mark but is plainly a question handle, and the starter check catches it. We accept some false positives ("What's New" is not really a question) because under-counting handles hurts more than over-counting them for this signal.

Beyond headings, two structural patterns count. Definition lists are counted directly by tallying <dl> elements, since a <dl> is term-and-definition pairing, which is a question-and-answer in HTML form. FAQ and accordion sections are detected by substring matching on class and id attributes, looking for the tokens sites actually use to mark these blocks. It is a heuristic, not a parser, and it leans on the convention that developers name these regions what they are.

The three signals combine into a single count, banded:

0 found scores 20.
1 to 2 found scores 50.
3 to 5 found scores 75.
6 or more found scores 100.

The bands reward presence, not density. We do not measure how good the answer under each question is, only that the page visibly poses questions and structures answers. Presence is what we can read from markup reliably; quality is what we cannot. Banding by count keeps the signal to the thing the HTML actually tells us.

Counting words honestly

Content depth is a word count, and the only interesting decision there is what counts as a word. We count text inside <p> tags and nothing else.

The reason is boilerplate. A page's navigation, footer, cookie banner, and sidebar are full of text, and if you count all of it, a thin article wrapped in a heavy template scores as deep when its actual body is two paragraphs. Restricting to paragraph text dodges almost all of that chrome, because nav links and footer columns are rarely marked up as <p>.

The trade-off is real and we accepted it knowingly: prose that lives in bare <div>s instead of paragraphs is invisible to the count. A page that is substantial but built without <p> tags will score thin. We decided that under-counting div-only pages is the lesser error, because counting boilerplate produces confidently wrong high scores, and a wrong high score is worse than a conservative low one. The bands:

Under 300 words scores 30.
300 to 800 words scores 60.
800 to 1,500 words scores 80.
1,500 words and up scores 100.

Semantic landmarks and media

Semantic HTML scores landmark coverage. We look for these elements: article, main, section, aside, figure, figcaption, nav. The score is the count found divided by three, times 100, capped at 100. Three landmarks present is full marks.

Why three and not all seven. The signal we want is "this page marks its regions semantically at all," and a page that uses a main, an article, and a section has clearly made that choice. Requiring all seven would punish a simple page that has no figure or aside to mark, which is most pages. Three is the threshold where the intent is unambiguous, so that is where we cap.

Media scores alt text coverage: images with an alt attribute divided by total images. A page with no images at all scores 50, a neutral middle rather than a zero, because no images is not a failure, it is just an absence of signal. Penalizing a text-only page for having no alt text to measure would be measuring nothing.

The known blind spot

The honest limitation, stated in the tool itself: we read server-rendered HTML only. The fetcher pulls the raw response and parses that. It does not run a browser, so it does not execute JavaScript, so any content hydrated on the client is invisible to every check above.

This means a page built as a client-side app, with its real content injected after load, scores as nearly empty: no headings, no paragraphs, no landmarks. That looks like a flaw until you remember what we are estimating. A crawler reading your raw response sees the same emptiness we do. Scoring the un-hydrated HTML is not a limitation of the measurement, it is the measurement. We surface it plainly rather than hide it, because a tool that quietly renders JavaScript would report a score the actual extractors never see. An honest blind spot beats a confident wrong number.

Try it

The Content Structure Analyzer is free and takes a URL, no signup. You get the heading tree, per-category scores, the issues it found, and a short list of recommendations. Once a page's structure is clean, the Schema Generator emits the JSON-LD to label what the page actually is.

Here is the thing worth doing with it. Run it against a page you are sure is well structured, one you wrote carefully and would defend. Then look at the heading tree it builds. The first time we ran it on our own docs, the tree showed two skipped levels and a second H1 we had styled down to look like a subhead. The page looked fine to us and read fine to a human. The tree did not lie. Go find out what yours looks like.

Mehul Jain is an AI entrepreneur and product builder. He works on Geology, a GEO platform.

Scoring a Page's Meta Tags 0-100: The Rubric Behind Our Analyzer

Mehul Jain — Thu, 04 Jun 2026 15:55:05 +0000

A meta tag audit is a pile of binary checks. Title present, yes or no. Title in range, yes or no. Description present. One H1. og:image set. Canonical present. Run them all and you get a few dozen booleans. The problem is that a wall of green and red checkmarks does not motivate anyone. People glance at it, feel vaguely bad, and close the tab. A single number does motivate. "You are at 62" is a thing a person will act on. But a number only works if it is honest, and a number is only honest if it is explainable.

So we set one hard constraint before writing any scoring code: every point a page loses has to trace back to a named check with a specific fix. No mystery deductions. If you are at 62 and not 100, the tool can point at the exact items that cost you the other 38. That constraint shaped every decision that followed, and it is the reason the rubric looks the way it does. This is the write-up of how we got from a pile of booleans to a number we are willing to defend.

Choosing the dimensions and the weights

The first decision was how to group the checks. We landed on five dimensions, each with a fixed weight, and the overall score is their weighted average:

Basic meta, 30 percent. Title tag and meta description.
Headings, 20 percent. H1 count and heading-level hierarchy.
Open Graph, 20 percent. og:title, og:description, og:image, og:url.
Twitter Card, 15 percent. twitter:card, twitter:title, twitter:description, twitter:image.
Technical, 15 percent. Canonical, html lang, viewport, robots.

The weights are the opinionated part, and they encode what we actually believe about how pages get found now. Basic meta gets 30 percent, the largest slice, because the title and description are the strings an AI engine quotes when it summarizes or cites a page. They are the highest-value characters on the whole page, so a gap there should cost the most.

Technical gets the smallest slice at 15 percent, but for a subtler reason than "it matters least." Technical failures are rarer, and most of the time they are minor: a missing canonical is a warning, not a catastrophe. The exception is the robots meta, where a noindex does not just dent the dimension, it ends the page's chances entirely. We did not solve that by inflating the technical weight, which would have penalized every page for a risk most do not carry. We solved it inside the dimension, by letting the noindex check fail hard rather than nudge an average. More on that split below.

Setting the thresholds

Within each dimension, individual checks need thresholds, and we tried to make them match observed reality rather than a style guide.

The character windows are the clearest case. A title passes between 50 and 60 characters and a description passes between 150 and 160. Outside those bands but present is a warning, not a fail, because a slightly long title still works, it just risks truncation. Missing entirely is a fail. The windows come from where truncation actually bites in search results and citation cards, not from a round number, with the character count standing in as a proxy for the pixel width that does the real clipping.

Headings use two checks. Exactly one H1 is a pass. Zero or multiple H1s is a fail or warning, because the H1 is the page's single claim and splitting or dropping it muddies what the page asserts. The second check walks the heading levels looking for skips, an H2 jumping straight to an H4, and flags them, because a broken outline is harder to parse as a structured document.

The penalties inside a dimension are graded, not binary. A failing item subtracts 40 from its dimension. A warning subtracts 20. That ordering is deliberate. It means a fail always outranks a warning when the tool sorts what to fix, and it means a dimension with one warning still scores well above a dimension with one outright fail. Graded penalties carry more information than a flat "this dimension is broken," and that information is what lets us rank fixes later.

Why a weighted average, not min()

The tempting alternative to a weighted average is to take the minimum across dimensions, or to zero a dimension on any failure. We tried thinking in those terms and rejected them, because they lie about the page.

Consider a page with flawless title and description, clean headings, a solid canonical, and no Twitter Card tags at all. Under a min() model, the missing Twitter Card drags the whole page toward zero, and the score screams emergency at a page that is in good shape and missing one nice-to-have. That is not honest, and worse, it is not actionable, because it tells you nothing about what to do first. A weighted average treats that page correctly: it is strong, with one soft spot worth 15 percent, and the score reflects that. Averaging ranks your fixes by impact. min() just punishes you for your weakest link regardless of how much that link matters.

The counter-case is noindex, and it is the one place we deliberately let a single check dominate. A noindex tag means the page has asked engines not to index it at all. No amount of perfect metadata matters if the page has opted out of being found. So that one check is allowed to fail its dimension hard, because unlike a missing og:url, it does zero the page's prospects. The rule we settled on: a weighted average everywhere, except the one check whose real-world effect is binary and total. Encoding that exception explicitly was cleaner than bending the whole model to accommodate it.

Generating exactly three recommendations

The score tells you where you stand. The recommendations tell you what to do, and there we capped the list at three.

The cap is a behavioral decision, not a technical limit. A list of fifteen fixes does not get done. It gets bookmarked. Three fixes get done this afternoon. So the tool returns at most three Top Recommendations, and they target the lowest-scoring categories.

The ranking is a simple priority score: dimension weight multiplied by the score deficit in that dimension. A big gap in a heavy dimension floats to the top, a small gap in a light dimension sinks. That product captures both halves of "what should I fix first," how much room there is to improve and how much that improvement is worth, in one number. We sort by it, take the top three, and attach the specific fix to each. The user gets a short, ordered, finishable list instead of an exhaustive one nobody completes.

What we left out on purpose

Two checks we could have added and chose not to: keyword density and a meta keywords audit.

Both are dead signals. Search engines stopped using the meta keywords tag for ranking many years ago, and keyword density as a target is a relic of an era when stuffing a term ten times moved rankings. Including either would have padded the check count and made the tool look thorough. It would also have cost us credibility with exactly the people we built this for. An SEO who knows the field would see "meta keywords" in our output and quietly conclude we did not know what year it was. A rubric earns trust partly by what it refuses to score. Leaving the dead signals out was as much a design decision as any weight we picked.

For the record on scope: extraction runs on server-rendered HTML, parsed with BeautifulSoup. We read the markup the page ships, not a post-JavaScript DOM, which is worth knowing if your tags are injected client-side.

Try it

You can run a page through it. The Meta Tag Analyzer is free and takes a URL, no signup. You get the 0 to 100 score, the per-item pass, warning, or fail with a specific fix, and up to three prioritized recommendations.

The Open Graph and Twitter Card scores tell you the tags are present and correct, but not how the card actually looks. For that, run the page through the OG Previewer, which renders the link card as it will appear on Facebook, X, LinkedIn, and Slack. "Are the tags present and in range" and "does the card render right" turned out to be two different questions, so we built two tools rather than bolt a renderer onto a rubric. If you disagree with the weights, that is the interesting part: run a page you know well, see what it scores, and tell us where the number feels wrong.

Mehul Jain is an AI entrepreneur and product builder. He works on Geology, a GEO platform.

Parsing robots.txt for 10 AI Crawlers: Wildcards, Partial Blocks, Line Numbers

Mehul Jain — Thu, 04 Jun 2026 15:48:06 +0000

robots.txt parsing looks like a weekend job. It is a flat text file. Each line is a directive. You split on the colon, match the user agent, check whether a path is disallowed. How hard can it be.

Then you start feeding it real files. You hit a group that opens with three User-agent lines and one rule block. You hit a Disallow: /*? that means more than its author thought. You hit a file that 404s over HTTPS but loads over HTTP. You hit comments mid-line, mixed casing, and a Disallow: with nothing after it. The weekend job grows teeth.

We built the AI Crawler Checker to answer one narrow question well: for a given domain, which of the major AI crawlers can read it, and which cannot. We grade against ten specific user agents:

GPTBot: ChatGPT and OpenAI, training and search
ChatGPT-User: ChatGPT live browsing
Google-Extended: Gemini and Google AI Overviews grounding
Googlebot: Google Search and AI Overviews
PerplexityBot: Perplexity
Anthropic-AI: Claude training
ClaudeBot: Claude web crawler
Bytespider: ByteDance and TikTok
CCBot: Common Crawl, which feeds many AI training sets
Applebot-Extended: Apple Intelligence

This is a write-up of the parts that were not trivial.

Grouping directives by user-agent

The thing that trips up a naive parser is that robots.txt is not a flat list of rules. It is a sequence of groups. A group opens with one or more User-agent lines, and the rule lines that follow apply to every user agent named in that opening run. So this:

User-agent: GPTBot
User-agent: CCBot
Disallow: /private/

is one group of rules shared by two bots, not two separate groups. The * group is the fallback that applies to any agent without its own group. Get the grouping wrong and you misattribute every rule.

The parser we settled on tracks state as it reads. While it is still seeing User-agent lines, it accumulates names. The first non-User-agent directive closes the agent list and starts collecting rules for all of them. Stripped down, it looks like this:

def parse_groups(lines):
    groups = []
    agents, rules = [], []
    reading_agents = False
    for lineno, raw in enumerate(lines, start=1):
        line = strip_comment(raw).strip()
        if not line:
            continue
        field, _, value = line.partition(":")
        field = field.strip().lower()
        value = value.strip()
        if field == "user-agent":
            if rules:                       # previous group closed
                groups.append((agents, rules))
                agents, rules = [], []
            agents.append(value.lower())
            reading_agents = True
        elif field in ("allow", "disallow"):
            reading_agents = False
            rules.append((field, value, lineno))
    if agents:
        groups.append((agents, rules))
    return groups

Illustrative, not the production code, but the shape is real. The detail that earns its keep is lineno, carried on every rule from the moment it is read. More on that below.

Three verdicts, not two

The obvious model is binary: a bot is allowed or it is blocked. Real files do not split that cleanly, so we report three states: allowed, blocked, and partial.

Blocked is a blanket Disallow: /. The bot gets nothing:

User-agent: Bytespider
Disallow: /

Partial is a scoped disallow. The bot can read most of the site but is shut out of specific paths:

User-agent: Googlebot
Disallow: /admin/
Disallow: /cart/

That is "partial" rather than "blocked", and the distinction is the whole point. A bot blocked only from /admin/ is fine. A bot blocked from /docs/ might be cut off from exactly the content you want it to read. Collapsing partial into blocked would cry wolf; collapsing it into allowed would hide a real problem. Three states is the smallest model that tells the truth, so that is what we report, with the matching path shown for every partial.

Line-number attribution

This was the UX decision that mattered most, and it shaped the parser.

A verdict of "GPTBot: blocked" is technically correct and operationally useless. The person reading it now has to open robots.txt, scan it, figure out which group applies to GPTBot, and find the offending line. For a long file with shared groups and wildcards, that is a few minutes of grepping and a decent chance of editing the wrong line.

A verdict of "GPTBot: blocked by Disallow: / on line 42" is a different object. It points at one line. The fix is one edit. There is nothing to investigate.

The implementation note that makes this work: you have to track line numbers during the parse, not reconstruct them after. Once you have normalized a file into groups and rules, the original line positions are gone, and trying to find them again by re-matching strings is fragile the moment the file has duplicate directives. So the line number rides along with each rule from the first read, as the lineno in that tuple above. It costs nothing to carry and it is impossible to recover later, which is the whole argument for doing it up front.

Fetch fallbacks and the messy middle

Before you parse anything you have to fetch the file, and fetching is where the real-world mess starts.

We try HTTPS first and fall back to plain HTTP, because a non-trivial number of sites serve robots.txt correctly over one protocol and 404 over the other. A checker that only tries HTTPS reports "no rules" for a site that has plenty.

Then there is the missing file. If robots.txt does not exist at all, the spec is clear: absence means allowed. No file is permission for everything, so a 404 over both protocols resolves to allowed-by-default for all ten bots, not to an error.

And the file itself is rarely clean. Comments appear at the end of otherwise valid lines, so you strip from the first # before parsing the directive. Field names arrive in every casing, so User-agent, user-agent, and USER-AGENT all have to match, which is why the parser lowercases the field. None of this is hard individually. It is the accumulation that turns a flat-file parser into something you actually test against captured real-world files.

What this method cannot see

The honest boundary, stated in the tool rather than buried: this reads robots.txt and nothing else. robots.txt is permission, not discovery, and permission is only one layer of access control.

If a site blocks a crawler at the CDN or WAF, or by IP, or through a bot-management product, none of that lives in robots.txt. It happens at the network edge, and our checker never sees it. So a clean report means your robots.txt is not blocking the bot. It does not prove the bot can reach you. We say exactly that in the result rather than letting a green checkmark overclaim, because a tool that quietly overstates what it knows is worse than no tool.

Try it

If you want to run your own domain through it, the AI Crawler Checker is free and takes a root domain, no signup. You get the per-bot verdict, the directive, and the line number for all ten crawlers.

The companion on the discovery side is the Sitemap Checker, which validates that your sitemap is findable and healthy. Permission and discovery are different problems, and a bot you have allowed in robots.txt still has to be able to find your pages. Geology is a full-stack SEO and GEO agency, and these free tools are the same checks we run in client audits, factored out so you can run them yourself.

Mehul Jain is an AI entrepreneur and product builder. He works on Geology, a GEO platform.

We Built 7 Free GEO Tools. Here's What Each One Actually Checks

Mehul Jain — Thu, 04 Jun 2026 15:41:25 +0000

Why we gave the audit away

We run a full-stack SEO and GEO agency. Before any strategy work could matter, clients kept failing the same handful of checks: a crawler blocked in robots.txt, a page with three H1s, a sitemap full of 404s, JSON-LD that parsed fine but was missing the one property the schema actually required. We were running the same manual checks at the start of every engagement, and they were the same checks anyone could run if they knew where to look.

So we turned them into tools. Seven of them, all free, all no-signup. You paste a domain or a page URL, and you get results in seconds. They are the same checks we run in our own client audits, which is the whole point: there is no watered-down public version. This post is the engineering write-up. For each tool we will cover what it actually checks, and the one design decision we had to get right (a few we got wrong first).

There was a quieter reason too. We were tired of explaining the same problems over email. "Your GPTBot is blocked" lands very differently when the prospect can run the check themselves and see the line number than when it arrives as an assertion in a deck. A tool that shows its work is more persuasive than a consultant who says trust me, and it costs the reader nothing to verify. In practice the tools do a chunk of the audit conversation before we ever get on a call.

The framing: find, read, cite

The tools map to a pipeline, not a menu. We kept coming back to three questions about any page, and they have to be asked in order. Can an AI system find it? Can it read it? Can it cite it? Access gates structure, structure gates extraction. There is no value in validating schema on a page a crawler cannot reach, so the suite is built to be run roughly in that sequence.

The architecture is deliberately boring. A FastAPI backend, and each tool is a self-contained analyzer sitting behind one endpoint shape: take a URL, fetch what you need, return a structured result. The analyzers do not share state with each other, which makes each one easy to reason about and test in isolation. The page fetches run with strict timeouts, because the user is staring at a spinner and a slow upstream site cannot be allowed to hang the request. The API is rate limited, and beyond that we will keep the operational details out of this post. The interesting decisions are inside the analyzers, so that is where we will spend the words.

Seven tools, seven decisions

AI Crawler Checker

What it checks: you give it a root domain, it fetches robots.txt (HTTPS first, HTTP fallback), parses the file line by line, and returns a per-bot status for ten AI crawlers:

GPTBot and ChatGPT-User: OpenAI
Google-Extended and Googlebot: Google search and AI Overviews grounding
PerplexityBot: Perplexity
Anthropic-AI and ClaudeBot: Anthropic
Bytespider: ByteDance
CCBot: Common Crawl
Applebot-Extended: Apple Intelligence

Each verdict is allowed, blocked, or partial, where partial means a path-specific Disallow.

The decision: every verdict has to cite the exact directive and its line number, or nobody acts on it. Our first internal version just returned "blocked" per bot. It was correct and useless. When we showed it to a client, the immediate response was "blocked by what?" and we had no answer in the UI. People do not edit a production robots.txt on the strength of a red label. They edit it when you can say "line 14, Disallow: / under the GPTBot group." So we rebuilt the parser to carry line numbers and the source directive all the way through to the result. The honest limitation we state plainly: this reads robots.txt only. If a CDN or WAF is blocking a bot at the network layer, we cannot see it, because it never appears in the file.

Meta Tag Analyzer

What it checks: a page URL, scored across five weighted dimensions to a 0 to 100 number.

Basic Meta, 30 percent: title 50 to 60 chars, description 150 to 160
Headings, 20 percent: exactly one H1, hierarchy skips flagged
Open Graph, 20 percent
Twitter Card, 15 percent
Technical, 15 percent: canonical, html lang, viewport, robots meta, where a noindex is a fail

It returns up to three prioritized recommendations.

The decision: weighted dimensions instead of a flat pass/fail. A pass/fail checklist treats a missing twitter:card as equal to a missing title, which is wrong. They are not the same severity, and an analyst with limited time needs to know what to fix first. We landed on 30/20/20/15/15 because it reflects how much each dimension moves real visibility. Basic meta is the page's pitch to both search and assistants, so it carries the most weight. Headings are structural and matter almost as much. Open Graph governs how a shared citation renders. Twitter Card and the technical tags matter but are the smaller levers, hence 15 each. The single 0 to 100 score plus three ranked recommendations is what makes the output actionable rather than a 20-item list nobody triages.

Content Structure Analyzer

What it checks: a page URL across six weighted dimensions to 0 to 100.

Heading Structure, 25 percent: minus 40 for a missing H1, minus 20 for multiple H1s, minus 15 per skipped level
Content Depth, 20 percent: paragraph word count banded, with under 300 scoring 30, the 300 to 800 range scoring 60, 800 to 1500 scoring 80, and 1500-plus scoring 100
Q&A Patterns, 20 percent: question headings, definition lists, and FAQ sections, scored zero, one to two, three to five, and six-plus as 20, 50, 75, 100
Semantic HTML, 15 percent
Lists, 10 percent
Media, 10 percent: alt coverage

The decision: model the headings as a tree, not a list. Our first pass collected the headings into a flat array and counted them. That tells you there are four H2s, which is almost meaningless. The problem with headings is structural: an H2 that jumps straight to an H4 has skipped a level, and a flat list cannot see that gap. So we build a nested heading tree and walk it, which is the only way to detect skipped levels and to dock 15 points each time one shows up. The tree is also what lets us assert exactly one H1 cleanly. It reads server-rendered HTML only, which we flag in the results, because a page whose body is injected by client-side JavaScript looks empty to us and frequently looks empty to crawlers too.

Sitemap Checker

What it checks: a root domain. Discovery follows the crawler's path: the Sitemap: directive in robots.txt, then /sitemap.xml, then /sitemap_index.xml. It parses both url sets and sitemap index files (following up to five children), reports total URLs, lastmod coverage as a percentage, and the freshest and stalest entries. Then it health-checks: it samples min(50, total) random URLs, fires parallel HEAD requests, and classifies each as healthy, redirected, or broken with the status code. It flags duplicate URLs and files that breach the 50,000-URL limit.

The decision: sample 50 URLs in parallel rather than crawl everything. A real sitemap can list tens of thousands of URLs. Fetching all of them is slow, hammers the target site, and is pointless when the user wants results in seconds. A random sample of fifty is enough to tell you whether the sitemap is broadly healthy or broadly broken, which is the question that actually matters at audit time. We use HEAD instead of GET so we get the status code without pulling response bodies, and we run them in parallel so the whole health check finishes inside the same few-second budget as the rest of the suite. If the sample comes back clean, the sitemap is almost certainly fine; if a chunk of it 404s, you have a problem worth a deeper look.

Schema Generator

What it checks: a page URL. It extracts signals (title, meta description, canonical, up to thirty headings, up to ten images, author selectors, dates, breadcrumbs, price markers, FAQ markers), detects the page type, then generates ready-to-paste JSON-LD with confidence scores. Output types span Article/BlogPosting, Product, FAQPage, LocalBusiness, HowTo, BreadcrumbList, Organization, and WebPage/WebSite.

The decision: heuristics pick the page type before any LLM call. This was the most important call in the whole suite. The naive build is to hand the page to Gemini and ask "what type is this and write the schema." It works, sometimes, and it is expensive and unpredictable. Instead we run cheap deterministic heuristics first: price markers mean product, FAQ markers mean an FAQ page, an author plus a date means a blog post, an about-or-team page means organization, a contact page means local business, two or more "Step N" headings mean how-to, and everything else falls back to generic. Only then do we call Gemini Flash, and we call it with the type already decided, so the model is writing schema for a known shape rather than guessing the shape and the content at once. That grounds the output and cuts cost. The rule we enforce hard: any fact the model cannot read off the page becomes an explicit placeholder. It never invents a value. A tool you paste into production cannot hallucinate an author name or a price.

Schema Validator

What it checks: a page URL or pasted JSON-LD. It extracts every ld+json block and runs three layers. JSON syntax. schema.org type rules, with required and recommended properties for fifteen common types, plus ISO 8601 date validation. Then content alignment. Each block scores syntax 40 + required 30 + recommended 20 + alignment 10, and the page averages its blocks.

The fifteen types group sensibly:

Articles: Article, BlogPosting, NewsArticle (require headline and author), and FAQPage
Commerce: Product (requires name)
Local: LocalBusiness (requires name and address) and Event
Media: Recipe, VideoObject, and HowTo
Site structure: BreadcrumbList, Organization, WebSite, and WebPage

The decision: valid JSON is not valid schema, and the output has to make that distinction loud. Early on we leaned on a JSON parse and called it a day. The trouble is that a block can be perfectly valid JSON and still be useless markup, because it is missing the headline an Article requires or the address a LocalBusiness requires. Those are the failures that actually keep you out of rich results and clean citations. So we built per-type required-property tables for the fifteen types and score against them, with required properties weighted far above recommended (30 versus 20) because a missing required property is a hard failure and a missing recommended one is a nudge. Gemini suggests markup you are missing on top of that, but those suggestions never block the result. A syntactically clean, type-complete block validates clean regardless of what the model would add.

OG Previewer

What it checks: a page URL. It extracts Open Graph and Twitter tags, falls back to title and meta description where tags are missing, and renders previews for Facebook, X/Twitter, LinkedIn, and Slack using each platform's real fallback chain. It checks og:image accessibility server-side and recommends 1200×630 (1.91:1). A missing og:title, description, image, or url is an error; a missing twitter:card is a warning.

The decision: implement each platform's real fallback chain rather than a single generic preview. It is tempting to read the OG tags once and draw one card. But the platforms do not agree. Each one has its own order for what it falls back to when a tag is missing, and a card that looks fine in one place can render blank in another. Since these cards are the first impression when an AI-cited link gets shared into a channel, "looks fine somewhere" is not good enough. So we model Facebook, X, LinkedIn, and Slack separately, each with its own fallback logic, and render four cards. The server-side image accessibility check exists because a perfectly tagged og:image that returns a 403 still produces a broken card, and you cannot see that by reading the HTML alone.

What we deliberately left out

There are no signups. No "enter your email to see results." No PDF gate, no "book a demo" wall between you and the numbers. The entire value of these tools is that you can run one on a hunch in the middle of an audit, and anything that interrupts that with a form kills the use case. The hard constraint we held to: results in seconds, or people leave. Every design decision above bends toward that, from the parallel HEAD requests in the Sitemap Checker to the heuristic type detection that keeps the Schema Generator from making unnecessary LLM calls.

We also left out a "score history" feature, at least for now. It is the obvious next ask: run a page weekly and chart the trend. We skipped it because it would have forced accounts, and accounts would have reintroduced the exact friction we were trying to delete. A tool you can run anonymously in ten seconds is a different product than a dashboard you log into, and we decided the anonymous tool was the one worth shipping first.

The honest trade-off in all of this is that these tools check readiness, not outcomes. They tell you whether AI systems can find, read, and cite a page. They cannot tell you your actual citation share inside ChatGPT or Perplexity, or the sentiment of how an assistant describes you. That is a different system, watching live answers over time, and we were careful not to over-promise it in the tool copy. A readiness check that is honest about its boundary is more useful than one that implies it measures the thing it cannot see.

Run it on your own site

The tools are meant to be run as a sequence, find then read then cite, and the whole pass takes about half an hour.

Start with access. Run the AI Crawler Checker on your root domain and clear any blocked AI bots, then run the Sitemap Checker on the same domain to catch broken URLs and stale lastmod coverage. Move to readability: put your two or three most important pages through the Meta Tag Analyzer and the Content Structure Analyzer, fix the title and description lengths, enforce one H1, and turn vague section headings into the questions your buyers actually ask. Then make yourself citable: run your top page through the Schema Generator, paste the JSON-LD into the page, confirm it with the Schema Validator, and check how the shared card renders with the OG Previewer.

All seven live on the tools hub. They are free and there is nothing to sign up for. If you build something with them or find an edge case we missed, we want to hear about it.

Mehul Jain is an AI entrepreneur and product builder working on Geology.

We Ran 3,000+ AI Prompts to Test GEO for B2B SaaS. Here's What We Found.

Mehul Jain — Wed, 27 May 2026 13:37:59 +0000

Every GEO guide tells you the same thing. Add FAQ schema. Sprinkle statistics. Get on G2. Keep it short and structured.

These tips all trace back to one study (the Princeton/IIT Delhi GEO paper from 2023), passed through hundreds of blog posts until the findings are unrecognizable.

We wanted to know if any of it holds up for B2B SaaS. So we tested it.

We queried ChatGPT, Google AI Overviews, and Perplexity with 124 unique B2B SaaS queries. Things like "best project management software for startups" and "CRM for small business." We collected 3,352 citations across 881 unique domains. Then we analyzed the top 50 most-cited pages for on-site signals and tested 12 specific hypotheses.

The results? Only 3 of 12 popular GEO claims survived contact with the data.

Here are the five findings that matter most.

1. Content length is the strongest signal. Not the weakest.

The most repeated GEO claim is that word count has near-zero correlation with AI citations. Ahrefs measured r=0.04.

Our data says otherwise. For B2B SaaS buyer queries, content length correlates with citations at r=0.393. Pages over 5,000 words average 15.3 citations versus 10.3 for mid-length pages. That's a 50% advantage.

The three most-cited pages in our dataset are all massive buyer's guides: project-management.com (9,227 words, 37 citations), wrike.com (11,017 words, 28 citations), paymoapp.com (17,890 words, 27 citations).

The explanation is simple. When someone asks AI "best project management software for startups," the AI needs a source that covers enough tools to build an answer. Short pages can't serve that function.

2. Reddit dominates ChatGPT, not Perplexity.

The popular claim: Reddit accounts for ~24% of Perplexity citations. Invest in Reddit for Perplexity visibility.

Our data flips this completely.

Reddit provides 14.7% of ChatGPT citations (4.7x more than vendor-owned content). On Perplexity? Zero. Literally 0%.

Perplexity favors vendor websites (18.1%) and YouTube (7.6%). ChatGPT favors Reddit and editorial content. If you're investing in Reddit content seeding for Perplexity, you're spending money on the wrong platform.

3. G2 and Capterra account for just 1.6% of citations.

Multiple guides claim that G2/Capterra presence is the strongest predictor of AI visibility for SaaS. Our data: review platforms account for 55 of 3,352 total citations. That's 1.6%.

Reddit alone has 6.1x more citations than G2 and 15.7x more than Capterra. The traditional review platform stack (G2 + Capterra + TrustRadius + Trustpilot + Software Advice) accounts for 78 citations combined. Reddit alone is 3.6x larger.

Having a G2 profile is table stakes for SaaS credibility. Optimizing it as your primary GEO strategy is a misallocation.

4. FAQ blocks and schema markup show zero advantage.

FAQ blocks: pages with them average 11.7 citations versus 12.4 without. The ratio is 0.94x. Slightly negative.

Schema markup: r=0.103 correlation with citations. Flat.

ChatGPT and Perplexity parse rendered HTML, not JSON-LD. They see your headings, tables, and lists. JSON-LD schema tells Google's parser about your content, but tells ChatGPT nothing it can't already extract from the page.

Stop spending engineering hours on FAQPage schema as a GEO tactic.

5. There are two AI ecosystems, not three.

Google AI Overviews and Perplexity share 55% domain overlap. They largely cite the same sources. ChatGPT overlaps with either at less than 6%.

This means you need two strategies, not one.

For ChatGPT: Reddit presence, long-form guides, comparison content. ChatGPT's web search only triggers on "best X" and comparison queries (94% and 75% trigger rates). FAQ queries trigger web search 0% of the time.

For Google AIO + Perplexity: vendor-owned content, YouTube, niche blog placements. These platforms reward professional publishing.

What to do with this

The winners in our dataset aren't the most "optimized" pages. They're the most useful ones. The page with 37 citations earned them by being the most thorough buyer's guide in its category. Not by adding FAQ schema or sprinkling statistics.

The full study, with all 12 hypotheses, per-platform breakdowns, the review source comparison table, and the complete playbook, is here:

Read the full report: GEO for B2B SaaS: What Actually Works

This research was conducted by Geology, the analytics platform for Generative Engine Optimization. Data collected May 2026.

GEO and AEO for Startups: A Founder's SEO Kickstart

Mehul Jain — Wed, 06 May 2026 06:06:46 +0000

Most early-stage founders treat SEO, GEO, and AEO as three separate budgets, three separate hires, three separate dashboards. At startup scale that is the wrong model. The three are one practice with three places to measure the same content. Trying to fund them as parallel workstreams is the most common waste of early-stage marketing time I see.

Here is the framing that holds at this stage. Classic SEO ranks pages on Google. GEO (generative engine optimization) earns brand mentions inside ChatGPT, Perplexity, and Gemini answers. AEO (answer engine optimization) targets the literal text of an AI response or a Google AI Overview. The work that earns one tends to earn the others. What changes between them is where you measure the result, not what you write.

Why early stage favors GEO and AEO over classic SEO

Domain authority compounds slowly. Backlinks take months. A six-month-old startup competing for "best [category] tool" against incumbents with 10,000 backlinks is a losing fight on Google.

AI answers do not work that way. ChatGPT, Perplexity, Gemini, and Google AI Overviews evaluate whether a source answers a specific question well. They weight recency, structured specificity, and third-party signal differently than Google's traditional ranking model. A startup that publishes 15 precisely targeted answer pages in a month can show up in AI responses well before it ranks page two of Google. The deeper argument for why this happens lives in our post on building AI visibility from zero, which walks through the topic-authority gap between the two systems.

That is the structural advantage. It is also why a startup founder should not start with a content calendar or a keyword strategy. They should start with a prompt list.

Step one. Map the prompts your buyers ask AI

Before you write anything, build a list of 30 to 60 prompts your buyers actually type into ChatGPT, Perplexity, or a Google AI search bar in the week before they buy.

Pull from these sources:

Sales call recordings, where you can hear the literal phrasing of buyer questions
Support tickets and customer interview transcripts
Reddit and Stack Overflow threads in your category
Your competitors' FAQ pages and help center entries

This is the foundation of AEO. Answer engines pull from sources that match the prompt closely. If your content is not built around the prompts your buyers actually use, it will not be cited.

Step two. Mine the founder's head

Most early-stage startups have already done the hard part of content. The founder has spent a year talking to users. They know which workflows the product handles well, which buyer objections come up most, and which competitor weaknesses are real.

The problem is that this knowledge sits in Notion docs, Slack threads, and call recordings. None of it is in a place an AI model can read. The unlock for early-stage GEO is converting that material into 10 to 15 source-of-truth pages on your own domain. Each page answers one prompt from the list above with depth and specificity that no aggregator article can match.

Founders often resist this because shipping 15 articles feels like more work than shipping five. It is not. The content is already in the founder's head. The job is transcription and editing, not research.

Step three. Earn the off-site signal

AI models do not trust new domains alone. They cross-reference whether the brand or the expert behind it appears credibly elsewhere. For a startup with no link history, that means showing up in places AI training and retrieval systems read: niche subreddits, GitHub discussions, Stack Overflow, Hacker News, IndieHackers, Product Hunt reviews, podcast transcripts, and small-creator newsletters.

Two practical rules:

The off-site activity has to be useful to the community first, promotional second. Mod rules matter. Account history matters. Helpful answers anchored to the source pages on your site are the shape that works.
Do not buy links. AI models seem unbothered by paid or low-quality link networks for now, which is the opposite of how Google works. Real conversations in real communities move the needle.

Where this goes wrong

A few patterns repeat in early-stage teams.

The first is hiring a full-time SEO person before there is any signal. The first dedicated GEO hire usually makes sense at month four to six, when there is weekly citation data and a backlog of plays the founder can hand off. Before that, the founder is the right person to write the source pages, because nobody else has the domain conviction.

The second is writing for keywords instead of prompts. Keyword tools optimize for what people type into Google. AI prompts are longer, more conversational, and more specific. If your brief uses Ahrefs keywords as the source of truth, you are optimizing for the wrong text.

The third is treating Reddit as a marketing channel. It is a community signal channel. The startups that benefit are the ones whose accounts have helpful comment history predating the product launch.

The 30-day plan, in one sentence

Pick 30 prompts your buyers actually ask AI, ship 10 to 15 source pages that answer them with founder-level specificity, and seed three communities with helpful answers anchored to those pages.

If you want the version with weekly artifacts, founder time commitments, and a citation dashboard, the Geology startups solution covers the four-week program we run with pre-seed to Series A teams. For the underlying difference between optimizing for search engines and optimizing for AI answers, GEO vs SEO is a useful read.

The window for early-mover advantage in AI discovery is still open in most categories. It will close. The founders who run this playbook in the next twelve months will compound a citation lead their later-funded competitors will spend a year trying to close.

Mehul Jain is an AI entrepreneur and product builder. He writes about how search is shifting from keywords to model-mediated discovery at Geology.