DEV Community: AgustaON

Four AI APIs, four completely different citation formats. Getting the data out was harder than the analysis.

AgustaON — Sat, 01 Aug 2026 09:03:48 +0000

I've been running a small monthly study: take ten buyer-intent questions from one industry, ask four AI assistants the identical question, and record every source each one actually read.

The analysis is the easy bit. Set overlap, count the domains, done.

Getting the citations out of four different APIs in a shape you can compare — that's where I lost a weekend. Writing it up because I couldn't find this anywhere when I needed it.

The problem

All four providers will tell you which URLs they used. None of them agree on where to put that, what to call it, or even what a "URL" is.

Here's roughly what each one gives you. (Shapes as of mid-2026 — they move, check the docs before you copy this.)

OpenAI, Responses API with the web search tool. Citations ride along as annotations on the text output:

const urls = [];
for (const item of resp.output ?? []) {
  for (const block of item.content ?? []) {
    for (const a of block.annotations ?? []) {
      if (a.type === "url_citation") urls.push(a.url);
    }
  }
}

Anthropic, Messages API with the web search tool. Search results come back as their own content block type, separate from the text:

const urls = [];
for (const block of msg.content ?? []) {
  if (block.type === "web_search_tool_result") {
    for (const r of block.content ?? []) {
      if (r.url) urls.push(r.url);
    }
  }
}

Gemini, with Google Search grounding. It's on the candidate, under grounding metadata:

const urls = (resp.candidates?.[0]?.groundingMetadata?.groundingChunks ?? [])
  .map(c => c.web?.uri)
  .filter(Boolean);

Perplexity is the friendly one — a flat array on the response:

const urls = resp.citations ?? resp.search_results?.map(r => r.url) ?? [];

Four different nesting depths, three different words for the same concept ("annotation", "search result", "grounding chunk"), and one of them isn't giving you the URL at all. More on that in a second.

So: one adapter per provider, normalise to { engine, question, urls[] }, and never let provider-shaped data past that boundary. Obvious in hindsight. I did not do this first and regretted it.

Gotcha 1: Gemini's grounding URIs aren't the source

This is the one that would have quietly wrecked my numbers.

Gemini's groundingChunks[].web.uri isn't the publisher's URL. It's a redirect through Google's own grounding endpoint. If you take it at face value, every single Gemini citation looks like it came from the same domain, and your "which sites does Gemini read" analysis returns exactly one site.

You have to resolve it:

async function resolveFinal(url) {
  try {
    const r = await fetch(url, { method: "HEAD", redirect: "follow" });
    return r.url;
  } catch {
    return url; // keep the original, flag it, move on
  }
}

Two notes from doing this at volume. HEAD is enough and much cheaper than GET — you only want the final URL. And cache aggressively, because the same redirect will come up over and over and you don't want to hammer anything.

I noticed this because Gemini's numbers looked absurd on the first run. Worth building a sanity check that screams when one engine's domain diversity collapses to near-zero — that's almost always an extraction bug, not a finding.

Gotcha 2: the same page in five costumes

Once you have real URLs, they still won't match each other. All of these are the same page:

https://www.example.com/clinic/
http://example.com/clinic
https://example.com/clinic?utm_source=chatgpt.com
https://example.com/clinic#reviews
https://example.com/Clinic

If you're comparing sets, five spellings of one page means five "different" sources, and your overlap number comes out lower than reality. Which, if your whole finding is "they overlap surprisingly little", is the exact direction you don't want an unforced error pointing.

The canonicaliser I settled on:

const TRACKING = /^(utm_|fbclid|gclid|msclkid|ref|source$)/i;

function canonical(raw) {
  const u = new URL(raw);
  u.protocol = "https:";
  u.hostname = u.hostname.replace(/^www\./, "").toLowerCase();
  u.hash = "";
  for (const k of [...u.searchParams.keys()]) {
    if (TRACKING.test(k)) u.searchParams.delete(k);
  }
  u.pathname = u.pathname.replace(/\/+$/, "") || "/";
  return u.toString();
}

Deliberately not lowercasing the path — paths are case-sensitive on plenty of servers and I'd rather over-count than merge two genuinely different pages.

One that made me laugh: several assistants append ?utm_source=chatgpt.com (or their own equivalent) to the URLs they hand back. The tools are tagging their own referral traffic. Strip it, or you'll have engine-specific duplicates of the same page.

Then the analysis, which really is boring

Set overlap per question, pairwise across engines:

const jaccard = (a, b) => {
  const A = new Set(a), B = new Set(b);
  const inter = [...A].filter(x => B.has(x)).length;
  const union = new Set([...A, ...B]).size;
  return union ? inter / union : 0;
};

Jaccard has a real flaw here: it treats a source read once exactly the same as one read by every engine on every question. I've gone back and forth on weighting it and haven't convinced myself either way. If you've got a better metric for "did these two systems consult the same evidence", I'd genuinely like to hear it.

What the numbers looked like

This month's run was cosmetic and aesthetic clinics. Ten questions, nine cities, four engines.

214 different websites read across just ten questions
75% of those were read by one engine and no other
Overlap between any two engines on the same question: 7%
Exactly 1 question out of ten produced a website all four read
About 13% of what they read was a directory; the rest was businesses' own sites

That last one flipped between industries, which I didn't expect — in the law firms run last month, directory domains dominated the top of the list.

The objection I had immediately, and you probably do too: LLMs are non-deterministic, so of course two calls differ — this could be noise.

So I re-ran a subset three times per engine and compared each engine against itself:

Same engine, asked again: 51% overlap
Different engines, same run: 7%

About 7.5× more consistent with itself than with its competitors. There's real run-to-run variance, but it's much smaller than the gap between engines. The divergence is structural, not sampling.

If you want to poke at that from the other direction, I wrote up whether a single assistant can even see a given site separately — same problem, one engine at a time.

What I can't claim

Ten questions in one vertical is a sample, not a census. The variance control was two questions re-run three times, not the whole set. "Read" only means the URL appeared in that response's citations — I can't see weighting, or how much of the page actually mattered. And all of it is a July 2026 snapshot; these systems change under you.

The takeaway for anyone building on this

If you're writing anything that compares retrieval across providers, budget most of your time for extraction and normalisation, not analysis. My rough split ended up 80/20 in favour of the boring part.

And build the sanity checks early. Both of my real bugs — the redirect one and the tracking-param one — were invisible in the output. The code ran fine. The numbers were just wrong, in a direction that happened to flatter my hypothesis. That's the kind of bug worth being paranoid about.

The wider point, if you take one thing away: treating "AI visibility" as a single channel doesn't survive contact with the data. Four engines built almost entirely separate reading lists from the same question, so it's four problems, not one.

Full write-up of this month's run, with the question-by-question breakdown, is here. Happy to share the raw JSONL if you want to check my counts — I'd rather be corrected than cited.

(Disclosure: I build a tool in this space. The study isn't gated and there's no signup on it.)

How do you measure something that gives a different answer every time?

AgustaON — Wed, 29 Jul 2026 06:44:55 +0000

I had a simple-sounding question: does ChatGPT recommend this business?

You'd think you just ask it. Ask ChatGPT "best personal injury law firm in NYC", see if the business is named, record yes or no.

That works exactly once. Ask again an hour later and you might get a different answer. Not slightly different — potentially a completely different set of firms and a completely different set of cited sources.

Which means the naive version of this measurement is worthless. You're not measuring visibility, you're sampling a distribution once and calling it a fact.

This is the same problem anyone gets when they try to test an LLM-backed feature. Your normal testing instinct — same input, assert on output — just doesn't apply. So here's how I ended up designing around it, and the numbers that came out, which surprised me.

The setup

I wanted to compare four assistants (GPT-4o, Claude Haiku 4.5, Gemini 2.5 Flash, Perplexity Sonar, all with web search on) across 10 buyer-intent questions in one vertical. Something like:

"Best personal injury law firm in New York City?"
"Top immigration lawyers in Mumbai?"

For each response I recorded two things: which businesses got named, and which URLs got cited. The cited sources come from each API's own citation metadata, so that part is structured — no scraping the prose.

First pass, the results looked dramatic. The four assistants barely agreed on anything. Different firms, different sources, almost no overlap.

Great finding. Except I couldn't publish it, because there was an obvious objection I couldn't answer:

Maybe they weren't disagreeing with each other. Maybe each one was just disagreeing with itself.

If a single assistant returns wildly different sources run to run, then "these four models cite different things" is a meaningless statement. You'd be measuring noise and calling it signal.

The control

The fix is the same idea as a control group. Measure the thing you're worried about, separately, and see if it explains your result.

So: re-run all 10 questions 3× per assistant, and compute two different numbers.

Cross-consistency — how much do two different assistants overlap on the same question?
Self-consistency — how much does the same assistant overlap with itself on the same question, asked again?

If those two numbers are close, the divergence is just non-determinism and I have nothing. If self-consistency is much higher, the models genuinely differ.

For "overlap" I used Jaccard similarity on the sets of cited domains — intersection over union. Standard, boring, and easy to defend when someone asks:

function jaccard(setA, setB) {
  const a = new Set(setA);
  const b = new Set(setB);
  if (a.size === 0 && b.size === 0) return 1;

  let intersection = 0;
  for (const x of a) if (b.has(x)) intersection++;

  const union = a.size + b.size - intersection;
  return intersection / union;
}

Cross-consistency for a question is the mean pairwise Jaccard across the 6 assistant pairs. Self-consistency is the mean pairwise Jaccard across the 3 runs of a single assistant. Then average across questions.

function meanPairwise(runs) {
  const scores = [];
  for (let i = 0; i < runs.length; i++) {
    for (let j = i + 1; j < runs.length; j++) {
      scores.push(jaccard(runs[i], runs[j]));
    }
  }
  return scores.reduce((a, b) => a + b, 0) / scores.length;
}

Two normalisation details that matter more than they sound:

Normalise to registrable domain. www.example.com/page-a and example.com/page-b are the same source for this purpose. If you don't collapse those, you'll measure URL formatting instead of source overlap and your numbers will be garbage.

Decide what counts as a source and write it down. I counted only domains that appeared in the API's citation metadata, not domains mentioned in prose. Defensible either way, but you have to pick one and say which.

What came back

Cross-assistant overlap: ~7%. Two different assistants asked the same question cited almost entirely different sources.
Self-consistency: ~51%. Same assistant, same question, asked again — about half the sources repeat.

So an assistant is roughly 7.5× more consistent with itself than with its rivals. The divergence between models is real. The control did its job.

But look at that self-consistency number again, because it's the more interesting one. Ask the same model the same question twice and half the cited sources change.

And it isn't uniform:

Assistant	Sources repeating on a second ask
Perplexity	~72%
Claude	~54%
Gemini	~42%
ChatGPT	~32%

Roughly two-thirds of what ChatGPT cites changes between two identical questions. Perplexity is more than twice as stable.

That spread is the practically useful finding. If you're building anything that depends on retrieval consistency, your choice of provider changes how reproducible your system is by a factor of two — before you've written a line of your own code.

The generalisable bit

If you're testing anything LLM-backed, single-run assertions are lying to you. A few things I'd carry to any similar problem:

Run n times and report a distribution, not a value. n=3 was enough to separate signal from noise here. It won't be for everything — if your effect size is smaller, you need more runs. Sanity-check by increasing n and seeing whether your headline number moves.

Measure your noise floor explicitly. The self-consistency number is the noise floor. Any difference between two things you're comparing has to clear it to mean anything. Most "we compared model A and B" posts skip this, which is why most of them are unreliable.

Assert on properties, not exact outputs. "Response cites at least one source from an allowed list" survives non-determinism. "Response equals this string" does not.

Pin what you can. Model version, temperature, whether web search is on, region. My biggest regret is not pinning model versions harder — a provider updating a model mid-study would have quietly invalidated everything, and I'd have had no way to detect it after the fact.

What I can't claim

Two things worth being explicit about, since this space is full of overclaiming:

This is a snapshot. July 2026, one vertical, 10 questions. Retrieval behaviour changes with model updates. The 32% figure for ChatGPT is not a constant of nature — it's what I measured, in that window, with that method.

More runs would be better. n=3 is enough to establish that self-consistency >> cross-consistency, which is the claim I'm making. It is not enough to say Claude's 54% is meaningfully different from Gemini's 42%. Those two could easily swap places with more sampling, and I wouldn't defend that ordering.

If someone wants to run it with n=20 and tell me I'm wrong, genuinely great — that's a better dataset than mine.

The data

Full write-up with all 10 questions, every business named, every source cited, and the method in more detail:
https://greaterthanservices.com/blog/ai-search-study-law-firms

Disclosure: I run Greater Than Services, which does this measurement as a product — the variance control exists because customers were reasonably asking "how do you know this isn't random?" The method above is the answer, and it's not proprietary, so take it and run it on your own vertical if it's useful.

If you've built variance controls for LLM-dependent systems I'd like to hear how you handled it — particularly how you picked n, which is the part I'm least confident I got right.

My company ranked #1 on Google but ChatGPT had never heard of us. Here's what I did about it.

AgustaON — Mon, 20 Jul 2026 08:25:14 +0000

Last month my manager forwarded me a Slack message from our founder that just said: "Ask ChatGPT what our company does. Then panic."

So I did. I opened ChatGPT and typed our company name. It confidently told me about a completely different company with a similar name. Then I asked it to "recommend a good [our category] tool" — the exact thing we sell — and it named three competitors. Not us. Never us.

Here's the kicker: we rank #1 on Google for our main keywords. Our SEO is genuinely good. And it turns out that means almost nothing to an LLM.

I went down a rabbit hole for two weeks. This is the practical version of what I learned, with the actual code that moved the needle.

Why "good SEO" doesn't transfer to AI

Google crawls, indexes, and ranks pages. Its algorithm is built around links, keywords, freshness, and hundreds of ranking signals. You can win that game with backlinks and on-page optimization.

LLMs work differently. When you ask ChatGPT or Perplexity "what's the best X," the model isn't ranking a list of pages — it's generating an answer from an internal representation of the world, sometimes augmented with live retrieval. For your business to show up, the model needs to have formed a clear, confident, consistent understanding of what you are.

That understanding comes from:

How clearly your own site describes you (in plain language, not marketing-speak)
How consistently you're described everywhere else (directories, social profiles, other sites)
Whether machines can parse your site at all (structured data, semantic HTML)

We were failing all three, and none of them showed up in a normal SEO audit.

Problem 1: Our homepage was written for humans who already knew us

Our hero section said something like "The intelligent growth platform for modern teams." Beautiful. Meaningless to a model. An LLM reading that has no idea what we sell.

The fix was boring but effective: we rewrote the first paragraph of the homepage to be almost aggressively literal. One sentence that a stranger — or a model — could repeat back accurately:

"[Company] is a [exact product category] that helps [specific audience] do [specific outcome]."

That's it. No adjectives doing load-bearing work. After the change, re-testing showed the model could actually describe us.

Problem 2: No structured data

This is the technical one and where devs can make the biggest difference fast.

Most sites have zero JSON-LD structured data. It's invisible to human visitors but it's a direct, machine-readable statement of what this entity is. If you give an AI crawler explicit facts, it doesn't have to guess.

Here's the minimum I'd add to any company homepage — a <script type="application/ld+json"> block in the <head>:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Your Company",
  "url": "https://yourcompany.com",
  "description": "A one-sentence, plain-language description of exactly what you sell and who it's for.",
  "sameAs": [
    "https://www.linkedin.com/company/yourcompany",
    "https://twitter.com/yourcompany",
    "https://github.com/yourcompany"
  ]
}
</script>

The sameAs array matters more than it looks — it links your entity to every other profile that describes you, which reinforces consistency (problem 3).

If you sell specific products, add Product schema too:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Product Name",
  "description": "What it does, in plain language.",
  "brand": { "@type": "Brand", "name": "Your Company" },
  "offers": {
    "@type": "Offer",
    "price": "4.99",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  }
}
</script>

For content pages, FAQPage and Article schema help LLMs extract answers directly. The pattern is always the same: stop making the machine infer, and just tell it.

Validate everything with Google's Rich Results Test or Schema.org's validator before shipping — malformed JSON-LD does nothing.

Problem 3: We described ourselves five different ways

Our website said one thing, our LinkedIn bio said another, our Crunchbase entry was three years out of date, and our Twitter bio was a joke (literally, it was a pun). To a model trying to build a confident picture, that inconsistency is noise. It'll favor a competitor whose story is the same everywhere.

Fixing this wasn't code — it was a boring afternoon of copy-pasting one canonical description into every profile we own. But it measurably helped.

The part where I admit I used a tool

I could check some of this manually — read our own copy, eyeball the markup, run the schema validators. But two things were painful to do by hand:

Writing correct JSON-LD for every page type from scratch, matching the right schema.org vocabulary
Actually measuring whether AI assistants mention us — you can't eyeball that, you have to query them with real buyer questions and check the answers

A teammate pointed me at Greater Than Services, which runs an audit specifically for this (they split it into CRO / GEO / ASO — human, AI-search, and AI-agent readability). The useful part for me as a dev: instead of a report that says "add structured data," it generates the actual JSON-LD and meta tags for your specific site, ready to paste. It also runs the "does AI name you" test automatically across a few assistants, which is the measurement I couldn't be bothered to script myself.

The free preview was enough to see our scores; the full report was a few dollars. I'm not affiliated with them — it just saved me from writing schema by hand for 6 page types, so it's earned a mention. There are other ways to do this; that's the one that worked for me. If you'd rather DIY, everything above is the manual path.

The 30-second test you should run right now

Regardless of tooling, do this before you close the tab:

Open ChatGPT (or Perplexity, or Gemini).
Type: "What does [your company] sell, and who is it for?" — do not paste your URL.
Read what comes back.

If it's vague, wrong, or blank, that's not a hypothetical future problem. That's real prospects, right now, asking an AI for exactly what you sell and getting pointed at someone else.

We went from "ChatGPT thinks we're a different company" to "ChatGPT describes us correctly" in about two weeks, mostly with the three fixes above. The code part — structured data — was genuinely the highest-leverage thing, which is why I'm writing this on dev.to and not on LinkedIn.

Ship the JSON-LD. Fix your copy. Be consistent. The models are already answering questions about your business — you just want to be in the answer.

Ran into weird edge cases with schema for a specific page type? Drop them in the comments — happy to compare notes.

We looked at where AI actually gets its answers. Your homepage barely matters.

AgustaON — Mon, 20 Jul 2026 08:24:45 +0000

If you've read anything about "getting cited by AI" in the last year, the advice was probably some combination of: add structured data, write clearer copy, fix your headings. All reasonable. We build a tool that does exactly that kind of analysis, so I'm biased toward it being useful.

Then we started logging where the AI answers were actually pulling from — the source types cited when an assistant answers a buying question in a given category — and the data made me rethink the priority order.

Company homepages were rarely the primary source.

What the citation mix actually looks like

When you ask ChatGPT, Perplexity, Gemini, or Claude something like "best project management tool for a small agency," the model (usually with web search on) assembles an answer from retrieved documents. Log what those documents are across a category and a pattern shows up fast. Roughly:

Review platforms — G2, Capterra, Trustpilot, category-specific review sites
Industry directories — niche listings, "best X tools" databases
Editorial roundups — blog posts titled "12 best X for Y in 2026"
Forums and communities — Reddit threads, Stack Overflow, niche forums
Brand-owned sites — usually the smallest slice

The exact mix varies a lot by category — developer tools skew heavily toward forums and GitHub, local services skew toward review sites and maps — but brand sites consistently underperform the intuition.

This makes sense once you think about retrieval rather than ranking. The model isn't looking for the most authoritative page about you. It's looking for documents that compare options in your category. Your homepage says "we're the best." A G2 category page says "here are 40 tools, ranked, with reviews." Guess which one is more useful for answering a comparison question.

The uncomfortable implication

You can have perfect technical SEO, flawless JSON-LD, a beautifully clear homepage — and still be absent from the answer, because the answer was assembled from four sources you're not on.

That doesn't make on-page work pointless. It makes it necessary but not sufficient, and it changes the order you should do things in:

Be present on the source types your category's answers are built from
Be parseable so that when the model does encounter you, it correctly understands what you are

Most advice covers step 2 exclusively. Step 1 is unglamorous, non-technical, and does more.

The technical half (step 2) — what actually helps

Since this is dev.to, here's the part you can ship this afternoon.

Make your entity unambiguous

The single highest-leverage markup is Organization schema with a populated sameAs. It links your site to every other profile that represents you — which is exactly how a retrieval system corroborates that scattered mentions are the same entity:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Your Company",
  "url": "https://yourcompany.com",
  "description": "One plain sentence: what you sell and who it's for.",
  "sameAs": [
    "https://www.g2.com/products/your-company/reviews",
    "https://www.crunchbase.com/organization/your-company",
    "https://github.com/yourcompany",
    "https://www.linkedin.com/company/yourcompany"
  ]
}
</script>

Note what's in that sameAs array: the review site and the directory. You're explicitly connecting yourself to the sources that actually get cited.

Write passages that survive extraction

Retrieval works on chunks, not pages. A paragraph that only makes sense after reading the three above it is a bad chunk. A self-contained paragraph that answers one question is a good one.

Practically, that means:

<!-- Bad: requires context to be useful -->
## How it works
It integrates seamlessly and handles this automatically for you.

<!-- Good: survives being ripped out of context -->
## How does [Product] handle rate limiting?
[Product] applies a token bucket per API key, defaulting to 100
requests per minute. Limits are configurable per plan, and 429
responses include a Retry-After header.

The second one can be lifted directly into an answer with attribution. The first can't be lifted at all.

FAQPage schema formalizes this — it's a machine-readable declaration of "here is a question and here is its self-contained answer":

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [{
    "@type": "Question",
    "name": "How does the API handle rate limiting?",
    "acceptedAnswer": {
      "@type": "Answer",
      "text": "A token bucket per API key, defaulting to 100 requests per minute, configurable per plan. 429 responses include a Retry-After header."
    }
  }]
}
</script>

Don't block the crawlers you want

Easy to get wrong, especially if someone added a blanket rule during a scraping panic. Check your robots.txt for the AI crawlers you actually want indexing you (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, among others). Blocking them is a legitimate choice — just make sure it's a choice and not a leftover.

Comparison pages are retrieval bait

If the cited documents are mostly comparison content, one thing you control is publishing genuinely useful comparison content yourself. An honest "X vs Y" page — including where you lose — is far more likely to be retrieved for a comparison query than a features page. It also tends to be the highest-intent traffic you'll get.

The honest caveats

Two things worth saying plainly, because a lot of content in this space overclaims:

Nobody can guarantee AI citations. You can improve the signals that correlate with being retrieved and understood. You cannot control what any model outputs next week. Anyone selling you a promised ranking in ChatGPT is selling you something they can't deliver.

This is a moving target. Retrieval behavior differs across engines and changes with model updates. What I described is a snapshot, not a law. Measure your own category rather than trusting a generic list — including this one.

How to check your own category

Free version, takes ten minutes: open each of the four assistants, ask three or four buying questions a customer in your category would ask, and note which domains get cited. Not which companies get named — which domains get linked. Tally them. The pattern usually appears within a dozen questions.

If you'd rather not do it by hand, Greater Than Services (what I work on) runs those queries across ChatGPT, Perplexity, Gemini and Claude and gives you the source-type breakdown for your specific category, plus the schema fixes as copy-paste code. Free preview, no signup. But the manual method above genuinely works and costs nothing — the point of this post is the finding, not the tool.

TL;DR

Your homepage's job is to be unambiguous when a model encounters it. Getting encountered mostly happens elsewhere — on the review sites, directories, roundups and forums that AI answers are actually assembled from.

Fix the markup. Then go get listed.

Curious what the citation mix looks like in other categories — if you run the manual check for yours, drop what you find in the comments. Especially interested in whether dev-tool categories really are as forum-heavy as they look.