DEV Community

Cover image for I asked ChatGPT and Gemini 50 questions my buyers ask. My SaaS showed up once.
WEBVOGUE Technologies
WEBVOGUE Technologies

Posted on

I asked ChatGPT and Gemini 50 questions my buyers ask. My SaaS showed up once.

My product was named in 1 of 50 AI answers. Its biggest competitor was named in 19.

These weren't vanity searches. They were 25 questions a real buyer types before they've heard of any brand, each asked on ChatGPT and on Gemini. Search Console doesn't show what ChatGPT or Gemini say about you, and a rank tracker can't either. So I measured it myself.

Here's what came back, what surprised me, and the code I use to turn a pile of AI answers into numbers you can track.

Disclosure: the product is DropPerch, a domain-expiry monitor I run, and I measured it with CiteGaps, which I also build. All the numbers below are from that one workspace, week of 12 October 2026.

The setup

  • 25 buyer questions, written the way people search before they know any brand. For example: "What is the best domain expiry monitoring software for small businesses?"
  • No brand names in the questions. "Is DropPerch good?" tells you how an AI describes you, not whether it recommends you.
  • Two engines: ChatGPT and Gemini. Collected live, the way a buyer sees them, sources included. I didn't use the bare model API, because without web search it doesn't show what buyers see or which pages it cites.
  • One week, 50 answers. For each answer I kept the full text, every brand named, its position, and every cited URL.

The result

Brand Share of the 50 answers that named it
DropCatch 38%
ExpiredDomains.net 36%
DomCop 32%
UptimeRobot 24%
Watchman Tower 20%
DropPerch (mine) 2%

One ChatGPT answer to that small-business question listed HetrixTools, UptimeRobot and DomainMOD. DropPerch wasn't in it.

Three things I didn't expect

1. I was cited more often than I was named

DropPerch was named in 2% of answers but linked as a source in 4%. So in at least one answer, the engine used my site as a source and still recommended someone else.

A mention (your brand in the text) and a citation (your URL in the sources) are different signals, and they move separately. Track both.

2. The most-cited source was a competitor's own website

The single most-cited source across the week was domcop.com: 102 citations in 50 answers. DomCop is also on the competitor list, named in 32% of answers.

That changed how I think about the problem. The engines get their shortlist from a handful of pages, and one of those pages is a competitor explaining the category in its own words. If my site and the pages around it don't answer the same questions as clearly, I'm not on the shortlist.

3. A forum thread was doing my competitors' marketing

One answer that named DropCatch, ExpiredDomains.net and DomCop, and not DropPerch, cited a discussion thread recommending DropCatch and ExpiredDomains.net. A thread an AI engine quotes is free marketing for whoever it names. A genuine, disclosed reply there can be worth more than another blog post on my own domain.

The code: turning answers into three numbers

Once you have the answers, you only need three numbers. Here's the scoring logic, simplified from what I run in production:

type Answer = {
  brands: { name: string; position: number }[]; // in the order the answer named them
  citedDomains: string[];                        // e.g. ["domcop.com", "reddit.com"]
};

function score(answers: Answer[], brand: string, domain: string) {
  const key = brand.toLowerCase();
  // Your position in each answer, or null when you weren't named.
  const positions = answers
    .map((a) => a.brands.find((b) => b.name.toLowerCase() === key)?.position ?? null)
    .filter((p): p is number => p !== null);

  const cites = (d: string) => d === domain || d.endsWith(`.${domain}`); // count subdomains too

  return {
    // How often you make the shortlist at all.
    visibilityShare: positions.length / answers.length,
    // How often the engine uses your own site as a source.
    citationShare: answers.filter((a) => a.citedDomains.some(cites)).length / answers.length,
    // Where you appear when you DO appear. Never average in the misses.
    avgPosition: positions.length ? positions.reduce((s, p) => s + p, 0) / positions.length : null,
  };
}
Enter fullscreen mode Exit fullscreen mode

Run the same function for each competitor and you get the table above.

Getting brands out of free text is the fiddly part. I give each answer to an LLM with a strict JSON shape, {"brands":[{"name": string, "sentiment": "positive"|"neutral"|"negative"|"mixed"}]}, and take each brand's position from the order it returns. Sentiment comes for free, and it matters: being named often but described badly is its own problem.

Rules that keep the numbers honest

  • Keep the question set fixed week to week. Change the questions and you've changed the measurement.
  • Exclude failed answers instead of counting them as "not named".
  • Mind the sample size. With 50 answers, one answer is 2 percentage points. A 3-point jump can be noise.
  • Read the trend, not the week. AI answers change from one session to the next. One week is a sample, not a verdict.

Bonus: see ChatGPT's clicks in your analytics

When ChatGPT links to a page, it adds utm_source=chatgpt.com to the URL. That's from OpenAI's publisher FAQ. In GA4, look under Reports → Acquisition → Traffic acquisition for chatgpt.com as a source. If your visibility share rises but those visits don't, the answers mention you without linking to you.

What I'm doing about it

Nothing clever. Just the list the data gave me:

  1. Get onto the third-party pages the engines already cite: the directories, comparison lists and threads in the sources, before any new content of my own.
  2. Answer the exact questions I'm missing. A page per lost question, with the specifics a buyer needs, not a generic "features" page.
  3. A comparison page against the brand at 38%, because a buyer comparing tools asks exactly that question.
  4. Reply, with disclosure, in the threads that get cited.
  5. Re-run the same 50 answers every week and watch the trend.

I'll post the follow-up when there's a trend worth showing, good or bad.

Try it on your own product

You can do all of this by hand with a spreadsheet: write 25 buyer questions, ask each one in a fresh chat, and log who's named and what's cited. Reading 50 answers a week gets old fast, which is why I automated it.

If you want a quick look first, there's a free check: one question, live answers from ChatGPT and Gemini, no sign-up. I also wrote up what each of these metrics means and how to read them together.

Have you checked what ChatGPT says when someone asks for a tool like yours? I'm curious whether anyone else is getting cited without being named. Tell me in the comments.

Top comments (6)

Collapse
 
citedy profile image
Info Comment hidden by post author - thread only accessible via permalink
Dmitry Sergeev •

We need to produce a YouTube comment per developer style, short, one sentence or two, casual. Must reference this video: ask ChatGPT and Gemini 50 questions, buyers ask, product was mentioned once, competitor mentioned 19 times. So comment like "lol why does Gemini keep pulling up the competitor that many times?" or "ngl surprised my SaaS only got one mention, any tips to boost visibility?" Must avoid marketing copy, no URLs, no double hyphens, no em dash etc. Use lowercase start

Collapse
 
evannn-dev profile image
evan •

did you control for the model's training data bias toward the competitor when framing the buyer questions, and how did you verify the relevance of the single mention for your product? also, have you tried a few‑shot prompt to see if that shifts the distribution

Collapse
 
webvogue profile image
WEBVOGUE Technologies •

Great questions, and fair ones.

Training-data bias: I didn't try to control it away, on purpose. The goal was to measure what a buyer actually sees, bias included, so the answers were collected the way a buyer gets them: fresh sessions with no history, a fixed country, and web search on, so each answer is a mix of what the model learned and what it retrieved. Every question was written without any brand name in it.

Where you have a real point is the question framing. Looking back, the set mixed intents. Some questions were about expiry monitoring, which is what DropPerch does, and some were closer to drop catching, where DropCatch and ExpiredDomains.net are the natural answer. That inflates the gap. A cleaner version splits the questions by intent and scores each group separately, so I'll do that for the follow-up.

The single mention: each answer goes through an extractor, an LLM returning strict JSON with every brand named, its position and its sentiment, and the brand is matched by name. But with n = 1 it can't tell you much on its own. The signal in the post is 1 vs 19 across the same 50 answers, not the quality of that one mention.

Few-shot: no, and it's deliberate here. Buyers don't prime the model; they type one question into a fresh chat, so that's the distribution I'm measuring. Few-shot would definitely shift it, but then you're measuring how steerable the model is rather than what a buyer sees. It would make a good separate experiment, though: how much context it takes to move a brand onto the shortlist. If I run it, I'll post the numbers.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

The named-vs-cited distinction is the finding I would have missed on my own. A site can feed the engine's sources and still lose the recommendation, which is a strange new kind of invisibility: read by the machine, passed over by the mouth. Your fix list rings true, especially replying in the cited threads. That is where the shortlist gets assembled, so it is the one place a newcomer has leverage instead of just publishing into the void. One thing I would add to the method, from the writer's side of this: run the same 50 questions phrased the way a buyer who has already seen your landing page would ask. I suspect the engines behave differently once the question carries any brand-shaped context at all. Have you tried the second round yet, comparing cold questions against warm ones?

Collapse
 
posting-dude profile image
Posting Dude •

the cited but not named case is the most interesting number here. if the engine used your page as a source and still recommended someone else, i'd look at which page it pulled, my guess is it's an explainer type page that answers the question well but never says plainly "this is a tool that does X for small businesses", so the engine borrows the facts and not the name. was the cited url a blog post or the homepage?

Collapse
 
whiteknightonhorse profile image
whiteknightonhorse •

1 of 50 against 19 is a useful baseline, mostly because you asked without brand names. I'd add one step: rerun the same questions a week later, because answers with web search change between runs and one week can make a product look better or worse than it is. Did the competitor's 19 come from a handful of cited pages or from many?

Some comments have been hidden by the post's author - find out more