I Wrote 40 Lines of Python to Find Out If ChatGPT Knows My Company Exists

A client sent me a screenshot in June. They had asked ChatGPT to name good agencies in their category and a competitor came up. They did not. The competitor is not bigger than them and does not rank better in Google. The question in the email was reasonable and I did not have an answer: why them and not us?

I run ESBO Ltd, a digital PR agency, so this is my problem twice over. Once for the client and once for us.

I could not check it properly by hand. Open a chat, ask the question, get an answer, close the tab. Ask again tomorrow and the answer is different because the model samples its next token instead of looking anything up. One conversation tells you nothing. You need the same question asked many times before the shape of the answer means anything.

That is a loop, and loops are the one thing I know how to ask for.

What the script does

Ask a set of category questions, many times each, and count which names come back. That's the entire idea. The reason it works is the reason a single chat doesn't: at any temperature above zero the model is sampling, so twenty runs of one prompt give you a distribution instead of an anecdote.

import json, collections
from anthropic import Anthropic

client = Anthropic()

QUESTIONS = [
    "Which agencies are best known for digital PR in Europe?",
    "Who should a SaaS company hire for link building?",
    "Name agencies that do multilingual PR placements.",
]

EXTRACT = """List only the company names mentioned in the text below.
Return a JSON array of strings, nothing else.

TEXT:
{text}"""

def ask(prompt):
    r = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1000,
        messages=[{"role": "user", "content": prompt}],
    )
    return r.content[0].text

counts = collections.Counter()

for q in QUESTIONS:
    for _ in range(20):
        answer = ask(q)
        names = json.loads(ask(EXTRACT.format(text=answer)))
        counts.update(n.strip().lower() for n in names)

for name, n in counts.most_common(25):
    print(f"{n:>4}  {name}")

Two calls per run: one for the answer, one to pull the names out of it. Using a model to parse a model's prose felt like cheating the first time I did it, but regex on free text was worse. Claude wrote both halves. I supplied the questions, which turned out to be the part that mattered.

The questions are the whole experiment

My first question list was garbage. I wrote things like "is ESBO a good agency," which is not a question anyone types, and the model obligingly said nice things about a company it barely knew. Vanity in, vanity out.

The questions that produce useful output are the ones a buyer would actually ask, with your company's name absent from them. "Who should I hire for X." "Best tools for Y." "Which vendors do Z in Germany." If your name only appears when you put it there, you have your answer already.

Write ten of those, run them twenty times each, and the counter tells you who owns your category in the model's head. The first time I ran mine, the top of the list was not who I expected, and two of the names were companies I would not have described as competitors at all.

What I did with the output

The counter gives you names. The useful part is the next question, which is where those names came from.

I dug into it and wrote up what I found for Entrepreneur a couple of weeks ago. Short version: when Ahrefs studied 75,000 brands, branded mentions across the web predicted AI visibility about three times better than backlinks did. Muck Rack looked at 25 million links cited by ChatGPT, Claude and Gemini and found 84% came from earned media, against 0.3% from paid and advertorial content.

That second number reorganized my week. Link building is what ESBO Ltd sells, so I have a direct commercial interest in links continuing to matter, and they do, for rankings and for referral traffic. But the thing that gets you named in an answer is being talked about in places the model read, whether or not anyone bothered to hyperlink you. For ten years my industry logged an unlinked brand mention as a failed campaign. Turns out we were throwing away the receipt.

Things that will bite you

Cost. Ten questions times twenty runs times two calls is 400 requests. Cheap on a small model, less cheap if you point it at the expensive one and forget it's in a nested loop.

Name normalization. Models write company names inconsistently, and my counter treated each spelling as a separate company until I noticed. Lowercasing helps. It does not fix "Acme" versus "Acme Digital."

The obvious trap. Do not ask the model whether it likes you. It will say something kind and you will feel great and learn nothing.

Sample size. Five runs is noise. Twenty starts to hold still. I have not tested where the curve flattens, and I would like to know.

Where I've landed

I run it monthly now. It is not a rank tracker and I have stopped wanting it to be one, because there is no rank, just a probability that your name comes up when someone asks. Watching that probability move is still more information than the screenshot my client sent me in June.

If you run something like this on your own category, I'd be curious whether your top three matches what you'd have guessed. Mine didn't, and that gap is the only genuinely useful thing the script has told me so far.