DEV Community: abhineet

How to track what ChatGPT says about your brand (open source, self-hosted)

abhineet — Mon, 19 Oct 2026 04:12:22 +0000

People now ask ChatGPT, Gemini and Claude things like "what's the best tool for X?" before they ever open Google. If your product is in that answer, you get a visit. If it isn't, you never find out.

I wanted to measure this properly, so I built CiteLadder and open sourced it under AGPL-3.0. This post walks through running it on your own machine and getting a first real result. No sign-up, and your data stays with you.

What you are measuring

You give it a set of real buyer questions (prompts). On a schedule it asks the answer engines and records every answer. From those answers you get:

which brands are named (yours and your competitors), your share of voice and rank
which pages and domains are cited
what the engines get wrong about you, checked against your own site
how the numbers change between runs that used the same prompts and engines

Engines covered: ChatGPT, Claude and Gemini through their APIs, plus ChatGPT Search, the Gemini app and Google AI Overviews through DataForSEO.

One honest note: this measures what the engines answer, observed over time. It does not prove why an answer changed. Site and demand data are leading indicators, not proof of cause.

Step 1: Run it

You need Docker with Compose v2.

git clone https://github.com/abhij1306/CiteLadder.git
cd CiteLadder
docker compose up -d --build

The first build takes a few minutes. Then open http://localhost:8100. There is no login. The Compose file binds to 127.0.0.1 only, so it is not exposed to your network.

No Docker? You can run it with Node.js 26+, pnpm 12 and PostgreSQL 16. The README has the commands.

Useful commands:

docker compose logs -f app   # follow logs
docker compose down          # stop, keep data
docker compose down -v       # stop and delete all data

Step 2: Create a project

Enter your website. CiteLadder researches the brand and proposes competitors and an ideal customer profile. You confirm or edit them. It is worth ten minutes here, because everything downstream depends on the competitor list.

For the Agent, brand research and prompt generation, set an LLM in .env (copy .env.example). Anthropic and OpenAI are fully supported. Any OpenAI-compatible endpoint works for research and generation, including a local Ollama server.

Step 3: Add your keys (bring your own)

Go to Settings > Providers and add the keys you have: OpenAI, Anthropic, Google, and DataForSEO if you want ChatGPT Search, the Gemini app or AI Overviews. You pay those providers directly. Keys are stored encrypted in your own database.

Back up the citeladder_data volume together with the database. The encryption key lives there.

Step 4: Write prompts like a buyer

Prompts are the whole experiment. Generate candidates from your site, market and competitors, then review them before they are tracked. Organise by topic, intent and buyer stage.

Good: "What are the best open source alternatives to X for a small team?"
Weak: "Tell me about MyBrand." Nobody asks that before they know you.

Step 5: Run an audit and read the answers

Run the audit. For each answer you can open the record: the engine, the prompt, the date, where your brand appeared and which sources were cited. Start with two numbers:

Visibility: how often you are named across tracked prompts
Owned citation rate: how often the cited pages are yours

Then look at who is named next to you, and which publishers keep showing up. Those are the pages that shape the answers in your category.

Run the same prompts again later. Comparing runs under the same prompts and engines is where the signal is.

Step 6: Check why

Site Health crawls your site (politely, with bounds) and checks things that affect AI answers: robots.txt and llms.txt access for AI crawlers, structure, evidence, freshness. It needs no keys. Findings roll into a ranked list of actions you can work through and mark done.

CiteLadder never publishes anything to your site for you.

Optional: use it from your AI assistant (MCP)

The MCP server has 46 tools. You can read visibility, sources, Site Health, actions, demand and traffic, and also make changes such as adding prompts or competitors, launching audits and updating actions. Every change is shown as a preview and applied only after you confirm.

For self-hosting, set MCP_TOKEN in .env to a long random string, restart, then for example:

claude mcp add --transport http citeladder http://localhost:8100/mcp \
  --header "Authorization: Bearer <MCP_TOKEN>"

Before you expose it

It is a single-workspace tool with no login. Put an authenticating reverse proxy in front of it before opening it to a network, and list any extra host names in ALLOWED_HOSTS.

Licence and hosted version

The code is AGPL-3.0. If you run a modified version as a network service, you must offer the source to its users. I am the only maintainer right now, so issues and PRs are welcome and I will read them.

If you would rather not run it, there is a hosted version at citeladder.com with a 7-day free trial on ChatGPT answers (limits apply).

Repo: https://github.com/abhij1306/CiteLadder
Hosted: https://citeladder.com
Docs: https://docs.citeladder.com

If you try it, tell me what was confusing in the setup. That is the most useful feedback I can get.

What an AI visibility score hides: mentions, citations, coverage and source evidence are different measurements

abhineet — Sun, 20 Sep 2026 03:30:01 +0000

A score can be directionally useful and still be operationally incomplete

AI visibility dashboards often compress a complicated observation into one number. That is convenient for a weekly report, but dangerous as a debugging interface.

If a score falls from 62 to 48, what changed?

Was the brand named less often?
Did the model cite fewer owned pages?
Did the prompt set change?
Did competitors gain mentions while the brand stayed flat?
Did the answer contain a citation that nobody can inspect?

Those are different questions. A single score cannot answer all of them unless its underlying observations remain available.

The practical distinction is between four measurements: mentions, citations, coverage, and source evidence.

1. Mentions: was the brand named?

A mention is a textual appearance of the brand, product, or organization in an answer. It answers a narrow question: did the answer say our name?

Mentions are useful for tracking recognition, recommendation language, and competitor co-occurrence. But a mention does not prove that the answer used the brand’s website as a source. An answer can name a company from general model knowledge, a directory, a review, or a retrieved page.

At minimum, store the exact answer text, engine, model or surface when available, prompt, run time, and the text span that triggered the mention classification. A percentage without those records is difficult to audit.

2. Citations: what sources did the answer point to?

A citation is a source link, source chip, or named reference attached to the answer. It answers: what did the engine show as grounding material?

Citation and mention are independent:

Result	Mentioned?	Owned page cited?
The brand is named and its documentation is linked	Yes	Yes
The brand is named but a review is linked	Yes	No
The brand is not named but its documentation is linked	No	Yes
Neither the brand nor an owned page appears	No	No

The third row is easy to miss. A page can be useful evidence for an answer even when the prose never names the company. Conversely, a name can appear without the company’s source supporting the claim.

3. Coverage: how much of the test surface was observed?

Coverage is about the measurement surface, not the answer’s wording. A team might define it as the percentage of prompt-and-engine cells that produced an observation, or as the percentage of relevant prompts where the brand was present. Those denominators must be stated.

For example, “48% visibility” could mean:

the brand appeared in 48 of 100 prompts;
the brand was mentioned in 48% of successful runs;
the brand received 48% of all cited sources;
48% of the monitored engines were reachable.

Each is a legitimate metric in the right context. None should be silently substituted for another.

Coverage also exposes sampling bias. A portfolio containing only branded queries will make a brand look visible. A portfolio containing only “best tools” queries may measure category competition instead. Keep the prompt portfolio versioned, and report changes to its size, intent mix, engines, locales, and schedules.

4. Source evidence: can another person re-check the observation?

Evidence is the audit layer. It connects a metric to the answer and the source behind it.

A useful evidence record includes:

the exact prompt and portfolio version;
the engine and surface used;
the timestamp and locale;
the raw or preserved answer;
extracted mentions and citations;
the URL for every cited source;
the claim-to-source relationship, where it can be determined;
the page snapshot or retrieval result used for review.

This is not the same as storing a screenshot of a dashboard. A screenshot shows a conclusion. Evidence lets a reviewer inspect the observation that produced it.

Why the four measurements should not be collapsed

Imagine two weekly reports:

Report A: mention rate is flat, owned citations rise, and the cited pages now cover more of the answer’s factual claims.

Report B: mention rate rises, but citations shift to third-party pages and the prompt set becomes smaller.

A composite score might rank both weeks as improvements. An operator would make different decisions: Report A suggests stronger source grounding; Report B needs investigation before it is called progress.

The same logic applies to competitors. Share of voice is not the same as citation share. A brand may be named often but receive little of the available source space. A competitor may appear less frequently but own the citations that answer the important questions.

A small data model beats a large mystery number

Teams do not need a perfect universal score to start. They need a stable observation table:

run_id
portfolio_version
prompt_id
engine
surface
locale
observed_at
answer_text_or_artifact
brand_mentioned
brand_recommended
owned_citation_urls
competitor_citation_urls
coverage_status
evidence_links

Derived metrics can then be calculated for a specific purpose. For example:

mention rate = prompts with a brand mention / eligible prompts;
owned citation rate = prompts with an owned citation / eligible prompts;
citation share = owned citations / all counted citations in the answer;
collection coverage = completed runs / scheduled runs.

The denominator and eligibility rule belong beside the metric. If a number cannot open to the records behind it, treat it as a summary, not as ground truth.

A practical review checklist

Before acting on an AI visibility score, ask:

What exactly is the numerator?
What exactly is the denominator?
Are mentions and citations separate fields?
Can I see the prompt and answer for a representative result?
Did the prompt portfolio, engine, locale, or model change?
Are the cited sources owned, earned, competitor, or unknown?
Can a colleague reproduce or challenge the classification?
Does the system preserve history instead of overwriting yesterday’s run?

This is the difference between a score that starts a conversation and a measurement system that supports a decision.

Closing thought

AI visibility is not one property of a brand. It is a set of observations about language, retrieval, source selection, and sampling. Mentions tell you whether the name appeared. Citations tell you what the engine pointed to. Coverage tells you how much of the test surface you actually observed. Source evidence tells you whether the result can be inspected.

Use the score as a summary. Keep the four measurements underneath it.

Disclosure: I work with CiteLadder, an AI-visibility product. The examples above are measurement guidance, not a claim that CiteLadder is the only valid implementation.

Further reading: CiteLadder’s AI visibility overview and FAQ on mentions, citations, and evidence.