ChatGPT cited our guide and recommended a competitor, so I built a monitor for it

We make a small app that clears your TikTok reposts in bulk. A few days ago installs dropped by about a quarter in one day, and the cause wasn't on our side at all.

Most of our website traffic doesn't come from Google. It comes from chatgpt.com. When ChatGPT changed how it answers "how do I delete all my TikTok reposts", we felt it by the next morning. I asked it the same question myself: the steps in the answer were taken from our guide, but the app it recommended was a competitor with thousands of App Store ratings.

So there are two different things an AI answer can give you: being the source and being the recommendation. Search Console tells you about neither. I wanted a number for both, every day, so I built a small monitor. Here's how it works and what surprised me.

The setup

• A Cloudflare Worker with a D1 database, run by cron every 30 minutes.
• 27 questions real users ask: 15 in English, 12 in ten other languages (Spanish, Portuguese, German, French, Italian, Dutch, Russian, Japanese, Turkish...).
• Two engines: OpenAI's Responses API with the web_search tool (gpt-5-mini), and Perplexity's API.
• Each question is asked once a day per engine. Every tick takes a batch of 8 pending pairs, with up to 3 attempts per pair per day.
• For every answer it stores the text, the sources the engine cited, and the search queries it ran.

A small admin page shows, per question, whether we were cited, whether we were named, and which domains were cited instead of us. That last list turned out to be the most useful part.

Lesson 1: "sources" and "citations" are not the same thing

My first Perplexity parser saved every URL from the search results. That gave 9–10 "sources" per answer and made us look great. It was wrong: the engine reads all of those results but only cites a few of them in the answer.

The answer text marks citations inline as [1], [2]... (sometimes [web:1]), and the number is the 1-based position in the search results. So the fix is to keep only the URLs the text actually points to:

export function parsePerplexity(json) {
  let answer = "";
  const byId = new Map();
  let pos = 0;
  for (const item of json.output || []) {
    if (item.type === "search_results") {
      for (const r of item.results || []) {
        pos++;
        const id = String(r.id ?? pos);
        if (r.url && !byId.has(id)) byId.set(id, r.url);
      }
    } else if (item.type === "message") {
      for (const c of item.content || [])
        if (c.type === "output_text") answer += c.text || "";
    }
  }
  const citations = [];
  for (const m of answer.matchAll(/\[(?:web:)?(\d+)\]/g)) {
    const url = byId.get(m[1]);
    if (url && !citations.includes(url)) citations.push(url);
  }
  return { answer, citations };
}

After the fix, a typical answer cites 2–4 sources. That's a much more honest picture. For OpenAI it's simpler: cited sources come back as url_citation annotations on the output text.

Lesson 2: run the slow engine one call at a time

On our usage tier Perplexity answered 429 Request rate limit exceeded as soon as 3–4 calls ran in parallel. OpenAI didn't mind. So each engine gets a serial flag, and the batch runs serial engines one call at a time while the others run in parallel:

const done = (await Promise.all(engines.map(async (e) => {
  const mine = batch.filter((t) => t.engine === e);
  if (!e.serial) return Promise.all(mine.map(run));
  const out = [];
  for (const t of mine) out.push(await run(t));
  return out;
}))).flat();

Since then there have been no failed pairs.

Lesson 3: most "losses" were pages we already had

When we weren't cited, I expected to find topics we had no page for. That was almost never the case. We had the page; the engine just didn't find it. The engines run their own web searches first (the monitor logs those queries), and a young domain with few links simply doesn't come up.

Bing matters more than I thought here. When I inspected our URLs in Bing Webmaster Tools, most of them were "discovered but not crawled": no technical problem, Bing just hadn't chosen to fetch them yet. So the work that actually moves the number is boring: getting pages crawled, and getting a few honest links from places crawlers already visit.

What I'd tell anyone starting this

• Count citations, not search results. Otherwise you'll celebrate numbers that aren't real.
• Log the engine's own search queries. They show you the exact phrasing to answer on your pages.
• Ask in the languages your users speak. Our results in Portuguese and Spanish were very different from English.
• Expect the recommendation to follow reviews and ratings, not your content. Content gets you cited; being recommended takes proof from real users, and that can't be faked or rushed.

If you're curious what the guides look like, they're here: repostless.app/blog. I also put the short answers in a public FAQ repo: github.com/repostless/tiktok-reposts-faq.

Is ChatGPT already a real traffic source for you? How are you measuring it?