ChatGPT Might Be Recommending Your Competitor, Not You — Here's How We Built a Tool to Check

Tabor Bachelor — Wed, 01 Jul 2026 15:36:43 +0000

Ask ChatGPT who the best project management tool is. Or the best CRM for startups. Or the best anything-in-your-category. It'll give you an answer — a specific one, with a specific brand in it. If yours isn't in that answer, you've lost a customer you never knew was in play.

That's the problem I set out to measure with Relevyn: how visible is a brand across the AI engines that are quietly becoming the new front page of the internet — ChatGPT, Claude, Perplexity, and Gemini. Turns out, actually measuring that is a more interesting engineering problem than it sounds.

Four APIs, four different ways to fail

The obvious approach: send the same prompt to all four engines, see who gets mentioned. The reality is each provider fails differently, on its own schedule.

Early versions ran everything in parallel with Promise.all — which meant one slow or erroring engine took the entire scan down with it. Swapping to Promise.allSettled fixed that: if Perplexity times out, you still get results from the other three instead of nothing.

const responses = await Promise.allSettled(
  engines.map((engine) => callEngine(engine, prompt))
);

Each engine call also gets its own retry logic (3 attempts, short backoff on a 503) and its own timeout, raced against the clock so one hung request doesn't stall the whole scan. Even with that, we had to drop from 6 prompts per scan down to 3, run sequentially instead of one big parallel burst — the combined latency of 4 engines × 6 prompts kept tripping serverless function timeouts.

Scoring text without paying another LLM to judge it

The tempting move here is to hand each response to an LLM and ask "was the brand mentioned, and was it positive?" We deliberately didn't do that for the core detection — it's slower, non-deterministic, and adds cost to something that needs to run thousands of times a day.

Instead: plain detection logic on the raw text. Is the brand name present. Where in the response does it first appear. Which sentences mention it, and do those sentences skew toward positive or negative language. Are competitor names present in the same response. It's less elegant than "ask an AI to grade the AI," but it's fast, free, and gives the same input the same score every time — which matters when you're about to tell someone their score changed.

What a scan actually costs

Running real queries against four commercial LLM APIs at scale isn't free, but it's cheaper than you'd think if you're deliberate about it: a full multi-engine scan costs somewhere in the $0.06–0.09 range, mixing cheaper models for high-volume detection work against pricier ones only where reasoning quality actually matters for the output. The expensive mistake we made early on was leaving prompt caching headers on for a low-volume beta — they added latency and occasional 503s with no caching benefit at that traffic level. Removing them fixed more reliability issues than any retry logic did.

The finding that changed how we think about this

Run the exact same query against the exact same brand two weeks apart and you will not get the exact same answer. These models aren't static lookups — they're generating an answer fresh each time, shaped by whatever's changed in their training and retrieval since the last time you asked.

Which means a one-time "AI visibility audit" tells you almost nothing. It's a photograph of a moving target. The only thing that's actually useful is watching the trend — which is the entire reason Relevyn runs as a monitor, not a single report.

If you want to see where your own brand lands, the scan's free: relevyn.com.

DEV Community: Tabor Bachelor

ChatGPT Might Be Recommending Your Competitor, Not You — Here's How We Built a Tool to Check

Four APIs, four different ways to fail

Scoring text without paying another LLM to judge it

What a scan actually costs

The finding that changed how we think about this