I asked AI assistants 220 times whether they'd recommend my Mac app. Here's what I learned.

I make FavTray, a Mac menu bar app for developers. It shows Claude, Codex and Cursor usage limits, GitHub PRs and CI, which process is holding a localhost port, and a few other things.

Google Search Console tells me how I rank on Google. It tells me nothing about the question I care about more and more:

When a developer asks an AI assistant "what's a good menu bar app to track my Claude Code limits?", does my app come up?

So I measured it. I took 50 questions, sent them to two AI models with web search turned on, and ran 220 calls in total. Everything below is from those runs; nothing is estimated.

The setup

I wrote 50 questions the way a developer would actually type them. 40 are generic, meaning they're about a problem and don't name my app:

• "What is the best Mac menu bar app to track my Claude Code usage limits?"
• "How do I find and kill the process using a localhost port on a Mac?"
• "Best notch app for MacBook to use the notch area"
• "Free alternative to Bartender for organising Mac menu bar icons"

The other 10 are branded: "What is FavTray?", "FavTray vs Raycast", "How much does FavTray Pro cost?"

For each answer I recorded three things:

1. Mentioned: does the answer name the app?
2. Cited: does it link to favtray.com?
3. Cited domains: which sites does it link to instead?

I ran the questions through two models:

• Claude Haiku 5.5 with Anthropic's web search tool. 50 questions once, and the first 10 twice more to measure variance: 70 calls.
• DeepSeek (deepseek-flash) with web search. All 50 questions, three times each: 150 calls.

Here's the core of the Claude probe in Python:

import anthropic

client = anthropic.Anthropic()

def probe(question: str) -> dict:
    r = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=2000,
        tools=[{"type": "web_search_20250305", "name": "web_search", "max_uses": 5}],
        messages=[{"role": "user", "content": question}],
    )
    text, cited = "", []
    for block in r.content:
        if block.type == "text":
            text += block.text
            cited += [c.url for c in (block.citations or []) if getattr(c, "url", None)]
    return {
        "mentioned": "favtray" in text.lower(),
        "cited": any("favtray.com" in u for u in cited),
        "cited_urls": cited,
        "searches": r.usage.server_tool_use.web_search_requests,
    }

(The full script also handles pause_turn, adds up token usage and runs the questions on a thread pool.)

Gotcha #1: the newer search tool gave me no citations

My first runs used the newer web_search_20260209 tool, which filters search results with code before Claude reads them. The answers were fine, but the text blocks came back with no citations, so I couldn't tell which sites Claude relied on. The older web_search_20250305 tool attaches citations to each sentence.

If you want to measure sources rather than just get an answer, check that citations actually come back before you scale up.

Result: branded questions are easy, generic ones aren't

On Claude Haiku 5.5:

Question type	Mentioned	Linked to favtray.com
Branded (10)	10 / 10	10 / 10
Generic (40)	7 / 40	9 / 40

Claude finds the app when you ask about it by name. On the generic questions, the ones that bring in new users, it's mostly absent.

Where it did show up was the broad, "bundle of tools" questions:

• "Best developer dashboard app for the Mac menu bar in 2026"
• "All-in-one Mac utility app for developers instead of many small menu bar apps"
• "iStat Menus alternative for developers"
• Eye-rest reminders, window snapping, cloud costs in the menu bar, Pomodoro breaks

Where it didn't show up was the app's sharpest features:

• tracking Claude Code / Cursor / Codex usage limits
• GitHub PR and CI notifications
• killing the process on a localhost port
• notch apps

For those, Claude recommended single-purpose tools: small dedicated sites, GitHub repos and App Store listings.

So the AI's picture of my product is "general dev dashboard". It doesn't connect it to the specific problems people actually search for.

Which sites do the AI answers cite?

The sites cited most often across the 40 generic Claude answers:

Domain	Citations
github.com	11
favtray.com	9
apps.apple.com	7
dev.to	7
sessionwatcher.com	5
alternativeto.net	4

GitHub, the App Store, dev.to and AlternativeTo do much of the work here. If your product isn't listed on the sites these models already trust, your own homepage has to do everything.

(Yes, that's part of why this post is on dev.to.)

Answers change a lot between runs

I asked the first 10 questions three times each on Claude and compared which domains were cited:

• The overlap of cited domains across the three runs ranged from 0% to 62% depending on the question.
• For "best Mac menu bar app to track Claude Code usage limits", my app was mentioned in 1 run of 3.

A single run tells you very little. If you track this over time, sample each question several times and report a rate ("mentioned in 33% of runs"), not a yes/no.

DeepSeek: a different index, a different picture

DeepSeek's API supports the same web search tool through its Anthropic-compatible endpoint, so the probe ran with just a different base_url. The results:

Question type	Mentioned (3 runs each)
Branded (10 × 3)	30 / 30
Generic (40 × 3)	0 / 120

Zero. Then I noticed something odd about the setup: through that endpoint, the search results come back empty to the caller, and DeepSeek rarely writes links in its answer. So there was no way to see what it found.

DeepSeek also has a standalone Search API (POST /v1/tools/search). It returns ranked URLs with a relevance score, so I queried it directly:

• "What is FavTray?" The top results were third-party listing pages. favtray.com itself wasn't in the top 8.
• "Mac menu bar app to track Claude Code usage limits" returned PyPI packages, GitHub repos and App Store apps. My app wasn't there.

Different engines have different indexes, and being indexed in one doesn't mean you're in another. A setup that works better for measuring is two steps: call the Search API yourself, then have the model answer from those results with numbered citations. That way you can see both whether you're found and whether you're used.

What it cost

Cheaper than I expected.

Claude Haiku 5.5:

• 70 calls cost $1.26 in total, about 1.8¢ per question on average.
• 86% of that is the web search fee ($10 per 1,000 searches), not tokens. The model choice barely matters; the real cost control is max_uses.
• Each answer averaged 1.5 searches, ~18k input tokens and ~8 seconds.

DeepSeek:

• Searches are $0.75 per 1,000 (free until 1 November 2026, according to their docs), and the tokens are cheap.
• Answers averaged 3.4 searches and ~14 seconds.
• The model spends a lot of output on reasoning, so give it a generous max_tokens. 4 of my 150 calls still hit a 6,000-token cap.

Ongoing tracking: 50 questions × 3 samples, weekly, on Claude Haiku comes to roughly $12 a month for one product.

What I'm changing

1. Describe the specific jobs, not just the bundle. Each sharp feature (Claude Code limits, kill a port, CI status, notch) gets its own clear page whose title matches how people actually ask.
2. Be where the models already look: GitHub, the App Store, AlternativeTo, and posts like this one.
3. Track a rate, not a snapshot. Same questions, several samples, every week, and watch whether the generic mention rate moves from 7/40.

Try it on your own product

You need about 50 honest questions, one API key and a couple of dollars. Write the questions without your product name. That's the test that matters, and it's the one you're probably failing.

If you run it, I'd like to know your branded-vs-generic numbers. Mine were 10/10 vs 7/40, and I don't think that's unusual.

---

FavTray is a free Mac menu bar dashboard for developers (Pro is a $4.99 one-time purchase). Plenty of other good menu bar apps exist too, as the AI models made clear.