I run PostaGen, a small SaaS for the Brazilian market, and last week I added an app idea generator to it. You pick a niche, an audience and a country. It searches the web and returns 5 to 8 app ideas, each with a score, competitors, real user complaints and a price range.
The feature rests on one rule: a number without a source is never shown as fact. If the search didn't back it up, the page labels it as an estimate.
Below: how I enforced that rule with the Claude API, the hole a manual check found in it, and the fix a reader suggested in the comments of my first post about it.
The pipeline
-
Research call. One request with Anthropic's server-side web search tool (
web_search_20260209, up to 8 searches,user_locationset to the chosen country). The prompt asks for a source for every fact and tells the model to keep its own analysis apart from facts. The answer comes back as text with citations. - Structuring call. The API won't return citations and JSON structured output in the same request, so a second call, with no search, turns the notes into JSON with a fixed schema. It receives the numbered list of URLs the search returned and may only use those.
- Validation, no model involved. Every URL in the JSON must be on the list the search returned. A competitor without a valid URL is dropped. A sentence without a valid source is marked as an estimate, and so is a price range.
- MVP playbook and launch plan. Generated on demand, also with a JSON schema.
One detail you have to handle: a long search sometimes stops with stop_reason: "pause_turn". You send the assistant content back as it came and the model carries on. Skip that and the research ends halfway.
What it costs
Three scans from October 27, Brazilian market:
| Niche | Searches | Sources read | Time | Cost |
|---|---|---|---|---|
| Beauty (B2B) | 8 | 74 | 103 s | US$ 0.36 |
| Pets (consumers) | 8 | 76 | 93 s | US$ 0.32 |
| Education (creators) | 8 | 56 | 89 s | US$ 0.33 |
Nearly all of it is input tokens, because the search results go into the context (the beauty scan used 76.8k input tokens and 12.2k output), plus US$ 0.01 per search. The playbook costs about US$ 0.02 and the launch plan about US$ 0.03.
What the manual check caught
I opened the 56 links the three scans showed. 52 loaded, 3 didn't respond and 1 blocked automated access. The market figures I checked held up: 236 thousand beauty businesses formalized in 2025 (Sebrae), R$ 75.4 billion in pet sector revenue in 2024 (Abinpet), and the fees of Hotmart, Kiwify and Eduzz.
Three things were wrong:
- A sentence about the pet market said "about 160 million pets, among them 60 million dogs and 30 million cats, in about 32 million homes". The sources confirm the pets, the dogs (62.2 million) and the cats (30.8 million). The 32 million homes are in neither source. Validation passed it because both URLs were real and came from the search.
- One competitor's "website" was a complaint page on Reclame Aqui, Brazil's big consumer complaint site.
- A blog post comparing platform fees was listed as a competitor.
So a valid URL proves the source exists and was read. It says nothing about whether that page contains the number in the sentence.
The fix came from a comment
A reader pointed out something the first call already returns. Every web_search_result_location citation carries cited_text: the passage from the page that supports the sentence, up to 150 characters. My code kept the URL and threw the passage away.
Now the server keeps the passages per URL:
for (const block of message.content) {
if (block.type !== "text") continue;
for (const c of block.citations ?? []) {
if (c.type !== "web_search_result_location") continue;
const source = sources.get(c.url) ?? { url: c.url, title: c.title ?? c.url, excerpts: [] };
if (c.cited_text && !source.excerpts.includes(c.cited_text)) source.excerpts.push(c.cited_text);
sources.set(c.url, source);
}
}
The passages also go to the structuring call next to each URL, and the prompt asks the model to write a number the way the passage writes it. After the JSON comes back, every number in a market sentence has to appear in a passage cited by that sentence's sources. If any is missing, the sentence becomes an estimate.
Most of the work is reading the numbers. Brazilian sources write "62,2" for a decimal and "236.000" for thousands, English sources do the opposite, and both mix in words like "mil", "milhões" and "billion":
const SCALES: [RegExp, number][] = [
[/^(mil|thousand|k)$/i, 1e3],
[/^(milh(ão|ões)|mi|mm|millions?)$/i, 1e6],
[/^(bilh(ão|ões)|bi|billions?)$/i, 1e9],
];
// "62,2" and "62.2" are decimals; "236.000" and "1,500" are thousands
function parse(raw: string): number {
const last = Math.max(raw.lastIndexOf(","), raw.lastIndexOf("."));
if (last < 0) return Number(raw);
const sep = raw[last];
const decimal =
raw.includes(sep === "," ? "." : ",") || (raw.indexOf(sep) === last && raw.length - last - 1 !== 3);
return decimal
? Number(`${raw.slice(0, last).replace(/[.,]/g, "")}.${raw.slice(last + 1)}`)
: Number(raw.replace(/[.,]/g, ""));
}
export function numbersIn(text: string): number[] {
const re =
/(?<![\p{L}\d])(\d+(?:[.,]\d+)*)(?:\s?(mil|milh(?:ão|ões)|bilh(?:ão|ões)|mi|bi|mm|k|thousand|millions?|billions?)(?![\p{L}]))?/giu;
const out: number[] = [];
for (const [, raw, scale] of text.matchAll(re)) {
const value = parse(raw);
if (!Number.isFinite(value)) continue;
// A bare year dates the fact, and a 150-character excerpt rarely repeats it
if (!scale && /^\d{4}$/.test(raw) && value >= 1900 && value <= 2100) continue;
const factor = scale ? (SCALES.find(([r]) => r.test(scale))?.[1] ?? 1) : 1;
out.push(value * factor);
}
return out;
}
export function numbersMatch(sentence: string, excerpts: string[]): boolean {
const wanted = numbersIn(sentence);
if (!wanted.length) return true;
const found = excerpts.flatMap((e) => numbersIn(e));
return wanted.every((n) => found.some((f) => Math.abs(f - n) <= Math.abs(n) * 1e-9));
}
The test I wrote from the pet sentence:
const excerpt =
"O Brasil tem 160 milhões de animais de estimação, sendo 62,2 milhões de cães e 30,8 milhões de gatos";
numbersMatch("160 milhões de pets, 62,2 milhões de cães e 30,8 milhões de gatos, em 32 milhões de lares.", [excerpt]); // false
numbersMatch("160 milhões de pets, 62,2 milhões de cães e 30,8 milhões de gatos.", [excerpt]); // true
What it still gets wrong
- 150 characters is short. A number can be on the page and outside the passage, and the sentence gets marked as an estimate for nothing. The reader and I agreed that this is the right side to err on: a good number labeled as an estimate costs less than an invented one presented as fact.
- Rounding fails the check. The original sentence said "60 million dogs" while the source says 62.2 million. Under the new rule that sentence becomes an estimate, which is why the prompt now asks for numbers written as the passage writes them.
- A number can be in the passage and still be misused, with the wrong year or the wrong unit. The check is necessary, not sufficient. Opening the links is still on the reader.
-
I didn't use
blocked_domains. The reader also suggested blocking Reclame Aqui at the search. I kept it, because complaint pages are where the user pain that justifies an idea comes from. The rule went into validation instead: a complaint page is a valid source for a complaint and never a competitor's website.
I haven't rerun the three scans yet, so I don't know how many sentences now drop to estimates on real data. That's the next measurement.
Try it
The page shows the three scans with their sources, the ranking, the build prompt and the launch weeks, with no signup: postagen.ia.br/gerador-de-apps. The interface and the output are in Portuguese, though a scan can target the US market. The free plan gives 1 scan, 2 playbooks and 1 launch plan per month.
Stack: Next.js 16 on Vercel, Supabase with RLS (the owner can only read; writes go through the server) and the Claude API with web search.
If you check LLM-cited numbers some other way, I'd like to hear how in the comments.
Top comments (2)
the pet homes example is a good catch, real URL but the claim isn't in it. one cheap check on top of URL validation: have the structuring call return the exact quote it got each figure from, then verify in code that the number string appears in that quote and the quote appears in the fetched page text. anything that fails falls back to estimate. does the search tool give you enough page text to do the substring match, or would you need to refetch the pages?
A real URL proving nothing about the number is the exact hole. Keeping the cited passage instead of just the link is the fix that matters.