Someone asked how I handle noise in search results when there's no clean dataset to fall back on. Here's the full answer: six layers, none of which trust the model.
After I wrote about building a car pricing engine for a market with no data, @topstar_ai asked something in the comments that deserved more than a reply:
I was struck by the idea that LLMs can synthesize a price at request time from live search results, which flips the traditional supervised learning approach on its head. The fact that the dataset is no longer a prerequisite, but rather the exhaust, is a game-changer for building AI products in data-poor markets. I've worked on similar projects where we had to rely on proxy data sources and clever feature engineering to overcome data scarcity, but the LLM approach seems to offer a more scalable and flexible solution. How do you handle the issue of data quality and noise in the search results, and what kinds of domain guardrails have you implemented to ensure the model's output is reliable and trustworthy?
Two questions in there, and they're the right two. If your ground truth is live search results instead of a curated dataset, then your data quality problem doesn't go away. It moves. It becomes a runtime problem you have to solve on every single request.
Here's what I actually do at AutoValue, layer by layer.
First, what the noise actually looks like
Generic advice about "filtering low-quality sources" is useless until you know the specific failure modes. Mine, when I search for Nigerian used car prices:
- Asking prices, not transaction prices. Every listed price on Jiji is an opening move. Nigerian buyers negotiate 10 to 20% below the ask as a matter of routine. A model that reports the listed number is systematically overpricing the market.
- Pre-devaluation listings that never got taken down. The naira went from around ₦460 to the dollar to around ₦1,650. Listings from 2022 are still indexed and still show 2022 prices. They aren't wrong, exactly. They're archaeology.
- Trim contamination. Search "Toyota Land Cruiser Prado price Nigeria" and most of what comes back is the VX full option, because that's what dealers advertise. If you take that as the base price for a standard trim, every Prado prices high forever.
- Currency mixing. Export and auction sites quote USD. Nigerian listing sites quote naira. Both land in the same result set.
- Scam bait. A ₦2.5 million Range Rover is not a data point. It's a lure. But it's a perfectly well-formed number that any naive parser will happily average in.
Notice that only the last one is "bad data" in the usual sense. The rest are all correct data that's wrong for my purpose. That distinction drives the whole design.
Layer 1: Constrain the search, not just the parsing
The cheapest filter is the one that stops noise from being retrieved at all.
const query = `${year} ${make} ${model} price Nigeria site:jiji.ng OR site:cars45.com`;
const searchRes = await fetch("https://google.serper.dev/search", {
method: "POST",
headers: { "X-API-KEY": serperKey, "Content-Type": "application/json" },
body: JSON.stringify({ q: query, gl: "ng", hl: "en", num: 10 }),
});
site: scoping to the two marketplaces that actually matter, and gl: "ng" to geo-locate the search in Nigeria. That last one matters more than it looks. Without it you get US results for the same car, in dollars, for a market with entirely different import economics.
I also prioritise Google's answer box and knowledge panel above organic results when they're present, because Google has already done a round of synthesis across sources:
const priorityLines: string[] = [];
if (answerBox?.snippet || answerBox?.answer) {
priorityLines.push(`GOOGLE SUMMARY: ${answerBox.title ?? ""}\n${answerBox.snippet ?? ""}`);
}
const organicLines = organic.slice(0, 7).map(r => `${r.title ?? ""}\n${r.snippet ?? ""}`);
const allLines = [...priorityLines, ...organicLines];
Layer 2: Separate extraction from reasoning
This is the structural decision everything else depends on, and it's the one I'd most want someone to take away.
There are two completely different jobs here. One is read these messy snippets and tell me what this car currently sells for. The other is given a base price, adjust it for this specific car's mileage, condition, trim and location. The tempting move is one prompt that does both. Don't.
They're two separate Claude calls in my pipeline, and the reason is auditability. When a price comes out wrong in a single-call design, you cannot tell whether the model misread a snippet or misapplied an adjustment rule. Split them and every bad output localises to one stage. My debugging time dropped more from this change than from any prompt improvement.
Extraction runs on Haiku with a procedure, not a request:
Extract the REALISTIC TRANSACTION PRICE RANGE, what this car actually sells
for, not what sellers hope to get.
- Naira devalued sharply: ₦460/$ in 2022 to ₦1,650/$ in 2026. Car prices are
now 3 to 4x higher than pre-2023.
- Jiji and Cars45 show ASKING prices. Nigerian buyers negotiate 10 to 20%
below the listed ask.
- STEP 1: any price more than 60% below the highest-priced cluster member is a
pre-devaluation or junk listing. SKIP IT.
- STEP 2: SKIP listings marked "5+ years on Jiji". ENTERPRISE is a dealer
badge, do NOT skip those, dealers are valid sellers.
- STEP 3: prices in $ multiply by ₦1,650.
- STEP 4: from remaining valid prices take the MIDDLE cluster. Ignore the
bottom 15% and anything above 2x the cluster median.
- price_low = 25th percentile. price_high = 75th percentile.
- If only 1 to 2 valid prices remain: low = price × 0.88, high = price × 1.08.
Every one of those rules is a scar. STEP 2's carve-out for ENTERPRISE exists because my first version of "skip listings with badges" quietly deleted all the dealer inventory, which is the most reliable pricing signal on the site.
Layer 3: Percentiles, never averages
Worth calling out on its own because it's a one-line change with outsized effect.
The output is the 25th and 75th percentile of the surviving cluster. Not the mean, not min and max. A single scam listing at ₦2.5M destroys a mean and it destroys a min. It barely moves a 25th percentile. Robustness to outliers is a property you get for free from the right statistic, and no amount of prompt engineering substitutes for picking it.
The narrow-data case gets handled explicitly too. With only one or two valid prices left, percentiles are meaningless, so the rule degrades to a fixed spread around the single observation. Say what should happen when the data is thin, or the model will invent something.
Layer 4: Make the model report, let code decide
Trim contamination is the subtlest failure in the list, and the fix generalises well beyond pricing.
I could ask the extraction model to "adjust the price down if the listings look like a high trim." That's asking for a silent, unauditable, unrepeatable adjustment buried inside a number. Instead the model reports what it saw as a separate field, and the arithmetic happens in code where I can read it:
const TRIM_FACTOR: Record<string, number> = {
full_option: 1.12,
limited: 1.10,
sport: 1.08,
standard: 1.00,
unknown: 1.00,
};
const trimFactor = TRIM_FACTOR[searched.detected_trim] ?? 1.00;
const normalizedLow = Math.round(searched.price_low / trimFactor / 50_000) * 50_000;
const normalizedHigh = Math.round(searched.price_high / trimFactor / 50_000) * 50_000;
Now every stored anchor is a base-trim price, the trim premium gets re-added later as a visible line item the seller can see, and I can change the multipliers without touching a prompt.
Ask the model for observations. Keep the calculations in code. If a model's judgment is going to move a number, make it move the number through a mechanism you can print.
Layer 5: A deterministic invariant after the model responds
The adjustment model is given explicit percentage rules: excellent interior is +2%, mileage below expected earns +0.3% per 10,000km for a 4 to 7 year old car, and so on.
It does not always follow its own rules. Verified live: on one car it applied a +7.8% mileage bonus where its own stated rule produced 0.3%. That's not hallucination in the dramatic sense. It's a model doing approximate arithmetic and landing somewhere plausible-looking.
So there's a clamp that runs in code after the response comes back:
const CONDITION_UPLIFT_KEYS = ["condition", "interior", "engine", "mileage"] as const;
function applyAgeConditionCeiling(estimate, base, carAge) {
// Younger car, smaller allowed uplift: "excellent" is the expected norm for a
// 2-year-old car and already priced into the anchor, but a genuine rarity on
// a 10-year-old one.
const capPct = carAge <= 3 ? 0.03 : carAge <= 7 ? 0.06 : 0.10;
const baseMedian = (base.price_low + base.price_high) / 2;
const upliftNGN = CONDITION_UPLIFT_KEYS.reduce(
(sum, key) => sum + Math.max(0, estimate.adjustments?.[key] ?? 0), 0
);
const upliftPct = upliftNGN / baseMedian;
if (upliftPct <= capPct) return { estimate, cappedPct: 0 };
// Subtract only the excess above the ceiling, proportionally.
const excessPct = upliftPct - capPct;
...
}
The domain reasoning matters as much as the code. A search anchor for a two-year-old car already assumes near-excellent condition, because that's what two-year-old cars on the market are. Stacking excellent body, excellent interior, excellent engine and a low-mileage bonus on top of that anchor prices a clean used car like a new one. The failure was worst on thin-data models, where the only search results are new-car dealer prices and the anchor is already the ceiling.
Trim and location uplifts are deliberately exempt from the cap, because they're structural rather than condition-based. Trim was divided out in layer 4 and is legitimately being re-added.
Never trust a model to guarantee an arithmetic invariant. If a relationship must hold, enforce it in code after the fact.
Layer 6: Fail closed
This is the one I'd argue about with most people building LLM products.
When the search returns nothing usable, the endpoint does not fall back to the model's own knowledge. It returns a 503:
if (!searched) {
return NextResponse.json({
success: false,
error: "pricing_unavailable",
message: "We couldn't find live market data for this car right now. Please try again in a few minutes.",
}, { status: 503 });
}
Most LLM products fail open. Retrieval comes back empty, and the model answers anyway from training data, and the user cannot tell the difference between a grounded answer and a guess. For a chatbot that's mildly bad. For a product whose entire value proposition is "this number is trustworthy," it's fatal, because the failure is invisible exactly when it's most damaging.
Related, and covered in more depth in the previous article: model-generated prices were once cached as anchors, then re-injected on the next request as if they were market data. The fix has since hardened from a runtime bypass into a type. The stored anchor's source can only be manual, serper_search, or google_cse, and the read query filters on those three explicitly:
.in("source", ["serper_search", "google_cse", "manual"])
A model's own output is no longer something the system is capable of treating as evidence.
What I'd still fix
Being honest about the holes, since that's the part these posts usually skip.
The exchange rate is a hardcoded constant, USD_TO_NGN = 1650. I wrote an entire article about how stale currency assumptions destroyed my prices, and then I put a stale currency assumption in my own code. It needs to be a live lookup, and it will be.
The percentile cutoffs, the 60% staleness threshold, the trim multipliers: those are calibrated from my own market knowledge, not fitted to outcome data. Once enough listings sell through the platform, actual transaction prices become the training signal that replaces my guesses. That's the point where the exhaust finally becomes the dataset.
The short version
- Know your specific noise. Most of my bad data isn't false, it's correct data that's wrong for the question.
- Separate extraction from reasoning. Two calls, two failure domains, debuggable.
- Use robust statistics. Percentiles over means. One scam listing shouldn't move your answer.
- Models report, code computes. Any adjustment that moves a number should be visible and versioned.
- Enforce invariants after the model, in code. Stated rules are not guaranteed rules.
- Fail closed. No data is a better answer than a confident guess, when trust is the product.
I build AutoValue in the open at autovalue.tech. If you're doing retrieval-grounded work in a market where the sources are this messy, I'd like to hear which layer you'd add.
Top comments (7)
This is a great breakdown of the engineering reality behind trustworthy AI systems. The point about separating extraction, reasoning, and deterministic validation really resonates with my experience building production AI workflows.
In projects like health-data-rag.vercel.app and other AI applications I have worked on, I have seen that the hardest part is rarely calling an LLM — it is designing the retrieval pipeline, evaluating data quality, controlling hallucinations, and creating reliable guardrails around model outputs.
I have also worked across AI-powered platforms and products such as automator.ai, spaboostai.com, quantinium.uk, and visit-luebeck.com, where the focus has been turning AI capabilities into practical user-facing systems.
Your idea that "correct data can still be wrong for the purpose" is especially important. In real-world AI products, the winning approach is not trusting the model blindly, but building systems where models provide intelligence and engineering provides reliability.
Great work documenting these decisions openly. I would enjoy exchanging ideas around retrieval-grounded AI, agent workflows, and production AI architecture.
Thanks. That line was the last thing I added and the one I nearly cut, so
it is useful to hear it landed.
The part that took me longest to accept: I kept treating the noise as a
filtering problem, which meant I kept writing better filters. It is actually
a specification problem. Once I wrote down what a valid price point is for
my question, a real transaction between two independent parties, in the
current currency regime, at base trim, most of the filter rules fell out of
that definition instead of out of me staring at bad output.
The useful split here is extraction as a data-quality step and reasoning as a policy step. I like that the validation layer is deterministic, because pricing is exactly the kind of domain where a model can be fluent and still be wrong for the request. The part I'd watch is exception handling. Once enough cases fall outside the guardrail stack, they usually deserve their own lane rather than another prompt patch.
That's the sharpest thing anyone has said about this, and you have named the
thing I am currently doing wrong.
The case is Chinese brands. Changan, Chery and others are entering Nigeria
faster than resale data about them exists, so search returns new-car dealer
prices and nothing else. My guardrails all assume the anchor is a used-market
price. The age ceiling in particular is calibrated on that assumption and it
misbehaves badly when the anchor is already a showroom number.
I have patched that in the prompt twice. Both times it half-worked, which is
the worst outcome, because half-working delays admitting that thin-data
pricing is a different problem needing its own path. Probably synthesising an
anchor from new price minus a depreciation curve, rather than retrieving one
at all.
So yes. Own lane. That is the one I am writing up next.
The 'no data is a better answer than a confident guess when trust is the product' principle is the single most important design decision in any system where an agent acts on behalf of a user. Your Layer 6, fail closed, is exactly the philosophy behind the human discovery module in Opportunity Skill. The search results rely on cosine similarities between queries and impressions, and the documentation states plainly that finding no one who meets the requirements is a common and expected outcome. The system does not fabricate a match to appear useful. It returns an empty list and lets the user decide. Your separation of extraction from reasoning in Layer 2 also mirrors the proposal and benefits separation in our human outreach module. The contacting agent must articulate the collaboration idea and the recipient's gains as two distinct parameters, making each independently auditable. The insight that most bad data is 'correct data that is wrong for the current question' applies perfectly to professional matching too. A person's skills may be real but irrelevant to the specific collaboration context. The query has to be focused enough to distinguish.
The Opportunity Skill parallel is interesting, and I think there is an
asymmetry in it worth naming.
Failing closed is cheap for you and expensive for me. An empty result in a
discovery system is a legitimate answer a user can act on: nobody matches
yet, so widen the query or wait. When someone asks what their car is worth,
they came for exactly one number. An empty result is not a modest answer,
it is an outage.
That changes what the principle costs. I still return nothing rather than
guess, but I have to spend real engineering effort making "nothing" rare,
which is why every resolved price is cached permanently and why coverage
matters more to me than accuracy on any single query.
Your proposal-and-benefits split is the same move as my
extraction-and-reasoning split, for the same reason: two things that can be
wrong independently should be able to fail independently. Anything you fuse
into one model call, you lose the ability to attribute.
Great point. I think you’re right that “fail closed” has different costs depending on the user expectation and the domain. In discovery systems, an empty result can still create a useful next action, but in transactional/value-driven systems, the expectation is much closer to certainty.
I like the comparison between your extraction/reasoning split and the proposal/benefit split — the key idea is keeping uncertainty isolated so failures remain explainable. The more complex the system becomes, the more important that separation is for debugging and trust. Thanks for adding this perspective — it’s a great example of how the same principle evolves across different AI products.