Scam Listings In, Trustworthy Price Out: The Guardrail Stack Behind an AI Pricing Engine

Someone asked how I handle noise in search results when there's no clean dataset to fall back on. Here's the full answer: six layers, none of which trust the model.

After I wrote about building a car pricing engine for a market with no data, @topstar_ai asked something in the comments that deserved more than a reply:

Comment on The Next Billion Users Don't Come With Datasets, and That's the Opportunity

Luis Cruz Jul 15

I was struck by the idea that LLMs can synthesize a price at request time from live search results, which flips the traditional supervised learning approach on its head. The fact that the dataset is no longer a prerequisite, but rather the exhaust, is a game-changer for building AI products in data-poor markets. I've worked on similar projects where we had to rely on proxy data sources and clever feature engineering to overcome data scarcity, but the LLM approach seems to offer a more scalable and flexible solution. How do you handle the issue of data quality and noise in the search results, and what kinds of domain guardrails have you implemented to ensure the model's output is reliable and trustworthy?

Two questions in there, and they're the right two. If your ground truth is live search results instead of a curated dataset, then your data quality problem doesn't go away. It moves. It becomes a runtime problem you have to solve on every single request.

Here's what I actually do at AutoValue, layer by layer.

First, what the noise actually looks like

Generic advice about "filtering low-quality sources" is useless until you know the specific failure modes. Mine, when I search for Nigerian used car prices:

• Asking prices, not transaction prices. Every listed price on Jiji is an opening move. Nigerian buyers negotiate 10 to 20% below the ask as a matter of routine. A model that reports the listed number is systematically overpricing the market.
• Pre-devaluation listings that never got taken down. The naira went from around ₦460 to the dollar to around ₦1,650. Listings from 2022 are still indexed and still show 2022 prices. They aren't wrong, exactly. They're archaeology.
• Trim contamination. Search "Toyota Land Cruiser Prado price Nigeria" and most of what comes back is the VX full option, because that's what dealers advertise. If you take that as the base price for a standard trim, every Prado prices high forever.
• Currency mixing. Export and auction sites quote USD. Nigerian listing sites quote naira. Both land in the same result set.
• Scam bait. A ₦2.5 million Range Rover is not a data point. It's a lure. But it's a perfectly well-formed number that any naive parser will happily average in.

Notice that only the last one is "bad data" in the usual sense. The rest are all correct data that's wrong for my purpose. That distinction drives the whole design.

Layer 1: Constrain the search, not just the parsing

The cheapest filter is the one that stops noise from being retrieved at all.

const query = `${year} ${make} ${model} price Nigeria site:jiji.ng OR site:cars45.com`;

const searchRes = await fetch("https://google.serper.dev/search", {
  method: "POST",
  headers: { "X-API-KEY": serperKey, "Content-Type": "application/json" },
  body: JSON.stringify({ q: query, gl: "ng", hl: "en", num: 10 }),
});

site: scoping to the two marketplaces that actually matter, and gl: "ng" to geo-locate the search in Nigeria. That last one matters more than it looks. Without it you get US results for the same car, in dollars, for a market with entirely different import economics.

I also prioritise Google's answer box and knowledge panel above organic results when they're present, because Google has already done a round of synthesis across sources:

const priorityLines: string[] = [];
if (answerBox?.snippet || answerBox?.answer) {
  priorityLines.push(`GOOGLE SUMMARY: ${answerBox.title ?? ""}\n${answerBox.snippet ?? ""}`);
}
const organicLines = organic.slice(0, 7).map(r => `${r.title ?? ""}\n${r.snippet ?? ""}`);
const allLines = [...priorityLines, ...organicLines];

Layer 2: Separate extraction from reasoning

This is the structural decision everything else depends on, and it's the one I'd most want someone to take away.

There are two completely different jobs here. One is read these messy snippets and tell me what this car currently sells for. The other is given a base price, adjust it for this specific car's mileage, condition, trim and location. The tempting move is one prompt that does both. Don't.

They're two separate Claude calls in my pipeline, and the reason is auditability. When a price comes out wrong in a single-call design, you cannot tell whether the model misread a snippet or misapplied an adjustment rule. Split them and every bad output localises to one stage. My debugging time dropped more from this change than from any prompt improvement.

Extraction runs on Haiku with a procedure, not a request:

Extract the REALISTIC TRANSACTION PRICE RANGE, what this car actually sells
for, not what sellers hope to get.

- Naira devalued sharply: ₦460/$ in 2022 to ₦1,650/$ in 2026. Car prices are
  now 3 to 4x higher than pre-2023.
- Jiji and Cars45 show ASKING prices. Nigerian buyers negotiate 10 to 20%
  below the listed ask.
- STEP 1: any price more than 60% below the highest-priced cluster member is a
  pre-devaluation or junk listing. SKIP IT.
- STEP 2: SKIP listings marked "5+ years on Jiji". ENTERPRISE is a dealer
  badge, do NOT skip those, dealers are valid sellers.
- STEP 3: prices in $ multiply by ₦1,650.
- STEP 4: from remaining valid prices take the MIDDLE cluster. Ignore the
  bottom 15% and anything above 2x the cluster median.
- price_low = 25th percentile. price_high = 75th percentile.
- If only 1 to 2 valid prices remain: low = price × 0.88, high = price × 1.08.

Every one of those rules is a scar. STEP 2's carve-out for ENTERPRISE exists because my first version of "skip listings with badges" quietly deleted all the dealer inventory, which is the most reliable pricing signal on the site.

Layer 3: Percentiles, never averages

Worth calling out on its own because it's a one-line change with outsized effect.

The output is the 25th and 75th percentile of the surviving cluster. Not the mean, not min and max. A single scam listing at ₦2.5M destroys a mean and it destroys a min. It barely moves a 25th percentile. Robustness to outliers is a property you get for free from the right statistic, and no amount of prompt engineering substitutes for picking it.

The narrow-data case gets handled explicitly too. With only one or two valid prices left, percentiles are meaningless, so the rule degrades to a fixed spread around the single observation. Say what should happen when the data is thin, or the model will invent something.

Layer 4: Make the model report, let code decide

Trim contamination is the subtlest failure in the list, and the fix generalises well beyond pricing.

I could ask the extraction model to "adjust the price down if the listings look like a high trim." That's asking for a silent, unauditable, unrepeatable adjustment buried inside a number. Instead the model reports what it saw as a separate field, and the arithmetic happens in code where I can read it:

const TRIM_FACTOR: Record<string, number> = {
  full_option: 1.12,
  limited:     1.10,
  sport:       1.08,
  standard:    1.00,
  unknown:     1.00,
};

const trimFactor = TRIM_FACTOR[searched.detected_trim] ?? 1.00;
const normalizedLow  = Math.round(searched.price_low  / trimFactor / 50_000) * 50_000;
const normalizedHigh = Math.round(searched.price_high / trimFactor / 50_000) * 50_000;

Now every stored anchor is a base-trim price, the trim premium gets re-added later as a visible line item the seller can see, and I can change the multipliers without touching a prompt.

Ask the model for observations. Keep the calculations in code. If a model's judgment is going to move a number, make it move the number through a mechanism you can print.

Layer 5: A deterministic invariant after the model responds

The adjustment model is given explicit percentage rules: excellent interior is +2%, mileage below expected earns +0.3% per 10,000km for a 4 to 7 year old car, and so on.

It does not always follow its own rules. Verified live: on one car it applied a +7.8% mileage bonus where its own stated rule produced 0.3%. That's not hallucination in the dramatic sense. It's a model doing approximate arithmetic and landing somewhere plausible-looking.

So there's a clamp that runs in code after the response comes back:

const CONDITION_UPLIFT_KEYS = ["condition", "interior", "engine", "mileage"] as const;

function applyAgeConditionCeiling(estimate, base, carAge) {
  // Younger car, smaller allowed uplift: "excellent" is the expected norm for a
  // 2-year-old car and already priced into the anchor, but a genuine rarity on
  // a 10-year-old one.
  const capPct = carAge <= 3 ? 0.03 : carAge <= 7 ? 0.06 : 0.10;
  const baseMedian = (base.price_low + base.price_high) / 2;

  const upliftNGN = CONDITION_UPLIFT_KEYS.reduce(
    (sum, key) => sum + Math.max(0, estimate.adjustments?.[key] ?? 0), 0
  );
  const upliftPct = upliftNGN / baseMedian;
  if (upliftPct <= capPct) return { estimate, cappedPct: 0 };

  // Subtract only the excess above the ceiling, proportionally.
  const excessPct = upliftPct - capPct;
  ...
}

The domain reasoning matters as much as the code. A search anchor for a two-year-old car already assumes near-excellent condition, because that's what two-year-old cars on the market are. Stacking excellent body, excellent interior, excellent engine and a low-mileage bonus on top of that anchor prices a clean used car like a new one. The failure was worst on thin-data models, where the only search results are new-car dealer prices and the anchor is already the ceiling.

Trim and location uplifts are deliberately exempt from the cap, because they're structural rather than condition-based. Trim was divided out in layer 4 and is legitimately being re-added.

Never trust a model to guarantee an arithmetic invariant. If a relationship must hold, enforce it in code after the fact.

Layer 6: Fail closed

This is the one I'd argue about with most people building LLM products.

When the search returns nothing usable, the endpoint does not fall back to the model's own knowledge. It returns a 503:

if (!searched) {
  return NextResponse.json({
    success: false,
    error: "pricing_unavailable",
    message: "We couldn't find live market data for this car right now. Please try again in a few minutes.",
  }, { status: 503 });
}

Most LLM products fail open. Retrieval comes back empty, and the model answers anyway from training data, and the user cannot tell the difference between a grounded answer and a guess. For a chatbot that's mildly bad. For a product whose entire value proposition is "this number is trustworthy," it's fatal, because the failure is invisible exactly when it's most damaging.

Related, and covered in more depth in the previous article: model-generated prices were once cached as anchors, then re-injected on the next request as if they were market data. The fix has since hardened from a runtime bypass into a type. The stored anchor's source can only be manual, serper_search, or google_cse, and the read query filters on those three explicitly:

.in("source", ["serper_search", "google_cse", "manual"])

A model's own output is no longer something the system is capable of treating as evidence.

What I'd still fix

Being honest about the holes, since that's the part these posts usually skip.

The exchange rate is a hardcoded constant, USD_TO_NGN = 1650. I wrote an entire article about how stale currency assumptions destroyed my prices, and then I put a stale currency assumption in my own code. It needs to be a live lookup, and it will be.

The percentile cutoffs, the 60% staleness threshold, the trim multipliers: those are calibrated from my own market knowledge, not fitted to outcome data. Once enough listings sell through the platform, actual transaction prices become the training signal that replaces my guesses. That's the point where the exhaust finally becomes the dataset.

The short version

• Know your specific noise. Most of my bad data isn't false, it's correct data that's wrong for the question.
• Separate extraction from reasoning. Two calls, two failure domains, debuggable.
• Use robust statistics. Percentiles over means. One scam listing shouldn't move your answer.
• Models report, code computes. Any adjustment that moves a number should be visible and versioned.
• Enforce invariants after the model, in code. Stated rules are not guaranteed rules.
• Fail closed. No data is a better answer than a confident guess, when trust is the product.

I build AutoValue in the open at autovalue.tech. If you're doing retrieval-grounded work in a market where the sources are this messy, I'd like to hear which layer you'd add.