DEV Community

GK
GK

Posted on AI-assisted

I asked an LLM which listings to skip. It kept getting the numbers wrong.

#ai #webdev #productivity #javascript

I waste a stupid amount of time reading pages just to find the one line that rules them out.

A job post where the budget is hidden at the very bottom. "Brand new" headphones where the seller note, three scrolls down, says refurbished. A

So I thought, easy, I'll paste the page into an LLM with my own rules and ask "should I bother with this?"

Attempt 1: just ask the model

Something like:

My rules: budget at least $500, max 3 revisions.
Here's the job post: ...
Should I apply? Answer yes or no and why.
Enter fullscreen mode Exit fullscreen mode

It worked most of the time. Then it said yes to a $300 job. It took "List price: $79" as the price when the real price was $129. It told me a product had a 30-day return window on a page that never mentioned returns.

The problem wasn't that it was wrong sometimes. It was that it sounded exactly as sure when it was wrong.

Attempt 2: the model doesn't get to touch the numbers

What fixed it was splitting the work:

  • Plain code reads the numbers (price, budget, days, counts) and compares them to my limits. If one fails, the answer is no and the model never gets asked.
  • The model reads the same page separately, but a value it finds only counts if that text is actually on the page. If code and model disagree, it shows me both and I pick.
  • Missing fact = "not yet", and it asks me for the value instead of guessing.
  • The model only judges the fuzzy stuff, like "is this scope realistic for the money" or "is there a catch", and only after the numbers pass. It has to quote the line it's basing that on.

Roughly:

const budget = readMoney(page, ['budget']); // plain parsing, not the LLM
if (budget !== null && budget < rules.minBudget) {
  return no(`Budget is $${budget}, under your $${rules.minBudget} minimum.`);
}
// only now ask the model about the soft rules
Enter fullscreen mode Exit fullscreen mode

Stuff that bit me along the way

  • "List Price: $79" and "Was $99" sit right next to the real price. The parser has to skip any line that's clearly not the price.
  • "21 315 participants" (space as a thousands separator) was being read as 21.
  • "$12 per user, billed monthly" needs "per user" and "billed" stripped before you can tell it's monthly.
  • Models go down. When the model is busy, the numbers still get checked and it says plainly that the AI part is missing, instead of just failing.

What it turned into

I wrapped it in a Chrome extension called Kriterio. You set your limits once, open a page, and it shows green, amber or red with the reason. I mostly use it for [job posts / shopping / whatever you actually use it for].

It's free to try if you want to poke holes in it and give me feedback: https://chromewebstore.google.com/detail/phmlfimaombpolnlcmgohocekndiinpf

https://getkriterio.com/

Curious how others handle this. Do you let the model decide anything numeric, or do you always check it in code?

Top comments (0)