A keyword scorer is the cheapest filter you can put in front of an LLM. You write terms, you give them weights, you sum the matches, and anything over a threshold gets read by the expensive model. It works on day one. The problem starts on day thirty, when your product has moved and your term list has not.
The fix is not to rewrite the list by hand every month. It is to capture the feedback you are already collecting and turn it into weight proposals you can approve or reject. Here is the mechanism, including the part that breaks if you skip it.
Log attribution at scoring time, not later
This is the step that decides whether any of the rest is possible.
When you score an item, you know exactly which terms fired and what each one contributed. If you store only the final score, that knowledge is gone, and later you are reduced to re-running a guess about which term earned the hit.
# weights: term -> {"dimension": "threat", "weight": 3.0}
DIMENSIONS = ("opportunity", "adoption", "pivot", "threat")
def score(text, weights):
lowered = text.lower()
totals = {d: 0.0 for d in DIMENSIONS}
hits = []
for term, cfg in weights.items():
if term in lowered:
totals[cfg["dimension"]] += cfg["weight"]
hits.append((term, cfg["dimension"], cfg["weight"]))
return totals, hits
Persist hits next to the item:
CREATE TABLE signal_terms (
item_id INTEGER NOT NULL,
term TEXT NOT NULL,
dimension TEXT NOT NULL,
weight REAL NOT NULL,
PRIMARY KEY (item_id, term)
);
Now a thumbs up or down on an item is a labelled example for every term that fired on it. Without this table you have a rating on a piece of text and no way to attach it to anything you can tune.
Split the credit, do not hand it out twice
The naive aggregation gives every term that fired a full vote. That quietly rewards your broadest terms. A generic word that appears on almost everything gets credited for every good item your sharp terms actually found, and its weight creeps up until it dominates the score.
Split the credit across the terms that fired on that item, proportional to what each contributed:
from collections import defaultdict
def term_feedback(rows):
"""rows: (item_id, term, weight, verdict) with verdict in {"up", "down"}"""
by_item = defaultdict(list)
for item_id, term, weight, verdict in rows:
by_item[item_id].append((term, weight, verdict))
up = defaultdict(float)
down = defaultdict(float)
for entries in by_item.values():
total = sum(w for _, w, _ in entries) or 1.0
for term, weight, verdict in entries:
credit = weight / total
if verdict == "up":
up[term] += credit
else:
down[term] += credit
return up, down
Votes are now fractional, which is the honest representation. A term that was one of several reasons an item scored is one of several reasons it was good.
Smooth before you rank
A term with one thumbs down and nothing else has a raw precision of zero. Act on that and you will delete a good term because of a single bad morning.
Add a prior, then require a minimum amount of evidence before a term is eligible for a proposal at all:
PRIOR_UP, PRIOR_DOWN = 1.0, 1.0
def precision(u, d):
return (u + PRIOR_UP) / (u + d + PRIOR_UP + PRIOR_DOWN)
The prior pulls thin evidence toward the middle, so terms with little feedback sit in the do-nothing band on their own. The minimum-evidence gate is still worth having, because it makes the reason a term was skipped explicit in the code instead of implicit in the arithmetic.
Propose a diff, do not apply one
Here is the design decision worth arguing about: the tuner should emit a proposal, not a write.
MIN_EVIDENCE = 5.0
STEP = 0.25
FLOOR, CEIL = 0.5, 5.0
UP_BAND, DOWN_BAND = 0.70, 0.40
def propose(up, down, weights):
out = []
for term in set(up) | set(down):
u, d = up[term], down[term]
if u + d < MIN_EVIDENCE:
continue
p = precision(u, d)
current = weights[term]["weight"]
if p >= UP_BAND:
new = min(current + STEP, CEIL)
elif p <= DOWN_BAND:
new = max(current - STEP, FLOOR)
else:
continue
if new != current:
out.append({"term": term, "from": current, "to": new,
"precision": round(p, 2), "up": round(u, 1),
"down": round(d, 1)})
return sorted(out, key=lambda r: r["precision"])
Three reasons to keep a human in the loop.
Keyword weights are coupled. Move several at once and you have not adjusted terms, you have moved the whole score distribution relative to your threshold. A reviewer looking at the batch notices that. A loop applying its own output does not.
Feedback is an opinion about relevance, and relevance is a business judgement. The person who knows whether "seed round" should matter more this quarter is the person reading the briefing, not the aggregator.
An auto-applying loop has no audit trail. When scoring goes strange, you want a list of accepted proposals to walk backwards through, not a weights file that has been drifting on its own.
The fixed step and the floor and ceiling do the rest of the safety work. No single round can move a term far, and nothing can decay to zero or run away.
The loop can measure precision and not recall
Your feedback is not a random sample. You only see thumbs on items that survived the filter and got delivered. Every item the keyword pass dropped is invisible, so the loop can measure precision and cannot measure recall at all.
Run it long enough and the tuner sharpens the terms you already have while the things you never wrote down stay unreachable. Precision goes up. Coverage quietly narrows.
The cheap mitigation is a holdout. Route a small random sample of below-threshold items into a review queue on a schedule, label them like anything else, and look at what comes back marked good. Those items are the only direct evidence you get about what your term list is missing. Weight tuning cannot fix a missing term; only a human reading a near-miss can add it.
What this does not solve
It does not invent vocabulary. A new competitor's name, a new framework, a term of art that appeared last week: none of that arrives from tuning weights on terms you already wrote.
It drifts toward whatever you happen to click. If you only rate the items that annoy you, you are training an annoyance detector.
It needs a cold start. Until terms clear the evidence gate, the tuner correctly proposes nothing, and a system that appears to do nothing for a while is a system people stop feeding.
And if you will not give feedback, skip the whole thing. A tuner with no labels is a scheduled job that reads an empty table. Static weights you revise by hand are a perfectly reasonable alternative, and honest about what they are.
This auto-tuning loop is one piece of the Market Radar Kit, a self-hosted market-intelligence agent I built and run in Docker against my own Claude subscription: https://fulcrumenterprises.tech/go/market-radar-kit/?c=devto
Top comments (0)