A tool should do one thing, do it well, and — this is the part everyone forgets — know exactly where its knowledge ends.
I built kilo for a morn...
For further actions, you may consider blocking this person and/or reporting abuse
The design you should defend harder is the one you glossed: NOT_OBSERVED being an offline lookup. That's the whole game, and it's also the failure mode. Your snapshot printed "Claims 3160" — a known C2 shows up because someone already observed and published it. But a freshly-provisioned redirect host for a lead-gen campaign is almost by definition not in any feed yet. So kilo's honest 0.0 on 203.0.113.90 isn't really the tool refusing to guess under pressure — it's the tool correctly reporting that infrastructure this ephemeral will basically always be absent from a signed local index. The humility is real, but the coverage story does a lot of the work you're crediting to the philosophy.
Which actually makes your conclusion stronger, not weaker: the value wasn't kilo saying "I don't know," it was kilo saying it fast and without lying, so you didn't burn an hour treating absence-of-evidence as evidence. That's the reframe I'd lead with — an offline reputation engine's superpower isn't catching the novel scam, it's cheaply ruling out the "is this a known-bad box" branch so you can spend your attention on the sender, where this threat actually lived.
One concrete thing worth stating plainly for readers who'll copy your workflow:
dig +shorton a redirect host gives you an IP, often a CDN or aggregator edge, not the origin — so a clean check there is a check on shared infrastructure. Fine here, since the whole trail was legit, but it's exactly the case where "clean IP" and "clean intent" quietly diverge.This is a really nice concept. It probably would have helped me reach an honest verdict a while back when I received an email that was claiming that I won a scholarship that i didn't apply for. I mean, the issue is obvious — it's a scam but as you said a gut feeling is just an opinion and an honest tool is a witness. The whole concept is simple, but also effective. I will definitely check and test your tool.
That "structurally different state" framing is exactly right — and it's a much cleaner design than a continuous confidence score that degrades. The key difference is: you're not trying to predict how "confident" the model is, you're asserting whether the index has coverage for this input class. No prediction, no calibration drift. The only thing that can drift is the index itself staying current, which is a much more tractable problem.
This reminds me of a post I made in the pre-AI days but totally prescient
Technology Must Know When to Say "I Don't Know"
Making abstention a first-class output instead of forcing a confident label is underused, and it maps directly to how I think about model outputs: an honest "I don't know" is far cheaper than a wrong verdict stated with certainty. The hard part is calibrating the threshold, since a tool that abstains too often becomes noise people learn to ignore. Did you tune where kilo draws the line between a real answer and a refusal, or is it a fixed rule per signal?
A tool that can say "I do not know" is often more trustworthy than one with perfect confidence language. The refusal creates a useful contract: the tool is allowed to preserve uncertainty instead of laundering it into a verdict. That matters especially in security, where a confident false positive and a confident false negative both create bad behavior.
The "offline + signed release" choice is the part most people would skip. I built something similar a few years ago â a small Rust CLI for checking package hashes before install â and almost shipped it with a fetch-on-demand model. The decision to lock the index at build time, even though it means occasional staleness, is the only reason I trust it. A tool that can phone home to "update" is a tool that can phone home for other reasons.
The four-stage pipeline mirrors what I'd expect from a serious codebase: ingest, normalize, compile, query. Most hobby CLIs collapse all of that into one "lookup" function and then degrade weirdly when the input doesn't fit. I like that you treat the immutable local index as a first-class artifact â it means the tool can be audited, snapshotted, and tested without a network.
The "vibe vs verdict" framing is more useful than I expected. I keep catching myself saying "this looks suspicious" without naming what I'd actually check. Did you find a way to surface the tool's confidence level to the user, or does it just say "I don't know" as a binary?
How's the "I don't know" threshold actually calibrated, a model confidence score or a rule-based fallback? A badly-calibrated refusal that fires 90% of the time is exactly as useless as one that never fires.
Neither — and that's the design point. There's no threshold because abstention isn't the low-confidence tail of a score. It's a structurally different state: the verdict is a set-membership check against a signed, offline index. If the index has zero observations for the IP, you get NOT_OBSERVED / confidence 0.0. There's nothing to calibrate because there's nothing to estimate — the tool is reporting coverage, not confidence.
Your "fires 90% of the time" concern is real, but it lives in the wrong layer if you aim it at the CLI. Refusal rate is a function of index coverage, and index coverage is a data-pipeline property you can measure and improve independently — add feeds, widen observation windows, re-sign, ship. The CLI's contract stays fixed: observed → typed classification through the severity lattice; not out lying. A calibratedmodel can drift; a coverage check can only be stale, and staleness is auditable.
The design property that carries this is making 'I don't know' a typed third verdict, not the gap between yes and no. Most tools collapse abstention into a soft pass, and a soft pass is how the one link that isn't a scam trains you to click the one that is. The offline signed-release part is what earns the refusal: an online lookup degrades to 'unknown means probably fine' the moment the network flakes, but an immutable local index either has the observation or it provably does not. The seam I would fixture hardest: does 'unknown' ever get read as 'clean' downstream, in an exit code or in how a caller branches on the result? That is the one place a refusal quietly turns back into a vibe.
Interesting take on uncertainty in CLI tools. In security and confidential computing, ambiguity can actually be a feature—especially when dealing with untrusted inputs. I've seen similar patterns when working with GPU-based inference: letting the system "opt out" of uncertain predictions can prevent cascading errors downstream.
"A vibe is not a verdict" is the kind of framing that should be tattooed on every developer's wrist. I've been deep in AI agent quality gates lately and the same pattern keeps surfacing — most failures aren't from the model being wrong, they're from the model being confidently wrong when it should have said "I don't know."
Your four-stage pipeline (DNS → whois → reputation → heuristic scoring) is the deterministic analogue of what we've been calling "task-type routing" in the agent space: figure out what kind of check this is, apply the cheapest reliable gate first, only escalate when the deterministic layer says it can't decide. The offline-first Rust binary approach is elegant — no API calls, no latency, no external dependency that might be down when you need it most.
Curious: how does the heuristic scoring stage handle edge cases where individual signals point in different directions? That's the part where my agent experiments tend to hit the most false negatives.
Great question, because it's exactly where I expected this design to get pressure-tested.
The honest answer: there is no threshold to tune, and that's deliberate. Abstention in kilo isn't the low-confidence tail of a score — it's a structurally different state. kilo check reads a local, versioned evidence index (compiled from bulk threat-intel publications by kilo-data). If that index contains zero observations for the IP, the verdict is unknown / NOT_OBSERVED / confidence 0.0. If it contains any, the tool answers. There's no confidence cutoff where an answer degrades into a refusal — so there's no line to calibrate, and nothing to drift.
Where numbers do exist, they're derived by fixed rule, not fitted:
Letting a tool say "I don't know" is underrated as a design choice, most AI tooling is optimized to always produce an answer because a confident wrong answer looks better in a demo. Building in the honest failure mode is closer to engineering discipline than most "AI code review" products on the market right now.