The guard-then-verifier split maps cleanly onto a distinction that deserves its own name: guards catch what's derivable from the text alone, swapped numbers, negation, mismatched entities, no world model required. The verifier exists for exactly the cases where two strings are lexically indistinguishable and only differ by a fact about the world, ethanol versus methanol, puppy versus adult dog. That's a real fault line, syntactic difference versus semantic difference, and it explains why guards can be cheap, pure text comparison, while the verifier can't, it needs to know something, not just compare something.
Worth being precise about what the 67%/88% numbers include, since the post reads ambiguously on this point: are those guard-only rejection rates, or guard-plus-verifier? If the world-knowledge third of near misses is exactly the residual the verifier exists to catch, the honest number to publish alongside 67% is the verifier's own catch rate on that specific third, since that's the piece actually closing the gap the guards structurally can't. Otherwise 67% quietly reads as the whole system's performance when it might be describing only the free layer, with the paid layer's contribution unmeasured in the same table.
The negative-cache and embed-failure-policy design is the part I'd bet earns its keep in production faster than the guards do, mostly because embedding-call failures are boring and constant where near-misses are rare and dramatic. Curious whether you're tracking which of the two actually drives more cache misses in real traffic once this ships somewhere.
Guard-only, and you're right that the post doesn't say so. The corpus test runs
MatchGuards.standard() and nothing else, so the verifier isn't in those numbers
at all. Which makes the README line about it covering the rest an unmeasured
claim sitting next to two measured ones, and that's the wrong thing for this
project of all things to be sloppy about.
The set you'd need is small and already sitting there: the held-out near misses
the guards don't reject, about 28 of 86. That's exactly the population the
verifier claims, so running one against those 28 gives the companion number you
asked for on the same split. Opened it as github.com/NaCode-Studios/Kmemo/is... and credited you.
On the third point, I think you're right and the telemetry half exists:
MissReason already separates REJECTED_BY_GUARD from REJECTED_BY_VERIFIER. Embed
failures aren't in there though, because FALL_BACK_TO_COMPUTE bypasses the cache
rather than producing a miss, so nobody's counting the boring failure. Nothing's
in real traffic yet, but that's the counter I'd want on day one.
Guard-only numbers next to an unmeasured verifier claim is exactly the gap worth catching before it ships anywhere serious, and running the verifier against precisely the 28 held-out near misses the guards already let through is the most direct test, since that's not a fresh sample, it's the exact population the verifier exists to cover. Whatever number comes back there is the real claim the README line was gesturing at.
The embed-failure gap is the more useful find of the two, honestly, because it's invisible in a way the guard/verifier split isn't. A rejected candidate produces a MissReason and gets counted somewhere. A fallback-to-compute produces nothing, a cache miss that never registers as a miss, so the failure mode with the least drama, an embedding call timing out, is the one with no telemetry pointing at it at all. Worth a counter before day one specifically because it's boring: nobody's going to notice it's missing until a postmortem needs it and it isn't there, same shape as most of what fails quietly in this thread all week.
Thanks, this is the kind of comment that actually moves something.
And it's worse than I described: there's no event either. CacheEvent only has
Hit, Miss, Write and Eviction, so a fallback never even reaches the listener
seam. Opened it as github.com/NaCode-Studios/Kmemo/is... and
credited you there.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The guard-then-verifier split maps cleanly onto a distinction that deserves its own name: guards catch what's derivable from the text alone, swapped numbers, negation, mismatched entities, no world model required. The verifier exists for exactly the cases where two strings are lexically indistinguishable and only differ by a fact about the world, ethanol versus methanol, puppy versus adult dog. That's a real fault line, syntactic difference versus semantic difference, and it explains why guards can be cheap, pure text comparison, while the verifier can't, it needs to know something, not just compare something.
Worth being precise about what the 67%/88% numbers include, since the post reads ambiguously on this point: are those guard-only rejection rates, or guard-plus-verifier? If the world-knowledge third of near misses is exactly the residual the verifier exists to catch, the honest number to publish alongside 67% is the verifier's own catch rate on that specific third, since that's the piece actually closing the gap the guards structurally can't. Otherwise 67% quietly reads as the whole system's performance when it might be describing only the free layer, with the paid layer's contribution unmeasured in the same table.
The negative-cache and embed-failure-policy design is the part I'd bet earns its keep in production faster than the guards do, mostly because embedding-call failures are boring and constant where near-misses are rare and dramatic. Curious whether you're tracking which of the two actually drives more cache misses in real traffic once this ships somewhere.
Guard-only, and you're right that the post doesn't say so. The corpus test runs
MatchGuards.standard() and nothing else, so the verifier isn't in those numbers
at all. Which makes the README line about it covering the rest an unmeasured
claim sitting next to two measured ones, and that's the wrong thing for this
project of all things to be sloppy about.
The set you'd need is small and already sitting there: the held-out near misses
the guards don't reject, about 28 of 86. That's exactly the population the
verifier claims, so running one against those 28 gives the companion number you
asked for on the same split. Opened it as
github.com/NaCode-Studios/Kmemo/is... and credited you.
On the third point, I think you're right and the telemetry half exists:
MissReason already separates REJECTED_BY_GUARD from REJECTED_BY_VERIFIER. Embed
failures aren't in there though, because FALL_BACK_TO_COMPUTE bypasses the cache
rather than producing a miss, so nobody's counting the boring failure. Nothing's
in real traffic yet, but that's the counter I'd want on day one.
Guard-only numbers next to an unmeasured verifier claim is exactly the gap worth catching before it ships anywhere serious, and running the verifier against precisely the 28 held-out near misses the guards already let through is the most direct test, since that's not a fresh sample, it's the exact population the verifier exists to cover. Whatever number comes back there is the real claim the README line was gesturing at.
The embed-failure gap is the more useful find of the two, honestly, because it's invisible in a way the guard/verifier split isn't. A rejected candidate produces a MissReason and gets counted somewhere. A fallback-to-compute produces nothing, a cache miss that never registers as a miss, so the failure mode with the least drama, an embedding call timing out, is the one with no telemetry pointing at it at all. Worth a counter before day one specifically because it's boring: nobody's going to notice it's missing until a postmortem needs it and it isn't there, same shape as most of what fails quietly in this thread all week.
Thanks, this is the kind of comment that actually moves something.
And it's worse than I described: there's no event either. CacheEvent only has
Hit, Miss, Write and Eviction, so a fallback never even reaches the listener
seam. Opened it as github.com/NaCode-Studios/Kmemo/is... and
credited you there.