The 100 USD vs 250 USD example is the whole argument in one line — the near-miss sits closer on the similarity axis than an honest paraphrase, so no threshold can separate them, and raising it just costs you real hits first. That's the part people miss when they bolt a semantic cache on and celebrate the hit rate: the failure isn't a lower hit rate, it's a confident wrong answer with nothing in the logs.
Grounding the defaults in cost asymmetry — one wrong rejection is one API call, one wrong acceptance is a wrong answer, so abstain — is exactly the right prior. It's the same reasoning that makes argument-validation gates worth their tedium: the expensive side is silent.
Two things I'm curious about. First, do the guards look at the candidate answer text as well as the two prompts? A swapped number in the stored response is the real tell, and it survives even when the prompts paraphrase cleanly. Second, how does this behave with stateful/multi-turn prompts where the "same" question depends on conversation history the embedding never sees — does context get folded into the key, or is that explicitly out of scope?
No on the first one, and you've put your finger on something. The guards only
ever see the two prompts. The response is sitting right there on the entry and
nothing in the match path touches it, not even the optional Verifier. So the
case you're describing is exactly the one they can't catch: if both prompts read
as clean paraphrases there's nothing for NumericGuard to compare, and the number
that actually matters is off in the stored answer where nobody's looking. I
think that interface needs to get wider.
The multi-turn one I've left alone on purpose, and I'm not convinced folding
context into the key is the right fix anyway. The only lever is scope, which is
an exact match rather than part of the similarity search, so you can put a
conversation id in there but then you've got a cache per conversation and you've
given up most of what you wanted. Embedding the history has the opposite
problem: every follow-up drifts further from its neighbours and the hit rate
quietly goes. What I'd do instead is rewrite the prompt into something
self-contained before it ever reaches the cache, so "and in euros?" becomes
"convert 100 USD to EUR", and cache that. You probably need that rewrite for
retrieval anyway.
Writing it up surfaced the thing that actually blocks it, which hadn't occurred
to me until I started: the labelled corpus is prompt pairs only, so the 67/88
numbers are measured on a shape with no answer text in it at all. A
response-aware guard can't be reported honestly until the corpus schema grows an
answer field first. Slower than just widening the interface, but I'd rather not
ship a guard I can't put a number on.
The guard-then-verifier split maps cleanly onto a distinction that deserves its own name: guards catch what's derivable from the text alone, swapped numbers, negation, mismatched entities, no world model required. The verifier exists for exactly the cases where two strings are lexically indistinguishable and only differ by a fact about the world, ethanol versus methanol, puppy versus adult dog. That's a real fault line, syntactic difference versus semantic difference, and it explains why guards can be cheap, pure text comparison, while the verifier can't, it needs to know something, not just compare something.
Worth being precise about what the 67%/88% numbers include, since the post reads ambiguously on this point: are those guard-only rejection rates, or guard-plus-verifier? If the world-knowledge third of near misses is exactly the residual the verifier exists to catch, the honest number to publish alongside 67% is the verifier's own catch rate on that specific third, since that's the piece actually closing the gap the guards structurally can't. Otherwise 67% quietly reads as the whole system's performance when it might be describing only the free layer, with the paid layer's contribution unmeasured in the same table.
The negative-cache and embed-failure-policy design is the part I'd bet earns its keep in production faster than the guards do, mostly because embedding-call failures are boring and constant where near-misses are rare and dramatic. Curious whether you're tracking which of the two actually drives more cache misses in real traffic once this ships somewhere.
Guard-only, and you're right that the post doesn't say so. The corpus test runs
MatchGuards.standard() and nothing else, so the verifier isn't in those numbers
at all. Which makes the README line about it covering the rest an unmeasured
claim sitting next to two measured ones, and that's the wrong thing for this
project of all things to be sloppy about.
The set you'd need is small and already sitting there: the held-out near misses
the guards don't reject, about 28 of 86. That's exactly the population the
verifier claims, so running one against those 28 gives the companion number you
asked for on the same split. Opened it as github.com/NaCode-Studios/Kmemo/is... and credited you.
On the third point, I think you're right and the telemetry half exists:
MissReason already separates REJECTED_BY_GUARD from REJECTED_BY_VERIFIER. Embed
failures aren't in there though, because FALL_BACK_TO_COMPUTE bypasses the cache
rather than producing a miss, so nobody's counting the boring failure. Nothing's
in real traffic yet, but that's the counter I'd want on day one.
Guard-only numbers next to an unmeasured verifier claim is exactly the gap worth catching before it ships anywhere serious, and running the verifier against precisely the 28 held-out near misses the guards already let through is the most direct test, since that's not a fresh sample, it's the exact population the verifier exists to cover. Whatever number comes back there is the real claim the README line was gesturing at.
The embed-failure gap is the more useful find of the two, honestly, because it's invisible in a way the guard/verifier split isn't. A rejected candidate produces a MissReason and gets counted somewhere. A fallback-to-compute produces nothing, a cache miss that never registers as a miss, so the failure mode with the least drama, an embedding call timing out, is the one with no telemetry pointing at it at all. Worth a counter before day one specifically because it's boring: nobody's going to notice it's missing until a postmortem needs it and it isn't there, same shape as most of what fails quietly in this thread all week.
Thanks, this is the kind of comment that actually moves something.
And it's worse than I described: there's no event either. CacheEvent only has
Hit, Miss, Write and Eviction, so a fallback never even reaches the listener
seam. Opened it as github.com/NaCode-Studios/Kmemo/is... and
credited you there.
This is the clearest articulation of the polarity problem I've seen. The "Convert 100 USD to EUR" vs "Convert 250 USD to EUR" pair at 0.99 cosine is the same shape as the one that drove my design: "activate user 4471" vs "deactivate user 4471" scores 0.853 cosine — embeddings capture topic, not polarity, so no threshold separates near-opposites from genuine paraphrases. I agree with the conclusion that the guard has to be structural, not a threshold tweak.
One thing I'd add from my side: I ran into this at the trajectory level, not the response level. When an agent has already solved a goal, replaying the cached plan is far cheaper than re-running the loop (a 14-step trajectory replays in 47ms vs 3.9s live in my measurements) — but a fuzzy match serving the wrong write is worse than no cache at all. So I made every stored tool step declare read vs write, and write-containing trajectories replay only on an exact goal+context repeat, never a fuzzy one. An 18-pair adversarial eval runs as a CI gate: a threshold-only baseline serves the wrong command 1/18 times, the gated version 0/18.
The cost of that stance is exactly what you describe — the cache abstains on paraphrases rather than risking the wrong action, so hit rate depends entirely on how repetitive your goals are. Curious whether your guard chain has a similar recall tradeoff, or whether the verifier in 2.0 bought back some of it.
Thanks Kyle, and yes, the tradeoff is there. Guards alone reject 67% of near misses and keep 88% of paraphrases. The verifier doesn’t recover recall: with a cross-encoder as the reference verifier, it stops 40 of the 50 near misses the guards let through on the held-out split (false-hit 0.291 → 0.058), but rephrasings kept drop from 0.881 to 0.452. The validation split shows the same pattern, 0.882 → 0.686. That’s why it’s opt-in. Your read/write split is a better answer for agents, since the exact-match rule only costs hit rate where a wrong hit is expensive. Kmemo caches single responses, so I can’t scope strictness by side effects like that.
Thanks for sharing the numbers — that tradeoff is exactly the crux. A verifier that cuts false hits from 0.291 to 0.058 while dropping kept rephrasings from 0.881 to 0.452 is buying correctness with recall across the board, which is the right call when every hit is equally risky. The read/write split is just a way to stop paying that recall price where a wrong hit is cheap: exact-match only where a wrong hit is expensive, and looser matching where it just costs a worse answer. And agreed on the scoping point — a single-response cache has no side-effect axis to scope on, so the gate has to live somewhere else.
Nice project! I really like the idea of using semantic guards instead of relying only on similarity scores. The explanation about the 100 USD vs. 250 USD example makes the problem very clear. I'll definitely test this approach in one of my AI projects. We also share free developer resources and tutorials on Codecan.net, so this was an interesting read. Thanks for making it open source!
Thanks, glad that example landed. It's the one that made me build the thing in
the first place. If you do try it on one of your projects I'd love to hear how
it goes, especially what your near misses look like: the guards are tuned on a
corpus that's mostly numbers, units and entities, and I have no idea yet how
well that carries into other domains. And if you like where it's heading, a star
genuinely helps people stumble on it.
We're a place where coders share, stay up-to-date and grow their careers.
The
100 USDvs250 USDexample is the whole argument in one line — the near-miss sits closer on the similarity axis than an honest paraphrase, so no threshold can separate them, and raising it just costs you real hits first. That's the part people miss when they bolt a semantic cache on and celebrate the hit rate: the failure isn't a lower hit rate, it's a confident wrong answer with nothing in the logs.Grounding the defaults in cost asymmetry — one wrong rejection is one API call, one wrong acceptance is a wrong answer, so abstain — is exactly the right prior. It's the same reasoning that makes argument-validation gates worth their tedium: the expensive side is silent.
Two things I'm curious about. First, do the guards look at the candidate answer text as well as the two prompts? A swapped number in the stored response is the real tell, and it survives even when the prompts paraphrase cleanly. Second, how does this behave with stateful/multi-turn prompts where the "same" question depends on conversation history the embedding never sees — does context get folded into the key, or is that explicitly out of scope?
No on the first one, and you've put your finger on something. The guards only
ever see the two prompts. The response is sitting right there on the entry and
nothing in the match path touches it, not even the optional Verifier. So the
case you're describing is exactly the one they can't catch: if both prompts read
as clean paraphrases there's nothing for NumericGuard to compare, and the number
that actually matters is off in the stored answer where nobody's looking. I
think that interface needs to get wider.
The multi-turn one I've left alone on purpose, and I'm not convinced folding
context into the key is the right fix anyway. The only lever is scope, which is
an exact match rather than part of the similarity search, so you can put a
conversation id in there but then you've got a cache per conversation and you've
given up most of what you wanted. Embedding the history has the opposite
problem: every follow-up drifts further from its neighbours and the hit rate
quietly goes. What I'd do instead is rewrite the prompt into something
self-contained before it ever reaches the cache, so "and in euros?" becomes
"convert 100 USD to EUR", and cache that. You probably need that rewrite for
retrieval anyway.
Opened an issue for the first one, with you credited in it:
github.com/NaCode-Studios/Kmemo/is...
Writing it up surfaced the thing that actually blocks it, which hadn't occurred
to me until I started: the labelled corpus is prompt pairs only, so the 67/88
numbers are measured on a shape with no answer text in it at all. A
response-aware guard can't be reported honestly until the corpus schema grows an
answer field first. Slower than just widening the interface, but I'd rather not
ship a guard I can't put a number on.
The guard-then-verifier split maps cleanly onto a distinction that deserves its own name: guards catch what's derivable from the text alone, swapped numbers, negation, mismatched entities, no world model required. The verifier exists for exactly the cases where two strings are lexically indistinguishable and only differ by a fact about the world, ethanol versus methanol, puppy versus adult dog. That's a real fault line, syntactic difference versus semantic difference, and it explains why guards can be cheap, pure text comparison, while the verifier can't, it needs to know something, not just compare something.
Worth being precise about what the 67%/88% numbers include, since the post reads ambiguously on this point: are those guard-only rejection rates, or guard-plus-verifier? If the world-knowledge third of near misses is exactly the residual the verifier exists to catch, the honest number to publish alongside 67% is the verifier's own catch rate on that specific third, since that's the piece actually closing the gap the guards structurally can't. Otherwise 67% quietly reads as the whole system's performance when it might be describing only the free layer, with the paid layer's contribution unmeasured in the same table.
The negative-cache and embed-failure-policy design is the part I'd bet earns its keep in production faster than the guards do, mostly because embedding-call failures are boring and constant where near-misses are rare and dramatic. Curious whether you're tracking which of the two actually drives more cache misses in real traffic once this ships somewhere.
Guard-only, and you're right that the post doesn't say so. The corpus test runs
MatchGuards.standard() and nothing else, so the verifier isn't in those numbers
at all. Which makes the README line about it covering the rest an unmeasured
claim sitting next to two measured ones, and that's the wrong thing for this
project of all things to be sloppy about.
The set you'd need is small and already sitting there: the held-out near misses
the guards don't reject, about 28 of 86. That's exactly the population the
verifier claims, so running one against those 28 gives the companion number you
asked for on the same split. Opened it as
github.com/NaCode-Studios/Kmemo/is... and credited you.
On the third point, I think you're right and the telemetry half exists:
MissReason already separates REJECTED_BY_GUARD from REJECTED_BY_VERIFIER. Embed
failures aren't in there though, because FALL_BACK_TO_COMPUTE bypasses the cache
rather than producing a miss, so nobody's counting the boring failure. Nothing's
in real traffic yet, but that's the counter I'd want on day one.
Guard-only numbers next to an unmeasured verifier claim is exactly the gap worth catching before it ships anywhere serious, and running the verifier against precisely the 28 held-out near misses the guards already let through is the most direct test, since that's not a fresh sample, it's the exact population the verifier exists to cover. Whatever number comes back there is the real claim the README line was gesturing at.
The embed-failure gap is the more useful find of the two, honestly, because it's invisible in a way the guard/verifier split isn't. A rejected candidate produces a MissReason and gets counted somewhere. A fallback-to-compute produces nothing, a cache miss that never registers as a miss, so the failure mode with the least drama, an embedding call timing out, is the one with no telemetry pointing at it at all. Worth a counter before day one specifically because it's boring: nobody's going to notice it's missing until a postmortem needs it and it isn't there, same shape as most of what fails quietly in this thread all week.
Thanks, this is the kind of comment that actually moves something.
And it's worse than I described: there's no event either. CacheEvent only has
Hit, Miss, Write and Eviction, so a fallback never even reaches the listener
seam. Opened it as github.com/NaCode-Studios/Kmemo/is... and
credited you there.
This is the clearest articulation of the polarity problem I've seen. The "Convert 100 USD to EUR" vs "Convert 250 USD to EUR" pair at 0.99 cosine is the same shape as the one that drove my design: "activate user 4471" vs "deactivate user 4471" scores 0.853 cosine — embeddings capture topic, not polarity, so no threshold separates near-opposites from genuine paraphrases. I agree with the conclusion that the guard has to be structural, not a threshold tweak.
One thing I'd add from my side: I ran into this at the trajectory level, not the response level. When an agent has already solved a goal, replaying the cached plan is far cheaper than re-running the loop (a 14-step trajectory replays in 47ms vs 3.9s live in my measurements) — but a fuzzy match serving the wrong write is worse than no cache at all. So I made every stored tool step declare read vs write, and write-containing trajectories replay only on an exact goal+context repeat, never a fuzzy one. An 18-pair adversarial eval runs as a CI gate: a threshold-only baseline serves the wrong command 1/18 times, the gated version 0/18.
The cost of that stance is exactly what you describe — the cache abstains on paraphrases rather than risking the wrong action, so hit rate depends entirely on how repetitive your goals are. Curious whether your guard chain has a similar recall tradeoff, or whether the verifier in 2.0 bought back some of it.
Thanks Kyle, and yes, the tradeoff is there. Guards alone reject 67% of near misses and keep 88% of paraphrases. The verifier doesn’t recover recall: with a cross-encoder as the reference verifier, it stops 40 of the 50 near misses the guards let through on the held-out split (false-hit 0.291 → 0.058), but rephrasings kept drop from 0.881 to 0.452. The validation split shows the same pattern, 0.882 → 0.686. That’s why it’s opt-in. Your read/write split is a better answer for agents, since the exact-match rule only costs hit rate where a wrong hit is expensive. Kmemo caches single responses, so I can’t scope strictness by side effects like that.
Thanks for sharing the numbers — that tradeoff is exactly the crux. A verifier that cuts false hits from 0.291 to 0.058 while dropping kept rephrasings from 0.881 to 0.452 is buying correctness with recall across the board, which is the right call when every hit is equally risky. The read/write split is just a way to stop paying that recall price where a wrong hit is cheap: exact-match only where a wrong hit is expensive, and looser matching where it just costs a worse answer. And agreed on the scoping point — a single-response cache has no side-effect axis to scope on, so the gate has to live somewhere else.
Nice project! I really like the idea of using semantic guards instead of relying only on similarity scores. The explanation about the 100 USD vs. 250 USD example makes the problem very clear. I'll definitely test this approach in one of my AI projects. We also share free developer resources and tutorials on Codecan.net, so this was an interesting read. Thanks for making it open source!
Thanks, glad that example landed. It's the one that made me build the thing in
the first place. If you do try it on one of your projects I'd love to hear how
it goes, especially what your near misses look like: the guards are tuned on a
corpus that's mostly numbers, units and entities, and I have no idea yet how
well that carries into other domains. And if you like where it's heading, a star
genuinely helps people stumble on it.