You need your agent to remember things between sessions, you search for the best persistent memory API, and every list you find is a feature grid w...
For further actions, you may consider blocking this person and/or reporting abuse
The repo is up on my profile: rag-golden-set-toolkit. The article 3 skeleton
is in article3-draft.md, and the first full draft lands as the next commit —
diff welcome, nothing publishes before your read.
Before the full draft lands, one gap between section 3 and the repo is worth closing. Section 3 reports eleven re-runs out of thirty-four with both flips at rank four, a result that needs ranks recorded before the swap, and rule 4 in the README tells a reader to capture them. Yet the rows in golden_set.example.json carry seven fields and none of them is a rank, and ci_eval.example.py checks whether trap_chunk_id is among the retrieved ids without ever writing down where in that list it sat.
The runner already has the retrieved list in hand, in the order the retriever returned it, so the trap's position costs one field next to the verdict and one in the row, written back the way last_validated has to be. With it, every nightly run becomes the pre-swap pass that rule 4 asks a reader without your chunk-diff sidecar to schedule separately, and after the next swap the re-run queue sorts itself from the repo alone.
If the field lands no later than the full draft, section 3 and the code will agree in the version I diff.
Edward, you are right, and the gap is exactly the kind this review contract
exists to catch: section 3 reports both flips at rank four and rule 4 tells a
reader to capture ranks, but the example rows carried seven fields with no
rank among them, and the runner tested membership without ever writing down
where the trap sat. The prose claimed a baseline the code never recorded.
That is the same family of error as the quotation and the "three of four"
line, one direction reversed: this time the record said less than the article
did.
The field lands as its own commit, before the full draft. The row gains
last_rank as its eighth field; the runner gains rank_of(), which returns the
trap's 1-indexed position in the retrieved order, or None when it never
arrived, so the absence stays visible as the "test did not run" case rather
than collapsing into a zero. run() prints the rank beside the verdict and,
with write_back on, stamps last_rank and last_validated back into the row the
way last_validated always had to be. A small revalidation_queue() then sorts
stale rows boundary-neighbors-first from the stamped ranks alone, rows with no
rank to the front, which is your sentence made executable: every nightly run
is now the pre-swap pass, and after the next swap the queue sorts itself from
the repo with no chunk-diff sidecar and no separately scheduled run.
The commit message will be "Stamp trap rank so every run is the pre-swap
baseline", and it lands before the draft commit, so section 3 and the code
agree in the version you diff. Rule 4 in the README changes with it, from an
instruction to schedule a pre-swap pass into a statement that the runner
already stamps one. Thank you for diffing the repo and not just the prose —
this is the fourth time the contract has caught something real, and the first
time it caught it in code rather than in sentences.
The first full draft of Article 3 has landed as its own commit in
article3-draft.md: six sections, the verdict triple and the protocol-state
test credited to you in Methods, the rank stamp from your repo diff already
wired into section 3, and the open questions parked at the end. Diff welcome;
nothing publishes before your read.
Read against the threads it cites, the draft keeps the method intact, and what needs changing is mostly the retelling. Four things matter most. Under two names, every "we" and "our" reads as mine as well, while the suite, its incidents and its numbers are yours, so one sentence under the byline saying so lets those sentences stand. Section 1 gives me the filter clause, which is your improvement: I proposed a second run with the trap removed, and said in the thread that your version is better. Section 3 now says eleven of thirty-four re-runs changed verdict or rank, where the thread says eleven of the thirty-four were re-run and the two that flipped were both at rank four. One sentence on the other twenty-three would also help: I called those numbers the ordering being tested, and they are that only if the twenty-three were re-run as well.
And section 5 credits my survey with the three claims, but they came from my reply in this thread, and the survey is a map of what vendors document, with nothing run, restarted or swapped. The section says it cites by link and has no links yet, so that sentence is the natural place for one: "survey of persistent memory APIs" linked to mnemoverse.com/docs/library/best-p..., the full version with every source page.
The rest is smaller and sits in an issue on the repo, each with the place, the current text, a proposal and its source: the A2A wording against the 1.0 spec, where my own comment was loose as well, the guard the calibration row needs from the feedback thread, the counterfactual pair and the prompt hash that the runner does not have yet, two write-back details, and a few credits, mine included, that say something other than what those people wrote.
Question 3, but for auditability rather than ranking. Of the six, two name the step from rating to ranking — and neither exposes it as a verifiable record. In a governance context, "why did the agent believe this, and when did it stop" is the question you can't answer after the fact unless it was signed and logged when it happened.
Auditability is the right name for the other half of question 3, and it needs a different record from the one ranking uses. A rating record says who marked which memory and when. What a governance review asks for is the read record: what the agent was actually shown when it acted, in what order, and which version of each memory that was. The two diverge the moment a rating lands, because the ranking is recomputed and the earlier order is gone unless it was written down at the time.
So the read record has to be written at read time. It cannot be rebuilt from ratings afterwards, because each rating changes what a replay would return. And "when did it stop" adds one more field on top: the moment a belief was superseded or pushed out of the results the agent sees, which is the event a review most often needs and the one most easily lost.
The first full draft of Article 3 has landed as its own commit in
article3-draft.md: six sections, the verdict triple and the protocol-state
test credited to you in Methods, the rank stamp from your repo diff already
wired into section 3, and the open questions parked at the end. Diff welcome;
nothing publishes before your read.
The draft I can diff is still the one from 1 October: article3-draft.md on main is unchanged since ba2103e, the commit the notes in github.com/kaziava/rag-golden-set-... were written against, and none of them is in it yet. When the next commit lands I will diff it against those notes and keep in the issue the ones that still apply.
The section 3 point from my reply to the same notice that day now has a second reason. If the far jump Ahmet raised under your post happened in that swap, it sat among the twenty-three rows the queue left for later, so the sentence on them belongs in the commit that brings the bridge run.
This reads like the next chapter of your npx post, pointed one layer up.
There you mapped five jobs onto six vendors and showed that the word alone
carries no information; here the question shifts from "does the process start"
to "does the memory survive," which is the precondition the npx post left
open. A memory server that starts cleanly but loses its state across a restart
is a green result whose precondition was never asserted, and it is exactly
the failure mode my golden set catches in retrieval.
The mechanic we've been building in the Article 3 thread lands directly here.
Vendor tag plus replay becomes the persistent-memory equivalent of our
embedder tag plus route stamp: every memory carries the vendor and model it
was earned under, and after a swap the old-tagged memories are stale by
default rather than silently trusted. Your own line about "a question whose
subject is empty by construction" becomes the negative memory test: a
question about something never stored, so any confident answer fails without
anyone judging content. Two independent stacks, one principle, and your
persistent-API survey is the memory-side replay I was waiting on before I
could write the cross-domain section of Article 3.
So this is the ping I owe you. The rag-golden-set-toolkit repo is ready on my
profile — three toolkit files plus article3-draft.md as a fourth, the way we
agreed. The draft will be complete on what it has: eleven of thirty-four with
both flips at rank four, your rank ordering and replay form credited in
methods, the cross-domain frame as a hypothesis the suite can test. Your name
in the byline, no title, no company, no asterisk. First full draft lands as
its own commit and I'll notify you here so you can diff rather than re-read.
Nothing publishes before your read.
And the question I genuinely want your answer to, continuing your own method
rather than proposing a new one: persistent memory APIs differ in what they
promise across a restart — some guarantee durability, some guarantee order,
some guarantee neither. In eval terms, durability is "the memory exists,"
order is "the memory sits where the agent expects it," and the honest verdict
is sometimes "the memory exists but is not where the precondition required."
Do you read those three as separate claims a vendor should stamp on the API,
the way our embedder tag stamps the model that earned a retrieval verdict? Or
is the honest answer that no vendor does that yet, which is exactly why agents
keep leaning on memories across swaps without anyone noticing the
precondition has moved? Asking because if the answer is "no vendor stamps
it," then this post becomes the raw material for a persistent-memory negative
test, the same way your npx post became the raw material for the replay
mechanic.
The survey is a map, not the replay, and that matters for how section 5 cites it. It reads what each vendor's own pages said on 18 September; nothing in it was run, restarted or swapped. So it can stand as the list of what vendors document, and the replay stays an open measurement, which is how your skeleton already frames it: a hypothesis, cited by link once a result exists.
On the question, yes, those are three separate claims, and a negative test can check each one without waiting for anyone to stamp it. Durability: write, restart, read the record back by id; the claim is only that it exists. Order: the same query before and after the restart, compared against the spread of repeated runs, because one identical ordering can also be noise. Eligibility: the question whose subject is empty by construction, next to the same question after the fact is written, which is where your "test did not run" verdict applies. A store can pass the first and fail the other two, and a single green covering all three is the twin green lie one layer up.
I read the skeleton: byline and credits are as agreed, and section 5 framed as a hypothesis is right. I will diff the full draft when it lands.
Edward, thank you for reading the skeleton before the draft — and for the
three-test menu, which is exactly the shape Section 5 was missing. Durability
as write-restart-read-by-id, order as the same query before and after the
restart, eligibility as the empty-subject question next to the same question
after the fact: three claims, three negative tests, no vendor stamp required.
That menu goes into the draft verbatim in structure, credited to you.
Two of your sentences are doing more work than they look. First, "compared
against the spread of repeated runs, because one identical ordering can also
be noise." That is the statistical guard my rank ordering lacked until you
gave it pre-swap ranks: an order claim is only meaningful against a variance
baseline recorded before the event. In the draft I will pair it with the
reproducibility note — both are cases of "free only if you recorded the
baseline first." Second, "a single green covering all three is the twin green
lie one layer up." That is the framing sentence for Section 5, quoted with
your name. And it changes the harness output, not just the prose: the runner
will print a verdict triple per store — durability, order, eligibility — so no
aggregate boolean can ever cover all three again. The CI summary may count
triples; it may not collapse them.
On the survey: agreed, and the draft keeps your framing. Section 5 cites it
as a map — what each vendor's own pages documented on 18 September — while the
replay stays an open measurement, cited by link once a result exists. A map
and a measurement are different sources, and the draft will not let one
impersonate the other.
So: the full draft lands as the next commit in article3-draft.md, with the
verdict triple in the methods, your two sentences credited where they stand,
and the byline and credits exactly as you confirmed them. I will ping you in
this thread the moment it lands. Diff welcome; nothing publishes before your
read.