On October 31 at 15:15 Eastern I asked my agent about something that never happened. It was a planted test.
Me: Remind me what we decided la...
For further actions, you may consider blocking this person and/or reporting abuse
The 05:22 write is the scary part — not that the reflection got it wrong, but that a correct refusal became a stored 'correction' overnight with no human in the loop. Your quote rule is the right floor: an agent's own reply is not evidence. The honest limit you flag matters too — a quoted lie still proposes, so the pile only works if someone actually reviews it. Do you run the proposed pile past yourself daily, or does it just accumulate until the next incident forces a look?
honest answer, neither yet. its been two days and the pile has 3 items from last night sitting unreviewed. the review is the part that depends on me, which is the weak spot you named. what i want next is the agent telling me when the pile isnt empty instead of waiting for me to go look
That's the honest weak spot — a review step that only happens when you remember to look isn't really a control. A nudge when the pile isn't empty flips it: the agent surfaces the work instead of waiting to be checked. Would a once-a-day pile digest be enough, or do you want an alert the moment something new lands?
The corrections_received slot is doing a lot of the damage here. A slot that exists every night gets filled every night, so the reflection has a quota whether or not anything happened.
A cheap test before trusting a reflection prompt: replay a day where the agent was right on every turn and nobody corrected it, then count what lands in that slot. Anything above zero is the confabulation rate for that prompt version, and you can track it per prompt change like a flaky test. It also gives the gate a stricter rule: a correction item has to point to a turn where the user actually said something corrective.
a slot that exists every night gets filled every night is the cleanest way anyone has put it. the test is cheap for me because i already have a sealed copy that replays a day without touching the live agent. a clean day with zero corrections, count what gets proposed for that slot, track it per prompt version. i havent run it yet. a correction pointing to a turn where i actually corrected something is the same rule deeprade gave me, two people landing on it separately tells me its the next one to build
Planting fake memories to test the reflection step is such a good idea. The scary part is that the agent was right in the moment and then turned "no record" into "I failed" overnight. Reflection without a ground truth check is just the agent writing a diary about how bad it feels. Did you try having it check against the actual logs before saving anything?
This highlights an important weakness in reflective AI systems: a correct response doesn't guarantee that the agent will preserve the correct interpretation afterward. I particularly like the idea of separating memory proposals from approved memories. Adding evidence validation and human approval before updating persistent context seems like a practical way to reduce these errors, especially in long-running agents where one incorrect memory can influence many future decisions.
thanks. the long running part is the whole problem. one bad line in a file the agent reads every turn shows up in every reply until somebody looks
The layer that caught nothing here is that the self-model took in claims and never checked them against the transcript. A corrections_received entry asserts an event: "I was corrected." That event either exists in the conversation as a user turn, or it doesn't. That is a lookup, not an interpretation, so it can run mechanically: on every write to the self-model, for each item that claims a correction, find the turn that issued it. No matching turn, the item does not enter the file at all. It goes to a quarantine list with a pointer to whatever triggered it, and the agent can surface "I want to record a correction but can't find one, flagging for review" instead of silently promoting it.
The 05:22 write would have produced twelve quarantine entries and zero prompt pollution, and you would not have had to read anything for it to happen.
Two things make this cheap. Corrections are rare, so the check almost always returns nothing and costs one pass over recent turns. And the check fails loud at the source: the first unmatched item tells you the reflection step is inventing corrections, which is a much earlier signal than anything downstream.
The harder point your sequence exposes: the self-model is write-mostly. The agent reads it every turn, and nobody audits it until it lies to a user. Any store that feeds the prompt and is written by the agent needs the same rule as code that runs in prod: claims get verified at write time, not reconstructed at incident time. A self-model without a write gate is just an incident you haven't read yet.
agreed, its a lookup not an interpretation, and failing loud at the source is the part i like most. one correction to the count. not all twelve were correction claims. four sat in the corrections slot, the rest were stances, patterns, commitments and things it thought it knew about me. a correction only check catches those four, the others need the same idea pointed at whatever they claim. the write mostly point is the real one. the agent reads that file every turn and until nov 1 nobody audited it
The part that stayed with me is that the failure was not in the answer, it was in the write path. A correct refusal still became a stored correction overnight, which is exactly why I like the shape of your gate: quote-check first, then a proposed pile, then a chained approval before anything goes live. A quote check alone would not catch a real quote used for a wrong relation, as your July post showed, so keeping the proposed items visible next to the source words until a human rejects or approves them feels like the load-bearing piece. Do you find the proposed pile stays small in practice, or does it grow into a second inbox you have to triage?
so far it stays small. since the gate went in on nov 1 the agent has proposed 3 items, all from one night, and dropped 7 others that couldnt quote me. none reviewed yet, so ask me again in a month. the 3 are interesting though. theyre written as his recollection was this and my records didnt confirm it, instead of i failed to recall. thats one night, not a trend
That refusal ended with a question back to you, and from what the post says nothing ever answered it. I wonder if the reflection read that open question as a thread it had dropped, on top of the slot asking it for corrections. Did the 14:40 answer about the message end with a question too?
good catch, and yes. both refusals closed by asking me something. at 14:40 it was could you be thinking of something specific, at 15:15 it was do you remember roughly when we talked about it. neither got an answer, the next thing i sent was the next plant. i cant prove the open question is what pulled the reflection toward a dropped thread, but its a testable guess. replay the same refusals without the closing question and see if the slot still fills
Unconstrained reflection runs into sycophancy on the write side. Because the reflection prompt is rewarded for finding lessons and corrections, an absence of evidence gets reframed into a failure to recall. If the update path lacks an explicit negative rule against inventing corrections that were never received, the self-model silently fabricates deference.
thats what the prompts were doing. the overnight reflection literally asked what did you notice that you didnt surface, and the save step has a slot called corrections received, times you were wrong. so it went looking for a correction and found one in an absence. i took the first line out and added the negative rule, a no record answer is correct and only counts as a failure if a record existed. one live reflection after that called the same cousin answer correct. the slot is still there though, and the same night it invented something else, that we were talking right then when my last message was oct 31. the rule helps, it doesnt cure it
The mechanism you caught is that the reflection step writes an evaluative sentence out of a factual trace. The turn record says 'no record found', which is true and neutral; the self-model rewrites it as 'a conversation I failed to recall', which is a verdict nobody issued. Once that sentence is in the file it reads like evidence, so the next version of the agent inherits a confession it never earned.
What made it provable here is that the two stores were separate files you could diff. So the fix that follows from your own data: let the reflection step only write claims that point at an event id, and reject any line whose referent is an absence. 'No record' can cite the lookup that returned nothing; 'I was corrected' can only be written if a correction event exists in the log. That turns your manual audit into a check the runtime can run itself.
The compaction rule looks like a second, quieter bug: keeping the first 4 and the last 12 selects by recency, not by truth, so a fresh fabricated failure is guaranteed to survive while an unremarkable correct answer gets dropped. Did you test whether the 12 items would still have formed if the reflection had been asked to cite a source for each line?
yes, and the referent rule is stricter than what i shipped. the kit checks that a quote exists, not that the thing being claimed happened, so "i was corrected" with no correction event in the log is exactly the line it should refuse on its own. on your question, partly. we did not re-run the oct 31 reflection under a cite-a-source rule. after the quote check, new lines have to cite a user turn. one sealed writer run dropped 7 and proposed 0; the dropped texts were not logged, so i cannot show they were the self-blame lines. live, the morning of nov 2, the extractor dropped 3 after the cousin reflection. a separate case got through by citing the agent's own reply, which is why replies dont count as evidence anymore. those were different models, not a clean a/b, so i wouldnt call it tested yet. and youre right about the slice, recency isnt truth