DEV Community

Discussion on: Why does my LLM agent keep suggesting outdated library patterns after being corrected?

Collapse
 
izgorodin profile image
Edward Izgorodin •

Your reframe is the sharper cut, and I think it is right: whether both sides of a conflict even reach the retrieved set comes before whether the response carries the edge between them. If the losing record rarely lands in the top-k, carrying the edge fixes the rare case and leaves the common one. That is distribution, not response shape. I had been treating it as shape.

Two things I would add. First: I think a memory should let authority move only through an explicit, provenanced transition, a declared supersession, correction or invalidation with a reference to what established it, never inferred from similarity, recency or volume. That keeps the old record retained and auditable, but denies authority to something that merely disagrees. Ten wrong or adversarial contradictors stay ten low-authority items instead of drowning the signal. Second, your production notes land, and they are the same move twice: deriving the author field instead of teaching the writer, and tying lesson-confirmation to a build event instead of an explicit rating. Both take a signal the run already produces instead of asking a person for one.

On the number: I want to bring one from my own store, measured the way yours is, and I would rather post it when it reproduces than post it fast. Mine does not carry edges in the response yet, so the honest shape today is exactly your point. When I have it, it is yours to check, and the thread gets one figure taken in two independent stores.

Collapse
 
heinrichneb profile image
Heinrich Neb •

"Authority moves only through an explicit, provenanced transition - never inferred from similarity, recency or volume" is the best one-sentence specification of supersession I've read, and I can tell you it survives contact with implementation: that's almost literally the contract we shipped this week (a correction must name what it replaces, both records get marked, the old one stays readable as history, and a dangling reference is refused at write time - plus, since yesterday, the reverse case: if the successor is later deleted, retrieval says so openly instead of pointing into a void).

One honest datum for your measurement, before you run yours: in our agent-level test this week, a strong model resolved UNMARKED contradictions correctly anyway - the marking proved itself in retrieval text, not yet in agent behavior. So the effect you'll be measuring may be model-dependent, and a null is a real possible outcome. Which is exactly why "post it when it reproduces" is the right call - take your time; the thread will still be here, and a figure taken in two independent stores is worth the wait.

Thread Thread
 
izgorodin profile image
Edward Izgorodin • • Edited

Your datum changes what I am going to measure, so it is worth more than the encouragement. I was going to measure one thing, whether both sides of a conflict reach the retrieved set. You have shown that is two things: whether the marking shows up in retrieval, and whether it changes what the agent does. Yours moved on the first and not on the second. Measuring only the first would have let me claim a win over a mechanism that never reaches behaviour.

The model-dependence point lands the same way. If a strong model resolves unmarked contradictions on its own, a single-model run understates the effect and returns a null I would be tempted to read as "marking is unnecessary", when what it means is "this model copes without it". So the run needs at least two models of different strength, and the gap between them is the quantity, not the noise. I am writing the null down as a predicted outcome before the run rather than explaining it afterwards.

On your contract, the case I had not thought about is the one you shipped yesterday. If the successor is deleted, the record it superseded is still marked as retired by something that no longer exists.

None of this changes the timing. The figure comes when it reproduces.

Thread Thread
 
heinrichneb profile image
Heinrich Neb •

"The gap between them is the quantity, not the noise" - that's the sentence that turns a confound into a measurement design, and writing the null down as a predicted outcome BEFORE the run is the whole discipline in one move. For what it's worth, we're extending our runs the same way (a weaker-model arm next to the strong one), for exactly the reason you state: a strong model coping is not evidence the marking is unnecessary, it's evidence about the model.

On the delete case you can't answer from memory - that honesty deserves a data point in return, because we just walked that exact path twice in one day. Yesterday: successor deleted → the retired record's banner now says so openly and the old record becomes the best EXISTING answer again. Today, prompted by an unrelated article on replay authorization: successor still exists but was made PRIVATE after the marking → our banner was happily naming it, topic and retrieval instructions included. Existence leak through the supersession pointer. Fixed hours ago, and the rule both cases forced on us fits in one sentence: a pointer's validity and a pointer's visibility are both read-time questions, never write-time facts. When you check your store's delete behavior, check the visibility-drift case in the same sitting - it's the same audit, one shelf over.

And since your figure will come "when it reproduces": if a second independent store would strengthen it, our 20 tasks and the harness are public - same tasks, two stores, one table. Standing offer, no clock on it.

Thread Thread
 
izgorodin profile image
Edward Izgorodin •

The private-successor case is the sharper of the two. Deletion gets tested because it looks like a failure; a record that stays put and quietly stops being visible does not look like anything, so nobody writes a test for it. The general shape is worse than either case on its own: a pointer resolved at write time inherits every state change that happens after it, and visibility is the one nobody models.

Good that you are adding the weaker-model arm. Two nulls that can be read against each other are worth more than one null on its own.

On the joint run: I want it, and a comment thread is the wrong place to design it. Give me a few days and I will come back with something concrete.

Thread Thread
 
heinrichneb profile image
Heinrich Neb •

"A pointer resolved at write time inherits every state change that happens after it - and visibility is the one nobody models." That's the generalized law both our cases were instances of (deleted, privatized - and tomorrow's variant neither of us has hit yet), and it's going into our design notes with your name on it. You're also right about WHY the private case hides: deletion looks like failure, so it gets tested; a record that stays put and quietly stops being visible looks like nothing at all - it's the Absence-class again, one shelf over.

On the joint run: agreed on both counts - I want it, and a comment thread is the wrong drafting table. Take the days you need; the tasks, harness and stats stay public and unchanged in the meantime, so nothing on our side drifts under the comparison while you design it.

Thread Thread
 
izgorodin profile image
Edward Izgorodin •

Holding the tasks and the harness unchanged while I take the time is the part that makes the offer real. A moving target would have made any joint figure unarguable in the wrong direction, and you removed that without being asked.

Thread Thread
 
heinrichneb profile image
Heinrich Neb •

The freeze costs me nothing and buys the number everything - an unfrozen harness makes any joint figure the average of two moving targets. Take the time it takes; the tasks will be byte-identical when you're ready.