DEV Community

Discussion on: How can I prevent my AI coding assistant from repeating fixed mistakes across sessions?

Collapse
 
mihai_leanzero profile image
Mihai Perdum •

The failure mode I've hit more than the ignored-correction one: the correction gets stored, retrieved, and applied with full confidence — but the diagnosis written into it was wrong in the first place, just close enough to look right on the case that produced it. It fixes the symptom, gets marked resolved, and reapplies wrong the next time a similar-but-not-identical case comes up. A negative verdict on a bad recall doesn't catch this, because the recall wasn't bad the first time — the write was. Worth a verification step before a correction becomes eligible for the store, not just an update step after.

Collapse
 
izgorodin profile image
Edward Izgorodin •

That is the case where the verdict arrives one step too late by design. The first application looks like a success, so the only signal the correction ever received was positive, and it enters the store with more credit than a fresh note would have had.

A verification step before eligibility needs something the correction was not written from. A diagnosis fitted to one case passes that case by construction, so the test that means anything is a second case of the same kind. A correction becomes eligible once it has held on a case it did not see, and until then it is a candidate.

That also changes what gets stored. Keeping the originating case next to the correction means that when it misfires on the similar but not identical case, both cases are available side by side, and the difference between them is usually where the real diagnosis was.