The application gap is the one that shows up in real agent runs. An agent can recite a constraint when you ask for it, then ignore that same constraint two turns later when the task never restates it. Direct recall tests pass while the patch quietly invents a cache, a timer, or a dependency that the architecture forbids. The useful memory check is whether the fact appears in the change without you having to re-prompt it.
The patch is where I’d score it too. I’d still record whether the constraint reached the model, because “never retrieved it” and “received it but ignored it” need different fixes. Then add a task where the constraint should not apply, so repeating every remembered rule does not look like success.
We're a place where coders share, stay up-to-date and grow their careers.
The application gap is the one that shows up in real agent runs. An agent can recite a constraint when you ask for it, then ignore that same constraint two turns later when the task never restates it. Direct recall tests pass while the patch quietly invents a cache, a timer, or a dependency that the architecture forbids. The useful memory check is whether the fact appears in the change without you having to re-prompt it.
The patch is where I’d score it too. I’d still record whether the constraint reached the model, because “never retrieved it” and “received it but ignored it” need different fixes. Then add a task where the constraint should not apply, so repeating every remembered rule does not look like success.