Tags:ai, productivity, agents, testing
Most "test your AI agent" advice stops at a single prompt. Ask it something, check if the answer is good, move on. That tells you almost nothing about whether the thing survives a real week of work, where the task you started on Monday isn't the task you're finishing on Thursday.
So I ran a messier test. I picked a refactor I was already doing (splitting one bloated module into three, with call sites scattered across a handful of files) and did the whole thing through an AI Agent instead of just asking it to write code in isolation. Not because I distrust writing code by hand, but because I wanted to see where a multi-day, multi-file task actually breaks an agent's grip on what it's doing, and where it doesn't.
The setup wasn't fair, on purpose
Day one, I described the target structure and asked for a plan. Fine, every tool handles day one fine. The part I actually cared about was day two and three: coming back after a break, changing my mind about one piece of the split partway through, and asking follow-up questions that referenced decisions from the first session without repeating them.
That's the actual shape of a refactor. You don't decide everything up front and execute in a straight line. You decide, get partway through, notice the plan was wrong about one file, and adjust.
Where it held up
Referencing an earlier decision without re-explaining it worked better than I expected. When I said "keep the naming convention we used for the second module" on day three, it didn't ask me what convention I meant, it pulled it from day one's session instead of guessing or defaulting to something generic. That's a small thing, but it's the thing that usually kills momentum: stopping to re-explain context you already gave, because the tool quietly forgot it.
The other thing that held up was catching an inconsistency I introduced myself. Partway through, I asked for a change to one file that technically contradicted a naming decision from earlier. Instead of just doing what I asked, it flagged the contradiction first and asked which one should win. I'll take being annoyed at a clarifying question over silently ending up with two different naming conventions in the same codebase.
Where it didn't
The plan itself, the actual "should this go here or there" decision, is still mine to make and re-make. When I changed my mind about which of the three new modules should own a shared utility function, the agent adjusted every file consistently once I told it, but it didn't push back or suggest the earlier placement was better. It executed the correction cleanly. It did not have an opinion I should have wanted it to have.
And it's worth saying plainly: I mentioned above that I work on this stuff (I'm the Product Lead at Noumi.AI], which builds AI Agents aimed at exactly this kind of multi-day, multi-file work), so take "held up better than expected" with the appropriate grain of salt. I tried to test it like I would test anyone else's tool, not like I was writing marketing copy, but I'm not a neutral party here.
What I'd actually check before trusting this on a real task
If you're evaluating any AI Agent for something longer than a single prompt, the useful test isn't "does it write good code once." It's:
· Does it remember a decision from an earlier session without you repeating it
· Does it notice when a new request contradicts an earlier one, instead of just complying
· When you change your mind halfway through, does the correction propagate everywhere it needs to, not just where you happened to ask
None of these show up in a one-shot demo. They only show up once the task has enough days and enough files that keeping it all straight becomes the actual hard part, which, if I'm honest, was the whole reason I ran this test in the first place instead of trusting a quick sample prompt.
I don't think one refactor on one codebase proves much on its own. If you've run something similar, a longer-running task where the agent had to hold onto context across sessions, I'd genuinely like to hear where it broke for you.
Top comments (1)
Multi-day refactors are a much better memory test than toy recall prompts. The question is not whether the agent remembers a fact, but whether it preserves intent across interruptions: why a change started, what invariant must survive, and which old decision should be challenged instead of obeyed.