DEV Community

Revin
Revin

Posted on

An agent rewrote a retry helper over a TODO from 2019: I counted 189 stale markers before the next run

Last Tuesday I gave a coding agent one task on a Node service I have read access to: add an optional source field to a POST endpoint and persist it. Two files and a migration. The pull request that came back touched 14 files and included a rewritten retry helper nobody had asked about.

I went looking for the reason. It was three lines above the handler:

// TODO(OPS-4412): retry logic here is wrong, fix when we drop the v1 client
Enter fullscreen mode Exit fullscreen mode

The v1 client was dropped in 2021. The tracker behind OPS-4412 was turned off the same year. Nothing in the repository says either of those things. The agent read a comment written by an engineer, about that exact file, and treated it as work in scope. That is what a comment is for.

InfoQ published a piece on this a few hours ago and the behaviour has a name now. Sam Saccone and Madhuri Garikapati, at Google, call it agent distraction: stale TODOs, obsolete bug references and specs nobody updated pull an agent into side quests nobody requested. Reading it is what made me stop rewriting the prompt and start counting what the repository was telling the agent.

Counting before cleaning

First question: how much of this is in there at all.

$ rg -n --no-heading -e 'TODO' -e 'FIXME' -e 'HACK' -e 'XXX' src | wc -l
211
Enter fullscreen mode Exit fullscreen mode

A raw count says nothing. What matters is age, because a marker written last month is a note and a marker written in 2018 is furniture. So I dated every one of them by the commit that introduced the line:

rg -n --no-heading -e 'TODO' -e 'FIXME' -e 'HACK' -e 'XXX' src \
  | cut -d: -f1,2 \
  | while IFS=: read -r file line; do
      git blame -L "$line,$line" --porcelain -- "$file" \
        | awk '/^author-time/ {print strftime("%Y", $2)}'
    done \
  | sort | uniq -c
Enter fullscreen mode Exit fullscreen mode
     12 2017
     19 2018
     31 2019
     44 2020
     38 2021
     22 2022
      9 2023
     14 2024
     12 2025
     10 2026
Enter fullscreen mode Exit fullscreen mode

189 of the 211 were written before 2025. The service has had 4 contributors in the last twelve months, and none of them wrote most of that.

Then the ones pointing at systems that no longer answer:

$ rg -o 'OPS-[0-9]+' src | wc -l
37
Enter fullscreen mode Exit fullscreen mode

Thirty seven references to a ticket prefix that resolves to nothing. Another 9 named feature flags that are not in the config file anymore. Those 46 lines are pure instruction with no way to verify it, which is the worst possible input for something that reads the repository as context and cannot ask a colleague.

Three kinds of rot, and only one is in the code

The comments were the obvious part. The documents did more damage per line:

$ for f in docs/*.md; do printf '%s %s\n' "$(git log -1 --format=%ad --date=short -- "$f")" "$f"; done | sort
2022-03-04 docs/billing-webhooks.md
2022-08-17 docs/payments-api.md
2024-11-02 docs/architecture.md
2026-02-09 docs/README.md
Enter fullscreen mode Exit fullscreen mode

Two specs frozen in 2022, describing response shapes that changed at least twice since. The agent trusts a markdown file more than it trusts the code, because the file reads like intent and the code reads like an accident.

And the file written specifically for the agent was wrong too:

$ grep -n 'yarn' AGENTS.md
12:Run `yarn test:unit` before proposing changes.

$ jq -r '.scripts | keys[]' package.json | grep test
test
test:integration
Enter fullscreen mode Exit fullscreen mode

That script was deleted some seven months ago. Part of two runs went into making a command that does not exist produce output. There is a post here on dev.to about AGENTS.md being an attack surface that pulled 32 comments, and the uncomfortable half of it is this: the same blind trust that makes a malicious file dangerous makes a merely outdated one expensive.

So the inventory came out in three piles:

  • markers that reference dead trackers, dead flags and dead clients (46 lines)
  • markers that still describe real unfinished work, written by people who left (a shade over 60, I stopped counting exactly at 63)
  • documents and agent instructions that contradict the running code (3 files)

What did not work

Two attempts before the one that stuck.

I added a line to AGENTS.md telling the agent to ignore TODO comments older than a year. It held on short tasks and fell apart on anything that ran more than a few minutes, which matches what the Google engineers describe: the distraction does not come from one instruction, it comes from the accumulated context, and my line was one sentence against 189.

Then I deleted all 211 markers in a single commit. That was worse. Two of them described a known race condition in the webhook consumer that was never ticketed anywhere, and I only noticed because I had the diff open when someone mentioned the symptom. Deleting a marker deletes the only record of the problem in about the frequency you would expect from an undocumented codebase.

The pass itself

In the order I ran it, over roughly two afternoons:

  1. Markers referencing the dead tracker, flags or clients: read each one, delete. 46 lines gone, no replacement.
  2. Markers describing real work: open an issue, replace the comment with one line carrying the issue URL and the date. The comment stops being an instruction and becomes a pointer.
  3. The two frozen specs: moved to docs/archive/ with a first line saying what they are and when they stopped being true. Archived beats deleted, because the next person still wants the history.
  4. AGENTS.md: every command in it executed by hand before the commit. If it does not run, it does not stay in the file.

No refactoring in that pass. Nothing in src changed behaviour, and that was on purpose, so the measurement afterwards would have one variable.

What changed

Twelve tasks before the cleanup, twelve after, same kind of work: small field additions, two bug fixes, one endpoint.

Before, the median pull request touched a shade over 9 files, and 11 of the 12 contained at least one change I had not asked for. After, the median was 3 files, and unrequested changes showed up in 2 of 12. Input tokens per task fell roughly 28%, from somewhere around 61k to around 44k, which is the number everyone quotes and the least interesting one here. The expensive part was never the tokens. It was reading a 14 file diff to find out that 12 of those files were a conversation from 2019.

I do not know how this holds up on a repository with twenty active contributors. This one has four, and the rot was almost entirely written by people who are gone, which makes it easy to delete and hard to argue with. On a busy codebase the markers are alive and the triage is a political problem, not a git blame problem.

The part I have not solved

The repository is clean today and it will rot back. The obvious fix is a CI rule that fails on a TODO without an issue reference and a date. I have not shipped it, because I suspect it teaches people to write TODOs without using the word TODO, and an undetectable stale marker is strictly worse than a detectable one.

So: do you gate markers in CI, do you expire them, or do you run a cleanup pass before each agent session and accept it as a cost of operating the thing? If you gate them, did the comments actually improve, or did they just move?

I write about the measurement side of this at the Revin blog: Technical debt from AI coding agents: how to measure the bill.

Top comments (3)

Collapse
 
chao_yan_0eca708448360655 profile image
Henry Yan •

The retry-helper rewrite is a clean case of agent distraction, and dating the markers by blame is the right measurement. The 14-file PR for a two-file task was not a prompt failure: a 2019 TODO three lines above the handler looked in-scope.

The part I would add is a scope contract that comments cannot enlarge. Even after a cleanup, a fresh TODO or a stale spec in docs/ will be read as intent. If the task record names the allowed write set — the endpoint, the migration, and nothing else — a comment in the same file can inform the change but cannot authorize rewriting a nearby helper.

I would not fail CI on the word TODO. I would fail a PR that touches files outside the task's write set, and treat leftover markers as untrusted context rather than tickets the agent may pick up. That avoids teaching people to disguise the word, which you are right is strictly worse.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

The count shows what got cleaned, not what the next tasks cost. Did you rerun those 12 tasks after the cleanup?

Collapse
 
ontomato profile image
ontomato •

if OPS-4412 is closed, or the date on the marker has passed, that line shouldn't be in the agent's context for the next run. re-read the live issue at session start, or drop the marker when you can't.