How I Sorted 1,200 TODO Comments with Free AI Models and Strict JSON

#ai #python #productivity #refactoring

Every codebase has a landfill. This one had 1,200 TODO comments. Each represented an unprocessed decision. Some were urgent. Most were forgotten. The refactor queue was invisible. Sorting it manually would take hours. Free AI models changed that math.

I am testing the free-tier workflow of an AI coding assistant. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides access to free models. It also offers a free server for heavy batch tasks. This combination fits one workflow perfectly. It classifies technical debt at scale. Here is the full pipeline, raw.

The Setup

The goal was simple. Turn a pile of comments into a prioritized list. No fancy ML training. Just a zero-shot prompt and a JSON parser. The script was the easy part. Python handles it in thirty lines.

import re, json
from pathlib import Path

pattern = re.compile(r'(//|#|/\*|--)\s*(TODO|FIXME|HACK|XXX)')
todos = []

for path in Path('src').rglob('*.ts'):
    if 'node_modules' in path.parts:
        continue
    for i, line in enumerate(path.read_text().splitlines(), 1):
        m = pattern.search(line)
        if m:
            todos.append({
                'file': str(path),
                'line': i,
                'type': m.group(2),
                'tag': line[m.end():].strip()
            })

open('/tmp/todos.json', 'w').write(json.dumps(todos, indent=2))
print(f"Extracted {len(todos)} TODO comments")

Run this against any codebase. Adjust the regex for your language. It exports a clean JSON file. That file becomes the input for the AI classifier.

The Decision Matrix

The model received a strict taxonomy. Four labels. Each maps to a clear action. The matrix was defined before writing the prompt. Clarity upfront prevents output drift.

Label	Effort	Risk	Action
URGENT	High	High	Refactor immediately
CANDIDATE	Medium	Medium	Schedule for next sprint
DEFERRED	Low	Low	Keep with a date
SPAM	Any	None	Delete the comment

This matrix is the core artifact. Free models handle repetitive labeling well. The matrix keeps their output structured. Without it, the AI returns prose. With it, the output is a clean list.

The Zero-Shot Prompt

A good prompt beats a bigger model. This prompt worked on the first attempt. It emphasizes strict JSON output. It also rejects prose entirely.

You are a senior software architect. Analyze the attached JSON array of TODO comments. Each entry has a file, line, type, and context. Assign exactly one label from [URGENT, CANDIDATE, DEFERRED, SPAM] to each entry. Justify nothing. Return a flat JSON array only. The output must be valid JSON.

Running this through MonkeyCode's free models took one minute. The free server handled the batch processing seamlessly. My local machine stayed completely free. The result was a 1,200-entry JSON array. Perfectly structured.

The Accuracy Audit

Raw output is unverified output. A 10% sample needed manual review. The provisional results were surprising. The model showed high precision on obvious spam. It also showed strong focus on high-risk files.

Bucket	Total Count	Manual Sample	Matches	Precision
URGENT	87	10	8	80%
CANDIDATE	415	42	35	83%
DEFERRED	491	49	43	88%
SPAM	207	21	20	95%

Overall precision settled at 85%. The biggest flaw was contextual. The model mislabeled deleted legacy code as CANDIDATE. It saw a server file and assumed business value. Humans know that file is a tombstone. This is the current ceiling for free models.

Lessons for Free-Model Workflows

Free models are not free maintenance. They offload toil. They don't offload judgment. Here are the hard rules from this run.

Define your taxonomy before prompting.
Always request strict JSON output.
Audit a random sample before executing changes.
Feed audit corrections back into the next prompt.

Do these four things. The free tier becomes a reliable junior engineer. Skip them. It becomes a confident intern.

Who Should Not Use This

A solo project with ten comments needs no AI. The overhead will exceed the value. A production monolith with years of debt is a perfect fit. Anything in between requires careful cost analysis. The setup took fifteen minutes. The payoff scales directly with comment volume.

The Final Verdict

The pipeline turned hours of triage into minutes. It produced clear labels and an actionable queue. The free server made batch classification painless. The 15% error rate was manageable with review. There was no bill, only a better backlog. The refactor queue finally wrote itself. Run this on your own repo tomorrow. The landfill will not clean itself.

DEV Community