I built TechDebt ML, an open-source Random
Forest model that predicts which files in a Python repo are likely to be carrying technical
debt. Along the way it hit 100% accuracy (a red flag, not a win), quietly turned into a "how
much git activity does this file have" detector, and only got genuinely useful once I stopped
trusting my own test set. This is the story of those three mistakes, with the numbers.
Why build this
I'm a systems engineering student, and I wanted a project that let me practice ML end-to-end
while building something a "vibe-coder", someone shipping fast with AI-assisted code, could
actually use. Technical debt felt like a good target: everyone agrees it's a problem, almost
nobody has a fast way to know which files in their repo have it before it bites them.
The plan: scrape a batch of public Python repos, extract code-quality and git-activity metrics
per file, label files based on how many bug-fix commits touched them, and train a classifier.
Mistake #1: a suspiciously perfect model
First training run: 100% accuracy. If you've done any ML, you already know this is bad
news, not good news. A model that never makes a mistake usually isn't learning a pattern, it's
found a shortcut.
The shortcut was embarrassingly direct. My label was computed as:
label = 1 if bug_fix_commits >= 2 else 0
And bug_fix_commits, a count of commits with "fix"/"bug"/"error" in the message, was also
one of my model's input features. I wasn't predicting technical debt. I was teaching the model
to look up its own answer key.
feature_importances_ confirmed it: that one feature held 59% of the model's decision weight.
I removed it (and a related refactor_commits feature with the same problem), retrained, and
the score dropped to a much more honest 88%, recall of 0.79 on the "has technical debt"
class. Lower number, real signal. That's the trade I'll take every time.
Mistake #2: it wasn't reading code, it was reading git blame
With a legitimate model in hand, I ran it against repos it had never seen. The results looked
plausible at first glance, until I checked two files by hand.
One file had zero cyclomatic complexity, a maintainability score of 80/100 (healthy by any
metric), and the model flagged it at 97% risk. Another file had the highest complexity in
its entire repo, and the model gave it a 19% risk score, a near-total pass.
The common thread: the first file had been touched by 5 different people across 17 commits; the
second had one author and one commit. I pulled a correlation table between raw features and
predicted risk across several unseen repos:
| Feature | Correlation with risk |
|---|---|
commit_count |
0.73 |
author_count |
0.73 |
churn_rate |
0.69 |
cyclomatic_complexity |
0.13 |
maintainability_index |
-0.41 |
The model had learned "how much git activity does this file have" far more than "how well is
this code written." It made sense in hindsight, my training repos were selected for having
lots of bug-fix activity, so activity and debt were entangled in exactly the data I gave it.
Fix: instead of feeding raw commit/author/churn counts, I normalized each one relative to
its own repo's median (file's commit count / that repo's median commit count). A file's
absolute activity isn't comparable across repos with very different overall activity levels;
its activity relative to its own project is. After retraining: author_count dropped from
0.73 to 0.28 correlation, and maintainability_index, an actual code-quality signal, became
the single strongest correlate at -0.58, ahead of any git-activity feature.
Mistake #3 (sort of): a trade-off I made on purpose
Even after that fix, cyclomatic_complexity was still barely moving the needle (0.16
correlation). So I tried something more ambitious: encoding each file's most complex functions
with a sentence embedding model (all-MiniLM-L6-v2) and feeding a PCA-reduced version of that
into the model alongside the tabular features.
It worked, on the metric I actually cared about, cyclomatic_complexity's correlation with
predicted risk nearly doubled, to 0.28, and maintainability_index strengthened further. But my
local test-set recall dropped slightly, from 0.76 to 0.75.
I adopted it anyway. That local number was measured on the same 22 repos the model trained on -
part of the earlier versions' higher scores came from exactly the shortcuts I'd just spent two
iterations removing. A model that's slightly "worse" on a biased test set while genuinely better
at reading real code is the version I want people using.
What I'd tell someone starting a similar project
A perfect score is a bug report, not a result. Check feature_importances_ before you
celebrate anything.
Your training data has its own biases, and your model will find them before it finds what you
actually wanted. Selecting repos by "has lots of bugs" quietly taught the model that activity
is debt.
Test on data your model has never seen, and look at individual predictions, not just
aggregate metrics. The two files that broke my assumptions, the complex-but-untouched file,
the simple-but-busy one, never would have shown up in a confusion matrix.
Document mistakes as you find them, not after. The exact numbers (59% feature importance,
not "very high") are what make a decision log useful later, and you lose that precision fast if
you wait.
Try it / contribute
The repo, the training dataset on Hugging Face,
and the full decision log (every mistake above, documented in detail) are here:
https://github.com/Chillipeeper1/techdebt-ml
It's early-stage and I'd genuinely like help, there are issues open for multi-language support,
expanding the training set, and a temporal-split redesign that would make the whole approach more
rigorous. If you try it on your own repo and it gets something obviously wrong, that's useful
feedback too, open an issue with the file.
Top comments (0)