Last week I published a paper claiming that a vulnerability detection model was mostly reading comments instead of code. I predicted accuracy would drop significantly after stripping comments.
I ran the follow-up experiment. I was wrong about the magnitude.
The Ablation
Five conditions, same model (TF-IDF + logistic regression), same dataset:
| Condition | Accuracy | Drop |
|---|---|---|
| Baseline (with comments) | 84.7% | -- |
| Comments stripped | 82.6% | -2.1% |
| Label words removed | 83.9% | -0.8% |
| Both combined | 82.6% | -2.1% |
| Full clean (comments + labels + identifiers) | 81.1% | -3.6% |
Only 3.6 percentage points was leakage. The model keeps 81.1% accuracy on fully cleaned code.
What Changed in the Features
Before cleaning (baseline):
Top vulnerability signals: substrings of "vulnerable", "this", string concatenation
Top safety signals: substrings of "safe", "the", "secure"
After full cleaning:
Top vulnerability signals: eval(, string concatenation (+), eva, ev
Top safety signals: if, if, else, els
Every top feature after cleaning is actual code syntax. The model learned three real patterns:
- eval() usage -- the strongest vulnerability signal after cleaning
- String concatenation -- precursor to injection (SQL, command, template)
- Conditional density -- safe code has more if/else branching (defensive validation)
Where I Was Wrong
In the original paper I said the accuracy was "partly measuring the model ability to read English comments, not detect code vulnerabilities" and implied the real capability was much lower.
The data says otherwise. 81.1% is real. The leakage existed but it was 3.6 percentage points, not 20+. I overestimated because I confused feature rank with feature contribution. Label-word substrings were the highest-ranked features, but they did not dominate the classification boundary, which uses thousands of features.
The Decomposition
- Comment removal: -2.1pp (biggest single source)
- Label word removal: -0.8pp (smaller than expected)
- Identifier normalization: -1.5pp
- Effects overlap (not purely additive)
Comment removal is the main leakage vector. But even comments carry weakly valid signal -- a comment describing a vulnerability often co-occurs with the vulnerability itself.
Revised Conclusion
The original paper was right that leakage exists and should be controlled for. It was wrong about the magnitude. TF-IDF character n-grams learn substantial real vulnerability patterns from code syntax: eval usage, string concatenation, and defensive branching.
A simple logistic regression trained in 0.5 seconds achieves 81.1% accuracy on pure code structure. That is a legitimate baseline for vulnerability detection.
Code
Full ablation code and all four papers: github.com/turingrtss/vulndetect
Correcting your own published results is more interesting than getting them right the first time. The original paper had a finding. This paper has data showing that finding was partially wrong. Both are useful.
Top comments (1)
Really good follow-up - the feature rank vs feature contribution distinction is the load-bearing insight, and it is easy to miss because the highest-ranked TF-IDF features are exactly the ones that move most under cleaning. One caveat worth adding to the decomposition: stripping comments is not a pure ablation of comment signal. Deletion also shifts the input distribution (shorter docs, different n-gram co-occurrence), so part of the -2.1pp could be distribution shift rather than lost information. A cleaner control is content substitution - swap comment text for length-matched filler or shuffle comments across documents - which isolates content contribution from structural change. The point is sharper because of your own observation that comments carry weakly valid signal: removal deletes both the signal and the context the rest of the code was scored against. The 81.1 percent full-clean result reads as a solid baseline regardless, and eval( / string-concat / conditional-density surviving as the top code signals is a genuinely useful takeaway. Curious whether rank-vs-contribution divergence behaves differently in a deeper model, where individual features are not inspectable the same way.