If you only need Pylint for duplicate-code detection, you don't necessarily need to run Pylint to get it.
Python tooling has changed quite a bit over the last few years.
Ruff can replace a large part of the traditional Python linting stack, and it does that work extremely quickly. For a lot of projects, adopting Ruff means there is much less reason to run several different linters over the same source tree.
There is one check I wasn't able to move out of Pylint, though: duplicate-code detection.
Pylint reports duplicated code as R0801, or duplicate-code. It comes from Pylint's similarities checker and looks for similar lines across Python source.
It's a useful check. The problem for me wasn't what R0801 did. It was having to run Pylint to get it.
Why I Still Wanted R0801
Duplicate code is easy to dismiss as a style problem until you've maintained enough of it.
Two copies of the same implementation mean two places that may need to change when the behavior changes. Three copies mean three. Eventually one gets fixed while another doesn't, and code that looked identical gradually stops behaving identically.
Pylint's own documentation makes essentially the same point: duplicated logic increases the number of places that have to be found, changed, and tested, and can make code harder to understand during review.
So turning off R0801 wasn't the solution I wanted.
I wanted to keep the check without paying for a much broader analysis pass just to get that one result.
Running Only Pylint's Duplicate-Code Check
If you already have Pylint installed, the first thing to try is simply running less Pylint.
You can enable only duplicate-code detection:
pylint --disable=all --enable=duplicate-code .
That is a perfectly reasonable solution, particularly if Pylint is already part of your project and its performance is acceptable.
There are also useful controls around what contributes to similarity. Pylint supports ignoring comments, docstrings, imports, and function signatures when constructing the source representation used for duplicate detection.
So before replacing anything, I'd start there.
If a dedicated R0801 pass is fast enough for your repository, you may already have the solution you need.
For me, it wasn't.
Why Duplicate Detection Can Become Noticeable
Duplicate-code detection is different from many ordinary lint rules.
A rule such as "this import is unused" can largely reason about a particular file or syntax tree. Duplicate detection has to compare source across a corpus because the code in one file may duplicate code somewhere else entirely.
That distinction becomes more important as a repository grows.
It also complicates the obvious answer to performance problems: just split the work across files.
Cross-file analysis needs a global view of the source being compared. There are real-world projects that have ended up separating Pylint's duplicate-code checker into its own single-process CI job because partitioning files among workers can change the detected clusters.
That doesn't mean Pylint is a bad tool. Pylint does far more than duplicate detection.
But it does raise a reasonable question if R0801 is the reason you're still running it:
Do you need a general-purpose linter for this particular job?
A Focused Alternative
That question is why I built Arid.
Arid is a Python duplicate-code checker written in Rust. It isn't intended to replace Pylint as a whole, and it isn't trying to replace Ruff.
It replaces one job:
Pylint R0801
with a tool dedicated to duplicate-code detection.
The workflow I use is simply:
ruff check .
arid .
Ruff handles the broad linting work. Arid handles duplication.
That separation is useful beyond performance. A focused tool can make its configuration, reporting, CI behavior, and machine interfaces specifically about the problem it's solving rather than accommodating an entire linting framework.
How Much Faster?
I don't think "written in Rust" is a benchmark, so I maintain a pinned performance campaign for Arid.
For Arid 2.0, I compared it with Pylint 4.0.6 while isolating Pylint's duplicate-code functionality rather than comparing Arid with an entire Pylint lint run.
Running serially, the results were:
Requests — 191.19x faster
Pydantic — 219.06x faster
Polaris — 249.68x faster
Those numbers aren't meant to establish that Pylint itself is "200x slower." That would be an unfair comparison because Pylint performs many checks Arid doesn't even attempt.
They answer a much narrower question:
If the job is duplicate-code detection, what does the focused implementation cost compared with Pylint's implementation of that job?
That's the comparison that mattered to me.
What Arid Actually Detects
There is another important qualification.
"Duplicate code" can mean several different things.
At one extreme is exact textual duplication. At the other are sophisticated clone detectors trying to recognize code that is structurally or semantically similar despite substantial differences in its source.
Arid isn't trying to solve the entire code-clone research problem.
It detects exact duplicated Python source after configurable Python-aware normalization. Depending on the configuration, comments, docstrings, imports, and function signatures can be excluded from duplicate identity.
That makes it much closer in purpose to the R0801 workflow I wanted to replace.
If you're looking for semantic clone detection—two differently written implementations that happen to do the same thing—Arid isn't the tool for that.
I think being explicit about that boundary is important. A focused tool is only useful if its focus matches the problem you actually have.
What About an Existing Codebase Full of Duplication?
This is where replacing a checker and actually adopting one become different problems.
Suppose you run duplicate detection on a mature project for the first time and discover 300 existing findings.
Technically, the tool worked.
Practically, you've just created 300 reasons for your team not to enable it in CI.
You could fix everything before enforcing the check, but that's often unrealistic. You could ignore duplicate detection entirely, but then new duplication continues accumulating.
The more useful approach is a baseline.
With Arid, existing findings can be recorded as accepted debt. CI can then reject new duplication without requiring you to eliminate everything that existed before the check was introduced.
As existing duplication gets refactored, stale baseline entries can be identified and pruned.
That changes adoption from:
fix all existing duplication
↓
enable the check
into:
record existing duplication
↓
prevent new duplication
↓
improve old duplication over time
For mature codebases, I think the second model is considerably more realistic.
I'll cover that workflow separately because it turns out to be a more general engineering problem than just configuring a duplicate-code checker.
Using It in CI
For a basic local check, Arid doesn't require much:
uv tool install arid
arid .
Arid 2.0 also has an official GitHub Action for CI:
- uses: sponge-b0b/arid@v2.0.0
with:
paths: .
V2 can produce text, JSON, Markdown, and SARIF reports, including multiple representations from a single analysis.
That matters if, for example, you want readable output for developers, structured JSON for another tool, and SARIF for code scanning. The source doesn't need to be analyzed independently for every consumer.
Again, none of that makes duplicate detection inherently better. It makes the detector easier to incorporate into the systems surrounding it.
Should You Replace Pylint R0801?
Not necessarily.
If you're already using Pylint extensively, R0801 performs well enough for your repository, and you like its behavior, replacing it solely because another implementation is faster may accomplish very little.
I'd consider a dedicated alternative when the situation looks more like this:
- you've moved most linting to Ruff;
- duplicate-code detection is one of the remaining reasons you're running Pylint;
- R0801 has become noticeable on your repository;
- you want duplicate detection in a fast local feedback loop;
- you need baseline-based adoption for an existing codebase; or
- you want structured duplicate-code output for CI or other tooling.
That's the situation Arid was built for.
I didn't start the project because I thought Python needed another general-purpose linter. Quite the opposite. I started it because I wanted one useful capability without carrying a general-purpose linter along just to get it.
If that's your problem too, Arid may be useful.
If it isn't, keep using R0801.
The goal isn't to replace a tool that already works for you. It's to avoid running more tooling than the problem actually requires.
Arid is an open-source Python duplicate-code checker written in Rust and designed to complement Ruff.
About the Author
Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing Arid and Polaris.
Top comments (0)