Pylint Was Too Slow, So I Built Arid

I didn't set out to write a duplicate-code checker.

I was working on Polaris, a fairly large Python project, and doing what I normally do: running code-quality tools against it. Ruff handles most of what I want from a Python linter these days, and it handles it very quickly.

There was one problem.

Ruff doesn't detect duplicated blocks of code.

That isn't an oversight in my configuration. Ruff simply doesn't support project-wide duplicate-code detection today.

Pylint does.

Its R0801 checker has been finding similar code for years, and Pylint also exposes the same capability through its standalone symilar tool. It can ignore comments, docstrings, imports, and function signatures while looking for repeated blocks.

There was just one small problem.

It was slow.

On Polaris, it was really slow.

Eventually I got tired of waiting for it.

So I built Arid.

How Hard Could It Be?

Those may be some of the most dangerous words in software development.

At first glance, duplicate-code detection doesn't sound particularly complicated. Read some files, compare some lines, find the parts that repeat, print them out.

Done.

Except we aren't comparing text files. We're analyzing Python source code.

Suppose these two functions contain the same executable logic:

def save_customer(value):
    # Persist the current customer
    result = serialize(value)
    database.save(result)

def save_account(value):
    # Store the account
    result = serialize(value)
    database.save(result)

Are they duplicates?

If comments are ignored and function signatures are ignored, they probably should be.

Now add docstrings. Imports. Blank lines. Decorators. Multiline function signatures. Parenthesized expressions. Different physical source ranges. Same-file duplicates that overlap one another.

Suddenly "compare some lines" needs a little more definition.

Pylint already has an answer to many of these questions, and I wasn't trying to invent a completely different meaning for duplicate code. Arid started with the intent of Pylint's R0801: find repeated Python source while allowing things such as comments, docstrings, imports, and signatures to be excluded from the comparison.

But I didn't need to reproduce Pylint's implementation.

That distinction turned out to matter.

One Job

Once I decided to build the tool, I had one significant advantage over Pylint.

Arid only needed to do one thing.

Pylint is a general-purpose static-analysis system. Duplicate-code detection is one capability among many.

Arid has exactly one production responsibility:

Find duplicated Python source code.

That's it.

No formatting. No import sorting. No type checking. No complexity analysis. No security scanner. No dead-code detector.

Ruff already does a lot of those things exceptionally well. I have no interest in building a slower, less capable Ruff just so Arid can have a longer feature list.

In fact, one of the rules I settled on for Arid is fairly simple:

If a feature naturally belongs in Ruff, it probably doesn't belong in Arid.

That decision had architectural consequences.

Arid supports one programming language, so it doesn't have a generic language abstraction.

It has one detector, so it doesn't have a detector hierarchy.

It doesn't have a plugin system.

It doesn't have a reporter registry.

It doesn't have a dependency-injection framework.

It doesn't have an async runtime.

And I did not create a parser abstraction just in case I wake up one morning three years from now and decide to replace the parser.

I realize this may be shocking.

The duplicate-code checker somehow manages to function anyway.

Those choices are explicit in Arid's technical architecture. Parser-specific knowledge stays in the Python frontend, while everything downstream operates on Arid-owned data structures rather than parser AST or token types.

That isn't an argument against abstraction.

It's an argument for paying for abstraction when you actually have a requirement for it.

Every abstraction has a cost. More types. More indirection. More concepts somebody has to understand. More extension points that have to remain stable. More opportunities to design for a future that never arrives.

If Arid eventually develops requirements that justify one of those abstractions, then I'll have a reason to build it.

Until then, I have a duplicate-code checker to write.

Small Scope Does Not Mean a Trivial Problem

Keeping the product small didn't make the underlying problem simple.

Arid still has to understand enough Python syntax to distinguish source that participates in duplicate identity from source that may be ignored.

The basic pipeline became:

Python source
    ↓
file discovery
    ↓
Python parse + tokenize
    ↓
Python-aware filtering + structural classification
    ↓
normalized source lines
    ↓
global exact-duplicate index
    ↓
maximal repeated blocks
    ↓
duplicate findings + metrics

The Python frontend uses syntax to identify comments, docstrings, imports, function declarations, structural scope, and their source ranges. It converts that parser-specific information into Arid's internal representation.

After that boundary, the duplicate-detection engine doesn't know what a Python AST node is.

That was deliberate.

The parser is a dependency.

Python is the domain.

Those aren't the same thing.

I want Arid to understand Python, but I don't want the entire application architecture to understand the implementation details of whichever parser happens to provide that information.

There's another important distinction in the design: structural information describes a duplicate, but it does not determine whether two blocks are duplicates.

Arid can tell you that repeated code is executable logic inside a function or declarative code associated with a class. That can be useful when deciding what deserves attention.

But "this looks like framework boilerplate" is a judgment.

So is "this duplicate is harmless."

So is "you need to refactor this."

Arid doesn't pretend to know your application's intent. Its job is to detect the duplication accurately and give you enough objective information to make your own decision.

Exact Means Exact

I also didn't want duplicate findings based solely on the assumption that two hashes being equal means two pieces of source code are equal.

Hashes can be useful internally.

They are not proof.

Arid v1 defines duplication as exact equality after the configured Python-aware normalization. Hashing can be an implementation technique, but equality eventually has to resolve to actual equality.

The global detection engine uses a generalized suffix array with longest-common-prefix analysis to identify repeated normalized source sequences. Arid then turns those candidates into maximal duplicate groups while handling things like overlapping occurrences and deterministic canonicalization.

That probably deserves an article of its own.

The important point here is that once duplicate detection became the entire problem instead of one feature inside a larger tool, I could choose an architecture specifically for that problem.

Fast Was Still the Point

All of this architectural discussion can make the project sound much more philosophical than it actually was.

Let's not rewrite history.

I built Arid because I was tired of waiting for Pylint.

So eventually I had to answer the obvious question:

Is it actually faster?

For Arid 1.0, I built a reproducible benchmark suite around three real Python repositories at fixed revisions: Requests, Pydantic, and Polaris.

The benchmark isolates Pylint's duplicate-code checker, pins tool versions and repository revisions, records environment metadata, uses repeated Hyperfine measurements, and distinguishes comparisons with approximately equivalent semantics from comparisons against tools whose clone-detection semantics differ.

Against Pylint 4.0.6, the published Arid 1.0 measurements were:

Project	Python files	Arid vs. Pylint
Requests	37	196.79× faster
Pydantic	404	192.88× faster
Polaris	1,452	264.45× faster

On the pinned Polaris benchmark, Arid completed the duplicate scan in about 442 milliseconds.

Pylint took about 117 seconds.

That's the difference between a check I don't mind running and one that interrupts my workflow.

But benchmark numbers need context.

Pylint is not standing still.

Its upcoming 4.1 release includes significant work on the duplicate-code checker: reuse of an already-parsed AST when running inside Pylint, rolling hashes with caching, and changes intended to avoid quadratic behavior in problematic inputs. The Pylint project reports improvements ranging from roughly 1.5× on small projects to 20× on large ones, along with lower memory usage.

Good.

I hope it gets even faster.

Arid doesn't need Pylint to be bad in order for Arid to be useful.

And those benchmark numbers are measurements of particular versions of two pieces of software on particular corpora.

They are not a law of physics.

If Arid's entire identity were "Pylint 4.0.6 is slow," the project would have a fairly short shelf life.

The original motivation was performance.

The resulting tool has a broader reason to exist: focused, Python-aware duplicate-code detection that can sit next to Ruff without requiring a general-purpose linter for that one remaining job.

Performance is still the reason I started.

It just isn't the only engineering property I care about now.

Yes, I Built It With AI

There's another part of the story worth being explicit about.

I built Arid with ChatGPT as an AI coding partner.

The same is true of Polaris.

I use AI extensively in my development workflow. It helps generate code, examine designs, write tests, review changes, reason about problems, and accelerate implementation.

I'm not particularly interested in pretending otherwise.

What I have found, though, is that generating code and engineering software are not the same activity.

The difficult parts don't disappear because an LLM can produce Rust.

You still have to decide what the product is supposed to do.

You have to define the invariants.

You have to recognize a bad abstraction when one appears.

You have to decide whether a result is actually correct.

You have to build validation that doesn't merely prove the implementation agrees with itself.

You have to benchmark honestly.

You have to reject unnecessary code.

And sooner or later, you have to decide that the thing is stable enough to put 1.0.0 on it.

AI changes how I write software.

It does not remove the need to engineer it.

That experience probably deserves an article too.

The Tool I Wanted to Use

Arid began with a very unremarkable engineering problem.

I had a useful tool.

One part of it was too slow for the way I wanted to work.

The tool I preferred to use for the rest of my Python linting didn't provide that capability.

So I wrote the missing piece.

I didn't need another Python linter.

I needed duplicate-code detection that was fast enough that I wouldn't think twice about running it.

That constraint eventually led to a Rust implementation, Python-aware normalization, exact matching, deterministic output, a suffix-array-based detector, a reproducible benchmark suite, and a deliberately small architecture.

But none of those things were the original idea.

The original idea was much simpler:

I was tired of waiting.

---

Arid is open source and available on GitHub. If duplicate-code detection is part of your Python workflow, give it a try. Feedback, bug reports, and contributions are welcome.

About the Author

Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing Arid, a fast Python duplicate-code checker written in Rust, and Polaris, an AI-assisted portfolio intelligence platform.

GitHub