DEV Community: Bob Taylor

A Fast Alternative to Pylint R0801

Bob Taylor — Tue, 25 Aug 2026 04:53:27 +0000

If you only need Pylint for duplicate-code detection, you don't necessarily need to run Pylint to get it.

Python tooling has changed quite a bit over the last few years.

Ruff can replace a large part of the traditional Python linting stack, and it does that work extremely quickly. For a lot of projects, adopting Ruff means there is much less reason to run several different linters over the same source tree.

There is one check I wasn't able to move out of Pylint, though: duplicate-code detection.

Pylint reports duplicated code as R0801, or duplicate-code. It comes from Pylint's similarities checker and looks for similar lines across Python source.

It's a useful check. The problem for me wasn't what R0801 did. It was having to run Pylint to get it.

Why I Still Wanted R0801

Duplicate code is easy to dismiss as a style problem until you've maintained enough of it.

Two copies of the same implementation mean two places that may need to change when the behavior changes. Three copies mean three. Eventually one gets fixed while another doesn't, and code that looked identical gradually stops behaving identically.

Pylint's own documentation makes essentially the same point: duplicated logic increases the number of places that have to be found, changed, and tested, and can make code harder to understand during review.

So turning off R0801 wasn't the solution I wanted.

I wanted to keep the check without paying for a much broader analysis pass just to get that one result.

Running Only Pylint's Duplicate-Code Check

If you already have Pylint installed, the first thing to try is simply running less Pylint.

You can enable only duplicate-code detection:

pylint --disable=all --enable=duplicate-code .

That is a perfectly reasonable solution, particularly if Pylint is already part of your project and its performance is acceptable.

There are also useful controls around what contributes to similarity. Pylint supports ignoring comments, docstrings, imports, and function signatures when constructing the source representation used for duplicate detection.

So before replacing anything, I'd start there.

If a dedicated R0801 pass is fast enough for your repository, you may already have the solution you need.

For me, it wasn't.

Why Duplicate Detection Can Become Noticeable

Duplicate-code detection is different from many ordinary lint rules.

A rule such as "this import is unused" can largely reason about a particular file or syntax tree. Duplicate detection has to compare source across a corpus because the code in one file may duplicate code somewhere else entirely.

That distinction becomes more important as a repository grows.

It also complicates the obvious answer to performance problems: just split the work across files.

Cross-file analysis needs a global view of the source being compared. There are real-world projects that have ended up separating Pylint's duplicate-code checker into its own single-process CI job because partitioning files among workers can change the detected clusters.

That doesn't mean Pylint is a bad tool. Pylint does far more than duplicate detection.

But it does raise a reasonable question if R0801 is the reason you're still running it:

Do you need a general-purpose linter for this particular job?

A Focused Alternative

That question is why I built Arid.

Arid is a Python duplicate-code checker written in Rust. It isn't intended to replace Pylint as a whole, and it isn't trying to replace Ruff.

It replaces one job:

Pylint R0801

with a tool dedicated to duplicate-code detection.

The workflow I use is simply:

ruff check .
arid .

Ruff handles the broad linting work. Arid handles duplication.

That separation is useful beyond performance. A focused tool can make its configuration, reporting, CI behavior, and machine interfaces specifically about the problem it's solving rather than accommodating an entire linting framework.

How Much Faster?

I don't think "written in Rust" is a benchmark, so I maintain a pinned performance campaign for Arid.

For Arid 2.0, I compared it with Pylint 4.0.6 while isolating Pylint's duplicate-code functionality rather than comparing Arid with an entire Pylint lint run.

Running serially, the results were:

Requests — 191.19x faster

Pydantic — 219.06x faster

Polaris — 249.68x faster

Those numbers aren't meant to establish that Pylint itself is "200x slower." That would be an unfair comparison because Pylint performs many checks Arid doesn't even attempt.

They answer a much narrower question:

If the job is duplicate-code detection, what does the focused implementation cost compared with Pylint's implementation of that job?

That's the comparison that mattered to me.

What Arid Actually Detects

There is another important qualification.

"Duplicate code" can mean several different things.

At one extreme is exact textual duplication. At the other are sophisticated clone detectors trying to recognize code that is structurally or semantically similar despite substantial differences in its source.

Arid isn't trying to solve the entire code-clone research problem.

It detects exact duplicated Python source after configurable Python-aware normalization. Depending on the configuration, comments, docstrings, imports, and function signatures can be excluded from duplicate identity.

That makes it much closer in purpose to the R0801 workflow I wanted to replace.

If you're looking for semantic clone detection—two differently written implementations that happen to do the same thing—Arid isn't the tool for that.

I think being explicit about that boundary is important. A focused tool is only useful if its focus matches the problem you actually have.

What About an Existing Codebase Full of Duplication?

This is where replacing a checker and actually adopting one become different problems.

Suppose you run duplicate detection on a mature project for the first time and discover 300 existing findings.

Technically, the tool worked.

Practically, you've just created 300 reasons for your team not to enable it in CI.

You could fix everything before enforcing the check, but that's often unrealistic. You could ignore duplicate detection entirely, but then new duplication continues accumulating.

The more useful approach is a baseline.

With Arid, existing findings can be recorded as accepted debt. CI can then reject new duplication without requiring you to eliminate everything that existed before the check was introduced.

As existing duplication gets refactored, stale baseline entries can be identified and pruned.

That changes adoption from:

fix all existing duplication
        ↓
enable the check

into:

record existing duplication
        ↓
prevent new duplication
        ↓
improve old duplication over time

For mature codebases, I think the second model is considerably more realistic.

I'll cover that workflow separately because it turns out to be a more general engineering problem than just configuring a duplicate-code checker.

Using It in CI

For a basic local check, Arid doesn't require much:

uv tool install arid
arid .

Arid 2.0 also has an official GitHub Action for CI:

- uses: sponge-b0b/arid@v2.0.0
  with:
    paths: .

V2 can produce text, JSON, Markdown, and SARIF reports, including multiple representations from a single analysis.

That matters if, for example, you want readable output for developers, structured JSON for another tool, and SARIF for code scanning. The source doesn't need to be analyzed independently for every consumer.

Again, none of that makes duplicate detection inherently better. It makes the detector easier to incorporate into the systems surrounding it.

Should You Replace Pylint R0801?

Not necessarily.

If you're already using Pylint extensively, R0801 performs well enough for your repository, and you like its behavior, replacing it solely because another implementation is faster may accomplish very little.

I'd consider a dedicated alternative when the situation looks more like this:

you've moved most linting to Ruff;
duplicate-code detection is one of the remaining reasons you're running Pylint;
R0801 has become noticeable on your repository;
you want duplicate detection in a fast local feedback loop;
you need baseline-based adoption for an existing codebase; or
you want structured duplicate-code output for CI or other tooling.

That's the situation Arid was built for.

I didn't start the project because I thought Python needed another general-purpose linter. Quite the opposite. I started it because I wanted one useful capability without carrying a general-purpose linter along just to get it.

If that's your problem too, Arid may be useful.

If it isn't, keep using R0801.

The goal isn't to replace a tool that already works for you. It's to avoid running more tooling than the problem actually requires.

Arid is an open-source Python duplicate-code checker written in Rust and designed to complement Ruff.

About the Author

Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing Arid and Polaris.

GitHub

Arid 2.0: From Fast Python Duplicate Detection to CI-Ready Tooling

Bob Taylor — Sun, 23 Aug 2026 08:43:32 +0000

Arid started because Pylint R0801 was too slow. Version 2.0 keeps the fast detector and builds the workflows around it.

A few months ago, I had a fairly simple problem: Pylint was too slow.

I use Pylint's R0801 duplicate-code detection on Polaris, a fairly large Python project I've been building. I also use Ruff, which handles most of the Python linting I care about and handles it very quickly.

Unfortunately, Ruff doesn't detect duplicate code. So every time I wanted that one check, I was back to waiting for Pylint.

Eventually I got tired of waiting and built Arid, a focused Python duplicate-code checker written in Rust. The idea was deliberately simple: do the job I was using Pylint R0801 for, do it accurately, and do it fast enough that I wouldn't mind running it all the time.

The first versions of Arid proved that idea worked. Arid 2.0 is about something different: what does a fast detector need around it before it becomes a tool you can comfortably build into real development workflows? That question ended up defining the release.

The Detector Didn't Need Reinventing

Major versions have a way of encouraging major rewrites: new architecture, new algorithm, new semantics. Everything is better because everything is new.

I didn't do that.

Arid 2.0 uses the same basic detection model as 1.2. It still detects exact duplicated Python source after configurable Python-aware normalization. It still reports DUP001. Comments, docstrings, imports, and function signatures can be excluded from duplicate identity. The detector isn't suddenly trying to find semantically equivalent code or fuzzy AST clones.

That's intentional. The detector was already solving the problem I wanted it to solve, so instead of redesigning the part that worked, I concentrated v2 on the things surrounding it: stable machine contracts, CI integration, baseline management, focused workflows, incomplete-analysis handling, project control, and better support for external tooling.

Arid 2.0 isn't really a new detector. It's the same detector with a much more useful development workflow around it.

Fast Still Matters

None of that would matter much if Arid stopped being fast.

The v2 performance campaign used the same pinned benchmark corpora and Hyperfine methodology used to qualify Arid 1.2. Against Pylint 4.0.6, running serially, Arid 2.0 measured:

Project	Arid v2 vs. Pylint
Requests	191.19x faster
Pydantic	219.06x faster
Polaris	249.68x faster

Those aren't comparisons against an entire Pylint run. The benchmark isolates Pylint's duplicate-code functionality so the comparison is actually about the job Arid replaces.

I also compared v2 directly against the qualified Arid 1.2 implementation. The additional v2 functionality introduced only low-single-digit overhead across the canonical corpora.

Could I have spent more time trying to recover that few percent? Sure. Would anybody using Arid notice the difference between roughly 219x faster than Pylint and slightly more than 219x faster than Pylint? Probably not. At some point optimization becomes an excellent way to avoid working on things users actually need.

Stable Identity for a Finding

One of those things is identity.

Suppose Arid finds the same duplicated code today and tomorrow, but somebody inserts 20 lines near the top of the file. The physical line numbers changed.

The duplicate didn't.

Or perhaps a file moves to another directory. Maybe the order of occurrences changes, or the same duplicate appears in another file. If external tooling identifies findings using locations, those findings become surprisingly unstable.

Arid 2.0 gives every finding a versioned fingerprint:

arid-finding-v1:sha256:...

That fingerprint identifies the normalized duplicate content independently of path, physical line number, occurrence ordering and multiplicity, structural metadata, output format, and worker mode. The same identity is exposed in SARIF through a versioned partial fingerprint.

This isn't particularly exciting when you're looking at a CLI report. It becomes considerably more useful when a CI system, reporting service, coding agent, or other tool needs to reason about the same finding across multiple runs.

Focus the Report, Not the Analysis

Large projects create another problem. Sometimes I don't care about every duplicate in the repository. I'm working on one package, directory, or file and want to know what's relevant to the thing I'm changing.

The obvious implementation is to scan only that path, but that's wrong for duplicate detection.

Suppose I'm working in:

src/payments/

and some code there duplicates code in:

src/customers/

If I analyze only src/payments/, I've removed half of the evidence.

Arid 2.0 therefore separates what gets analyzed from what gets reported:

arid . --focus src/payments

Arid still performs whole-corpus duplicate detection. Baseline enforcement still happens against the complete result, and only afterward does focus filtering determine which groups are reported. If a focused finding also occurs outside the focused path, those occurrences remain part of the finding.

In other words, focus changes what you ask Arid to show you without changing the corpus Arid uses to determine whether the code is duplicated. That's an important distinction for CI jobs and coding agents operating on a specific part of a larger repository.

Existing Duplicate Debt Is a Lifecycle

Baselines were already part of Arid before v2. The idea is straightforward: perhaps you're introducing duplicate-code enforcement into a mature project that already has 300 duplicate groups.

You could fix all 300 before adopting the tool.

Or don't adopt the tool.

Neither is especially compelling.

A baseline gives you a third option: accept the existing debt temporarily while preventing new duplicate debt from being introduced. Arid 2.0 extends that into an actual lifecycle.

arid . --baseline-status arid-baseline.json

can distinguish accepted duplicate debt, active/new findings, and stale baseline entries. Then:

arid . --prune-baseline arid-baseline.json

removes stale acceptance when the corresponding duplication no longer exists. It never silently accepts new debt.

That gives a team a useful progression:

existing duplication
        ↓
baseline it
        ↓
prevent new duplication
        ↓
refactor existing duplication over time
        ↓
prune stale baseline entries
        ↓
smaller baseline

You don't have to make an old codebase perfect before you're allowed to stop making it worse. I suspect that principle applies to considerably more than duplicate code.

Failure Doesn't Have to Mean "Tell Me Nothing"

Source analysis has another annoying edge case. Imagine scanning 3,000 Python files and one cannot be read, parsed, or normalized. Should the entire analysis disappear?

Sometimes yes. If you're enforcing a complete quality gate, an incomplete analysis cannot be treated as success. But that doesn't mean the useful results from the other 2,999 files need to vanish.

Arid 2.0 adds:

arid . --keep-going --json

Independent source failures are collected while valid files continue through detection. The important part is that Arid doesn't pretend partial analysis is complete analysis.

A report-v4 result explicitly says:

{
  "complete": false
}

and includes structured source errors. The process still exits with operational status 2, and incomplete reports cannot be emitted as SARIF.

The intent is simple: produce as much useful information as you safely can, but be explicit about how complete that information is. A human can inspect the partial result, and an automated consumer can make its own decision. Neither has to guess whether the analysis silently skipped something.

One Scan, Several Consumers

A CI pipeline often wants more than one representation of the same result. Maybe developers want readable console output, the build system wants JSON, GitHub code scanning wants SARIF, and the job summary wants Markdown.

The inefficient answer is to run the analyzer four times.

Arid 2.0 can instead produce multiple outputs from one in-memory report:

arid . \
  --format text \
  --report json=artifacts/arid.json \
  --report markdown=artifacts/arid.md \
  --report sarif=artifacts/arid.sarif

The source isn't reparsed four times and duplicate detection isn't repeated four times. The analysis happens once, and the result is rendered for the consumers that need it.

Obvious in retrospect? Probably. Still worth doing.

Machine-Readable Means Having a Contract

Once other software starts consuming CLI output, "it happens to be JSON" isn't enough.

Arid 2.0 introduces report schema v4 with explicit fields for things like the schema version, tool version, analysis metadata, completion state, structured errors, and finding fingerprints. The schema itself is published:

schemas/report-v4.schema.json

Arid also publishes contracts for capabilities and fatal JSON-mode operational errors. And:

arid --capabilities

allows tooling to discover deterministic build capabilities without first discovering or analyzing a project.

This is partly about ordinary CI integration, but there's another consumer I care about more now than I would have a few years ago: coding agents.

I use AI heavily in my own development workflow. An agent interacting with a tool shouldn't have to scrape human-readable console output and hope a sentence doesn't change in the next release. If we're increasingly going to have software using software on our behalf, the interfaces between those tools need to become more explicit, not less.

JSON gives us a machine-readable format. Publishing the schema tells the consumer what that format actually promises.

Arid Now Has an Official GitHub Action

Of course, the easiest integration is the one you don't have to assemble yourself.

Arid 2.0 ships an official composite GitHub Action:

- uses: sponge-b0b/arid@v2.0.0
  with:
    paths: .

The Action installs the exact Arid release associated with its tag and performs one scan. It can expose core metrics as outputs, write a job summary, and produce SARIF when configured.

Before v2, Arid could certainly be used in CI. Now there's a supported integration that makes doing it considerably simpler.

Sometimes You Want to Analyze Code That Isn't on Disk

Arid normally discovers and analyzes Python files in a project, but editors, coding agents, and other tools frequently have source that doesn't exist on disk yet—or source that differs from what's currently there.

V2 adds virtual Python source through standard input:

cat src/example.py | arid . --stdin-path src/example.py

The virtual source goes through the same Python parser and normalizer as disk-backed source. If an equivalent disk path exists, the virtual version replaces it for that scan. Otherwise it can be added to the corpus when the resolved project context permits it. Arid never writes that source to disk.

That means another tool can effectively ask, "If this were the contents of src/example.py, what duplicates would exist?" without first modifying the working tree. That's useful for editors and automation, and yes, it's particularly useful for coding agents.

Explicit When You Need It

Convention is great until automation needs certainty.

Arid still supports its existing nearest-config behavior, but v2 adds explicit control over project and configuration context:

arid . --config path/to/pyproject.toml
arid . --no-config
arid . --project-root path/to/project
arid . --show-config
arid . --list-files

For somebody running Arid manually in a normal repository, most of this can stay invisible. For CI, monorepos, editor integrations, and agents, being able to ask exactly which project and configuration are being used becomes considerably more important.

The common case can still rely on convenient defaults. The less common cases now have a way to be explicit.

Not Everything Became Public

There was one place where v2 deliberately became less extensible.

Arid is primarily a CLI application, but its Rust crate naturally exposes Rust code too. In v2, I narrowed the semver-supported Rust surface to a small crate-root application API. Implementation modules, detector internals, and reporting internals are no longer promises to downstream Rust consumers.

That decision fits something I've been thinking about a lot lately: every public interface creates an obligation. Once implementation details become supported API, changing your own internals becomes somebody else's breaking change.

If Arid were intended to be a general duplicate-detection framework, that would be a different conversation. It isn't, and not everything another developer could call needs to become something they should depend on.

What Didn't Change May Matter More

For a major release, the compatibility list is almost as important as the feature list.

Arid 2.0 preserves the things ordinary users depend on:

exact normalized duplicate semantics;
DUP001;
normal CLI invocation;
[tool.arid] configuration;
normalization behavior;
source suppression;
existing baseline-v1 files;
serial execution by default;
worker controls;
the 0 / 1 / 2 exit meanings;
pre-commit integration;
supported release platforms.

For CLI-only users who don't consume Arid's machine contracts or Rust internals, upgrading from 1.2 may require no migration work at all.

The intentional breaking changes are concentrated where a major version gives us room to make contracts cleaner: report JSON, SARIF finding identity, and the supported Rust API. That's the kind of major version I prefer: break what you have a good reason to break and leave everything else alone.

I Tried It on Real Projects

I don't want Arid's correctness story to be based entirely on unit tests and a repository containing six carefully selected Python files.

The v2 validation campaign exercised Arid against Black, Django, mypy, Rich, Unicode and space-containing paths, and combinations of the new workflow features.

For equivalent settings, canonical duplicate groups from Arid 2.0 were compared directly with qualified Arid 1.2 results across Black, Django, mypy, and Rich. No detector-semantic regression was found.

Validation also covered focus behavior, baseline-before-focus ordering, virtual-source replacement without disk mutation, controlled malformed source with --keep-going, multi-output on Django, worker determinism, and the published GitHub Action.

Performance matters, but so does knowing that the fast answer is still the right answer.

Where Arid Fits

Arid isn't trying to replace Ruff. Quite the opposite.

My normal mental model is:

ruff check .
arid .

Ruff handles the broad Python linting problem extraordinarily well. Arid handles one problem Ruff currently doesn't: duplicate code.

Because Arid is deliberately focused on that problem, I can make decisions around its detection model, reporting, baselines, and automation without turning it into another general-purpose linter. That's still the philosophy behind v2:

Cleaner contracts. Better automation. Same focused detector.

Try It

With uv:

uv tool install "arid==2.0.0"
arid .

Or with pip:

python -m pip install "arid==2.0.0"
arid .

If you're already using Ruff and want duplicate-code detection, try Arid on a real project. If you're still running Pylint primarily because you need R0801, I'm particularly interested in what you think.

I'm also interested in people integrating static-analysis tools into CI, editors, or coding-agent workflows. A large part of Arid 2.0 exists because once a command-line tool starts participating in larger development systems, speed isn't the only thing that matters anymore.

Arid started because I didn't want to wait for Pylint.

Version 2.0 is what happened after the detector became fast enough that speed stopped being the most interesting problem.

Arid is an open-source Python duplicate-code checker written in Rust. Version 2.0 is available now.

Documentation: Arid 2.0 release notes

About the Author

Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing Arid and Polaris.

GitHub

Every Abstraction Is a Bet on the Future

Bob Taylor — Wed, 19 Aug 2026 06:30:22 +0000

Your architecture shouldn't solve requirements that don't exist yet.

Software developers love abstractions.

I know. I'm one of them.

Give us a small enough problem and eventually somebody will propose an interface, a factory, a plugin architecture, dependency injection, and perhaps a message bus just in case the three functions need to communicate asynchronously someday.

You know. For flexibility.

I've been thinking about this while building Arid, a Python duplicate-code checker written in Rust.

Arid has a deliberately narrow job:

Detect duplicated Python source code quickly and accurately.

That's it.

It doesn't format Python.

It doesn't sort imports.

It doesn't type-check anything.

It doesn't detect dead code.

It doesn't scan JavaScript.

And unless something changes dramatically, it isn't going to make coffee either.

That narrow scope created an interesting architectural question:

How much architecture does a small tool actually need?

My answer has increasingly become:

Enough to make the current problem clean. No more.

That sounds like YAGNI.

It is.

But I think there's a more useful way to look at it.

Every Abstraction Is a Prediction

Suppose Arid had started with this:

Language
    ├── Python
    ├── JavaScript
    ├── TypeScript
    └── ...

Looks reasonable.

Maybe even responsible.

After all, duplicate-code detection isn't inherently a Python problem. Why couple the architecture to Python?

So we introduce something like:

LanguageFrontend
    parse()
    normalize()
    classify()

Python implements it today.

JavaScript can implement it tomorrow.

Look at us. Future-proof already.

Except there is currently no JavaScript version of Arid.

There is no TypeScript version.

There is no requirement for either one.

There isn't even a roadmap item for another language.

What exactly did the abstraction buy us?

It bought us an interface.

It bought us indirection.

It bought us a contract that future implementations now have to fit.

It bought us tests for a generic concept that currently has exactly one implementation.

And perhaps most importantly, it quietly made a prediction:

Arid will need multiple language frontends.

Maybe that's true someday.

Maybe it isn't.

Either way, we've paid for part of that future before we know whether it's coming.

That's the thing about abstractions.

They're not free flexibility. They're bets about where the software is going.

Architecture Has Carrying Costs

The cost of an abstraction isn't just the code required to create it.

Code is usually the cheap part.

The real cost is that somebody has to understand it.

Consider a hypothetical detector architecture:

DuplicateDetector
    ├── ExactDetector
    ├── StructuralDetector
    ├── SemanticDetector
    └── FuzzyDetector

Nice.

Except Arid has one detector.

It performs exact duplicate detection after configurable Python-aware normalization.

There is no structural detector.

There is no semantic detector.

There is no fuzzy detector.

In fact, those are explicitly outside Arid's current scope.

So if I introduce DuplicateDetector, what have I modeled?

Not the software that exists.

I've modeled software I can imagine.

That's a subtle but important difference.

Now every developer reading the code has additional questions:

Why is this an interface?

Are there other implementations?

Can the implementation change at runtime?

Am I expected to add new detectors this way?

What guarantees does the abstraction make?

Which behavior belongs to the interface and which belongs to the implementation?

The abstraction has increased the number of concepts required to understand the system without increasing what the system can do.

That's architectural debt too.

We just don't usually call it that because the code looks clean.

But Don't Just Put Everything in `main()`

There is an obvious bad interpretation of this argument:

Small software doesn't need architecture.

I don't believe that.

Arid 1.1.0 has an architecture.

The main application pipeline is roughly:

discover
   ↓
read
   ↓
normalize
   ↓
build corpus
   ↓
detect duplicates
   ↓
apply baseline
   ↓
build report
   ↓
render output

Those are real boundaries because they represent different responsibilities in the problem.

File discovery shouldn't know how suffix arrays work.

Duplicate detection shouldn't know how Python comments are parsed.

Normalization shouldn't know how Markdown reports are rendered.

Reporting shouldn't decide what counts as a duplicate.

Those separations aren't predictions about hypothetical future products.

They're properties of the problem Arid solves today.

That's the distinction I care about.

Good architecture separates things that are actually different. Overarchitecture separates things because they might become different someday.

Those are not the same thing.

Concrete Is Not a Dirty Word

At some point, "concrete" became suspicious in software design.

If one module directly calls another module, perhaps we're too tightly coupled.

Better introduce an interface.

But coupling isn't automatically bad.

Incorrect coupling is bad.

Arid's duplicate detector depends on Arid's corpus representation.

Of course it does.

That's the data it detects duplicates in.

The normalization layer produces Arid's normalized representation.

Again: yes.

That's its job.

Those relationships aren't architectural mistakes waiting to be abstracted away.

They're the architecture.

The question shouldn't be:

How do I eliminate coupling?

It should be:

Are these things coupled for a reason that belongs to the domain?

If the answer is yes, hiding that relationship behind another interface doesn't necessarily improve anything.

Sometimes it just makes the coupling harder to see.

One Implementation Is a Clue

I don't subscribe to a hard rule that an interface must always have multiple implementations.

There are legitimate reasons to put a boundary in front of a single implementation.

External systems are an obvious example.

Testing can be another.

A meaningful architectural seam can exist before the second implementation arrives.

But one implementation should at least make you ask a question:

What variation am I modeling?

If the answer is:

Well, someday we might...

I become suspicious.

Someday is responsible for a lot of software.

Someday we'll support another database.

Someday we'll have multiple cloud providers.

Someday this will become a distributed system.

Someday users will write plugins.

Someday we'll support seventeen programming languages.

Maybe.

When someday becomes a requirement, we can design for someday with considerably more information than we have now.

And if the current design makes that future literally impossible without rewriting the entire system, that's worth considering.

But there is an enormous amount of territory between:

Don't make the future impossible.

and:

Implement the future now.

We seem to confuse those surprisingly often.

The Plugin System Nobody Asked For

Plugin systems are one of my favorite examples.

Imagine adding plugins to Arid.

What can a plugin do?

Add a language?

Change normalization?

Replace duplicate detection?

Add output formats?

Filter findings?

Modify configuration?

Now we need a plugin API.

Then we need to decide which internal concepts are public.

Then those concepts need stability guarantees.

Then plugins need version compatibility.

Then failures need isolation.

Then documentation.

Then testing.

Then somebody writes a plugin that depends on behavior we thought was an implementation detail.

Congratulations.

Our tiny duplicate-code checker now has an ecosystem to govern.

For what requirement?

There isn't one.

A plugin architecture would not make Arid more flexible today.

It would create an obligation to remain flexible tomorrow.

That's different.

Generalization Can Make the Current Problem Worse

There's another cost to premature abstraction that bothers me more than the extra code.

It can make the abstraction less correct.

Arid's frontend is Python-aware for a reason.

A comment isn't simply "text following a comment delimiter."

A docstring isn't simply "a string."

A function signature has Python-specific syntax.

Structural context depends on Python syntax.

If I had started by demanding a language-neutral abstraction, I would have needed to decide what all programming languages have in common before I had completely solved the Python problem.

What is a Function in the generic model?

What is a Comment?

What is a Docstring in a language that doesn't have docstrings?

What is StructuralScope across Python, Rust, JavaScript, SQL, and whatever somebody asks for next?

Now we're not merely implementing duplicate detection.

We're designing a theory of programming languages.

All because somebody might want TypeScript someday.

No thanks.

Arid can understand Python correctly.

If another language becomes a real requirement later, then we can compare two concrete implementations and discover which concepts are genuinely shared.

That's usually a much better time to generalize.

The second implementation teaches you things the first one can't.

Duplication Isn't Always Worse Than the Wrong Abstraction

This is where DRY can get us into trouble.

We're trained to see duplication and eliminate it.

I'm literally building a tool that finds duplicated code, so I'm probably supposed to be careful here.

But eliminating duplication by creating the wrong abstraction can be worse than the duplication itself.

Two pieces of code can look similar today and evolve for completely different reasons tomorrow.

Combine them too early and you've coupled their futures.

The same thing happens architecturally.

We see two concepts that might eventually share behavior, so we manufacture a common parent before we understand either one.

Then reality arrives.

One implementation needs a special case.

Then another.

The abstraction starts accumulating flags.

Then optional methods.

Then configuration.

Eventually the "generic" abstraction is mostly a complicated description of the differences it was supposed to hide.

Sometimes duplication is information.

It tells you:

These things look similar.

It does not necessarily tell you:

These things are the same concept.

That's a decision we still have to make.

Extensibility Is a Feature

We often talk about extensibility as though every system should have as much of it as possible.

I don't think that's true.

Extensibility is a product capability.

Like any other capability, it has users, requirements, costs, and tradeoffs.

If third-party developers need to extend your system without modifying it, extensibility may be essential.

If your organization has five implementations behind a stable contract, abstraction may be essential.

If you're publishing a framework whose entire purpose is to support unknown use cases, flexibility may be the product.

But Arid is a CLI that finds duplicate Python code.

Its value isn't proportional to the number of ways it can be extended.

Its value comes from doing its one job correctly, quickly, and predictably.

That changes the architecture I want.

Small Doesn't Mean Crude

Architectural restraint doesn't mean throwing everything into one file until it becomes unbearable.

Look at Arid's internal model.

A prepared file owns its original source, normalized source, normalized lines, and segments.

A normalized line carries things the detector and reporting pipeline genuinely need: its range in normalized text, original source line, whether it's effective, and its structural context and scope.

A duplicate occurrence identifies a file and a normalized range.

A duplicate group contains its effective size and occurrences.

These are small types.

They exist because the domain has those concepts.

That's very different from introducing types whose primary purpose is to make the architecture look sophisticated.

The test I increasingly like is:

Can I explain why this abstraction exists without talking about a hypothetical future?

If I can say:

We need this boundary because parsing Python and detecting repeated normalized sequences are different responsibilities.

Good.

If I say:

We need this because eventually we may support arbitrary parser backends selected dynamically from third-party plugins...

I'm going to need considerably more evidence.

What Happens When Requirements Change?

The obvious objection is:

Isn't this shortsighted? What happens when Arid needs another language?

Then I change the architecture.

Seriously.

Architecture is not a one-time ceremony performed before implementation begins.

It's the structure of a living system.

If Arid someday has a legitimate requirement to support Rust source, I'll have something incredibly valuable that I don't have today:

a second real language implementation.

Then I can look at Python and Rust and ask:

What is actually common?

What must vary?

Where does the boundary belong?

Which assumptions in the Python implementation were language-specific?

Which concepts really are universal?

The abstraction designed from those answers is likely to be better than the one I invent today while staring at a single Python implementation.

Will refactoring cost something?

Of course.

So does maintaining an unnecessary abstraction for three years waiting for a requirement that never arrives.

Architecture is tradeoffs.

There is no option where we pay nothing.

You Can Always Add Code Later

This sounds ridiculously obvious, but software development sometimes behaves as if there's a code shortage.

There isn't.

If a requirement appears later, we're allowed to write more code.

If a second implementation appears, we're allowed to extract an interface.

If users need plugins, we're allowed to design a plugin model.

If the pipeline needs asynchronous execution, we're allowed to introduce it.

If Arid needs another language, we're allowed to refactor the frontend.

We don't receive bonus points for having predicted every requirement five years early.

In fact, predictions made too early can make the real requirement harder to implement because now it has to fit the imaginary one.

The goal isn't to avoid changing the architecture.

The goal is to make the architecture easy to change when reality gives us a reason to change it.

Those are very different objectives.

Build the Architecture You Can Defend

Arid 1.1.0 isn't architecturally small because I don't care about architecture.

It's small because I do.

Its pipeline has boundaries.

Its Python-specific parsing is isolated from duplicate detection.

Its internal representation carries the information downstream stages actually need.

Detection doesn't decide presentation.

Structural metadata describes findings without changing duplicate identity.

Those are architectural decisions.

But there is no generic language framework.

There is no detector hierarchy.

There is no plugin system.

There is no dependency-injection framework.

There is no async runtime.

Not because those things are bad.

Because Arid doesn't currently have problems that they solve.

That's the standard I want to apply more often:

Don't ask whether an abstraction could be useful. Ask what requirement makes it necessary.

If you can't name one, maybe don't build it yet.

Your small tool doesn't need a framework.

It needs an architecture that makes the problem it actually solves obvious.

Build that.

When the problem changes, change the architecture.

Arid is an open-source Python duplicate-code checker written in Rust. This article discusses the architecture of Arid 1.1.0. Its intentionally small application pipeline can be seen in lib.rs, and its core domain representation is in model.rs.

About the Author

Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing Arid and Polaris.

GitHub

Detecting Duplicate Python Code Is Harder Than Comparing Text

Bob Taylor — Mon, 17 Aug 2026 16:59:54 +0000

Detecting Duplicate Python Code Is Harder Than Comparing Text

The hard part isn't finding repeated lines. It's deciding what "the same code" actually means.

Duplicate-code detection sounds easy.

You have some source files. Find sequences of lines that occur more than once.

How hard could it be?

I asked essentially that question when I started building Arid, a duplicate-code checker for Python.

The answer, as it often is in software, was: it depends on what you mean.

What exactly is a duplicate?

Consider these two functions:

def save_customer(value):
    # Persist the customer
    result = serialize(value)
    database.save(result)

def save_account(value):
    # Store the account
    result = serialize(value)
    database.save(result)

As text, they're different.

As executable logic, they're the same.

If I change the comment again, is it suddenly different code?

What about the function name?

What about a docstring?

Imports?

Blank lines?

Decorators?

Formatting?

At some point duplicate-code detection stops being a string-comparison problem and becomes a language problem.

That was one of the first lessons I learned building Arid.

First, Define "Duplicate"

There are a lot of ways two pieces of code can be similar.

These are obviously identical:

value = calculate()
save(value)

value = calculate()
save(value)

Now change the variable name:

value = calculate()
save(value)

result = calculate()
save(result)

Are those duplicates?

A human can look at them and reasonably say yes.

Arid says no.

That's deliberate.

Arid 1.1.0 detects exact duplicate source after configured normalization. It is not trying to determine whether two pieces of code are semantically equivalent, structurally similar, or suspiciously alike.

Changing value to result changes the code being compared.

Changing a literal from 10 to 20 changes it.

Changing an expression changes it.

That's an important boundary because "find code that means roughly the same thing" is a very different problem from "find source that has actually been duplicated."

The first problem starts taking you toward AST similarity, clone classification, semantic analysis, and eventually a fairly interesting discussion about what "equivalent" even means.

I wasn't trying to solve that problem.

I wanted a fast replacement for the duplicate-code functionality I was using in Pylint.

Pylint's similarity checker already has useful concepts for this. By default, it can exclude comments, docstrings, imports, and function signatures from similarity calculation. Those options are part of Pylint's similarity checker.

Arid keeps that general idea.

The interesting part is figuring out how to do it correctly.

A `#` Is Not Necessarily a Comment

Let's start with comments.

This looks easy:

value = 42  # this is a comment

Remove everything after #.

Done.

Until this shows up:

value = "# this is not a comment"

Now our sophisticated duplicate-code checker has helpfully converted valid Python source into:

value = "

Excellent.

You can keep adding increasingly clever text-processing rules, or you can ask Python what the thing actually is.

Arid takes the second approach.

Its Python frontend tokenizes the source and identifies tokens whose kind is actually Comment. The comment's source range can then be removed from the representation used for duplicate matching.

So:

value = "# this is not a comment"
other = 42  # actual comment

normalizes to:

value = "# this is not a comment"
other = 42

when comments are ignored.

The distinction seems obvious when you see the example.

But that's the point.

The distinction is obvious because you understand Python syntax.

A text processor doesn't.

A String Is Not Necessarily a Docstring

Docstrings get more interesting.

Consider:

def calculate():
    """Calculate the current value."""
    value = 42
    return value

If docstrings are configured to be ignored, we don't want that string to participate in duplicate identity.

Now consider:

def calculate():
    value = 42
    """This is an ordinary string expression."""
    return value

Should that string disappear too?

No.

They're both string literals.

Only one is a docstring.

The difference isn't the quotes. The difference is where that expression exists in the Python program.

Arid identifies structural docstrings as string-expression statements in the docstring position of a module, class, or function body.

So this:

"""module documentation"""

class Customer:
    """class documentation"""

    def save(self):
        """method documentation"""
        persist()

can have all three docstrings excluded from matching.

But this:

def save():
    persist()
    """ordinary string expression"""
    finish()

keeps the string expression.

You can't make that distinction reliably by looking for triple quotes.

You have to understand the syntax tree.

And now our simple "compare some lines" project has a parser.

That escalated quickly.

Function Signatures Are Worse Than They Look

Ignoring function signatures sounds simple too.

Remove the def line:

def calculate(value):

Except Python doesn't require a function declaration to fit on one line.

It can look like this:

def calculate(
    value: dict[str, tuple[int, int]],
    multiplier: int = 2,
) -> list[int]:
    return transform(value, multiplier)

Or:

async def calculate(
    value: dict[str, tuple[int, int]],
) -> list[int]:
    return await transform(value)

There can be nested brackets, type annotations, default values, and plenty of colons inside those expressions before you reach the colon that actually terminates the function signature.

So "remove everything through the first colon" isn't going to last very long.

Arid walks the parser tokens beginning at def or async, tracks bracket nesting, and finds the colon that terminates the declaration at nesting level zero.

Then it masks that source range.

The body remains.

There's another detail I like here.

Decorators remain significant.

Given:

@transactional
def save(value):
    persist(value)

and:

@cached
def save(value):
    persist(value)

ignoring function signatures doesn't silently erase the decorators.

Arid removes the function declaration.

It doesn't remove everything vaguely associated with the function.

That distinction is easier to maintain when syntax tells you where the boundaries actually are.

Then There Are Imports

Imports have their own collection of small traps.

This:

import os

is easy.

This:

from package import (
    first,
    second,
    third,
)

takes several physical lines.

Imports can also occur inside control flow or functions:

if enabled:
    import optional_backend
    run()

If imports are ignored, Arid removes the import statement while retaining:

if enabled:
    run()

Then somebody writes:

import os; run()

If you remove only the AST range belonging to import os, you're left with:

; run()

Wonderful.

Or:

run(); import os

becomes:

run();

So Arid's frontend deliberately consumes the appropriate adjacent semicolon when an ignored statement shares a logical line with retained code.

The result in both cases is:

run()

It's a tiny implementation detail.

It's also exactly the kind of tiny implementation detail that separates "works on my example" from "works on Python source."

Normalization Is Not Parsing

At this point it would be easy to let the parser take over the entire design.

I didn't want that either.

Arid uses Python syntax to answer questions that require knowledge of Python:

Is this actually a comment?
Is this string expression actually a docstring?
What source range belongs to this import?
Where does this function signature end?
Is this code associated with a module, class, or function?

Then that knowledge crosses a boundary.

The Python frontend converts parser-specific information into ordinary source ranges and structural regions owned by Arid.

The normalization layer doesn't operate on AST nodes.

The duplicate detector certainly doesn't.

The basic relationship is:

Python source
    ↓
parser + tokenizer
    ↓
source masks + structural regions
    ↓
normalized lines
    ↓
duplicate detector

I think that distinction matters.

Python syntax determines what participates in comparison. It does not perform the comparison.

The parser is there because text alone can't reliably tell me which text should be ignored.

Once that question has been answered, the rest of Arid doesn't need to know which parser answered it.

The Source You Compare Is Not the Source You Report

Normalization creates another problem.

Suppose the original code is:

def calculate(
    first: int,
    second: int,
) -> int:
    # Add the values
    result = first + second

    return result

With comments and signatures ignored, the meaningful normalized representation might effectively be:

result = first + second
return result

That's useful for matching.

It's terrible for reporting if you lose the relationship to the original file.

A developer doesn't want a diagnostic that says:

Duplicate found at normalized line 2.

Normalized line 2 does not exist in the file they're editing.

Arid therefore keeps the original physical source-line location on every normalized line.

The detector works with the normalized representation.

The report maps the result back to the original Python source.

That means ignored comments, docstrings, signatures, imports, and blank lines can disappear from duplicate identity without making the diagnostic point somewhere imaginary.

This sounds like bookkeeping.

It is bookkeeping.

Bookkeeping is architecture when getting it wrong makes the product useless.

Four Lines Isn't Always Four Lines

There's another deceptively small question:

What does --min-lines 4 mean?

Four physical lines?

Four normalized lines?

Four lines containing actual code?

Consider:

values = [
    first,
    second,
]

The closing bracket is a normalized physical line.

But should ] count as one of the four meaningful lines required to declare a duplicate?

Arid distinguishes a normalized line from an effective line.

A line is effective when it contains at least one alphanumeric character or underscore.

So punctuation-only lines can remain part of a repeated source sequence without artificially helping that sequence satisfy the configured minimum duplicate size.

Blank lines don't count either.

That means a reported four-line duplicate means four effective normalized lines satisfied the threshold, even if the corresponding physical source range spans more lines.

Again, this isn't difficult because the algorithm is exotic.

It's difficult because apparently simple words like line turn out to require definitions.

Python-Aware Does Not Mean Semantic

This is probably the most important boundary in Arid's normalization model.

Using a Python parser does not mean Arid performs semantic duplicate detection.

These two blocks:

value = calculate()
save(value)

result = calculate()
save(result)

are not duplicates to Arid.

Nor are:

if ready:
    execute()

and:

if is_ready:
    execute()

They may represent the same pattern.

They may deserve refactoring.

They may even have been created by copy and paste followed by renaming a variable.

Arid still says they're different.

Why?

Because the parser is being used to interpret Python syntax accurately, not to erase meaningful Python source until everything vaguely similar starts matching everything else.

There's a line somewhere between useful normalization and inventing equivalence.

As of Arid 1.1.0, that line sits in a fairly conservative place.

Comments, structural docstrings, imports, and function signatures can be configured out.

What remains must match exactly.

Could Arid eventually detect renamed-variable clones, structural clones, or fuzzy AST similarity?

Sure.

It could also become an IDE, package manager, database, and small accounting system.

The question isn't whether those things can be built.

The question is whether they're the problem Arid is supposed to solve.

For now, they're not.

Description Is Not Identity

Arid does use the Python structure for one other purpose: describing what it found.

A duplicate can be reported as declarative or executable and associated with module, class, or function scope.

For example, repeated class-level assignments might be described differently from repeated executable logic inside functions.

But that metadata doesn't change duplicate identity.

Two blocks don't become equal because they're both executable.

They don't stop being equal because one occurs in a different structural context.

First Arid answers:

Is this source duplicated?

Then it can help answer:

What kind of source did I find?

I deliberately keep those questions separate.

The moment classification starts deciding whether something "really counts" as duplication, the tool starts making domain judgments it doesn't have enough information to make.

Repeated declarative code might be framework boilerplate.

It might be accidental duplication.

It might be exactly what the project wants.

Arid doesn't know.

Neither does the AST.

The Algorithm Wasn't the First Hard Part

Arid ultimately uses a generalized suffix array and longest-common-prefix analysis to find repeated normalized sequences.

That's the algorithmically interesting part of the project, and I'll get into it separately.

But something surprised me while building the tool.

Before you can efficiently find repeated sequences, you have to decide what the sequence is.

That decision contains a surprising amount of the product's behavior.

Do comments matter?

Which strings are docstrings?

Do imports matter?

What is a function signature?

What happens to decorators?

What does a line mean?

How do normalized lines map back to physical source?

Does structural context affect equality?

How much difference is allowed before code stops being "the same"?

Those aren't suffix-array questions.

They're product-definition questions.

And they're language questions.

The duplicate detector can only be as correct as the representation you give it.

Feed it bad normalization quickly and all you've built is a very fast way to produce bad answers.

The Parser Isn't the Point

I started Arid because Pylint's duplicate-code checker was too slow for my workflow.

I expected performance to be the interesting problem.

And performance certainly mattered.

But building the tool reinforced something I keep running into in software architecture:

Before optimizing the solution, define the problem precisely.

"Find duplicate code" isn't precise enough.

"Find exact repeated Python source after removing a configurable set of syntactically understood constructs" is considerably closer.

Once that definition existed, a lot of architectural decisions became easier.

Use Python syntax where Python syntax matters.

Turn that knowledge into a small internal representation.

Keep parser details out of the detector.

Preserve the mapping back to the source developers actually edit.

And don't quietly turn exact duplicate detection into semantic similarity because the parser happens to make that possible.

It turns out duplicate-code detection is harder than comparing text.

Not because comparing text is hard.

Because deciding which text means what is.

Arid is an open-source Python duplicate-code checker written in Rust. This article describes the normalization model in Arid 1.1.0. The implementation is available in normalize.rs, and the Python syntax frontend is in python.rs.

About the Author

Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing Arid and Polaris.

GitHub

Pylint Was Too Slow, So I Built Arid

Bob Taylor — Sun, 16 Aug 2026 15:11:47 +0000

I didn't set out to write a duplicate-code checker.

I was working on Polaris, a fairly large Python project, and doing what I normally do: running code-quality tools against it. Ruff handles most of what I want from a Python linter these days, and it handles it very quickly.

There was one problem.

Ruff doesn't detect duplicated blocks of code.

That isn't an oversight in my configuration. Ruff simply doesn't support project-wide duplicate-code detection today.

Pylint does.

Its R0801 checker has been finding similar code for years, and Pylint also exposes the same capability through its standalone symilar tool. It can ignore comments, docstrings, imports, and function signatures while looking for repeated blocks.

There was just one small problem.

It was slow.

On Polaris, it was really slow.

Eventually I got tired of waiting for it.

So I built Arid.

How Hard Could It Be?

Those may be some of the most dangerous words in software development.

At first glance, duplicate-code detection doesn't sound particularly complicated. Read some files, compare some lines, find the parts that repeat, print them out.

Done.

Except we aren't comparing text files. We're analyzing Python source code.

Suppose these two functions contain the same executable logic:

def save_customer(value):
    # Persist the current customer
    result = serialize(value)
    database.save(result)

def save_account(value):
    # Store the account
    result = serialize(value)
    database.save(result)

Are they duplicates?

If comments are ignored and function signatures are ignored, they probably should be.

Now add docstrings. Imports. Blank lines. Decorators. Multiline function signatures. Parenthesized expressions. Different physical source ranges. Same-file duplicates that overlap one another.

Suddenly "compare some lines" needs a little more definition.

Pylint already has an answer to many of these questions, and I wasn't trying to invent a completely different meaning for duplicate code. Arid started with the intent of Pylint's R0801: find repeated Python source while allowing things such as comments, docstrings, imports, and signatures to be excluded from the comparison.

But I didn't need to reproduce Pylint's implementation.

That distinction turned out to matter.

One Job

Once I decided to build the tool, I had one significant advantage over Pylint.

Arid only needed to do one thing.

Pylint is a general-purpose static-analysis system. Duplicate-code detection is one capability among many.

Arid has exactly one production responsibility:

Find duplicated Python source code.

That's it.

No formatting. No import sorting. No type checking. No complexity analysis. No security scanner. No dead-code detector.

Ruff already does a lot of those things exceptionally well. I have no interest in building a slower, less capable Ruff just so Arid can have a longer feature list.

In fact, one of the rules I settled on for Arid is fairly simple:

If a feature naturally belongs in Ruff, it probably doesn't belong in Arid.

That decision had architectural consequences.

Arid supports one programming language, so it doesn't have a generic language abstraction.

It has one detector, so it doesn't have a detector hierarchy.

It doesn't have a plugin system.

It doesn't have a reporter registry.

It doesn't have a dependency-injection framework.

It doesn't have an async runtime.

And I did not create a parser abstraction just in case I wake up one morning three years from now and decide to replace the parser.

I realize this may be shocking.

The duplicate-code checker somehow manages to function anyway.

Those choices are explicit in Arid's technical architecture. Parser-specific knowledge stays in the Python frontend, while everything downstream operates on Arid-owned data structures rather than parser AST or token types.

That isn't an argument against abstraction.

It's an argument for paying for abstraction when you actually have a requirement for it.

Every abstraction has a cost. More types. More indirection. More concepts somebody has to understand. More extension points that have to remain stable. More opportunities to design for a future that never arrives.

If Arid eventually develops requirements that justify one of those abstractions, then I'll have a reason to build it.

Until then, I have a duplicate-code checker to write.

Small Scope Does Not Mean a Trivial Problem

Keeping the product small didn't make the underlying problem simple.

Arid still has to understand enough Python syntax to distinguish source that participates in duplicate identity from source that may be ignored.

The basic pipeline became:

Python source
    ↓
file discovery
    ↓
Python parse + tokenize
    ↓
Python-aware filtering + structural classification
    ↓
normalized source lines
    ↓
global exact-duplicate index
    ↓
maximal repeated blocks
    ↓
duplicate findings + metrics

The Python frontend uses syntax to identify comments, docstrings, imports, function declarations, structural scope, and their source ranges. It converts that parser-specific information into Arid's internal representation.

After that boundary, the duplicate-detection engine doesn't know what a Python AST node is.

That was deliberate.

The parser is a dependency.

Python is the domain.

Those aren't the same thing.

I want Arid to understand Python, but I don't want the entire application architecture to understand the implementation details of whichever parser happens to provide that information.

There's another important distinction in the design: structural information describes a duplicate, but it does not determine whether two blocks are duplicates.

Arid can tell you that repeated code is executable logic inside a function or declarative code associated with a class. That can be useful when deciding what deserves attention.

But "this looks like framework boilerplate" is a judgment.

So is "this duplicate is harmless."

So is "you need to refactor this."

Arid doesn't pretend to know your application's intent. Its job is to detect the duplication accurately and give you enough objective information to make your own decision.

Exact Means Exact

I also didn't want duplicate findings based solely on the assumption that two hashes being equal means two pieces of source code are equal.

Hashes can be useful internally.

They are not proof.

Arid v1 defines duplication as exact equality after the configured Python-aware normalization. Hashing can be an implementation technique, but equality eventually has to resolve to actual equality.

The global detection engine uses a generalized suffix array with longest-common-prefix analysis to identify repeated normalized source sequences. Arid then turns those candidates into maximal duplicate groups while handling things like overlapping occurrences and deterministic canonicalization.

That probably deserves an article of its own.

The important point here is that once duplicate detection became the entire problem instead of one feature inside a larger tool, I could choose an architecture specifically for that problem.

Fast Was Still the Point

All of this architectural discussion can make the project sound much more philosophical than it actually was.

Let's not rewrite history.

I built Arid because I was tired of waiting for Pylint.

So eventually I had to answer the obvious question:

Is it actually faster?

For Arid 1.0, I built a reproducible benchmark suite around three real Python repositories at fixed revisions: Requests, Pydantic, and Polaris.

The benchmark isolates Pylint's duplicate-code checker, pins tool versions and repository revisions, records environment metadata, uses repeated Hyperfine measurements, and distinguishes comparisons with approximately equivalent semantics from comparisons against tools whose clone-detection semantics differ.

Against Pylint 4.0.6, the published Arid 1.0 measurements were:

Project	Python files	Arid vs. Pylint
Requests	37	196.79× faster
Pydantic	404	192.88× faster
Polaris	1,452	264.45× faster

On the pinned Polaris benchmark, Arid completed the duplicate scan in about 442 milliseconds.

Pylint took about 117 seconds.

That's the difference between a check I don't mind running and one that interrupts my workflow.

But benchmark numbers need context.

Pylint is not standing still.

Its upcoming 4.1 release includes significant work on the duplicate-code checker: reuse of an already-parsed AST when running inside Pylint, rolling hashes with caching, and changes intended to avoid quadratic behavior in problematic inputs. The Pylint project reports improvements ranging from roughly 1.5× on small projects to 20× on large ones, along with lower memory usage.

Good.

I hope it gets even faster.

Arid doesn't need Pylint to be bad in order for Arid to be useful.

And those benchmark numbers are measurements of particular versions of two pieces of software on particular corpora.

They are not a law of physics.

If Arid's entire identity were "Pylint 4.0.6 is slow," the project would have a fairly short shelf life.

The original motivation was performance.

The resulting tool has a broader reason to exist: focused, Python-aware duplicate-code detection that can sit next to Ruff without requiring a general-purpose linter for that one remaining job.

Performance is still the reason I started.

It just isn't the only engineering property I care about now.

Yes, I Built It With AI

There's another part of the story worth being explicit about.

I built Arid with ChatGPT as an AI coding partner.

The same is true of Polaris.

I use AI extensively in my development workflow. It helps generate code, examine designs, write tests, review changes, reason about problems, and accelerate implementation.

I'm not particularly interested in pretending otherwise.

What I have found, though, is that generating code and engineering software are not the same activity.

The difficult parts don't disappear because an LLM can produce Rust.

You still have to decide what the product is supposed to do.

You have to define the invariants.

You have to recognize a bad abstraction when one appears.

You have to decide whether a result is actually correct.

You have to build validation that doesn't merely prove the implementation agrees with itself.

You have to benchmark honestly.

You have to reject unnecessary code.

And sooner or later, you have to decide that the thing is stable enough to put 1.0.0 on it.

AI changes how I write software.

It does not remove the need to engineer it.

That experience probably deserves an article too.

The Tool I Wanted to Use

Arid began with a very unremarkable engineering problem.

I had a useful tool.

One part of it was too slow for the way I wanted to work.

The tool I preferred to use for the rest of my Python linting didn't provide that capability.

So I wrote the missing piece.

I didn't need another Python linter.

I needed duplicate-code detection that was fast enough that I wouldn't think twice about running it.

That constraint eventually led to a Rust implementation, Python-aware normalization, exact matching, deterministic output, a suffix-array-based detector, a reproducible benchmark suite, and a deliberately small architecture.

But none of those things were the original idea.

The original idea was much simpler:

I was tired of waiting.

Arid is open source and available on GitHub. If duplicate-code detection is part of your Python workflow, give it a try. Feedback, bug reports, and contributions are welcome.

About the Author

Bob Taylor is a software engineer and architect who builds developer tools and AI systems. He is currently developing Arid, a fast Python duplicate-code checker written in Rust, and Polaris, an AI-assisted portfolio intelligence platform.

GitHub

DEV Community: Bob Taylor

A Fast Alternative to Pylint R0801

Why I Still Wanted R0801

Running Only Pylint's Duplicate-Code Check

Why Duplicate Detection Can Become Noticeable

A Focused Alternative

How Much Faster?

What Arid Actually Detects

What About an Existing Codebase Full of Duplication?

Using It in CI

Should You Replace Pylint R0801?

About the Author

Arid 2.0: From Fast Python Duplicate Detection to CI-Ready Tooling

The Detector Didn't Need Reinventing

Fast Still Matters

Stable Identity for a Finding

Focus the Report, Not the Analysis

Existing Duplicate Debt Is a Lifecycle

Failure Doesn't Have to Mean "Tell Me Nothing"

One Scan, Several Consumers

Machine-Readable Means Having a Contract

Arid Now Has an Official GitHub Action

Sometimes You Want to Analyze Code That Isn't on Disk

Explicit When You Need It

Not Everything Became Public

What Didn't Change May Matter More

I Tried It on Real Projects

Where Arid Fits

Try It

About the Author

Every Abstraction Is a Bet on the Future

Every Abstraction Is a Prediction

Architecture Has Carrying Costs

But Don't Just Put Everything in main()

Concrete Is Not a Dirty Word

One Implementation Is a Clue

The Plugin System Nobody Asked For

Generalization Can Make the Current Problem Worse

Duplication Isn't Always Worse Than the Wrong Abstraction

Extensibility Is a Feature

Small Doesn't Mean Crude

What Happens When Requirements Change?

You Can Always Add Code Later

Build the Architecture You Can Defend

About the Author

Detecting Duplicate Python Code Is Harder Than Comparing Text

Detecting Duplicate Python Code Is Harder Than Comparing Text

First, Define "Duplicate"

A # Is Not Necessarily a Comment

A String Is Not Necessarily a Docstring

Function Signatures Are Worse Than They Look

Then There Are Imports

Normalization Is Not Parsing

The Source You Compare Is Not the Source You Report

Four Lines Isn't Always Four Lines

Python-Aware Does Not Mean Semantic

Description Is Not Identity

The Algorithm Wasn't the First Hard Part

The Parser Isn't the Point

About the Author

Pylint Was Too Slow, So I Built Arid

How Hard Could It Be?

One Job

Small Scope Does Not Mean a Trivial Problem

Exact Means Exact

Fast Was Still the Point

Yes, I Built It With AI

The Tool I Wanted to Use

About the Author

But Don't Just Put Everything in `main()`

A `#` Is Not Necessarily a Comment