Engineering decisions behind dsweep — a three-stage detection pipeline, reversible quarantine, and why I keep reaching for .NET when I want a fast CLI.
The obvious approach, and why it's wrong
The naive implementation of a duplicate-file scanner goes like this: walk the directory tree, hash every file with SHA-256, group files by hash, done.
It works. It's also brutally expensive. If you're scanning 50,000 files and most of them are different, you've computed full SHA-256 hashes of every single one — reading every byte of every file — to find a few hundred matches.
The interesting engineering problem isn't "how do I hash files." It's "how do I avoid hashing files I don't need to."
This post walks through the design of dsweep, a .NET 10 duplicate-file scanner with a three-stage detection funnel and a reversible quarantine system. I'll focus on the decisions I found genuinely interesting rather than a feature walkthrough.
Stage 1: File size — zero hashing, pure filesystem metadata
The cheapest possible check: two files with different sizes cannot be identical. No hashing, no reading — just a number from the filesystem stat.
List<FileEntry> bySize = entries
.GroupBy(e => e.Length)
.Where(g => g.Count() > 1)
.SelectMany(g => g)
.ToList();
This one LINQ expression eliminates the majority of files in a typical scan. A 3 KB config file and a 14 MB video file will never match, and after this stage they never compete for hash time again.
The pattern: group, filter groups of 1, flatten back to a list. Only files that share a size with at least one other file move to the next stage.
Stage 2: Quick hash — first 64 KB only
For files that share a size, we sample the first 64 KB and compute a SHA-256 of that sample. The sample size is deliberate:
private const int QuickHashSampleBytes = 64 * 1024;
public static string ComputeQuickHash(string path, long fileLength, CancellationToken ct)
{
using SHA256 sha256 = SHA256.Create();
using FileStream stream = File.OpenRead(path);
int sampleSize = (int)Math.Min(fileLength, QuickHashSampleBytes);
if (sampleSize == 0)
return Convert.ToHexString(sha256.ComputeHash([]));
byte[] buffer = new byte[sampleSize];
int read = ReadFully(stream, buffer, ct);
return Convert.ToHexString(sha256.ComputeHash(buffer, 0, read));
}
Why 64 KB? It's a pragmatic calibration. Large files (videos, disk images, archives) that differ will almost always differ within the first 64 KB — the content starts diverging quickly. The quick hash is cheap enough that a false positive (two different files that happen to share the same first 64 KB) doesn't hurt much: it just means both files go to stage 3 for the full hash. The only cost of a false positive is an unnecessary full-file read. The benefit of a true negative is avoiding that full-file read for every file that doesn't need it.
For a 4 GB video file, this is the difference between reading 64 KB and reading 4 GB. For 1,000 video files where 998 are unique, that's roughly 63.9 GB of I/O saved.
Stage 3: Full SHA-256 — only on confirmed quick-hash collisions
Files that share both size and quick hash are the real candidates. Only these get the full-file SHA-256 treatment:
public static async Task<string> ComputeFullHashAsync(string path, CancellationToken ct)
{
await using FileStream stream = File.OpenRead(path);
byte[] hash = await SHA256.HashDataAsync(stream, ct).ConfigureAwait(false);
return Convert.ToHexString(hash);
}
A few things worth noticing:
SHA256.HashDataAsync — this is the modern static one-liner available since .NET 5. The old pattern (using var sha = SHA256.Create(); sha.ComputeHashAsync(stream)) is still valid but requires managing the disposable SHA256 instance. The static version is cleaner and does the same thing.
Convert.ToHexString — available since .NET 5, replaces the classic BitConverter.ToString(hash).Replace("-", "").ToLower() pattern that I still see in a lot of code. It returns uppercase hex but that's consistent throughout the codebase.
await using — the async dispose pattern. FileStream implements IAsyncDisposable so the stream gets closed asynchronously when we're done, which matters when you're running many of these concurrently.
The full pipeline in 40 lines
The three stages compose cleanly in DuplicateFinder.FindGroupsAsync:
// Stage 1: group by size — zero I/O
List<FileEntry> bySize = entries
.GroupBy(e => e.Length)
.Where(g => g.Count() > 1)
.SelectMany(g => g)
.ToList();
// Stage 2: quick hash (64 KB sample)
(Dictionary<string, List<FileEntry>> byQuickHash, List<string> quickWarnings) =
await GroupByAsync(bySize, e => Task.FromResult(Hashing.ComputeQuickHash(e.FullPath, e.Length, ct)), ct);
List<FileEntry> candidates = byQuickHash.Values
.Where(g => g.Count > 1)
.SelectMany(g => g)
.ToList();
// Stage 3: full hash — only on confirmed quick-hash collisions
(Dictionary<string, List<FileEntry>> byFullHash, List<string> fullWarnings) =
await GroupByAsync(candidates, e => Hashing.ComputeFullHashAsync(e.FullPath, ct), ct);
The repeated shape — group by key, filter groups of 1, flatten — is the core abstraction. GroupByAsync handles the parallel execution, error capture, and dictionary construction for both the quick and full hash stages.
Parallel execution with bounded concurrency
Both hashing stages use Parallel.ForEachAsync with a configurable parallelism limit:
await Parallel.ForEachAsync(
Enumerable.Range(0, entries.Count),
new ParallelOptions { MaxDegreeOfParallelism = options.Parallelism, CancellationToken = ct },
async (i, token) =>
{
try
{
keys[i] = await keySelector(entries[i]).ConfigureAwait(false);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
warnings.Add($"skipped unreadable file: {entries[i].FullPath} ({ex.Message})");
}
});
A few design decisions here:
Parallel.ForEachAsync (not Task.WhenAll) — Task.WhenAll with a large file list creates all tasks immediately, which can overwhelm the I/O subsystem and the thread pool. Parallel.ForEachAsync runs at most MaxDegreeOfParallelism tasks concurrently, keeping the I/O queue manageable. On an SSD this matters less; on an HDD the difference is large (random reads kill rotational disk performance).
Indexed output array, not a concurrent dictionary — each worker writes to keys[i] at its own index. Since no two workers share an index, there's no race condition, no lock, and no concurrent collection overhead. The dictionary construction happens in a single-threaded loop afterward. This is the "results by position" pattern: safer and faster than ConcurrentDictionary for this use case.
Exception filter, not catch-all — only IOException and UnauthorizedAccessException are caught. These are the expected failures (file deleted during scan, permission denied). Any other exception propagates up and fails the scan, because unexpected exceptions deserve to be seen.
ConcurrentBag<string> for warnings — the warning bag needs to be written from multiple threads; ConcurrentBag handles that without locks. Warnings are surfaced to the user at the end ("skipped 3 unreadable files") rather than crashing the scan.
The quarantine model: delete is the wrong default
Most "remove duplicates" tools delete. dsweep doesn't — at least not directly.
The quarantine model:
- Create a
.dsweep-quarantine/directory - Move duplicates into it, preserving filenames (with a suffix counter for filename collisions within the quarantine folder)
- Write a JSON manifest recording the original path, quarantine path, file size, and hash for each moved file
-
dsweep restorereads the manifest and moves everything back
public static IReadOnlyList<ManifestEntry> Quarantine(
IReadOnlyList<DuplicateResolution> resolutions, string quarantineDir)
{
Directory.CreateDirectory(quarantineDir);
var manifest = new List<ManifestEntry>();
for (int groupIndex = 0; groupIndex < resolutions.Count; groupIndex++)
{
DuplicateResolution resolution = resolutions[groupIndex];
string groupDir = Path.Combine(quarantineDir, groupIndex.ToString());
Directory.CreateDirectory(groupDir);
foreach (FileEntry duplicate in resolution.Duplicates)
{
string destination = UniqueDestination(groupDir, Path.GetFileName(duplicate.FullPath));
File.Move(duplicate.FullPath, destination);
manifest.Add(new ManifestEntry(duplicate.FullPath, destination, duplicate.Length, resolution.Group.Hash));
}
}
return manifest;
}
Why not delete? The main failure mode of any duplicate finder isn't missing duplicates — it's false positives. A file that appears identical to another isn't always safe to delete. Two files that share a hash are identical in content, but they might have different metadata you care about (creation time, permissions) or they might be in locations that have semantic significance ("original" vs "backup"). Move is reversible; delete is not.
The --dry-run flag on the restore command (RestoreService.Restore(manifest, dryRun: true)) lets you see what would be moved before committing. The manifest also serves as an audit trail: every file that was quarantined, when, and where it went.
Keep strategy: deterministic tiebreaking
For each duplicate group, one file survives — the "keeper." dsweep supports four strategies:
public static FileEntry SelectKeeper(IReadOnlyList<FileEntry> files, KeepStrategy strategy) => strategy switch
{
KeepStrategy.First => files[0],
KeepStrategy.Oldest => files.OrderBy(f => f.LastWriteTimeUtc)
.ThenBy(f => f.FullPath, StringComparer.Ordinal).First(),
KeepStrategy.Newest => files.OrderByDescending(f => f.LastWriteTimeUtc)
.ThenBy(f => f.FullPath, StringComparer.Ordinal).First(),
KeepStrategy.ShortestPath => files.OrderBy(f => f.FullPath.Length)
.ThenBy(f => f.FullPath, StringComparer.Ordinal).First(),
_ => throw new ArgumentOutOfRangeException(nameof(strategy), strategy, "unknown keep strategy"),
};
The .ThenBy(f => f.FullPath, StringComparer.Ordinal) secondary sort exists to ensure determinism. If two files share the same LastWriteTimeUtc (common with copied files) or the same path length, the selection would otherwise depend on enumeration order — which is filesystem-dependent and not stable across runs. The ordinal path sort makes the selection reproducible: given the same inputs, the same file is always the keeper.
Determinism matters here because a user might run the tool, review the quarantine, and then run it again. Getting a different answer on the second run without any file changes would be disorienting.
Why .NET for a CLI in 2025
The honest answer: the tooling has gotten good enough that it's no longer a disadvantage.
Native AOT — dotnet publish -c Release -r win-x64 --self-contained produces a single-file binary with a typical startup time under 10ms and no runtime install requirement. The published binary is ~8MB for dsweep. That's a long way from the "requires .NET Framework 4.7.2 to be installed" era.
Parallel.ForEachAsync — the I/O-bound parallel pattern I described above exists natively in .NET since .NET 6. No third-party concurrency library needed.
The standard library breadth — SHA256.HashDataAsync, Path.GetRelativePath, JsonSerializer, Convert.ToHexString, File.OpenRead returning an IAsyncDisposable stream — all the pieces you need for a file tool are there and well-designed.
Pattern matching and switch expressions — the KeepStrategy switch above is representative. The combination of discriminated unions (via sealed classes + switch), exhaustiveness checking, and expression-bodied members produces code that's almost F#-level terse while staying in the C# ecosystem.
The thing I'd caution about: if your CLI needs a rich TUI (terminal UI), Rust has a better story (Ratatui, indicatif). If your CLI does heavy text processing, Go's startup profile and goroutine model can be a better fit. For a CPU-and-I/O intensive data processing CLI where you want cross-platform binary distribution and you know the .NET type system, it's a solid choice.
Results
On a test run over a 180 GB media archive (82,000 files):
- Stage 1 eliminated 71,000 files from hashing entirely
- Stage 2 quick-hashed ~11,000 files, eliminated ~8,200 from full hashing
- Stage 3 full-hashed ~2,800 files
- Found 340 duplicate groups (18 GB reclaimable)
- Wall-clock time: 23 seconds with
--parallelism 8on an NVMe SSD
The majority of the savings came from stage 1 (size filter). Stage 2 contributed meaningfully for files in common sizes (600 KB JPEG thumbnails, for example, where many files share a size but differ in content after the first 64 KB).
Source
amasen02/dupesweep — MIT licensed. C#, .NET 10, ~600 lines.
The design applies beyond duplicate finding: any "compare by content" problem benefits from a cheap-first funnel. Dedup systems in databases (MinHash, LSH), spell checkers (edit distance gating on length difference first), image similarity search (perceptual hash before deep feature comparison) all use the same shape. Cheap gate first, expensive confirmation only for survivors.
I'm a full-stack engineer building .NET microservices and TypeScript frontends. Most of what I write about comes from real implementation work rather than tutorial derivation.
Open-source: a11y-scope — a free, self-hosted WCAG 2.2 accessibility monitor.
Top comments (0)