This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
What I Built
Somewhere in your files sit contract.pdf, contract_final.pdf, contract_final_v2(1).pdf, and contract_FINAL (copy).pdf β the same document, four times over. You know one of them is the real one. You don't know which one. So you open all four, squint at the footer, find the signed copy, close three tabs β or worse, you pick wrong and sign the wrong version.
Keeper answers that question for you. Point it at a folder: it clusters version-clone files by actual content (not filename), asks a local open-weight model which copy is the keeper and why, and quarantines the rest. Dry-run by default. It never deletes a thing.
That's the whole sloth move β a task you've been putting off for years, done by one command while you get a coffee.
The model suggests; you keep the final say. All of this is in v0.1.6, released Friday:
-
--interactiveβ Keeper walks each cluster in turn and asks you. Press Enter to accept the verdict, type a number to keep a different copy,sto leave that cluster alone,qto stop. It stays a dry run the whole way β nothing moves without--apply. -
--only 1,3,5/--skip 2β act on a subset of clusters (ranges work too:2-4). Numbering is stable across runs, so--only 7still reads as cluster 7 in the table, the JSON and the Markdown report. -
Targeting β
--ext pdf,jpg,--min-size 10k,--include 'rental*',--exclude 'reports/*'. The filters run during the walk, so a file you excluded is never hashed and never read. -
--max-hash-mb 512β the byte-identical pass used to stop at 64 MB, which quietly hid the big files: a re-downloaded installer, two copies of the same video. The default is now 512 MB;0means no limit. -
--quietβ nothing but errors, for cron and CI. -
--fail-on-clutterβ exits3when there's work to do and0when the folder is tidy, so a script can tell "findings" apart from "it broke".
If you disagree with it
Real output from a small test folder β the model and the rules both preferred report_v2.txt, I didn't:
$ keeper client-docs --interactive
Cluster 1 of 4 β 3 copies
1. [loser] report.txt (73 B)
2. [loser] report_final.txt (92 B)
3. [keep] report_v2.txt (109 B)
why: newest signed copy [rule, confidence 0.60]
[Enter] accept Β· [number] keep that copy Β· [s] skip Β· [q] stop > 1
keeper is now report.txt
Cluster 2 of 4 β 2 copies
1. [loser] data.bin (4.0 KB)
2. [keep] data.bin.bak (4.0 KB)
why: all copies are byte-identical; kept the newest timestamp [identical, confidence 1.00]
[Enter] accept Β· [number] keep that copy Β· [s] skip Β· [q] stop > q
Review stopped β 2 cluster(s) after this one will not be touched.
DRY RUN β nothing was moved (2 loser(s) would be quarantined Β· 201 B of duplicate clutter).
Run again with --apply to quarantine them, or --restore to undo.
That row comes back in the table as β you β you chose copy 1. You were looking at the actual files, size and name in front of you, instead of a confidence score β and the override still went through the same dry run, manifest and --restore as every other verdict. Quit early and the clusters you never reviewed are left alone, not swept up.
Demo
In my case it takes time β I'm on a low-end device, and local LLMs are heavy. Sorry for the bad quality!
A full run on my test folder: scan β spinner with a self-measured ETA β verdict table β --apply quarantines the 8 losers. Nothing was deleted.
Code
atif-1402
/
keeper
Which file do I keep? Finds version-clone files in a folder and asks a local open-weight model (Ollama + Gemma) which copy wins β and why. Dry-run by default, never deletes, fully reversible.
Finds version-clone files in any folder, asks a local open-weight model which copy to keep, and quarantines the rest β dry-run by default, 100% offline
Demo Β· Install Β· How it works Β· Safety model Β· CLI Β· Why open weights
Demo
In my case it takes time β I'm on a low-end device, and local LLMs are heavy. Sorry for the bad quality!
Why this exists
Duplicate finders tell you files are identical. Diff tools make you pick the two files to compare by hand. Neither one answers the question you actually have: "I have five versions of the same document β which one do I keep?"
Keeper walks the folder itself, clusters the versions, and gives you a colored verdict table with a one-line reason per cluster. While it works you get a live spinner β one line that updates itself (hashing files β extracting text ββ¦
curl -fsSL https://raw.githubusercontent.com/atif-1402/keeper/main/install.sh | bash
ollama pull gemma3:1b # ~800 MB, runs on 4 GB RAM
keeper ~/Documents # dry run β moves nothing
Windows runs it today via pipx install git+https://github.com/atif-1402/keeper; release v0.1.6 ships standalone binaries for Linux (x64/arm64), macOS and Windows. No Ollama? Keeper falls back to its rule engine and tells you so.
- Repo: github.com/atif-1402/keeper
- Release: v0.1.6
- This challenge: Slow and Steady on the HF26 hub
How I Built It
gemma3:1b through local Ollama β an ~800 MB open-weight model that runs in 4 GB RAM. No API keys, no cloud, no ML framework in the codebase: grouping is TF-IDF cosine similarity over filename tokens and extracted text, image matching is a 64-bit perceptual hash, and everything is pure Python.
folder ββΆ scan (hash + metadata) ββΆ extract (txt/md/csv/pdf/docx)
ββΆ group (filename tokens + TF-IDF cosine) ββΆ decide ββΆ report
β
`--apply` ββΆ _keeper_quarantine/ + manifest
-
Scan β walks the folder (skipping
.git,node_modules, and its own quarantine), records size, mtime, sha256. - Extract β pulls text from documents; images, video and archives are recognized as media up front so their bytes are never decoded into garbage "text". A scanned PDF with no text layer is honestly flagged instead of treated as empty.
-
Group β byte-identical files cluster by hash; near-duplicates cluster by filename-token overlap (browser
notes (1).mdsuffixes stripped) and content similarity β and since v0.1.5, visually near-identical images cluster by perceptual hash: a screenshot saved asphoto.pngand re-exported asphoto.jpgis the same picture with different bytes, and a 64-bit difference hash sees it. Names carrying different identity numbers (S01E01vsS01E02,Lesson13vsLesson14) are never merged β episodes and numbered lessons stay apart. -
Decide β only clusters with readable text ever reach the model. Byte-identical clusters and unreadable ones (zips, images, videos) are settled instantly from metadata, so a low-end machine runs Gemma only when it actually matters. Each call is temperature 0, capped output, with a server-side JSON schema β the model structurally cannot answer in any other shape than
{keep, reason, confidence}. A verdict it isn't confident about gets one richer second look (more copies, longer excerpts); still unsure, Keeper says "the model was unsure after two looks" and the rules decide. An agent that knows when it doesn't know. -
Quarantine β
--applymoves losers into_keeper_quarantine/<group>/with amanifest.jsonand a portablerestore.sh.
The safety net
If the model's pick contradicts hard evidence β the signed copy, the newest timestamp β a transparent rule engine overrules it. Look at row 3 in the demo: the model disagreed on the contract, and the newest copy won on metadata, with the override stated plainly in the table (β rule β model disagreed). I'd rather ship a tool that's right than one that's confidently wrong. Rule modes are yours too: --no-model (rules only, instant) and --model-only (hard-error if Ollama is down).
And the three guarantees everything else rests on:
-
Dry-run by default β nothing moves without
--apply, including after an--interactivereview. -
Never deletes β losers are moved into
_keeper_quarantine/<group>/; the manifest records origin, size and SHA-256 for every one. -
Fully reversible β
keeper <folder> --restorereturns the folder to its exact prior state, verified byte-for-byte by tests (95 of them).
Why Does Open Innovation Matter?
The files Keeper reads are contracts, passports, medical letters, tax returns. Shipping filenames and text excerpts to a closed cloud model is exactly the wrong default for this job. gemma3:1b (~800 MB, runs in 4 GB RAM) reads your lease agreement on your laptop and forgets it when the process exits β no API keys, no telemetry, no network calls beyond localhost:11434.
That's what an open-weight model buys you: a private assistant with no exfiltration surface and no vendor who can change the terms next quarter. And the whole reasoning pipeline β grouping, prompting, fallback rules β is in one repo, auditable in an afternoon.
The Lazy Payoff
Honest numbers from my own machine β a 4-core low-end box:
- A real 211-file folder of mine β 35 duplicate clusters, 92 losing copies identified, in about a minute. Only 9 clusters called the model; everything else was decided instantly from metadata.
- Doing that by hand means opening 92 files, comparing them, deciding. That's an afternoon I'm never getting back. Keeper: one command.
- Every run ends with the weight:
8 loser(s) would be quarantined Β· 37.3 KB of duplicate clutterβ one number that says how much you've been carrying. - On a fast machine the model calls take seconds; on mine the spinner honestly shows minutes (
[3/9] Β· about 40s left) β measured throughput, no fake progress bar.
Prize Categories
Best Use of Gemma β featured category.
gemma3:1b is Keeper's default model, run through local Ollama. Every cluster the hash and the metadata can't settle goes to Gemma, which returns {keep, reason, confidence} through a server-side JSON schema and writes the one-line explanation you read in the verdict table β the "newest signed copy" style reasons are its words, not a template. --model gemma3:4b swaps in the larger Gemma for a second opinion. No cloud model is involved at any point.
Thanks for reading β point it at the folder you keep meaning to sort, and enjoy International Sloth Day.
Disclosure: this article was created with the help of AI.


Top comments (0)