When I look for an old note or a screenshot, I often remember what it was about but not a single word in it. ripgrep is instant if you remember the word. That doesn't help when I remember the topic but not the wording, like "the meeting note where we decided to cut the budget" or "that dialog saying the disk was full".
So I built anygrep, a CLI that puts notes, PDF pages and images into one local index and searches them by meaning. It runs one model, Google's EmbeddingGemma 2, as an int8 ONNX file on the CPU. No PyTorch, no GPU.
To try it: pipx install anygrep, then anygrep "your query" ~/notes. The rest of this post is how it works and where it went wrong.
Text, PDFs and images in one list
EmbeddingGemma 2 gives text and images 768-dimensional embeddings in the same space. anygrep stores paragraphs, PDF pages and pictures in one index, compares each with the query using cosine similarity, and shows them in one ranked list.
These are from the synthetic demo corpus in the repo (12 notes, 3 PDFs, 8 images), with anygrep 0.1.0. Snippets are cut to the terminal width.
$ anygrep 'running out of disk space'
1 0.767 screenshots/error-dialog.png [image]
$ anygrep 'パン生地の発酵時間'
1 0.713 notes/sourdough.md:1
# Sourdough bread, the version that finally worked Starter: feed 1:1:1...
$ anygrep 'how many vacation days do I get'
1 0.762 docs/employee-handbook.pdf p.3 [pdf]
Vacation and leave New employees receive 25 paid vacation days per c...
The first query finds a screenshot of a "Disk full" dialog without any OCR. The second is Japanese ("fermentation time for bread dough") and finds an English note. The third finds the right page of a PDF.
It works the other way too. An English query, a Japanese note in second place:
$ anygrep "when is the tax return due" -m 3
1 0.732 notes/finance/tax-checklist.md:1
# Tax return checklist - Salary statement from the employer - Receip...
2 0.691 notes/finance/kakutei-shinkoku.md:1
...16日から3月15日まで。e-Taxならスマホとマイナンバーカードで提出できる...
int8 ONNX on a CPU
Before writing the tool I benchmarked three ways to run the model on a CPU: PyTorch fp32 through sentence-transformers, the int8 ONNX export from onnx-community, and the q4 ONNX export. That run was on 4 threads of a Ryzen 7 7700 while another heavy job was using the machine, so I recorded the CPU time each run actually got and estimated the idle numbers from it.
- int8 was about 2x faster than fp32 for text and about 1.4x faster for images.
- q4 was faster for a single short query (38 ms vs 97 ms for int8), but slower than int8 on documents and images, and its image embeddings drifted further from fp32.
I used int8 for both text and images.
The model was ONNX, but resizing and cutting images into patches still went through the transformers processor, which pulls in transformers and PyTorch. That is not a light pipx install. I ported it to numpy: resize (keeping the aspect ratio) so the number of patches fits the chosen number of image tokens (more on that below), normalize to [0, 1], cut into 16x16 patches, attach (x, y) position IDs. Compared with the original processor, the embeddings agree to a cosine similarity of 0.99995 or better.
Installed with pipx, anygrep takes about 213 MB. The model files (about 540 MB: 346 MB for text, 195 MB for images) are downloaded on first use.
One thing that bit me in the port was special tokens. If a note happened to contain the literal string <|image|> or <eos>, the tokenizer treated it as a control token. <|image|> with no image features was an ONNX Runtime error, and <eos> broke the embedding. The tokenizer now runs with encode_special_tokens = True, so text in a file is always just text.
Speed
These numbers are for anygrep itself, on the same Ryzen 7 7700 (8 cores, 16 threads) under WSL2, with the default 8 threads and nothing else running. For images, the image token budget controls how much visual detail the model processes; I explain the choice below.
| What | Result |
|---|---|
| Text | about 1,100 tokens/s |
| Image, 140-token budget (default) | about 1.2 s per image |
| One search command | about 1.5 s, mostly loading the model; the search itself takes tens of milliseconds on the demo corpus |
| Demo corpus (12 notes, 3 PDFs, 8 images) | about 13 s to index |
Re-running index with nothing changed |
0.2 s (demo corpus) |
| Memory | about 1.3 GB peak while indexing, about 0.5 GB for a text search |
Images are resized to fit the token budget before they reach the encoder, so the time per image depends little on the original image size. In these measurements, reading and resizing were about 5% of the time.
If you search a lot in a row, anygrep -Q is an interactive mode that keeps the model loaded.
Estimated from those per-item speeds, 1,000 notes of about 300 tokens each take around 5 minutes. 10,000 images take around 3.5 hours at the default 140 tokens (about 2 hours at the faster 70-token setting described below).
So text is fine on a CPU, and images are not fast. About 80% of the time per image is the vision encoder, so there was not much to win in my own code. I tried batching several same-size images through the encoder. It was not measurably faster and needed up to twice the memory at 280 tokens, so images go one at a time.
More threads help up to a point: at the default 140 tokens an image takes 1.52 s with 4 threads, 1.21 s with 8 and 1.00 s with 12, but 1.11 s with 16. I left the default at 8 so the machine stays usable while it indexes.
Picking the image token budget
EmbeddingGemma 2 lets you choose how many tokens an image becomes. A smaller budget is faster but keeps less visual detail. To choose a default I made a harder synthetic set: 50 images of shapes and 28 px text drawn with Pillow, where you have to see small details to get the answer. Top-1 accuracy below is the share of text queries whose top-ranked image was the right one.
| Budget | Seconds per image (8 threads) | Top-1 text-to-image accuracy |
|---|---|---|
| 70 | about 0.7 | 0.78 |
| 140 | about 1.2 | 0.90 |
| 280 | about 2.6 | 0.88 |
The difference between 140 and 280 was one result out of 50, too small to justify twice the time. 70 drops visibly on small objects and text. So 140 is the default, and --image-tokens 70|140|280 is there for a fast first pass or for screenshots full of small text. These are drawn shapes and text, not real photos; I have not measured the budgets on a real photo collection.
Embeddings of the same image at different budgets have a cosine similarity of only 0.85 to 0.96, so mixing budgets in one index makes scores incomparable. The budget is stored per image, and changing it re-embeds the images that don't match.
Where pure embedding search failed
Exact strings
Some things have no meaning to match: an invoice number, an error code, a part number. Embeddings are bad at these. So anygrep also looks up any text query of 3 or more characters as a plain string, in the file names and in the indexed text chunks (the short pieces each file is split into), using SQLite's FTS5 index with the trigram tokenizer. Files that contain it come first, marked (exact), whatever their score. anygrep INV-20391 finds the one note that mentions it.
That created the opposite problem: the is in almost every file. So if a query is only plain words and appears in many of the searched files, anygrep skips the exact match and searches by meaning only. A query with a digit or a symbol, like an ID, always gets the exact match, however many files contain it.
Something always comes first
With cosine similarity, every query has a top result, even "asdf". A single cutoff failed for two reasons. The score range depends on the pair: text queries score lower against images than against text, and images score higher against other images, so one threshold either hides most images or lets through many unrelated text results. And short or meaningless queries ("asdf", "hello", "...") score a little higher against everything, so their best results pass a fixed minimum.
My first rule was "a margin above the median score of the indexed chunks". On the corpus I tuned it on, it looked fine. Then I built held-out corpora that I had not used for tuning, all synthetic: a mixed folder, 10 recipes, 8 sourdough notes plus 2 unrelated ones, and 6 Japanese notes on one topic. The rule broke on the folder that is mostly about one topic. There, the median is the score of the right answers, so "sourdough bread" showed only 2 of the 8 sourdough notes.
The current rule needs two things: a minimum score per query/result type pair, and a margin above the query's reference level. The reference level is the median of the query's scores against 64 varied texts and 12 pictures that ship with the package. It measures how high "everything" scores for this query, and it doesn't depend on what you indexed. A result must reach the minimum and also be at least the margin above the query's reference level.
| Query -> result | Minimum score (70 / 140 / 280 image tokens) | Margin |
|---|---|---|
| text -> text | 0.62 | 0.10 |
| text -> pdf | 0.64 | 0.11 |
| text -> image | 0.63 / 0.62 / 0.62 | 0.06 |
| image -> image | 0.76 / 0.75 / 0.70 | 0.09 |
On the tuning corpus at budget 140, none of the 40 unrelated queries shows a result, and text-to-text precision goes from 0.02 without the cutoff to 0.87 with it. But those values were chosen on those same queries, so of course they look good. I trust the held-out numbers more:
| Held-out corpora | Old rule (index median) | Current rule |
|---|---|---|
| Relevant files shown | 64 of 72 | 67 of 72 |
| Unrelated queries that showed something | 8 of 34 | 2 of 34 |
| "sourdough bread", sourdough notes shown | 2 of 8 | 6 of 8 |
The 2 unrelated queries that still get through are about ice hockey rules and sumo rankings, and both match a picture of a chess board. The 2 missing sourdough notes are about bulk fermentation and cold proofing, don't use the word "sourdough", and score just under the minimum.
When nothing passes, anygrep says so and shows the closest few, marked weak, instead of an empty screen:
$ anygrep 'qwzx plorb fnord' -m 2; echo "exit=$?"
No strong matches; showing the closest results.
1 0.628 photos/whiteboard.jpg [image] (weak)
2 0.615 screenshots/terminal.png [image] (weak)
exit=1
In an interactive terminal, if some results were hidden just under the cutoff (less than 0.02 below it), the list ends with a dim line saying how many, so you know a file you expected may be just below the cutoff. --weak includes results below the cutoff.
Near-duplicates, and the payslip problem
Real folders have copies: an exported version, a slightly edited one, the same note in two places. If they all rank separately, the top 10 fills up with one file. At first I grouped files by embedding cosine alone (0.95 for text and PDFs, 0.98 for images).
Then monthly payslips from the same template collapsed into one result. Only the month and the amounts differ, so their cosine similarity was 0.96 to 0.98. The embedding sees layout and topic, not numbers.
So text and PDF files are now grouped only if they also share almost all their words. At index time each file gets a small signature (1 KB): the 256 smallest hashes of its adjacent word pairs. At search time, comparing two signatures estimates what share of the smaller file's word pairs is missing from the other file, and the files are grouped only if that share is 18% or less.
My first version compared full texts with difflib, and on 20 long reports from one template (8,000 words each) a single search took 8 to 48 seconds. The signature comparison takes a few microseconds per pair and never reads the text.
| Pair | Replaced words | Changed word pairs |
|---|---|---|
| Payslips or bank statements from one template | 14 to 36% | 28% or more |
| Intended copies (largest: a recipe with scaled amounts) | up to 5% | up to 10% |
| The sourdough note rewritten for rye | 9.4% | 11% |
It still over-groups some: when a template has enough fixed text that only a few amounts and the month change (a 15-line payslip, say), fewer than 18% of the word pairs differ, so those months are still collapsed into one result with a (+N similar: ...) note. --no-collapse shows every file.
Images have no words to compare, and the embeddings can't help: a resized copy of a chart and a chart with different bar heights both scored 0.999 against the original. The different numbers might be exactly what you are looking for, so images are grouped only when their bytes are identical.
Running it
pipx install anygrep # or: uv tool install anygrep
anygrep "when is the tax return due" ~/notes # new files are indexed first
anygrep --image whiteboard.jpg ~/Screenshots # find similar pictures
anygrep index --background ~/Pictures # big folder: index in a low-priority process
It needs Python 3.10+. Ctrl-C during indexing keeps what is done, and the next run continues. Text and PDFs are indexed before images, newest images first, so text is searchable while images are still going.
anygrep has no telemetry. Its only network access is the one-time model download from huggingface.co, pinned to a fixed revision and checked against SHA-256 hashes. onnxruntime has its own usage telemetry, which anygrep switches off before loading the model. With ANYGREP_OFFLINE=1, anygrep never touches the network. The index stores the text of your files unencrypted in one SQLite file, so treat it like the files themselves when you back up or sync.
Limitations
- No OCR. Small text inside images is often missed (
--image-tokens 280helps). Scanned PDFs without a text layer are skipped. - Image indexing is slow on a CPU (see Speed).
- The relevance cutoff was tuned on small synthetic corpora. Expect borderline misses and the odd chess board. Use
--weakwhen something is missing. - Search compares the query with every stored vector in memory, with no approximate index. On a synthetic index of 100,000 chunks (16,000 files), the first search after the model loads took about 0.4 s, mostly reading the vectors.
- No audio, video or Office documents yet (it tells you how many Office files it skipped), and no file watching.
- I develop on Linux. CI runs the tests on Windows and macOS too, but there are few real-world reports from those yet.
If a file you expected is missing, try the same query with --weak. If that brings the file back, please open an issue, especially if your folder is unlike the synthetic ones I tested. The output of anygrep "..." --weak --json makes it much easier to see why.
The code is on GitHub.
anygrep is MIT-licensed. The model is downloaded separately: an int8 ONNX conversion of Google's EmbeddingGemma 2, under the Apache License 2.0. anygrep is not affiliated with Google.
Top comments (1)
Porting the image processing to pure NumPy to ditch the massive PyTorch dependency for a CLI tool is an absolute chef's kiss move.