⚡ Why Another Search Tool?
I love ripgrep and grep — they are legendary for one-off manual terminal lookups. But while designing local AI coding agents and automated developer loops, I hit a fundamental bottleneck: traditional search tools have zero memory.
When an autonomous coding agent or CI test runner performs dozens of searches across the exact same 100,000-file codebase within a few minutes, re-traversing the directory tree and re-scanning gigabytes of files from disk on every single turn wastes seconds of user latency.
To solve this, I built poler-harvest in Rust.
🎯 How It Works
poler-harvest combines high-throughput multi-threaded search with content-addressed L1–L3 persistent caching:
-
Cold Run (Zero Cache):
- Multi-threaded traversal powered by
rayon,aho-corasick, andregex. - Automatically segments matching anchor maps using FastCDC (Fast Content-Defined Chunking) with BLAKE3 deduplication.
- Compiles anchors into an ultra-compact binary crystal file (
.t5a/.tar.poler).
- Multi-threaded traversal powered by
-
Warm Run (Instant Lookups):
- Returns exact matches in 2 milliseconds — streaming results directly through memory-mapped L1–L3 cache.
- Up to 300×–400× faster than cold searches on large codebases.
-
Honest Invalidation (7/7 Regression Tests):
- The cache never serves stale data.
- Automatically validates files by
size,mtime, and content hash, selectively re-scanning only modified, added, or deleted files in sub-milliseconds.
📊 Benchmarks (Debian 12, 100,000 Files / 12 MB)
| Tool | Cold Search | Warm Search (Repeated) | Speedup |
|---|---|---|---|
GNU grep 3.8 (grep -r) |
407–451 ms | 407–451 ms | 1× |
ripgrep 13.0 (rg) |
728–760 ms | 728–760 ms | 1× |
| poler-harvest | 633–760 ms | 2 ms | 364× faster 🚀 |
(Measurements taken with hyperfine on Linux 6.x, x86_64, NVMe).
🚀 Quick Install (Linux x86_64)
One-line installer:
curl -sSL https://raw.githubusercontent.com/poler-engine-org/poler-harvest/main/scripts/install.sh | bash
Arch Linux / CachyOS (Pacman):
sudo pacman -U https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest-0.96.1-1-x86_64.pkg.tar.zst
Debian / Ubuntu / Mint (.deb):
curl -LO https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest_0.96.1_amd64.deb
sudo dpkg -i poler-harvest_0.96.1_amd64.deb
Python Wheels:
pip install https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler_harvest-0.96.1-py3-none-manylinux_2_34_x86_64.whl
🛠 Usage Examples
# Exact search with 2ms warm lookups:
poler-harvest . "TODO: fix this bug" address
# Contextual markdown search for documentation:
poler-harvest src/ "network timeout" markdown -c 2
# Regex search with machine-readable JSONL stream:
poler-harvest src/ "struct.*Error" -r -f json
# One-off scan without touching persistent cache:
poler-harvest /var/log "kernel error" --no-cache
💎 Source-Available & Open Tooling
The project is published under the POLER Custom Source-Available License v1.0 — completely free for personal use, security audits, academic research, and community developer tooling:
⭐ GitHub Repository: https://github.com/poler-engine-org/poler-harvest
I would love to hear feedback, architectural suggestions, or questions about the FastCDC chunking pipeline from the community!
Top comments (0)