DEV Community

Kotokvit
Kotokvit

Posted on

Poler-harvest: A Content-Addressed Search Tool in Rust with 2ms Warm Cache for AI Agents

⚡ Why Another Search Tool?

I love ripgrep and grep — they are legendary for one-off manual terminal lookups. But while designing local AI coding agents and automated developer loops, I hit a fundamental bottleneck: traditional search tools have zero memory.

When an autonomous coding agent or CI test runner performs dozens of searches across the exact same 100,000-file codebase within a few minutes, re-traversing the directory tree and re-scanning gigabytes of files from disk on every single turn wastes seconds of user latency.

To solve this, I built poler-harvest in Rust.


🎯 How It Works

poler-harvest combines high-throughput multi-threaded search with content-addressed L1–L3 persistent caching:

  1. Cold Run (Zero Cache):

    • Multi-threaded traversal powered by rayon, aho-corasick, and regex.
    • Automatically segments matching anchor maps using FastCDC (Fast Content-Defined Chunking) with BLAKE3 deduplication.
    • Compiles anchors into an ultra-compact binary crystal file (.t5a / .tar.poler).
  2. Warm Run (Instant Lookups):

    • Returns exact matches in 2 milliseconds — streaming results directly through memory-mapped L1–L3 cache.
    • Up to 300×–400× faster than cold searches on large codebases.
  3. Honest Invalidation (7/7 Regression Tests):

    • The cache never serves stale data.
    • Automatically validates files by size, mtime, and content hash, selectively re-scanning only modified, added, or deleted files in sub-milliseconds.

📊 Benchmarks (Debian 12, 100,000 Files / 12 MB)

Tool Cold Search Warm Search (Repeated) Speedup
GNU grep 3.8 (grep -r) 407–451 ms 407–451 ms 1×
ripgrep 13.0 (rg) 728–760 ms 728–760 ms 1×
poler-harvest 633–760 ms 2 ms 364× faster 🚀

(Measurements taken with hyperfine on Linux 6.x, x86_64, NVMe).


🚀 Quick Install (Linux x86_64)

One-line installer:

curl -sSL https://raw.githubusercontent.com/poler-engine-org/poler-harvest/main/scripts/install.sh | bash
Enter fullscreen mode Exit fullscreen mode

Arch Linux / CachyOS (Pacman):

sudo pacman -U https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest-0.96.1-1-x86_64.pkg.tar.zst
Enter fullscreen mode Exit fullscreen mode

Debian / Ubuntu / Mint (.deb):

curl -LO https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest_0.96.1_amd64.deb
sudo dpkg -i poler-harvest_0.96.1_amd64.deb
Enter fullscreen mode Exit fullscreen mode

Python Wheels:

pip install https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler_harvest-0.96.1-py3-none-manylinux_2_34_x86_64.whl
Enter fullscreen mode Exit fullscreen mode

🛠 Usage Examples

# Exact search with 2ms warm lookups:
poler-harvest . "TODO: fix this bug" address

# Contextual markdown search for documentation:
poler-harvest src/ "network timeout" markdown -c 2

# Regex search with machine-readable JSONL stream:
poler-harvest src/ "struct.*Error" -r -f json

# One-off scan without touching persistent cache:
poler-harvest /var/log "kernel error" --no-cache
Enter fullscreen mode Exit fullscreen mode

💎 Source-Available & Open Tooling

The project is published under the POLER Custom Source-Available License v1.0 — completely free for personal use, security audits, academic research, and community developer tooling:

⭐ GitHub Repository: https://github.com/poler-engine-org/poler-harvest

I would love to hear feedback, architectural suggestions, or questions about the FastCDC chunking pipeline from the community!

Top comments (0)