<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Kotokvit</title>
    <description>The latest articles on DEV Community by Kotokvit (@kotokvit).</description>
    <link>https://dev.to/kotokvit</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254627%2F2611b740-52f1-4154-a5eb-f8aa21e51216.png</url>
      <title>DEV Community: Kotokvit</title>
      <link>https://dev.to/kotokvit</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/kotokvit"/>
    <language>en</language>
    <item>
      <title>Poler-harvest: A Content-Addressed Search Tool in Rust with 2ms Warm Cache for AI Agents</title>
      <dc:creator>Kotokvit</dc:creator>
      <pubDate>Fri, 20 Nov 2026 00:10:02 +0000</pubDate>
      <link>https://dev.to/kotokvit/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai-agents-87d</link>
      <guid>https://dev.to/kotokvit/poler-harvest-a-content-addressed-search-tool-in-rust-with-2ms-warm-cache-for-ai-agents-87d</guid>
      <description>&lt;h1&gt;
  
  
  ⚡ Why Another Search Tool?
&lt;/h1&gt;

&lt;p&gt;I love &lt;code&gt;ripgrep&lt;/code&gt; and &lt;code&gt;grep&lt;/code&gt; — they are legendary for one-off manual terminal lookups. But while designing local AI coding agents and automated developer loops, I hit a fundamental bottleneck: &lt;strong&gt;traditional search tools have zero memory&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;When an autonomous coding agent or CI test runner performs dozens of searches across the exact same 100,000-file codebase within a few minutes, re-traversing the directory tree and re-scanning gigabytes of files from disk on every single turn wastes seconds of user latency.&lt;/p&gt;

&lt;p&gt;To solve this, I built &lt;strong&gt;&lt;a href="https://github.com/poler-engine-org/poler-harvest" rel="noopener noreferrer"&gt;poler-harvest&lt;/a&gt;&lt;/strong&gt; in Rust.&lt;/p&gt;




&lt;h2&gt;
  
  
  🎯 How It Works
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;poler-harvest&lt;/code&gt; combines high-throughput multi-threaded search with content-addressed L1–L3 persistent caching:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Cold Run (Zero Cache):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Multi-threaded traversal powered by &lt;code&gt;rayon&lt;/code&gt;, &lt;code&gt;aho-corasick&lt;/code&gt;, and &lt;code&gt;regex&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Automatically segments matching anchor maps using &lt;strong&gt;FastCDC (Fast Content-Defined Chunking)&lt;/strong&gt; with BLAKE3 deduplication.&lt;/li&gt;
&lt;li&gt;Compiles anchors into an ultra-compact binary crystal file (&lt;code&gt;.t5a&lt;/code&gt; / &lt;code&gt;.tar.poler&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Warm Run (Instant Lookups):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Returns exact matches in &lt;strong&gt;2 milliseconds&lt;/strong&gt; — streaming results directly through memory-mapped L1–L3 cache.&lt;/li&gt;
&lt;li&gt;Up to &lt;strong&gt;300×–400× faster&lt;/strong&gt; than cold searches on large codebases.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Honest Invalidation (7/7 Regression Tests):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The cache never serves stale data.&lt;/li&gt;
&lt;li&gt;Automatically validates files by &lt;code&gt;size&lt;/code&gt;, &lt;code&gt;mtime&lt;/code&gt;, and content hash, selectively re-scanning only modified, added, or deleted files in sub-milliseconds.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  📊 Benchmarks (Debian 12, 100,000 Files / 12 MB)
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;Cold Search&lt;/th&gt;
&lt;th&gt;Warm Search (Repeated)&lt;/th&gt;
&lt;th&gt;Speedup&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;GNU grep 3.8&lt;/strong&gt; (&lt;code&gt;grep -r&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;407–451 ms&lt;/td&gt;
&lt;td&gt;407–451 ms&lt;/td&gt;
&lt;td&gt;1×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;ripgrep 13.0&lt;/strong&gt; (&lt;code&gt;rg&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;728–760 ms&lt;/td&gt;
&lt;td&gt;728–760 ms&lt;/td&gt;
&lt;td&gt;1×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;poler-harvest&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;633–760 ms&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;364× faster&lt;/strong&gt; 🚀&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;(Measurements taken with &lt;code&gt;hyperfine&lt;/code&gt; on Linux 6.x, x86_64, NVMe).&lt;/em&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  🚀 Quick Install (Linux x86_64)
&lt;/h2&gt;

&lt;h3&gt;
  
  
  One-line installer:
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sSL https://raw.githubusercontent.com/poler-engine-org/poler-harvest/main/scripts/install.sh &lt;span class="o"&gt;|&lt;/span&gt; bash
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Arch Linux / CachyOS (Pacman):
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo pacman -U https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest-0.96.1-1-x86_64.pkg.tar.zst
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Debian / Ubuntu / Mint (.deb):
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -LO https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest_0.96.1_amd64.deb
sudo dpkg -i poler-harvest_0.96.1_amd64.deb
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Python Wheels:
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler_harvest-0.96.1-py3-none-manylinux_2_34_x86_64.whl
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  🛠 Usage Examples
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Exact search with 2ms warm lookups:&lt;/span&gt;
poler-harvest . &lt;span class="s2"&gt;"TODO: fix this bug"&lt;/span&gt; address

&lt;span class="c"&gt;# Contextual markdown search for documentation:&lt;/span&gt;
poler-harvest src/ &lt;span class="s2"&gt;"network timeout"&lt;/span&gt; markdown -c 2

&lt;span class="c"&gt;# Regex search with machine-readable JSONL stream:&lt;/span&gt;
poler-harvest src/ &lt;span class="s2"&gt;"struct.*Error"&lt;/span&gt; -r -f json

&lt;span class="c"&gt;# One-off scan without touching persistent cache:&lt;/span&gt;
poler-harvest /var/log &lt;span class="s2"&gt;"kernel error"&lt;/span&gt; --no-cache
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  💎 Source-Available &amp;amp; Open Tooling
&lt;/h2&gt;

&lt;p&gt;The project is published under the &lt;strong&gt;POLER Custom Source-Available License v1.0&lt;/strong&gt; — completely free for personal use, security audits, academic research, and community developer tooling:&lt;/p&gt;

&lt;p&gt;⭐ &lt;strong&gt;GitHub Repository:&lt;/strong&gt; &lt;a href="https://github.com/poler-engine-org/poler-harvest" rel="noopener noreferrer"&gt;https://github.com/poler-engine-org/poler-harvest&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I would love to hear feedback, architectural suggestions, or questions about the FastCDC chunking pipeline from the community!&lt;/p&gt;

</description>
      <category>rust</category>
      <category>cli</category>
      <category>opensource</category>
      <category>performance</category>
    </item>
  </channel>
</rss>
