Local RAG Over Audit Reports: Searching Five Years of Vulnerabilities Offline

Last month I was reviewing a vault contract and had that itch: I have seen this exact rounding bug before, in some audit report, two or three years ago, something with ERC4626 share math. I spent forty minutes grepping through a folder of PDFs and markdown files and never found it. The knowledge existed on my disk. I just couldn't query it.

Public audit reports are one of the most underused resources in this field. Sherlock contest reports, Code4rena findings, Trail of Bits publications, OpenZeppelin audits: thousands of real vulnerabilities, described by the people who found them, with the exact code patterns that caused them. But they're scattered across PDFs, GitHub repos, and judging platforms, and keyword search fails you because the same bug gets described ten different ways. "Rounding direction favors attacker", "share price inflation", "first depositor attack", "donation attack": four phrasings, one family of bugs.

That's an embeddings problem. So I built a local RAG over my report collection. Ollama for embeddings, SQLite for storage, everything offline. No API costs, no rate limits, and I can query it on a plane. Here's how, including the one decision that matters more than all the others.

Chunk by finding, not by page

Every RAG tutorial tells you to split documents into fixed-size chunks with overlap. For audit reports that's actively harmful. A 500-token window will happily slice a finding in half, gluing the tail of "H-02: Reentrancy in withdraw" to the head of "H-03: Oracle staleness not checked". Your embedding then represents a chimera that matches nothing well.

Audit reports have a natural atomic unit: the finding. One title, one severity, one description, one code snippet, one fix. Findings are almost always short enough to embed whole, and they're exactly the granularity you want back as a search result. So the chunker's job is to detect finding boundaries, and audit reports are formulaic enough that this works well:

interface Finding {
  id: string;          // "H-01", "M-07", "TOB-XYZ-3"
  severity: string;    // parsed from the id or heading
  title: string;
  body: string;        // full finding text incl. code blocks
  source: string;      // report file path
  protocol: string;    // from report metadata
  date: string;
}

const FINDING_HEADING = /^#{1,4}\s*\[?((?:H|M|L|QA|G)-\d+|TOB-[A-Z0-9-]+-\d+|C4-\d+)\]?[:.\s-]+(.+)$/;

export function chunkReport(markdown: string, meta: ReportMeta): Finding[] {
  const lines = markdown.split("\n");
  const findings: Finding[] = [];
  let current: { id: string; title: string; start: number } | null = null;

  const flush = (end: number) => {
    if (!current) return;
    const body = lines.slice(current.start, end).join("\n").trim();
    if (body.length < 200) return; // skip stubs and withdrawn findings
    findings.push({
      id: current.id,
      severity: severityFromId(current.id),
      title: current.title,
      body,
      source: meta.path,
      protocol: meta.protocol,
      date: meta.date,
    });
  };

  lines.forEach((line, i) => {
    const m = line.match(FINDING_HEADING);
    if (m?.[1] && m[2]) {
      flush(i);
      current = { id: m[1], title: m[2].trim(), start: i };
    }
  });
  flush(lines.length);
  return findings;
}

That regex covers Code4rena and Sherlock conventions plus Trail of Bits IDs. Real life needs a couple more variants (some firms use "Finding 3:", some use severity words as headings), and PDFs need a text-extraction pass first (I use a CLI converter and accept the mess). It doesn't have to be perfect. A chunker that correctly isolates 90% of findings beats fixed-size windows by a mile.

One more trick: I prepend the metadata into the text that gets embedded, so "protocol: , severity: High" is part of what the vector represents. Queries that mention severity or protocol type then work for free.

Embeddings and storage: Ollama plus SQLite is plenty

You do not need a vector database for this. My whole corpus is a few thousand findings, and brute-force cosine similarity over a few thousand vectors takes milliseconds. SQLite holds everything:

import Database from "better-sqlite3";

const db = new Database("audits.db");
db.exec(`
  CREATE TABLE IF NOT EXISTS findings (
    id INTEGER PRIMARY KEY,
    finding_id TEXT, severity TEXT, title TEXT, body TEXT,
    protocol TEXT, source TEXT, date TEXT,
    embedding BLOB
  )
`);

async function embed(text: string): Promise<Float32Array> {
  const res = await fetch("http://localhost:11434/api/embed", {
    method: "POST",
    body: JSON.stringify({ model: "nomic-embed-text", input: text }),
  });
  const json = await res.json();
  return new Float32Array(json.embeddings[0]);
}

export async function indexFinding(f: Finding): Promise<void> {
  const text = `protocol: ${f.protocol}\nseverity: ${f.severity}\n${f.title}\n\n${f.body}`;
  const vec = await embed(text.slice(0, 8000));
  db.prepare(
    `INSERT INTO findings
     (finding_id, severity, title, body, protocol, source, date, embedding)
     VALUES (?, ?, ?, ?, ?, ?, ?, ?)`
  ).run(f.id, f.severity, f.title, f.body, f.protocol, f.source, f.date,
        Buffer.from(vec.buffer));
}

Query is the same embed call plus a scan:

function cosine(a: Float32Array, b: Float32Array): number {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i]! * b[i]!; na += a[i]! * a[i]!; nb += b[i]! * b[i]!;
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

export async function search(query: string, k = 10) {
  const qv = await embed(query);
  const rows = db.prepare("SELECT * FROM findings").all() as StoredFinding[];
  return rows
    .map(r => ({ ...r, score: cosine(qv, new Float32Array(r.embedding.buffer)) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k);
}

If your corpus grows past tens of thousands of findings, sqlite-vec gives you indexed vector search in the same file. I haven't needed it yet.

What querying this actually feels like

The query that started all this: "protocols that got burned by ERC4626 rounding". Keyword search over the same corpus returns findings that literally contain "ERC4626". The RAG returns those, plus first-depositor share inflation findings that never mention the standard by name, plus a vault bug described entirely as "assets-to-shares conversion truncates in the wrong direction". That last category is the payoff, the bugs described in words you didn't think to search for.

Other queries that have earned their keep: "signature replay across chains", "reward accounting broken by direct token transfers", "pausable functions that brick withdrawals". Before an audit or a Sherlock contest, I now query the corpus for the protocol's category and read the top twenty findings as a warm-up. It loads exactly the right prior into my head.

I also wired the search into a small answer step: take the top five findings, stuff them into qwen2.5-coder:7b with the question, get a synthesized answer with report citations. Honestly, that part is optional. Ranked raw findings with sources are usually more useful to me than the model's summary, and the retrieval is where all the value lives. Some of this plumbing later fed into how spectr-ai gives its analysis engine known-vulnerability context, but the standalone version is a weekend project, maybe two hundred lines total.

The corpus is the real moat here, and it compounds. Every report I read now gets dropped into the folder and indexed, thirty seconds of effort for a permanently searchable memory of every vulnerability I've ever studied.

What's sitting in your bookmarks or download folder that you'd query weekly if it were actually searchable?