DEV Community: Pavel Espitia

I Point a Local LLM at Every Repo Before Opening It in My Editor

Pavel Espitia — Tue, 04 Aug 2026 16:18:32 +0000

In May a "recruiter" sent me a take-home project for a Web3 role. Nice README, plausible Next.js structure, a real-sounding company. Buried in the build tooling was a postinstall script that decoded a base64 blob and pulled a second stage from a hardcoded IP. If I had done what 99% of candidates do, git clone then npm install then open it in my editor, an infostealer would have been running on my machine before I read a single line of code.

That was not the last one either. These fake-recruiter lures are an industry now, and the payload almost never lives in src/. It lives in the places you skim: lifecycle scripts, config files, a "utils" file with one weird function. So I changed my default. Every unknown repo now goes through a local LLM triage pass before my editor ever touches it. No code execution, no install, just static reading.

Here's the workflow.

Rule zero: never let the repo run anything

The whole point is that the repo stays inert. Two safe ways to get the files:

# Option 1: clone without checkout, inspect the tree first
git clone --no-checkout https://github.com/some-org/take-home-task.git
cd take-home-task
git ls-tree -r HEAD --name-only

# Option 2: download the tarball, no git hooks, no clone at all
curl -L https://github.com/some-org/take-home-task/archive/refs/heads/main.tar.gz \
  | tar -xz -C ./quarantine/

I prefer the tarball. It cannot execute anything, and extracting into a quarantine/ directory keeps me honest. Also worth saying explicitly: do not open the folder in an editor with plugins that auto-run tasks. VS Code will happily execute workspace settings, launch configs, and some extensions will run npm install for you as a favor. Read the files with cat, bat, or the LLM pipeline below.

Step 1: the boring checks that catch most payloads

Before any AI is involved, three cheap checks catch the majority of these campaigns:

# Lifecycle scripts are the #1 delivery mechanism
cat package.json | jq '.scripts | with_entries(
  select(.key | test("install|prepare|prepublish|postpack")))'

# Dependencies present in package.json but missing from the lockfile
# (a classic evasion: the malicious dep gets resolved fresh at install time)
jq -r '.dependencies, .devDependencies | keys[]?' package.json | sort > deps.txt
jq -r '.packages | keys[]' package-lock.json | sed 's|node_modules/||' | sort > locked.txt
comm -23 deps.txt locked.txt

# Long encoded blobs anywhere in the tree
rg -n --max-columns=200 '[A-Za-z0-9+/]{120,}={0,2}' --glob '!*.lock' --glob '!*.map'

The lockfile mismatch check matters more than people think. Several campaigns I've dissected ship a clean-looking lockfile and a dirty package.json, or reference a typosquatted package only from a script. When I built Argus Lens (lens.noctis.biz), a scanner for exactly this class of repo, deps-missing-from-lockfile turned out to be one of the highest-signal checks in the whole tool.

Step 2: feed the suspicious files to a local model

Regex gets you candidates. Judgment is where a model earns its keep, and this has to be a local model, because I'm sometimes triaging repos under NDA or repos whose mere URL I don't want leaving my machine.

I run Ollama on WSL2 with qwen2.5-coder in two sizes: 1.5b for the fast pass over everything, 7b when the small one flags something. The prompt is a classifier, not a chat:

triage_file() {
  local file="$1"
  ollama run qwen2.5-coder:7b <<EOF
You are a supply-chain malware analyst. Classify the following file
from an UNTRUSTED repository. Do not summarize what the code claims
to do. Focus on what it actually does.

Answer in exactly this format:
VERDICT: CLEAN | SUSPICIOUS | MALICIOUS
SIGNALS: <comma-separated list, or "none">
EXPLANATION: <max 3 sentences>

Signals to look for:
- decoding of base64/hex strings followed by eval, Function, or child_process
- network calls to raw IPs or unusual domains at import/build time
- reading of environment variables, keychains, browser profile paths,
  .ssh, .aws, or wallet files
- code that only runs during install/build, not at runtime
- obfuscation: string array shuffling, charCode arithmetic, packed code

FILE: ${file}
---
$(cat "$file")
EOF
}

Then it's just a loop over the candidates:

rg -l 'child_process|eval\(|Function\(|fromCharCode|atob|Buffer\.from' \
  --glob '!node_modules' quarantine/ | while read -r f; do
  echo "=== $f"
  triage_file "$f"
done

The strict output format is doing real work here. Small models ramble, and "answer with VERDICT on the first line" turns a rambling model into something you can grep and script against.

What signals actually matter

After feeding a few dozen of these repos through this pipeline (and building spectr-ai, my open-source contract auditor, which taught me a lot about prompting small models for security work), the signals that separate real payloads from noise:

Install-time execution. Legitimate projects rarely need postinstall beyond native module builds. A postinstall that touches the network or decodes strings is close to a guaranteed conviction.

Deps in manifest but not in lockfile. Covered above. It means the attacker wants resolution to happen fresh on your machine.

Encoded blobs plus a decoder. A base64 string alone is often fine (inlined images, test fixtures). A base64 string within reach of eval, new Function, or child_process.exec is not.

Env harvesting. Loops over process.env, or path building toward ~/.ssh, browser extension folders, or wallet data directories like Local Storage/leveldb. There is no honest reason for a take-home CRUD app to know where MetaMask keeps its state.

Effort asymmetry. The app code is boilerplate quality, but one config or helper file is dense, minified, or oddly sophisticated. Attackers copy the app and hand-craft the payload, and the seam shows.

Honest limitations

The 1.5b model misses things. It's fine as a fast filter over many files, but I've watched it label a charCode-obfuscated dropper as "string manipulation utilities." The 7b catches most of what I throw at it, but a determined attacker who tests their payload against open models will eventually get past this too. That's fine. I'm not trying to build a perfect oracle, I'm trying to make sure the lazy, mass-produced lures (which is most of them) get caught in under two minutes without me executing anything.

Also: the model reads what you give it. If you only scan .js files, the payload will be in a .node binary or a build config. Cast the net wide first, then classify.

The whole thing costs me maybe three minutes per unknown repo, runs entirely offline, and has already paid for itself twice. Cheap insurance.

Do you actually inspect repos from strangers before installing, or does npm install still happen on autopilot?

A Local AI Pre-Commit Hook That Blocks Secrets Without Annoying You

Pavel Espitia — Mon, 03 Aug 2026 16:32:16 +0000

My regex secret scanner once blocked a commit because a test file contained the string sk_test_EXAMPLE_KEY_DO_NOT_USE. The same week, a colleague on another project committed a real Etherscan API key inside a hardcoded URL, and no scanner caught it because it didn't match any known key format. That pair of failures sums up regex-based secret scanning: loud where it doesn't matter, quiet where it does.

The standard fix is an allowlist file that grows forever, plus developers who learn to type git commit --no-verify from muscle memory. Once people bypass the hook by habit, the scanner is decoration.

So I tried something different: keep the regex scanner, but add a small local LLM as a second opinion. The regex stage decides what's worth looking at. The model decides whether it's actually a secret. Only flagged files ever reach the model, so the hook stays fast, and because the model is Ollama running on my own machine, no staged diff ever leaves my laptop. That last part is non-negotiable for me, sending your possibly-secret-containing diffs to a cloud API to check for secrets is a joke that writes itself.

The architecture in one paragraph

Stage 1 is a deliberately paranoid regex pass over staged changes: high-entropy strings, known key prefixes, PRIVATE KEY blocks, suspicious variable names. Stage 2 sends each flagged hunk, with a few lines of surrounding context, to qwen2.5-coder via Ollama with a classification prompt. Verdict SECRET blocks the commit, FALSE_POSITIVE lets it through. Most commits never trigger stage 1 at all, so most commits pay zero latency.

The hook

.git/hooks/pre-commit (or wire it through your hook manager of choice):

#!/usr/bin/env bash
set -euo pipefail

MODEL="${SECRET_HOOK_MODEL:-qwen2.5-coder:7b}"

# Stage 1: cheap and paranoid. Wide patterns, we WANT false positives here.
PATTERNS=(
  'AKIA[0-9A-Z]{16}'                      # AWS access key
  '-----BEGIN( RSA| EC| OPENSSH)? PRIVATE KEY-----'
  '(api[_-]?key|secret|token|passwd|password)["'"'"']?\s*[:=]\s*["'"'"'][^"'"'"']{16,}'
  '0x[a-fA-F0-9]{64}'                     # possible EVM private key
  '[A-Za-z0-9+/]{40,}={0,2}'              # high-entropy base64-ish
)

flagged=()
while IFS= read -r file; do
  [[ -f "$file" ]] || continue
  for p in "${PATTERNS[@]}"; do
    if git show ":$file" | grep -qE "$p"; then
      flagged+=("$file")
      break
    fi
  done
done < <(git diff --cached --name-only --diff-filter=ACM)

[[ ${#flagged[@]} -eq 0 ]] && exit 0   # fast path: nothing suspicious

# Stage 2: ask the local model about each flagged file's staged content.
block=0
for file in "${flagged[@]}"; do
  verdict=$(git show ":$file" | ollama run "$MODEL" "$(cat <<'PROMPT'
You review a file staged for a git commit. Decide if it contains a REAL
credential that must not be committed.

REAL secrets: live API keys, private keys (including 0x-prefixed 64-hex
EVM keys), tokens, passwords, connection strings with embedded passwords.

NOT secrets: placeholders (YOUR_KEY_HERE, xxx, changeme), documented
example keys, test fixtures clearly labeled as fake, public addresses,
hashes of public data, template variables like ${API_KEY}, lockfile
integrity hashes.

First line of your answer must be exactly SECRET or FALSE_POSITIVE.
Second line: one short reason.
PROMPT
)")
  if [[ "$verdict" == SECRET* ]]; then
    echo "BLOCKED: $file"
    echo "$verdict" | sed -n '2p' | sed 's/^/  reason: /'
    block=1
  fi
done

if [[ $block -eq 1 ]]; then
  echo ""
  echo "Commit blocked. If this is wrong, re-run with SECRET_HOOK_MODEL"
  echo "set to a bigger model, or use --no-verify and accept the risk."
  exit 1
fi
exit 0

Two details in there matter more than they look.

Scan the staged content, not the working tree. git show ":$file" reads the index. If you scan the file on disk, you'll block commits over unstaged scratch content, and you'll miss the case where the secret is staged but already deleted from the working copy.

The exit-code contract is the whole interface. Exit 0 commits, exit 1 blocks, and the model's freeform text never decides anything by itself. I parse only the first line and demand it be one of two tokens. Small local models will occasionally produce a paragraph of hedging, forcing a machine-readable first line is what makes them usable in a pipeline.

The prompt is mostly a list of non-secrets

Notice the prompt spends more words on what is NOT a secret than on what is. That's deliberate. The regex stage already guarantees everything the model sees looks secret-ish, so the model's real job is recognizing placeholders, fixtures, and templates. Framing it that way cut my false blocks dramatically. I learned this pattern building spectr-ai: small models do much better when you tell them what to exclude than when you ask them open-ended "is this dangerous?" questions.

The 0x + 64 hex chars pattern deserves a note for the Web3 crowd. A transaction hash, a storage slot, and a private key all look identical to a regex. The model can use the variable name and surrounding code to tell DEPLOYER_PRIVATE_KEY = 0x... apart from KNOWN_TX_HASH = 0x.... That single distinction is most of the value I get from this hook, because regex scanners either flag every 32-byte hex value in a blockchain codebase (unbearable) or none (useless).

Honest tradeoffs

Latency. On my machine (WSL2, mid-range GPU), a flagged file costs roughly two to four seconds with the 7b model once it's warm, more if Ollama has to load the model first. Clean commits pay nothing because stage 1 short-circuits. Commits that touch a .env.example or a test fixture pay a few seconds. I find that acceptable, you might not, and if your team commits forty times an hour you should keep the model stage async or advisory.

1.5b vs 7b. I tried qwen2.5-coder:1.5b first because it responds almost instantly. It was too eager to please: it labeled obviously fake fixtures as SECRET often enough that I would have started bypassing my own hook, which defeats the entire purpose. The 7b is noticeably better at reading context like "this is in tests/fixtures/ and the variable is called FAKE_KEY". For this job the model runs rarely, so I pay for the bigger one.

It can still be wrong both ways. A local 7b model is not a security boundary. A weird real key can slip through, and this hook is a complement to your platform-side scanning (GitHub push protection and friends), not a replacement. What it fixes is the human layer: the hook complains so rarely that when it does, I actually stop and look instead of reflexively reaching for --no-verify. A scanner people trust and obey beats a stricter one everybody bypasses.

Determinism. LLM verdicts can flip between runs on borderline inputs. For a blocking hook I accept that, borderline cases are exactly the ones I want a human to re-examine anyway.

I've run this for a couple of months now. The regex stage fires a few times a week, the model overrules it almost every time, and the one time it said SECRET it was right: a real RPC URL with an embedded API key inside an old test I was resurrecting.

What's your current secret-scanning setup, and be honest, how often do you bypass it?

Local RAG Over Audit Reports: Searching Five Years of Vulnerabilities Offline

Pavel Espitia — Sun, 02 Aug 2026 15:09:53 +0000

Last month I was reviewing a vault contract and had that itch: I have seen this exact rounding bug before, in some audit report, two or three years ago, something with ERC4626 share math. I spent forty minutes grepping through a folder of PDFs and markdown files and never found it. The knowledge existed on my disk. I just couldn't query it.

Public audit reports are one of the most underused resources in this field. Sherlock contest reports, Code4rena findings, Trail of Bits publications, OpenZeppelin audits: thousands of real vulnerabilities, described by the people who found them, with the exact code patterns that caused them. But they're scattered across PDFs, GitHub repos, and judging platforms, and keyword search fails you because the same bug gets described ten different ways. "Rounding direction favors attacker", "share price inflation", "first depositor attack", "donation attack": four phrasings, one family of bugs.

That's an embeddings problem. So I built a local RAG over my report collection. Ollama for embeddings, SQLite for storage, everything offline. No API costs, no rate limits, and I can query it on a plane. Here's how, including the one decision that matters more than all the others.

Chunk by finding, not by page

Every RAG tutorial tells you to split documents into fixed-size chunks with overlap. For audit reports that's actively harmful. A 500-token window will happily slice a finding in half, gluing the tail of "H-02: Reentrancy in withdraw" to the head of "H-03: Oracle staleness not checked". Your embedding then represents a chimera that matches nothing well.

Audit reports have a natural atomic unit: the finding. One title, one severity, one description, one code snippet, one fix. Findings are almost always short enough to embed whole, and they're exactly the granularity you want back as a search result. So the chunker's job is to detect finding boundaries, and audit reports are formulaic enough that this works well:

interface Finding {
  id: string;          // "H-01", "M-07", "TOB-XYZ-3"
  severity: string;    // parsed from the id or heading
  title: string;
  body: string;        // full finding text incl. code blocks
  source: string;      // report file path
  protocol: string;    // from report metadata
  date: string;
}

const FINDING_HEADING = /^#{1,4}\s*\[?((?:H|M|L|QA|G)-\d+|TOB-[A-Z0-9-]+-\d+|C4-\d+)\]?[:.\s-]+(.+)$/;

export function chunkReport(markdown: string, meta: ReportMeta): Finding[] {
  const lines = markdown.split("\n");
  const findings: Finding[] = [];
  let current: { id: string; title: string; start: number } | null = null;

  const flush = (end: number) => {
    if (!current) return;
    const body = lines.slice(current.start, end).join("\n").trim();
    if (body.length < 200) return; // skip stubs and withdrawn findings
    findings.push({
      id: current.id,
      severity: severityFromId(current.id),
      title: current.title,
      body,
      source: meta.path,
      protocol: meta.protocol,
      date: meta.date,
    });
  };

  lines.forEach((line, i) => {
    const m = line.match(FINDING_HEADING);
    if (m?.[1] && m[2]) {
      flush(i);
      current = { id: m[1], title: m[2].trim(), start: i };
    }
  });
  flush(lines.length);
  return findings;
}

That regex covers Code4rena and Sherlock conventions plus Trail of Bits IDs. Real life needs a couple more variants (some firms use "Finding 3:", some use severity words as headings), and PDFs need a text-extraction pass first (I use a CLI converter and accept the mess). It doesn't have to be perfect. A chunker that correctly isolates 90% of findings beats fixed-size windows by a mile.

One more trick: I prepend the metadata into the text that gets embedded, so "protocol: , severity: High" is part of what the vector represents. Queries that mention severity or protocol type then work for free.

Embeddings and storage: Ollama plus SQLite is plenty

You do not need a vector database for this. My whole corpus is a few thousand findings, and brute-force cosine similarity over a few thousand vectors takes milliseconds. SQLite holds everything:

import Database from "better-sqlite3";

const db = new Database("audits.db");
db.exec(`
  CREATE TABLE IF NOT EXISTS findings (
    id INTEGER PRIMARY KEY,
    finding_id TEXT, severity TEXT, title TEXT, body TEXT,
    protocol TEXT, source TEXT, date TEXT,
    embedding BLOB
  )
`);

async function embed(text: string): Promise<Float32Array> {
  const res = await fetch("http://localhost:11434/api/embed", {
    method: "POST",
    body: JSON.stringify({ model: "nomic-embed-text", input: text }),
  });
  const json = await res.json();
  return new Float32Array(json.embeddings[0]);
}

export async function indexFinding(f: Finding): Promise<void> {
  const text = `protocol: ${f.protocol}\nseverity: ${f.severity}\n${f.title}\n\n${f.body}`;
  const vec = await embed(text.slice(0, 8000));
  db.prepare(
    `INSERT INTO findings
     (finding_id, severity, title, body, protocol, source, date, embedding)
     VALUES (?, ?, ?, ?, ?, ?, ?, ?)`
  ).run(f.id, f.severity, f.title, f.body, f.protocol, f.source, f.date,
        Buffer.from(vec.buffer));
}

Query is the same embed call plus a scan:

function cosine(a: Float32Array, b: Float32Array): number {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i]! * b[i]!; na += a[i]! * a[i]!; nb += b[i]! * b[i]!;
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

export async function search(query: string, k = 10) {
  const qv = await embed(query);
  const rows = db.prepare("SELECT * FROM findings").all() as StoredFinding[];
  return rows
    .map(r => ({ ...r, score: cosine(qv, new Float32Array(r.embedding.buffer)) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k);
}

If your corpus grows past tens of thousands of findings, sqlite-vec gives you indexed vector search in the same file. I haven't needed it yet.

What querying this actually feels like

The query that started all this: "protocols that got burned by ERC4626 rounding". Keyword search over the same corpus returns findings that literally contain "ERC4626". The RAG returns those, plus first-depositor share inflation findings that never mention the standard by name, plus a vault bug described entirely as "assets-to-shares conversion truncates in the wrong direction". That last category is the payoff, the bugs described in words you didn't think to search for.

Other queries that have earned their keep: "signature replay across chains", "reward accounting broken by direct token transfers", "pausable functions that brick withdrawals". Before an audit or a Sherlock contest, I now query the corpus for the protocol's category and read the top twenty findings as a warm-up. It loads exactly the right prior into my head.

I also wired the search into a small answer step: take the top five findings, stuff them into qwen2.5-coder:7b with the question, get a synthesized answer with report citations. Honestly, that part is optional. Ranked raw findings with sources are usually more useful to me than the model's summary, and the retrieval is where all the value lives. Some of this plumbing later fed into how spectr-ai gives its analysis engine known-vulnerability context, but the standalone version is a weekend project, maybe two hundred lines total.

The corpus is the real moat here, and it compounds. Every report I read now gets dropped into the folder and indexed, thirty seconds of effort for a permanently searchable memory of every vulnerability I've ever studied.

What's sitting in your bookmarks or download folder that you'd query weekly if it were actually searchable?

Function Calling With a Local LLM to Drive Foundry: Fuzz, Read, Repeat

Pavel Espitia — Sat, 01 Aug 2026 15:09:05 +0000

The first time I let a local model drive Foundry unsupervised, it spent eleven turns trying to fix a fuzz test by renaming the test function. Not changing the logic. Renaming it. testFuzz_withdraw, then test_fuzz_withdraw, then testWithdrawFuzz, each time running the suite and reading the same compiler error with fresh optimism.

That experiment still turned into one of the more useful tools in my workflow, once I accepted what a small local model can and cannot do in an agent loop. The idea is simple: give qwen2.5-coder three tools (run forge tests, read a contract file, write a fuzz test) and point it at a target contract. It reads the code, writes property tests, runs them, reads the failures, and iterates. When it works, it surfaces broken invariants I would have gotten to eventually, but it gets there while I make coffee, and everything stays on my machine.

The three tools

Ollama supports function calling through its chat API. You describe tools in JSON schema, the model returns tool_calls, you execute and feed results back. Keep the toolset minimal, every extra tool is another way for a 7b model to get confused.

const tools = [
  {
    type: "function",
    function: {
      name: "read_file",
      description: "Read a Solidity source file from the project.",
      parameters: {
        type: "object",
        properties: {
          path: { type: "string", description: "Path relative to project root" },
        },
        required: ["path"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "write_fuzz_test",
      description:
        "Write the complete contents of the fuzz test file test/Fuzz.t.sol. " +
        "Always write the FULL file, never a fragment.",
      parameters: {
        type: "object",
        properties: {
          content: { type: "string", description: "Full Solidity file content" },
        },
        required: ["content"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "run_forge_test",
      description: "Compile and run the test suite. Returns compiler errors or test results.",
      parameters: { type: "object", properties: {}, required: [] },
    },
  },
];

Two design decisions hiding in there. First, write_fuzz_test takes no path: the model writes exactly one file, test/Fuzz.t.sol, always in full. Letting a small model choose file paths or emit diffs is asking for contracts/test/../test/Fuzz.sol and patches against lines that don't exist. Second, run_forge_test takes no arguments at all, so the model can't invent flags.

The executors are thin wrappers. The only interesting one is the test runner, which truncates aggressively:

import { execFileSync } from "node:child_process";

function runForgeTest(): string {
  try {
    const out = execFileSync("forge", ["test", "--fuzz-runs", "256"], {
      cwd: PROJECT_ROOT,
      timeout: 120_000,
      encoding: "utf8",
    });
    return tail(out, 3000);
  } catch (e: unknown) {
    const err = e as { stdout?: string; stderr?: string };
    return "FAILED\n" + tail((err.stdout ?? "") + (err.stderr ?? ""), 3000);
  }
}

Truncation matters because a forge trace on a failing fuzz test can be enormous, and dumping 40k tokens of trace into a small model's context doesn't inform it, it lobotomizes it. I keep the tail, which is where Foundry puts the counterexample and the summary.

The loop

const messages: Message[] = [
  { role: "system", content: SYSTEM_PROMPT },
  { role: "user", content: `Target contract: src/${target}. Read it, identify invariants, write fuzz tests, run them, iterate until they compile and either pass or reveal a real counterexample.` },
];

for (let turn = 0; turn < MAX_TURNS; turn++) {
  const res = await ollama.chat({ model: "qwen2.5-coder:7b", messages, tools });
  messages.push(res.message);

  if (!res.message.tool_calls?.length) break; // model thinks it's done

  for (const call of res.message.tool_calls) {
    const result = dispatch(call.function.name, call.function.arguments);
    messages.push({ role: "tool", content: result });
  }
}

The system prompt is where most of the iteration went. The parts that earned their place:

You write Foundry fuzz tests for one target contract.
Rules:
- ALWAYS read the target contract before writing any test.
- Test properties and invariants, not specific values. Good properties:
  balance accounting sums correctly, access control cannot be bypassed,
  state transitions preserve solvency, no operation mints value from nothing.
- Use vm.assume or bound() to constrain inputs, never require statements.
- After every write, run the tests. Read the ACTUAL error before editing.
- If the same error appears twice in a row, change your approach entirely.
- A failing fuzz test with a counterexample is a SUCCESS. Report it and stop.

That last line is important and counterintuitive. The model's instinct is to "fix" a failing test by weakening the assertion until it passes, which is the exact opposite of what a security tool should do. Telling it explicitly that a legitimate counterexample is the win condition mostly (not always) suppresses that reflex. I hit the same failure mode building spectr-ai: small models are trained to make errors go away, and in security work the error is frequently the product.

Where it gets stuck, and how I bound it

Small models lose the plot in long agent loops. Around eight to twelve turns in, qwen2.5-coder:7b starts forgetting constraints from the system prompt, re-reading files it already read, or cycling between two broken versions of the same test. The 1.5b model is worse, it rarely survives past turn four before emitting a tool call with malformed arguments. This isn't a bug I can prompt away, it's what limited context handling and small parameter counts look like in practice.

So I stopped fighting it and bounded the loop instead:

Hard turn cap. MAX_TURNS = 12. Past that, whatever's in test/Fuzz.t.sol is the output, and I take over.

Loop detection. I hash each write_fuzz_test payload. Same hash twice means the model is cycling, and the harness injects a user message saying so, one time. Second repeat kills the run.

Compile-first gating. The most common death spiral is chasing compiler errors (wrong import path, wrong pragma). The harness pre-seeds the file with a known-good skeleton (correct imports, a deployed instance of the target in setUp), so the model starts from something that compiles and only has to fill in properties. This single change roughly doubled how often runs end usefully.

One target, one file. I don't ask it to fuzz a protocol. I ask it to fuzz one contract, sometimes one function. Small scope is the difference between a focused property test and turn-eleven function renaming.

With those bounds, a run takes a few minutes on my machine (WSL2, Ollama, 7b model) and ends usefully most of the time: either compiling property tests I edit and keep, or a genuine counterexample. It found an unchecked rounding case in one of my own vault experiments that my hand-written tests missed, mostly because the model, having no idea what the code was "supposed" to do, tested the boring invariant I had skipped.

That's the honest pitch. It's not an autonomous auditor, it's a tireless, slightly dim intern with a Foundry license, and bounded correctly that's genuinely worth having.

Have you tried giving a local model tools yet, and if so, at what turn count does yours start eating its own tail?

Qwen2.5-Coder vs DeepSeek-Coder for Solidity Review: What I Actually See Locally

Pavel Espitia — Fri, 31 Jul 2026 16:06:04 +0000

I keep a folder of ten small Solidity contracts with bugs I planted myself: a classic reentrancy in a withdraw function, a missing access modifier on an initializer, an ERC4626 vault that rounds in the depositor's favor, a signature check that never validates the signer, a few subtler logic bugs. Whenever I'm deciding which local model earns a slot on my disk, I run every candidate against all ten with the same prompts and compare notes. It's not a benchmark, there's no percentage at the end, but after doing this for months the patterns are consistent enough to be worth writing down.

This round: qwen2.5-coder against deepseek-coder, both running through Ollama on WSL2, both at the sizes that actually fit comfortably on my hardware, meaning 7B and below. If you can run 33B models locally, your conclusions will differ and you should mostly stop reading benchmarks of small models anyway.

Methodology, such as it is

Same ten contracts, none over 150 lines. Same three prompts per contract: an open "review this contract for security issues", a targeted "check specifically for reentrancy and access control problems", and a structured one demanding findings as a severity-tagged list. Each combination run a few times because small models are noisy, one lucky sample proves nothing. Temperature low but not zero. I score by hand: did it find the planted bug, did it describe it correctly, and how much noise came along for the ride.

Two honest caveats. First, these contracts are small and self-contained, which flatters every model. Real review means cross-contract call graphs and 2000-line diffs, and no 7B model survives contact with that regardless of family. Second, my planted bugs skew toward canonical patterns, the kind that appear in training data constantly. Both facts mean my setup measures "useful assistant on digestible code", not "auditor replacement". That's fine, it's the only job I'd give these models anyway.

Where qwen2.5-coder wins

Instruction following. This is the biggest practical gap and it has nothing to do with security knowledge. When I demand output like:

Report findings as a list. For each:
SEVERITY: HIGH | MEDIUM | LOW
LINE: <number or range>
ISSUE: <one sentence>
Report at most 5 findings. If none, say NONE.

qwen at 7B complies almost every run. deepseek at comparable size complies most of the time but drifts more, adding preambles, ignoring the cap, or wrapping the list in commentary I have to strip. If you're piping model output into a pipeline, which I am (this comparison started as model selection for spectr-ai, my open-source auditor), format discipline is worth more than a marginally better eye for bugs, because a finding you can't parse is a finding you don't have.

Targeted checks. Ask qwen specifically about reentrancy and it stays on reentrancy. deepseek more often wanders into gas remarks and style nits mid-answer, which pollutes the signal when I asked a narrow question.

Explanation quality. When both models find the same bug, qwen's explanation more often names the actual attack sequence (call withdraw, reenter through the fallback before the balance update) while deepseek at small sizes tends toward the generic textbook sentence about external calls. Both are "correct", one is more useful in a report.

Where deepseek-coder holds its own or wins

Raw pattern recall on the classics. On the bluntest bugs, unguarded selfdestruct-style privileged functions, missing modifiers, the crude reentrancy, deepseek is every bit as reliable and occasionally phrases the risk more concretely. Its training clearly included plenty of Solidity.

Skepticism. deepseek flags slightly more, which cuts both ways. On one of my subtler contracts, a logic bug where a reward calculation uses a stale balance snapshot, deepseek raised a suspicious eyebrow at the right function more often than qwen, even though it rarely articulated the exact bug. As a "look here, human" pointer, that has real value. The cost is more noise on clean code: deepseek invented plausible-sounding problems on my one deliberately clean contract more often than qwen did.

Hallucination flavor. Both hallucinate, differently. qwen's failure mode is confident omission, a clean bill of health on buggy code. deepseek's is confident invention, citing a reentrancy that structurally cannot happen because there's no external call. I mildly prefer deepseek's failure mode for security triage (false alarms cost minutes, misses cost everything) but prefer qwen's for automation, where noise erodes trust in every alert.

Size beats family, and it's not close

The most consistent finding across every round of this exercise: the gap between 1.5B and 7B within the same family dwarfs the gap between qwen and deepseek at the same size.

qwen2.5-coder:1.5b, which I use daily and like a lot for quick classification and triage, is simply not a code reviewer. On these ten contracts it catches only the most textbook patterns, misses anything requiring two steps of reasoning, and frequently mangles the structured output format. The subtler planted bugs, the stale snapshot, the rounding direction, might as well be invisible to it. Same story for deepseek's smallest variants. Below roughly 7B, "which family" is the wrong question, both are too small for multi-step reasoning about state and control flow.

At 7B, both families cross a threshold where they reliably nail the canonical top-ten-style bugs in small contracts and sometimes surprise you on harder ones. Neither crosses the next threshold: consistent detection of business logic bugs, the ones that require understanding what the code is supposed to do rather than matching a known-dangerous shape. That's exactly where real audit findings live, which is why I keep saying these are triage assistants, not auditors.

The practical corollary: if you're choosing a local model for Solidity work, spend your effort getting the biggest model your hardware runs comfortably, and only then compare families. Family choice is a tiebreaker.

What I actually run

qwen2.5-coder:7b is my default for anything structured: review passes, classification, the pre-commit and repo-triage pipelines I've written about before. The instruction following is what earns it the slot. qwen2.5-coder:1.5b stays installed for fast, low-stakes filtering where a wrong answer costs nothing. deepseek-coder stays on disk as a second opinion: when a contract feels off and qwen shrugs, I run deepseek and read what it points at, treating it as a suspicion generator rather than a truth source. Two differently-wrong models disagreeing is often exactly where I should look harder, and both together still cost me zero dollars and zero data leaving my machine.

Would a wrong-but-suspicious model or a quiet-but-precise one fit your workflow better?

How Much GPU You Actually Need for Local Code Review in 2026

Pavel Espitia — Thu, 30 Jul 2026 15:56:20 +0000

Last month a guy in a security Discord asked what GPU he should buy to run code review models locally. Budget: around $1,600. I told him to try it first on the laptop he already had, and he was almost offended. He'd read enough threads to believe local AI starts at 24GB of VRAM. Three days later he messaged me again: the 7b model running on his four-year-old machine with an 8GB card was already catching the things he wanted it to catch.

That conversation happens a lot, so here's my honest breakdown after running local models for code review daily for over a year, on WSL2, on unglamorous hardware.

Review is not generation

First, an important distinction. Most benchmarks and most YouTube videos are about code generation: can the model write a working function, can it scaffold an app. Review is a different job. The code already exists. The model's task is to read it, hold it in context, and reason about what's wrong with it.

That changes the hardware math in two ways:

You care less about raw output speed. A review that takes 90 seconds is still faster than a human reviewer who gets to your PR tomorrow.
You care much more about context. A model that writes beautiful functions but can only see 200 lines of your file at a time will miss the bug that spans an import at the top and a call at the bottom.

Keep that in mind while reading the tiers below.

Tier 0: no GPU at all

Yes, this works. A 1.5b model like qwen2.5-coder:1.5b runs on CPU with a few GB of RAM. On my machine it's slow enough that you feel it, we're talking coffee-sip pauses, not real-time streaming, but it runs.

What a 1.5b model is actually good for in review:

Flagging obvious smells: unchecked return values, empty catch blocks, string-built SQL
Summarizing what a diff does before you read it yourself
Pre-filtering: "does this file touch auth or money?" so a bigger model only sees what matters

What it's not good for: multi-step reasoning. Ask it whether a reentrancy guard actually protects the function that moves funds and it will confidently say yes to almost anything. Small models are pattern matchers, not reasoners, and review quality lives in the reasoning.

If you're budget-zero, start here. It will teach you the workflow, and the workflow transfers to bigger models unchanged.

Tier 1: 8GB VRAM, the sweet spot

This is where I'd point almost everyone. An 8GB card runs a 7b coder model quantized to 4 bits with room to spare, and a 7b model is a genuinely different animal from a 1.5b one. It follows structured output instructions, it can hold an argument across a function, and when I feed it a Solidity contract it reasons about access control instead of just spotting keywords.

Most of the local pipeline in spectr-ai, my open-source contract auditor, was developed and tested against a 7b model on exactly this class of hardware. Not because I couldn't get bigger, but because a tool that requires a $2,000 GPU isn't a tool most people will run.

The practical experience at this tier: reviews of a single file complete in a comfortable time, you can leave a directory-wide scan running while you work, and the model is smart enough that its false positives are at least interesting to read.

Tier 2: 16-24GB, and where the returns diminish

With 16GB you run 14b models comfortably and 32b models quantized. Are they better at review? Yes, measurably in my experience: fewer hallucinated issues, better at cross-function reasoning, better at explaining severity.

But here's the thing nobody selling GPUs will tell you: the jump from 1.5b to 7b transforms what review tasks are possible. The jump from 7b to 32b mostly improves quality on tasks the 7b could already attempt. For generation, big models pull far ahead. For review, where the model reads more than it writes, the gap is narrower than the price gap.

If review is your main use case and you're choosing between a 24GB card and an 8GB card plus $1,000 in your pocket, I'd take the second option and spend part of that grand on cloud API calls for the hard cases.

Quantization, explained without the religion

Quantization compresses the model's weights so it fits in less memory. q8 keeps roughly full quality. q4 halves the memory again with some loss.

For code review the question is: does q4 hurt reasoning? In my daily use, on 7b models, the honest answer is "a little, sometimes." A q4 model occasionally loses the thread on longer chains of logic where q8 keeps it. But q4 is what lets a 7b model fit on an 8GB card with context to spare, and a q4 7b beats a q8 1.5b every single time. Rule of thumb: pick the largest parameter count that fits at q4 before you spend VRAM on higher precision.

One exception: if you're doing structured output (JSON findings, for example), heavier quantization seems to increase format errors slightly. I handle that with a retry loop in code rather than with more VRAM.

The real bottleneck is context, not parameters

Here's what actually limits local code review in 2026: context window memory. The KV cache, the memory the model uses to remember your input, grows with context length and it competes with the model weights for VRAM.

A 900-line contract plus your prompt plus room for the response gets big fast. On an 8GB card with a 7b model, you can raise the context, but you'll feel it. When the context you request doesn't fit, Ollama silently offloads layers to CPU and your fast review becomes a slow one, and worse, if you don't raise the default context at all, the model silently truncates your file and reviews half of it without telling you.

Check what context length you're actually getting. Set it explicitly:

ollama run qwen2.5-coder:7b
# in the session:
/set parameter num_ctx 16384

Or in the API call. And for large files, chunk deliberately (by contract, by class, by function group) instead of hoping the window is big enough. A model that fully reads 300 lines beats a model that half-reads 900.

WSL2 notes

I run everything on WSL2 and it's been solid for a long time now. Things worth knowing:

Install the NVIDIA driver on Windows only. Do not install a Linux driver inside WSL, the passthrough handles it.
Run nvidia-smi inside WSL to confirm the GPU is visible before blaming Ollama.
Run Ollama inside WSL, not on Windows, if your tooling lives in WSL. Crossing the boundary through localhost works but adds friction.
WSL caps its RAM by default. If you're doing CPU offload or running CPU-only, raise the limit in .wslconfig.

What I'd actually buy in mid-2026

If I were starting from zero today: nothing, first. Run the 1.5b on CPU for a week and learn the workflow. Then, if it sticks, a used 8GB or 12GB card, whatever's cheap in your market. Go bigger only when you can name the specific task your 7b keeps failing at, because for review, most people never hit that wall.

What's the smallest model that's earned a permanent place in your workflow?

My Local AI Stack, Mid-2026: What Survived and What I Dropped

Pavel Espitia — Wed, 29 Jul 2026 15:54:05 +0000

Six months ago I wrote up my local AI setup and a reader bookmarked it, tried to reproduce it last week, and emailed me confused because half of it no longer matched what I actually run. Fair. Stacks rot quietly. So here's the mid-2026 state of mine: what's still earning its place on disk, what I deleted, and where I quietly went back to the cloud.

Context for the numbers and opinions below: I do smart contract security work, I run everything on WSL2 on a machine with a modest GPU, and I've been doing the local-model thing daily for over a year, not as a hobby but as part of shipping.

Still here: Ollama as the runtime

Ollama remains the center of the local stack and honestly it's not close. I've tried the alternatives, llama.cpp directly for control, a couple of the newer serving layers for speed, and I keep coming back for one boring reason: the API is stable and everything I've built talks to it. My audit tooling, my shell scripts, my editor config, they all point at localhost:11434 and they've pointed there for a year without breaking.

That stability matters more than a marginal tokens-per-second win. When a model update lands, ollama pull and I'm done. The day something meaningfully better appears with the same API shape, I'll switch in an afternoon, which is exactly the position you want to be in.

Still here: qwen2.5-coder, both sizes, different jobs

I run two models and the split has stayed remarkably stable:

qwen2.5-coder:1.5b is the reflex model. It handles anything where speed matters more than depth: quick "what does this diff do" summaries, commit message drafts, pre-filtering files before a heavier pass, and the small classification jobs inside my pipelines ("does this file handle user input, yes or no"). It's fast enough on my machine that I never think about invoking it, and that's the whole point. A model you hesitate to call is a model you stop calling.

qwen2.5-coder:7b is the thinking model. Code review, security triage, structured findings extraction, explaining unfamiliar code. It's the smallest model I trust to follow a JSON schema and to hold an argument across a whole function. The first-pass local analysis in spectr-ai runs on it.

Have newer models tempted me? Constantly. I try most of the coder-tuned releases that fit my VRAM. A few scored better on paper. None have been enough better at my actual tasks to justify re-tuning the prompts and the output parsing that took months to get reliable. Switching costs are real, and "5% better in a benchmark" doesn't pay them.

Dropped: the local autocomplete experiment

This one hurt to admit. I spent a while running a local model as inline editor autocomplete, the ghost-text kind, and I turned it off around February and never missed it.

The problem wasn't quality, it was latency psychology. Autocomplete has to be near-instant or it interrupts more than it helps. On my hardware, with the models good enough to suggest something useful, completions arrived just late enough that I'd already started typing my own version. So I was paying GPU memory around the clock for suggestions I raced past. For autocomplete specifically, either you have hardware fast enough for real-time inference or you're better off without it. I write the code myself and save the models for reading it.

Dropped: the model zoo

At one point I had a lot of models on disk. General chat models, an embedding model I benchmarked once, three abandoned fine-tune experiments, a vision model I used twice. Combined, a triple-digit number of gigabytes doing nothing.

The realization: every model I actually used was one of two. Everything else was collected, not used, the AI equivalent of unread books. I did a purge, kept my two, and made a rule that a new model gets thirty days to earn permanence or it gets ollama rm. Disk space is cheap but attention isn't, and a zoo invites you to fiddle instead of work.

Dropped: local RAG over my codebase

I built an embedding-based "chat with my repos" setup, and it was fun to build and useless to use. For code, grep and ripgrep with a bit of thought beat semantic search almost every time, because in code I usually know the literal string I'm hunting for. The RAG stack added an index to maintain and returned nearby-but-wrong chunks just often enough to erode trust. Deleted without regret.

Where the cloud still wins, clearly

Here's the honest part. I'm not local-only and I don't pretend to be.

Long context. When I need a model to hold an entire protocol in its head, multiple contracts, the interactions between them, that's beyond what my hardware runs well. Claude gets those jobs.

Agentic work. Multi-step tasks where the model plans, uses tools, reads results, and adjusts course. Small local models fall apart at step three, they lose the plot or loop. Anything agent-shaped goes to the cloud without a second thought.

Final-pass judgment. My security pipeline is a funnel: local models do the wide, cheap, private first pass over everything, and the highest-signal candidates go to a frontier model for verdicts. The local layer's job is to make the expensive layer's input small.

The pattern that emerged wasn't planned, but it's clean: local models read and filter, cloud models reason and decide.

The money

Before I had local models in the loop, sending everything to cloud APIs during heavy audit work produced monthly bills that made me wince, the kind where you check the dashboard mid-month.

Now the local layer absorbs the bulk of raw token volume: every summarize, every classify, every first-pass review costs me electricity and nothing else. What's left for the cloud is a much smaller number of much harder calls, plus a subscription for interactive work. My total AI spend is a fraction of what the all-cloud version of my workflow would cost, and just as important, the cost is flat and predictable. I don't ration my own tooling anymore, and unrationed tooling gets used more, which compounds.

There's also a non-money line item: contract code under NDA never leaves my machine during the first pass. For security work, some clients care about that more than I do.

The stack, in one box

Runtime:      Ollama on WSL2
Fast model:   qwen2.5-coder:1.5b  (summaries, classification, pre-filtering)
Deep model:   qwen2.5-coder:7b    (review, structured findings, triage)
Cloud:        Claude              (long context, agents, final judgment)
Dropped:      local autocomplete, model zoo, local RAG
Rule:         new models get 30 days to earn their disk space

If there's a theme to the last six months, it's subtraction. The stack got better by getting smaller, and every deletion made the remaining pieces easier to trust.

What have you dropped from your setup this year, and do you miss it?

Structured Security Findings From a 7B Model: The Schema That Stopped Breaking

Pavel Espitia — Tue, 28 Jul 2026 16:08:43 +0000

For about two weeks last year, roughly one in five runs of my audit pipeline died on the same line: JSON.parse. The model's security analysis was often good. The JSON wrapping it was a coin flip: a trailing comma here, a severity of "pretty high" there, one memorable case where the model wrapped valid JSON in a polite paragraph explaining that it had produced JSON.

I was asking a 7b model, qwen2.5-coder:7b running on Ollama, to emit security findings in a nested schema I'd designed the way I'd design an API response. That was the mistake. Here's the schema that eventually stopped breaking, and the loop around it that catches the rest.

Why nested schemas fail on small models

My first schema looked reasonable:

{
  "findings": [
    {
      "vulnerability": {
        "type": "reentrancy",
        "details": { "function": "withdraw", "lines": [42, 57] }
      },
      "assessment": {
        "severity": { "level": "high", "confidence": 0.8 },
        "exploitability": { "requires": ["external call"], "notes": "..." }
      },
      "remediation": { "suggestion": "...", "references": [] }
    }
  ]
}

A frontier model handles this fine. A 7b model fails it in ways that took me a while to categorize:

Every level of nesting is a place to lose track. The model is closing braces from memory. Three levels deep, mid-generation, with its attention mostly on the security reasoning, it forgets whether it's inside assessment or vulnerability. Frontier models have capacity to spare for bookkeeping. Small models spend theirs on the actual task, and the structure decays.

Optional fields invite improvisation. Give a small model a details object with loosely defined contents and it will invent keys, one per run, all different.

Mixed types compound errors. A confidence float next to a string next to an array of numbers means three formatting rules active at once. Each is easy. Together, at temperature, they're where the trailing commas came from.

The fix wasn't a better parser. It was accepting that schema complexity is a budget, and a 7b model has a small one.

The flat schema

Here's what I run now, essentially unchanged for months:

import { z } from "zod";

const Severity = z.enum(["critical", "high", "medium", "low", "info"]);

const Finding = z.object({
  title: z.string().min(1).max(120),
  severity: Severity,
  category: z.enum([
    "reentrancy",
    "access-control",
    "arithmetic",
    "unchecked-call",
    "oracle",
    "logic",
    "gas",
    "other",
  ]),
  location: z.string(),        // "withdraw(), lines 42-57" as plain text
  description: z.string(),
  recommendation: z.string(),
});

const FindingsResponse = z.object({
  findings: z.array(Finding),
});

Design rules that came out of the failures:

One level of nesting, total. An array of flat objects. The model never has to remember where it is.

Enums for anything you'll branch on. Severity as free text gave me "High", "HIGH", "high-ish", and "severe". Severity as a closed enum, with the options listed in the prompt, comes back clean almost every run, and when it doesn't, validation catches it instead of my sorting logic silently misfiling a critical.

Strings for anything descriptive. I originally wanted lines as an array of numbers. The model would emit ranges, or strings, or line numbers that didn't exist. Now location is prose and I treat it as a hint for a human, which is what it actually was all along. Parse precision out of model output only where you'll act on it programmatically.

No optional fields. Everything required, every run. Uniformity is legibility for a small model.

The severity enum earns a special mention because it's load-bearing: everything downstream in spectr-ai sorts and filters on it, so it's the one field where I accept zero creativity.

Validate at the boundary, and only there

Model output is untrusted input, the same as a network request. It crosses one boundary, gets parsed and validated once, and after that the rest of the codebase works with a typed value and never second-guesses it.

type Finding = z.infer<typeof Finding>;

function parseFindings(raw: string): z.SafeParseReturnType<unknown, { findings: Finding[] }> {
  // Small models love wrapping JSON in prose or markdown fences.
  // Extract the outermost JSON object before parsing.
  const start = raw.indexOf("{");
  const end = raw.lastIndexOf("}");
  if (start === -1 || end <= start) {
    return { success: false, error: new z.ZodError([]) } as never;
  }
  const candidate = raw.slice(start, end + 1);

  let json: unknown;
  try {
    json = JSON.parse(candidate);
  } catch {
    return FindingsResponse.safeParse(undefined); // uniform failure shape
  }
  return FindingsResponse.safeParse(json);
}

The slice-to-braces trick is inelegant and it eliminates the entire "here is your JSON:" class of failure in one move. I've made peace with it.

The retry loop that feeds errors back

The piece that took reliability from good to boring: when validation fails, don't just retry, tell the model what it got wrong.

async function getFindings(code: string, maxAttempts = 3): Promise<Finding[]> {
  let feedback = "";

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const raw = await ollamaGenerate({
      model: "qwen2.5-coder:7b",
      prompt: buildPrompt(code) + feedback,
      format: "json",
      options: { temperature: 0.1 },
    });

    const result = parseFindings(raw);
    if (result.success) return result.data.findings;

    const issues = result.error.issues
      .map((i) => `- ${i.path.join(".")}: ${i.message}`)
      .join("\n");

    feedback =
      `\n\nYour previous response failed validation:\n${issues}\n` +
      `Respond with ONLY a JSON object matching the schema. No prose.`;
  }

  throw new Error(`Model output failed validation after ${maxAttempts} attempts`);
}

Details that matter in practice:

Zod's error messages are the feedback. "severity: Invalid enum value. Expected 'critical' | 'high' | ..." is exactly the correction a model needs, and I get it for free from the validator I already run. No separate error-explaining code to maintain.

Low temperature for extraction. Creativity belongs in the analysis prompt, not the formatting one. Near-zero temperature cut my malformed-output rate noticeably on its own.

Ollama's format: "json" helps but doesn't save you. It constrains output to valid JSON syntax, which kills trailing commas, but valid JSON with a wrong shape or a hallucinated severity still gets through. Syntax enforcement and schema validation solve different layers, you want both.

Three attempts, then fail loudly. Early on I had a fallback that logged a warning and returned an empty findings array. That's the worst possible behavior for a security tool: a parse failure quietly became "no vulnerabilities found." If the pipeline can't produce validated output, it should say so at full volume.

On my machine, with all of this in place, first-attempt success is the strong norm and the retry loop handles nearly everything else. Parse failures went from a daily annoyance to something I check the logs for out of curiosity.

The general lesson transfers beyond security findings: with small models, don't fight for the output format you want, negotiate down to the format they can reliably produce, then enforce it mechanically at the boundary.

What's the most cursed thing a model has handed your JSON parser?

Sandboxing Patterns for Local AI Agents With Filesystem Access

Pavel Espitia — Mon, 27 Jul 2026 16:27:50 +0000

A while back I wrote about running local AI agents on your own code, and it became the most-read thing I've published. The most common follow-up question, by a wide margin: "okay, but you gave it write access to your disk, doesn't that terrify you?"

It should, a little. The first week I had a local agent with real filesystem tools, it "cleaned up" a directory by rewriting a config file I hadn't asked it to touch. Nothing was lost, git had my back, but I sat there looking at the diff thinking: this thing was three characters away from editing .env instead. A 7b model doesn't need to be malicious to hurt you. It just needs to be confidently wrong once, with write permissions.

So this is the follow-up: the patterns I actually use to give local agents filesystem access without holding my breath. None of this is exotic. All of it is the same boring security thinking I apply to smart contracts, pointed inward.

The threat model is dumber than you think

With cloud agents people worry about prompt injection and exfiltration. Those matter locally too, especially if your agent reads untrusted files (a cloned repo can absolutely contain text aimed at your agent, I see variants of this in the wild through my repo-scanner work on Argus Lens). But for local agents the dominant risk is more mundane: the model misunderstands, and its misunderstanding is executed with your permissions.

Wrong file, right operation. Right file, too-broad operation. A path that resolves somewhere you didn't expect. Plan your defenses for confident stupidity first and malice second, and you'll cover most of both.

Pattern 1: allowlist roots, never denylist paths

The instinct is to block dangerous places: not in /etc, not in home config. Denylists fail the way they always fail, you forget a case. Invert it. The agent gets an explicit list of directories it may touch, and everything outside them is denied by default:

const ALLOWED_ROOTS = [
  "/home/pavel/projects/current-audit",
  "/tmp/agent-scratch",
];

Two roots is typical for me: the project under work, and a scratch directory the agent can mess up freely. That's it. The agent doesn't need your whole home directory any more than a contract needs an unrestricted delegatecall.

The critical implementation detail: resolve paths before checking them. projects/current-audit/../../.ssh/id_ed25519 passes a naive prefix check. Canonicalize first, then compare, and treat symlinks with suspicion because a symlink inside an allowed root can point anywhere.

Pattern 2: deny dotfiles and secrets by default, even inside allowed roots

Inside an allowed project directory there are still files the agent has no business touching. My rule: anything starting with a dot, plus known secret-bearing names, is invisible to the agent unless I explicitly grant it per session.

.env is the obvious one. Also .git (an agent that writes into .git can corrupt your repo or, worse, plant hooks), credentials files, key material. Deny reads too, not just writes: an agent that reads .env will happily paste your API key into a generated file, a commit message, or a summary that later leaves your machine.

Pattern 3: dry-run mode that prints the diff

Every write tool in my setup has a mode where it doesn't write. It prints what it would do, as a unified diff, and stops. New agent, new prompt, new model version: dry-run stays on until I've watched enough proposed changes to trust the combination.

The diff format matters. "I will update config.ts" tells you nothing. Seeing the actual before-and-after lines is what let me catch that config rewrite in week one. Cheap to build, and it converts "trust me" into "check me."

Pattern 4: read-only bind mounts for reference material

Agents often need to read things they should never write: dependency sources, a reference repo, documentation trees. Instead of adding those to the allowlist and hoping, mount them read-only:

mkdir -p /home/pavel/agent-ro/reference-repo
sudo mount --bind -o ro /home/pavel/projects/reference-repo /home/pavel/agent-ro/reference-repo

Now enforcement lives in the kernel, not in my TypeScript. Even if my wrapper has a bug, a write to that tree fails at the OS level. Defense in depth means the second layer catches what the first one misses. If you'd rather go further, running the whole agent in a container with explicit volume mounts gets you the same property plus process isolation, but the bind mount is the eighty-percent version you can set up in a minute.

Pattern 5: the blast radius checklist

Before I enable any tool for an agent, I answer five questions in writing:

What's the worst single call this tool can make?
Is that worst case reversible? (git-tracked file: yes. rm outside the repo, or a pushed commit: no.)
What does this tool get to read, and could any of it be secret?
Can output from this tool influence a later, more dangerous call? (read tool feeding a write tool means injection through file contents is on the table)
What's the narrowest scope that still does the job?

If question 2 comes back "irreversible," the tool either doesn't get enabled or gets a human-confirmation gate. This is exactly how I think about reviewing a contract's external calls, and it transfers cleanly: enumerate what can go wrong before it's live, not after.

A wrapper that enforces the policy

Here's a trimmed version of the wrapper every filesystem tool goes through. The point is the shape: one choke point where policy lives, so individual tools stay policy-free.

import { realpath } from "node:fs/promises";
import path from "node:path";

interface FsPolicy {
  allowedRoots: string[];
  deniedPatterns: RegExp[];
  dryRun: boolean;
}

const policy: FsPolicy = {
  allowedRoots: ["/home/pavel/projects/current-audit", "/tmp/agent-scratch"],
  deniedPatterns: [
    /(^|\/)\.[^/]+/,          // any dotfile or dot-directory
    /(^|\/)\.env(\.|$)/,      // .env and variants, redundant on purpose
    /id_(rsa|ed25519)/,
    /\.(pem|key)$/,
  ],
  dryRun: true,
};

async function authorize(requested: string, mode: "read" | "write"): Promise<string> {
  const resolved = await realpath(path.resolve(requested)).catch(() => {
    throw new Error(`denied: cannot resolve ${requested}`);
  });

  const inRoot = policy.allowedRoots.some(
    (root) => resolved === root || resolved.startsWith(root + path.sep),
  );
  if (!inRoot) throw new Error(`denied (${mode}): ${resolved} outside allowed roots`);

  if (policy.deniedPatterns.some((p) => p.test(resolved))) {
    throw new Error(`denied (${mode}): ${resolved} matches denied pattern`);
  }
  return resolved;
}

async function writeFileTool(requested: string, content: string): Promise<string> {
  const target = await authorize(requested, "write");
  if (policy.dryRun) {
    return `DRY RUN, would write ${content.length} bytes to ${target}:\n` +
      renderDiff(await currentContent(target), content);
  }
  await backupThenWrite(target, content);
  return `wrote ${target}`;
}

Note that realpath resolves symlinks before the root check, that denial errors go back to the model as tool results (models actually adapt when told "denied: outside allowed roots"), and that the real version backs up every file before writing because git doesn't cover untracked files.

None of this makes an agent safe in some absolute sense. What it does is bound the damage of any single bad decision to a space you've consciously chosen and can recover from. That's all sandboxing has ever been, and it's enough to let you use these tools without flinching.

Which tool in your agent setup has the biggest blast radius right now, and have you actually written it down?

Streaming Long AI Jobs to the Browser: SSE Patterns From Building an Audit Tool

Pavel Espitia — Sun, 26 Jul 2026 15:12:56 +0000

The first version of the spectr-ai web frontend had a spinner. You uploaded a contract, the spinner spun, and several minutes later results appeared, or didn't. My test users (friends, so they were honest) all did the same thing: around the ninety-second mark they refreshed the page, killing the audit that was about to finish. A spinner with no progress is indistinguishable from a hang, and users act accordingly.

So I rebuilt the pipeline around Server-Sent Events, and most of what I learned wasn't in the SSE tutorials, because tutorials stream a chat completion for ten seconds and call it a day. Multi-minute jobs with real state are a different problem. Here's what actually mattered.

Why SSE and not WebSockets

Short version: the browser never needs to talk back mid-job. The client uploads a contract, then listens. That's exactly the shape SSE was built for, one-directional server-to-client over plain HTTP. No connection upgrade, no socket lifecycle management, and the browser's EventSource gives you automatic reconnection for free, which turns out to be the most valuable feature of the whole protocol. WebSockets would work, but I'd be maintaining bidirectional machinery to use ten percent of it.

Design the event vocabulary before writing any code

My first attempt streamed whatever the pipeline felt like emitting: raw model tokens, log lines, half-thoughts. The frontend became a parser for an undocumented format that changed whenever I touched the backend. Bad.

Second attempt: a fixed vocabulary of typed events, treated as a real API contract. Four types cover everything:

type AuditEvent =
  | { type: "progress"; step: string; current: number; total: number }
  | { type: "partial-finding"; finding: Finding }   // validated, complete finding
  | { type: "done"; summary: AuditSummary }
  | { type: "error"; message: string; recoverable: boolean };

Decisions inside that shape that earned their place:

progress carries semantic steps, not percentages. "Analyzing withdraw(), function 3 of 7" keeps a user at the screen through minute four. A bar crawling from 41% to 43% doesn't, and honest percentages are impossible anyway when you don't know how long each model call takes.

partial-finding is the retention feature. The pipeline finds issues one at a time, so I ship each one the moment it's validated. Users start reading the first finding while the model is still chewing on the rest, and the perceived wait collapses even though total time is unchanged. If your long job produces incremental results, streaming them beats any progress bar you could design.

Findings stream only after validation. Early on I forwarded findings as raw model output and occasionally streamed garbage straight into the UI. Now everything passes schema validation server-side first. Stream results, never stream your parsing problem.

error distinguishes recoverable from fatal. A single model call failing means retrying and telling the user, the audit continues. Out of budget or malformed input means the job is dead. The frontend does very different things with those, so the event must say which it is.

Reconnection: the part everyone skips

Laptops sleep. Phones switch networks. Proxies kill idle connections. Over several minutes, disconnection is a certainty at scale, and EventSource will reconnect automatically, sending a Last-Event-ID header with the last event it received.

That header is only useful if you built for it. Two requirements: every event gets a monotonic ID, and the server keeps a replayable log of events per job, independent of any connection. Which forces the real architectural insight: the job must not live inside the HTTP request. The audit runs somewhere durable and appends events to a log. The SSE endpoint is just a cursor over that log:

// app/api/audits/[id]/events/route.ts
export async function GET(req: Request, { params }: { params: { id: string } }) {
  const lastId = Number(req.headers.get("last-event-id") ?? -1);
  const job = await getJob(params.id);
  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      const send = (id: number, e: AuditEvent) =>
        controller.enqueue(
          encoder.encode(`id: ${id}\nevent: ${e.type}\ndata: ${JSON.stringify(e)}\n\n`),
        );

      // 1. Replay whatever this client missed
      for (const { id, event } of await job.eventsAfter(lastId)) send(id, event);

      // 2. Then follow the live log
      const unsubscribe = job.subscribe((id, event) => {
        send(id, event);
        if (event.type === "done" || (event.type === "error" && !event.recoverable)) {
          unsubscribe();
          controller.close();
        }
      });

      // 3. Heartbeat comment so proxies don't kill an idle connection
      const heartbeat = setInterval(
        () => controller.enqueue(encoder.encode(`: hb\n\n`)),
        15_000,
      );
      req.signal.addEventListener("abort", () => {
        clearInterval(heartbeat);
        unsubscribe();
      });
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
    },
  });
}

Replay-then-follow is the whole trick. A client that drops at event 12 reconnects, replays 13 through wherever the job is now, and continues live. Refresh the page mid-audit and you lose nothing. The heartbeat comment line (SSE ignores lines starting with :) keeps intermediaries from declaring the connection dead during a long model call, and no-transform stops well-meaning proxies from buffering your stream into uselessness.

Serverless will fight you on this

Everything above assumes the job outlives the request, and default serverless assumes the opposite. A function that dies at its time limit takes your four-minute audit with it, and "keep the function alive by streaming" only stretches the ceiling, it doesn't remove it, and now a client disconnect can kill the job for every other viewer of that audit.

What I landed on: separate the worker from the stream. The job runs in something with a long lifetime (a worker process, a queue consumer, a container, whatever your platform offers), writes events to shared storage, and the SSE function does nothing but read and forward. SSE endpoints become cheap and stateless, the job becomes durable, and reconnection falls out naturally because the log is the source of truth. If you're on a platform where a long-lived worker is genuinely unavailable, chunk the pipeline into resumable stages and accept the added complexity, but know that you're paying it to avoid a worker, not because streaming requires it.

Backpressure, or when the model outruns the browser

Local models on a decent GPU can emit events faster than a busy tab renders them, especially token-level progress. Two things saved me:

Coalesce chatty events server-side. Nobody needs 40 progress updates a second. I batch progress events on a short interval and send only the latest state per tick. Findings are never coalesced, every one ships. Classify your events as "latest value wins" or "every one matters" and throttle only the first kind.

Respect the stream's own signals. controller.enqueue piles into a buffer if the consumer is slow, and unbounded buffering on a multi-minute job is a slow memory leak. Check controller.desiredSize before enqueueing low-priority events and drop stale progress ticks when it goes negative. The client that skipped some progress frames catches up instantly at the next one, and nobody notices.

The pattern that ties all of this together: treat the event log as the product and the SSE connection as a disposable view of it. Every hard problem (reconnection, serverless limits, backpressure, even multiple tabs watching one audit) gets easy once the connection stops being where state lives.

What's the longest-running job you've had to keep a browser honest about, and what broke first?

A Typosquat Package Almost Got My Keys: Dissecting the Attack Safely

Pavel Espitia — Sat, 25 Jul 2026 15:12:20 +0000

A recruiter DM led to a "take-home" repo. Standard stuff, or so it looked. I cloned it, opened package.json, and one line stopped me: a dependency named clx-cookieparser. The real package is cookie-parser. That extra clx- prefix and the missing hyphen were the whole attack. I never installed it. Here is how I read it apart without running a single line, and the checklist that has saved me twice now.

I do smart contract security, but the boring truth is most attacks against developers do not touch the chain at all. They touch your machine, your environment variables, and your wallet files. This one wanted all three.

The bait

The repo was framed as a coding assessment for a "client." Clean README, a couple of real features, tests that passed. The kind of thing you skim and trust because it looks like work, not like a trap. That framing is the point. You are in "let me finish this task" mode, not "let me audit a stranger's code" mode.

The dependency list had mostly normal packages. Express, a test runner, a couple of utilities. And then clx-cookieparser, sitting in the middle like it belonged. Typosquatting works because your eyes autocorrect. You read "cookie parser," your brain fills in the canonical name, and you move on.

I did not move on, mostly out of habit. I keep npm configured so that installing is not a one-command reflex, which buys me time to look before anything executes.

Why I could read it without running it

Two settings do the heavy lifting here, and I recommend both to everyone:

# never run install/postinstall scripts automatically
npm config set ignore-scripts true

# with pnpm, add a cooldown so brand-new versions can't hit you instantly
pnpm config set minimumReleaseAge 1440   # 24 hours

ignore-scripts matters because the classic move is a postinstall hook that fires the moment you run install. Turn that off and cloning plus reading is safe, because nothing runs on its own. The cooldown matters because a lot of these malicious versions get yanked within hours of publication once someone reports them, so a 24-hour delay quietly dodges the freshest poison.

So I read. Reading is not running. That distinction is the entire safety model.

What the payload was shaped like

I want to be careful here, so I am describing structure, not handing anyone a recipe. No real payload code.

The package had two layers. The first layer, the code visible in the published tarball, was almost boring. It wrapped a real cookie-parsing function so the thing actually worked if you used it. That is camouflage. If the library breaks your app, you rip it out and the attacker loses. If it works, you keep it and stop looking.

The interesting part was a second dependency the first package pulled in, and here is the tell that made my neck prickle: that second dependency was present in the resolved dependency tree but did not appear in the lockfile the repo shipped. In other words, the code referenced a package that the committed lockfile did not account for. A supply chain that does not reconcile with its own lockfile is lying to you about something.

That second-stage package is where the real behavior lived. Reading its structure (not executing it), the intent was obvious from the surface it reached for:

It looked for environment variables and process env, the place people stash API keys, tokens, and RPC URLs.
It probed common wallet and keystore file locations in the home directory.
It assembled that data and prepared to ship it outbound to a remote endpoint.

Two-stage is a deliberate design. Stage one is quiet and passes a casual glance. Stage two carries the theft and hides behind an install-time or first-run trigger, one step removed from the package you actually named in your file. You have to follow the thread to see it.

The signals that gave it away

I did not need a fancy tool for the first pass. I needed to read like the code was guilty until proven innocent. The signals, roughly in the order they hit me:

The name. clx-cookieparser is not cookie-parser. Any prefix, swapped hyphen, or singular/plural flip on a popular package name is a five-alarm reason to stop.
Lockfile mismatch. A dependency resolving in the tree that the committed lockfile does not describe means the manifest and reality disagree. Legitimate projects reconcile.
Obfuscation and entropy. The second stage had chunks of high-entropy strings, the dense base64-looking blobs and hex that normal utility code just does not carry. Human-written cookie parsing is low entropy and readable. A wall of encoded bytes is a place to hide behavior from a reader.
Reach that does not match purpose. A cookie parser has no business reading your home directory or touching env beyond what it is handed. Capability that exceeds the stated job is intent.

For the entropy check you do not need to run anything either. You can eyeball it, or score strings statically. A rough sketch of the idea:

import math
from collections import Counter

def shannon_entropy(s: str) -> float:
    if not s:
        return 0.0
    counts = Counter(s)
    n = len(s)
    return -sum((c / n) * math.log2(c / n) for c in counts.values())

# high entropy long string literals in a "utility" package are a smell
for line in open("suspicious.js"):
    for token in line.split('"'):
        if len(token) > 120 and shannon_entropy(token) > 4.5:
            print("high-entropy blob:", token[:40], "...")

That is reading, scoring, and flagging. Never executing.

The checklist that saved me

This is the routine now, and it takes maybe five minutes on a fresh repo:

Read package.json dependencies out loud. Any name that is close-but-not-exact to a popular package gets verified against the real registry name before anything else.
Diff the dependency tree against the lockfile. Anything present in one and missing from the other is a stop sign.
Never let install scripts run by default. ignore-scripts true globally, opt in per project only when you trust it.
Grep for high-entropy string literals and unexpected network or filesystem calls in dependencies before install, not after.
Assume any repo from an unsolicited recruiter is hostile until proven otherwise. The social framing is part of the exploit.

I ended up feeding the repo through Argus Lens, the static repo scanner I built (lens.noctis.biz) partly because I got tired of doing this by hand. It flags build-time execution, lockfile-missing dependencies, and obfuscation without cloning or installing. But the tool just automates the checklist above. The checklist is the actual defense, and you can run it with your eyes.

I did not lose anything this time. The uncomfortable part is how close normal, get-the-task-done behavior came to running it. One npm install on autopilot and I would have shipped my env vars to someone.

What is your default posture when a stranger sends you a repo to run: trust and clone, or hostile until proven safe?

I'm Entering My First Sherlock Audit Contest: The Setup, the Plan, the Fear

Pavel Espitia — Fri, 24 Jul 2026 20:25:15 +0000

I have shipped software for 18 years. Roughly 8 of those in crypto. And this week I signed up for my first Sherlock audit contest, and my hands were a little cold when I clicked in. That gap tells you something about how competing in public feels versus building things.

I write secure code for a living. I built spectr-ai, an open-source AI smart contract auditor. I have read more Solidity than I can remember. None of that is the same as putting my name on a public leaderboard next to people who do this full time and win five figures per contest. So this post is me being honest about the setup, the plan, and the fear, before I have any results to brag about or hide.

Why a contest at all

There are a few ways to make money auditing. Firms hire you. Bug bounties pay you when you find a live bug. And contest platforms like Sherlock run time-boxed competitions where a protocol puts its code in scope, a pile of auditors (Watsons, in their language) hunt for bugs at the same time, and the prize pool gets split based on what you find.

The model is the part that pulled me in. Payouts are severity based. A valid High is worth more than a valid Medium, and the pot for a given issue gets shared among everyone who found it. If five people report the same High, they split that issue's reward. So the incentive is not just to find bugs, it is to find the bugs other people miss.

There is also a gate that matters if you care about ranking, not just cash. To climb from the entry tier you generally need to land 2 valid issues and clear a points threshold (around 20% of the top performer on a contest, from what I have read). That is a real bar. It stops the leaderboard from filling up with people who got lucky once. It also means my goal for the first contest is not "win." It is "submit 2 issues I can fully defend." Everything else is noise.

How I prepared

I did not open the code first. I spent the first two days reading old contest reports. Sherlock publishes them, and they are gold. You get to see what actually counted as a High, how judges reasoned about severity, which reports got downgraded to informational and why, and the shape of the mistakes that recur across protocols.

A few patterns jumped out fast:

Rounding and precision bugs in share/asset math show up constantly.
Access control that looks fine until you trace who can call an internal admin path through a proxy.
Oracle assumptions that hold on mainnet but break on an L2 or during sequencer downtime.
Reentrancy that is not the classic kind, but cross-function or cross-contract.

Reading reports also recalibrated my sense of what a "real" finding looks like. A lot of things I would flag in a code review ("this could be clearer," "add a check here") are not valid contest issues unless I can show funds at risk or an invariant broken. Severity is about impact plus likelihood, not tidiness.

The workflow: AI proposes, Foundry proves

Here is the rule I set for myself, and I am not breaking it: nothing gets submitted without a working Foundry proof of concept.

I use LLMs heavily, but as an idea generator, not an oracle. I run Ollama locally on WSL2 with qwen2.5-coder for the cheap fast passes, and I reach for a bigger model when a path looks promising. The loop looks like this:

I feed the model a contract plus the invariants I wrote down (more on that in another post) and ask it to enumerate attack paths, not to "find bugs." Open-ended "is this safe" prompts produce garbage. Specific "how could an attacker make totalAssets diverge from the sum of balances" prompts produce leads.
The model gives me candidate attacks. Most are wrong. Some are hallucinated functions that do not exist. I throw those out immediately.
For anything that survives, I write a Foundry test that either triggers the bug or fails to. If I cannot make it fail, it is not a finding.

A skeleton PoC looks like this:

// test/PocShareInflation.t.sol
pragma solidity ^0.8.20;

import "forge-std/Test.sol";
import {Vault} from "../src/Vault.sol";
import {MockToken} from "./mocks/MockToken.sol";

contract PocShareInflation is Test {
    Vault vault;
    MockToken token;
    address attacker = makeAddr("attacker");
    address victim = makeAddr("victim");

    function setUp() public {
        token = new MockToken();
        vault = new Vault(address(token));
        token.mint(attacker, 100 ether);
        token.mint(victim, 100 ether);
    }

    function test_firstDepositorInflation() public {
        // attacker deposits 1 wei, mints 1 share
        vm.startPrank(attacker);
        token.approve(address(vault), type(uint256).max);
        vault.deposit(1);
        // then donates directly to inflate share price
        token.transfer(address(vault), 50 ether);
        vm.stopPrank();

        // victim deposits and gets rounded down to 0 shares
        vm.startPrank(victim);
        token.approve(address(vault), type(uint256).max);
        vault.deposit(50 ether);
        vm.stopPrank();

        assertEq(vault.balanceOf(victim), 0, "victim minted zero shares");
        // attacker withdraws everything, including victim funds
    }
}

If that assert holds, I have something. If it does not, the model was wrong and I move on. The PoC is also what I paste into the report, because a judge should be able to run forge test --match-test test_firstDepositorInflation and watch it happen.

This is the same discipline I use everywhere. spectr-ai flags things, but a flag is a hypothesis, not a verdict. The PoC is the verdict.

The fear, handled honestly

The impostor feeling is real, and pretending it is not would be dishonest. Eighteen years of shipping does not translate to "I will place well against people who audit for a living." Those are different sports.

What settled me down was reframing the goal. I am not trying to top the board. I am trying to submit two issues I can defend line by line, learn how the judging actually works from the inside, and read every other Watson's report after the contest closes to see what I missed. The reports of a closed contest are the best paid course in the space, except it is free.

There is also a floor to the downside. The worst case is I submit nothing valid, lose a week, and learn a lot from the post-contest reports. That is a cheap tuition. The upside is a payout and a data point that I can actually do this.

I picked the contest this week. I am not naming it because I do not want to color anyone else's read of the same code, and honestly because I want to talk about method here, not about one protocol. Next posts will be about how I read the scope and how I turn invariants into findings.

If you have competed on a contest platform, what is the one thing you wish someone had told you before your first one?