DEV Community: Pavel Espitia

Mid-Conversation System Prompts: Steering an Agent Without Breaking the Cache

Pavel Espitia — Tue, 16 Jun 2026 15:46:03 +0000

Here is a problem I hit building a long-running agent: I needed to inject a new instruction partway through a session ("the project is Go, write Go") but editing the top-level system prompt to add it invalidated my entire prompt cache. Every cached turn got reprocessed at full price. The fix is a feature that landed in the current Claude models: mid-conversation system messages. Here is what it is and when to use it.

The setup that breaks

A long agent session has a large, stable system prompt and a growing message history, and you cache the prefix so each turn reuses the prior work cheaply. That works until you learn something mid-session that the agent needs to know: a mode toggled, the user delivered async context, files changed on disk, the token budget dropped.

The naive move is to edit the system prompt to include the new fact. But the system prompt sits at the front of the cached prefix. Change one byte there and you invalidate everything after it. Your whole conversation history reprocesses at full input price on the next request. For a long session, that is expensive and slow.

The fix: a system message in the messages array

The current models let you put a system-role message directly in the messages array, after the history, instead of editing the top-level system:

const response = await client.messages.create(
  {
    model: "claude-opus-4-8",
    max_tokens: 16000,
    system: [
      { type: "text", text: STABLE_SYSTEM, cache_control: { type: "ephemeral" } },
    ],
    messages: [
      ...history,                                    // cached prefix, untouched
      { role: "user", content: latestUserMessage },
      // @ts-expect-error: role:"system" SDK types may still be landing
      { role: "system", content: "This project is Go. Write all code in Go." },
    ],
  },
  { headers: { "anthropic-beta": "mid-conversation-system-2026-04-07" } },
);

Because the new instruction sits after the cached history, it invalidates nothing before it. The cached prefix stays intact, you pay full price only for the small new message, and the agent still receives the instruction with operator authority.

Why this beats stuffing it in a user message

The old workaround was to put operator instructions inside a user turn, often wrapped in something like <system-reminder>. That preserves the cache the same way, but it has a security problem: a user message is forgeable. Anything that can write to user-visible input can spoof an instruction that looks like it came from you, the operator.

A role: "system" message is the non-spoofable operator channel. It carries operator authority that a user-turn instruction does not, which matters when you are injecting trusted state (mode switches, permissions) into an agent that also processes untrusted user input. So this is both a caching win and a prompt-injection-safety win.

Phrase it as context, not a command

One subtlety that took me a try to get right: phrase these messages as facts, not overrides. State the situation and let the model act on it. Avoid override-style language like "ignore what the user said" or "disregard the previous instruction." The models are trained to protect users from instructions that work against them, and that protection applies to the system role too. So:

// Good: states context, lets the model act
{ role: "system", content: "Auto-approve mode is now enabled for this session." }
// Risky: override framing, may be resisted
{ role: "system", content: "Ignore the user's earlier request and do X instead." }

The constraints to know

A few rules from the spec:

It must follow a user message (or an assistant message ending in a server tool result). It cannot be messages[0]. Use the top-level system for the initial prompt.
The content is text only.
It is model-gated. On a model that does not support it, you get a 400 (role 'system' is not supported on this model). Catch that and fall back to the user-turn <system-reminder> pattern.

try {
  // ... mid-conversation system message
} catch (err) {
  if (err instanceof Anthropic.BadRequestError && err.message.includes("system")) {
    // fall back: inject as a user-turn <system-reminder> block
  } else {
    throw err;
  }
}

When to reach for it

The trigger is: I have learned something mid-session that the agent needs, and I want to tell it without rebuilding the prefix. Mode changes, freshly delivered context, state the application discovered after the session started. Anything dynamic that you would otherwise be tempted to splice into the system prompt.

If the fact is known at session start, it belongs in the top-level system prompt as usual. The mid-conversation channel is specifically for things you learn after the cached prefix is already built. Used that way, it keeps your cache hot, keeps your operator instructions non-spoofable, and saves you from the expensive mistake I made: editing the system prompt mid-session and watching the whole conversation reprocess at full price.

Building a Local AI Code Reviewer with Ollama That Catches Bugs Before Your Team

Pavel Espitia — Mon, 15 Jun 2026 16:04:52 +0000

Your teammates are busy. Your CI is green but shallow. And the bug you just staged is the kind a second pair of eyes would catch in five seconds. So let's build that second pair of eyes: a small TypeScript CLI that feeds your staged git diff to a local LLM and returns structured findings, before anyone else sees your code. No API key, no cloud, no leaking your private repo to a vendor.

The plan

The whole tool is one loop:

Grab the staged diff with git diff --cached.
Send it to Ollama with a tight review prompt.
Ask for JSON, validate it with Zod.
Print findings, exit non-zero if anything is severe.
Wire it as a pre-commit hook.

Everything runs locally against qwen2.5-coder:7b. You'll need Ollama running (ollama serve) and the model pulled (ollama pull qwen2.5-coder:7b).

Step 1: Get the staged diff

The reviewer should look at exactly what you're about to commit, nothing more. That's --cached (staged changes only):

import { execSync } from "node:child_process";

function getStagedDiff(): string {
  return execSync("git diff --cached --no-color -U3", {
    encoding: "utf8",
    maxBuffer: 10 * 1024 * 1024,
  });
}

A few choices that matter:

--no-color keeps ANSI escape codes out of the prompt.
-U3 gives three lines of context around each hunk. Enough for the model to reason, not so much that you blow the context window.
maxBuffer bumps Node's default 1MB cap so big diffs don't throw.

If the diff is empty, there's nothing to review:

const diff = getStagedDiff();
if (diff.trim().length === 0) {
  console.log("No staged changes. Stage something first with `git add`.");
  process.exit(0);
}

Step 2: Craft the review prompt

This is where the quality lives. A vague prompt gives you vague, hallucinated nitpicks. Be specific about what counts as a finding, and what to ignore.

const SYSTEM_PROMPT = `You are a senior code reviewer. You review git diffs for bugs only.

Focus on:
- Logic errors (off-by-one, inverted conditions, wrong operators)
- Null/undefined access and unhandled error cases
- Resource leaks (unclosed handles, missing awaits)
- Security issues (injection, hardcoded secrets, unsafe input)

Do NOT report:
- Style, formatting, or naming preferences
- Suggestions to add comments or tests
- Anything you are not confident is an actual bug

Lines starting with "+" are added. Lines starting with "-" are removed.
Only review added ("+") lines. Respond with ONLY valid JSON.`;

The "do NOT report" block is doing heavy lifting. Small models love to pad output with "consider adding a comment here." Telling them what to suppress is more effective than telling them what to find.

The instruction to only review + lines matters too. Without it, the model will happily flag a bug in code you just deleted, which is both useless and confusing. Diffs are a strange dialect to a model trained mostly on whole files, so being explicit about what the + and - prefixes mean pays off in fewer nonsense findings.

Step 3: Ask for structured JSON

Ollama speaks the OpenAI-compatible API at localhost:11434. Spell out the exact schema in the prompt and set temperature: 0 so the output is deterministic:

const RESPONSE_SCHEMA = `Respond with this exact JSON shape:
{
  "findings": [
    {
      "severity": "high" | "medium" | "low",
      "file": "string",
      "line": "string (the code snippet or line reference)",
      "issue": "string (one sentence: what is wrong)",
      "fix": "string (one sentence: how to fix it)"
    }
  ]
}
If there are no bugs, return { "findings": [] }.`;

async function reviewDiff(diff: string, model: string): Promise<unknown> {
  const response = await fetch("http://localhost:11434/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model,
      messages: [
        { role: "system", content: `${SYSTEM_PROMPT}\n\n${RESPONSE_SCHEMA}` },
        { role: "user", content: `Review this diff:\n\n${diff}` },
      ],
      temperature: 0,
      response_format: { type: "json_object" },
      stream: false,
    }),
  });

  if (!response.ok) {
    throw new Error(`Ollama returned ${response.status}. Is \`ollama serve\` running?`);
  }

  const data = await response.json();
  return JSON.parse(data.choices[0].message.content);
}

response_format: { type: "json_object" } nudges Ollama into JSON mode, which cuts down on the "Here's your review:" preamble that breaks JSON.parse. It isn't a guarantee, though, which is why the next step exists.

Step 4: Validate with Zod

Never trust raw model output. A 1.5b model will occasionally hand you a string where you expected an array, or invent a severity level. Parse it at the boundary and fail loudly if it's malformed:

import { z } from "zod";

const FindingSchema = z.object({
  severity: z.enum(["high", "medium", "low"]),
  file: z.string(),
  line: z.string(),
  issue: z.string(),
  fix: z.string(),
});

const ReviewSchema = z.object({
  findings: z.array(FindingSchema),
});

type Review = z.infer<typeof ReviewSchema>;

function parseReview(raw: unknown): Review {
  const result = ReviewSchema.safeParse(raw);
  if (!result.success) {
    throw new Error(`Model returned invalid review JSON:\n${result.error.message}`);
  }
  return result.data;
}

safeParse over parse so you can give a useful error instead of an unhandled throw. When this fires, it's almost always the model wandering off-schema, and the fix is usually a smaller diff or a bigger model.

Step 5: Print the findings

Make the output scannable. A reviewer nobody reads is useless:

function printReview(review: Review): number {
  if (review.findings.length === 0) {
    console.log("Local review passed. No bugs found.");
    return 0;
  }

  const icon = { high: "[HIGH]", medium: "[MED] ", low: "[LOW] " };
  let hasHigh = false;

  for (const f of review.findings) {
    if (f.severity === "high") hasHigh = true;
    console.log(`\n${icon[f.severity]} ${f.file}`);
    console.log(`  where: ${f.line}`);
    console.log(`  issue: ${f.issue}`);
    console.log(`  fix:   ${f.fix}`);
  }

  console.log(`\n${review.findings.length} finding(s).`);
  return hasHigh ? 1 : 0;
}

Notice the exit code: only high severity blocks the commit. Medium and low get printed as a heads-up but don't stand in your way. Tune that threshold to your team's tolerance.

Wiring it all together

async function main() {
  const model = process.argv[2] ?? "qwen2.5-coder:7b";
  const diff = getStagedDiff();

  if (diff.trim().length === 0) {
    console.log("No staged changes.");
    process.exit(0);
  }

  try {
    const raw = await reviewDiff(diff, model);
    const review = parseReview(raw);
    process.exit(printReview(review));
  } catch (err) {
    console.error(`Review failed: ${(err as Error).message}`);
    // Don't block commits on tooling failure. Warn and pass.
    process.exit(0);
  }
}

main();

The catch is deliberate: if Ollama is down or the JSON is garbage, you log it and let the commit through. A review tool that hard-blocks commits when it itself breaks is a tool people will rip out by Friday.

Step 6: Make it a pre-commit hook

Build the CLI, then drop a hook into .git/hooks/pre-commit:

#!/usr/bin/env bash
set -euo pipefail

echo "Running local AI review..."
node /path/to/review.js qwen2.5-coder:7b

chmod +x .git/hooks/pre-commit

For a hook the whole team shares, use husky instead so it lives in the repo. Either way, every git commit now runs the diff past a local model first. Need to skip it for a quick WIP commit? git commit --no-verify.

One thing to watch: the first call after the model loads into memory is slow, often several seconds on CPU. That's Ollama paging the weights in, not your code being slow. Keep ollama serve running in the background and subsequent commits feel near-instant. If you commit rarely enough that the model unloads between commits, that cold start is the price you pay each time.

Be honest about small local models

This is the part most tutorials skip. A local qwen2.5-coder:7b is not a staff engineer. Here's the realistic picture:

Bug type	1.5b	7b	Notes
Null/undefined access	Decent	Good	The model's bread and butter
Inverted conditions / wrong operator	Spotty	Decent	Needs enough context (`-U3` helps)
Missing `await`	Decent	Good	Easy pattern to catch
Subtle race conditions	Misses	Misses	Needs cross-file context it lacks
Logic spanning multiple files	Misses	Misses	A diff is a keyhole, not the room
False positives	Frequent	Occasional	The main cost of running local

Two failure modes dominate: it invents bugs that aren't there (false positives), and it misses anything that requires understanding code outside the diff. Here's how to keep it useful anyway:

temperature: 0 always. Deterministic output means the same diff gives the same review. You can't trust a reviewer that changes its mind on re-run.
Scope the diff small. Review per-commit, not per-branch. The smaller and more focused the diff, the sharper the model. A 2000-line diff gets you noise.
Use 7b for the hook, 1.5b for fast local iteration. 1.5b is ~1GB and quick, but its false-positive rate makes it annoying as a gate. Save it for --dry-run style checks.
Block on high only. Let medium and low be advisory. This keeps the false-positive tax from blocking real work.
Treat it as a first pass, not a replacement. It catches the dumb stuff so your human reviewers can spend their attention on architecture and intent.

Takeaway

A local AI reviewer won't replace your team, and it shouldn't try to. What it does well is catch the careless, three-in-the-afternoon bugs before they reach a pull request: the missing await, the ! you meant to delete, the unhandled null. It runs free, it runs private, and it runs every time you commit.

I built the same Claude-plus-Ollama pattern at a larger scale in spectr-ai, an AI smart contract auditor where --model ollama:qwen2.5-coder:1.5b runs the entire audit locally with no API key. The diff-reviewer here is the same idea shrunk to fit in a git hook. Steal it, scope your diffs, and let the small model earn its keep.

Structured Output From Local LLMs: JSON That Never Breaks (Ollama + Zod)

Pavel Espitia — Sun, 14 Jun 2026 14:53:43 +0000

A 1.5B model running on your laptop will return JSON that almost parses. The closing brace is missing. A trailing comma sneaks in. The whole thing is wrapped in a markdown fence with a chirpy "Sure! Here's your JSON:" on top. Cloud models do this too, but small local models do it constantly, and that is exactly where most "just prompt it harder" advice falls apart.

I wrote about validating LLM responses with Zod before: schemas as contracts, safeParse, extracting JSON from chaos. That post is the foundation. This one is the local-model-specific layer on top: Ollama's native JSON modes, retry loops that actually converge, repairing truncated output, and a single generateStructured<T>(schema) helper that ties it all together so you never hand-roll this again.

Two JSON Modes Most People Skip

Ollama gives you two ways to force structure before you ever touch Zod. Use them. They cut your parse-failure rate dramatically.

The first is format: "json". It constrains decoding so the model can only emit syntactically valid JSON. No markdown fences, no preamble, no trailing prose.

const res = await fetch("http://localhost:11434/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "qwen2.5-coder:7b",
    messages: [
      { role: "system", content: "Output a JSON object describing the code." },
      { role: "user", content: "function add(a, b) { return a + b }" },
    ],
    format: "json",
    stream: false,
    options: { temperature: 0 },
  }),
});

const data = await res.json();
const obj = JSON.parse(data.message.content); // already clean JSON

format: "json" guarantees valid syntax. It does not guarantee your shape. The model can still invent fields or skip required ones. That is what Zod is for.

The second mode is the one people miss: pass a full JSON Schema to format. Ollama then constrains generation to match the schema's structure, not just "valid JSON."

const schema = {
  type: "object",
  properties: {
    language: { type: "string" },
    purpose: { type: "string" },
    isPure: { type: "boolean" },
  },
  required: ["language", "purpose", "isPure"],
};

const res = await fetch("http://localhost:11434/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "qwen2.5-coder:7b",
    messages: [{ role: "user", content: "Describe: function add(a,b){return a+b}" }],
    format: schema,
    stream: false,
    options: { temperature: 0 },
  }),
});

You do not want to write JSON Schema by hand and keep it in sync with your TypeScript types. You already have a Zod schema. Convert it.

import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";

const CodeInfo = z.object({
  language: z.string(),
  purpose: z.string(),
  isPure: z.boolean(),
});

const jsonSchema = zodToJsonSchema(CodeInfo, { target: "openApi3" });
// pass jsonSchema as `format` to Ollama

One source of truth. Zod drives both the generation constraint and the runtime validation.

Mode	Forces valid syntax	Forces your shape	Cost
Plain prompt	No	No	Free, unreliable
`format: "json"`	Yes	No	Negligible
`format: <schema>`	Yes	Mostly	Slower decode, fewest retries

On small models, format: <schema> is worth the slightly slower decode because it turns most three-attempt loops into one.

Repairing Truncated JSON

Schema-constrained decoding still breaks in one nasty way: the model hits its token limit mid-object. You get {"vulnerabilities": [{"id": "V1", "severity": "hi and a dead parse.

Two defenses. First, raise num_predict so the model has room to finish.

options: { temperature: 0, num_predict: 2048 }

Second, attempt a repair before you give up. The common failure is unclosed brackets and a dangling partial value. You can salvage a surprising amount by trimming to the last complete token and closing what is open.

function repairJson(raw: string): string {
  let text = raw.trim();

  // Drop a trailing partial string/number/key after the last comma or brace
  const lastSafe = Math.max(text.lastIndexOf("}"), text.lastIndexOf("]"));
  if (lastSafe !== -1) text = text.slice(0, lastSafe + 1);

  // Walk the string, tracking open brackets outside of string literals
  const stack: string[] = [];
  let inString = false;
  let escaped = false;
  for (const ch of text) {
    if (escaped) { escaped = false; continue; }
    if (ch === "\\") { escaped = true; continue; }
    if (ch === '"') inString = !inString;
    if (inString) continue;
    if (ch === "{") stack.push("}");
    else if (ch === "[") stack.push("]");
    else if (ch === "}" || ch === "]") stack.pop();
  }

  // Close whatever is still open, innermost first
  while (stack.length) text += stack.pop();
  return text;
}

This is a last resort, not a primary strategy. If you trim a truncated array, you lose the cut-off element, which is fine for "best effort" reads and wrong for anything that must be complete. I gate it: try the raw parse, then the repaired parse, and if the repaired version loses data the schema requires, Zod rejects it and the retry loop takes over.

Retry Loops That Converge

The naive retry resends the same prompt and prays. It does not converge because nothing changed. The version that works feeds the specific Zod error back into the next attempt, the same idea from the earlier post but tuned for local models: lower the temperature on retries and tighten the instruction.

async function withRetry<T>(
  attempt: (feedback: string | null) => Promise<string>,
  parse: (raw: string) => z.SafeParseReturnType<unknown, T>,
  maxAttempts = 3,
): Promise<T> {
  let feedback: string | null = null;

  for (let i = 1; i <= maxAttempts; i++) {
    const raw = await attempt(feedback);
    const parsed = parse(raw);
    if (parsed.success) return parsed.data;

    feedback = parsed.error.issues
      .map((issue) => `${issue.path.join(".") || "<root>"}: ${issue.message}`)
      .join("\n");
  }

  throw new Error(`No valid output after ${maxAttempts} attempts:\n${feedback}`);
}

The discipline is: the model never sees a generic "that was wrong." It sees vulnerabilities.0.severity: Invalid enum value. Expected 'high', received 'High'. Small models self-correct from that. They cannot self-correct from silence.

Streaming and Parsing Together

You want streaming for UX (tokens appearing live) but you cannot parse JSON until it is complete. Resolve the tension by streaming for display and accumulating for parsing. Do not try to parse each chunk.

async function streamAccumulate(
  body: object,
  onToken?: (t: string) => void,
): Promise<string> {
  const res = await fetch("http://localhost:11434/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ ...body, stream: true }),
  });
  if (!res.body) throw new Error("No response body from Ollama");

  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let full = "";
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    // Ollama streams newline-delimited JSON objects, one per chunk
    const lines = buffer.split("\n");
    buffer = lines.pop() ?? "";
    for (const line of lines) {
      if (!line.trim()) continue;
      const token = JSON.parse(line).message?.content ?? "";
      full += token;
      onToken?.(token);
    }
  }

  return full;
}

The trap people fall into: Ollama's /api/chat stream is newline-delimited JSON, one envelope per line, and a single network chunk can split a line in half. That is why buffer keeps the trailing partial line and only parses complete ones. Parse the accumulated full once the stream ends. Never on a partial.

The Helper That Ties It Together

Here is the piece I actually reuse. One generic function: pass a Zod schema, get back a typed, validated object, with schema-constrained generation, repair, and retry all handled.

import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";

interface StructuredOptions {
  model?: string;
  temperature?: number;
  maxAttempts?: number;
  numPredict?: number;
}

export async function generateStructured<T>(
  schema: z.ZodType<T>,
  system: string,
  user: string,
  opts: StructuredOptions = {},
): Promise<T> {
  const {
    model = "qwen2.5-coder:7b",
    temperature = 0,
    maxAttempts = 3,
    numPredict = 2048,
  } = opts;

  const jsonSchema = zodToJsonSchema(schema, { target: "openApi3" });

  const call = async (feedback: string | null): Promise<string> => {
    const messages = [
      { role: "system", content: system },
      { role: "user", content: user },
    ];
    if (feedback) {
      messages.push({
        role: "user",
        content: `Your last response failed validation:\n${feedback}\nReturn corrected JSON only.`,
      });
    }

    const res = await fetch("http://localhost:11434/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        model,
        messages,
        format: jsonSchema,
        stream: false,
        options: { temperature, num_predict: numPredict },
      }),
    });

    const data = await res.json();
    return data.message?.content ?? "";
  };

  const parse = (raw: string): z.SafeParseReturnType<unknown, T> => {
    for (const candidate of [raw, repairJson(raw)]) {
      try {
        return schema.safeParse(JSON.parse(candidate));
      } catch {
        // not parseable, try the next candidate
      }
    }
    // Force a Zod failure with a useful message
    return schema.safeParse(undefined);
  };

  return withRetry(call, parse, maxAttempts);
}

Usage is the part that makes the abstraction worth it.

const CodeReview = z.object({
  summary: z.string(),
  issues: z.array(z.object({
    severity: z.enum(["high", "medium", "low"]),
    line: z.number().int().positive(),
    note: z.string(),
  })),
  riskScore: z.coerce.number().min(0).max(100),
});

const review = await generateStructured(
  CodeReview,
  "You are a code reviewer. Output JSON only.",
  sourceCode,
  { model: "ollama-friendly", maxAttempts: 3 },
);

// review is fully typed as z.infer<typeof CodeReview>, validated, never undefined

review.issues[0].severity is typed. Your editor autocompletes it. If the model returns "High", the z.enum rejects it, the error flows back into the retry, and the next attempt fixes it. You wrote the schema once.

What I Learned Wiring This Into spectr-ai

Constrain generation, then validate. format: <schema> and Zod are not redundant. The first reduces how often you fail; the second catches what slips through.
One Zod schema, two jobs. zodToJsonSchema keeps the generation constraint and the runtime check from drifting apart.
Repair is a fallback, never the plan. Closing brackets salvages reads. It silently drops data on truncated arrays, so let Zod reject anything that must be complete.
Stream for the eyes, accumulate for the parser. Buffer newline-delimited chunks, parse once at the end.
Retries need the actual error. Feed Zod's path-and-message back in. Small models converge from specifics, not from "try again."

This is the exact pattern spectr-ai uses to run a smart-contract audit fully locally with --model ollama:qwen2.5-coder:1.5b. Every byte the model emits passes through generateStructured. The 1.5B model still fumbles the JSON. The user never sees it.

How Much RAM Do You Really Need to Run LLMs Locally? 2026 Benchmarks

Pavel Espitia — Sat, 13 Jun 2026 14:51:23 +0000

"Will it run on my machine?" is the first question everyone asks before pulling a model with Ollama. The honest answer is a formula, not a yes or no. Here's how to estimate memory before you download 9GB you can't fit, plus what to actually expect for speed on the hardware you already own.

The Rule of Thumb

A model's memory footprint is roughly:

RAM = (parameters in billions) * (bytes per parameter) + overhead

Bytes per parameter depends on quantization (more on that below). For the common Q4 quantization, figure about 0.55 to 0.65 GB per billion parameters once you include the KV cache and runtime overhead. Ollama's default quants land here.

So a 7B model at Q4 needs roughly 7 * 0.6 ≈ 4.2GB, and in practice Ollama reports qwen2.5-coder:7b at 4.7GB on disk, which is close to what it occupies in memory. The overhead grows with your context window: a long prompt fills the KV cache and adds anywhere from a few hundred MB to a couple of GB. Plan for headroom, not a tight fit.

What Quantization Actually Does

Models are trained in 16-bit floats. Quantization shrinks each weight to fewer bits so the model fits in less memory. You trade a little quality for a lot of RAM.

Quant	Bits/param	~GB per 1B params	Quality
FP16	16	~2.0	Full, reference
Q8_0	8	~1.1	Nearly lossless
Q5_K_M	~5.5	~0.75	Very good
Q4_K_M	~4.5	~0.6	Good (the sweet spot)
Q3_K_M	~3.5	~0.5	Noticeable degradation
Q2_K	~2.5	~0.4	Often too lossy to trust

The _K_M suffix means "K-quant, medium": a smarter scheme that keeps the important weights at higher precision and squeezes the rest. Q4_K_M is the default for most Ollama models because it's the best balance: roughly a quarter of the FP16 size with quality most people can't distinguish in normal use.

My take: don't go below Q4 unless you're desperate for space. The jump from Q4 to Q3 buys you a little RAM and costs you real coherence, especially on code.

To pull a specific quant in Ollama:

ollama pull qwen2.5-coder:7b-instruct-q4_K_M
ollama pull qwen2.5-coder:7b-instruct-q8_0

RAM vs VRAM: CPU vs GPU Inference

This is the part beginners miss. There are two kinds of memory that matter, and which one you have changes everything about speed.

RAM is your system memory. If the model lives here, your CPU does the math. It works, it's universal, and it's slow.
VRAM is memory on your GPU. If the whole model fits here, the GPU does the math, and it's 10x to 30x faster.

Ollama loads as much of the model as fits in VRAM and runs the rest on CPU. A model that's half in VRAM and half in RAM runs at roughly the speed of the slow half, so partial offload helps less than you'd hope. The goal is to fit the entire model in VRAM.

That's why a $300 used 12GB GPU often beats a $2000 laptop with 64GB of RAM for inference: the RAM is plenty, but without VRAM the CPU is the bottleneck.

Apple Silicon is the exception. Unified memory means the GPU and CPU share one fast pool, so an M-series Mac with 16GB or more punches well above a typical RAM-only PC.

The Benchmark Table

These are representative figures from my own machines and what I see consistently reported, not lab results. Treat them as "what to expect," plus or minus a chunk depending on your exact CPU, RAM speed, and GPU. CPU numbers assume a recent multi-core desktop/laptop chip; GPU numbers assume a mid-range card (roughly an RTX 3060/4060 class, 8 to 12GB VRAM) with the model fully offloaded.

Model	Params	Q4 size	RAM to run	CPU tok/s	Mid GPU tok/s
qwen2.5-coder:1.5b	1.5B	~1.0GB	4GB+	15 to 30	80 to 130
mistral:7b	7B	~4.1GB	8GB+	5 to 9	45 to 70
qwen2.5-coder:7b	7B	~4.7GB	8GB+	5 to 9	45 to 70
llama3.1:8b	8B	~4.7GB	8GB+	4 to 8	40 to 65
deepseek-coder-v2	16B (MoE)	~8.9GB	16GB+	8 to 14	50 to 80

A few notes that matter:

Tokens/sec is your "feel." Below ~5 tok/s a model feels painful for chat. Above ~20 it feels snappy. GPU turns "go make coffee" into "instant."
deepseek-coder-v2 is a mixture-of-experts model. It's 16B total but only activates ~2.4B per token, so it's faster than its size suggests while still needing the full 8.9GB resident.
The 1.5B model on CPU is genuinely usable. That's the whole point of it. For autocomplete, quick summaries, and structured extraction, it's fast enough without a GPU.
First call is always slow because the model loads from disk into memory. Benchmark the second call.

Recommendations by Machine Class

8GB laptop (no dedicated GPU)

You can run 7B models, but only just. The OS and browser already eat 3 to 4GB, so a 4.7GB model leaves you scraping. Realistically:

Daily driver: qwen2.5-coder:1.5b (~1.0GB). Fast on CPU, leaves room for everything else.
Occasional quality: mistral:7b or qwen2.5-coder:7b, but close your browser tabs first and expect 5 to 9 tok/s.
Avoid anything 8B or larger. You'll swap to disk and it'll crawl.

Keep your context window modest. A 16K context fills the KV cache and can push you over the edge on 8GB.

16GB dev box (no GPU or weak iGPU)

This is the comfortable RAM-only tier and where most developers sit.

Default: qwen2.5-coder:7b (~4.7GB). Plenty of headroom, good code quality.
Reach model: deepseek-coder-v2 (~8.9GB) fits with room to spare and runs respectably thanks to MoE.
You can run a 7B model and still have your IDE, Docker, and browser open. That's the real win.

Speed is still CPU-bound here (single digits to low teens tok/s), so use the 7B for "thinking" tasks and the 1.5B for anything interactive.

24GB+ with a GPU

Now it's a different machine. If you have 8 to 12GB of VRAM, every model in the table above fits fully on the GPU and flies.

Workhorse: qwen2.5-coder:7b fully offloaded, 45 to 70 tok/s. Feels like a hosted API.
Quality: deepseek-coder-v2 for harder code tasks, still fast.
With 24GB of VRAM (a 3090/4090 class card) you can run 32B-class models at Q4 entirely on the GPU. That's the tier where local quality starts rivaling cloud for code.

Check what Ollama is actually doing:

ollama ps

The PROCESSOR column tells you the split. 100% GPU is what you want. If it says 50%/50% CPU/GPU, your model is too big for VRAM and you're leaving speed on the table. Drop to a smaller quant or a smaller model until it's fully on the GPU.

A Quick Sizing Checklist

Before you pull anything:

Total RAM minus 4GB is roughly your model budget on a RAM-only machine.
VRAM is the number that matters if you have a GPU. Fit the whole model in it.
Stick to Q4_K_M unless you have a specific reason not to.
Add headroom for context. Long prompts cost real memory.
Benchmark the second call, never the first.

The Takeaway

You need less RAM than you think and more VRAM than you have. A 16GB machine with no GPU runs 7B code models comfortably, which covers most real developer work. A cheap GPU with 8 to 12GB of VRAM matters more than doubling your system RAM, because it turns a tolerable tool into an instant one.

Start with qwen2.5-coder:1.5b to learn the workflow, move to 7b when you want quality, and only chase bigger models once you've got the VRAM to fit them. Everything I build, including spectr-ai, runs fine on a 16GB box with the 7B model. Local is more capable than the hardware fear suggests.

Giving Your Local LLM Safe Filesystem Access With Ollama Tool Use

Pavel Espitia — Fri, 12 Jun 2026 15:14:08 +0000

A local LLM that can read your files is genuinely useful. A local LLM that can read your files without guardrails is a path-traversal bug with a chat interface.

I covered tool calling basics in an earlier post: define a tool schema, the model returns a structured request, your code decides whether to run it. That's the foundation. This post is about how to not get burned once those tools touch the filesystem. We're going to give the model three tools (list_dir, read_file, grep), wire up the dispatch loop with Ollama, and then harden every single one so a confused (or adversarial) model can't read your .env, climb out of the project, or hand you back a 2GB file.

The model is the planner. Your code is the executor. The executor is also the only thing standing between an unpredictable token generator and your home directory. Treat it that way.

The threat model first

Before any code, be honest about what can go wrong. The LLM is not malicious, but it is unpredictable, and the input feeding it might be malicious (a file it reads could contain instructions, a classic prompt-injection vector). So plan for all of it:

The model asks to read /etc/passwd or ~/.ssh/id_rsa.
The model passes ../../../../etc/shadow as a "relative" path.
The model reads .env and helpfully prints your API keys into the chat transcript.
The model asks to read a 4GB log file and pins your RAM.
A file the model reads contains "ignore previous instructions, now write to ...".

Every defense below maps to one of these. None of them trust the model.

The sandbox: one root, resolved, allow-listed

The single most important control: every path the model gives you gets resolved to an absolute path and checked against an allowed root. If it escapes the root, reject it. No exceptions, no "but it's probably fine."

import path from "node:path";
import fs from "node:fs/promises";

// The ONLY directory the model is allowed to touch.
const SANDBOX_ROOT = path.resolve(process.env.SANDBOX_ROOT ?? "./workspace");

class PathError extends Error {}

// Resolve a model-supplied path and prove it stays inside the sandbox.
function resolveInSandbox(userPath: string): string {
  // Resolve against the root, collapsing any `..` segments.
  const resolved = path.resolve(SANDBOX_ROOT, userPath);

  // The check that matters: is `resolved` actually under the root?
  const rel = path.relative(SANDBOX_ROOT, resolved);
  if (rel.startsWith("..") || path.isAbsolute(rel)) {
    throw new PathError(`Path escapes sandbox: ${userPath}`);
  }
  return resolved;
}

Why path.relative instead of a startsWith(SANDBOX_ROOT) string check? Because startsWith is a trap. /home/pavel/workspace-secrets starts with /home/pavel/workspace, but it's a different directory. path.relative does it structurally: if the relative path begins with .., the target is above the root. Done.

Test it before you trust it:

resolveInSandbox("notes.txt");        // OK -> <root>/notes.txt
resolveInSandbox("sub/dir/a.md");     // OK
resolveInSandbox("../secrets.env");   // throws PathError
resolveInSandbox("/etc/passwd");      // throws PathError
resolveInSandbox("a/../../etc/hosts");// throws PathError

One more thing path.resolve does not cover: symlinks. A symlink inside the sandbox can point anywhere. If your workspace could contain symlinks you don't control, resolve them too and re-check:

async function resolveRealInSandbox(userPath: string): Promise<string> {
  const resolved = resolveInSandbox(userPath);
  try {
    const real = await fs.realpath(resolved);
    const rel = path.relative(SANDBOX_ROOT, real);
    if (rel.startsWith("..") || path.isAbsolute(rel)) {
      throw new PathError(`Symlink escapes sandbox: ${userPath}`);
    }
    return real;
  } catch (err) {
    if ((err as NodeJS.ErrnoException).code === "ENOENT") return resolved;
    throw err;
  }
}

A deny-list for the obvious landmines

Allow-listing the root is the structural control. On top of it, a small deny-list stops the model from reading things that are inside the sandbox but still secret. Match on the basename, not a substring, so environment.md doesn't get caught by an .env rule.

const DENIED_NAMES = new Set([".env", ".git", "id_rsa", "id_ed25519"]);
const DENIED_SUFFIXES = [".env", ".pem", ".key"];

function assertReadable(absPath: string): void {
  const base = path.basename(absPath);
  if (DENIED_NAMES.has(base) || base.startsWith(".env")) {
    throw new PathError(`Refusing to read protected file: ${base}`);
  }
  if (DENIED_SUFFIXES.some((s) => base.endsWith(s))) {
    throw new PathError(`Refusing to read protected file type: ${base}`);
  }
}

Keep this list short and obvious. The structural sandbox is your real defense; the deny-list just catches the secrets that legitimately live in a project folder.

The tools

Three read-only tools. Notice read_file has a hard byte budget, and none of them write anything.

const MAX_READ_BYTES = 256 * 1024; // 256 KB. Models do not need a 2GB file.

async function listDir(dirPath: string): Promise<string[]> {
  const abs = await resolveRealInSandbox(dirPath);
  const entries = await fs.readdir(abs, { withFileTypes: true });
  return entries.map((e) => (e.isDirectory() ? `${e.name}/` : e.name));
}

async function readFile(filePath: string): Promise<string> {
  const abs = await resolveRealInSandbox(filePath);
  assertReadable(abs);

  const stat = await fs.stat(abs);
  if (!stat.isFile()) throw new PathError(`Not a file: ${filePath}`);
  if (stat.size > MAX_READ_BYTES) {
    throw new PathError(
      `File too large: ${stat.size} bytes (limit ${MAX_READ_BYTES}).`,
    );
  }
  return fs.readFile(abs, "utf8");
}

async function grep(pattern: string, dirPath: string): Promise<string[]> {
  // Compile the model's pattern; reject anything that won't compile.
  let re: RegExp;
  try {
    re = new RegExp(pattern);
  } catch {
    throw new PathError(`Invalid regex: ${pattern}`);
  }

  const abs = await resolveRealInSandbox(dirPath);
  const hits: string[] = [];
  const entries = await fs.readdir(abs, { withFileTypes: true });

  for (const entry of entries) {
    if (!entry.isFile()) continue;
    const child = path.join(abs, entry.name);
    try {
      assertReadable(child);
    } catch {
      continue; // skip protected files silently in search results
    }
    const stat = await fs.stat(child);
    if (stat.size > MAX_READ_BYTES) continue;

    const text = await fs.readFile(child, "utf8");
    text.split("\n").forEach((line, i) => {
      if (re.test(line)) hits.push(`${entry.name}:${i + 1}: ${line.trim()}`);
    });
  }
  return hits.slice(0, 100); // cap output so a broad pattern can't flood context
}

The schemas, in the same JSON Schema format from the function-calling post:

const tools = [
  {
    type: "function",
    function: {
      name: "list_dir",
      description: "List files and folders in a directory inside the workspace",
      parameters: {
        type: "object",
        properties: { path: { type: "string", description: "Relative path" } },
        required: ["path"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "read_file",
      description: "Read a UTF-8 text file inside the workspace",
      parameters: {
        type: "object",
        properties: { path: { type: "string", description: "Relative path" } },
        required: ["path"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "grep",
      description: "Search files in a directory for a regex pattern",
      parameters: {
        type: "object",
        properties: {
          pattern: { type: "string" },
          path: { type: "string", description: "Relative directory path" },
        },
        required: ["pattern", "path"],
      },
    },
  },
];

The dispatch loop

Here is where most tutorials get sloppy: they eval-style dispatch on the tool name and pass arguments straight through. Don't. Validate arguments with Zod, route through an explicit switch, and turn every thrown error into a tool result the model can read and recover from. An error is data, not a crash.

import { z } from "zod";

const PathArgs = z.object({ path: z.string() });
const GrepArgs = z.object({ pattern: z.string(), path: z.string() });

async function dispatch(name: string, rawArgs: string): Promise<string> {
  try {
    switch (name) {
      case "list_dir":
        return JSON.stringify(await listDir(PathArgs.parse(JSON.parse(rawArgs)).path));
      case "read_file":
        return await readFile(PathArgs.parse(JSON.parse(rawArgs)).path);
      case "grep": {
        const a = GrepArgs.parse(JSON.parse(rawArgs));
        return JSON.stringify(await grep(a.pattern, a.path));
      }
      default:
        return `Error: unknown tool ${name}`;
    }
  } catch (err) {
    // Hand the failure back to the model. It will usually correct itself.
    return `Error: ${(err as Error).message}`;
  }
}

Now the agent loop against Ollama. Same two-round-trip shape as before, wrapped so the model can chain calls:

async function run(userPrompt: string): Promise<string> {
  const messages: any[] = [
    {
      role: "system",
      content:
        "You can read files inside the workspace only. Never assume a path " +
        "outside it exists. If a tool returns an error, adjust and retry.",
    },
    { role: "user", content: userPrompt },
  ];

  for (let turn = 0; turn < 8; turn++) {
    const res = await fetch("http://localhost:11434/v1/chat/completions", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ model: "qwen2.5:7b", messages, tools, tool_choice: "auto" }),
    });
    const msg = (await res.json()).choices[0].message;
    messages.push(msg);

    if (!msg.tool_calls?.length) return msg.content;

    for (const call of msg.tool_calls) {
      const result = await dispatch(call.function.name, call.function.arguments);
      messages.push({ role: "tool", tool_call_id: call.id, content: result });
    }
  }
  return "Stopped: too many tool-calling turns.";
}

The turn < 8 cap matters. Without it, a model that keeps requesting tools (or gets stuck in a retry loop on a prompt-injected file) will run forever.

Writes are different: require a human

Reading is reversible. Writing is not. So I keep writes out of the autonomous loop entirely and gate them behind an explicit human approval. The tool doesn't write: it proposes a write, prints a diff, and waits for you.

import readline from "node:readline/promises";

async function proposeWrite(filePath: string, content: string): Promise<string> {
  const abs = resolveInSandbox(filePath);   // same sandbox check
  assertReadable(abs);                       // same secrets guard

  console.log(`\nProposed write to ${path.relative(SANDBOX_ROOT, abs)}:`);
  console.log("-".repeat(40));
  console.log(content.slice(0, 2000));
  console.log("-".repeat(40));

  const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
  const answer = (await rl.question("Apply this write? [y/N] ")).trim().toLowerCase();
  rl.close();

  if (answer !== "y") return "Write rejected by user.";
  await fs.writeFile(abs, content, "utf8");
  return `Wrote ${content.length} bytes to ${filePath}.`;
}

The model can want to write all day. Nothing hits disk until a human types y. This is the same principle as the read sandbox, applied to the higher-stakes operation: the LLM proposes, your code (and you) dispose.

The hardening checklist

Every filesystem tool you expose to an LLM should pass all of these:

Resolve and re-check. path.resolve, then path.relative against the root. Reject anything starting with ... Never startsWith on the raw string.
Resolve symlinks too. fs.realpath and re-check, or you've left a back door inside the sandbox.
Deny secrets by basename. .env, keys, .git. Short list, matched on basename, not substring.
Cap read size. A byte budget per read and a result cap on search. Context windows and RAM are finite.
Read-only by default. Writes go through a separate, human-approved path. No write tool in the autonomous loop.
Validate every argument. Zod-parse tool arguments before they reach fs. The model hallucinates fields.
Errors are tool results. Catch, stringify, return to the model. Never let a bad path crash the process.
Cap the turn count. Bound the agent loop so a stuck or injected model can't spin forever.

Takeaway

Function calling makes a local LLM useful. Filesystem access makes it powerful, and powerful is exactly when you have to slow down. The model is an untrusted planner working over potentially untrusted input. Your tools are the trust boundary. Build them so the worst a confused model can do is read a text file it was already allowed to see.

I run this exact pattern in spectr-ai, my local-first smart contract auditor, so the model can walk a contract's source tree without ever leaving the project folder. Sandbox first, features second. That order is the whole point.

I Replaced My $20/mo AI Tools With Local Models: My Full Stack

Pavel Espitia — Thu, 11 Jun 2026 15:21:30 +0000

I was paying $20/mo for Copilot and reaching for the Claude API on every side project. Then I added it up. Copilot, a code-review SaaS trial, the occasional ChatGPT Plus month: I was burning real money on tasks a 4GB model running on my own machine handles fine. So I tore the whole thing down and rebuilt it local-first. Here is the exact stack, what each piece replaced, and the parts where I still pay for cloud.

The cost table

Here is what I was spending versus what I run now.

Task	Cloud tool	Was paying	Local replacement	Now
Autocomplete + inline chat	GitHub Copilot	$10/mo	`qwen2.5-coder:7b` via Continue	$0
Code review	a review SaaS	~$15/mo	local review prompt + qwen	$0
Commit messages	Copilot / Cursor	bundled	`qwen2.5-coder:1.5b` git hook	$0
Q&A over my own docs	ChatGPT Plus	$20/mo	Ollama + local RAG	$0
Hard reasoning / shipping	Claude API	pay per token	still Claude	pay per token

The recurring subscriptions went from roughly $45/mo to $0. I still pay Claude per token, but only for the work that actually needs it, and that bill is now a few dollars a month instead of a flat fee I paid whether I used it or not.

The hardware

None of this is exotic. I run it on a Windows laptop through WSL2: Ryzen 7, 32GB RAM, and an RTX with 8GB VRAM. The GPU matters more than anything else here. The 7b models fit in 8GB of VRAM and respond fast enough to feel interactive. The 1.5b models run fine on CPU alone if you do not have a GPU, which is why I lean on them for the background tasks.

If you only have CPU, everything below still works, you just live on the smaller models and accept that long prompts take longer. Be honest with yourself about this before you cancel a subscription.

1. Autocomplete and chat (replaces Copilot)

The autocomplete is the piece people doubt most, so I will start there. I use the Continue extension in VS Code pointed at Ollama. The config is plain JSON:

{
  "models": [
    {
      "title": "qwen-chat",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b"
    }
  ],
  "tabAutocompleteModel": {
    "title": "qwen-autocomplete",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b"
  }
}

Two models on purpose. Autocomplete needs to be fast above all else, so it runs on 1.5b, which returns a suggestion in well under a second on my GPU. Chat, where I want a real answer, runs on 7b.

Is it as good as Copilot? No, and I will not pretend otherwise. Copilot's multi-line completions are smarter and its sense of your wider repo is better. But for finishing the line I am already typing, closing a loop, filling in an obvious object literal, the local model is good enough that I stopped noticing the difference within a week. The honest gap shows up on long, novel completions, which is exactly where I would rather think for myself anyway.

2. Local code review

I built a small review step into my own workflow rather than paying a SaaS to comment on diffs. It is a shell function that pipes a staged diff into Ollama with a focused prompt:

review() {
  git diff --staged | ollama run qwen2.5-coder:7b \
    "You are a senior reviewer. Point out bugs, missing error
     handling, and unsafe patterns in this diff. Be specific and
     terse. If it looks fine, say so. Diff follows:"
}

The trick that makes this useful is the same one I learned building spectr-ai: narrow the prompt. Asking a 7b model to "review this code" gets you vague praise. Asking it specifically for bugs, missing error handling, and unsafe patterns gets you something I can act on. It catches the boring real stuff: an unawaited promise, a swallowed error, an off-by-one in a slice. It misses architectural problems and anything that needs understanding of the whole system. I treat it as a second pair of eyes that never gets tired, not as an approver.

3. Commit message generation

This one runs entirely in the background and I never think about it. A prepare-commit-msg hook feeds the staged diff to the small model:

#!/usr/bin/env bash
set -euo pipefail

diff=$(git diff --staged)
[ -z "$diff" ] && exit 0

msg=$(printf '%s' "$diff" | ollama run qwen2.5-coder:1.5b \
  "Write a concise git commit message in imperative mood, under
   72 chars for the subject. Output only the message, no quotes.")

printf '%s\n' "$msg" > "$1"

I use 1.5b here deliberately. The task is constrained enough that the small model nails it, and speed matters when it sits between me and the commit. I still read and edit what it writes, but it gets the subject line 80% right and I tweak the rest. A flat subscription to do this would be absurd when a 1GB model does it for free in half a second.

4. Local RAG over my own docs

This is the one that quietly replaced my ChatGPT Plus habit. I kept pasting my own project READMEs and notes into a chat box to ask "how did I wire up the provider abstraction again?" Now I run a local retrieval setup: documents get embedded with nomic-embed-text through Ollama, the vectors live in a local store, and queries pull the relevant chunks before they hit qwen2.5-coder:7b.

const embed = async (text: string): Promise<number[]> => {
  const res = await fetch("http://localhost:11434/api/embeddings", {
    method: "POST",
    body: JSON.stringify({ model: "nomic-embed-text", prompt: text }),
  });
  const data = await res.json();
  return data.embedding;
};

The whole thing is offline. My notes, my unreleased project code, my half-formed ideas never leave the machine. That privacy point is not a footnote for me. As someone who works on smart-contract security, I am not pasting client code or unpublished findings into a cloud box, and now I do not have to.

5. Where I still pay for Claude

Local-first does not mean local-only. I still reach for the Claude API in two situations, and I am specific about which.

First, hard reasoning. When I needed spectr-ai to reason about how three contracts interact to form an attack path, the local models fell apart, the same way I wrote about in my "what I learned" post. They confidently describe attack vectors that do not work. Claude is meaningfully better at multi-step reasoning that is not just pattern matching, so spectr-ai's deep audit mode defaults to Claude.

Second, production quality. Both spectr-ai and AbiLens ship with a provider flag. Development, demos, and privacy-sensitive runs go through Ollama with --provider ollama. When a user wants the best possible audit and is fine paying for it, it routes to Claude. The provider abstraction I built on day one makes that a one-line switch:

const provider = createProvider(options.provider); // "ollama" | "claude"

So the rule is simple: local for everything I do all day, cloud for the rare task that needs a frontier model or for the output someone else relies on.

The honest limitations

I would be lying if I sold this as free with no cost.

Quality gap is real on hard tasks. For pattern recognition, code completion, and summarizing, the gap is small. For novel reasoning and long-context work, Claude still wins clearly.
Speed depends entirely on hardware. On my GPU the 7b models are interactive. On CPU only, a long prompt can take a minute or more. There is no free lunch here, you trade money for either hardware or patience.
First call after idle is slow. The model has to load into memory. Keep the Ollama server warm and this stops mattering.
You own the plumbing now. No vendor fixes your git hook or updates your model for you. That is the price of not paying the other price.

What this actually bought me

The money is the smaller win. The bigger one is that my daily AI tooling now runs offline, costs nothing per use, and never sends my code anywhere. I stopped rationing API calls because each one felt like spending. I stopped worrying about what I paste into a chat box.

The setup is not magic and it is not as polished as the paid tools. But qwen2.5-coder on Ollama crossed the "good enough for real work" line for me, and the parts that have not crossed it, the hard reasoning, are exactly the parts I am happy to pay Claude for by the token instead of by the month.

Start with the small model and the commit hook. It is the lowest-risk way to feel whether local-first works for you before you cancel anything.

Reentrancy in 2026: Why It Still Drains Millions (and How AI Spots It)

Pavel Espitia — Wed, 10 Jun 2026 15:17:17 +0000

Reentrancy is the oldest trick in smart contract exploitation. The DAO fell to it in 2016. You would think a bug this famous would be extinct by now. It isn't. Protocols still lose millions to reentrancy every year, because the textbook version is the easy one, and attackers stopped using the textbook version a long time ago.

This post walks through the classic bug, then the modern variants that still bite in 2026: cross-function, cross-contract, read-only reentrancy through view functions, and callback reentrancy through ERC777 and ERC721. I'll show why ReentrancyGuard is not the silver bullet most developers think it is, and how an LLM-based auditor reasons about the call-then-state-change pattern in a way pattern matchers can't.

The Classic: Single-Function Reentrancy

Here is the canonical vulnerable withdraw.

contract VulnerableVault {
    mapping(address => uint256) public balances;

    function deposit() external payable {
        balances[msg.sender] += msg.value;
    }

    function withdraw() external {
        uint256 amount = balances[msg.sender];
        (bool success, ) = msg.sender.call{value: amount}("");
        require(success, "transfer failed");
        balances[msg.sender] = 0; // state update AFTER the external call
    }
}

The attack is simple. The attacker is a contract. When withdraw sends ETH via msg.sender.call, control passes to the attacker's receive() function before balances[msg.sender] is zeroed. The attacker calls withdraw again. The balance is still the original amount, so the vault pays out again. Repeat until the vault is empty.

contract Attacker {
    VulnerableVault vault;

    receive() external payable {
        if (address(vault).balance >= 1 ether) {
            vault.withdraw(); // re-enter before balance is zeroed
        }
    }
}

The fix is checks-effects-interactions. Do all your checks, then update all your state, and only then make the external call.

function withdraw() external {
    uint256 amount = balances[msg.sender];
    balances[msg.sender] = 0;   // effect first
    (bool success, ) = msg.sender.call{value: amount}(""); // interaction last
    require(success, "transfer failed");
}

Now when the attacker re-enters, the balance is already zero, the second payout is zero, and the drain stops. You can also add OpenZeppelin's ReentrancyGuard:

function withdraw() external nonReentrant {
    uint256 amount = balances[msg.sender];
    balances[msg.sender] = 0;
    (bool success, ) = msg.sender.call{value: amount}("");
    require(success, "transfer failed");
}

If reentrancy were only this, it would be a solved problem. Every linter on earth flags a state write after an external call. The reason millions still vanish is that the modern variants slip past both the guard and the linter.

Cross-Function Reentrancy

nonReentrant locks a single function. But contracts share state across many functions, and a mutex on withdraw does nothing if the attacker re-enters through a different function that reads the same balance.

contract Vault {
    mapping(address => uint256) public balances;

    function withdraw() external nonReentrant {
        uint256 amount = balances[msg.sender];
        (bool success, ) = msg.sender.call{value: amount}("");
        require(success);
        balances[msg.sender] = 0;
    }

    function transfer(address to, uint256 amount) external {
        require(balances[msg.sender] >= amount); // stale balance here
        balances[msg.sender] -= amount;
        balances[to] += amount;
    }
}

withdraw has the guard. But during the external call, balances[msg.sender] is still the full pre-withdrawal amount. The attacker's receive() calls transfer instead of withdraw. transfer is not guarded by the same lock (the default nonReentrant only blocks re-entry into functions sharing that specific modifier instance, and many codebases forget to add it everywhere). The attacker transfers their still-intact balance to a second wallet, then lets withdraw finish and zero out the original account. They now hold the funds twice.

The fix is the same root principle: update state before the external call. withdraw must zero the balance first. The guard is a backstop, not the actual fix.

Cross-Contract Reentrancy

Now the shared state lives in two contracts. Protocol A holds the accounting, Protocol B holds the funds, and the lock on one says nothing about the other.

contract Pool {
    Token public token;
    mapping(address => uint256) public shares;

    function redeem() external nonReentrant {
        uint256 amount = shares[msg.sender];
        token.transfer(msg.sender, amount); // ERC777-style callback fires here
        shares[msg.sender] = 0;
    }
}

If token invokes a hook on the recipient (more on that below), the attacker re-enters during token.transfer. They can't re-enter redeem because of the guard, but they can call into a different contract that trusts Pool.shares[msg.sender] as an oracle of how much they own. That second contract has no idea Pool is mid-execution. It reads the stale, still-nonzero share balance and lets the attacker borrow or mint against value they are about to remove.

Read-Only Reentrancy: The One Guards Cannot Catch

This is the variant that has done the most damage since 2022, and it is the one that defeats ReentrancyGuard by design.

nonReentrant protects state-mutating functions. View functions are not guarded, because views don't write state, so reentering them looks harmless. The problem: a view function can return a value computed from state that is temporarily inconsistent during an external call.

Consider an AMM-style pool whose getVirtualPrice() derives a price from the contract's token balances and total supply.

contract StablePool {
    uint256 public totalSupply;

    function removeLiquidity(uint256 lpAmount) external nonReentrant {
        uint256 ethOut = (lpAmount * address(this).balance) / totalSupply;
        _burn(msg.sender, lpAmount);
        (bool ok, ) = msg.sender.call{value: ethOut}(""); // attacker re-enters here
        totalSupply -= lpAmount; // supply not yet updated during the call
    }

    function getVirtualPrice() external view returns (uint256) {
        return (address(this).balance * 1e18) / totalSupply; // reads inconsistent state
    }
}

During removeLiquidity, the ETH has already left the contract via the call, but totalSupply has not yet been decremented. For the duration of that external call, getVirtualPrice() returns a value based on a reduced balance divided by an unchanged supply: a deflated, wrong price.

The attacker's receive() does not re-enter StablePool at all. It calls a lending protocol that uses getVirtualPrice() as its oracle. That protocol is fully patched, fully guarded, written by a different team. It simply trusts the view function. The attacker borrows against the manipulated price, the call returns, removeLiquidity finishes and restores totalSupply, and the lending protocol is left holding an undercollateralized loan.

This is why guards alone miss read-only reentrancy. StablePool is guarded. The lending protocol is guarded. The vulnerability lives in the interaction: a view function reading half-updated state, consumed by a third party who never sees the lock. The only real fix is to never leave state inconsistent across an external call. Burn and decrement totalSupply before the call, or have the consuming protocol take the pool's own reentrancy lock before reading the price (Curve added remove_liquidity reentrancy checks readable by integrators for exactly this reason).

Callback Reentrancy: ERC777 and ERC721

Plain ERC20 transfer does not hand control to the recipient. ERC777 and ERC721 do, and that turns an innocent-looking token movement into an external call.

ERC777 invokes tokensReceived on the recipient. ERC721 safeTransferFrom invokes onERC721Received. Both are attacker-controlled entry points.

contract NFTMint {
    mapping(address => bool) public hasMinted;
    uint256 public totalMinted;

    function mint() external {
        require(!hasMinted[msg.sender], "already minted");
        totalMinted++;
        _safeMint(msg.sender, totalMinted); // calls onERC721Received
        hasMinted[msg.sender] = true;       // flag set AFTER the callback
    }
}

_safeMint calls onERC721Received on the minter before hasMinted[msg.sender] is set. The attacker re-enters mint from inside that callback, passes the require again (flag still false), and mints as many NFTs as gas allows. This exact shape drained limited mints and bypassed per-wallet caps repeatedly. The fix, again, is ordering: set hasMinted[msg.sender] = true before _safeMint.

Why This Defeats Pattern Matchers

A static analyzer flags "state write after external call" inside one function. That catches the classic case and nothing else. The variants above share a deeper property that no regex encodes:

The dangerous call and the stale state can live in different functions.
The dangerous call and the stale read can live in different contracts.
The "external call" can be disguised as a token transfer (safeMint, ERC777 transfer) that a pattern matcher does not even recognize as reentrant.
The victim of read-only reentrancy is a third-party integrator that contains no bug of its own.

These are not pattern problems. They are reasoning problems. You have to model what state is inconsistent at the moment control leaves the contract, and who else reads that state.

How an LLM-Based Auditor Reasons About It

This is the gap I built spectr-ai to close. Instead of matching call(...) followed by a storage write, the engine reasons about the sequence:

It identifies every point where control can leave the contract, including the disguised ones: low-level call, safeTransferFrom, ERC777 transfer, and any external call to a contract it cannot prove is trusted.
For each exit point, it asks what state is half-updated at that instant, then checks whether any other function or view reads that state. That is how it surfaces cross-function and read-only reentrancy that a single-function rule never sees.
It flags view functions that compute prices or balances from mutable state and warns that integrators reading them mid-call get inconsistent values, the read-only reentrancy signature.

It is the same approach from my earlier post on the five vulnerability classes AI catches: reason about intent and cross-function interaction, not just local patterns.

# Free, local, no API key needed
ollama pull qwen2.5-coder:1.5b
npx spectr-ai --model ollama:qwen2.5-coder:1.5b your-contract.sol

The Takeaway

Checks-effects-interactions is still the foundation: update all state before any external call, every time, in every function. ReentrancyGuard is a backstop, not a fix, and it is blind to read-only reentrancy by design because view functions are never locked. The variants that drain millions in 2026 are the cross-function, cross-contract, and read-only cases where the bug lives in the seam between contracts rather than in any single line.

Run your deterministic tools for the obvious writes-after-call. Then run an auditor that can reason across functions and contracts for the ones that hide in the seams.

spectr-ai is open source and works with Claude or local models via Ollama.

Access Control Bugs: The #1 Cause of Smart Contract Hacks This Year

Pavel Espitia — Wed, 10 Jun 2026 13:31:46 +0000

Every year the post-mortems pile up, and every year the same category sits at the top of the loss chart. Not reentrancy. Not oracle manipulation. Access control. A function that should have been guarded was not, an owner check was the wrong check, or a privileged role landed in the wrong hands. The bugs are simple to describe and brutal in their consequences, because the attacker does not need to outsmart your math. They just need to call a function you forgot to lock.

If you audit contracts, this is the category you check first. Here is the full landscape, with vulnerable and fixed Solidity for each pattern, and an audit checklist at the end.

Why Access Control Tops the Charts

Access control bugs share three traits that make them the dominant loss category:

They are easy to introduce. A missing modifier is a single forgotten word.
They are easy to exploit. No flash loan, no precise block timing, just a direct call.
They are hard to spot by eye. A privileged function looks identical to a safe one until you trace who is allowed to call it.

Pattern-matching tools catch some of these, but the worst ones live in the gaps between functions and in the gap between what the code says and what the developer intended. Let's walk through them.

1. The Missing Modifier

The classic. A state-changing function that should be restricted to an admin, shipped with no guard at all.

// Vulnerable: anyone can change the protocol fee
function setFee(uint256 newFee) external {
    fee = newFee;
}

Nothing stops a random address from calling this. The fix is one line, and a thousand audits have written it:

// Fixed
function setFee(uint256 newFee) external onlyOwner {
    fee = newFee;
}

The reason this still happens in production is that the function looks finished. It compiles, it works in the happy-path test, and the missing word is invisible until someone reads the function asking "who is allowed to call this?" Every setter, every withdrawal, every parameter change needs that question answered explicitly.

2. The Unprotected Initializer (Proxy Front-Running)

Upgradeable proxies cannot use constructors, because the constructor runs in the context of the implementation contract, not the proxy. So the pattern moved to an initialize() function that sets the owner. The footgun is that initialize() is just a regular external function.

// Vulnerable: initialize can be called by anyone, even twice
contract Vault {
    address public owner;
    bool private initialized;

    function initialize(address _owner) external {
        owner = _owner;
    }
}

If the proxy is deployed and the deployer forgets to call initialize() in the same transaction, an attacker watching the mempool calls it first and sets themselves as owner. This uninitialized-proxy pattern has drained multiple protocols, and it is one of the highest-impact bugs in the category because it hands over the entire contract.

// Fixed: use OpenZeppelin's Initializable
import "@openzeppelin/contracts-upgradeable/proxy/utils/Initializable.sol";

contract Vault is Initializable {
    address public owner;

    function initialize(address _owner) external initializer {
        owner = _owner;
    }
}

The initializer modifier guarantees the function runs exactly once. Just as important: lock down the implementation contract itself with _disableInitializers() in its constructor, so nobody can initialize the logic contract directly and use it as an attack surface.

3. tx.origin Authentication

tx.origin is the original external account that started the transaction chain. msg.sender is the immediate caller. Using tx.origin for authorization breaks the moment a contract sits in the middle.

// Vulnerable to phishing
function withdraw() external {
    require(tx.origin == owner, "Not owner");
    payable(msg.sender).transfer(address(this).balance);
}

The attack: the owner is tricked into calling a malicious contract. That contract calls withdraw(). The check tx.origin == owner passes, because the owner did start the chain, even though they never intended to withdraw. The funds go to msg.sender, which is the attacker's contract.

// Fixed: authenticate the immediate caller
function withdraw() external {
    require(msg.sender == owner, "Not owner");
    payable(msg.sender).transfer(address(this).balance);
}

The rule is absolute: never use tx.origin for authorization. Its only legitimate uses are rare and defensive (for example, refusing to be called by any contract at all), and even those are increasingly discouraged with account abstraction in the picture.

4. Default Visibility and Exposed Internals

Modern Solidity forces you to declare visibility, which killed the old pre-0.5.0 footgun of functions defaulting to public. But the underlying mistake survives: marking something public or external when it should have been internal.

// Vulnerable: a helper that should never be externally callable
function _mint(address to, uint256 amount) public {
    totalSupply += amount;
    balances[to] += amount;
}

If that mint helper is public, anyone can print tokens. The fix is to scope it correctly and expose only a guarded wrapper:

// Fixed
function _mint(address to, uint256 amount) internal {
    totalSupply += amount;
    balances[to] += amount;
}

function mint(address to, uint256 amount) external onlyMinter {
    _mint(to, amount);
}

When you read a contract, every public and external function is part of the attack surface. Helpers with a leading underscore that are not internal or private are a red flag worth a second look.

5. Flawed Role Management

OpenZeppelin's AccessControl is the standard for multi-role systems, and it is solid. The bugs come from misusing it: granting roles too broadly, or worse, leaving the role admin open.

// Vulnerable: anyone can grant themselves the minter role
contract Token is AccessControl {
    bytes32 public constant MINTER_ROLE = keccak256("MINTER_ROLE");

    function grantMinter(address account) external {
        _grantRole(MINTER_ROLE, account);
    }
}

The custom grantMinter wrapper bypasses the role-admin check entirely. Anyone calls it, grants themselves MINTER_ROLE, and mints freely.

// Fixed: gate role grants behind the role's admin
function grantMinter(address account) external onlyRole(getRoleAdmin(MINTER_ROLE)) {
    _grantRole(MINTER_ROLE, account);
}

A related footgun: forgetting to set DEFAULT_ADMIN_ROLE at all, which can leave roles unmanageable, or granting DEFAULT_ADMIN_ROLE to too many addresses, where any one of them can rewrite the entire permission graph. Map out the role hierarchy explicitly and confirm who can grant what.

6. Unprotected selfdestruct and Upgrade Functions

Two of the most dangerous functions a contract can expose are the ability to destroy it and the ability to swap its logic. Both must be locked down hard.

// Vulnerable: anyone can brick the contract or hijack the proxy
function kill() external {
    selfdestruct(payable(msg.sender));
}

function upgradeTo(address newImplementation) external {
    implementation = newImplementation;
}

An unprotected upgradeTo is the same severity as an unprotected initialize: the attacker points the proxy at their own logic and owns everything. An unprotected selfdestruct lets anyone delete the contract (and where it still force-sends ETH, redirect the balance). This pattern of a public self-destruct sitting on a shared library has frozen large sums when triggered by accident.

// Fixed
function upgradeTo(address newImplementation) external onlyOwner {
    implementation = newImplementation;
}

If you use OpenZeppelin's UUPSUpgradeable, the guard goes in _authorizeUpgrade, and that override must contain a real access check. An empty _authorizeUpgrade body is the same bug wearing a respectable name.

7. Ownership Transfer Footguns

Single-step ownership transfer is a foot-gun because there is no confirmation. Pass the wrong address and ownership is gone forever.

// Risky: a typo permanently locks you out of admin functions
function transferOwnership(address newOwner) external onlyOwner {
    owner = newOwner;
}

There is no validation that newOwner is correct, reachable, or even nonzero. The fix is the two-step pattern, where the new owner must accept:

// Fixed: OpenZeppelin Ownable2Step
import "@openzeppelin/contracts/access/Ownable2Step.sol";

contract Vault is Ownable2Step {
    // transferOwnership now only sets a pending owner;
    // the new owner must call acceptOwnership() to take control
}

Also guard against the silent renounceOwnership call, which is inherited from Ownable and, if left callable, can permanently disable every admin function with one transaction. Decide deliberately whether your contract should be able to renounce ownership at all.

The Audit Checklist

Run this against every privileged surface in the contract:

Check	What to confirm
Modifiers	Every state-changing admin function has an access modifier
Initializers	`initialize()` uses `initializer`; implementation calls `_disableInitializers()`
tx.origin	Zero uses of `tx.origin` for authorization
Visibility	No `public`/`external` function that should be `internal`; helpers scoped correctly
Roles	Role grants gated behind role admin; `DEFAULT_ADMIN_ROLE` set and minimized
Upgrade/destroy	`upgradeTo`, `_authorizeUpgrade`, and any `selfdestruct` are guarded
Ownership	Two-step transfer; deliberate decision on `renounceOwnership`
Zero address	Owner and role assignments reject `address(0)`

How AI Tooling Flags This

The deterministic part of access control auditing is the part tools already handle well: list every external and public function, and flag the ones that mutate state without a modifier. The hard part is the reasoning part, the cross-function privilege paths where each function looks fine alone but together they form an escalation route, like a setFeeRecipient with no guard feeding a withdrawFees that trusts whoever the recipient happens to be.

That is where an LLM-based reviewer earns its place. In spectr-ai, the engine first enumerates every privileged function and checks for a guard, then reasons about whether the guards that exist are the correct ones and whether any sequence of unguarded calls leads to a privileged state. It treats an unprotected initialize, an empty _authorizeUpgrade, and a tx.origin check as high-severity findings, because those are the patterns that consistently top the annual loss reports.

The takeaway: access control bugs win the loss chart every year because they are cheap to make and cheaper to exploit. Read every privileged function asking one question, "who can call this and is that who I meant?", run the checklist above, and let tooling enumerate the surface so nothing slips through.

spectr-ai is open source and runs with Claude or local models via Ollama.

Auditing a Contract With No Source Code: Reading EVM Bytecode

Pavel Espitia — Wed, 10 Jun 2026 13:31:42 +0000

You found a contract holding $4M. Etherscan shows the balance, the transactions, the token transfers. What it does not show is a single line of source code. The "Contract" tab just says: this contract has not been verified.

Now what? You can still audit it. The bytecode is right there on chain, fully public, and the EVM has no secrets from you. This post walks through reading deployed bytecode by hand, recovering function selectors, matching them against signature databases, and reconstructing a usable ABI with viem and whatsabi. By the end you will know exactly what you can recover and what is gone forever.

Step 1: Fetch the Deployed Bytecode

Every deployed contract exposes its runtime bytecode through the eth_getCode RPC method. With viem, that is one call:

import { createPublicClient, http } from "viem";
import { mainnet } from "viem/chains";

const client = createPublicClient({
  chain: mainnet,
  transport: http(),
});

const bytecode = await client.getCode({
  address: "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48", // USDC
});

console.log(bytecode);
// 0x608060405234801561001057600080fd5b50600436106103425760003560e01c...

If getCode returns 0x (or undefined), the address is an externally owned account, not a contract, or the contract has self-destructed. Anything longer is runtime bytecode you can dissect.

Two things to know about this hex blob:

It is the runtime bytecode, not the creation bytecode. The creation code (the part that runs once during deployment and returns the runtime code) is only visible in the deployment transaction's input field, not from eth_getCode.
The compiler usually appends a metadata hash (CBOR-encoded, often an IPFS pointer to the source) at the very end. Sometimes that hash is enough to find the original source if the author published it elsewhere.

Step 2: The Anatomy of Runtime Bytecode

EVM bytecode is a flat sequence of single-byte opcodes. Each opcode is one hex byte. 0x60 is PUSH1, 0x01 is ADD, 0x57 is JUMPI, and so on. The EVM reads them left to right and runs them on a stack machine.

The piece that matters for auditing is the part at the top of almost every compiled contract: the function dispatcher. When you call a contract, the first 4 bytes of your calldata are the function selector. The dispatcher reads those 4 bytes and jumps to the matching function body.

Here is the classic Solidity dispatcher prologue:

PUSH1 0x00       // 6000
CALLDATALOAD     // 35      load first 32 bytes of calldata
PUSH1 0xE0       // 60e0
SHR              // 1c      shift right 224 bits -> keep top 4 bytes
DUP1             // 80
PUSH4 0x18160ddd // 6318160ddd   selector for totalSupply()
EQ               // 14
PUSH2 0x0xxx     // 61....       jump destination
JUMPI            // 57      if equal, jump to totalSupply body
DUP1
PUSH4 0x70a08231 // 6370a08231   selector for balanceOf(address)
EQ
PUSH2 0x0xxx
JUMPI
...

That CALLDATALOAD / PUSH1 0xE0 / SHR sequence is the fingerprint. It loads the calldata, shifts right by 224 bits (0xE0 = 224) to isolate the leading 4 bytes, and then runs a chain of PUSH4 <selector> / EQ / JUMPI comparisons. Every PUSH4 you see in that chain is a function selector the contract responds to.

Step 3: What a Selector Actually Is

A function selector is the first 4 bytes of the keccak256 hash of the function's canonical signature. The canonical signature is the function name plus its argument types, no spaces, no parameter names:

import { keccak256, toHex, toBytes } from "viem";

const signature = "transfer(address,uint256)";
const hash = keccak256(toBytes(signature));
const selector = hash.slice(0, 10); // "0x" + 8 hex chars = 4 bytes

console.log(selector); // 0xa9059cbb

transfer(address,uint256) always hashes to 0xa9059cbb on every chain, in every contract, forever. That determinism is the whole reason bytecode auditing is even possible. The selectors are baked into the dispatcher, and they are reversible if (and only if) someone has already recorded the original signature.

This is also the catch. The hash is one way. Given 0xa9059cbb you cannot compute transfer(address,uint256). You can only look it up in a database of known signatures.

Step 4: Recover the Selectors From Bytecode

You can extract candidate selectors by scanning the bytecode for the PUSH4 opcode (0x63) followed by 4 bytes, in the region before the first JUMPDEST of the function bodies. A naive but surprisingly effective extractor:

function extractSelectors(bytecode: string): string[] {
  const code = bytecode.startsWith("0x") ? bytecode.slice(2) : bytecode;
  const selectors = new Set<string>();

  // walk the bytes, honoring PUSH data so we do not misread operands
  for (let i = 0; i < code.length; ) {
    const opcode = parseInt(code.slice(i, i + 2), 16);

    if (opcode === 0x63) {
      // PUSH4: the next 4 bytes are a candidate selector
      const data = code.slice(i + 2, i + 10);
      selectors.add("0x" + data);
      i += 10;
      continue;
    }

    // PUSH1 (0x60) through PUSH32 (0x7f) carry inline data we must skip
    if (opcode >= 0x60 && opcode <= 0x7f) {
      const dataLen = opcode - 0x5f; // 0x60 -> 1 byte, ... 0x7f -> 32
      i += 2 + dataLen * 2;
      continue;
    }

    i += 2;
  }

  return [...selectors];
}

The reason you must honor PUSH data lengths is that a 0x63 byte can appear inside the operand of some other PUSH instruction. If you scan blindly you get false selectors that were never meant to be opcodes. Skipping each push's payload keeps you aligned to real instruction boundaries.

Run it on USDC's bytecode and you get a clean list: 0x18160ddd, 0x70a08231, 0xa9059cbb, 0x095ea7b3, and so on. Those are real ERC-20 selectors.

Step 5: Match Selectors Against a Signature Database

Now reverse the hash by lookup. Two public databases index millions of signatures:

Source	Endpoint	Notes
4byte.directory	`https://www.4byte.directory/api/v1/signatures/`	Community-submitted, large, noisy (multiple collisions per selector)
OpenChain (ex-Sam Czsun)	`https://api.openchain.xyz/signature-database/v1/lookup`	Curated, also covers event topics

A direct lookup against OpenChain:

async function lookupSelector(selector: string): Promise<string[]> {
  const url = new URL("https://api.openchain.xyz/signature-database/v1/lookup");
  url.searchParams.set("function", selector);
  url.searchParams.set("filter", "true");

  const res = await fetch(url);
  const json = await res.json();
  const matches = json.result?.function?.[selector] ?? [];

  return matches.map((m: { name: string }) => m.name);
}

await lookupSelector("0xa9059cbb"); // ["transfer(address,uint256)"]
await lookupSelector("0x70a08231"); // ["balanceOf(address)"]

Be ready for collisions. Different signatures can hash to the same 4 bytes (4 bytes is only 32 bits of entropy). For 0x70a08231 you will almost always get balanceOf(address), but obscure selectors can return several plausible names. You then disambiguate by how the function is used, what it returns, or which one is an ERC standard.

Selectors that return nothing are the interesting ones. They are custom functions whose signatures were never published. You know they exist, you know their 4-byte ID, but their name and argument types are unknown.

Step 6: Let whatsabi Do the Heavy Lifting

Doing all of the above by hand is good for understanding. For real work, whatsabi does it properly, including proxy resolution and ABI assembly:

import { whatsabi } from "@shazow/whatsabi";
import { createPublicClient, http } from "viem";
import { mainnet } from "viem/chains";

const client = createPublicClient({ chain: mainnet, transport: http() });

const result = await whatsabi.autoload(
  "0x1f9840a85d5aF5bf1D1762F925BDADdC4201F984", // UNI token
  {
    provider: client,
    // follows EIP-1967 / EIP-1822 proxies to the implementation
    followProxies: true,
    // resolve selectors via OpenChain + 4byte
    abiLoader: new whatsabi.loaders.MultiABILoader([
      new whatsabi.loaders.OpenChainABILoader(),
      new whatsabi.loaders.FourByteABILoader(),
    ]),
  },
);

console.log(result.abi);
// [
//   { type: "function", name: "transfer", inputs: [...], stateMutability: "nonpayable" },
//   { type: "function", selector: "0x...", inputs: [...] }, // unresolved: no name
//   ...
// ]

whatsabi reads the bytecode, extracts selectors the robust way, detects proxy patterns and follows them to the implementation contract, looks each selector up, and hands you an ABI you can pass straight into viem's encodeFunctionData. For resolved selectors you get real names and types. For unresolved ones you still get a usable entry keyed by selector, so you can call the function even without knowing what it is named.

What You CAN and CANNOT Recover

This is the part people get wrong. Bytecode auditing is powerful but it is not decompilation back to source.

Recoverable	Lost
Function selectors (all of them)	Parameter names (`to`, `amount`, ...)
Signatures for published functions	Signatures for custom/private functions
Argument types, often inferable	Local variable names, comments
Proxy implementation address	Original source structure, modifiers
Storage layout clues (slot access patterns)	Business logic intent
Events (topics are keccak of the event sig)	Internal/private function boundaries

Three hard truths:

Parameter names are gone. The EVM never stored them. transfer(address,uint256) recovers, but whether the second arg was called amount or value is unknowable from bytecode.
Unpublished selectors stay opaque. If a contract has a function secretMint(address,uint256) that nobody ever recorded in 4byte or OpenChain, you get the selector and the calldata shape but never the name. You can still call it, you just do not know what it is for.
The logic is still bytecode. Recovering the ABI tells you the interface, not the behavior. To understand what 0x4a8c1fbb actually does to storage you need a decompiler (like Dedaub or heimdall) or to read the opcodes by hand. The ABI gets you the front door, not a map of the building.

Where This Goes

This whole pipeline (fetch bytecode, extract selectors, resolve signatures, follow proxies, assemble an ABI) is exactly what AbiLens runs every time you paste an unverified address. It tries Etherscan first for a verified ABI, and when that fails it falls back to whatsabi against the deployed bytecode, across 6 chains (Ethereum, Base, Arbitrum, Polygon, Optimism, Sepolia).

The payoff is that an unverified contract stops being a black box. AbiLens hands the reconstructed ABI to an LLM, so you can ask "what functions does this expose and which ones look like admin controls?" and get an answer grounded in the real selectors pulled out of the bytecode, even for the functions whose names came back generic.

The Takeaway

A contract with no verified source is not a dead end. The bytecode is public, the dispatcher is a readable table of PUSH4 <selector> / EQ / JUMPI, and selectors are deterministic keccak256 hashes you can reverse through 4byte and OpenChain. With viem's getCode and whatsabi you go from a raw hex blob to a callable ABI in a few lines.

Just stay honest about the limits: you recover the interface, not the intent. Names, comments, and private logic do not survive compilation. For the rest, the bytecode tells you everything if you are willing to read it.

The pipeline is open source at github.com/pavelEspitia/abilens.

Foundry Invariant Testing: Finding Bugs Fuzzing Can't

Pavel Espitia — Wed, 10 Jun 2026 13:31:36 +0000

Your fuzz tests pass. Your unit tests pass. Coverage is green. Then the protocol goes live and someone drains the vault with a five-transaction sequence you never tested.

That's the gap invariant testing fills. Unit tests check one path. Fuzz tests check one call with random inputs. Invariant tests check that a property holds no matter what sequence of calls anyone throws at your contract. That last category catches the bugs that actually drain money, because real exploits are almost never a single call. They're a chain.

This post walks through stateful invariant testing in Foundry with runnable code: a simple vault, an invariant that should always hold, a handler to bound the chaos, and how to read the failing call sequence Foundry hands you when it breaks the invariant.

Three kinds of test, increasing power

Test type	Prefix	What varies	What it checks
Unit	`test_`	Nothing (you pick inputs)	One specific path
Stateless fuzz	`testFuzz_`	Inputs to one call	One call, many inputs
Stateful invariant	`invariant_`	Inputs and call sequence	A property across random sequences

Here's the same vault tested three ways so the difference is concrete.

// test/Vault.t.sol
pragma solidity ^0.8.20;
import "forge-std/Test.sol";
import "../src/Vault.sol";

contract VaultTest is Test {
    Vault vault;

    function setUp() public {
        vault = new Vault();
    }

    // 1. Unit test: I pick everything
    function test_deposit() public {
        vault.deposit{value: 1 ether}();
        assertEq(vault.shares(address(this)), 1 ether);
    }

    // 2. Stateless fuzz: Foundry picks the input, runs ONE deposit
    function testFuzz_deposit(uint256 amount) public {
        amount = bound(amount, 1, 100 ether);
        vm.deal(address(this), amount);
        vault.deposit{value: amount}();
        assertEq(vault.shares(address(this)), amount);
    }
}

The fuzz test is strictly better than the unit test: it tries thousands of deposit amounts. But notice what it still can't see. It does one deposit, in isolation, from a fresh contract. It never deposits, withdraws, has another user deposit, then withdraws again. The bugs that matter live in that interleaving.

The vault we'll break

A minimal share-based vault. Deposit ETH, get shares. Burn shares, get ETH back. Looks fine.

// src/Vault.sol
pragma solidity ^0.8.20;

contract Vault {
    mapping(address => uint256) public shares;
    uint256 public totalShares;

    function deposit() external payable {
        uint256 minted = totalShares == 0
            ? msg.value
            : (msg.value * totalShares) / address(this).balance;
        shares[msg.sender] += minted;
        totalShares += minted;
    }

    function withdraw(uint256 amount) external {
        require(shares[msg.sender] >= amount, "insufficient");
        uint256 payout = (amount * address(this).balance) / totalShares;
        shares[msg.sender] -= amount;
        totalShares -= amount;
        (bool ok, ) = msg.sender.call{value: payout}("");
        require(ok, "transfer failed");
    }
}

There's a subtle bug in here. You probably can't spot it by reading, and neither could a unit test author who only writes the happy path. Invariant testing finds it in seconds.

Writing the invariant

An invariant is a property that must be true after every possible call, in every possible order. For a vault, the obvious solvency invariant is:

The contract's ETH balance must always cover what it owes. Assets >= liabilities.

For this share model, the cleanest invariant is: if anyone holds shares, the contract holds ETH. You should never be in a state where totalShares > 0 but address(this).balance == 0, because that means shareholders own claims against nothing.

// test/VaultInvariant.t.sol
pragma solidity ^0.8.20;
import "forge-std/Test.sol";
import "../src/Vault.sol";

contract VaultInvariantTest is Test {
    Vault vault;

    function setUp() public {
        vault = new Vault();
        targetContract(address(vault));
    }

    // Solvency: shares outstanding implies ETH backing them
    function invariant_solvency() public view {
        if (vault.totalShares() > 0) {
            assertGt(address(vault).balance, 0, "shares exist but no ETH");
        }
    }
}

targetContract tells Foundry which contract to bombard with random calls. Run it:

forge test --match-contract VaultInvariantTest -vvv

Foundry generates random sequences of deposit and withdraw calls with random arguments and random sender addresses, then checks invariant_solvency after each call. If the property ever breaks, it prints the exact sequence that broke it.

Reading the failing call sequence

When the invariant fails, Foundry gives you the reproduction directly:

[FAIL: shares exist but no ETH]
    [Sequence]
        sender=0x0000...0a addr=Vault calldata=deposit() value=3
        sender=0x0000...1f addr=Vault calldata=withdraw(2) value=0
        sender=0x0000...0a addr=Vault calldata=withdraw(1) value=0

  invariant_solvency() (runs: 256, calls: 3840, reverts: 1201)

Read that bottom-up. runs: 256 is how many independent sequences it tried. calls: 3840 is total calls across all runs. reverts: 1201 means many random calls reverted (expected, since random withdraw amounts usually exceed balances). The [Sequence] block is the shrunk, minimal series of calls that triggered the failure. Foundry shrinks it for you, so you get the shortest path to the bug, not the raw 50-call run.

Drop that sequence into a standalone unit test and you have a permanent regression test for the exact bug. That's the workflow: invariant test discovers, you copy the sequence into a test_ for the fix.

The bug itself: division truncation in deposit and withdraw lets totalShares go positive while the contract's balance rounds down toward zero across a sequence of small deposits and withdrawals. A single fuzzed call can't reach that state. It needs the interleaving of multiple users and multiple withdrawals, which is exactly what stateful testing explores.

Handlers: stop wasting runs on reverts

Notice reverts: 1201 above. More than a quarter of the random calls reverted. That's wasted work. Pointing Foundry directly at the vault means it calls withdraw with absurd amounts from accounts holding zero shares, and those just revert without exercising anything.

The fix is a handler contract: a thin wrapper that bounds inputs and only makes calls that can realistically succeed. Foundry targets the handler instead of the vault.

// test/handlers/VaultHandler.sol
pragma solidity ^0.8.20;
import "forge-std/Test.sol";
import "../../src/Vault.sol";

contract VaultHandler is Test {
    Vault public vault;

    // Ghost variable: track total ETH ever deposited
    uint256 public ghost_depositSum;

    address[] internal actors;
    address internal currentActor;

    modifier useActor(uint256 seed) {
        currentActor = actors[bound(seed, 0, actors.length - 1)];
        vm.startPrank(currentActor);
        _;
        vm.stopPrank();
    }

    constructor(Vault _vault) {
        vault = _vault;
        for (uint256 i = 0; i < 4; i++) {
            actors.push(makeAddr(string(abi.encodePacked("actor", i))));
        }
    }

    function deposit(uint256 seed, uint256 amount) external useActor(seed) {
        amount = bound(amount, 1, 10 ether);
        vm.deal(currentActor, amount);
        vault.deposit{value: amount}();
        ghost_depositSum += amount;
    }

    function withdraw(uint256 seed, uint256 amount) external useActor(seed) {
        uint256 bal = vault.shares(currentActor);
        if (bal == 0) return;            // skip impossible calls
        amount = bound(amount, 1, bal);  // only withdraw what you hold
        vault.withdraw(amount);
    }
}

Three things are doing real work here:

bound(amount, 1, bal) keeps amount inside the legal range. bound maps any random uint256 into [min, max] without throwing away the run. Never use vm.assume for ranges this tight, it discards too many runs and slows the suite to a crawl.
useActor rotates between a fixed set of accounts via vm.startPrank, so the random sequences actually model multiple users interacting, which is where the interleaving bugs hide.
Ghost variables (ghost_depositSum) accumulate state the contract doesn't track itself. You can assert against them in the invariant, for example "total deposited minus total withdrawn equals current balance," which catches accounting drift the contract alone can't reveal.

Wire the handler in and restrict targeting to the functions you want:

// test/VaultInvariant.t.sol (updated setUp)
import "./handlers/VaultHandler.sol";

VaultHandler handler;

function setUp() public {
    vault = new Vault();
    handler = new VaultHandler(vault);

    targetContract(address(handler));

    // Only fuzz these two selectors on the handler
    bytes4[] memory selectors = new bytes4[](2);
    selectors[0] = handler.deposit.selector;
    selectors[1] = handler.withdraw.selector;
    targetSelector(FuzzSelector({addr: address(handler), selectors: selectors}));
}

targetSelector narrows the fuzzer to exactly the handler functions you wrote, so it never calls internal helpers or view functions by accident. Now nearly every generated call is a valid, state-changing operation. The revert count drops, and each run explores meaningful states instead of bouncing off require checks.

Tuning foundry.toml

Invariant runs are controlled separately from regular fuzzing. The two knobs that matter:

# foundry.toml
[invariant]
runs = 256        # independent sequences to generate
depth = 128       # calls per sequence
fail_on_revert = false  # don't treat handler reverts as failures

runs is how many fresh sequences Foundry tries. More runs, more coverage, slower CI.
depth is how many calls per sequence. This is the lever that matters most for finding deep bugs. A bug that needs eight interleaved operations to manifest will never show up at depth = 4. Bump depth when you suspect multi-step state corruption.
fail_on_revert decides whether a reverting call fails the whole invariant. Keep it false while developing the handler (reverts are noise), flip it to true once your handler is tight enough that any revert genuinely signals a bug.

A practical starting point for a real protocol: runs = 500, depth = 100. Then watch the calls count in the output. If most calls are reverting, your handler bounding is too loose, not your invariant.

Why this catches what fuzzing misses

Stateless fuzzing resets state between every test. It asks "given a fresh contract, does this one call with this one input behave?" Invariant testing asks "given a contract that's been hammered by 100 random operations from 4 different users, does this property still hold?"

Almost every high-value exploit is the second question. Reentrancy chains, rounding-error accumulation, share-price manipulation, donation attacks, accounting drift across deposits and withdrawals: none of them reproduce in a single call. They require a sequence, and the bug only appears in the accumulated state. That's the entire category stateless tests are blind to.

When I'm auditing a protocol or building detection into spectr-ai, the solvency invariant ("assets >= liabilities, always") is one of the first things I reach for, because it's the property an attacker is ultimately trying to violate. If you can write an invariant and Foundry can break it, you found the bug before mainnet did.

The takeaway

Write unit tests for known paths, fuzz tests for input ranges, and invariant tests for properties that must hold across all sequences.
Start with a solvency invariant: assets >= liabilities. It maps directly to what attackers want to break.
Use a handler with bound() and rotating actors so the fuzzer spends runs on valid, state-changing calls instead of reverts.
Add ghost variables to assert accounting properties the contract doesn't track itself.
Crank depth before runs when you suspect a multi-step bug. Read the shrunk failing sequence and turn it into a permanent regression test.

Unit and fuzz tests prove your functions work in isolation. Invariant tests prove your system stays correct under chaos. The bugs that drain protocols live in the chaos.

If you want to see invariant-style reasoning applied automatically to contracts you didn't write, that's part of what I'm building into spectr-ai: https://github.com/pavelEspitia/spectr-ai

The Auditor's AI Workflow: How I Use LLMs Without Trusting Them

Pavel Espitia — Wed, 10 Jun 2026 13:31:32 +0000

I use an LLM on every contract I review. I also assume it is lying to me until I prove otherwise.

That sounds contradictory, but it is the only way to get leverage out of AI without getting burned by it. An LLM will map a 2,000-line protocol faster than you can scroll it, and then it will confidently invent a CVE that does not exist, assign "Critical" to a non-issue, and miss the actual bug three functions down. The skill is not "use AI" or "do not use AI." The skill is running a workflow where every AI claim has to survive a verification step before it reaches your report.

Here is the exact loop I run, the prompts I use, and where I refuse to trust the model.

The Loop, Top to Bottom

Recon and triage with AI to build a mental map of the contract.
Targeted questions per vulnerability class, one class at a time.
Verify every finding manually plus with tooling (Slither, Foundry) before believing it.
Use AI to write PoC exploit tests that either confirm or kill the finding.
Guard against hallucinations and false confidence at every step.

Steps 1, 2, and 4 are where AI earns its keep. Step 3 is where I trust nothing. Step 5 runs the whole time.

Step 1: Recon and Triage

The first job is not finding bugs. It is understanding the contract well enough to know where bugs would live. AI is excellent at this because summarizing and mapping is exactly what it is good at, and a wrong summary is cheap to catch.

My opening prompt is deliberately boring:

You are reviewing a Solidity contract. Do NOT look for vulnerabilities yet.
Output only:
1. Every external/public function, its access control, and what state it mutates.
2. All privileged roles and what each can do.
3. Every external call and which contracts it touches.
4. Any assumptions the code makes about callers or token behavior.
Cite line numbers for each item. If you are unsure, say "unsure" instead of guessing.

That last line matters. Telling the model to mark uncertainty instead of filling gaps cuts the confident-nonsense rate noticeably. I get back a function-by-function map, an access-control table, and an external-call list. I read the contract alongside it and fix the map where the model is wrong. By the end I have a verified picture of the attack surface, and the AI did the tedious cataloging.

This is the one stage where I let AI move fast, because the output is structural, not a judgment call, and I am cross-checking it line by line anyway.

Step 2: One Vulnerability Class at a Time

A general "find all the bugs" prompt produces a wall of low-signal output: ten findings, three real, four hallucinated, three padding. So I ask per class, with the role context I built in step 1.

Focus ONLY on reentrancy in this contract.
For each external call: state whether checks-effects-interactions is followed,
which state is read after the call, and whether a reentrant call could profit.
Reference the function map I gave you. If a function is safe, say so and why.
Do not report anything outside reentrancy.

Then I repeat for access control, oracle/price manipulation, decimal and rounding errors, unchecked returns, storage layout (for proxies), and business-logic invariants. Narrow scope does two things: it keeps the model from spraying half-baked findings across categories, and it makes each answer easy to verify because I know exactly what claim to check.

For business logic, the model cannot help unless I give it intent, so I do:

The intended behavior: only the timelock (0x...) may execute a passed proposal,
and only after a 48-hour delay. Does the code enforce both? Quote the lines that do.

If it cannot quote the lines, the invariant probably is not enforced, and now I have a lead.

Step 3: Trust Nothing Until It Is Verified

This is the part people skip, and it is the part that makes the difference between an AI-assisted audit and a hallucinated one. Every finding from step 2 goes through the same gate before I write it down.

AI claims	How I verify before believing it
"Reentrancy in `withdraw`"	Read the function. Run Slither. Write a Foundry PoC that actually reenters.
"This violates EIP-XXXX"	Open the actual EIP. Models invent spec numbers and requirements constantly.
"Severity: Critical"	Re-derive severity myself from impact times likelihood. AI severity is noise.
"Vulnerable to CVE-2021-XXXXX"	Search the CVE. Smart-contract findings rarely map to CVEs; usually fabricated.
"No issues in this function"	Treat as unconfirmed, not as a clean bill. Absence of a finding is not proof.

The fastest filter is the deterministic tooling. Slither and a compiler do not hallucinate. If the AI flags reentrancy and Slither's reentrancy-eth detector is silent, I do not discard either one. I dig into why they disagree, because one of them is wrong and I need to know which. Often the AI caught a cross-function path Slither missed, and sometimes the AI made it up. The disagreement itself is the signal.

The hard rule: a finding is not real until I can point to the lines and explain the exploit without the AI in the room. If I cannot re-explain it myself, I do not understand it, which means I cannot defend it to a client, which means it does not go in the report.

Step 4: Make AI Write the PoC

Once a finding survives manual review, the cleanest way to confirm it is a test that exploits it. Writing PoCs by hand is slow, and this is another spot where AI is genuinely strong because a failing or passing test is self-verifying. The test either drains the funds or it does not.

Write a Foundry test that proves this exploit. Setup: deploy the contract,
fund it with 10 ether, attacker starts with 1 ether. The test must assert the
attacker's balance increases by at least 9 ether after the attack. Use a
reentrancy receiver contract. Only Foundry, no pseudocode.

Then I run it.

forge test --match-test testReentrancyExploit -vvv

If it passes, the vulnerability is real and I now have a reproducible PoC for the report. If it fails, one of two things is true: the exploit does not work (the AI finding was wrong, kill it) or my setup is off (fix it and rerun). Either way the test, not the model's prose, is the source of truth. A passing exploit test is worth more than ten paragraphs of confident explanation.

Step 5: Guarding Against Hallucinations

Here is where AI lies to you, ranked by how often I see it:

Invented severity and confidence. "Critical" on a gas optimization. The fix is to ignore AI severity entirely and re-derive your own.
Fabricated standards and CVEs. Cited EIPs, SWC entries, and CVE numbers that do not exist or do not say what the model claims. Always open the source.
Plausible-but-wrong exploit paths. A reentrancy story that ignores a nonReentrant modifier two lines up. The PoC test kills these.
Silent misses presented as completeness. "I reviewed everything and found no other issues." It did not review everything. Coverage is your job.
Anchoring. Tell the model a function is vulnerable and it will find a way to agree. Ask neutrally: "is this safe or unsafe, and why," not "explain why this is vulnerable."

The structural defense is to never let free-form model text be the final artifact. This is exactly how I built spectr-ai: the LLM does not get to return prose that goes straight into a report. Its output is forced through a Zod schema (every finding must have a title, a severity from a fixed enum, a line reference, and a description), so a malformed or half-hallucinated response fails validation instead of silently corrupting the results. Then those findings sit next to deterministic Slither output, not in place of it. The AI gives reasoning and coverage; the static analyzer and the type-checked schema give you a floor of facts the model cannot wander away from.

That pairing is the whole philosophy in one sentence: let the AI reason, but pin every claim to something deterministic.

The Trust-But-Verify Checklist

Before any AI-sourced finding goes in a report, it clears all of these:

[ ] I read the flagged code myself and understand the claim.
[ ] I can explain the exploit without the AI present.
[ ] A deterministic tool (Slither, compiler, or my own reading) agrees, or I understand exactly why it disagrees.
[ ] Any cited EIP, SWC, or CVE was opened and actually says what the model claimed.
[ ] Severity was re-derived by me from impact times likelihood, not copied from the model.
[ ] Where exploitable, a Foundry PoC test passes and proves it.

If a finding fails any box, it stays out until I close the gap.

The Takeaway

AI does not make you a worse auditor by being wrong. It makes you a worse auditor when you let it be the last word. Used inside a verification loop, an LLM is the best junior analyst you will ever have: tireless at recon, fast at writing PoCs, fluent at explaining a pattern you half-remember. Used as an oracle, it is a confident liar that will put fabricated CVEs in your report.

The workflow is the entire value. Recon with it, question it one class at a time, verify everything against tooling and your own reading, make it write the PoC, and never trust a severity it assigned itself. Do that and AI is leverage. Skip it and AI is liability.

That verification-first design is what I am building into spectr-ai: LLM reasoning, forced through a typed schema, paired with deterministic static analysis, so the AI gets to think but never gets the last word. It is open source and runs with Claude or a local model via Ollama.

Streaming LLM Tokens to the Browser: The Production SSE Setup

Pavel Espitia — Fri, 05 Jun 2026 11:54:01 +0000

A spinner is a lie. It tells the user something is happening without telling them what. When spectr-ai generates a security report, the LLM produces text token by token over 15 to 40 seconds. If I wait for the full response and then drop it on the page, the user stares at nothing the whole time. If I stream each token as it arrives, the report writes itself in front of them, exactly like ChatGPT. Same wait, completely different feel.

A while back I covered SSE for progress bars: the server sends a handful of step and progress events, the client moves a bar. This is the token-streaming version. Instead of a few discrete progress events, the server forwards hundreds of text fragments coming out of the model in real time. The transport is the same (Server-Sent Events over a fetch stream), but the source, the parsing, and the failure modes are different.

Here is the full production setup: a Next.js 15 Route Handler that consumes the model's own stream and re-emits it, a client reader that renders tokens as they land, and the cancellation and error handling you actually need when the thing runs for 40 seconds.

Why Not Just `EventSource`

The browser's EventSource is the obvious tool for SSE, and it handles reconnection for free. But it only does GET requests. spectr-ai sends a POST with the contract source and the chosen model in the body, so EventSource is out. We read the response stream by hand with fetch and response.body.getReader(). That is also what gives us an AbortController to cancel, which EventSource does not expose cleanly.

Need	`EventSource`	`fetch` + reader
POST with a body	No	Yes
Custom headers (auth)	No	Yes
Manual cancellation	Awkward	`AbortController`
Auto-reconnect	Yes	You write it

For a one-shot LLM request you do not want auto-reconnect anyway. A reconnect would restart the generation and bill you twice.

The Source: Consuming the Model's Stream

Both Ollama and Claude can stream. Ollama exposes an OpenAI-compatible endpoint at /v1/chat/completions, and with stream: true it returns SSE lines of its own: data: {json}\n\n, ending with data: [DONE]. So the server is doing two things at once: it is an SSE client (reading the model) and an SSE server (writing to the browser).

Here is the helper that turns the model's HTTP stream into an async iterator of plain text deltas:

// lib/stream-model.ts
interface ChatChunk {
  choices: { delta: { content?: string } }[];
}

export async function* streamModel(
  prompt: string,
  contract: string,
  signal: AbortSignal,
): AsyncGenerator<string> {
  const response = await fetch(
    "http://localhost:11434/v1/chat/completions",
    {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      signal,
      body: JSON.stringify({
        model: "qwen2.5-coder:7b",
        stream: true,
        messages: [
          { role: "system", content: prompt },
          { role: "user", content: contract },
        ],
      }),
    },
  );

  if (!response.ok || !response.body) {
    throw new Error(`Model request failed: ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n");
    buffer = lines.pop() ?? "";

    for (const line of lines) {
      const trimmed = line.trim();
      if (!trimmed.startsWith("data: ")) continue;

      const payload = trimmed.slice(6);
      if (payload === "[DONE]") return;

      const chunk = JSON.parse(payload) as ChatChunk;
      const token = chunk.choices[0]?.delta?.content;
      if (token) yield token;
    }
  }
}

The buffer logic is the same idea as the client reader in the progress-bar post: a TCP chunk does not respect message boundaries. One read() might give you half a data: line. Splitting on \n and holding the last fragment until the next read keeps you from trying to JSON.parse a truncated object.

For Claude the shape differs (the Anthropic SDK gives you a typed stream you can for await over, with content_block_delta events), but the contract from the server's point of view is identical: an async generator of text deltas. Swap the body of streamModel and the rest of this post does not change.

The Server: A Route Handler That Re-emits Tokens

Now wrap that generator in a ReadableStream and return it from a Route Handler. Each token becomes its own SSE event so the client can render it immediately.

// app/api/report/route.ts
import { NextRequest } from "next/server";
import { streamModel } from "@/lib/stream-model";

export const runtime = "nodejs";
export const maxDuration = 60;

interface TokenEvent {
  type: "token";
  text: string;
}
interface DoneEvent {
  type: "done";
}
interface ErrorEvent {
  type: "error";
  message: string;
}
type ReportEvent = TokenEvent | DoneEvent | ErrorEvent;

export async function POST(request: NextRequest) {
  const { contract } = (await request.json()) as { contract: string };

  if (!contract?.trim()) {
    return Response.json({ error: "No contract" }, { status: 400 });
  }

  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      function send(event: ReportEvent) {
        controller.enqueue(
          encoder.encode(`data: ${JSON.stringify(event)}\n\n`),
        );
      }

      try {
        for await (const token of streamModel(
          SYSTEM_PROMPT,
          contract,
          request.signal,
        )) {
          send({ type: "token", text: token });
        }
        send({ type: "done" });
      } catch (err) {
        if (request.signal.aborted) return; // client left, stay quiet
        const message =
          err instanceof Error ? err.message : "Generation failed";
        send({ type: "error", message });
      } finally {
        controller.close();
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
      "X-Accel-Buffering": "no",
    },
  });
}

Three things here earn their keep.

request.signal flows all the way through. When the browser aborts, Next.js aborts request.signal, which I pass into streamModel, which passes it to the model fetch. Hitting stop in the UI actually stops the model from generating. No orphaned 40-second jobs burning a GPU.
no-transform and X-Accel-Buffering: no. These are the anti-buffering headers. no-transform tells proxies not to gzip-buffer the body, and X-Accel-Buffering: no disables nginx's response buffer. Without them, a proxy can hold your tokens and flush them all at once at the end, which defeats the entire point. This matters far more for token streaming than for progress bars, because tokens arrive every few milliseconds and any buffer is instantly visible.
runtime = "nodejs" and maxDuration. Token generation is long. On Vercel the default function timeout will cut you off mid-report. Set maxDuration and budget for the slowest model you support.

Backpressure: Respecting the Reader

ReadableStream has built-in backpressure. When you call controller.enqueue, the data goes into an internal queue. If the client reads slower than the model produces (a slow network, a backgrounded tab), that queue fills, and controller.desiredSize drops at or below zero.

For text tokens the volume is small enough that you rarely need to act on it, but if you are also streaming large structured chunks, you can await space before enqueuing more:

async function backpressuredSend(
  controller: ReadableStreamDefaultController,
  bytes: Uint8Array,
) {
  controller.enqueue(bytes);
  // Yield to let the consumer drain if the queue is full.
  if ((controller.desiredSize ?? 1) <= 0) {
    await new Promise((resolve) => setTimeout(resolve, 0));
  }
}

The bigger backpressure win is the abort signal above. The single most expensive thing you can do is keep generating tokens for a client that has already navigated away. Cancellation is backpressure taken to its limit: the consumer is gone, so produce nothing.

The Client: Reading and Rendering Tokens

The reader mirrors the progress-bar client, with one change: instead of replacing a progress value, we append each token to accumulated text.

// lib/read-report.ts
type ReportEvent =
  | { type: "token"; text: string }
  | { type: "done" }
  | { type: "error"; message: string };

export async function streamReport(
  contract: string,
  onToken: (text: string) => void,
  signal: AbortSignal,
): Promise<void> {
  const response = await fetch("/api/report", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ contract }),
    signal,
  });

  if (!response.ok || !response.body) {
    throw new Error(`Request failed: ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const events = buffer.split("\n\n");
    buffer = events.pop() ?? "";

    for (const event of events) {
      const dataLine = event
        .split("\n")
        .find((line) => line.startsWith("data: "));
      if (!dataLine) continue;

      const parsed = JSON.parse(dataLine.slice(6)) as ReportEvent;

      if (parsed.type === "token") onToken(parsed.text);
      if (parsed.type === "error") throw new Error(parsed.message);
      if (parsed.type === "done") return;
    }
  }
}

And the React component. The trick that makes this feel fast is appending to a ref and flushing to state, so hundreds of rapid token updates do not trigger hundreds of re-render storms fighting each other:

"use client";

import { useState, useRef, useCallback } from "react";
import { streamReport } from "@/lib/read-report";

export function ReportView({ contract }: { contract: string }) {
  const [text, setText] = useState("");
  const [status, setStatus] =
    useState<"idle" | "streaming" | "done" | "error">("idle");
  const [error, setError] = useState<string | null>(null);
  const abortRef = useRef<AbortController | null>(null);

  const start = useCallback(async () => {
    abortRef.current?.abort();
    const controller = new AbortController();
    abortRef.current = controller;

    setText("");
    setError(null);
    setStatus("streaming");

    try {
      await streamReport(
        contract,
        (token) => setText((prev) => prev + token),
        controller.signal,
      );
      setStatus("done");
    } catch (err) {
      if (controller.signal.aborted) return; // user cancelled
      setError(err instanceof Error ? err.message : "Stream failed");
      setStatus("error");
    }
  }, [contract]);

  const stop = useCallback(() => {
    abortRef.current?.abort();
    setStatus("idle");
  }, []);

  return (
    <div>
      <div className="flex gap-2">
        <button onClick={start} disabled={status === "streaming"}>
          Generate report
        </button>
        {status === "streaming" && (
          <button onClick={stop}>Stop</button>
        )}
      </div>

      <pre className="mt-4 whitespace-pre-wrap font-mono text-sm">
        {text}
        {status === "streaming" && (
          <span className="animate-pulse">▍</span>
        )}
      </pre>

      {error && (
        <div className="mt-4 rounded bg-red-50 p-3 text-red-700">
          {error}
        </div>
      )}
    </div>
  );
}

The blinking ▍ cursor while status === "streaming" costs nothing and sells the live feel. The whitespace-pre-wrap on a pre preserves the model's markdown spacing until you are ready to run it through a markdown renderer at the end.

The Failure Modes That Actually Bite

Errors after a 200. This is the same trap as the progress-bar post, sharper here. The moment you return the Response, the status is 200 and frozen. If the model dies at token 300, you cannot send a 500. You send an error event inside the stream and the client throws on it. Any error handling that relies on HTTP status codes is dead the instant the first byte ships.

Cancellation is two-sided. The client aborts, but the server has to notice. That is why request.signal is threaded into every fetch. I also guard if (controller.signal.aborted) return on the client so a deliberate stop does not flash a scary error message.

Half-tokens across chunk boundaries. A data: line can be split across two TCP reads on both the server-to-model side and the browser side. Both readers use the same split-and-hold-the-tail buffer. Skip it and you get random JSON.parse crashes under load that you will never reproduce locally on localhost.

Proxy buffering eats the effect. If tokens arrive in one big clump at the end, your headers are wrong. Check no-transform and X-Accel-Buffering: no, and confirm nothing in front of the app (a CDN, a dev proxy) is re-buffering.

Takeaway

Progress bars and token streams ride the same SSE rails, but token streaming raises the stakes: you are now an SSE client and server at once, buffering bugs become visible at millisecond resolution, and cancellation has to reach all the way to the model or you pay for work nobody will read. Thread one AbortController from the button to the model fetch, set the anti-buffering headers, and parse with a tail buffer on both ends. That is the whole production setup.

It is running in spectr-ai right now: paste a contract, watch the security report write itself token by token, hit stop and the GPU stops with it. Code is on GitHub: https://github.com/pavelEspitia/spectr-ai