Function Calling With a Local LLM to Drive Foundry: Fuzz, Read, Repeat

The first time I let a local model drive Foundry unsupervised, it spent eleven turns trying to fix a fuzz test by renaming the test function. Not changing the logic. Renaming it. testFuzz_withdraw, then test_fuzz_withdraw, then testWithdrawFuzz, each time running the suite and reading the same compiler error with fresh optimism.

That experiment still turned into one of the more useful tools in my workflow, once I accepted what a small local model can and cannot do in an agent loop. The idea is simple: give qwen2.5-coder three tools (run forge tests, read a contract file, write a fuzz test) and point it at a target contract. It reads the code, writes property tests, runs them, reads the failures, and iterates. When it works, it surfaces broken invariants I would have gotten to eventually, but it gets there while I make coffee, and everything stays on my machine.

The three tools

Ollama supports function calling through its chat API. You describe tools in JSON schema, the model returns tool_calls, you execute and feed results back. Keep the toolset minimal, every extra tool is another way for a 7b model to get confused.

const tools = [
  {
    type: "function",
    function: {
      name: "read_file",
      description: "Read a Solidity source file from the project.",
      parameters: {
        type: "object",
        properties: {
          path: { type: "string", description: "Path relative to project root" },
        },
        required: ["path"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "write_fuzz_test",
      description:
        "Write the complete contents of the fuzz test file test/Fuzz.t.sol. " +
        "Always write the FULL file, never a fragment.",
      parameters: {
        type: "object",
        properties: {
          content: { type: "string", description: "Full Solidity file content" },
        },
        required: ["content"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "run_forge_test",
      description: "Compile and run the test suite. Returns compiler errors or test results.",
      parameters: { type: "object", properties: {}, required: [] },
    },
  },
];

Two design decisions hiding in there. First, write_fuzz_test takes no path: the model writes exactly one file, test/Fuzz.t.sol, always in full. Letting a small model choose file paths or emit diffs is asking for contracts/test/../test/Fuzz.sol and patches against lines that don't exist. Second, run_forge_test takes no arguments at all, so the model can't invent flags.

The executors are thin wrappers. The only interesting one is the test runner, which truncates aggressively:

import { execFileSync } from "node:child_process";

function runForgeTest(): string {
  try {
    const out = execFileSync("forge", ["test", "--fuzz-runs", "256"], {
      cwd: PROJECT_ROOT,
      timeout: 120_000,
      encoding: "utf8",
    });
    return tail(out, 3000);
  } catch (e: unknown) {
    const err = e as { stdout?: string; stderr?: string };
    return "FAILED\n" + tail((err.stdout ?? "") + (err.stderr ?? ""), 3000);
  }
}

Truncation matters because a forge trace on a failing fuzz test can be enormous, and dumping 40k tokens of trace into a small model's context doesn't inform it, it lobotomizes it. I keep the tail, which is where Foundry puts the counterexample and the summary.

The loop

const messages: Message[] = [
  { role: "system", content: SYSTEM_PROMPT },
  { role: "user", content: `Target contract: src/${target}. Read it, identify invariants, write fuzz tests, run them, iterate until they compile and either pass or reveal a real counterexample.` },
];

for (let turn = 0; turn < MAX_TURNS; turn++) {
  const res = await ollama.chat({ model: "qwen2.5-coder:7b", messages, tools });
  messages.push(res.message);

  if (!res.message.tool_calls?.length) break; // model thinks it's done

  for (const call of res.message.tool_calls) {
    const result = dispatch(call.function.name, call.function.arguments);
    messages.push({ role: "tool", content: result });
  }
}

The system prompt is where most of the iteration went. The parts that earned their place:

You write Foundry fuzz tests for one target contract.
Rules:
- ALWAYS read the target contract before writing any test.
- Test properties and invariants, not specific values. Good properties:
  balance accounting sums correctly, access control cannot be bypassed,
  state transitions preserve solvency, no operation mints value from nothing.
- Use vm.assume or bound() to constrain inputs, never require statements.
- After every write, run the tests. Read the ACTUAL error before editing.
- If the same error appears twice in a row, change your approach entirely.
- A failing fuzz test with a counterexample is a SUCCESS. Report it and stop.

That last line is important and counterintuitive. The model's instinct is to "fix" a failing test by weakening the assertion until it passes, which is the exact opposite of what a security tool should do. Telling it explicitly that a legitimate counterexample is the win condition mostly (not always) suppresses that reflex. I hit the same failure mode building spectr-ai: small models are trained to make errors go away, and in security work the error is frequently the product.

Where it gets stuck, and how I bound it

Small models lose the plot in long agent loops. Around eight to twelve turns in, qwen2.5-coder:7b starts forgetting constraints from the system prompt, re-reading files it already read, or cycling between two broken versions of the same test. The 1.5b model is worse, it rarely survives past turn four before emitting a tool call with malformed arguments. This isn't a bug I can prompt away, it's what limited context handling and small parameter counts look like in practice.

So I stopped fighting it and bounded the loop instead:

Hard turn cap. MAX_TURNS = 12. Past that, whatever's in test/Fuzz.t.sol is the output, and I take over.

Loop detection. I hash each write_fuzz_test payload. Same hash twice means the model is cycling, and the harness injects a user message saying so, one time. Second repeat kills the run.

Compile-first gating. The most common death spiral is chasing compiler errors (wrong import path, wrong pragma). The harness pre-seeds the file with a known-good skeleton (correct imports, a deployed instance of the target in setUp), so the model starts from something that compiles and only has to fill in properties. This single change roughly doubled how often runs end usefully.

One target, one file. I don't ask it to fuzz a protocol. I ask it to fuzz one contract, sometimes one function. Small scope is the difference between a focused property test and turn-eleven function renaming.

With those bounds, a run takes a few minutes on my machine (WSL2, Ollama, 7b model) and ends usefully most of the time: either compiling property tests I edit and keep, or a genuine counterexample. It found an unchecked rounding case in one of my own vault experiments that my hand-written tests missed, mostly because the model, having no idea what the code was "supposed" to do, tested the boring invariant I had skipped.

That's the honest pitch. It's not an autonomous auditor, it's a tireless, slightly dim intern with a Foundry license, and bounded correctly that's genuinely worth having.

Have you tried giving a local model tools yet, and if so, at what turn count does yours start eating its own tail?