slothaudit: a night-shift auditor for your backtest, on a 1.5B model in your own laptop

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

slothaudit is a command-line auditor for trading-strategy repos that runs entirely offline. Point it at a folder; it runs five deterministic source checks, then a local 1.5B open-weight model reads the code each finding points at and answers one narrow question about it: is this finding actually true here?

You start it before you go to bed. In the morning you read six lines.

Why a night shift and not a chat window: the bugs that kill a backtest are boring and repetitive — a fill taken on the signal bar, a ledger that never mentions fees, one market event counted as eleven samples, a drawdown that resets its own peak. You find them when you already suspect them, which is exactly when you don't need help. So the boring pass runs while you're asleep, at sloth speed, on your own CPU.

Demo

Rendered from a real run's output on 2026-10-23 — CPU only, network interface idle.

$ slothaudit audit samples/broken_lookahead
# slothaudit report — samples/broken_lookahead
Generated 2026-10-23 12:30, offline.

## Morning summary

FIX FIRST: samples/broken_lookahead/strategy.py:10 — fill price read on the signal bar
ALSO: samples/broken_lookahead/strategy.py:12 — no fee or slippage term in the ledger
ALSO: samples/broken_lookahead/strategy.py:8 — entries without a cooldown

## Findings

### CO-01 [high] no fee or slippage term in the ledger
samples/broken_lookahead/strategy.py:12
This file records PnL but never mentions fee, commission, slippage or spread. Searched 24 lines: 0 matches for fee|commission|slippage|spread|cost.
Evidence:
    samples/broken_lookahead/strategy.py:12  pnl = (exit_ - entry) * 100
Model check: **YES**

A clean repo gets the other answer, and it is deliberately not reassuring:

$ slothaudit audit samples/clean_strategy
No v0 check fired. That is not the same as clean: it means this run had
nothing to point at, not that the strategy survives an audit.

Both runs: under a minute on a laptop CPU, no network.

Code

github.com/rytjy/slothaudit — Python standard library only, no dependencies beyond the local model binary.

The split of labour is the point:

• Locations come from code, never from the model. The report prints path:line from the checker. The model is never asked to name a file or a line.
• Every finding prints the line it matched. The checker already knows the exact source line, so the report shows it under Evidence:. For an absence check (no fee term, no cooldown) it prints the ledger line and the number of lines searched. The model's prose is never the reason to open a file.
• The model answers one question per finding, with the surrounding six lines as context, and must start its reply with yes, no or unsure. That is a task a 1.5B model can do. Asking it to "write a summary" produced confident nonsense — so I stopped asking.
• Every report ends with the blind spot, in fixed text: this tool does not run the strategy, does not see your data, and cannot tell you the edge is real.

How I Built It

Prior work, credited. The five checks are distilled from my own earlier kit, backtest-audit-kit, which documents the failing cases they came from. slothaudit is a new project written this week during the challenge; the checks were rewritten as source-level heuristics and the model layer is new.

• Models: two open-weight GGUFs, both served by llama.cpp build b11804 as a subprocess — llama-cli in and out, one process per audit, nothing left running: Qwen2.5-Coder-1.5B-Instruct (1.1 GB) and Gemma 2 2B IT Q4_K_M (1.6 GB). --model qwen or --model gemma picks one; --compare asks both.
• Checks: five regex/AST-level heuristics distilled from real audit cases: same-bar fills, missing cost terms, entries without a cooldown, peak resets in drawdown, and where(x > 0, 0) turning NaN into zero.
• Everything else: Python 3.12 standard library. No key, no account, no telemetry, no upload.

Four decisions came from being honest about what a small local model can do:

1. Few-shot beat instruction. "Reply with VERDICT: YES | NO | UNSURE" made it echo the template back verbatim. A worked example about different code fixed it.

2. When the model has nothing to add, the tool says so. An empty summary is rendered as "deterministic findings only", not as a confident paragraph.

3. An unstable verdict is a finding, not noise. Asked the same yes/no question about the same code three times, the 2B answered NO, NO, YES (temperature 0.2). That is why every finding now prints the source line the check matched: the model's answer is a second opinion, the line is the evidence — and when the two disagree, the line is what you act on.

4. The slow path is the design. Four checks through the 1.5B took about 48 seconds on this laptop's CPU. That number is why the tool is built to run overnight and print six lines in the morning, instead of pretending to be a chat window.

The Lazy Payoff

An audit of the broken sample takes about a minute and produces four findings, one of which — the missing cost term — is the kind of thing I would otherwise notice three weeks later, after wondering why a strategy that "works" loses money.

The honest comparison is not a minute vs 20 minutes of expert review. It is a minute vs the zero minutes I actually spend looking, because after the fifth project nobody re-reads the fill logic. That is the effort it saves: the pass that never happened.

I ran it on both sample repos this week; the clean one fired nothing, which is the result that makes the other one trustworthy.

Prize Categories

Best Use of Gemma. slothaudit asks the same yes/no question of two open-weight models on the same laptop — Qwen2.5-Coder-1.5B-Instruct and Gemma 2 2B IT — both through llama.cpp, with no API key and no network. --compare prints both answers for each finding and marks whether they agreed:

Model check: qwen YES · gemma YES
Agreement: yes

That is a confidence signal, not proof: agreement means neither model disputed the checker's reading of those six lines; disagreement points you at the line to read yourself. Neither model is asked where the bug is, and neither writes the report — they only answer the one question the deterministic checker cannot answer alone.

On the bundled broken sample, two runs of the same audit gave 3 of 4 and then 4 of 4 agreement. The code changed between them, but asking Gemma the cost question again on its own returned NO in both variants — at 2B the verdict is not stable enough to be a metric. So the report leads with the checker's evidence line and treats agreement as a hint. Both runs are committed: 2026-10-22 and 2026-10-23.

Asked on its own on the same code, Gemma returned:

$ slothaudit audit samples/broken_lookahead --model gemma
### LA-01 [high] fill price read on the signal bar
Model check: gemma **YES** — the code fills on the bar that produced the signal.

### CO-01 [high] no fee or slippage term in the ledger
Model check: gemma **YES**

### SI-01 [medium] entries without a cooldown
Model check: gemma **YES**

### MD-01 [medium] peak reset inside the drawdown loop
Model check: gemma **YES** — this code calculates a running peak by adding the current pnl to the previous peak.

Same code, same question, different run: this one says YES on the cost finding, while the 10-22 run and one isolated re-ask said NO. That flip is why the report prints the evidence line instead of a confidence score.

Full report: docs/gemma_only.md.

Why Open Innovation Matters

Three concrete reasons, not slogans:

1. The code never leaves the machine. A trading-strategy repo is a person's edge, and its data includes fills and account sizes. Uploading that to a hosted API to get a lint report is a bad trade. Here the audit runs on the same laptop the repo lives on — I tested it with the network interface down.
2. It costs nothing per run. A nightly audit is 20 model calls a day. On a hosted API that is a meter running; locally it is the laptop you already own, asleep at 3 a.m.
3. The model is swappable. The interface is one function — chat(prompt, model) — with llama.cpp as the local path and an Ollama HTTP server as the fallback. Qwen 1.5B and Gemma 2B are the same three lines of wiring. When a better small model appears, I drop in a GGUF. No re-platforming, no migration, no vendor.

The 1.5B is not a placeholder for a bigger model I wish I could afford. It is a deliberate choice: the checks find the problem, the model only has to answer "is this true here?" — and a small model that answers a narrow question honestly beats a large one that writes a beautiful paragraph I cannot verify.

---

Built during Week 3. The repo includes the two sample strategies used above, and the audit output is reproducible byte-for-byte on any machine with the same GGUF.

AI disclosure: the code and the draft of this post were produced with an AI coding assistant (OpenClaw), working from my direction. Every number here — the timings, the verdicts, the two-model comparison — comes from runs committed in the repo, and the demo GIF is rendered from real output, not staged.