I run an auditor that reads trading-strategy repos while I sleep. A 1.5B model is part of it, and for a long time I described that part wrong.
The tool runs five deterministic checks over the source, then hands each finding to a local model with exactly six lines of context and one yes/no question: is this actually true here? Nothing leaves the machine. No API key, no upload, no account. The model is Qwen2.5-Coder-1.5B-Instruct Q4_K_M, loaded through llama.cpp, on a laptop CPU.
The interesting part is not that it works. It is where it stops working, and what I changed once I measured that.
The 48-second constraint
Four model checks on the bundled sample take about 48 to 50 seconds on this laptop. That is roughly twelve seconds per question. The deterministic pass, --no-llm, takes under a second.
Twelve seconds per question is the fact that shaped everything else. You cannot put that inside a loop over every file in a repository. You cannot put it behind a save button. What you can do is run it at night, or while you make coffee, and then read a report that is already sorted.
Latency also removes a whole class of bad ideas for free. I stopped considering "ask the model to look around and tell me what seems off" not because it is philosophically wrong but because it costs twelve seconds and returns a paragraph, and I do not want a paragraph at that price. I want a pointer.
One question, six lines, three answers
The model never sees the repository. It sees the finding's evidence line plus five lines around it, and it can answer YES, NO, or UNSURE. It is never asked to name a file, never asked for a line number, never asked for a summary.
That last restriction came from a failure. In an early build I asked it to summarise the file. The output was fluent, specific, and wrong about which function did what — the kind of answer that is expensive to read because you have to verify every clause. A checker that points at line 9 and a model that confirms or denies a claim about line 9 is a pipeline. A model that writes prose about code is a different product, and not one a 1.5B can deliver.
So the division is blunt:
- Locations come from code, never from the model.
- The model answers one narrow question per finding, and "UNSURE" is a real answer, not a failure.
- The report prints the evidence line verbatim, so the reason to read a finding is never the model's prose.
The instability I could not engineer away
Here is the measurement that changed my design.
I asked Gemma 2 2B IT Q4_K_M (temperature 0.2) the same yes/no question about the same code, three times, in isolated contexts. The answers were NO, NO, YES.
Same weights, same prompt, same evidence line. Two of three said the cost bug was not there. It was there; the checker had already printed the ledger line with no fee term in it.
I also run both models and print their answers side by side, as a confidence signal. Two runs of the same audit gave 3 of 4 agreement and then 4 of 4. It would be tidy to call that sampling variance, but it is not: the code changed between the runs — I moved the context window to the evidence line — so the difference is confounded. I would rather write that down than round it into a nicer story.
Both runs are committed in the repository, and so are the three re-asks. That is the only claim I am willing to make: at 2B, on this task, the verdict is not a stable measurement. It is a hint about where to look, from a reader who has read six lines.
Once you accept that, the design stops fighting the model.
Five rules that follow
-
The checker owns the finding. Every
path:linein the report comes from code that matched a pattern. The model cannot create a finding, move one, or delete one. - One question per finding. Narrow scope keeps a 1.5B inside the task. Broad scope turns it into a confident narrator.
- The model is allowed to say nothing. "UNSURE" and "nothing to add" are valid outputs, and the report still finishes without them.
- Agreement is a hint, not a vote. Two models agreeing means neither disputed the checker. It does not mean the finding is real, and the report never prints a confidence score I cannot compute.
- Absence checks print what they searched. "No fee term in this file" is a weaker statement than "no match for fee, commission, slippage, or spread across 214 lines". For negative findings the evidence is the search, so the search has to be in the output.
The blind spot is written in fixed text, not generated: the tool does not run the strategy, does not see your data, and cannot tell you whether the edge is real. All it can do is point at the places where a backtest usually lies and ask a small model whether the pointer is correct.
What I have not measured yet
The honest gap is the threshold. Right now a finding either fires or it does not, and the model's answer is read by a human. If I ever wanted to automate the decision — skip the ones the model denies, say — I would need two numbers I do not have:
- the margin around the cutoff, meaning the distribution of scores for true findings versus near-misses, not three anecdotes;
- the flip rate under prompt shift, meaning the same question with a quoted line added or the field order changed.
Both are cheap to measure and both are missing from this post. That is the difference between a demo and something I would let run unattended on a repository I care about. I would rather say that here than discover it in production.
When to skip the model entirely
--no-llm runs the deterministic checks only, instantly, and it is the right mode more often than you would think:
- when you need the same output twice, byte for byte;
- when the repository is small enough that reading the report is faster than waiting;
- when you are about to argue with someone about a number and cannot afford a witness who changes its mind.
The model is worth its twelve seconds when the finding is ambiguous and you want a second reader at 3am. It is not worth it when you already know what to look at.
Try it
pip install -e .
slothaudit audit samples/broken_lookahead # ~50s, four model checks
slothaudit audit samples/broken_lookahead --compare # both models, one report
slothaudit audit samples/broken_lookahead --no-llm # deterministic only, instant
python3 -m slothaudit audit samples/broken_lookahead # works without install
A clean repository gets a deliberately un-reassuring answer: no v0 check fired; that means this run had nothing to point at, not that the strategy survives an audit.
Code: github.com/rytjy/slothaudit
Disclosure: I used an AI assistant to help draft and edit this post. The measurements, the code, and the command output are mine, and the output above is pasted verbatim from runs on this machine.
Top comments (20)
The 'locations come from code, never from the model' rule is the load-bearing one — your summarize experiment shows exactly why: fluent, specific, and wrong is the worst failure mode for a checker, because reading it costs more than the check saves. Twelve seconds per question is a great constraint to design against; it forces the discipline that a fast model would let you skip. What does the UNSURE rate look like at temperature 0.2, and did you calibrate it against a larger judge to see whether UNSURE means 'hard question' or 'weak model'?
Temperature is already 0.2 — that is what the CLI passes, so this one is answerable rather than hypothetical: zero UNSURE in every committed run. Thirteen verdict lines across the five reports, all YES or NO, none UNSURE. At N=13 the rule of three puts the upper bound near 20%, so I cannot claim the model is never unsure — only that this sample never saw it.
There is a second problem inside that zero. The prompt allows yes/no/unsure, and the parser defaults to UNSURE when it cannot find a keyword — so UNSURE currently means both "the model declined" and "we failed to parse it". In the artifact those two are identical, which is the same binary-schema complaint as the other thread, one level up.
No, I have not calibrated it against a larger judge. My only cross-check is qwen vs gemma on the same findings, and both are small — that measures disagreement between two weak judges, not whether UNSURE tracks difficulty. Doing it properly needs a stronger judge plus a difficulty label on the same findings, and at four findings the sample is too small to settle it either way. Widening the sample has to come first.
Zero UNSURE at N=13 is a useful bound, not a verdict — the rule-of-three caveat you state is exactly right. The parser conflation is the sharper problem though: until 'model declined' and 'we failed to parse' are separate codes, the UNSURE rate isn't measuring the model at all, so a larger judge would just calibrate a corrupted signal. Is splitting those two the fix you'd do before widening the sample, or do both have to move together?
Agreed that the conflation is the sharper problem — with one correction: in this sample it has not fired yet. Zero UNSURE means zero of all three sources, so those thirteen verdicts are clean. What is corrupted is the next sample, not the current one: if UNSURE ever appears, the stored record cannot say whether the model declined, the parser missed a keyword, or the process died.
So: split first, and do not wait for the widening to move with it. The split is roughly an hour — three separate codes (unsure, parse_failed, model_error), then re-run the same four findings and check that nothing that used to parse now falls through. Widening is the expensive half and it produces data that is unclassifiable under the current schema, so the order matters more than doing both at once.
To be clear about what the split buys: it does not make UNSURE calibratable at N=13 — nothing does. It only makes the next N worth collecting.
Fair correction — the current thirteen are clean, and the corruption is prospective, not retrospective. The sequencing argument is the convincing part: an hour's split now is what makes the expensive widening worth paying for later, since unclassifiable data at larger N is just a bigger version of the same problem. One thing I'd check on the re-run: do the three new codes change anything about how the thirteen read, or is it purely a forward-looking fix?
Forward-looking only — with one gap the split does not close. The thirteen already parsed to yes or no, and re-running the same four findings should reproduce them; if one lands in parse_failed instead, that tells us the old parser was coercing something, which is worth knowing.
But there is a fourth failure mode the three codes miss: keyword coercion. The parser looks for yes, then no, then unsure, and takes the first hit — so a reply like "no, but if you count partial matches then yes" gets stored as NO with no trace of the ambiguity. Splitting the codes does not fix that, because the parse succeeded; it just succeeded on the wrong word.
The cheap fix is to store the raw line next to the parsed label, so any future question about a verdict is answered from the record instead of by re-running the model. That is also what makes the widening auditable later.
So: split the codes, keep the raw line, then widen.
Keyword coercion is the right fourth mode — first-hit parsing doesn't just mislabel ambiguity, it erases it, so a hedged 'no, but...' becomes a clean NO with no trace it was ever contested. Storing the raw line next to the label fixes the audit problem at its root: future questions get answered from the record, not from a re-run that may not reproduce the same reply. Split the codes, keep the raw, then widen is the right order — would you also flag those mixed-signal replies for human review, or is the raw record enough once widening starts?
Flag them, but as a review queue, not a second gate. If the raw line is stored, a mixed-signal reply is already separable after the fact — the queue can be filled from the record instead of blocking the run.
Once widening starts I'd cap it: the first N a day get human eyes, the rest stay flagged. Otherwise the review backlog becomes the bottleneck and the widest sample stalls on the narrowest resource.
Would you cap by count, or by how strong the disagreement is?
By strength of disagreement, not count — a first-N cap cuts the queue at an arbitrary point, while ranking by how sharply the reply contradicts the label (or how consequential the finding is) means the human sees the cases most likely to flip a verdict first. The count still matters as a daily budget, but as the order you work through, not as a cutoff that drops the rest. The rest just stay flagged — since the queue isn't a gate, an unreviewed flag costs nothing except a later look.
Strength over count is the better rule, and I'll take it: rank by how hard a reply contradicts the label, keep count as the daily budget, and let the tail stay flagged.
The part I'd add is that "sharpness" needs a cheap operational definition — a hedged no, a mid-sentence reversal, a yes/no split inside one line. Otherwise the ranking becomes its own judgment call, and the queue quietly starts picking the easy cases.
That's the right pressure on the ranking — without a cheap definition of sharpness, 'rank by disagreement' just moves the judgment call one level up. Your three markers are a good start: a hedged no, a mid-sentence reversal, a yes/no split in one line are all detectable without a second model call. The test I'd add: can two reviewers apply the same markers to the same reply and rank it the same way — if not, the definition isn't cheap enough yet?
Agreed, and that's the test I can't run honestly — I don't have two reviewers. What I can do is a weaker proxy: same item scored twice, with the reply shown first in one pass and the check shown first in the other, then compare the two rankings. If the order flips, the marker isn't cheap; it's just a different opinion per prompt.
One thing I'd add: decide in advance what happens on disagreement. Averaging two ranks hides the failure. Better to mark the item unranked and publish the disagreement rate next to the daily queue — a definition that needs a tie-breaker isn't operational yet.
Would you set a floor (say, agreement on 4 of 5 before the queue opens), or just report the rate and let it float?
Report the rate and let it float, I'd say — a floor like 4-of-5 turns a measurement into a gate, and then the queue only opens on the days the definition already agrees with itself. Publishing the disagreement rate next to the queue is the honest version: it lets a reader discount a day's rankings without hiding that the definition is still contested. The order-flip proxy is weaker than two reviewers, but as a smoke test it still catches a definition that's just prompt-order noise — if it flips, you know before you spend a reviewer's day on it.
The line in the title has a second checkpoint at write time. A model emits a hint, and then a record format can quietly promote that hint into a verdict. The append-only ledger here took the other fork, and its machine-generated verdict records say something useful about the storage contract before denied findings start getting skipped automatically.
Measured today: across 1,787 stored verdict records the
verdictfield has only ever held two values, 1,749 "passed" and 37 "failed", plus one lone row on a different schema. The third answer has never been written once. Rule 3 makes UNSURE a legal model output. That is a separate thing from a legal stored value, and a binary write schema gives it nowhere faithful to land, so automation collapses it into one of the other two without anything in the artifact recording that it happened. Worth tracing one UNSURE response through the writer and looking at what the saved report actually contains.Those records cover 1,686 distinct task ids. 67 of them were judged more than once, and 4 of those repeat pairs disagree with each other. Stored instability does not average out. It sits there as two equally authoritative rows.
The part that took longest to notice is that a correction mechanism already exists. The signed tag vocabulary includes a tag named
replaces, and in all history it has been used exactly once, on a different record kind, never on a verdict. So not one of those four contradictions is marked as a correction. No field name across the 1,787 contains version, rule, revision, supersede, retract or amend either. Nothing inside a record separates "corrected" from "contradicted", which leaves the tie to be broken by timestamp, a convention that lives outside the record and is not signed with it. Four in 1,686 is 0.24%, quiet enough that no aggregate ever flags it, and it is exactly the subset where the answer was contested.The limit on all of that: the ledger has no truth labels either. A disagreement rate shows the format dropped information, and it cannot say which of the two rows was right. It argues for keeping the supersede slot rather than for picking a winner.
On the flip rate you list as missing, the re-asks are already a measurement. One differing answer in three, on one finding. Widening that to the bundled sample is 3 re-asks x 12 seconds x 4 findings, roughly 144 seconds against the 48-second baseline. Under three minutes, inside the nightly budget the design already assumes, provided each answer is stored next to the exact prompt variant that produced it rather than folded into a single per-finding field.
In the committed pair of runs, does a regenerated report look any different for a finding whose answer flipped than for one answered the same way twice, or does the artifact lose the fact that it was contested?
Short answer: the artifact loses it.
In the two committed reports each finding carries one line per model —
Model check: qwen YES · gemma YES,Agreement: yes— and nothing else. The 10-22 report has CO-01 as gemma NO (3 agree / 1 disagree); the 10-23 report has the same finding as YES (4 agree). Regenerate and the contested state is gone: no history field, no supersede slot, no per-verdict timestamp, no record of which prompt variant produced it. The flip survives only because the two reports were committed as separate files, so git is doing the job the record should do, from outside it.That is your 1,787 records at a smaller scale, and I think your framing is right: the write schema is the bug, not the model. If UNSURE cannot be stored, a contested answer cannot be stored either, and a correction becomes indistinguishable from a contradiction.
One caveat on the pair you are asking about: it is not a clean re-ask. The code changed between the runs (I moved the context window to the evidence line, so line numbers moved from :1/:9 to :10/:12), which is why I would not count 3-of-4 to 4-of-4 as a flip rate. Your 3 re-asks x 12s x 4 findings is the right way to get one — and storing each answer next to its prompt variant is the only version of that which is still checkable a week later.
Taking locations from the code and never from the model is the design choice that makes the rest of this safe. The model can be wrong about whether something matters, but it cannot invent where it is.
The NO, NO, YES result on the same question is a good reminder that a small model gives you a distribution, not an answer. Efe's suggestion of five votes with a 3 to 2 split going to a human turns that instability into a routing rule, which is easier to defend than a confidence score.
On client builds we write down early what the model is allowed to decide and what it may only flag. Triage, ordering and summarising are fine. Anything that closes a ticket or changes data waits for a person or a deterministic check. Agentic Affairs' point about separating UNSURE from a parse failure belongs in that same list, since the two need different handling.
At 12 seconds per question, did batching change the vote pattern at all?
No — and that is the honest answer rather than a modest one. The 12 seconds was one narrow yes/no prompt at a time: four isolated calls, roughly 48–50 seconds total. I never put the four checks into one prompt, so I cannot claim batching is neutral.
I would expect it to move. A batched prompt is longer, and later answers can anchor on earlier ones — which is exactly the contamination the isolated re-asks were there to avoid. The test is cheap: same four checks, five votes each, run isolated vs batched, then compare per-item agreement and how often an item flips. If agreement drops in the batch, the votes have to stay per-question and the 12 seconds is the real cost.
NO, NO, YES from three re-asks at temperature 0.2 is hard to read as a rate. With five votes per finding the split carries more than any single verdict: 5-0 and 3-2 are different statements, and the 3-2 findings can go to the human first. I grade model output with five votes for that reason. At twelve seconds a question, that is a minute per finding, which fits an overnight run. You list the flip rate under prompt shift as unmeasured. Would you report it per check, as the share of findings whose majority changes when the context line moves, instead of one global number? Unstable findings may cluster in a few checks.
That is the better design, and I would take five votes over three re-asks: 5-0 and 3-2 are different claims, and routing the 3-2 findings to a human is a decision the tool can make from the votes alone.
On per-check reporting: yes, but with one condition. The variation I would expect is across checks rather than across findings — a check whose criterion is fuzzy ("no cooldown") should flip more than one that is structural ("the fill reads index i"). A global flip rate averages exactly that away.
The catch is sample size per cell. I have four findings and four checks, so a per-check table is one finding per cell — descriptive, not a rate, and I would label it that way rather than dress it up. If I widen, the target is ten findings per check at five votes each: 4 checks x 10 x 5 = 200 model calls, about 40 minutes at twelve seconds. That fits an overnight run, and it produces a table instead of a number.
The sentence that carries this post concerns locations: they come from code, never from the model. A small model can then triage, because a wrong guess costs a check, not a shipped claim.