The lazy dream is an AI that judges your tests. I measured what it costs

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

The laziest dream in software testing goes like this: stop checking your tests yourself — just let an AI read the output and tell you PASS or FAIL. The effort it promises to save is the most tedious part of QA: deciding whether a result is actually correct. The person who gets to be lazier is every tester who'd rather not eyeball one more assertion.

So I built trust-judge to take that lazy dream seriously and measure what it actually costs. It runs a local, open-weight LLM (Llama 3.1 8B via Ollama) as a test "judge" and scores how often you can believe its verdict — against objective ground truth the model never sees.

The theme is "Slow and Steady," and that became the finding, not just the vibe: the lazy shortcut (trust the fast AI green and move on) quietly misses real bugs. The slow, steady habit — measure it, keep a ground-truth check — is what actually works.

Demo

A local Llama 3.1 8B judged 81 test outcomes (correct vs. buggy), scored against a reference implementation it never saw:

• Accuracy 69% — wrong about 1 in 3 times.
• Miss rate 26% — it waved through a quarter of genuinely wrong outputs as PASS. Silent false passes: dashboard green, bug shipped.
• False-pass rate 10.6% (95% CI 4.6–22.6%).
• Calibration ECE 0.31 — it answered with confidence ≈ 1.00 almost every time, even when wrong.

It fails exactly where it matters — arithmetic and inclusive boundaries:

discount (rounding) collapses to 40% accuracy with 80% of wrong answers missed; on letter_grade it decided a score of exactly 60 was an "F" — failing the correct "D" and passing the wrong "F".

Run it yourself in ~5 minutes:

ollama serve &
ollama pull llama3.1:8b
pip install -e ".[dev]"
trust-judge --model llama3.1:8b -o results/run.json

Code

rollamadugu / trust-judge

Measure how much you can trust a local open-weight LLM (Ollama) acting as a software test oracle (LLM-as-judge), against objective ground truth. Built for the DEV Hacktoberfest 2026 open-source AI challenge.

trust-judge

"LLM-as-judge" is everywhere now. But can you trust the judge?

trust-judge points a local, open-weight LLM (Llama 3.1 8B via Ollama running entirely on your machine) at software test outcomes and measures how often you can believe its verdict — against objective ground truth (a reference implementation). No cloud, no API key, no data leaving your laptop.

Built for the DEV Hacktoberfest 2026 "open-source AI" challenge: an open-weight model running locally is the engine, and that openness is exactly what makes a trust study like this possible — you can inspect it, pin it, and reproduce it for free.

The result (Llama 3.1 8B, 81 labelled cases)

• Accuracy 69% — the judge is wrong about 1 in 3 times.
• Miss rate 26% — it waved through a quarter of genuinely wrong outputs as PASS. Those are silent false passes: the dashboard stays green, the bug ships.
• False-pass rate…

View on GitHub

How I Built It

Open-weight model, 100% local, pure standard library (plus pytest):

• Subjects — small functions with a plain-English spec, a correct reference implementation, and a realistic buggy variant.
• Cases — for each input, a correct case (reference output) and a wrong case (buggy output when it differs). Truth is objective: correct ⟺ output equals the reference.
• The judge sees only spec + call + returned value — never the reference — and must answer PASS/FAIL with a confidence, through the local Ollama API (stdlib urllib, no SDK).
• Metrics — false-pass rate (+ Wilson CI), miss rate, false-alarm rate, accuracy, confidence ROC-AUC vs. a naive baseline, and Expected Calibration Error.

The test suite (12 tests) is self-contained and never calls the model, so CI stays green without a GPU.

Why Does Open Innovation Matter?

A study about trust only works if the thing you're studying is open:

• No internet, no API key — the model runs on my laptop. Nothing leaves the machine, which matters when the "outputs" under test could be real, sensitive code.
• Reproducible for free, forever — anyone can ollama pull llama3.1:8b and get the exact setup. No vendor can silently swap the model under my results.
• Swappable — change one flag (--model qwen2.5:7b) and re-run. Open weights turn "trust the AI" from faith into an experiment you can rerun and challenge.

You cannot do an honest, reproducible trust study on a black box you rent by the token. Open models are what let you check my numbers instead of taking my word.

The Lazy Payoff

Here's the honest version of the payoff, because the whole project is a test of it.

Does an LLM judge save effort? Yes — it does the tedious "is this output right?" check for you, instantly, at zero marginal cost. That's the sloth dream.

But the bill comes due quietly: it let 26% of real bugs through and was ~100% confident while doing it. So the fully-lazy move — fire the judge into your pipeline and stop checking — doesn't actually save you; it just moves the cost to the bug that ships.

The payoff that survives contact with the data is the slow-and-steady one: use the judge as a cheap first-pass filter, keep a ground-truth backstop, and distrust its confidence. That genuinely saves time and stays honest. Lazy where you can afford it; steady where it counts.

Built by Krishnasai Rollamadugu — Test Automation Architect researching trust in AI-driven software testing. Related: TrustLab · awesome-ai-software-testing.