DEV Community: Krishnasai Rollamadugu

LLM-as-judge, responsibly: a checklist before you trust an AI to grade anything

Krishnasai Rollamadugu — Wed, 21 Oct 2026 08:33:48 +0000

"LLM-as-judge" has quietly become infrastructure. We use one model to grade another's answers, to decide if a test really passed, to score RAG relevance, to moderate, to rank. It's cheap, fast, and scales to volumes no human can touch.

It's also a measurement instrument — and most teams deploy it without ever checking whether the instrument is accurate. Here's a practical checklist before you let an LLM grade anything that matters. (Context: I built a small harness that runs a local LLM as a test judge and scores it against ground truth; some numbers below come from that.)

1. Measure it against ground truth at least once

You cannot manage what you haven't measured. Assemble a set of cases where you know the right verdict (labelled correct/wrong) and run the judge over them. You're looking for:

False-pass rate — of everything it approved, how much was actually wrong? (The dangerous direction.)
Miss rate — of the genuinely wrong items, how many did it wave through?
False-alarm rate — how often does it reject good work? (The annoying direction.)

In my run, a capable open model missed 26% of wrong outputs — a quarter of real failures, passed. You want your number, on your task, before you rely on it.

2. Check calibration, not just accuracy

Ask the judge for a confidence, then check whether that confidence means anything. A well-calibrated judge that says "90% sure" should be right about 90% of the time. Measure Expected Calibration Error (ECE).

The common failure: models are massively overconfident. Mine answered with ~100% confidence on nearly every call — including the wrong ones (ECE ≈ 0.31). If the confidence is always "very sure," it carries no signal and you can't use it to route hard cases to a human.

3. Pin everything and version it

An LLM judge whose model, prompt, or temperature drifts is a ruler that changes length. Pin:

the exact model (and version),
the prompt template,
temperature (use 0 for grading — you want determinism, not creativity).

Open-weight models help here: you can pin a local model and know it won't change under you. A hosted endpoint can be swapped silently.

4. Watch for the known biases

LLM judges have documented, reproducible biases:

Position bias — prefers the first (or last) option in a pairwise comparison. Mitigate by running both orders and averaging.
Verbosity bias — rates longer answers higher regardless of correctness.
Self-preference — a model tends to favour outputs that look like its own.
Leniency — many judges default to "PASS" when unsure.

If your eval drives a leaderboard or a release decision, these biases become your biases.

5. Never let it be the only gate

The safe pattern isn't "judge decides, ship." It's:

Use the judge as a cheap first-pass filter, not the final word.
Keep a ground-truth backstop on a sample — spot-check the judge continuously, not once.
Route low-confidence or high-stakes cases to a human or a stronger check.
Treat a PASS as "probably fine," not "verified."

The one-line version

An LLM judge is a fast, cheap, biased, overconfident grader that is right often enough to be useful and wrong often enough to be dangerous. Measure it, pin it, debias it, and keep a human backstop — then it's a great tool. Trust it blindly and it will quietly approve your worst bug with total confidence.

Are you using LLM-as-judge in production? What did you find when you checked it against ground truth — or have you checked yet?

I'm Krishnasai Rollamadugu — a test automation architect researching trust and measurement in AI-driven software testing. Related: an open experiment measuring a local LLM judge — trust-judge.

The oracle problem: testing's oldest unsolved question, and why AI just made it urgent

Krishnasai Rollamadugu — Wed, 21 Oct 2026 08:33:25 +0000

Here's a question that sounds trivial and isn't: when your program produces an output, how do you know it's the right one?

That's the test oracle problem, and it's one of the oldest open problems in software testing (the definitive survey is Barr, Harman, McMinn, Shahbaz & Yoo, IEEE TSE 2015). Every test you've ever written quietly answered it somehow — and AI testing tools are now answering it for you, which is exactly why it's worth understanding.

What an "oracle" actually is

An oracle is the mechanism that decides pass or fail. When you write:

assert add(2, 3) == 5

the == 5 is your oracle. You, the human, knew the right answer and hard-coded it. Easy.

Now scale that up. What's the correct output of a route planner for a 10,000-node graph? A tax calculation across 40 interacting rules? An ML model's ranking? A PDF renderer? You often cannot just write down the expected answer. That's the oracle problem: the hard part of testing isn't running the program, it's knowing what correct looks like.

How we actually cope (the oracle taxonomy)

We rarely have a perfect oracle, so we use partial ones:

Specified oracle — a spec or reference implementation says the exact answer. Strongest, rarest.
Derived oracle — infer correctness from other artifacts: a previous version (regression), documentation, or a slower-but-trusted implementation.
Metamorphic oracle — when you can't check one output, check a relation between outputs. You may not know sin(x), but you know sin(x) == sin(x + 2π). Hugely useful for scientific code, ML, and anything without a ground truth.
Implicit oracle — the "obvious" failures anyone can detect without a spec: crashes, hangs, memory corruption. Cheap but weak — "it didn't crash" is a very low bar.
Human oracle — a person eyeballs it. Expensive, unscalable, inconsistent.

Most real suites are a pile of implicit + specified oracles, which is why they catch crashes and the handful of cases someone thought to assert — and miss the subtle wrong-but-plausible outputs.

Why AI makes this urgent

Two shifts just collided with the oracle problem:

AI writes the oracle. When an LLM generates your assertions, it is deciding what "correct" means — and models love weak, behaviour-echoing assertions (type checks, "no exception", loose ranges) that look like tests but barely constrain the output. A green suite full of weak oracles is the oracle problem wearing a disguise.
AI is the oracle. "LLM-as-judge" makes the model itself the pass/fail mechanism. That's an implicit-ish oracle with opinions — convenient, scalable, and unaccountable unless you measure it. (I've measured one: a local LLM judge waved through ~1 in 4 genuinely wrong outputs while ~100% confident.)

In both cases we've quietly handed the oldest hard problem in testing to a system that's fluent, fast, and frequently wrong — without noticing we did it.

The practical takeaway

Name your oracle. For any important test, ask: what decides correct here, and how strong is it? "It returns a string" is not an oracle; it's a vibe.
Reach for metamorphic relations when there's no ground truth — they're underused and powerful.
If an AI is your oracle, treat it as an oracle under test: check it against real ground truth, measure how often it's wrong, and don't trust its confidence.

The oracle problem never went away. We just built tools that paper over it more convincingly. Knowing it's there is most of the battle.

How do you solve the oracle problem on your hardest-to-verify code? Metamorphic testing? A reference implementation? I'd love to hear the approaches.

I'm Krishnasai Rollamadugu — a test automation architect researching trust and measurement in AI-driven software testing.

Flaky tests aren't random: a field guide to finding and killing them

Krishnasai Rollamadugu — Wed, 21 Oct 2026 08:31:47 +0000

A flaky test is one that passes and fails on the same code with no change in between. It's the most corrosive thing in a test suite — not because any single failure is scary, but because flakiness teaches your team the worst possible habit: ignoring red.

Once "just re-run it" becomes the reflex, your suite stops being a safety net and becomes a slot machine. Here's a practical field guide to the usual culprits and how to actually kill them — built from years of chasing these in real automation suites.

Flakiness is never truly random

It feels random, but it almost always traces to one of a handful of non-determinisms. The job is to find which one.

1. Timing / async (the #1 cause in UI and integration tests)

The test checks for something before the app is ready. It passes when the machine is fast, fails when CI is loaded.

Fix: never sleep for a fixed time. Wait for a condition — an element state, a network response, a value. Modern frameworks (Playwright, Cypress) auto-wait on assertions; lean on that instead of sleep(2).
Smell to grep for: sleep, setTimeout, hard-coded waits.

2. Test order dependence / shared state

Test B passes only because Test A ran first and left data behind. Run them in a different order (or in parallel) and B falls over.

Fix: each test sets up and tears down its own state. No reliance on execution order. Tools like pytest-randomly or iDFlakies surface these by shuffling order.

3. Shared external resources

A real database, a shared test account, a live third-party API. Two runs collide, or the API has a bad day.

Fix: isolate data per test (unique keys), and stub the network at the boundary (cy.intercept, WireMock, VCR). Test your logic, not someone's uptime.

4. Time, timezones, randomness, locale

Tests that use now(), random seeds, or locale-dependent formatting fail at midnight, on a CI box in another timezone, or on the one day the random value hits an edge.

Fix: inject the clock and the seed. Freeze time (freezegun, sinon), pin the seed, pin the locale.

5. Resource leaks and ordering in the app itself

Sometimes the test is honest and the code is flaky — an unawaited promise, a race in the app, a dictionary iteration order assumption. The test just exposed it.

Fix: this one's a gift. The flaky test found a real concurrency bug. Fix the app, not the test.

A triage process that works

Quarantine, don't ignore. Move the flaky test to a separate suite that doesn't block the pipeline — but tracks it. The goal is to stop the "ignore red" habit, not to hide the problem.
Reproduce deterministically. Run it 50 times locally (pytest --count=50, or loop it). If it never fails locally but fails in CI, the difference is the clue — load, parallelism, environment.
Bisect the cause. Run it in isolation vs. with the suite (order dependence?). Disable the network (external dependency?). Freeze the clock (time?).
Fix the root cause, then de-quarantine. And add the reason to your commit message, so the next person learns the pattern.

Measure it, don't vibe it

Track a flaky rate per test (failures ÷ runs over the last N runs). It turns "the suite feels flaky" into a number you can drive down, and it tells you which tests to fix first. A test that's failed 8% of the last 200 runs is a priority; one that failed once six weeks ago is noise.

The mindset

Every flaky test is a small lie your suite tells. The fix isn't more retries — retries just make the lie quieter. The fix is removing the non-determinism, one category at a time, until green means green again.

What's the flakiest test you've ever had to hunt down — and what turned out to be the cause? I collect these war stories.

I'm Krishnasai Rollamadugu — a test automation architect who writes about test reliability and trust in AI-driven testing.

Mutation testing in 5 minutes: catch the tests that secretly test nothing

Krishnasai Rollamadugu — Wed, 21 Oct 2026 08:20:45 +0000

Your test suite is green. 100% passing. Maybe even 90% coverage.

Here's the uncomfortable question: if someone introduced a real bug right now, would your tests actually go red?

Coverage can't answer that — it only tells you a line ran, not that anything checked the result. A test can execute every line and assert nothing meaningful. It'll be green forever, including when the code is broken.

The technique that does answer it is mutation testing, and you can try it on your own code in about five minutes.

The idea in one sentence

Change the code to introduce a bug on purpose, run your tests, and see if they catch it.

Each small deliberate change is a "mutant" — flip a + to a -, a > to a <, a True to a False. For every mutant you run your suite:

tests fail → the mutant is killed → good, your tests caught the bug.
tests still pass → the mutant survived → a blind spot. Your tests didn't notice the code changed behaviour.

Your mutation score = killed / total. It's an honest measure of fault-detection, not just execution.

Try it in 5 minutes

I'll use a tiny open-source tool I built, trust-gate — zero dependencies, works with any test runner. (If you're on the JVM or JS, use PIT or Stryker instead — same idea.)

git clone https://github.com/rollamadugu/trust-gate
cd trust-gate
pip install -e ".[dev]"

The repo ships a tiny pricing.py with a weak test suite and a strong one. Both are 100% green. Watch what mutation testing says about each.

Weak suite — it runs the code but barely checks the values:

trust-gate examples/pricing.py -c "pytest examples/test_pricing_weak.py -q" --cwd examples

mutants         : 7
caught (killed) : 0
survived        : 7
mutation score  : 0.00%
gate: FAIL

Zero. Every injected bug slipped through while the suite stayed green.

Strong suite — same code, same "all passing", but it checks real values across branches:

trust-gate examples/pricing.py -c "pytest examples/test_pricing_strong.py -q" --cwd examples

mutants         : 7
caught (killed) : 7
survived        : 0
mutation score  : 100.00%
gate: PASS

Same green checkmark on your dashboard. Completely different safety nets.

Reading the result

The useful part isn't the score — it's the surviving mutants. trust-gate prints each one with a line number and what it changed:

SURVIVED  line 9: Gt -> Lt
SURVIVED  line 11: 2 -> 3

Each survivor is a precise to-do: "nothing you wrote would notice if line 9's comparison flipped." That points you straight at the assertion you're missing — far more actionable than a coverage percentage.

Make it a CI gate

Because trust-gate exits non-zero when the score is below a threshold, you can fail the build when test strength drops:

- run: pip install trust-gate
- run: trust-gate src/your_module.py -c "pytest -q" --min-score 0.75

Now "green" means "and the tests would actually catch a bug," not just "ran without error."

Caveats (so you use it well)

Equivalent mutants: some changes don't alter behaviour, so they can't be killed. Don't chase 100%; chase understanding your survivors.
Speed: mutation testing re-runs your suite many times. Start on one important module, not the whole repo.
It grades the tests, not the code. A surviving mutant means your tests have a gap — exactly what you want to know.

The takeaway

Coverage tells you what ran. Mutation testing tells you what your tests would actually catch. Point it at your most important module today and look at the survivors — it's usually a humbling five minutes, in the best way.

Tool used: trust-gate (MIT, open source). Feedback and PRs welcome.

rollamadugu / trust-gate

Mutation-based test-strength gate: fail CI when your passing tests are too weak to catch real bugs. Zero dependencies, any test runner.

trust-gate

Your tests are green. But are they strong enough to catch a real bug?

trust-gate injects small faults (mutants) into your code, runs your test command for each one, and measures how many faults your passing tests actually catch. Green tests that miss injected faults are weak tests — trust-gate makes that measurable and lets you fail CI when test strength drops below a threshold.

It's a focused, zero-dependency mutation-testing gate. Pure Python standard library. Works with any test runner (pytest, unittest, a shell script — anything that exits non-zero on failure).

Part of a broader line of work on trust in AI-driven testing (TrustLab). As AI writes more of our tests, "it passed" needs to mean "it would have caught the bug." This gate checks exactly that.

Install

pip install trust-gate        # from PyPI (coming soon)
# or, from source:
pip install -e ".[dev]"

…

View on GitHub

I'm Krishnasai Rollamadugu — a test automation architect researching trust and measurement in AI-driven software testing. If weak-but-green tests are your kind of problem, I write about this a lot.

The lazy dream is an AI that judges your tests. I measured what it costs

Krishnasai Rollamadugu — Wed, 21 Oct 2026 07:52:40 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

The laziest dream in software testing goes like this: stop checking your tests yourself — just let an AI read the output and tell you PASS or FAIL. The effort it promises to save is the most tedious part of QA: deciding whether a result is actually correct. The person who gets to be lazier is every tester who'd rather not eyeball one more assertion.

So I built trust-judge to take that lazy dream seriously and measure what it actually costs. It runs a local, open-weight LLM (Llama 3.1 8B via Ollama) as a test "judge" and scores how often you can believe its verdict — against objective ground truth the model never sees.

The theme is "Slow and Steady," and that became the finding, not just the vibe: the lazy shortcut (trust the fast AI green and move on) quietly misses real bugs. The slow, steady habit — measure it, keep a ground-truth check — is what actually works.

Demo

A local Llama 3.1 8B judged 81 test outcomes (correct vs. buggy), scored against a reference implementation it never saw:

Accuracy 69% — wrong about 1 in 3 times.
Miss rate 26% — it waved through a quarter of genuinely wrong outputs as PASS. Silent false passes: dashboard green, bug shipped.
False-pass rate 10.6% (95% CI 4.6–22.6%).
Calibration ECE 0.31 — it answered with confidence ≈ 1.00 almost every time, even when wrong.

It fails exactly where it matters — arithmetic and inclusive boundaries:

discount (rounding) collapses to 40% accuracy with 80% of wrong answers missed; on letter_grade it decided a score of exactly 60 was an "F" — failing the correct "D" and passing the wrong "F".

Run it yourself in ~5 minutes:

ollama serve &
ollama pull llama3.1:8b
pip install -e ".[dev]"
trust-judge --model llama3.1:8b -o results/run.json

Code

rollamadugu / trust-judge

Measure how much you can trust a local open-weight LLM (Ollama) acting as a software test oracle (LLM-as-judge), against objective ground truth. Built for the DEV Hacktoberfest 2026 open-source AI challenge.

trust-judge

"LLM-as-judge" is everywhere now. But can you trust the judge?

trust-judge points a local, open-weight LLM (Llama 3.1 8B via Ollama running entirely on your machine) at software test outcomes and measures how often you can believe its verdict — against objective ground truth (a reference implementation). No cloud, no API key, no data leaving your laptop.

Built for the DEV Hacktoberfest 2026 "open-source AI" challenge: an open-weight model running locally is the engine, and that openness is exactly what makes a trust study like this possible — you can inspect it, pin it, and reproduce it for free.

The result (Llama 3.1 8B, 81 labelled cases)

Accuracy 69% — the judge is wrong about 1 in 3 times.
Miss rate 26% — it waved through a quarter of genuinely wrong outputs as PASS. Those are silent false passes: the dashboard stays green, the bug ships.
False-pass rate…

View on GitHub

How I Built It

Open-weight model, 100% local, pure standard library (plus pytest):

Subjects — small functions with a plain-English spec, a correct reference implementation, and a realistic buggy variant.
Cases — for each input, a correct case (reference output) and a wrong case (buggy output when it differs). Truth is objective: correct ⟺ output equals the reference.
The judge sees only spec + call + returned value — never the reference — and must answer PASS/FAIL with a confidence, through the local Ollama API (stdlib urllib, no SDK).
Metrics — false-pass rate (+ Wilson CI), miss rate, false-alarm rate, accuracy, confidence ROC-AUC vs. a naive baseline, and Expected Calibration Error.

The test suite (12 tests) is self-contained and never calls the model, so CI stays green without a GPU.

Why Does Open Innovation Matter?

A study about trust only works if the thing you're studying is open:

No internet, no API key — the model runs on my laptop. Nothing leaves the machine, which matters when the "outputs" under test could be real, sensitive code.
Reproducible for free, forever — anyone can ollama pull llama3.1:8b and get the exact setup. No vendor can silently swap the model under my results.
Swappable — change one flag (--model qwen2.5:7b) and re-run. Open weights turn "trust the AI" from faith into an experiment you can rerun and challenge.

You cannot do an honest, reproducible trust study on a black box you rent by the token. Open models are what let you check my numbers instead of taking my word.

The Lazy Payoff

Here's the honest version of the payoff, because the whole project is a test of it.

Does an LLM judge save effort? Yes — it does the tedious "is this output right?" check for you, instantly, at zero marginal cost. That's the sloth dream.

But the bill comes due quietly: it let 26% of real bugs through and was ~100% confident while doing it. So the fully-lazy move — fire the judge into your pipeline and stop checking — doesn't actually save you; it just moves the cost to the bug that ships.

The payoff that survives contact with the data is the slow-and-steady one: use the judge as a cheap first-pass filter, keep a ground-truth backstop, and distrust its confidence. That genuinely saves time and stays honest. Lazy where you can afford it; steady where it counts.

Built by Krishnasai Rollamadugu — Test Automation Architect researching trust in AI-driven software testing. Related: TrustLab · awesome-ai-software-testing.

Can you trust a green test written by AI? I measured it.

Krishnasai Rollamadugu — Tue, 20 Oct 2026 10:41:06 +0000

I've spent eleven years building test automation for enterprise systems. The whole job rests on one quiet assumption: a passing test is evidence. Green means safe. We ship on green.

Lately that assumption has been getting shakier, because more and more of the test itself is being produced by AI:

AI generates the test oracle (the assertion that decides pass/fail).
AI heals broken UI locators when the page changes (self-healing frameworks).
AI adjudicates whether a run actually passed or was "just flaky" (LLM-as-judge).

Each of these can hand you a GREEN result that isn't actually sound:

An AI-written assertion can be too weak to catch a real bug — it checks the type but not the value, so the test passes even when the output is wrong.
A self-healing locator can "repair" itself onto the wrong element and sail through, silently testing nothing.
An LLM judge can wave a genuine failure through as a flake.

In all three cases the dashboard stays green while the safety net quietly grows holes. I wanted to stop hand-waving about this and actually measure it. So I built a small, open research prototype called TrustLab.

The idea: the False-Confidence Gap

The core concept is simple:

False-Confidence Gap = how often tests appear to pass − how often they are actually sound.

To measure it honestly you need ground truth for "sound." I use mutation testing: an engine injects real faults into a program (flip a + to a -, a < to a <=, and so on), and a verdict only counts as sound if it actually catches those faults. No opinions, no self-report — a test either kills the mutant or it doesn't.

I ran this across the three modalities above, on a set of small subject programs, repeated over 15 random seeds with held-out evaluation.

What came out

AI-generated oracles: ~62% gap. This is the scary one. A huge share of "passing" came from assertions that don't actually catch injected faults.
LLM-as-judge: ~41% gap.
AI self-healing: ~11% gap — lower, but with silent mis-heals hiding in the tail (the worst kind, because nothing looks wrong).

A detail I didn't expect: for AI-generated oracles, the danger tracks code structure. Branch-heavy logic like a grading function (99.7% gap) or a triangle classifier (93.6%) was far more dangerous than a simple BMI bucket (5.2%). The more branches your code has, the more room a weak AI oracle has to look green while missing faults.

Can we do anything about it?

Measuring the problem is only half of it. The other half: can we predict which green verdicts to believe, using only signals we can actually observe at runtime (not the ground truth, which in real life we don't have)?

I trained a small, calibrated trust metric on observable features and compared it to the naive policy everyone uses today — "it's green, trust it."

If you keep the 70% most-trusted green verdicts and set aside the least-trusted 30%, the silent false-pass rate drops hard:

Self-healing: 11% → 0.5%
LLM-judge: 40% → 19%

The naive "trust any green" baseline can't separate sound from unsound at all (ROC-AUC 0.50 by construction). The trust metric reaches ROC-AUC up to ~0.95, with calibration (Expected Calibration Error) reported so the scores actually mean something. Effect sizes are "large" (Vargha–Delaney A12 0.76–0.95), Mann–Whitney p < 0.001.

Being honest about what's real

I think credibility matters more than hype, so I'm explicit about this in the repo too:

The oracle results are real: a genuine AST mutation engine injects faults, and runtime coverage decides whether a verdict is sound.
The self-healing and adjudication results are empirically-shaped models of documented tool and LLM behaviour, placed behind an adapter interface. That means you can drop a real Playwright self-healer or a real LLM judge into the same seam and re-run the exact experiments without touching the metric. (That's literally one of the open issues on the repo — contributions welcome.)

I'd rather you know exactly where the solid ground is.

Why it matters for the rest of us

If your pipeline uses AI to generate assertions, heal locators, or judge flakiness, "green" is no longer self-evidently trustworthy. Two practical takeaways:

Weak oracles are the biggest risk, and they're worst exactly where your code is most complex. AI-generated assertions deserve review in proportion to branch complexity.
A trust score beats a raw green/red. Even a simple calibrated metric over observable signals can tell you which passes to re-examine — turning a binary into a prioritised queue.

Try it / break it

Everything is open and reproducible — pure Python standard library core, 19 passing tests, the benchmark dataset, and the experiment outputs:

👉 github.com/rollamadugu/trustlab

I'd genuinely value feedback from people who build test oracles, self-healing frameworks, or LLM-evaluation pipelines — especially if you can poke holes in the methodology. There are a couple of "good first issues" open if you want to plug in a real tool. A fuller write-up (preprint) is in preparation.

I'm Krishnasai Rollamadugu, a test automation architect (11+ years) researching trust and measurement in AI-driven software testing. If this resonates, follow along — this is the first step of a longer research direction.