DEV Community: Zohair

Why I Stopped Using LLM-as-a-Judge (And What I Use Instead)

Zohair — Tue, 04 Aug 2026 10:45:03 +0000

Six months ago I was running evals on a coding Q&A dataset and my OpenAI bill hit $47 before I finished my test run. I wasn't doing anything exotic — just asking GPT-4 to score 500 model responses on a rubric. That's the hidden cost of LLM-as-a-Judge: every evaluation call is itself an API call, which means the more you evaluate, the more you pay.

I spent a weekend building an alternative. The idea was simple: train a local binary classifier using Sentence Transformers and logistic regression to separate "good" from "bad" responses. No API. No cloud. No credit card. The result was a tool that runs at ~8ms per sample and costs $0 to operate indefinitely. It hit 75% accuracy on coding Q&A out of the box. Not perfect — but good enough to catch regressions, and fast enough to run in CI on every commit.

Here's the full story of how it works, where it breaks, and when you should (and shouldn't) use it instead of GPT-4-based eval.

The Problem with LLM-as-a-Judge in 2026

1. Cost compounding: eval calls multiply with dataset size and iteration count
2. Latency kills CI: 2-5 second API latency per sample = minutes for a 1,000-sample suite
3. Bias and instability: 2026 research showing LLM judges are prompt-sensitive, exhibit taxonomic bias, and can flip scores with minor prompt rewording
4. Lock-in: if the model changes upstream (GPT-4 → GPT-5), your eval scores drift even if your app didn't change
5. The OpenAI-acquires-Promptfoo moment: what it means for "neutral" open-source eval

The Architecture of LLM Judge

1. Step 1: embed query + response pairs using sentence-transformers (all-MiniLM-L6-v2)
2. Step 2: train a logistic regression classifier on labeled good/bad examples
3. Step 3: at eval time, embed and classify — no network call, no API key
4. Why logistic regression (not a neural classifier): interpretability, speed, no GPU required, works with small label sets
5. Code snippet: full training loop in ~20 lines of Python

Benchmark: 75% Accuracy on Coding Q&A

1. Dataset description: coding question + model answer pairs, human-labeled
2. How accuracy compares to random (50%) and human-agreement ceiling (~85%)
3. Where the model is wrong: ambiguous cases, style vs. correctness confusion, edge cases in multi-step reasoning
4. Honest take: this is not a replacement for human review on ambiguous cases — it's a regression detector

When to Use It (And When Not To)

Use it: CI smoke tests, catching obvious regressions, high-volume eval where API cost is prohibitive, air-gapped / local-first environments, early-stage projects without budget
Don't use it: nuanced long-form eval, multi-modal, tasks requiring domain expertise to label, when you need explainability on individual failures (logistic regression gives a score, not a reason)
Hybrid approach: use LLM Judge for volume, LLM-as-a-judge for spot-checks on borderline cases

How to Extend It to Your Domain

1. Labeling your own dataset: minimum viable label set (~100–200 examples), how to use weak supervision to bootstrap
2. Swapping the embedding model: when to use domain-specific embeddings (legal, medical, code-specific)
3. Adding multi-class scoring: extending from binary (good/bad) to rubric-based (0–3)
4. Integration patterns: pytest plugin, pre-commit hook, GitHub Actions step

What's Next

1. Pre-trained domain packs (coding, customer support, RAG faithfulness)
2. LangChain / LlamaIndex integration
3. Active learning loop: flag low-confidence samples for human review, retrain
4. Link to GitHub: github.com/Zoh007/llm-judge — PRs and issues welcome

What's your current eval setup? Are you paying for API-based eval, running something local, or just skipping evals entirely? I'm curious what's actually working for people in production.

The Promptfoo Acquisition Made Me Realize I Was Evaluating LLMs on Easy Mode

Zohair — Mon, 03 Aug 2026 23:04:01 +0000

In March 2026, OpenAI acquired Promptfoo — the tool that 350,000 developers use to evaluate and red-team OpenAI's own models.

I don't think that makes Promptfoo suddenly useless. But it made me ask a question I should have asked earlier: who is the eval tool accountable to?

Most LLM eval frameworks — DeepEval, Ragas, LangSmith — have something in common. They all route scoring through an LLM API call. That means your eval pipeline has at least three dependencies you don't control: the tool vendor, the judge model vendor, and whatever pricing changes either of them makes next quarter.

When I started building LLM Judge, I wasn't thinking about vendor independence. I was thinking about cost. I was running evaluations on a coding Q&A dataset, and every DeepEval run was making LLM API calls I hadn't budgeted for. It added up fast.

So I tried something simpler: train a classifier. Sentence Transformers to embed the responses, logistic regression to classify quality. The result: 75% accuracy on coding Q&A, ~8ms per sample, $0 per run, runs fully offline.

It's not magic. It won't replace human review for high-stakes decisions. But for fast feedback loops in CI/CD — the place where LLM eval matters most — it's surprisingly good.

What this approach gives you:

No API key required. Ship to air-gapped environments, run in restricted CI, deploy anywhere Python runs.
No vendor lock-in. You own the model. You trained it. No one can deprecate it out from under you.
Speed that fits CI. 8ms/sample means you can eval 1,000 outputs in under 10 seconds.
Customizable to your domain. Fine-tune on your data and your definitions of "good."

The tradeoff: you need labeled training data. If you have zero examples, start with a small hand-labeled set. 50–100 examples is enough to get a useful baseline classifier.

How to get started:

pip install llm-judge

Then train on your data and run evals locally. No signup. No API key. No monthly bill.

The Promptfoo situation is a useful reminder: the best eval tool is one you understand, own, and can run independently of whoever built the model you're testing.

→ github.com/Zoh007/llm-judge

What's your current eval setup? Especially curious if anyone's moved away from API-dependent evals — drop a comment.

How I Built a No-Execution LLM Eval Judge (75% Accuracy, No API Calls)

Zohair — Fri, 31 Jul 2026 15:01:50 +0000

Evaluating LLM outputs is one of those problems that sounds simple until you actually try to do it at scale.

Most approaches fall into three buckets:

Run the code — works for coding tasks but requires a sandbox, is slow, and breaks on edge cases constantly.

GPT-4 as a judge — surprisingly effective but costs money, adds latency, and feels wrong to use one LLM to evaluate another.

Manual review — fine for 50 samples. Completely falls apart at 1,500+.

I wanted something different. Fast, local, no API calls, no execution environment, no cost per eval. So I built LLM Judge.

How it works

The core insight is simple: semantic similarity between a model output and a reference answer is a surprisingly strong signal for correctness — especially for coding questions where there's usually one right approach.

Here's the pipeline:

Model output + Reference answer
↓
Sentence Transformers (all-MiniLM-L6-v2)
↓
Cosine similarity score
↓
Logistic regression classifier
↓
Correct / Partially correct / Incorrect + calibrated confidence score

I trained the logistic regression on a merged Hugging Face coding Q&A dataset, validated on 1,500+ labeled submissions.

Results
~75% accuracy on held-out coding Q&A split
~58% agreement with human judges on 1,500+ labeled submissions
Runs entirely locally — no API calls, no sandbox
Fast enough for batch evaluation at scale

The 58% human agreement number is the interesting one. Human judges disagree with each other more than you'd expect on coding questions — so 58% agreement with humans is actually competitive with inter-human agreement on ambiguous cases.

What I built

Three interfaces for different use cases:

CLI batch scoring — point it at a CSV of model outputs and reference answers, get scores back:

bash
python evaluate_csv.py --input outputs.csv --output scores.csv

Training your own judge:

bash
python train_judge.py --dataset your_dataset.csv

Streamlit UI — visual interface with calibrated scores and 3-tier feedback for each eval:

bash
streamlit run app.py
Stack
Python
Sentence Transformers
Scikit-Learn
Streamlit
Pandas
What's next

The open source version handles local evaluation. I'm building a hosted API version next — upload your dataset, get scores back via API, no setup required. If that's something you'd use, let me know in the comments.

Try it

GitHub: https://github.com/Zoh007/RAG_PRAC

Would love feedback from anyone doing LLM evaluation at scale — especially curious:

Is 75% accuracy good enough for production eval pipelines?
What edge cases do you run into that simple similarity scoring misses?
Would a hosted API be worth paying for or would you always self-host?