Why I Stopped Using LLM-as-a-Judge (And What I Use Instead)

Six months ago I was running evals on a coding Q&A dataset and my OpenAI bill hit $47 before I finished my test run. I wasn't doing anything exotic — just asking GPT-4 to score 500 model responses on a rubric. That's the hidden cost of LLM-as-a-Judge: every evaluation call is itself an API call, which means the more you evaluate, the more you pay.

I spent a weekend building an alternative. The idea was simple: train a local binary classifier using Sentence Transformers and logistic regression to separate "good" from "bad" responses. No API. No cloud. No credit card. The result was a tool that runs at ~8ms per sample and costs $0 to operate indefinitely. It hit 75% accuracy on coding Q&A out of the box. Not perfect — but good enough to catch regressions, and fast enough to run in CI on every commit.

Here's the full story of how it works, where it breaks, and when you should (and shouldn't) use it instead of GPT-4-based eval.

1. The Problem with LLM-as-a-Judge in 2026

• 1. Cost compounding: eval calls multiply with dataset size and iteration count
• 2. Latency kills CI: 2-5 second API latency per sample = minutes for a 1,000-sample suite
• 3. Bias and instability: 2026 research showing LLM judges are prompt-sensitive, exhibit taxonomic bias, and can flip scores with minor prompt rewording
• 4. Lock-in: if the model changes upstream (GPT-4 → GPT-5), your eval scores drift even if your app didn't change
• 5. The OpenAI-acquires-Promptfoo moment: what it means for "neutral" open-source eval

1. The Architecture of LLM Judge

• 1. Step 1: embed query + response pairs using sentence-transformers (all-MiniLM-L6-v2)
• 2. Step 2: train a logistic regression classifier on labeled good/bad examples
• 3. Step 3: at eval time, embed and classify — no network call, no API key
• 4. Why logistic regression (not a neural classifier): interpretability, speed, no GPU required, works with small label sets
• 5. Code snippet: full training loop in ~20 lines of Python

1. Benchmark: 75% Accuracy on Coding Q&A

• 1. Dataset description: coding question + model answer pairs, human-labeled
• 2. How accuracy compares to random (50%) and human-agreement ceiling (~85%)
• 3. Where the model is wrong: ambiguous cases, style vs. correctness confusion, edge cases in multi-step reasoning
• 4. Honest take: this is not a replacement for human review on ambiguous cases — it's a regression detector

1. When to Use It (And When Not To)

• Use it: CI smoke tests, catching obvious regressions, high-volume eval where API cost is prohibitive, air-gapped / local-first environments, early-stage projects without budget
• Don't use it: nuanced long-form eval, multi-modal, tasks requiring domain expertise to label, when you need explainability on individual failures (logistic regression gives a score, not a reason)
• Hybrid approach: use LLM Judge for volume, LLM-as-a-judge for spot-checks on borderline cases

1. How to Extend It to Your Domain

• 1. Labeling your own dataset: minimum viable label set (~100–200 examples), how to use weak supervision to bootstrap
• 2. Swapping the embedding model: when to use domain-specific embeddings (legal, medical, code-specific)
• 3. Adding multi-class scoring: extending from binary (good/bad) to rubric-based (0–3)
• 4. Integration patterns: pytest plugin, pre-commit hook, GitHub Actions step

1. What's Next

• 1. Pre-trained domain packs (coding, customer support, RAG faithfulness)
• 2. LangChain / LlamaIndex integration
• 3. Active learning loop: flag low-confidence samples for human review, retrain
• 4. Link to GitHub: github.com/Zoh007/llm-judge — PRs and issues welcome

What's your current eval setup? Are you paying for API-based eval, running something local, or just skipping evals entirely? I'm curious what's actually working for people in production.