How I Built a No-Execution LLM Eval Judge (75% Accuracy, No API Calls)

Zohair — Fri, 31 Jul 2026 15:01:50 +0000

Evaluating LLM outputs is one of those problems that sounds simple until you actually try to do it at scale.

Most approaches fall into three buckets:

Run the code — works for coding tasks but requires a sandbox, is slow, and breaks on edge cases constantly.

GPT-4 as a judge — surprisingly effective but costs money, adds latency, and feels wrong to use one LLM to evaluate another.

Manual review — fine for 50 samples. Completely falls apart at 1,500+.

I wanted something different. Fast, local, no API calls, no execution environment, no cost per eval. So I built LLM Judge.

How it works

The core insight is simple: semantic similarity between a model output and a reference answer is a surprisingly strong signal for correctness — especially for coding questions where there's usually one right approach.

Here's the pipeline:

Model output + Reference answer
↓
Sentence Transformers (all-MiniLM-L6-v2)
↓
Cosine similarity score
↓
Logistic regression classifier
↓
Correct / Partially correct / Incorrect + calibrated confidence score

I trained the logistic regression on a merged Hugging Face coding Q&A dataset, validated on 1,500+ labeled submissions.

Results
~75% accuracy on held-out coding Q&A split
~58% agreement with human judges on 1,500+ labeled submissions
Runs entirely locally — no API calls, no sandbox
Fast enough for batch evaluation at scale

The 58% human agreement number is the interesting one. Human judges disagree with each other more than you'd expect on coding questions — so 58% agreement with humans is actually competitive with inter-human agreement on ambiguous cases.

What I built

Three interfaces for different use cases:

CLI batch scoring — point it at a CSV of model outputs and reference answers, get scores back:

bash
python evaluate_csv.py --input outputs.csv --output scores.csv

Training your own judge:

bash
python train_judge.py --dataset your_dataset.csv

Streamlit UI — visual interface with calibrated scores and 3-tier feedback for each eval:

bash
streamlit run app.py
Stack
Python
Sentence Transformers
Scikit-Learn
Streamlit
Pandas
What's next

The open source version handles local evaluation. I'm building a hosted API version next — upload your dataset, get scores back via API, no setup required. If that's something you'd use, let me know in the comments.

Try it

GitHub: https://github.com/Zoh007/RAG_PRAC

Would love feedback from anyone doing LLM evaluation at scale — especially curious:

Is 75% accuracy good enough for production eval pipelines?
What edge cases do you run into that simple similarity scoring misses?
Would a hosted API be worth paying for or would you always self-host?

DEV Community: Zohair

How I Built a No-Execution LLM Eval Judge (75% Accuracy, No API Calls)