In March 2026, OpenAI acquired Promptfoo — the tool that 350,000 developers use to evaluate and red-team OpenAI's own models.
I don't think that makes Promptfoo suddenly useless. But it made me ask a question I should have asked earlier: who is the eval tool accountable to?
Most LLM eval frameworks — DeepEval, Ragas, LangSmith — have something in common. They all route scoring through an LLM API call. That means your eval pipeline has at least three dependencies you don't control: the tool vendor, the judge model vendor, and whatever pricing changes either of them makes next quarter.
When I started building LLM Judge, I wasn't thinking about vendor independence. I was thinking about cost. I was running evaluations on a coding Q&A dataset, and every DeepEval run was making LLM API calls I hadn't budgeted for. It added up fast.
So I tried something simpler: train a classifier. Sentence Transformers to embed the responses, logistic regression to classify quality. The result: 75% accuracy on coding Q&A, ~8ms per sample, $0 per run, runs fully offline.
It's not magic. It won't replace human review for high-stakes decisions. But for fast feedback loops in CI/CD — the place where LLM eval matters most — it's surprisingly good.
What this approach gives you:
No API key required. Ship to air-gapped environments, run in restricted CI, deploy anywhere Python runs.
No vendor lock-in. You own the model. You trained it. No one can deprecate it out from under you.
Speed that fits CI. 8ms/sample means you can eval 1,000 outputs in under 10 seconds.
Customizable to your domain. Fine-tune on your data and your definitions of "good."
The tradeoff: you need labeled training data. If you have zero examples, start with a small hand-labeled set. 50–100 examples is enough to get a useful baseline classifier.
How to get started:
pip install llm-judge
Then train on your data and run evals locally. No signup. No API key. No monthly bill.
The Promptfoo situation is a useful reminder: the best eval tool is one you understand, own, and can run independently of whoever built the model you're testing.
→ github.com/Zoh007/llm-judge
What's your current eval setup? Especially curious if anyone's moved away from API-dependent evals — drop a comment.
Top comments (0)