The Evolution of an LLM Judge: From "It Works on My Laptop" to Under a Rupee a Conversation

How I built an LLM to grade tens of thousands of AI support conversations a day, then spent two months making it cheap enough to actually run.

The words you'll need

If you know these, skip ahead. One plain line each.

• LLM-as-a-judge — an LLM that scores another LLM's answers against rules, instead of a human.
• Conversation vs turn — one conversation can hold eight or ten back-and-forth turns. My judge reads the whole thing.
• Compliance file — a rules file that says, for each kind of question, roughly what a good answer looks like.
• Token — the unit a model reads, and the unit you pay for. Context, rules, the chat, all tokens.
• Prompt caching — reusing the work a model already did on text it has seen, so you don't pay for it twice.
• Batch API — many independent requests sent as one job. Cheaper, and each is still answered on its own.

The whole story in 7 lines

1. We ran an AI help desk with tens of thousands of chats a day.
2. No test can tell you if an answer was correct, helpful, and kind. Only judgment can.
3. So I built an LLM to judge whole conversations against a rules file.
4. It worked. Then the bill came: about 3 rupees per conversation.
5. Everything is tokens. A paper on cost, accuracy and performance showed me which to cut. Down to ~1.5.
6. I wanted caching, but one independent call per chat made normal caching unsafe.
7. The Gemini Batch API gave me the discount without the risk. Under a rupee a conversation.

1. The breaking point

We were running an AI help desk. Real users, real questions, all day.

Behind the chat sat a three-agent pipeline. The user never saw it. They just saw answers.

On a busy day, tens of thousands of conversations.

And I could not tell you how many of those answers were any good.

That sentence should bother you. It bothered me.

Here is what people get wrong about testing an AI product. You can test the plumbing. Did the API return? Did the call time out? Was there a network error? That is a normal test. Green or red.

Quality is not like that.

• Was the answer actually informative?
• Did it show any empathy, or just cold facts?
• Could the user finish the thing they came to do?
• Was the data we showed them correct?
• Did the answer fit the whole conversation, not just the last line?

None of those have a green checkmark. You cannot assert them in a unit test.

The obvious fallback is to read the conversations. Fine. Read tens of thousands. Then do it again tomorrow. Then sort them into a hundred buckets by hand.

It is like being handed every letter a post office moves in a day, and asked by tonight which ones made the reader happy. One person cannot. A hundred people cannot. Not every day, not the same way twice.

And there was a trap inside the question. An answer can be completely correct and still fail. If the bot only states facts, with no warmth, the user leaves unhappy. Low satisfaction. Lower rating. The fact was right and the conversation was still a loss.

So the test suite was out. Manual review was out.

Then one plain thought: if a human can judge an answer but a test cannot, maybe another model can.

That was the decision. Build an LLM to judge the conversations our AI was having. Point it at all of them, every day, and score what no test and no team ever could.

That decision started two months of getting it wrong before I got it right.

2. Deciding what the judge actually judges

Most people open a blank file and start writing a prompt. That is the mistake. The first question is not how to judge. It is what.

I wrote down three decisions before any code.

One: judge the whole conversation, not a single reply. A user does not ask one question and leave. One conversation can run eight or ten turns. The third answer only makes sense if you saw the first. So the judge reads the entire conversation and scores it as one thing.

Two: give the judge enough context to have an opinion. A judge that does not know your product cannot tell a right answer from a confident wrong one. So I fed it the context: what the product is, what the APIs do, what fields exist, what lives in the knowledge base. The more it knew, the fairer it got.

Three, the interesting one: a compliance file. Even with all that context, the judge needs a reference for "what does good look like here?" I did not want one hard-coded correct answer. Real support answers vary. So I wrote a compliance file, a YAML, that says: for this kind of question, a good answer lives in roughly this zone. Not an exact string. A shape to land inside.

# compliance.yaml (illustrative)
- intent: reset_password
  must_cover: [verify identity, send reset link, say where to look]
  tone: warm, step-by-step
  never: [ask for the full password, promise instant delivery]
  good_answer_shape: >
    Acknowledge the problem, verify who they are,
    send the link, tell them where it lands, offer a next step.

Think of a teacher's answer key. A good key does not demand the student's words match letter for letter. It says what a strong answer must contain, and how far it can wander. That is what the compliance file is for the judge.

Then one dimension that is easy to forget: tone. If your AI gives only cold, factual answers, the user does not feel helped. Satisfaction drops, ratings drop, even when every fact is right. So the judge scores tone and satisfaction next to correctness, not as an afterthought.

Every judgment takes the same ingredients: system prompt and context, the compliance file, and the full conversation. One call. One verdict, with scores and a reason.

I built it. I ran it locally. The judgments were good.

And then I did the math.

3. It worked. Then the bill arrived.

Working locally on a few conversations felt great. The judge read a chat, scored it, explained itself. I thought the hard part was behind me.

It was not. The hard part was the bill.

Each judgment carried a lot of tokens: the context, the compliance file, the API descriptions, the system prompt, and a full multi-turn conversation. All of it goes in on every single call.

The cost landed at about 3 rupees per conversation.

One conversation. Now multiply by tens of thousands, every day. It stopped being a rounding error and became a problem I had to solve before any of this could ship.

So the project quietly changed. Phase one was "can a model judge this?" Yes. Phase three became "can I afford to let it?"

4. Everything is a token

Here is the idea that reframed the whole cost problem for me.

Everything is a token. The context is tokens. The compliance file is tokens. The API docs, the system prompt, the conversation, all of it. And you pay per token.

So cost cutting is not a dark art. It is one question, asked over and over: which token can I remove without changing the verdict?

It is like an old telegram, priced by the word. You keep every word that carries meaning and cut every word that does not. The message still arrives. It just costs less.

One part you cannot touch: the conversation itself. That is the thing being judged. You do not get to shrink the evidence. The savings come from the context wrapped around it.

Easy to state, hard to do well. Cut too much and the judge gets dumber. Cut the wrong thing and your scores drift. I needed more than instinct. I needed a method.

5. The paper on cost, accuracy, and performance

I went looking for research instead of guessing. I found a paper framed around exactly my three tensions: cost, accuracy, and performance. It read like it was written for my problem.

Paper Link: https://arxiv.org/abs/2405.11299?utm_source=chatgpt.com

The framing alone was worth the read. You rarely improve all three at once. Push cost down and accuracy can slip. Push accuracy up and cost climbs. The question it taught me to ask about every token and every call was simple: is this paying its rent?

Here is what I actually changed, with that lens on.

1. I cut the context down to what the judge used. I had been pasting in the whole product context and every API description on every call, as if the judge needed all of it to grade one chat. It did not. I kept the fields and endpoints a judgment leans on, and dropped the rest. The biggest single saving, because it was the biggest block of tokens.
2. I compressed the compliance file. The first version was verbose and explained itself to a human reader. The judge does not need prose. I rewrote the rules tight: intents, must-cover points, tone, hard "never" lines. Same guidance, far fewer tokens.
3. I constrained the output. Early on the judge wrote long, chatty explanations. I only need a score per dimension and a short reason. Less output is fewer output tokens, which you also pay for, and the verdicts got easier to parse.

Cost fell from about 3 rupees to about 1.5 rupees per conversation. Half.

Half is a lot. It was also not enough. At my volume, that is still expensive to run every day. I wanted it under a rupee.

6. The caching puzzle, and the trick that solved it

The next idea was obvious: cache something.

Caching is simple in spirit. You keep the result of work that repeats, so you do not redo it. The catch is the word repeats. My judge did not repeat anything in the way caching wants.

Cache the results? Dead on arrival. Every conversation is different. Nothing to look up.

Cache the prompt, then. The context and the compliance file are identical on every call. In theory I could cache that big static block and only vary the conversation. But here is the problem, and it is subtle.

I call the judge once per conversation, on purpose. I never want it to carry one chat's context into the next. A fresh juror for every case. If a cached prefix let the last judgment bleed into this one, the judge could start forming opinions across chats it was never meant to connect. That is not a cost bug. That is a correctness bug, the worst kind, because the numbers still look fine while the verdicts quietly go wrong.

So normal prompt caching was off the table. One independent call per chat was a line I would not cross for a discount.

I kept digging. The answer was not a cache at all. It was the Gemini Batch API.

Here is why it fit. You hand the batch API many independent requests as one job. The shared instructions get the pricing benefit of being sent together, but each request is still answered on its own, cold, with no memory of the others. I got the discount and kept the one-call-per-chat isolation I refused to give up.

Think of a judge who gets the same rulebook printed once for the whole stack of cases, then reads every case from a clean desk. Same rules, no bleed.

Cost dropped to about 90 paise per conversation. Under a rupee, from 3 rupees at the start. About a 70% cut, and the judgments held.

7. Where it stands, and what I don't fully trust yet

I run this in production now. It reads the conversations our AI has, every day, and scores what no test suite could. But let me be honest about the edges, because a judge you trust blindly is a judge that will burn you.

• The numbers are mine. One product, one domain, my own traffic. Yours will differ.
• A judge is only as fair as its rubric. The compliance file encodes my idea of good. If that is wrong, the judge is confidently wrong.
• Approximate matching needs spot checks. The compliance file describes a zone, not a single answer. Edge cases slip. So I still read a sample by hand and compare.
• One provider, one tradeoff. The batch API trades latency for cost. Verdicts are not instant. For daily monitoring, fine. For a real-time gate, no.

8. What I'm taking away

If you are about to build one of these, here is what I wish I had known on day one.

1. Decide what you judge before you write a prompt. For me, the whole conversation, never a single reply.
2. Turn "good" into a file. A compliance rubric beats a vibe. It makes the judge arguable, and that is a feature.
3. Judge tone, not just facts. A correct, cold answer still loses the user.
4. Treat every token as money. Cut the ones that do not change the verdict. You cannot cut the evidence, so cut the context.
5. Read the paper. The cost-accuracy-performance framing saved me more than another week of guessing would have.
6. Caching is not one thing. If you cannot cache the normal way, batching might hand you the same discount without the risk.

The whole thing runs on judgment. Mine, encoded into a file. A model's, applied at scale. The judge now reads every conversation, every day. I still read a handful myself. Someone has to judge the judge.