DEV Community: Anu Kumar

LLM evaluation: evals that catch regressions, and when to trust LLM-as-a-judge

Anu Kumar — Fri, 16 Oct 2026 06:59:40 +0000

Without evals, every prompt change is a guess. You fix the example in front of you and quietly break three you aren't looking at. An eval set turns that guess into a number you can watch.

Start from real failures

The best eval cases come from real inputs, especially the ones that went wrong. Collect 50 to 200 of them, write down what a correct outcome looks like for each, and tag them by type: language, task, customer segment, whatever splits your traffic. An average score hides a feature that's broken for one slice.

Synthetic cases are useful for coverage, but they tend to be cleaner than real users. Keep real ones as the core.

Check with code first

A lot of quality is checkable without another model. Is the output valid JSON with the required fields? Did the agent call the right tool with the right arguments? Does every citation point to a document that was actually retrieved? Is the answer under the length limit? These checks are fast, free and give the same result every run. Write them before reaching for a judge.

LLM-as-a-judge, carefully

For open-ended qualities like helpfulness or tone, a model grading another model's output is often the only practical option. It has known biases, documented in the MT-Bench paper by Zheng et al. (2023):

Position bias: when comparing two answers, judges tend to favour one position regardless of content.
Verbosity bias: longer answers look better to judges even when they aren't.
Self-enhancement bias: a judge can favour answers written by its own model.

What helps: a rubric with specific, checkable criteria instead of "rate quality from 1 to 10". Pass/fail or a three-point scale rather than fine-grained scores. Asking for the reasoning before the verdict. In pairwise comparisons, running both orders and only counting consistent wins.

Grade the answer against the reference.
Criteria, each PASS or FAIL:
1. States the refund window that the reference states.
2. Does not promise anything the reference does not support.
3. Tells the user the next step to take.
Give one line of reasoning per criterion, then the verdicts.

Then calibrate. Grade 50 to 100 cases by hand, run the judge on the same cases and measure how often it agrees with you. If agreement is poor, fix the rubric before you trust a single number it produces.

For agents, grade the trajectory too

An agent can reach the right answer the wrong way: ten tool calls where two would do, a write it shouldn't have made, a lucky guess after a failed lookup. Score the path as well as the result. Did it call the expected tools? How many steps did it take? Did it stop when it should? What did the run cost and how long did it take?

Run evals like tests

Run the suite on every prompt change, model change and tool change, and compare against the last known good run.
Model output varies between runs. Run each case several times and track the pass rate, not a single pass or fail.
Watch the slices, not only the total.
Every production failure becomes a new case. The set should grow every week.

An eval set doesn't need to be big or clever on day one. Twenty real cases with honest checks will catch more regressions than a polished benchmark nobody runs.

Originally published at anukumar.site.

Multi-agent systems: when splitting an agent helps and when it just costs more

Anu Kumar — Fri, 16 Oct 2026 06:52:55 +0000

Splitting one agent into several is one of the most popular moves in agent design, and often the most expensive one. It helps in a specific situation. Outside it, you pay more for a system that's harder to debug.

What multiple agents actually buy

The real benefit is separate context windows. A subagent researching one question works in a clean context that holds only that question, its tools and its findings. It hands back a short summary, and the lead agent never sees the hundred search results behind it. For broad tasks that would otherwise overflow one context, that's a genuine gain.

The second benefit is parallelism. If a task breaks into independent pieces, like checking five sources or reviewing five files, subagents can run at the same time.

What it costs

Tokens. Every agent reads its own instructions, tools and context. Anthropic reported that its multi-agent research system used roughly 15 times the tokens of a plain chat interaction. That can be worth it for high-value work. It isn't for routine tasks.
Compounding errors. If each handoff is right 95% of the time, a chain of five is right about 77% of the time (0.95 to the power of 5).
Lost context. The lead agent knows why a task matters. A subagent knows only what the handoff told it.
Debugging. A wrong answer now has several transcripts behind it, and the mistake might be in any of them or in a handoff between them.

Patterns that hold up

Orchestrator and workers: A lead agent plans, sends independent subtasks to workers that run in parallel, then combines their results. Good for research and wide searches.
Pipeline: Fixed stages in a fixed order, like extract, then check, then write. Often this doesn't need agents at all, just a sequence of model calls.
Router: One step decides which specialist handles a request, then gets out of the way. Good when requests fall into clearly different types.

The pattern to avoid is several agents talking freely to each other until they agree. It's hard to predict, hard to bound and hard to test.

Write handoffs like task specs

"Research competitor pricing" sent to three workers produces three overlapping reports. A handoff needs the objective, the boundaries, the expected output and when to stop:

{
  "objective": "Find current list prices for the 3 cheapest plans of vendor X",
  "scope": "Vendor's own pricing page and docs only. Not reviews or forums.",
  "output": "JSON: [{plan, monthly_price, currency, source_url}]",
  "stop_when": "All 3 plans found, or 5 tool calls used"
}

Start with one agent

Build one agent with good tools first and measure it. Split it only when you can point to the problem splitting solves: the context keeps overflowing, or the work has clearly independent parts that would finish faster in parallel. Then run the same evals on both versions. If the multi-agent version isn't measurably better, keep the simpler one.

Originally published at anukumar.site.

RAG retrieval: hybrid search, reranking and measuring recall

Anu Kumar — Fri, 16 Oct 2026 06:48:26 +0000

When a RAG system gives a wrong answer, the model usually isn't the problem. It answered from the chunks it was given, and the right chunk wasn't among them. Fix retrieval and most of the bad answers go away.

Measure retrieval on its own

Before changing prompts or models, write down 50 to 100 real questions and, for each one, the chunk or chunks that actually answer it. Then measure retrieval without the model in the way.

recall@k: The share of questions where at least one correct chunk appears in the top k results. If recall@5 is 60%, the model is missing the answer on four questions out of ten before it writes a word.
MRR: Mean reciprocal rank: 1 divided by the position of the first correct chunk, averaged over questions. It tells you whether the right chunk is near the top or scraping in at rank 9.

Every change below should move one of these numbers. If it doesn't, it isn't helping.

Chunk by structure, not by character count

Fixed 500-character splits cut tables in half and separate step 4 from steps 1 to 3. Split on the document's own structure instead: headings, paragraphs, list items, table rows kept together. Prefix each chunk with its heading path, like "Billing > Refunds > Partial refunds", so a chunk that says "this takes 5 days" still knows what "this" is.

Size is a trade-off. Tiny chunks lose context. Huge chunks blur several topics into one embedding, and the one sentence that matters gets diluted.

Hybrid search: keywords and vectors together

Embeddings are good at meaning and bad at exact strings. Ask about error E1042, SKU 88-210 or a person's surname and vector search often returns things that are about the same topic but not that exact thing. Keyword search such as BM25 handles exactly those cases.

Run both and merge the result lists. Reciprocal rank fusion is the simple, robust way to do it, because it uses only the rank in each list, not the scores, which aren't comparable between the two systems:

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

k=60 is the value from the original RRF paper and a reasonable default.

Rerank the candidates

Embedding search compares a query vector with chunk vectors that were computed separately. A cross-encoder reranker reads the query and each chunk together, so it judges relevance much better, but it's too slow to run over the whole corpus. Retrieve 50 candidates cheaply, rerank them, keep the top 5.

Filter before you rank

If the question is about one product, one year or one customer's documents, apply that as a metadata filter, not as something you hope similarity will sort out. Permissions especially: filter out documents the user isn't allowed to see before retrieval. A prompt that says "don't mention confidential documents" is not access control.

Plan for finding nothing

Some questions have no answer in your data. If the best reranked score is below a threshold you've checked against your test set, tell the model nothing relevant was found and let it say so. "I couldn't find this in the documentation" is a good answer. A confident answer stitched together from loosely related chunks is the worst one.

Originally published at anukumar.site.

Agentic AI in practice: tool design matters more than the prompt

Anu Kumar — Fri, 16 Oct 2026 06:45:15 +0000

When an agent misbehaves, the usual reaction is to rewrite the system prompt. Most of the time the problem is the tools: what they're called, what they accept, and what they say back when something goes wrong.

The model only sees names and descriptions

An agent picks a tool from its name, its description and its argument schema. Nothing else. Give it search_customers and find_customer with similar descriptions and every lookup becomes a coin flip.

Write descriptions the way you'd brief a new teammate: what the tool does, when to use it, when not to, and what a good argument looks like. "Searches customers" is a label. "Find customers by name, email or phone. Use this before any action that needs a customer_id. Partial names work." is an instruction.

Fewer, higher-level tools

If answering "what did this customer order last week" takes list_orders, then get_order for each one, then list_order_items, the agent makes a dozen calls. Each call is a round trip, more tokens in context, and one more chance to go wrong. One get_recent_orders(customer_id, days) tool does the same job in one step.

Don't take this too far. A tool that both looks things up and changes them is hard to describe and hard to approve. Group by task, keep reads and writes apart.

Return what the model needs, not the raw API

Drop fields the agent will never use. A forty-field JSON record costs tokens on every call and buries the three fields that matter.
Prefer names over opaque IDs where you can, and include the ID too when a later tool needs it.
Cap list sizes and say so: "Showing 20 of 312. Narrow the search or call again with page=2."

Errors are instructions

An agent reads a tool's error and decides what to do next. A bare error gives it nothing to work with, so it retries the same call or invents an answer. A good error says what went wrong and what to try instead.

# unhelpful
{"error": "not found"}

# helpful
{"error": "No customer with email ravi@exmple.com. The domain may be misspelled. Try search_customers with the name or phone number."}

Make writes safe to retry

Agents retry. Networks time out after a request has already landed. Any tool that writes should take an idempotency key, so the same call twice does the work once. Validate arguments inside the tool instead of trusting the model to send sensible values, and for risky actions offer a dry_run flag that shows what would change without changing it.

Put the limits outside the model

A model asked to stop after ten steps will sometimes take thirty. Enforce a maximum number of steps, a timeout per tool and a token or cost budget in the code that runs the loop. When a limit is hit, stop and return what was done so far with a plain explanation. A partial answer with a reason beats an agent that loops until something else kills it.

Before you touch the prompt again, read a few transcripts of the agent failing. Usually you'll see it misread a tool, drown in a tool's output, or retry an error it couldn't understand. Those are tool fixes.

Originally published at anukumar.site.

Human-in-the-loop AI agents: where approval gates belong

Anu Kumar — Fri, 16 Oct 2026 06:35:49 +0000

Most agent demos add a human by putting a confirm button at the end. By then the agent has already sent the email or written the row. A useful approval gate sits inside the loop, right before the one step that changes the outside world.

Gate side effects, not answers

An agent does two kinds of work. It reads: searches, retrieves documents, calls APIs that only return data. And it acts: sends messages, updates records, moves money, deletes things. Reading is cheap to get wrong because you can throw the result away. Acting isn't.

So the gate belongs on tools, not on the conversation. Let the agent plan and read freely. When it decides to call a tool that writes, stop, save the exact call it wants to make, and wait.

def call_tool(run, tool, args):
    if tool.writes and needs_approval(tool, args):
        request = save_pending(run.id, tool.name, args)
        run.pause(waiting_on=request.id)
        return  # resumes later from the saved request
    return tool.execute(args)

Not every write needs a person. Creating a draft is different from sending it. A useful split is reversible versus irreversible, and internal versus visible to a customer. An irreversible, customer-facing action always waits. A reversible, internal one usually doesn't.

What the reviewer has to see

"The agent wants to update a record. Approve?" gets approved every time, which makes it worthless. The reviewer needs enough to say no.

The exact action and its arguments, shown as a diff against current state where possible. "Change the due date from 12 Nov to 3 Nov" beats a JSON blob.
Why the agent wants to do it, in a sentence, with the sources it used. If a retrieved document drove the decision, link it.
What happens if nobody answers. An approval request with no expiry turns into a stale action that fires weeks later.

Let the reviewer edit the arguments, not only approve or reject. Most corrections are small: a wrong date, one wrong recipient. Rejecting the whole action and rerunning the agent to fix one field wastes everyone's time.

Don't trust the model's confidence on its own

It's tempting to skip approval whenever the model says it's sure. A model asked to rate its own confidence will often give a high number for a wrong answer. Use signals you can check from outside the model instead.

Did retrieval find anything relevant, or is the answer coming from the model's memory?
Do the arguments pass validation: does the record exist, is the amount inside normal limits?
Has this exact kind of action been approved unchanged many times before?
Would running the step twice cause harm?

Combine those into a score if you like. The point is that every part of it can be verified without asking the model.

Resuming after a yes

Approval can arrive minutes or days later. By then the process that asked may be gone, the record may have changed, and the reviewer may click the button twice.

Persist the pending call, not the agent's memory. Resume by executing the saved call, not by asking the model to redo its reasoning and hoping it lands on the same answer.
Check the world again before executing. If the record changed since the request was made, ask again.
Give every approved action an idempotency key, so a double click or a retry executes it once.
Write an audit row with who approved what, when, and what actually ran. When something goes wrong later, that row is the first thing anyone asks for.

Watch for approval fatigue

A gate that fires fifty times a day gets clicked through without reading. That's worse than no gate, because it looks like oversight. Track how often each kind of request is approved unchanged. When that rate stays very high for a low-risk action, automate it and spend the human attention on the requests that actually get edited or rejected.

The goal isn't a person in every loop. It's a person on the few steps where a wrong action is expensive, with enough in front of them to catch it.

Originally published at anukumar.site.

Multi-tenant architecture: shared schema vs database per tenant

Anu Kumar — Fri, 16 Oct 2026 06:35:24 +0000

Every SaaS product has to decide where one customer's data stops and the next one's starts. There are three common answers. On a good day they look much the same. They differ in what happens on a bad one.

The three models

Shared schema: One database, one set of tables, and a tenant_id column on every row. Cheapest to run and simplest to migrate. Isolation depends on every query filtering by tenant_id.
Schema per tenant: One database server with a separate schema for each tenant. Tables are separate, connections are usually shared, and every migration runs once per schema.
Database per tenant: A separate database for each tenant, sometimes with a separate application process as well. The strongest isolation, and the most to operate.

Most teams start with a shared schema, and most should. The question is when it stops being enough.

Where a shared schema breaks

The classic leak isn't in the main screens. Those get tested. It's in the CSV export someone added on a Friday, the background job that loops over invoices, or the admin report that joins three tables and filters only one of them. One missing tenant_id filter and a customer sees someone else's data.

Postgres can enforce the filter for you with row-level security, so a forgotten WHERE clause returns nothing instead of everything:

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON invoices
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

-- per request, inside the transaction:
SET LOCAL app.tenant_id = '6f1c...';

Two details catch people out. Without FORCE, the table's owner skips the policy, and apps often connect as the owner. Superusers skip it regardless. And SET LOCAL only lasts for the transaction, which is what you want with a connection pool: a plain SET would leak one request's tenant into the next request that borrows the connection.

The other shared-schema problem is the noisy neighbour. One tenant's slow report or bulk import competes with everyone else for the same tables, indexes and connections.

What database per tenant buys you

Isolation by connection. A pool opened against one database can't read another, so cross-tenant access stops being something you can write by accident.
Clean exits. Deleting a customer is DROP DATABASE, not a delete script across forty tables. That matters when a customer, or a data protection law, asks for erasure.
Per-tenant backup and restore. You can roll one customer back to yesterday without touching anyone else.
Bounded noisy neighbours. Give each tenant a small pool and a heavy tenant mostly slows itself down.
Room for different schemas, if different kinds of customer genuinely need different tables. Use this sparingly, because every variation is a code path to test.

There are two ways to run it. One application can keep a pool per tenant and pick the right one per request:

const pools = new Map();

function poolFor(tenant) {
  if (!pools.has(tenant.id)) {
    pools.set(tenant.id, mysql.createPool({
      database: tenant.dbName,
      connectionLimit: 5,
    }));
  }
  return pools.get(tenant.id);
}

Or each tenant gets its own process, started with its database name and holding exactly one pool. A process per tenant also isolates memory leaks and crashes, at the price of idle memory for every customer who isn't using the product right now.

What it costs

Migrations run N times. Skip one tenant and its schema drifts. Drift fails quietly: the one feature that needs the missing column returns a 500, and nobody connects the error to a migration that never ran.
Connections multiply. Fifty tenants with a pool of ten is five hundred possible connections. MySQL and MariaDB default max_connections to 151.
Cross-tenant questions get hard. "How many invoices did all customers send last month" becomes one query per database, or a separate analytics store fed from all of them.
Login needs a directory. With no global users table, something has to map an email address or a subdomain to the right tenant before authentication can even start.
Provisioning has many steps and has to survive failing halfway. Record each step as it completes, and save the database name the moment it exists, so a retry resumes instead of leaving an orphaned database behind.

Drift is the one to plan for on day one. Keep a migrations table in every tenant database, compare each tenant against the repository before a deploy, and refuse to deploy code that needs a column some tenants don't have yet. Reads that tolerate a missing optional column turn a broken module into a missing feature.

How to choose

How many tenants? Tens or hundreds of business customers can each have a database. Tens of thousands of free accounts can't, at least not cheaply.
What does one leaked row cost? For health, legal or financial records, the answer may be the company.
Do customers ask for their own backups, their own region, or proof of deletion? Those are much easier with a database each.
Do tenants really need different schemas, or just different settings?

My default is a shared schema with row-level security, plus a test that runs every query path as two tenants and fails if either sees the other. Move to a database per tenant when the answers above point there, not before. Schema per tenant sits in between, and it inherits the migration cost of the strict option without all of its isolation, so I rarely pick it.

Originally published at anukumar.site.