Automating LLM A/B Testing: How to Build a Cross-Model Evaluation Harness Programmatically

If you are building LLM-powered products, you already know that benchmark leaderboards are not enough. What matters is how a model behaves on your prompts, your tone, your domain data, and your budget.

This guide is for engineers and product teams who want to build a repeatable evaluation harness that can compare several open-weight models on the same dataset, score them automatically, and choose a winner using accuracy, latency, and token cost. By the end, you will have a working mental model for building a cross-model evaluation system that runs asynchronously, stores its results locally, and uses Featherless.ai as the model execution layer.

The problem with ad hoc model testing

Most teams evaluate models in the least reliable way possible:

• copy a prompt into a chat UI
• compare a couple of responses by eye
• maybe keep a spreadsheet
• repeat the process later with a different model

That approach does not scale. It breaks down when you need to compare many rows, many models, or multiple scoring criteria at once. It also makes it hard to answer the real question:

Which model performs best on this specific dataset, under this specific cost and latency profile?

That is the problem this harness solves.

What Featherless contributes to the system

Featherless is the model execution layer for this project. Instead of provisioning GPUs, managing serving stacks, or wiring together one-off model hosts, the harness sends requests directly to Featherless’s OpenAI-compatible API and lets the platform handle model availability, concurrency limits, and serverless inference.

That matters because the evaluation harness can focus on orchestration, scoring, and reporting rather than infrastructure. The app uses four Featherless surfaces:

• GET /v1/models for live model discovery and filtering
• OpenAI-compatible chat completions for candidate generation and judging
• GET /account/concurrency for concurrency snapshots before a run
• GET /account/concurrency/stream for live concurrency telemetry during a run

What we are building

The project is a full-stack LLM evaluation dashboard built with FastAPI on the backend and Next.js on the frontend.

The workflow is straightforward:

1. upload a CSV dataset with prompts and expected answers
2. select 2 to 10 Featherless models
3. fan out requests concurrently across the selected models
4. score each candidate against ground truth with an LLM judge
5. store logs and results in SQLite
6. compare models in a dashboard ranked by weighted score

How the async fan-out works

The core performance trick is concurrent model execution. A naive loop would query one model at a time, which is slow and wastes the fact that the evaluation of each row is independent.

Instead, the harness evaluates a row by dispatching one task per selected model and waiting for them together with asyncio.gather.

tasks = [
    evaluate_model(run, row, model, semaphore)
    for model in selected_models
]
results = await asyncio.gather(*tasks)

To keep the system safe against plan limits, the app wraps model calls in a semaphore. That lets users choose how many requests can run in parallel without exceeding their Featherless concurrency budget.

parallel_model_calls = max(
    1,
    min(
        len(models) or 1,
        settings.max_parallel_models,
        int(run.get("parallel_model_calls") or 1),
    ),
)

semaphore = asyncio.Semaphore(parallel_model_calls)

This is where Featherless’s cost-based concurrency model matters. A large model can consume more concurrency units than a smaller one, so “parallel” does not simply mean “unlimited.”

How candidate outputs are collected and scored

Each candidate model receives the same prompt and system instruction. The goal is consistency, not creativity.

response = await client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "Answer the user's prompt directly. Be concise and factual."},
        {"role": "user", "content": prompt},
    ],
    temperature=0.0,
    max_tokens=768,
)

For each candidate response, the harness records:

• output text
• latency
• prompt tokens
• completion tokens
• total tokens
• any provider error

Then the judge model compares the candidate answer to the ground truth and returns a score from 0.0 to 1.0.

messages=[
    {
        "role": "system",
        "content": (
            "You are a strict LLM evaluation judge. Compare a candidate answer against "
            "the expected ground truth for semantic correctness. Return JSON only with "
            'shape {"score": 0.0, "rationale": "short reason"}.'
        ),
    },
    {
        "role": "user",
        "content": (
            f"Prompt:\n{prompt}\n\n"
            f"Expected Ground Truth:\n{expected}\n\n"
            f"Candidate Output:\n{candidate_output}\n\n"
            "Return only the JSON score object."
        ),
    },
]

The parser is deliberately tolerant because real models sometimes return fenced JSON, not bare JSON.

{
  "score": 1.0,
  "rationale": "The answer matches the ground truth."
}

That is important in production. If your parser assumes perfect formatting, your judge becomes a brittle point of failure.

How the dataset is structured

The harness expects a CSV with three columns:

• ID
• Prompt Input
• Expected Ground Truth

Example:

ID,Prompt Input,Expected Ground Truth
ticket-001,"Summarize this customer complaint: The user cannot reset their password after three attempts.","The user cannot reset their password because repeated attempts triggered a lockout."
ticket-002,"What is 18% of 250?","45"
ticket-003,"Rewrite this answer to sound more professional: 'I can't help with that right now.'","I’m unable to assist with that request at the moment."

This format is intentionally simple. It works for math, summarization, rewriting, extraction, and support workflows.

A real example run

Here is the kind of comparison this harness is meant to produce.

Scenario: a support team wants to compare two models on ticket summarization quality.

Dataset: 25 rows of internal support tickets
Candidates:

• deepseek-ai/DeepSeek-V4-Flash
• google/gemma-4-E4B-it

Judge:

• a separate judge model optimized for rubric-based evaluation

Observed result:

Model	Semantic Accuracy	Avg Latency	Total Tokens	Rows	Errors	Weighted Score
deepseek-ai/DeepSeek-V4-Flash	0.560	3.60s	12,749	25	10	0.442
google/gemma-4-E4B-it	0.393	4.63s	17,987	25	9	0.151

What this tells you:

• deepseek-ai/DeepSeek-V4-Flash was the better fit overall for this dataset
• google/gemma-4-E4B-it used more tokens and still scored lower
• the winner was selected by a custom weighting of accuracy, latency, and token use

That is the point of the harness. A leaderboard can tell you what is popular. This tells you what works for your task.

Why the judge model is separate

A strong candidate model is not automatically a strong judge. Those are different jobs.

A good candidate model should:

• generate useful task outputs
• follow the prompt closely
• be fast enough for your workflow
• fit your cost constraints

A good judge model should:

• apply a rubric consistently
• compare output to reference data
• be stable at low temperature
• minimize variance across evaluations

That is why the harness uses a dedicated judge model rather than reusing one of the candidates. If you let the same model generate and judge, you mix two different roles and make the evaluation less trustworthy.

How the backend handles failures

The backend is designed to keep a run alive even when individual requests fail.

It catches:

• connection timeouts
• rate limiting
• HTTP status errors
• schema validation failures

That matters because distributed model inference fails in different ways:

• a model can be temporarily at capacity
• a model can return invalid structured output
• a model can be unavailable on the current plan
• the provider can change response shape

The right behavior is not to crash the entire run. It is to record the failure, continue the other model calls, and expose the issue in the final report.

How concurrency visibility helps

The /account/concurrency/stream endpoint is useful because model limits are not static. A model may be available in the catalogue but temporarily unavailable under current load.

The dashboard uses the stream to show:

• current concurrency limit
• used concurrency cost
• number of active requests
• the model currently consuming capacity
• how long each in-flight request has been running

That gives the user a live view of whether the run is healthy or headed toward 429s.

Why the storage layer is local

The harness stores datasets, run metadata, and events in SQLite. That keeps the system reproducible and easy to inspect without adding a separate database dependency.

This is enough for:

• dataset uploads
• saved run history
• per-row event logs
• summary calculations
• debugging failed runs

For a team workflow, that is usually the right tradeoff until you need shared multi-user access.

What to extend next

The most useful next features are:

• row sampling controls, so you can test on 25 or 50 rows before spending tokens on the full dataset
• run export to CSV or JSON, so results can move into spreadsheets or BI tools
• historical comparisons, so teams can see whether a model is improving or regressing over time

These are practical extensions because they improve decision quality without complicating the core harness.

The useful way to think about model selection is not “which model is best?” It is:

Which model performs best on this dataset, under this latency budget, with this concurrency budget, using this scoring policy?

Featherless handles the model serving side. FastAPI handles orchestration. SQLite handles auditability. The async harness handles throughput. The judge model handles scoring. Together, they turn model evaluation from an ad hoc exercise into a repeatable engineering workflow.

Checkout Featherless.ai: https://featherless.ai/?ref=harishkotra