DEV Community: Quantiles.io

Build an AI Evaluation from a Hugging Face Dataset Without Writing Python

Quantiles.io — Tue, 04 Aug 2026 16:48:38 +0000

AI benchmarks and evaluations have varying datasets, prompts, measurement techniques, and more, but core execution logic rarely changes. In most cases, maintaining multiple custom implementations of the same evaluation pattern increases complexity and maintenance overhead.

Quantiles' Custom No-code Evaluations provide a purely configuration-based way to build evaluations without writing any code yourself. Below, we'll walk through creating one using the MMLU-Pro dataset, but you can use any dataset compatible with the evaluation styles described below.

Supported custom no-code evaluation styles

Quantiles custom no-code evaluations currently support two deterministic scoring styles:

exact_match: each sample has a golden answer, and that answer is one fixed string, number, or boolean, and each sample has a golden answer
multiple_choice: each sample has an answer selected from a finite set of choices, and each sample includes possible answers and the correct choice

If you're building an evaluation that fits either of these two styles, we encourage you to use custom no-code evaluations. Otherwise, if your evaluation requires the use of retrieval, multi-step agents, judges, highly specialized scoring logic, or any other logic that doesn't fit the custom no-code framework, use Quantiles custom code evaluations.

Prerequisites

We introduced the Quantiles CLI installation process in our previous Hugging Face article. Here is the command again for reference:

curl -fsSL https://cli.quantiles.io/install.sh | bash

After you have the qt CLI installed, create a working directory containing these two files:

.
├── quantiles.toml
└── prompts/
    └── mmlu-pro.txt

The example in this article uses the public quantiles/MMLU-Pro mirror of the canonical MIT-licensed TIGER-Lab/MMLU-Pro dataset.

1. Define the evaluation

Each evaluation is built around four core configuration fields:

type: set this value to custom_nocode.
dataset: define the dataset source, configuration, split, and revision.
prompt_template_file: specifies how the prompt should be rendered for each sample.
style: configures the specific style (see above for details on valid styels), maps dataset fields to choices and expected labels as appropriate, and selects parsing and scoring behavior.

The following quantiles.toml configuration shows how to define the MMLU-Pro evaluation using these four fields:

# MMLU-Pro Custom No-Code Evaluation Example
[benchmarks.mmlu-pro-nocode]

# This field must be set to "custom_nocode"
type = "custom_nocode"

# Identifies the Hugging Face dataset to evaluate
dataset = { name = "quantiles/MMLU-Pro", config_name = "default", split = "test" }

# This example uses the built-in demo model.
# To evaluate another model (e.g., OpenAI or Anthropic), specify its model identifier below.
# See the following documentation for details on configuring models:
#
# https://quantiles.io/documentation/model-configuration
model = "random"

# The relative path to the Jinja-formatted prompt template for this dataset.
prompt_template_file = "prompts/mmlu-pro.txt"

# Limit this smoke test to 10 dataset rows. Remove this to run the full dataset.
limit = 10

# Configure the benchmark's multiple-choice question and answer structure.
[benchmarks.mmlu-pro-nocode.style]
type = "multiple_choice"
choices = { column = "options" }
choice_labels = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]
answer = { label_column = "answer" }

Important: This walkthrough uses the MMLU-Pro dataset schema but does not match its implementation with 100% precision. If you need 100% fidelity to the original benchmark, see the authors’ reference implementation.

Write the prompt template

Create prompts/mmlu-pro.txt:

{{ row.question }}

{% for choice in choices %}{{ choice.label }}. {{ choice.text }}
{% endfor %}
Answer with only the letter of the correct choice.

Every field in the current dataset sample is available to the prompt template through the row object, as shown above. For multiple-choice evaluations, Quantiles also provides a normalized choices list containing each configured answer label and its corresponding choice text.

The configuration file also defines how dataset fields map to the evaluation inputs and expected answer, and the prompt template controls how those values are presented to the model. This separation allows you to modify the prompt without changing the evaluation logic and makes the fields used to determine the expected answer explicit in the configuration.

2. Run the evaluation

To run the new custom no-code benchmark you've just specified, simply invoke qt run from the same directory as your new quantiles.toml file:

qt run mmlu-pro-nocode

This example uses the built-in demo model, which randomly selects from the configured answer labels so you can validate the evaluation workflow without model credentials or inference costs. Step 4 below shows how to replace the demo model with your own AI model.

Immediately after you start the qt run command, you'll see a progress bar, and when the evaluation completes, the output will look similar to the following:

Created run 1
mmlu-pro-nocode: 100%|████████████████████████████| 10/10
Run 8 completed successfully in 0.46s

Aggregate metrics
 METRIC             VALUE        UNIT
 accuracy           0.82         -
 max_latency_ms     0.059792     ms
 mean_latency_ms    0.022361     ms
 median_latency_ms  0.003959     ms
 min_latency_ms     0.003334     ms
 p95_latency_ms     0.054208     ms
 p99_latency_ms     0.058675     ms

Run `qt show 1 --json` for sample-level details.

3. Inspect the results

When the run finishes, the CLI prints a run_id and aggregate metrics. To inspect the complete run record, including per-sample results, run:

qt show <run_id> --json

Each sample includes several metrics, with these three providing a helpful starting point:

is_correct: whether the parsed label matches the expected answer.
response_parsed: whether the response could be mapped to a configured label.
latency_ms: time spent processing the sample, including provider latency when a hosted model is used.

You can also opt-in to more metrics for multiple_choice evaluations. See this document for more details.

4. Evaluate a hosted model

The random demo model is useful for validating your setup, but its results do not reflect the quality of your own model. To evaluate a hosted model, first configure the required API key then pass its model identifier through --input:

export OPENAI_API_KEY="<your_openai_api_key>"

qt run mmlu-pro-nocode --input '{"model":"openai:gpt-5.6-luna","limit":10}'

The --input flag applies only to the current evaluation run, and overrides just the values you pass. If you omit it, only the configuration defined in quantiles.toml will be used.

Use coding agents to create and run custom evaluations

Your favorite coding agent, such as Codex or Claude Code, can handle most of the work required to create and run Quantiles custom no-code evaluations. Given a dataset and prompt template, it can inspect the available fields, configure the appropriate exact-match or multiple-choice scorer, and run the evaluation. The Quantiles agent skill provides the workflow and safeguards needed to complete those steps reliably.

If you haven't done so yet, install the Quantiles agent skill by passing this prompt to your coding agent:

Please install the Quantiles skill at github.com/quantiles-evals/skill

After you've installed the skill, give your coding agent the following prompt for a multiple-choice evaluation:

Create and run a custom no-code multiple-choice evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <choice_source> as the choice source, <choice_labels> as the choice labels, <answer_source> as the correct-answer source, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.

And use the following prompt to create an exact-match evaluation:

Create and run a custom no-code exact-match evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <answer_column> as the golden-answer column, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.

Conclusion

Quantiles custom no-code evaluations provide a high-performance execution pipeline for evaluations defined only through configuration, reducing the need for custom code for each benchmark or evaluation. You can specify your dataset, prompt, model, and scoring format in a configuration file, run the evaluation with a single command, and be sure it runs efficiently and correctly, with standardized metrics, reproducibility, and resilience. Coding agents can also handle both setup and execution with the Quantiles agent skill.

Custom no-code evaluations currently support exact-match and multiple-choice scoring as detailed above, and more evaluation styles are in development.

Documentation and References

Custom No-code Evaluations Documentation - A guide to Custom No-Code Evaluations
Model Configuration Documentation - Use hosted AI models with Quantiles
Quantiles Agent Skill Repository - Coding-Agent Skill and Workflow Guide

Run and Compare AI Evaluations with a CLI for Developers and Coding Agents

Quantiles.io — Tue, 28 Jul 2026 21:10:55 +0000

TL;DR: This walkthrough shows how developers and coding agents can use Quantiles, an open-source AI evaluation platform licensed under Apache 2.0, to quickly run, analyze, and compare AI evaluations locally. We'll use the SimpleQA Verified benchmark as an example throughout this post, letting you follow the commands, inspect the evaluation results, and configure your own model for the same workflow.

Running an AI evaluation is rarely as simple as sending prompts to a model. Developers must connect datasets, model APIs, scoring logic, result storage, and comparison tooling before they can answer a basic question: did the system get better? When those pieces are spread across scripts, notebooks, and logs, every rerun becomes harder to reproduce and diagnose. A score alone cannot reveal whether the model changed or whether the dataset, prompt, scorer, or sample set changed with it.

Quickstart: Run an example benchmark

The Quantiles CLI is called qt on the command line. A simple curl ... | bash command supports macOS and Linux on X86-64 and Arm64 systems. First, use it to install the CLI:

curl -fsSL https://cli.quantiles.io/install.sh | bash

If you don't want to run code directly sourced from the internet, see the install.sh source code first.

Next, let’s run a built-in benchmark from start to finish using a single command. SimpleQA Verified is a 1,000-prompt benchmark created by Google DeepMind and Google Research. It re-curates questions from OpenAI's SimpleQA benchmark to reduce problems such as incorrect labels, topical bias, redundant questions, and ambiguous source evidence. Each example includes a short factual question in problem, its reference answer, topic and answer-type metadata, and supporting URLs.

Use the following command to run simpleqa-verified using the built-in Quantiles demo model, which doesn't incur any usage charges:

qt run simpleqa-verified

Results from the demo model are intended only to demonstrate the evaluation workflow because its outputs are generated randomly. See the configuration section to run evaluations with your own model.

The current Quantiles integration reads problem and answer columns from the quantiles/simpleqa-verified dataset, which is a fork of the canonical Google dataset. The benchmark remains the work of the original authors.

The above qt run command embeds each response and reference answer using a local embedding model powered by fastembed, and then compares the two using cosine similarity. It then records a sample-level similarity score and aggregates the score distribution. This flow exercises dataset loading, sample execution, scoring, aggregation, and evaluation metadata and metrics storage, without using an LLM provider. It does not reproduce the benchmark's canonical GPT-4.1 autorater, so does not report the resulting similarity metrics as SimpleQA Verified model performance.

When you’re ready to evaluate LLMs, refer to the model configuration documentation section below for detailed setup instructions.

Quantiles calculates and displays the aggregate metrics after all samples have been evaluated and reports the evaluation name, status, timestamps, duration, workflow input and output, error state, and aggregate metrics.

$ qt run simpleqa-verified

Created run 1
simpleqa-verified: 100%|████████████████████| 1000/1000 [00:18<00:00, 1195.52it/s]
Run 1 completed successfully in 19.21s

Aggregate metrics
 METRIC                VALUE     UNIT
 max_similarity        0.8351    -
 mean_similarity       0.5647    -
 median_similarity     0.5589    -
 min_similarity        0.4497    -
 p95_similarity        0.6359    -
 p99_similarity        0.6734    -
 stdev_similarity      0.0396    -
 variance_similarity   0.0015    -

Run `qt show 1 --json` for sample-level details.

Inspect and analyze sample-level results

Each completed evaluation reports its run_id. If the ID is no longer visible, use the following command to list previous runs and their IDs:

qt list

Use the following command to display the sample-level details for a specific run. Here, run_id 1 refers to the SimpleQA Verified evaluation completed above.

qt show 1 --json

Use the --json flag to retrieve sample-level results and execution details as structured, machine-readable output. The JSON document has three top-level fields: run, metrics, and samples. Depending on the recorded step, a sample might include its step key, status, input hash, timestamps, stored output, error, and associated metrics.

Compare two benchmark runs

Many system changes can affect model behavior, including changes to prompts, datasets, code, infrastructure, and the model itself. Comparing two runs makes it possible to measure and inspect the effects of a change.

Run the same simpleqa-verified evaluation a second time and retain both run IDs:

qt run simpleqa-verified

In this example, the two SimpleQA Verified runs have run_id values of 1 and 2:

qt compare 1 2

Comparing runs 1 and 2
                       Run 1              Run 2              Delta
Eval                   simpleqa-verified  simpleqa-verified  SAME
Status                 COMPLETE           COMPLETE
Duration               23.201s            18.720s            -4.481s
Model                  demo-builtin       demo-builtin       SAME
max_similarity         0.7409             0.7618             +0.0209
mean_similarity        0.5634             0.5647             +0.0013
median_similarity      0.56               0.56               -0.000041
min_similarity         0.4501             0.4341             -0.0161
p95_similarity         0.6387             0.6397             +0.001
p99_similarity         0.6779             0.6798             +0.0018
stdev_similarity       0.0409             0.0407             -0.000228
variance_similarity    0.0017             0.0017             -0.000019

Run 'qt compare 1 2 --json' for sample-level details

qt compare exits with status 1 when any checked input, output, step, or aggregate metric differs, and 0 when the runs match, which can be useful for scripts or CI jobs. It's important to remember, however, that a 1 status means "different," and doesn't necessarily indicate that the model got worse. Even a metric improvement counts as a difference. Make sure to apply your own release policy to the JSON output and the status code.

Customize the benchmark

So far, the examples have used the default demo model and the full evaluation dataset. You can instead configure a model from a supported provider, such as OpenAI or Anthropic, and set a sample limit when you do not need to evaluate all 1,000 prompts. Provider-backed evaluations send prompts to an external service and may incur usage charges, so start with a small smoke test.

`--input` flag

The quickest way to customize a built-in benchmark such as SimpleQA Verified is to use --input to override the model or sample limit. These overrides apply only to the current run and the value passed must be a JSON object. To reuse the settings in future SimpleQA Verified runs, add them to quantiles.toml configuration file.

For example, this command limits SimpleQA Verified to 10 samples while still using the built-in demo model:

qt run simpleqa-verified --input '{"limit":10}'

To evaluate the same subset with a specific model, such as one from OpenAI, add the model to the JSON object and ensure that OPENAI_API_KEY is already set:

qt run simpleqa-verified --input '{"limit":10,"model":"openai:gpt-5.6-luna"}'

Configuration file

You can use a configuration file when you want the same model and sample settings to apply across repeated runs. To do this, create exactly one of either quantiles.toml or .quantiles.toml in the directory where you run qt (the CLI will throw an error if it finds both).

The below quantiles.toml configuration runs the first 10 SimpleQA Verified samples with an OpenAI model:

[benchmarks.simpleqa-verified]
samples = 10
model = "openai:gpt-5.6-luna"

Omit the samples key from configuration if you intend to run the evaluation over its complete dataset.

Quantiles loads the matching benchmark section automatically when you run:

qt run simpleqa-verified

Choose a model and configure its API key

Quantiles currently supports four model provider prefixes. It forwards the text after the prefix as the provider's model ID, so the available models depend on the provider's current catalog and your account access. The repository examples below show identifiers that work with the current configuration format.

Supported providers and model ID formats

Model source	`model` value	Required environment variables	Repository example
Built-in demo	Omit `model`, or use `random`	None	Generates random text for workflow validation only
OpenAI	`openai:<model-id>`	`OPENAI_API_KEY`	OpenAI
Anthropic	`anthropic:<model-id>`	`ANTHROPIC_API_KEY`	Anthropic
Google Gemini	`gemini:<model-id>`	`GEMINI_API_KEY`	Gemini
Cloudflare AI Gateway and Workers AI	`cloudflare_ai_gateway:<model-id>`	`CLOUDFLARE_API_KEY`, `CLOUDFLARE_ACCOUNT_ID`, and `CLOUDFLARE_GATEWAY_ID`	Cloudflare AI Gateway

Cloudflare includes models from families such as OpenAI's gpt-oss, Llama, Mistral, Gemma, DeepSeek, Qwen, and GLM. Use the Cloudflare model catalog to choose an exact model ID because availability and access can change independently of Quantiles.

Keep API keys out of quantiles.toml, --input JSON, and source control. Quantiles reads provider credentials from environment variables, so set the variable listed in the table for the selected provider.

For the OpenAI examples below, export the key in the same terminal session where you will run your qt commands:

export OPENAI_API_KEY="<your-openai-api-key>"

Credentials are configured the same way whether you pass the model through a configuration file or the --input flag.

Use a coding agent for evaluation workflows

Quantiles provides an open-source agent skill that gives coding agents, such as Codex or Claude Code, reusable instructions for running evaluations, inspecting sample-level results, comparing runs, and recovering interrupted work with the qt CLI. Run the agent from your project root so it can find your Quantiles configuration and local run history.

First, ask your coding agent to install the skill:

Please install the Quantiles skill at github.com/quantiles-evals/skill

Then prompt your agent to run the SimpleQA Verified benchmark with the demo model:

Run the simpleqa-verified benchmark and summarize the results.

When you are ready to evaluate your model, replace the placeholder in the following prompt and run a small smoke test to verify the model configuration:

Run 10 samples of the simpleqa-verified benchmark using <model provider and model_id>. Confirm that the required provider credentials are available without showing their values; if they are not, stop and tell me what is missing. When finished, summarize the results. Do not run the evaluation until I confirm.

After you have two comparable runs, ask the agent to analyze the change:

Compare the two most recent runs for 'simpleqa-verified'. Summarize the aggregate metrics, sample-level results, failures, and any notable errors. Identify the highest-impact issues to review first, and recommend specific next steps.

Limitations and reproducibility

The built-in demo model generates random, unseeded text. Run IDs, timings, responses, and similarity values will differ between runs, and none of these results should be interpreted as evidence of model quality.
SimpleQA Verified is designed to test answers produced without web search or external tools. When evaluating a model, disable tool access so the run measures the intended task. A coding agent can orchestrate the evaluation workflow, but it should not use tools to answer the benchmark questions.
The first run with a new dataset requires network access to load dataset metadata and any uncached rows. The qt CLI uses the default Hugging Face revision, so be sure to record the dataset repository, configuration, split, and revision in your configuration file when strict reproduction matters.
Model evaluations send prompts to an external service and may incur usage charges. Provider availability and behavior can also change independently of Quantiles.

Conclusion

The Quantiles stack provides a structured, local-first system for running, analyzing, comparing, and reproducing AI evaluations. You can use the built-in demo model to verify that your workflow is configured correctly, then switch to a model from a supported provider (e.g., OpenAI or Anthropic) with a fixed configuration to evaluate real model behavior consistently. Since running evaluations, analyzing results, and comparing runs all use the same CLI, the entire process can be carried out through your preferred coding agent.

If you’d like your benchmark included in Quantiles so others can run it with a single command, submit a request through the Quantiles issue tracker.

Documentation and references

Quantiles - overview of the local-first AI evaluation platform
Quantiles documentation - installation, configuration, CLI, SDK, and evaluation workflow guides
Quantiles GitHub repository - source code, examples, releases, and issue tracker
Quantiles agent skill repository - reusable instructions and installation files for coding-agent evaluation workflows

DEV Community: Quantiles.io

Build an AI Evaluation from a Hugging Face Dataset Without Writing Python

Supported custom no-code evaluation styles

Prerequisites

1. Define the evaluation

Write the prompt template

2. Run the evaluation

3. Inspect the results

4. Evaluate a hosted model

Use coding agents to create and run custom evaluations

Conclusion

Documentation and References

Run and Compare AI Evaluations with a CLI for Developers and Coding Agents

Quickstart: Run an example benchmark

Inspect and analyze sample-level results

Compare two benchmark runs

Customize the benchmark

--input flag

Configuration file

Choose a model and configure its API key

Supported providers and model ID formats

Use a coding agent for evaluation workflows

Limitations and reproducibility

Conclusion

Documentation and references

`--input` flag