DEV Community

Cover image for Build an AI Evaluation from a Hugging Face Dataset Without Writing Python
Quantiles.io
Quantiles.io

Posted on

Build an AI Evaluation from a Hugging Face Dataset Without Writing Python

#ai #llm #opensource #tutorial

AI benchmarks and evaluations have varying datasets, prompts, measurement techniques, and more, but core execution logic rarely changes. In most cases, maintaining multiple custom implementations of the same evaluation pattern increases complexity and maintenance overhead.

Quantiles' Custom No-code Evaluations provide a purely configuration-based way to build evaluations without writing any code yourself. Below, we'll walk through creating one using the MMLU-Pro dataset, but you can use any dataset compatible with the evaluation styles described below.

Supported custom no-code evaluation styles

Quantiles custom no-code evaluations currently support two deterministic scoring styles:

  • exact_match: each sample has a golden answer, and that answer is one fixed string, number, or boolean, and each sample has a golden answer
  • multiple_choice: each sample has an answer selected from a finite set of choices, and each sample includes possible answers and the correct choice

If you're building an evaluation that fits either of these two styles, we encourage you to use custom no-code evaluations. Otherwise, if your evaluation requires the use of retrieval, multi-step agents, judges, highly specialized scoring logic, or any other logic that doesn't fit the custom no-code framework, use Quantiles custom code evaluations.

Prerequisites

We introduced the Quantiles CLI installation process in our previous Hugging Face article. Here is the command again for reference:

curl -fsSL https://cli.quantiles.io/install.sh | bash
Enter fullscreen mode Exit fullscreen mode

After you have the qt CLI installed, create a working directory containing these two files:

.
├── quantiles.toml
└── prompts/
    └── mmlu-pro.txt
Enter fullscreen mode Exit fullscreen mode

The example in this article uses the public quantiles/MMLU-Pro mirror of the canonical MIT-licensed TIGER-Lab/MMLU-Pro dataset.

1. Define the evaluation

Each evaluation is built around four core configuration fields:

  • type: set this value to custom_nocode.
  • dataset: define the dataset source, configuration, split, and revision.
  • prompt_template_file: specifies how the prompt should be rendered for each sample.
  • style: configures the specific style (see above for details on valid styels), maps dataset fields to choices and expected labels as appropriate, and selects parsing and scoring behavior.

The following quantiles.toml configuration shows how to define the MMLU-Pro evaluation using these four fields:

# MMLU-Pro Custom No-Code Evaluation Example
[benchmarks.mmlu-pro-nocode]

# This field must be set to "custom_nocode"
type = "custom_nocode"

# Identifies the Hugging Face dataset to evaluate
dataset = { name = "quantiles/MMLU-Pro", config_name = "default", split = "test" }

# This example uses the built-in demo model.
# To evaluate another model (e.g., OpenAI or Anthropic), specify its model identifier below.
# See the following documentation for details on configuring models:
#
# https://quantiles.io/documentation/model-configuration
model = "random"

# The relative path to the Jinja-formatted prompt template for this dataset.
prompt_template_file = "prompts/mmlu-pro.txt"

# Limit this smoke test to 10 dataset rows. Remove this to run the full dataset.
limit = 10

# Configure the benchmark's multiple-choice question and answer structure.
[benchmarks.mmlu-pro-nocode.style]
type = "multiple_choice"
choices = { column = "options" }
choice_labels = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]
answer = { label_column = "answer" }
Enter fullscreen mode Exit fullscreen mode

Important: This walkthrough uses the MMLU-Pro dataset schema but does not match its implementation with 100% precision. If you need 100% fidelity to the original benchmark, see the authors’ reference implementation.

Write the prompt template

Create prompts/mmlu-pro.txt:

{{ row.question }}

{% for choice in choices %}{{ choice.label }}. {{ choice.text }}
{% endfor %}
Answer with only the letter of the correct choice.
Enter fullscreen mode Exit fullscreen mode

Every field in the current dataset sample is available to the prompt template through the row object, as shown above. For multiple-choice evaluations, Quantiles also provides a normalized choices list containing each configured answer label and its corresponding choice text.

The configuration file also defines how dataset fields map to the evaluation inputs and expected answer, and the prompt template controls how those values are presented to the model. This separation allows you to modify the prompt without changing the evaluation logic and makes the fields used to determine the expected answer explicit in the configuration.

2. Run the evaluation

To run the new custom no-code benchmark you've just specified, simply invoke qt run from the same directory as your new quantiles.toml file:

qt run mmlu-pro-nocode
Enter fullscreen mode Exit fullscreen mode

This example uses the built-in demo model, which randomly selects from the configured answer labels so you can validate the evaluation workflow without model credentials or inference costs. Step 4 below shows how to replace the demo model with your own AI model.

Immediately after you start the qt run command, you'll see a progress bar, and when the evaluation completes, the output will look similar to the following:

Created run 1
mmlu-pro-nocode: 100%|████████████████████████████| 10/10
Run 8 completed successfully in 0.46s

Aggregate metrics
 METRIC             VALUE        UNIT
 accuracy           0.82         -
 max_latency_ms     0.059792     ms
 mean_latency_ms    0.022361     ms
 median_latency_ms  0.003959     ms
 min_latency_ms     0.003334     ms
 p95_latency_ms     0.054208     ms
 p99_latency_ms     0.058675     ms

Run `qt show 1 --json` for sample-level details.
Enter fullscreen mode Exit fullscreen mode

3. Inspect the results

When the run finishes, the CLI prints a run_id and aggregate metrics. To inspect the complete run record, including per-sample results, run:

qt show <run_id> --json
Enter fullscreen mode Exit fullscreen mode

Each sample includes several metrics, with these three providing a helpful starting point:

  • is_correct: whether the parsed label matches the expected answer.
  • response_parsed: whether the response could be mapped to a configured label.
  • latency_ms: time spent processing the sample, including provider latency when a hosted model is used.

You can also opt-in to more metrics for multiple_choice evaluations. See this document for more details.

4. Evaluate a hosted model

The random demo model is useful for validating your setup, but its results do not reflect the quality of your own model. To evaluate a hosted model, first configure the required API key then pass its model identifier through --input:

export OPENAI_API_KEY="<your_openai_api_key>"
Enter fullscreen mode Exit fullscreen mode 
qt run mmlu-pro-nocode --input '{"model":"openai:gpt-5.6-luna","limit":10}'
Enter fullscreen mode Exit fullscreen mode

The --input flag applies only to the current evaluation run, and overrides just the values you pass. If you omit it, only the configuration defined in quantiles.toml will be used.

Use coding agents to create and run custom evaluations

Your favorite coding agent, such as Codex or Claude Code, can handle most of the work required to create and run Quantiles custom no-code evaluations. Given a dataset and prompt template, it can inspect the available fields, configure the appropriate exact-match or multiple-choice scorer, and run the evaluation. The Quantiles agent skill provides the workflow and safeguards needed to complete those steps reliably.

If you haven't done so yet, install the Quantiles agent skill by passing this prompt to your coding agent:

Please install the Quantiles skill at github.com/quantiles-evals/skill
Enter fullscreen mode Exit fullscreen mode

After you've installed the skill, give your coding agent the following prompt for a multiple-choice evaluation:

Create and run a custom no-code multiple-choice evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <choice_source> as the choice source, <choice_labels> as the choice labels, <answer_source> as the correct-answer source, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.
Enter fullscreen mode Exit fullscreen mode

And use the following prompt to create an exact-match evaluation:

Create and run a custom no-code exact-match evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <answer_column> as the golden-answer column, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.
Enter fullscreen mode Exit fullscreen mode

Conclusion

Quantiles custom no-code evaluations provide a high-performance execution pipeline for evaluations defined only through configuration, reducing the need for custom code for each benchmark or evaluation. You can specify your dataset, prompt, model, and scoring format in a configuration file, run the evaluation with a single command, and be sure it runs efficiently and correctly, with standardized metrics, reproducibility, and resilience. Coding agents can also handle both setup and execution with the Quantiles agent skill.

Custom no-code evaluations currently support exact-match and multiple-choice scoring as detailed above, and more evaluation styles are in development.

Documentation and References

Top comments (0)