DEV Community: Quantiles.io

AI Agent Documentation: A Guide to llms.txt, AGENTS.md, SKILL.md, and MCP

Quantiles.io — Thu, 15 Oct 2026 17:38:45 +0000

Documentation, READMEs, and API references have long helped developers get started with new tools. As coding agents take on more of the work, new formats are emerging to help them navigate that information and use developer products successfully.

You've probably encountered some already: llms.txt for finding documentation, AGENTS.md for repository instructions, SKILL.md for reusable workflows, and the Model Context Protocol (MCP) for connecting agents to tools and data. Newer proposals, including auth.md, are beginning to tackle how agents sign up for services and manage credentials. Understanding what each does helps you decide which ones would be useful for your product.

Types of AI agent documentation and protocols

Helping coding agents use a developer product involves more than writing good documentation. Agents need to know where to find information, how to work within a repository, and which tools are available to them. A growing collection of resources aims to make that easier, from documentation files and agent-specific instructions to protocols like MCP. The table below offers a look at what's available, followed by a closer examination of how each works and when it might be worth using.

AI Agent Resources and Their Purpose

Resource	Primary purpose	Where it lives
llms.txt	Helps agents find relevant documentation	Website or documentation site
llms-full.txt	Provides documentation in a single file	Website or documentation site
AGENTS.md	Provides instructions for working in a repository	Repository, with agent-specific loading rules
SKILL.md	Defines reusable workflows and supporting resources	Skill directory
CLAUDE.md	Provides project instructions for Claude Code	Repository or Claude Code configuration
GEMINI.md	Provides project instructions for Gemini CLI	Project or user configuration
Cursor Rules (.mdc)	Provides persistent or context-specific instructions	.cursor/rules/
Copilot instructions	Provides repository-wide or path-specific instructions	.github/copilot-instructions.md or .github/instructions/
MCP	Connects agents to tools, resources, and prompts	MCP server connected to a client
auth.md	Describes service registration and authentication	Service website

What is `llms.txt` and how do AI agents use it?

Finding the right documentation isn't always straightforward for an AI agent. llms.txt is a Markdown file that introduces a product and links to its most useful documentation, giving agents a starting point without having to search through an entire site.

For example, the Quantiles website itself provides an llms.txt that begins like this:

# Quantiles

> Quantiles helps developers evaluate how reliably coding agents complete developer tasks.

## Getting started
- [Quickstart](https://quantiles.io/documentation/quickstart): Get started with agent evaluations.
- [Agent Quickstart](https://quantiles.io/documentation/agent-quickstart): Coding agent instructions to configure, execute, and inspect evaluations.

Agents can follow the links relevant to their task rather than loading everything into context. The proposal also supports discovering these resources through HTML links or HTTP headers and recommends providing Markdown versions of documentation pages. A related convention, llms-full.txt, combines documentation into a single file. Though it seems convenient, a full export may consume unnecessary context. Platforms like Mintlify can generate both files.

Having an llms.txt or llms-full.txt doesn't guarantee that an agent will use it. Evaluate whether agents discover these resources, follow the relevant links, and find the information needed to complete their tasks.

How `AGENTS.md` guides coding agents

When an agent starts working in a repository, it needs to understand how the project works, from installing dependencies and running tests to following coding conventions. The AGENTS.md format provides a dedicated place for these instructions.

For example, here is the beginning of the Quantiles AGENTS.md:

# Quantiles Agent Guide

This file is for coding assistants such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and similar tools. It gives agents a short, public-safe map of the Quantiles repository and the rules for making focused, reviewable changes.

Before attempting tasks in this repository, read these files:

- README.md: public product overview, quickstart, CLI examples, SDK summary, docs links, and agent guidance.
- CONTRIBUTING.md: contributor expectations, development workflow, and review norms.

The instructions depend on the project. They might cover development workflows, testing requirements, or instructions on how to use a particular tool. Unlike website documentation, AGENTS.md is intended to guide agents working in a repository, and not every agent will automatically read it.

Codex layers repository guidance, while other coding agents may load and prioritize them differently. Regardless of the agent, keeping instructions focused and up to date is important. Outdated commands and unnecessary rules can introduce workflow inefficiencies, increase task complexity, and lead to failures when agents interact with your product.

How `SKILL.md` helps agents complete workflows

Some tasks require more than repository instructions. An agent might need to install a CLI, configure credentials, complete several steps, and verify the results. An agent skill provides reusable instructions and supporting resources to help agents complete these workflows.

quantiles-skill/
├── SKILL.md
├── references/
│   ├── configuration.md
│   └── troubleshooting.md
└── scripts/
    └── verify-install.sh

The SKILL.md file contains a name, description, and instructions for completing a workflow. Supporting files can provide additional documentation or scripts when needed. The description is especially important because it helps agents determine when to use the skill.

Both OpenAI Codex and Anthropic's skills implementation use a process called progressive disclosure to discover and use skills. Agents initially see a brief description of available skills and load the full instructions and supporting resources when needed. For this to work, the skill must be installed where the agent can discover it. Simply adding a SKILL.md file to a repository doesn't mean an agent will find or use it. Even when a skill is available, there's still the question of whether an agent will recognize when to use it and follow the instructions correctly.

How MCP connects AI agents to tools and data

While documentation helps agents understand a product, the Model Context Protocol (MCP) provides a standardized way for AI applications to connect to its tools and data. MCP servers can expose three types of capabilities:

Tools: Operations agents can invoke, such as creating a project or starting an evaluation.
Resources: Information applications can retrieve, such as evaluation results or execution logs.
Prompts: Reusable templates that guide specific tasks or interactions.

For example, a Quantiles MCP server could allow an agent to create and compare the results of evaluations using structured tool calls rather than manually composing API requests or using a CLI. Which capabilities an agent can actually use depends on what its MCP client supports and makes available.

MCPs and SKILL.md files can also work together. A skill can guide an agent through a workflow, explaining which operations to use, in what order, and how to verify the results. MCP provides access to the tools needed to carry out those steps.

Instruction files for different coding agents

The resources we've covered so far, including llms.txt, AGENTS.md, SKILL.md, and MCP, are designed to work across different AI agents, although support varies. But coding agents also have their own instruction files and configuration options. These can differ in how instructions are loaded, where they apply, and which instructions take priority. Here's how some popular coding agents handle them:

Claude Code: Uses CLAUDE.md for project instructions and .claude/rules/ for more specific rules. It can also support AGENTS.md, depending on the version and configuration.

Gemini CLI: Uses GEMINI.md for project context and instructions, with options to configure alternative filenames.

Cursor: Supports AGENTS.md and rules stored in .cursor/rules/. Rules can apply automatically, to specific files, when relevant to a task, or when explicitly requested.

GitHub Copilot: Uses .github/copilot-instructions.md for repository-wide guidance and .github/instructions/*.instructions.md for instructions that apply to specific paths.

With several agents to support, maintaining separate instruction files can quickly become difficult. Where possible, keep shared guidance in one place and add agent-specific instructions only when needed. And just as with AGENTS.md or SKILL.md, test whether each agent discovers and follows the instructions as intended.

How auth.md supports AI agent authentication

One of the newer proposals is auth.md, introduced by WorkOS in May 2026. Unlike the more established resources we've covered, it's still in the early stages of adoption. It addresses a familiar problem where an agent can find the documentation and understand an API but still get stuck when it needs to create an account or obtain credentials.

The proposal gives services a way to describe registration, authentication, and available permissions in a Markdown file. It supports two main approaches:

Agent verified: A trusted agent provider verifies the user's identity, allowing the service to register the agent without additional user interaction.
User claimed: The agent starts registration, then asks the user to sign in and confirm a code to authorize access.

Both approaches ultimately allow the agent to obtain a scoped access token. But publishing an auth.md alone isn't enough. The service still needs to implement the authentication endpoints, token issuance, and permission checks.

It's also worth distinguishing auth.md from MCP authorization. While MCP authorization governs access to protected MCP servers, auth.md focuses on helping agents register with a service in the first place, so that they have an identity that can be later authorized.

As an emerging proposal, auth.md is worth watching, but compatibility and agent support still need to be tested. And even with automated registration, some workflows may still require human approval, particularly when account ownership, permissions, or billing are involved.

Which AI agent documentation standards to use, and how to evaluate them

The best place to start is with the problems agents encounter when using your product. A simple library may only need accessible documentation, while a developer platform with more complex workflows might benefit from agent skills, structured tools, and authentication guidance.

Different resources address different parts of the developer experience:

Matching AI Agent Resources to Developer Workflows

Agent needs to...	Relevant resources	What to evaluate
Find information	`llms.txt`, Markdown documentation	Can it find the right documentation?
Understand a workflow	`AGENTS.md`, `SKILL.md`, agent-specific rules	Does it follow the right steps and prerequisites?
Establish access	Authentication flows, `auth.md`	Can it authenticate and obtain the required permissions?
Complete and verify tasks	MCP, CLI, APIs	Can it perform the operation and verify the result?

These resources aren't mutually exclusive, and not every product needs all of them. Adding more instructions isn't always the best solution, either. What matters is whether the ones you do have actually help agents complete developer tasks.

To find out, start by evaluating representative workflows, such as installing your product, completing the quickstart, configuring authentication, or using a more advanced capability. Compare performance with and without the resource to understand where it helps, where it doesn't, and what might need improvement.

Here are a few key questions to consider:

Discovery and activation: Did the agent find and use the relevant documentation, skill, or tool?
Task success and reliability: Did it complete the task correctly and consistently across repeated trials?
Efficiency: How much time, token usage, and tool activity did the task require?
Errors and recovery: Where did the agent get stuck, and could it recover without human intervention?

As AI agents evolve, documentation will too

AI agents are evolving quickly, and the resources that help them use your product will need to keep up. What works for one agent today may not work as well for another, or even for the same agent after an update.

Keeping a representative set of developer tasks makes it easier to spot those changes. Rerun evaluations as your product, documentation, and agents evolve, and use execution traces to understand where things go wrong. Solving these issues will sometimes require instructions. Other times, it might be a clearer error message, a simpler setup process, or a more useful tool response.

Supporting more formats only matters if they improve the developer experience. What matters most is whether agents can navigate your product and complete tasks reliably and successfully.

Using Token Data to Improve the AI Agent Experience

Quantiles.io — Thu, 08 Oct 2026 21:25:35 +0000

Token usage can tell you a lot about what happened between the start of a task and the final answer. An agent may complete the task successfully, but only after searching the documentation several times, retrying failed steps, or pulling large amounts of irrelevant tool output into context. Looking at token usage metrics across repeated trials helps surface that extra work and distinguish a direct, efficient workflow from one that depends on additional context, retries, and recovery.

Tokens add up

Tokens are the units a model uses to process content. In text, a token is roughly 3/4 of a single word, but different models use different tokenizers, so the same text can produce different token counts on different systems. That difference also means equal token totals across models don't necessarily represent the same amount of content or computation.

Token usage is usually measured by numbers of input and output tokens, and some models can also report reasoning tokens, which are used in reasoning models' chains of thought. Input tokens are everything sent to the model, including the prompt, conversation history, documentation, code, and tool results. Output tokens are what the model generates in response. In agent workflows, input tokens can add up quickly as more context is carried across multiple model calls.

The way different hosted model providers price tokens also varies by provider, though generally output tokens are often more expensive per token than input tokens, even when input tokens make up most of the overall usage.

The example below shows a single agent evaluation task through Quantiles and breaks down the input and output tokens used at each step of the workflow.

Tracing Token Usage Through a Quantiles Evaluation Task

Using tokens to understand agent behavior

In an agent workflow, token usage can help reflect the path the agent takes through a task: the context it reads, the tools it calls, the retries it makes, and the information it carries forward. That makes it a useful signal for behavior beyond task completion. For example, two trials may both succeed, even though one requires more context, additional exploration, more recovery steps, and longer execution time.

Recent research shows just how much that path can vary. A 2026 study of eight frontier models on SWE-bench Verified found that repeated executions of the same task could differ by as much as 30 times in total token usage. More tokens didn't necessarily lead to better results either. Accuracy often peaked at an intermediate level of token usage and then plateaued as consumption continued to increase.

Some patterns in the evaluation worth looking for include:

Repeated documentation reads. The agent may be struggling to find, read, or interpret the instructions it needs.
Large tool responses. A tool may be returning much more context than the agent needs for the task.
Retries and recovery. Token usage can climb quickly when the agent encounters an error, revisits earlier steps, or tries several approaches.
Growing context. Long conversations, tool outputs, and intermediate results can accumulate across model calls and increase input-token usage.
Unnecessary exploration. The agent may inspect files, APIs, or documentation that are unrelated to the task. In some cases, the model may not be able to read the data it requires without bringing large amounts of irrelevant data into context as well.
Extra verification. Higher usage isn't always waste. Sometimes the additional tokens come from checking work, testing an implementation, or validating the final result.

What matters most is the pattern, not whether a token count is simply high or low. Running the same task multiple times lets you compare usage across trials and spot meaningful differences. If one attempt uses substantially more tokens than the others, the trace can help show whether the agent needed more context, retries, or effort to recover from an error to reach the same result.

Using token data to improve the agent experience

For a developer tool, token usage is more than a model cost. It can tell you how much work an agent has to do to successfully use your product.

If agents consistently need large amounts of context, repeated tool calls, or several attempts to complete common workflows, that can be a sign of friction or barriers to entry in the product itself. The agent may be spending extra tokens because the documentation is hard to navigate, tool responses contain more information than it needs, an API requires too many steps, or an error message doesn't provide enough guidance to recover cleanly. In that sense, excess token usage can point to places where the product is making the agent work harder than necessary.

The table below shows how token usage data can point to opportunities for improving the agent experience.

Opportunities to Improve the Agent Experience

Opportunity	How it helps
Reduce unnecessary context	Smaller, more focused tool responses and clearer documentation can keep irrelevant information out of the model context.
Make workflows more direct	Fewer unnecessary steps, searches, and retries can reduce both token usage and completion time.
Improve reliability	High or highly variable token usage can point to places where agents are getting confused or taking inconsistent, inefficient paths.
Lower customer costs	When customers pay for model usage, a more token efficient product can make every agent interaction less expensive.
Make your product easier for agents to use	Clear interfaces, useful errors, focused tools, and discoverable documentation can help agents reach the right outcome with less exploration.

Using fewer tokens isn't the goal on its own. It's more important to create a product that agents can understand, navigate, and use efficiently. By tracking token usage alongside task success and execution traces, you can see where the experience is already working well and where improvements could make agent workflows even faster, more reliable, and more successful.

Does your SKILL.md help coding agents?

Quantiles.io — Thu, 01 Oct 2026 22:08:29 +0000

How to evaluate whether agents find and use your skill, how it affects their behavior, and whether it helps or harms the use of your product.

Agent skills introduce a new layer between coding agents and developer products. Instead of leaving an agent to piece everything together from documentation or other hints, a skill can give it targeted instructions, scripts, and resources when they become relevant. A skill goes beyond documentation by helping shape how an agent discovers, navigates, and uses your product.

Of course, a SKILL.md doesn’t necessarily mean the agent will find it, use it, or use your product more effectively just because it exists. Early evidence suggests skills can change agent behavior, but not always for the better. That raises a more useful set of questions about how coding agents discover and use skills, starting with whether they find them at all.

Can the agent find the skill?

Before we can ask whether a skill helps, the agent has to know it exists. And that starts with something surprisingly mundane: where the skill lives.

A SKILL.md sitting somewhere in a repository isn't automatically discoverable. The exact location still depends on the agent and runtime. Claude Code, for example, supports project skills under .claude/skills/skill-name/SKILL.md, while OpenAI Codex uses the same basic skill format, discovers repository skills in .agents/skills, and can also load skills bundled in plugins. When testing a skill, the first thing to verify is that it has been installed somewhere the agent is designed to discover it.

`SKILL.md` Description

Once the skill is available, there's still another step. Both OpenAI and Anthropic use progressive disclosure rather than putting every installed skill’s full instructions into context at once. The agent initially sees lightweight metadata, particularly the skill’s name and description, and uses that information to decide whether the skill is relevant enough to load. That makes the skill description especially important. Both OpenAI and Anthropic recommend using skill metadata to explain what it does and when the agent should use it.

Ask the right questions

Test skill activation at different levels of specificity:

Name the skill: Tests whether the agent can use the skill when explicitly directed. Example: “Use the Quantiles skill to run an evaluation.”
Describe the task: Tests whether the agent identifies the relevant skill from the task. Example: “Configure authentication for this application.”
Describe the goal: Tests whether the agent infers the skill’s relevance from the developer’s goal. Example: “Get this service connected so I can start evaluating agents.”

That’s much closer to how we actually use coding agents. We tell them what we want done, not which skill, MCP server, documentation page, or tool to use along the way. OpenAI recommends testing this range, from explicit and contextual requests to cases where the skill shouldn’t activate at all.

Does the skill make the product easier for agents to use?

Skill discovery is only one part of the evaluation. You also want to know whether the skill changes the agent’s ability to use your product successfully. Compare the same task with and without the skill while holding the agent, repository, documentation, tools, and environment constant. This isolates the skill as the primary experimental variable and makes differences in agent behavior and outcomes easier to interpret.

So how do we know if the skill actually helped? The execution trace gives us a few clues. Looking at these signals can show how the skill changed the agent’s behavior and whether it made the workflow more efficient, including its time to completion, or more reliable.

Measuring the Effect of a Skill

Measurement	What to look for
Task success	Did the agent reach the verified outcome, and was success more consistent with the skill?
Tool calls	Did the skill change how much work the agent needed to complete the task?
Tool errors	Did the skill help the agent avoid incorrect commands, arguments, APIs, or operations?
Repeated tool calls	Did the skill reduce unnecessary retries or loops, or help the agent recover more efficiently?
Latency	Did the skill shorten the path to completion or introduce additional overhead?
Token usage	Did the skill reduce searching and reasoning, or add more context and processing?
Human intervention	Did the agent need less clarification, redirection, or assistance to finish the task?
Skill activation	Did the agent load the skill, and at what point in the task did it become useful?

Where `SKILL.md` helps and where it hurts

Skills may affect agent performance differently across task types. For installation and authentication, a defined product workflow can reduce ambiguity and help the agent avoid incorrect tool calls or unnecessary steps. Recovering from errors often requires more flexibility because the appropriate next action depends on the error and the current environment. Advanced workflows may benefit more from guidance about product conventions and important decision points than from detailed step-by-step instructions.

The effect can also vary by coding agent. One agent may already navigate your documentation well and gain little from additional guidance, while another benefits significantly from the structure. That can change as models improve too. The goal isn't for the skill to improve everything. It's to find where a little extra guidance makes a meaningful difference and where the agent is already better off figuring things out on its own.

The table below shows examples of where skills may help or hurt and how the right level of guidance can reduce uncertainty while leaving room for the agent to adapt.

How `SKILL.md` Affects Different Tasks

More likely to help	More likely to hurt
There is a preferred path. Several approaches are possible, but one is recommended or supported.	The path is already obvious. The agent can infer the correct approach from the environment and normal documentation.
The task has important sequencing. Steps need to happen in a particular order.	The task is simple or direct. Extra instructions add process without adding useful information.
There are non-obvious prerequisites. Authentication, setup, state, dependencies, or permissions must exist first.	The skill repeats easily discoverable information. It mostly duplicates CLI help, API schemas, or straightforward documentation.
There are product-specific decisions. The agent needs to know which API, command, configuration, or workflow is appropriate in a particular situation.	The skill overprescribes the solution. It tells the agent exactly how to solve situations where adapting to the environment would be better.
There are important constraints. Some approaches appear reasonable but are unsupported, unsafe, expensive, destructive, or likely to produce the wrong state.	The instructions reduce flexibility. The agent keeps following the documented path even when errors or environment state suggest it should change course.
Success is not obvious. A successful command or API response does not necessarily mean the task is complete, and the agent needs to know what to verify.	The skill adds unnecessary verification or steps. The agent already has reliable signals that the task succeeded.
The product has unusual conventions. Important behavior differs from what an agent might reasonably infer from similar products.	The skill describes conventional behavior. It spends context explaining patterns the agent already understands.
A small amount of guidance generalizes. A rule or decision principle can help across many variations of the task.	The skill is tutorial-shaped. It encodes one specific trajectory that does not transfer well to unfamiliar situations.
The guidance is current and stable. The instructions describe behavior unlikely to change unexpectedly.	The guidance becomes stale easily. Commands, parameters, defaults, or workflows change frequently and can conflict with the current product.
The skill is concise and targeted. It supplies information the agent needs and leaves the rest to the agent.	The skill is large and noisy. Relevant guidance competes with examples, edge cases, explanations, and instructions that do not apply to the current task.

`SKILL.md` should evolve with your product

Changes to your product and the services it depends on, including AI models, can change how agents experience it. Keep a stable set of evaluation tasks for your most important workflows and use them as regression tests when the product, documentation, or SKILL.md changes. When a new agent failure appears, turn it into another evaluation task so you can test for it going forward.

Coding agents are constantly changing too. Guidance that helps today may become unnecessary as models improve, while new agent behavior may expose problems you've never seen before. That makes agent evaluation an ongoing part of building your product. Keep testing important workflows as both the product and the agents using it change. Sometimes that will lead you to update SKILL.md. Other times, the better fix will be in your documentation, API, error messages, or the product itself.

Evaluating AI Agent Tool Use

Quantiles.io — Thu, 24 Sep 2026 17:32:29 +0000

Understand how coding agents discover, select, and use tools, and how those behaviors affect successful use of your product.

A developer asks their coding agent to integrate your product with their application. The agent finds your docs, installs the SDK, and starts the integration work. Then, it hits an error it can't resolve. After a few attempts, it switches to another product, completes the integration, and moves on. The overall task passed, but the agent failed to use your product successfully.

Looking at tool use helps explain how the agent arrived at the final outcome. You can see which tools it discovered, how it used them, and ultimately where it changed course. These signals help you go deeper than simple task success, to see how successfully agents are using your products.

Is the agent struggling, or is the product?

One failed tool call may say little, but patterns across agent evaluation runs can reveal where they consistently struggle to use the product. When agents repeatedly choose the wrong API, miss the same required parameter, or misinterpret the same response, those patterns may point to parts of the product that are difficult for agents to discover or use. Looking across evaluation runs helps distinguish isolated agent behavior from friction that appears consistently.

These evaluation results give product teams concrete changes to test. Update a tool description, error message, documentation page, or SKILL.md, then rerun the same tasks under comparable conditions to measure the effect. The goal is not to make every agent follow the same path, but to determine whether the change helps agents reach the correct outcome more reliably and with less unnecessary work.

Did the agent find and choose the right tool?

Before an agent can use a tool correctly, it has to find and select the right one. Comparing the tools available at each decision point with the tool the agent chose helps distinguish discovery problems from selection problems. Across evaluation runs, these patterns can reveal tools that agents consistently overlook, confuse, or choose at the wrong time, pointing to where discovery, descriptions, or guidance may need improvement.

The decision path below shows how to distinguish discovery issues from selection issues during an agent workflow.

Tracing Agent Tool Discovery and Selection

Did the agent use the tool correctly?

A successful tool call does not necessarily mean the tool was used correctly. In multi-step workflows, each call must use the right inputs, carry forward relevant state, and remain consistent with the task. Evaluating calls in sequence can reveal errors that individual tool responses may not show.

A few things are especially useful to check:

Arguments: Did the agent provide the correct values for the task?
State: Did it preserve and use relevant information from earlier steps?
Defaults: Did omitted values change the tool's behavior or outcome?
Scope: Did the call stay within the required resources and permissions?

What do repeated tool calls tell us?

Repeated tool calls can reveal where an agent is having difficulty using a product, but repetition alone is not a failure. Retries are often part of agent recovery, especially when the underlying state has changed or the agent is testing a new approach. But when the same call repeats under essentially the same conditions without advancing the task, it can indicate that the agent is stuck. Execution traces help distinguish productive recovery from repetition that adds work without progress.

Did the agent know what to do next?

A successful tool call, or even several, doesn't always mean the task, or even that step of the task, is complete. The agent still has to interpret what the response actually tells it and decide what should happen next. A request that was accepted but is still processing, for example, should lead to another check rather than a claim that the work is finished.

For product builders, this makes tool and response design part of how well agents can use a product. A useful response should give the agent enough information to understand what happened and decide what to do next. That might include the current status, an identifier needed for a later step, or an error that explains how to recover. The clearer that information is, the easier it is for the agent to continue without guessing and/or wasting time and resources.

The final outcome should still be verified against what actually happened in the environment. An agent can interpret a response incorrectly and report success even when the task is incomplete. Evaluating the end-to-end execution trace helps connect the agent's interpretation to the actual outcome and identify where the workflow broke down.

Connecting Agent Behavior to Product Outcomes

Tool-use metrics show how an agent worked through a task, but no single metric tells you whether it used the product effectively. More calls may indicate unnecessary work or careful verification; repeated calls may reflect a loop or successful recovery; and errors may block progress or be quickly resolved. Interpreting these signals requires connecting them to the execution trace and the final task outcome.

Below are common tool-use measurements, what they can help you investigate, and what requires additional context to interpret.

Measurement	What to investigate	What it can't tell you
Tool calls	How much tool interaction the task required	Whether fewer calls would be better
Unique tools	Which tools the agent used	Whether it chose the right tools
Repeated calls	Where the agent retried, recovered, polled, or looped	Whether repetition was necessary
Tool errors	Where tool interactions failed	Whether they prevented progress
Tool duration	Where the agent spent time using tools	Whether that time contributed to progress

Rather than optimizing for fewer calls, fewer errors, or shorter tool time in isolation, evaluate whether a product change helps agents complete tasks more reliably and with less unnecessary work. This connects changes to docs, tools, APIs, or agent instructions to the resulting agent behavior and task outcomes.

Using Execution Traces to Evaluate AI Agent Behavior

Quantiles.io — Thu, 17 Sep 2026 18:05:42 +0000

Building a developer tool today means designing for more than the developer. Coding agents are increasingly part of the interface to the product, reading documentation, invoking CLIs and APIs, modifying configuration, and deciding what tools and workflows to use to achieve tasks. GitHub's 2024 developer survey found that 99% of U.S. respondents had tried AI coding tools at work, which increasingly makes agent compatibility a core requirement of your product.

When we created SKILL.md, AGENTS.md, and llms.txt for Quantiles, we needed to know whether coding agents could actually discover those resources, interpret the instructions correctly, and use them to complete real tasks. We started using Quantiles to evaluate how coding agents interacted with our own documentation. That work made it clear that evaluating a coding agent required an approach that was very different from that of evaluating an LLM.

Inside an Agent Trace

One of the biggest changes we had to make was to capture and evaluate the agent's execution trace. An agent trace is a structured record of all the things that happened during an agent run. Rather than storing only the task input and final output, a trace captures the sequence of operations that produced the result. Depending on the agent and instrumentation, a trace can include:

Model requests and responses
Tool calls, arguments, and results
Retrievals
Errors and retries
Timing and token usage
Context compactions
Changes to external state

Spans

A trace is typically organized into spans, where each span represents an operation such as an LLM call, tool invocation, retrieval, or validation step. Spans contain attributes such as timestamps, status, inputs, outputs, and operation-specific metadata, and parent-child relationships preserve how those operations fit into the overall execution.

This figure shows an execution trace for a coding agent running and analyzing a Quantiles evaluation, with each child span representing an operation performed during the workflow:

Quantiles Evaluation Agent Trace Example

Capturing Agent Traces

There are several ways to capture execution traces. Some agent runtimes and frameworks provide built-in tracing or telemetry, while others require additional instrumentation. A common approach is to emit telemetry using OpenTelemetry, often with agent-specific semantic conventions like those defined in OpenInference that add standardized attributes for LLM calls, tools, retrievals, and related operations. Traces can be exported to Quantiles or other observability systems for storage and analysis across runs.

When native tracing is unavailable, an evaluation harness can capture many of the same events by wrapping model clients, tool interfaces, subprocesses, agent hooks, and beyond. The important requirements in all cases are to preserve stable identifiers and timestamps, parent-child relationships between operations, tool inputs and outputs (without leaking secrets and other private data), errors and status codes, and enough metadata to reconstruct the observable execution of each trial.

Evaluate at the Right Level

A trace contains several kinds of evidence, and not every metric needs the entire trace. The goal is to grade agent behavior where it can be observed most directly. Some properties require the complete execution trajectory, while others can be determined from a single tool call, error event, final environment state, or final output.

For example, end-to-end measures can show whether the coding agent successfully completed the requested Quantiles workflow, how long it took, how much it cost, and whether it recovered from failures along the way. At the span level, we can examine whether the agent found and used the appropriate Quantiles resources, such as the SKILL.md, llms.txt, or the appropriate documentation, and whether it executed the correct qt commands with valid arguments. Individual events can capture narrower signals such as a failed command, invalid configuration, permission error, or approval request. Across repeated trials, these measurements can show how reliably and efficiently coding agents can use Quantiles, the nature of failures, and where failures consistently occur.

Validating behavior vs. Prescribing a trajectory

Agents can often complete the same task through several valid sequences of actions, so requiring one canonical sequence of tool calls can make an evaluation unnecessarily brittle. Anthropic cautions against enforcing specific execution paths when multiple approaches can be correct. Instead, define the constraints the agent must satisfy regardless of the path it takes. These can include permitted tool use, valid arguments, required approvals, retry limits, prohibited side effects, and the expected final state.

Different agent behaviors require different forms of evaluation. Anthropic recommends combining code-based, model-based, and human graders, using each where its form of evidence and judgment is most appropriate.

Grader methods for agent evaluation

Method	Use when	Examples
Code-based graders	The criterion can be evaluated deterministically from observable evidence	Schema validity, tool arguments, exit codes, file changes, environment state, required operations, exact outputs
Model-based graders	The criterion requires semantic or qualitative judgment that cannot be reliably expressed as deterministic checks at scale	Instruction following, relevance of selected documentation, quality of a response, appropriate handling of ambiguity
Human graders	The behavior is nuanced, novel, high-stakes, or requires review or calibration of automated graders	Edge cases, unexpected valid solutions, subjective quality judgments, disagreements about rubric interpretation

Interpret Traces Carefully

Traces provide a large amount of detailed insight into an agent's execution, but they have limits. What appears in a trace depends on how the system is instrumented, and recorded activity doesn't always establish the agent's reasoning or the final state of the environment. Keep these guidelines in mind when using traces for evaluations:

Treat traces as observations, not complete records. A trace can only show what the instrumentation records. Sampling, telemetry limits, unsupported operations, and instrumentation failures can leave gaps in the execution history.
Separate execution from reasoning. A trace can show what the agent did, but since model calls sometimes only show reasoning summaries, a trace can't always tell you why. Base evaluations on behavior you can observe rather than assumptions about the agent's reasoning.
Validate outcomes against the environment. A trace can show that the agent took the right actions without proving that the task actually succeeded. When possible, verify the final environment state. Doing so might require customized tests, filesystem checks, database queries, or API responses.
Allow more than one valid path. Focus on whether the agent met the task requirements rather than whether it followed one specific path, unless the path itself is part of the success criteria.
Keep the measurement system stable. Keep the evaluation setup consistent and versioned across runs. Changes to the runtime, tools, resources, or instrumentation can change the trace even when the agent's behavior hasn't changed.

Keep in mind that OpenTelemetry SDKs can limit or sample the data retained for a span. When one of those things happens, traces can be left with gaps, even if they're not readily apparent.

Conclusion

Developer products are increasingly being used by agents, not just people. That means testing them also requires understanding whether agents can navigate their resources, use their tools, recover from problems, and complete intended workflows reliably. Execution traces make those behaviors observable, while validations confirm whether the task actually succeeded. Together, they provide evidence of both the outcome and the path the agent took to reach it.

How to Evaluate AI Agents

Quantiles.io — Fri, 04 Sep 2026 18:27:42 +0000

Evaluate agent performance using representative tasks, controlled environments, observable execution traces, and repeated trials.

Today’s AI agents extend language models with tools, instructions, and an execution loop for completing multi-step tasks. They can search the web, query customer records, edit code, invoke APIs, and modify external systems. As a result, an agent produces not only an output but a sequence of actions that can alter system state.

As agents shift from producing answers to taking action, evaluation has to evolve with them. A correct outcome tells us that an agent reached its destination, but not what happened along the way. Did it use the right tools? Follow the required steps? Recover when something went wrong? Answering these questions starts with understanding the core components of a well-designed agent evaluation.

Start with the right tasks

The foundation of an agent evaluation is the task set. Much like a test set in conventional model evaluation, it defines the distribution against which performance will be measured. For agents, however, task coverage extends beyond input and expected output.

The table below outlines the types of tasks a well-designed evaluation set should include. Within each area, tasks should vary in complexity, workflow length, tool dependencies, required resources, ambiguity, state, constraints, failure conditions, and expected difficulty.

Evaluation Task Set

Task	Description	What it tests
Core workflows	Representative tasks from the primary workflows the agent is expected to perform.	Whether the agent reliably completes expected work.
High-impact edge cases	Less common scenarios where failure has meaningful consequences, including ambiguous inputs, conflicting constraints, tool failures, and cases requiring escalation.	Whether the agent handles difficult or unusual conditions safely and appropriately.
Capability boundaries	Tasks the agent cannot or should not complete because of missing capabilities, tools, permissions, information, or policy constraints.	Whether the agent recognizes its limits and responds appropriately rather than fabricating results or taking unsupported actions.

Real-world tasks

Real-world tasks are valuable because they capture the complexity of using a product as it actually exists: unexpected requests, incomplete information, hidden dependencies, and constraints that are difficult to anticipate in a designed test. But production data tells an uneven story. Common workflows appear everywhere, while rare but consequential failures may barely appear at all. Abandoned tasks can leave less evidence than successful ones, and some cases require expert judgment to establish what should have happened.

Fortunately, much of the material for these tasks already exists. Useful sources include:

Support cases
Operational logs
Incident reviews
Audit trails
Shadow traces
Workflow documentation
Interviews with domain experts

Production failures are often the best source of new evaluation tasks. When an agent struggles with a real workflow, that failure can be created into a task and added to the evaluation suite. Anthropic’s agent-evaluation guidance recommends combining these observed failures with tasks derived from product requirements, then moving established capability tests into the regression suite over time. If the original case contains sensitive data, remove or replace that information carefully. The goal is to protect the underlying records without simplifying away the conditions that caused the agent to struggle.

Synthetic tasks

Synthetic tasks allow evaluators to create targeted scenarios, vary one condition at a time, and expand coverage without waiting for those cases to appear naturally. They are especially useful when production data is limited or when important conditions are too rare, costly, or risky to reproduce in a live system. But the control that makes synthetic tasks useful can also make them less realistic. Generated cases may use repetitive language, combine conditions in unlikely ways, or simply fail to capture the distribution of work an agent will encounter in the real world.

To reduce this gap, build synthetic tasks from real evidence. Ground them in:

Validated workflows
Product requirements
Company policies
Failures observed in real traces

Then systematically vary inputs, permissions, system state, tool responses, and task complexity. LLMs can generate variations efficiently, but domain experts should verify that each scenario is realistic, its expected outcome is correct, and the task does not leak the solution.

Environment and evaluation harness

A task specifies the goal, but it is only one part of the evaluation. The agent needs an environment in which to act, and the evaluation harness manages the experiment as the agent interacts with that environment. The table below describes the role of each and their components.

Evaluation Environment and Harness

Component	Role	Evaluation requirement
Environment	Provides the state, data, tools, permissions, constraints, external systems, and tool behavior the agent encounters while completing the task.	Reproduce the conditions that could materially change agent behavior or task outcomes, including relevant failures and edge conditions.
Harness	Initializes the environment, presents the task, mediates agent and tool interactions, enforces execution limits, records the trace and resulting state, determines when the run ends, and passes evidence to graders.	Run each trial from a controlled, isolated state and preserve the evidence needed for reproducibility and grading. Validate the harness with known-pass and known-fail cases to catch reset, tool, instrumentation, and grading errors.

The evaluation also needs to be reproducible. Version the task, environment, tool schemas, policies, grader, and agent configuration so that changes in results can be traced back to changes in the system. When production reveals a new failure, reduce it to the smallest reproducible task and add it to the evaluation task set. Over time, these tasks can become regression tests, and changes to a model, prompt, tool, or policy can be measured easily.

Example

Imagine evaluating a support agent asked to refund a damaged order. Its environment contains the customer account, order history, refund policy, permissions, and payment tools, including real constraints such as approval limits and API failures. Without those conditions, the agent may succeed in the evaluation using a workflow that would fail in production.

The harness establishes the starting state, presents the request, gives the agent access to the appropriate tools, records what happens, and determines when the run ends. It then preserves the evidence for grading and resets the environment before the next trial. Grading can examine more than the final state. Apple’s ToolSandbox benchmark, for example, evaluates both intermediate milestones and the final outcome, making it possible to verify not only whether the agent reached the goal but whether important steps occurred along the way.

Choose the right metrics

Task success is necessary, but it is not sufficient to characterize agent performance. Evaluation should separately capture whether the agent reaches the intended outcome, respects required constraints, recovers from failures, and uses time and resources efficiently. This multi-metric approach follows the broader principle demonstrated by Stanford’s HELM framework, which evaluates language models across multiple dimensions rather than treating accuracy as a complete measure of performance. For agent systems, a useful metric stack includes:

Outcome metrics: Measure whether the agent reaches the required, verifiable final state. Award partial credit only when partial completion has meaningful value.
Constraint metrics: Measure violations of required policies, permissions, privacy rules, and security boundaries. Keep these separate from task success.
Trajectory metrics: Measure how the agent executes the task, including invalid tool calls, retries, recovery, escalation, and unnecessary steps. OpenAI’s trace-grading guidance shows how examining an agent’s decisions and tool calls can reveal behaviors that are not visible from the final outcome alone.
Efficiency metrics: Measure latency, cost, token usage, tool calls, and human intervention. Compare efficiency only across runs that meet the required outcome and constraints.

It's tempting to reduce an evaluation to one number, particularly when comparing agents or deciding whether to ship a new version. But a composite score can hide the failures an evaluation is supposed to expose. An agent, for example, might reduce latency and cost while introducing a critical security vulnerability. Averaging those changes together creates a better score without necessarily creating a better agent.

Test for Consistent Performance

Agent behavior is inherently variable. Even with the same task, repeated executions can produce different trajectories and outcomes because of model stochasticity, tool behavior, interaction state, and environmental conditions. A successful trial therefore establishes capability but does not, by itself, establish reliability.

For critical tasks, evaluate repeated trials under controlled conditions and report success across the resulting distribution. pass@k estimates whether at least one of k attempts succeeds and is appropriate when repeated attempts are part of the workflow. Tau-bench introduced pass^k to measure the probability that all k attempts succeed, placing a much stricter requirement on consistency. It's particularly useful for workflows where reliable behavior across repeated executions matters and where a failed run cannot simply be discarded and retried.

Reliability often declines as tasks become longer and more complex. METR’s time-horizon research captures this by estimating the length of human-completable tasks that agents can complete at a given success rate. The same principle applies to product evaluations. Results should be sliced by factors such as workflow length, tool dependencies, ambiguity, and opportunities for recovery. Otherwise, improvements on shorter or simpler tasks can raise aggregate success rates while longer, stateful workflows remain unreliable.

Conclusion

The goal of agent evaluation is not just to produce a score, but to understand how an agent behaves throughout a task. Representative tasks show whether the right capabilities are being tested. Controlled environments make those tests reproducible. Execution traces reveal what happens along the way, and repeated trials show whether that behavior is dependable. Combined, this evidence helps distinguish occasional success from reliable performance when conditions change, tools fail, and tasks become more difficult.

Build an AI Evaluation from a Hugging Face Dataset Without Writing Python

Quantiles.io — Tue, 04 Aug 2026 16:48:38 +0000

AI benchmarks and evaluations have varying datasets, prompts, measurement techniques, and more, but core execution logic rarely changes. In most cases, maintaining multiple custom implementations of the same evaluation pattern increases complexity and maintenance overhead.

Quantiles' Custom No-code Evaluations provide a purely configuration-based way to build evaluations without writing any code yourself. Below, we'll walk through creating one using the MMLU-Pro dataset, but you can use any dataset compatible with the evaluation styles described below.

Supported custom no-code evaluation styles

Quantiles custom no-code evaluations currently support two deterministic scoring styles:

exact_match: each sample has a golden answer, and that answer is one fixed string, number, or boolean, and each sample has a golden answer
multiple_choice: each sample has an answer selected from a finite set of choices, and each sample includes possible answers and the correct choice

If you're building an evaluation that fits either of these two styles, we encourage you to use custom no-code evaluations. Otherwise, if your evaluation requires the use of retrieval, multi-step agents, judges, highly specialized scoring logic, or any other logic that doesn't fit the custom no-code framework, use Quantiles custom code evaluations.

Prerequisites

We introduced the Quantiles CLI installation process in our previous Hugging Face article. Here is the command again for reference:

curl -fsSL https://cli.quantiles.io/install.sh | bash

After you have the qt CLI installed, create a working directory containing these two files:

.
├── quantiles.toml
└── prompts/
    └── mmlu-pro.txt

The example in this article uses the public quantiles/MMLU-Pro mirror of the canonical MIT-licensed TIGER-Lab/MMLU-Pro dataset.

1. Define the evaluation

Each evaluation is built around four core configuration fields:

type: set this value to custom_nocode.
dataset: define the dataset source, configuration, split, and revision.
prompt_template_file: specifies how the prompt should be rendered for each sample.
style: configures the specific style (see above for details on valid styels), maps dataset fields to choices and expected labels as appropriate, and selects parsing and scoring behavior.

The following quantiles.toml configuration shows how to define the MMLU-Pro evaluation using these four fields:

# MMLU-Pro Custom No-Code Evaluation Example
[benchmarks.mmlu-pro-nocode]

# This field must be set to "custom_nocode"
type = "custom_nocode"

# Identifies the Hugging Face dataset to evaluate
dataset = { name = "quantiles/MMLU-Pro", config_name = "default", split = "test" }

# This example uses the built-in demo model.
# To evaluate another model (e.g., OpenAI or Anthropic), specify its model identifier below.
# See the following documentation for details on configuring models:
#
# https://quantiles.io/documentation/model-configuration
model = "random"

# The relative path to the Jinja-formatted prompt template for this dataset.
prompt_template_file = "prompts/mmlu-pro.txt"

# Limit this smoke test to 10 dataset rows. Remove this to run the full dataset.
limit = 10

# Configure the benchmark's multiple-choice question and answer structure.
[benchmarks.mmlu-pro-nocode.style]
type = "multiple_choice"
choices = { column = "options" }
choice_labels = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]
answer = { label_column = "answer" }

Important: This walkthrough uses the MMLU-Pro dataset schema but does not match its implementation with 100% precision. If you need 100% fidelity to the original benchmark, see the authors’ reference implementation.

Write the prompt template

Create prompts/mmlu-pro.txt:

{{ row.question }}

{% for choice in choices %}{{ choice.label }}. {{ choice.text }}
{% endfor %}
Answer with only the letter of the correct choice.

Every field in the current dataset sample is available to the prompt template through the row object, as shown above. For multiple-choice evaluations, Quantiles also provides a normalized choices list containing each configured answer label and its corresponding choice text.

The configuration file also defines how dataset fields map to the evaluation inputs and expected answer, and the prompt template controls how those values are presented to the model. This separation allows you to modify the prompt without changing the evaluation logic and makes the fields used to determine the expected answer explicit in the configuration.

2. Run the evaluation

To run the new custom no-code benchmark you've just specified, simply invoke qt run from the same directory as your new quantiles.toml file:

qt run mmlu-pro-nocode

This example uses the built-in demo model, which randomly selects from the configured answer labels so you can validate the evaluation workflow without model credentials or inference costs. Step 4 below shows how to replace the demo model with your own AI model.

Immediately after you start the qt run command, you'll see a progress bar, and when the evaluation completes, the output will look similar to the following:

Created run 1
mmlu-pro-nocode: 100%|████████████████████████████| 10/10
Run 8 completed successfully in 0.46s

Aggregate metrics
 METRIC             VALUE        UNIT
 accuracy           0.82         -
 max_latency_ms     0.059792     ms
 mean_latency_ms    0.022361     ms
 median_latency_ms  0.003959     ms
 min_latency_ms     0.003334     ms
 p95_latency_ms     0.054208     ms
 p99_latency_ms     0.058675     ms

Run `qt show 1 --json` for sample-level details.

3. Inspect the results

When the run finishes, the CLI prints a run_id and aggregate metrics. To inspect the complete run record, including per-sample results, run:

qt show <run_id> --json

Each sample includes several metrics, with these three providing a helpful starting point:

is_correct: whether the parsed label matches the expected answer.
response_parsed: whether the response could be mapped to a configured label.
latency_ms: time spent processing the sample, including provider latency when a hosted model is used.

You can also opt-in to more metrics for multiple_choice evaluations. See this document for more details.

4. Evaluate a hosted model

The random demo model is useful for validating your setup, but its results do not reflect the quality of your own model. To evaluate a hosted model, first configure the required API key then pass its model identifier through --input:

export OPENAI_API_KEY="<your_openai_api_key>"

qt run mmlu-pro-nocode --input '{"model":"openai:gpt-5.6-luna","limit":10}'

The --input flag applies only to the current evaluation run, and overrides just the values you pass. If you omit it, only the configuration defined in quantiles.toml will be used.

Use coding agents to create and run custom evaluations

Your favorite coding agent, such as Codex or Claude Code, can handle most of the work required to create and run Quantiles custom no-code evaluations. Given a dataset and prompt template, it can inspect the available fields, configure the appropriate exact-match or multiple-choice scorer, and run the evaluation. The Quantiles agent skill provides the workflow and safeguards needed to complete those steps reliably.

If you haven't done so yet, install the Quantiles agent skill by passing this prompt to your coding agent:

Please install the Quantiles skill at github.com/quantiles-evals/skill

After you've installed the skill, give your coding agent the following prompt for a multiple-choice evaluation:

Create and run a custom no-code multiple-choice evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <choice_source> as the choice source, <choice_labels> as the choice labels, <answer_source> as the correct-answer source, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.

And use the following prompt to create an exact-match evaluation:

Create and run a custom no-code exact-match evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <answer_column> as the golden-answer column, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.

Conclusion

Quantiles custom no-code evaluations provide a high-performance execution pipeline for evaluations defined only through configuration, reducing the need for custom code for each benchmark or evaluation. You can specify your dataset, prompt, model, and scoring format in a configuration file, run the evaluation with a single command, and be sure it runs efficiently and correctly, with standardized metrics, reproducibility, and resilience. Coding agents can also handle both setup and execution with the Quantiles agent skill.

Custom no-code evaluations currently support exact-match and multiple-choice scoring as detailed above, and more evaluation styles are in development.

Documentation and References

Custom No-code Evaluations Documentation - A guide to Custom No-Code Evaluations
Model Configuration Documentation - Use hosted AI models with Quantiles
Quantiles Agent Skill Repository - Coding-Agent Skill and Workflow Guide

Run and Compare AI Evaluations with a CLI for Developers and Coding Agents

Quantiles.io — Tue, 28 Jul 2026 21:10:55 +0000

TL;DR: This walkthrough shows how developers and coding agents can use Quantiles, an open-source AI evaluation platform licensed under Apache 2.0, to quickly run, analyze, and compare AI evaluations locally. We'll use the SimpleQA Verified benchmark as an example throughout this post, letting you follow the commands, inspect the evaluation results, and configure your own model for the same workflow.

Running an AI evaluation is rarely as simple as sending prompts to a model. Developers must connect datasets, model APIs, scoring logic, result storage, and comparison tooling before they can answer a basic question: did the system get better? When those pieces are spread across scripts, notebooks, and logs, every rerun becomes harder to reproduce and diagnose. A score alone cannot reveal whether the model changed or whether the dataset, prompt, scorer, or sample set changed with it.

Quickstart: Run an example benchmark

The Quantiles CLI is called qt on the command line. A simple curl ... | bash command supports macOS and Linux on X86-64 and Arm64 systems. First, use it to install the CLI:

curl -fsSL https://cli.quantiles.io/install.sh | bash

If you don't want to run code directly sourced from the internet, see the install.sh source code first.

Next, let’s run a built-in benchmark from start to finish using a single command. SimpleQA Verified is a 1,000-prompt benchmark created by Google DeepMind and Google Research. It re-curates questions from OpenAI's SimpleQA benchmark to reduce problems such as incorrect labels, topical bias, redundant questions, and ambiguous source evidence. Each example includes a short factual question in problem, its reference answer, topic and answer-type metadata, and supporting URLs.

Use the following command to run simpleqa-verified using the built-in Quantiles demo model, which doesn't incur any usage charges:

qt run simpleqa-verified

Results from the demo model are intended only to demonstrate the evaluation workflow because its outputs are generated randomly. See the configuration section to run evaluations with your own model.

The current Quantiles integration reads problem and answer columns from the quantiles/simpleqa-verified dataset, which is a fork of the canonical Google dataset. The benchmark remains the work of the original authors.

The above qt run command embeds each response and reference answer using a local embedding model powered by fastembed, and then compares the two using cosine similarity. It then records a sample-level similarity score and aggregates the score distribution. This flow exercises dataset loading, sample execution, scoring, aggregation, and evaluation metadata and metrics storage, without using an LLM provider. It does not reproduce the benchmark's canonical GPT-4.1 autorater, so does not report the resulting similarity metrics as SimpleQA Verified model performance.

When you’re ready to evaluate LLMs, refer to the model configuration documentation section below for detailed setup instructions.

Quantiles calculates and displays the aggregate metrics after all samples have been evaluated and reports the evaluation name, status, timestamps, duration, workflow input and output, error state, and aggregate metrics.

$ qt run simpleqa-verified

Created run 1
simpleqa-verified: 100%|████████████████████| 1000/1000 [00:18<00:00, 1195.52it/s]
Run 1 completed successfully in 19.21s

Aggregate metrics
 METRIC                VALUE     UNIT
 max_similarity        0.8351    -
 mean_similarity       0.5647    -
 median_similarity     0.5589    -
 min_similarity        0.4497    -
 p95_similarity        0.6359    -
 p99_similarity        0.6734    -
 stdev_similarity      0.0396    -
 variance_similarity   0.0015    -

Run `qt show 1 --json` for sample-level details.

Inspect and analyze sample-level results

Each completed evaluation reports its run_id. If the ID is no longer visible, use the following command to list previous runs and their IDs:

qt list

Use the following command to display the sample-level details for a specific run. Here, run_id 1 refers to the SimpleQA Verified evaluation completed above.

qt show 1 --json

Use the --json flag to retrieve sample-level results and execution details as structured, machine-readable output. The JSON document has three top-level fields: run, metrics, and samples. Depending on the recorded step, a sample might include its step key, status, input hash, timestamps, stored output, error, and associated metrics.

Compare two benchmark runs

Many system changes can affect model behavior, including changes to prompts, datasets, code, infrastructure, and the model itself. Comparing two runs makes it possible to measure and inspect the effects of a change.

Run the same simpleqa-verified evaluation a second time and retain both run IDs:

qt run simpleqa-verified

In this example, the two SimpleQA Verified runs have run_id values of 1 and 2:

qt compare 1 2

Comparing runs 1 and 2
                       Run 1              Run 2              Delta
Eval                   simpleqa-verified  simpleqa-verified  SAME
Status                 COMPLETE           COMPLETE
Duration               23.201s            18.720s            -4.481s
Model                  demo-builtin       demo-builtin       SAME
max_similarity         0.7409             0.7618             +0.0209
mean_similarity        0.5634             0.5647             +0.0013
median_similarity      0.56               0.56               -0.000041
min_similarity         0.4501             0.4341             -0.0161
p95_similarity         0.6387             0.6397             +0.001
p99_similarity         0.6779             0.6798             +0.0018
stdev_similarity       0.0409             0.0407             -0.000228
variance_similarity    0.0017             0.0017             -0.000019

Run 'qt compare 1 2 --json' for sample-level details

qt compare exits with status 1 when any checked input, output, step, or aggregate metric differs, and 0 when the runs match, which can be useful for scripts or CI jobs. It's important to remember, however, that a 1 status means "different," and doesn't necessarily indicate that the model got worse. Even a metric improvement counts as a difference. Make sure to apply your own release policy to the JSON output and the status code.

Customize the benchmark

So far, the examples have used the default demo model and the full evaluation dataset. You can instead configure a model from a supported provider, such as OpenAI or Anthropic, and set a sample limit when you do not need to evaluate all 1,000 prompts. Provider-backed evaluations send prompts to an external service and may incur usage charges, so start with a small smoke test.

`--input` flag

The quickest way to customize a built-in benchmark such as SimpleQA Verified is to use --input to override the model or sample limit. These overrides apply only to the current run and the value passed must be a JSON object. To reuse the settings in future SimpleQA Verified runs, add them to quantiles.toml configuration file.

For example, this command limits SimpleQA Verified to 10 samples while still using the built-in demo model:

qt run simpleqa-verified --input '{"limit":10}'

To evaluate the same subset with a specific model, such as one from OpenAI, add the model to the JSON object and ensure that OPENAI_API_KEY is already set:

qt run simpleqa-verified --input '{"limit":10,"model":"openai:gpt-5.6-luna"}'

Configuration file

You can use a configuration file when you want the same model and sample settings to apply across repeated runs. To do this, create exactly one of either quantiles.toml or .quantiles.toml in the directory where you run qt (the CLI will throw an error if it finds both).

The below quantiles.toml configuration runs the first 10 SimpleQA Verified samples with an OpenAI model:

[benchmarks.simpleqa-verified]
samples = 10
model = "openai:gpt-5.6-luna"

Omit the samples key from configuration if you intend to run the evaluation over its complete dataset.

Quantiles loads the matching benchmark section automatically when you run:

qt run simpleqa-verified

Choose a model and configure its API key

Quantiles currently supports four model provider prefixes. It forwards the text after the prefix as the provider's model ID, so the available models depend on the provider's current catalog and your account access. The repository examples below show identifiers that work with the current configuration format.

Supported providers and model ID formats

Model source	`model` value	Required environment variables	Repository example
Built-in demo	Omit `model`, or use `random`	None	Generates random text for workflow validation only
OpenAI	`openai:<model-id>`	`OPENAI_API_KEY`	OpenAI
Anthropic	`anthropic:<model-id>`	`ANTHROPIC_API_KEY`	Anthropic
Google Gemini	`gemini:<model-id>`	`GEMINI_API_KEY`	Gemini
Cloudflare AI Gateway and Workers AI	`cloudflare_ai_gateway:<model-id>`	`CLOUDFLARE_API_KEY`, `CLOUDFLARE_ACCOUNT_ID`, and `CLOUDFLARE_GATEWAY_ID`	Cloudflare AI Gateway

Cloudflare includes models from families such as OpenAI's gpt-oss, Llama, Mistral, Gemma, DeepSeek, Qwen, and GLM. Use the Cloudflare model catalog to choose an exact model ID because availability and access can change independently of Quantiles.

Keep API keys out of quantiles.toml, --input JSON, and source control. Quantiles reads provider credentials from environment variables, so set the variable listed in the table for the selected provider.

For the OpenAI examples below, export the key in the same terminal session where you will run your qt commands:

export OPENAI_API_KEY="<your-openai-api-key>"

Credentials are configured the same way whether you pass the model through a configuration file or the --input flag.

Use a coding agent for evaluation workflows

Quantiles provides an open-source agent skill that gives coding agents, such as Codex or Claude Code, reusable instructions for running evaluations, inspecting sample-level results, comparing runs, and recovering interrupted work with the qt CLI. Run the agent from your project root so it can find your Quantiles configuration and local run history.

First, ask your coding agent to install the skill:

Please install the Quantiles skill at github.com/quantiles-evals/skill

Then prompt your agent to run the SimpleQA Verified benchmark with the demo model:

Run the simpleqa-verified benchmark and summarize the results.

When you are ready to evaluate your model, replace the placeholder in the following prompt and run a small smoke test to verify the model configuration:

Run 10 samples of the simpleqa-verified benchmark using <model provider and model_id>. Confirm that the required provider credentials are available without showing their values; if they are not, stop and tell me what is missing. When finished, summarize the results. Do not run the evaluation until I confirm.

After you have two comparable runs, ask the agent to analyze the change:

Compare the two most recent runs for 'simpleqa-verified'. Summarize the aggregate metrics, sample-level results, failures, and any notable errors. Identify the highest-impact issues to review first, and recommend specific next steps.

Limitations and reproducibility

The built-in demo model generates random, unseeded text. Run IDs, timings, responses, and similarity values will differ between runs, and none of these results should be interpreted as evidence of model quality.
SimpleQA Verified is designed to test answers produced without web search or external tools. When evaluating a model, disable tool access so the run measures the intended task. A coding agent can orchestrate the evaluation workflow, but it should not use tools to answer the benchmark questions.
The first run with a new dataset requires network access to load dataset metadata and any uncached rows. The qt CLI uses the default Hugging Face revision, so be sure to record the dataset repository, configuration, split, and revision in your configuration file when strict reproduction matters.
Model evaluations send prompts to an external service and may incur usage charges. Provider availability and behavior can also change independently of Quantiles.

Conclusion

The Quantiles stack provides a structured, local-first system for running, analyzing, comparing, and reproducing AI evaluations. You can use the built-in demo model to verify that your workflow is configured correctly, then switch to a model from a supported provider (e.g., OpenAI or Anthropic) with a fixed configuration to evaluate real model behavior consistently. Since running evaluations, analyzing results, and comparing runs all use the same CLI, the entire process can be carried out through your preferred coding agent.

If you’d like your benchmark included in Quantiles so others can run it with a single command, submit a request through the Quantiles issue tracker.

Documentation and references

Quantiles - overview of the local-first AI evaluation platform
Quantiles documentation - installation, configuration, CLI, SDK, and evaluation workflow guides
Quantiles GitHub repository - source code, examples, releases, and issue tracker
Quantiles agent skill repository - reusable instructions and installation files for coding-agent evaluation workflows

DEV Community: Quantiles.io

AI Agent Documentation: A Guide to llms.txt, AGENTS.md, SKILL.md, and MCP

Types of AI agent documentation and protocols

What is llms.txt and how do AI agents use it?

How AGENTS.md guides coding agents

How SKILL.md helps agents complete workflows

How MCP connects AI agents to tools and data

Instruction files for different coding agents

How auth.md supports AI agent authentication

Which AI agent documentation standards to use, and how to evaluate them

As AI agents evolve, documentation will too

Using Token Data to Improve the AI Agent Experience

Tokens add up

Tracing Token Usage Through a Quantiles Evaluation Task

Using tokens to understand agent behavior

Using token data to improve the agent experience

Opportunities to Improve the Agent Experience

Does your SKILL.md help coding agents?

Can the agent find the skill?

SKILL.md Description

Ask the right questions

Does the skill make the product easier for agents to use?

Measuring the Effect of a Skill

Where SKILL.md helps and where it hurts

How SKILL.md Affects Different Tasks

SKILL.md should evolve with your product

Evaluating AI Agent Tool Use

Is the agent struggling, or is the product?

Did the agent find and choose the right tool?

Did the agent use the tool correctly?

What do repeated tool calls tell us?

Did the agent know what to do next?

Connecting Agent Behavior to Product Outcomes

Using Execution Traces to Evaluate AI Agent Behavior

Inside an Agent Trace

Spans

Capturing Agent Traces

Evaluate at the Right Level

Validating behavior vs. Prescribing a trajectory

Interpret Traces Carefully

Conclusion

How to Evaluate AI Agents

Start with the right tasks

Evaluation Task Set

Real-world tasks

Synthetic tasks

Environment and evaluation harness

Evaluation Environment and Harness

Example

Choose the right metrics

Test for Consistent Performance

Conclusion

Build an AI Evaluation from a Hugging Face Dataset Without Writing Python

Supported custom no-code evaluation styles

Prerequisites

1. Define the evaluation

Write the prompt template

2. Run the evaluation

3. Inspect the results

4. Evaluate a hosted model

Use coding agents to create and run custom evaluations

Conclusion

Documentation and References

Run and Compare AI Evaluations with a CLI for Developers and Coding Agents

Quickstart: Run an example benchmark

Inspect and analyze sample-level results

Compare two benchmark runs

Customize the benchmark

--input flag

Configuration file

Choose a model and configure its API key

Supported providers and model ID formats

Use a coding agent for evaluation workflows

Limitations and reproducibility

Conclusion

Documentation and references

What is `llms.txt` and how do AI agents use it?

How `AGENTS.md` guides coding agents

How `SKILL.md` helps agents complete workflows

`SKILL.md` Description

Where `SKILL.md` helps and where it hurts

How `SKILL.md` Affects Different Tasks

`SKILL.md` should evolve with your product

`--input` flag