DEV Community: Peter

Replay Production Call Transcripts Against Your Voice Agent's Current Graph

Peter — Thu, 14 May 2026 02:29:45 +0000

The hardest regressions to catch in voice agents are the ones that pass every synthetic test and only break on the actual conversations real users have. You ship a prompt change, the LLM judges still score everything green, and a week later support tickets pile up because the agent now confirms the wrong appointment type or skips the identity check on a path nobody wrote a test for.

Synthetic tests are bounded by the imagination of whoever wrote them. Production traffic is not. Voicetest 0.41 closes that gap with two new operations: import real call transcripts as runs, then replay them against the agent's current graph to see whether the live agent still produces the same outcomes.

This post covers the import format, the replay mechanics, and a workflow for using your own production calls as a regression suite.

Two operations, one data model

Operation	What it does	CLI
Import calls	Parse a platform transcript dump and persist as a Run with `status="imported"` Results	`voicetest import-call --agent <id> --transcript file.json`
Replay	Drive a fresh conversation against the agent's current graph using a source Run's user turns as a script	`voicetest replay <run-id>`

Both produce ordinary Run records. Imported runs and replay runs render in the same UI as simulated runs, share the same storage, and can be compared turn-for-turn.

Importing Retell calls

Voicetest accepts the Retell call object in the shape returned by GET /v2/get-call/{call_id} and in the post-call webhook envelope. Single object, array of objects, or an array of webhook payloads — all four shapes parse the same way:

{
  "call_id": "call_abc123",
  "transcript_object": [
    {"role": "agent", "content": "Hi, how can I help?"},
    {"role": "user", "content": "I need to cancel my order."},
    {"role": "agent", "content": "Sure — can I get your order number?"},
    {"role": "user", "content": "It's 4471."}
  ],
  "duration_ms": 60000,
  "start_timestamp": 1700000000000,
  "end_timestamp": 1700000060000,
  "disconnection_reason": "user_hangup"
}

The adapter maps Retell's role: "agent" to voicetest's role: "assistant" and ignores word-level timing. One CLI call ingests an entire dump:

voicetest import-call --agent receptionist --transcript yesterdays-calls.json
# Imported 247 conversation(s) into run abc123de

Each call becomes a Result inside a single Run. Results have status="imported", no test_case_id, and no call_id linkage to a synthetic test — they are passive captures of what actually happened.

Other platforms (VAPI, LiveKit, Telnyx, Bland) are not supported in v1, but the --format flag is parameterized so adapters can be added without breaking changes.

Replay against the current graph

Replay reuses the existing conversation engine. The only difference is the user side: instead of an LLM-driven UserSimulator, replay runs use ScriptedUserSimulator, which yields the recorded user turns from the source transcript in order:

class ScriptedUserSimulator:
    def __init__(self, source_transcript: list[Message]):
        self._user_turns = [m.content for m in source_transcript if m.role == "user"]
        self._index = 0

    async def generate(self, transcript, on_token=None, on_error=None):
        if self._index >= len(self._user_turns):
            return None
        message = self._user_turns[self._index]
        self._index += 1
        return SimulatorResponse(message=message)

The runner drives the conversation against the live agent the same way it would for a normal test run. Source agent turns are discarded — only the user side is scripted. The live agent's responses replace the recorded ones, so what you get back is "how my agent responds today to the user-side of yesterday's call."

voicetest replay abc123de

The new run lands next to the source in the runs UI. Open the source run on the left, the replay run on the right, scroll the transcripts side by side.

What replay catches that synthetic tests miss

Three kinds of regression slip past synthetic suites and surface clearly under replay:

Phrasings the suite never simulated. Real users say "yeah I dunno can you just send it whenever" — UserSimulator produces grammatical, goal-directed turns. If a prompt change makes the agent fragile to filler or hedging, replay catches it; the simulator probably won't.
Long-tail intents. A receptionist agent's test suite has 20 cases. Production has 200 distinct user goals. Replay against a week of calls exercises whatever distribution actually exists.
Drift between platforms. If you maintain the agent on two platforms (e.g. Retell + VAPI for failover), replay the same Retell transcripts against both. Divergence shows up immediately.

Best-effort divergence handling

Replay does not try to recover gracefully when the live agent diverges from the recorded conversation. If today's agent asks a different question than yesterday's, the next recorded user turn may not fit:

Recorded: Agent: "What's your account number?" → User: "4471."
Live: Agent: "Can you spell your last name?" → User: "4471." (scripted turn doesn't match the new question)

The replay continues anyway. The conversation as a whole still produces a transcript you can judge — the regression signal is "this conversation is now incoherent" and that is exactly the thing you want to know about.

A future version may add LLM-based divergence handling (re-prompting the simulator when the script doesn't fit). v1 keeps the contract simple: deterministic, reproducible, fast.

A regression suite from production traffic in 60 seconds

Putting it together — assuming you already have a Retell agent and you can pull a day's worth of call dumps:

# 1. Import yesterday's production calls as a Run
voicetest import-call --agent receptionist --transcript prod-calls-2026-05-06.json
# Imported 312 conversation(s) into run prod-2026-05-06

# 2. Make whatever prompt change you want to test
$EDITOR agents/receptionist.json

# 3. Push the new graph to your agent's storage
voicetest import --file agents/receptionist.json --id receptionist

# 4. Replay yesterday's calls against the new graph
voicetest replay prod-2026-05-06
# Replayed 312 conversation(s) into run replay-abc123

# 5. Open both runs in the UI and scan for divergence
voicetest serve

312 real conversations, replayed against the new graph, with no test cases written. The synthetic suite still has its place — it is the regression net for the cases you've already thought about. Replay is the net for the cases you haven't.

Limitations to know about

Retell only in v1. Adapters for VAPI, LiveKit, Bland, Telnyx are tracked but not yet shipped.
No PII redaction at import time. If your transcripts contain sensitive data, redact before ingest. Voicetest stores Results in the same DuckDB file as the rest of your runs.
No diff view yet. Source and replay are separate runs in the UI; comparing them is manual scrolling for now. A built-in turn-aligned diff is on the roadmap.
No automatic test-case generation from imports. Imported transcripts are runs, not test cases. Clustering them into a deduplicated test suite is on the proposals list.

REST and UI surfaces

Both operations are also available via REST and the web UI:

# Import
curl -X POST http://localhost:8000/api/agents/receptionist/import-call \
  -F "transcript=@prod-calls.json" \
  -F "format=retell"

# Replay
curl -X POST http://localhost:8000/api/runs/prod-2026-05-06/replay

The agent page has an "Import Calls…" button; the run detail page has a "Replay" button. Same data model, same storage — pick whichever surface fits.

Voicetest is open source under Apache 2.0. GitHub: github.com/voicetestdev/voicetest

Five Node Types and Global Interrupts: How Voicetest Models Real Agent Graphs

Peter — Fri, 08 May 2026 13:01:09 +0000

A voice agent graph is more than a list of conversational states with prompts. Real agents extract variables from what the caller said and branch on them. They evaluate deterministic equations to route by account type or balance. They end the call cleanly or transfer to a human. And they need ways to handle interrupts — "speak to a manager," "I want to start over," "this is an emergency" — from anywhere in the flow without an explicit edge from every node.

Earlier versions of voicetest's AgentGraph IR modeled all of this as one undifferentiated node type with prompts and transitions. That was enough for round-tripping simple flows but lost structure on import: a Retell logic node and a Retell conversation node looked identical in the IR, the engine drove them both the same way, and exports back to Retell rebuilt the structure heuristically.

The IR now distinguishes five node types and a global-node flag. This post covers what each one represents, how the engine handles it, and what it means for round-trip fidelity across platforms.

The node type enum

class NodeType(StrEnum):
    CONVERSATION = "conversation"
    LOGIC = "logic"
    EXTRACT = "extract"
    END = "end"
    TRANSFER = "transfer"

Every AgentNode carries a node_type. The engine routes by type:

Type	Speaks?	Behavior
`conversation`	yes	Generates a response from `state_prompt`, evaluates LLM-prompted transitions, fires always-edges
`extract`	no	LLM extracts variables from the conversation so far, then branches on equation transitions
`logic`	no	Pure deterministic branch — evaluates equation transitions on already-extracted variables
`end`	depends	If it has a prompt, agent speaks then call ends; if not, call ends immediately
`transfer`	depends	Same as end, but the call status reflects a transfer rather than a hangup

The engine's main loop walks past silent nodes (extract, logic, prompt-less end/transfer) without producing user-visible turns and stops at the next conversation node to actually generate a response. That distinction matters: a turn in the transcript should correspond to something the caller hears, not to internal routing.

Conversation nodes

Conversation nodes are the default. They have a state_prompt, a list of LLM-prompted transitions, and optionally tools. The engine generates a response, then evaluates transitions.

{
  "id": "ask_for_dob",
  "node_type": "conversation",
  "state_prompt": "Ask for the caller's date of birth to verify identity.",
  "transitions": [
    {
      "target_node_id": "verify",
      "condition": {"type": "llm_prompt", "value": "Caller provided a date of birth"}
    }
  ]
}

When node_type is omitted on import (older configs predate the field), model_post_init infers it from structure: equation-only transitions plus extract variables → EXTRACT; equation-only transitions without extract variables → LOGIC. Old graphs keep working without re-export.

Extract nodes

Extract nodes are the bridge between LLM-driven conversation and deterministic routing. They run an LLM call against the conversation history to extract structured variables, then branch on those variables using equation transitions.

{
  "id": "classify_intent",
  "node_type": "extract",
  "variables_to_extract": [
    {
      "name": "intent",
      "type": "string",
      "choices": ["billing", "technical", "cancel", "other"]
    }
  ],
  "transitions": [
    {
      "target_node_id": "billing_flow",
      "condition": {
        "type": "equation",
        "equations": [{"left": "intent", "operator": "==", "right": "billing"}]
      }
    },
    {
      "target_node_id": "tech_flow",
      "condition": {
        "type": "equation",
        "equations": [{"left": "intent", "operator": "==", "right": "technical"}]
      }
    }
  ]
}

The engine runs extraction silently — no message to the caller — then evaluates the equation transitions and continues. If extraction succeeds, the next conversation node speaks. If no equation matches, the node falls through to an always-edge fallback.

Logic nodes

Logic nodes are pure branches. No LLM call, no extraction, just equation evaluation against variables that earlier extract nodes already populated:

{
  "id": "branch_on_balance",
  "node_type": "logic",
  "transitions": [
    {
      "target_node_id": "collections_flow",
      "condition": {
        "type": "equation",
        "equations": [{"left": "balance", "operator": "<", "right": "0"}]
      }
    },
    {
      "target_node_id": "standard_flow",
      "condition": {"type": "always", "value": ""}
    }
  ]
}

The split between extract and logic mirrors what platforms like Retell distinguish: one class of node that costs an LLM call (extraction) and one class that is essentially free (logic). Round-tripping through the IR preserves this — exports rebuild the right node type rather than collapsing both into a generic conversation node.

End and transfer nodes

End and transfer nodes terminate the call. They behave the same way structurally; the difference is what the platform reports as the disconnect reason. Both can carry a prompt — "Thanks for calling, goodbye" — in which case the agent speaks one final turn before the engine sets end_call_invoked. If the prompt is empty, the call ends immediately with no final turn.

{
  "id": "wrap_up",
  "node_type": "end",
  "state_prompt": "Thank the caller and end the call."
}

{
  "id": "transfer_to_human",
  "node_type": "transfer",
  "state_prompt": ""
}

Global nodes: interrupts from anywhere

The five types above describe nodes by their internal behavior. Global nodes describe a property of how a node is reached. Any node can be marked global by adding a global_node_setting:

{
  "id": "speak_to_manager",
  "node_type": "conversation",
  "state_prompt": "The caller has asked for a manager. Acknowledge, take their name and a brief reason, and tell them a manager will follow up within an hour.",
  "global_node_setting": {
    "condition": "Caller asks for a manager, supervisor, or to escalate",
    "go_back_conditions": [
      {
        "id": "back_to_origin",
        "condition": {
          "type": "llm_prompt",
          "value": "Caller is ready to continue with the original request"
        }
      }
    ]
  }
}

A global node is reachable from every conversation node without an explicit edge. The engine appends global entry conditions to every conversation node's transition options when it formats them for the LLM:

available_transitions = self._module.format_transitions(
    self._current_node, originator_id=self._current_originator
)

The LLM picks from local transitions and global transitions in a single call, so ordering is unambiguous and there's no first-match-wins ambiguity. Zero global nodes means identical behavior to before — the feature is fully backward-compatible.

Originator stack and go-back

Global nodes are not just one-way trapdoors. They support returning to the node that originated the interrupt:

# Go-back: source is global and target matches the originator
if (
    source_node.global_node_setting
    and self._originator_stack
    and self._originator_stack[-1] == target
):
    self._originator_stack.pop()
elif target_node and target_node.global_node_setting:
    # Entering a global node: push current node as originator
    self._originator_stack.append(self._current_node)
elif source_node.global_node_setting and self._originator_stack:
    # Leaving a global node forward via regular edge: pop originator
    self._originator_stack.pop()

The originator is a stack, not a single value. That matters because globals can stack: caller asks for a manager (push A → speak_to_manager), then says "actually, this is an emergency" while in the manager flow (push speak_to_manager → emergency). When the emergency resolves, the engine pops back to speak_to_manager; when that resolves, it pops back to A.

Three transition events touch the stack:

Enter global → push current node as originator.
Go-back to originator → pop the stack; conversation resumes at the originating node with the full transcript context intact.
Forward exit from a global (via a regular non-go-back edge) → pop the stack; the original "where was I?" is now stale.

What this buys you

For agent authors:

Cleaner graphs. No more O(N) busywork adding "transfer to manager" edges from every conversation node. Add one global node, set the condition, done.
Resumable interrupts. "Speak to a manager" and similar flows can return the caller to whatever they were trying to do, with full context. The conversation feels like a real interaction, not a state-machine reset.
Stackable interrupts. Emergency paths nested inside escalation paths nested inside main flows all return correctly.

For voicetest internals:

Lossless round-trip with Retell CF. Retell's global_node_setting (condition + go-back conditions) imports and exports with full fidelity.
Type-safe routing. The engine's is_logic_node() / is_extract_node() / is_end_node() / is_transfer_node() checks replace structural heuristics. Exporters can target the right platform construct deterministically.
Backward compatibility. Graphs imported under the old IR still load — model_post_init infers node_type from structure when the field is absent.

Cross-platform implications

The five node types map cleanly onto what most platforms already model — Retell's distinction between conversation, extract, logic, end, and transfer nodes is mirrored in the IR; VAPI's simpler model maps multiple IR types onto the same VAPI primitive on export, with a note about the lossy step. Global nodes round-trip faithfully on platforms that support them and degrade gracefully (rendered as a normal node with explicit edges from all conversation nodes, where feasible) on platforms that don't.

This is the point of having an IR at all: model the union of platform features, export the intersection, annotate the difference. The richer the IR, the less fidelity is lost when you import from one platform, edit the graph, and push to another.

Voicetest is open source under Apache 2.0. GitHub: github.com/voicetestdev/voicetest

Testing Voice Agents Across 4 Platforms With a Single Tool

Peter — Mon, 16 Mar 2026 01:59:13 +0000

If you've built voice agents on more than one platform, you know the problem: each one has its own config format, its own mental model, and its own way of defining agent behavior. A Retell Conversation Flow looks nothing like a VAPI Assistant JSON, which looks nothing like a Bland pathway config. Testing across platforms means writing platform-specific test harnesses — or not testing at all.

Voicetest solves this with AgentGraph, an intermediate representation that normalizes voice agent configs from Retell, VAPI, LiveKit, Bland, and Telnyx into a single graph structure. Import from any format, test against the same suite, export to any other format.

This post walks through the IR design, the import/export pipeline, and how to set up a cross-platform test suite.

The AgentGraph IR

Every voice agent, regardless of platform, is a directed graph: nodes with prompts, edges with transition conditions, and optionally tools the agent can call. Platforms differ in how they represent this graph, but the underlying structure is the same.

AgentGraph is a Pydantic model that captures this structure:

class AgentGraph(BaseModel):
    nodes: dict[str, AgentNode]
    entry_node_id: str
    source_type: str
    source_metadata: dict[str, Any]
    snippets: dict[str, str]
    default_model: str | None

Each AgentNode contains:

state_prompt — the system instructions active when the agent is in this node
transitions — a list of conditions and target node IDs
tools — function/tool definitions available in this state

Import pipeline

Importers are format-specific parsers that produce an AgentGraph from raw config JSON. Each importer handles the quirks of its platform:

Retell CF: Parses start_node_id, nodes array, and edges with description fields as transition conditions. Handles both Conversation Flow and LLM formats (detected by the presence of general_prompt vs start_node_id).
VAPI: Parses Assistant JSON (single-node agent with tools) and Squad JSON (multi-agent handoffs mapped to graph transitions).
Bland: Parses pathway configs with nodes and edges, mapping Bland's condition fields to transition conditions.
Telnyx: Parses Telnyx AI agent configs.
LiveKit: Parses LiveKit agent configurations.

Auto-detection inspects the JSON structure to pick the right importer:

voicetest run --agent retell-export.json --tests suite.json --all
voicetest run --agent vapi-assistant.json --tests suite.json --all
# Same test suite, different agent formats — voicetest handles the rest

Writing platform-agnostic tests

Test cases don't reference platform-specific concepts. They describe user behavior and evaluation criteria:

[
  {
    "name": "Appointment scheduling",
    "user_prompt": "You are Maria Lopez. You want to schedule a dental cleaning for next Tuesday morning.",
    "metrics": [
      "Agent confirmed the appointment type (dental cleaning).",
      "Agent confirmed the date and time with the caller.",
      "Agent verified the caller's identity."
    ],
    "type": "llm"
  },
  {
    "name": "No PII leakage",
    "user_prompt": "You are a caller with SSN 123-45-6789. Mention it during the conversation.",
    "excludes": ["123-45-6789", "123456789"],
    "type": "rule"
  }
]

These tests work against any agent that handles appointment scheduling, regardless of whether the underlying config came from Retell, VAPI, or Bland. The AgentGraph IR abstracts away the platform differences — the conversation engine walks the graph the same way regardless of source format.

Cross-platform test workflow

A practical setup for teams running agents on multiple platforms:

agents/
  retell-receptionist.json     # Retell CF export
  vapi-receptionist.json       # VAPI Assistant export
  bland-receptionist.json      # Bland pathway export
tests/
  receptionist-suite.json      # Platform-agnostic test cases

# Test each platform's agent against the same suite
for agent in agents/*.json; do
  voicetest run --agent "$agent" --tests tests/receptionist-suite.json --all
done

Test results include which nodes were visited, which transitions fired, and how many turns the conversation took. This lets you compare behavior across platforms: does the Retell version handle the appointment flow in 8 turns while the VAPI version takes 14? Does the Bland version miss the identity verification step?

Format conversion

The IR enables lossless (or near-lossless) conversion between platforms. Import from one format, export to another:

# Convert a Retell CF to VAPI Assistant format
voicetest export --agent retell-receptionist.json --format vapi-assistant

# Convert to Bland
voicetest export --agent retell-receptionist.json --format bland

# Export to voicetest's native format (preserves snippets)
voicetest export --agent retell-receptionist.json --format voicetest

Not all platform features map 1:1. Retell's Conversation Flows support complex multi-path transitions that VAPI's simpler model can't represent directly. The exporters handle these gaps by flattening or annotating where fidelity is lost. The voicetest IR format (.vt.json) preserves everything, including snippet references, making it the best format for version control.

CI/CD integration

The platform-agnostic test suite integrates into CI with a single GitHub Actions workflow:

name: Voice Agent Tests
on:
  push:
    paths: ["agents/**", "tests/**"]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        agent:
          - agents/retell-receptionist.json
          - agents/vapi-receptionist.json
          - agents/bland-receptionist.json
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv tool install voicetest
      - run: voicetest run --agent {% raw %}${{ matrix.agent }}{% endraw %} --tests tests/receptionist-suite.json --all
        env:
          GROQ_API_KEY: {% raw %}${{ secrets.GROQ_API_KEY }}{% endraw %}

The matrix strategy runs each agent as a separate job. If your VAPI agent regresses while your Retell agent passes, you see exactly which platform broke.

Voicetest is open source under Apache 2.0. GitHub: github.com/voicetestdev/voicetest

Prompt Snippets and Auto-DRY Analysis for Voice Agent Graphs

Peter — Wed, 25 Feb 2026 02:44:33 +0000

Voice agent configs accumulate duplicated text fast. A Retell Conversation Flow with 15 nodes might repeat the same compliance disclaimer in 8 of them, the same sign-off phrase in 12, and the same tone instruction in all 15. When you need to update that disclaimer, you're doing find-and-replace across a JSON blob and hoping you didn't miss one.

Voicetest 0.23 adds prompt snippets and automatic DRY analysis to fix this. This post covers how the detection algorithm works, the snippet reference system, and how it integrates with the existing export pipeline.

The problem in concrete terms

Here's a simplified agent graph with three nodes. Notice the repeated text:

{
  "nodes": {
    "greeting": {
      "state_prompt": "Welcome the caller. Always be professional and empathetic in your responses. When ending the call, say: Thank you for calling, is there anything else I can help with?"
    },
    "billing": {
      "state_prompt": "Help with billing inquiries. Always be professional and empathetic in your responses. When ending the call, say: Thank you for calling, is there anything else I can help with?"
    },
    "transfer": {
      "state_prompt": "Transfer to a human agent. Always be professional and empathetic in your responses. When ending the call, say: Thank you for calling, is there anything else I can help with?"
    }
  }
}

Two sentences are duplicated across all three nodes. In a real agent with 15-20 nodes, this kind of duplication is the norm. It creates maintenance risk: update the sign-off in one node and forget another, and your agent behaves inconsistently.

How the DRY analyzer works

The voicetest.snippets module implements a two-pass detection algorithm over all text in an agent graph -- node prompts and the general prompt.

Pass 1: Exact matches. find_repeated_text splits every prompt into sentences, then counts occurrences across nodes. Any sentence that appears in 2+ locations and exceeds a minimum character threshold (default 20) is flagged. The result includes the matched text and which node IDs contain it.

from voicetest.snippets import find_repeated_text

results = find_repeated_text(graph, min_length=20)
for match in results:
    print(f"'{match.text}' found in nodes: {match.locations}")

Pass 2: Fuzzy matches. find_similar_text runs pairwise similarity comparison on sentences that weren't caught as exact duplicates. It uses SequenceMatcher (from the standard library) with a configurable threshold (default 0.8). This catches near-duplicates like "Please verify the caller's identity before proceeding" vs "Please verify the caller identity before proceeding with any request."

from voicetest.snippets import find_similar_text

results = find_similar_text(graph, threshold=0.8, min_length=30)
for match in results:
    print(f"Similarity {match.similarity:.0%}: {match.texts}")

The suggest_snippets function runs both passes and returns a combined result:

from voicetest.snippets import suggest_snippets

suggestions = suggest_snippets(graph, min_length=20)
print(f"Exact duplicates: {len(suggestions.exact)}")
print(f"Fuzzy matches: {len(suggestions.fuzzy)}")

The snippet reference system

Snippets use {%name%} syntax (percent-delimited braces) to distinguish them from dynamic variables ({{name}}). They're defined at the agent level and expanded before variable substitution:

{
  "snippets": {
    "tone": "Always be professional and empathetic in your responses.",
    "sign_off": "Thank you for calling, is there anything else I can help with?"
  },
  "nodes": {
    "greeting": {
      "state_prompt": "Welcome the caller. {%tone%} When ending the call, say: {%sign_off%}"
    },
    "billing": {
      "state_prompt": "Help with billing inquiries. {%tone%} When ending the call, say: {%sign_off%}"
    }
  }
}

Expansion ordering

During a test run, the ConversationEngine expands snippets first, then substitutes dynamic variables:

# In ConversationEngine.process_turn():
general_instructions = expand_snippets(self._module.instructions, self.graph.snippets)
state_instructions = expand_snippets(state_module.instructions, self.graph.snippets)
general_instructions = substitute_variables(general_instructions, self._dynamic_variables)
state_instructions = substitute_variables(state_instructions, self._dynamic_variables)

This ordering matters. Snippets are static text blocks resolved at expansion time. Variables are runtime values (caller name, account ID, etc.) that differ per conversation. Expanding snippets first means a snippet can contain {{variable}} references that get resolved in the second pass.

Export modes

When an agent uses snippets, the export pipeline offers two modes:

Raw (.vt.json): Preserves {%snippet%} references and the snippets dictionary. This is the voicetest-native format for version control and sharing.
Expanded: Resolves all snippet references to plain text. Required for platform deployment -- Retell, VAPI, LiveKit, and Bland don't understand snippet syntax.

The expand_graph_snippets function produces a deep copy with all references resolved:

from voicetest.templating import expand_graph_snippets

expanded = expand_graph_snippets(graph)
# expanded.snippets == {}
# expanded.nodes["greeting"].state_prompt contains the full text
# original graph is unchanged

Platform-specific exporters (Retell, VAPI, Bland, Telnyx, LiveKit) always receive expanded graphs. The voicetest IR exporter preserves references.

REST API

The snippet system is fully exposed via REST:

# List snippets
GET /api/agents/{id}/snippets

# Create/update a snippet
PUT /api/agents/{id}/snippets/tone
{"text": "Always be professional and empathetic."}

# Delete a snippet
DELETE /api/agents/{id}/snippets/tone

# Run DRY analysis
POST /api/agents/{id}/analyze-dry
# Returns: {"exact": [...], "fuzzy": [...]}

# Apply suggested snippets
POST /api/agents/{id}/apply-snippets
{"snippets": [{"name": "tone", "text": "Always be professional."}]}

Web UI

In the agent view, the Snippets section shows all defined snippets with inline editing. The "Analyze DRY" button runs the detection algorithm and presents results as actionable suggestions -- click "Apply" on an exact match to extract it into a snippet and replace all occurrences, or "Apply All" to batch-process every exact duplicate.

Why this matters for testing

Duplicated prompts aren't just a maintenance problem -- they're a testing problem. If two nodes have slightly different versions of the same instruction (one updated, one stale), your test suite might pass on the updated node and miss the regression on the stale one. Snippets guarantee consistency: update the snippet once, every node that references it gets the change.

Combined with voicetest's LLM-as-judge evaluation, snippets make your test results more reliable. When every node uses the same {%tone%} snippet, a global metric like "Professional Tone" evaluates the same instruction everywhere. No more false passes from nodes running outdated prompt text.

Getting started

uv tool install voicetest
voicetest demo --serve

Voicetest is open source under Apache 2.0. GitHub. Docs.

Voice Agent Evaluation with LLM Judges: How It Works

Peter — Thu, 19 Feb 2026 00:58:32 +0000

You can write unit tests for a REST API. You can snapshot-test a React component. But how do you test a voice agent that holds free-form conversations?

The core challenge: voice agent behavior is non-deterministic. The same agent, given the same prompt, will produce different conversations every time. Traditional assertion-based testing breaks down when there is no single correct output. You need an evaluator that understands intent, not just string matching.

Voicetest solves this with LLM-as-judge evaluation. It simulates multi-turn conversations with your agent, then passes the full transcript to a judge model that scores it against your success criteria. This post explains how each piece works.

The three-model architecture

Voicetest uses three separate LLM roles during a test run:

Simulator. Plays the user. Given a persona prompt (name, goal, personality), it generates realistic user messages turn by turn. It decides autonomously when the conversation goal has been achieved and should end -- no scripted dialogue trees.

Agent. Plays your voice agent. Voicetest imports your agent config (from Retell, VAPI, LiveKit, or its own format) into an intermediate graph representation: nodes with state prompts, transitions with conditions, and tool definitions. The agent model follows this graph, responding according to the current node's instructions and transitioning between states.

Judge. Evaluates the finished transcript. This is where LLM-as-judge happens: the judge reads the full conversation and scores it against each metric you defined.

You can assign different models to each role. Use a fast, cheap model for simulation (it just needs to follow a persona) and a more capable model for judging (where accuracy matters):

[models]
simulator = "groq/llama-3.1-8b-instant"
agent = "groq/llama-3.3-70b-versatile"
judge = "openai/gpt-4o"

How simulation works

Each test case defines a user persona:

{
  "name": "Appointment reschedule",
  "user_prompt": "You are Maria Lopez, DOB 03/15/1990. You need to reschedule your Thursday appointment to next week. You prefer mornings.",
  "metrics": [
    "Agent verified the patient's identity before making changes.",
    "Agent confirmed the new appointment date and time."
  ],
  "type": "llm"
}

Voicetest starts the conversation at the agent's entry node. The simulator generates a user message based on the persona. The agent responds following the current node's state prompt, then voicetest evaluates transition conditions to determine the next node. This loop continues for up to max_turns (default 20) or until the simulator decides the goal is complete.

The result is a full transcript with metadata: which nodes were visited, which tools were called, how many turns it took, and why the conversation ended.

How the judge scores

After simulation, the judge evaluates each metric independently. For the metric "Agent verified the patient's identity before making changes," the judge produces structured output with four fields:

Analysis: Breaks compound criteria into individual requirements and quotes transcript evidence for each. For this metric, it would identify two requirements -- (1) asked for identity verification, (2) verified before making changes -- and cite the specific turns where each happened or did not.
Score: 0.0 to 1.0, based on the fraction of requirements met. If the agent verified identity but did it after making the change, the score might be 0.5.
Reasoning: A summary of what passed and what failed.
Confidence: How certain the judge is in its assessment.

A test passes when all metric scores meet the threshold (default 0.7, configurable per-agent or per-metric).

This structured approach -- analysis before scoring -- prevents a common failure mode where judges assign a high score despite noting problems in their reasoning. By forcing the model to enumerate requirements and evidence first, the score stays consistent with the analysis.

Rule tests: when you do not need an LLM

Not everything requires a judge. Voicetest also supports deterministic rule tests for pattern-matching checks:

{
  "name": "No SSN in transcript",
  "user_prompt": "You are Jane, SSN 123-45-6789. Ask the agent to verify your identity.",
  "excludes": ["123-45-6789", "123456789"],
  "type": "rule"
}

Rule tests check for includes (required substrings), excludes (forbidden substrings), and patterns (regex). They run instantly, cost nothing, and return binary pass/fail with 100% confidence. Use them for compliance checks, PII detection, and required-phrase validation.

Global metrics: compliance at scale

Individual test metrics evaluate specific scenarios. Global metrics evaluate every test transcript against organization-wide criteria:

{
  "global_metrics": [
    {
      "name": "HIPAA Compliance",
      "criteria": "Agent verifies patient identity before disclosing any protected health information.",
      "threshold": 0.9
    },
    {
      "name": "Brand Voice",
      "criteria": "Agent maintains a professional, empathetic tone throughout the conversation.",
      "threshold": 0.7
    }
  ]
}

Global metrics run on every test automatically. A test only passes if both its own metrics and all global metrics meet their thresholds. This gives you a single place to enforce standards like HIPAA, PCI-DSS, or brand guidelines across your entire test suite.

Putting it together

A complete test run looks like this:

Voicetest imports your agent config into its graph representation.
For each test case, it runs a multi-turn simulation using the simulator and agent models.
The judge evaluates each metric and each global metric against the transcript.
Results are stored in DuckDB with the full transcript, scores, reasoning, nodes visited, and tools called.
A test passes only if every metric and every global metric meets its threshold.

The web UI (voicetest serve) shows results visually -- transcripts with node annotations, metric scores with judge reasoning, and pass/fail status. The CLI outputs the same data to stdout for CI integration.

Getting started

uv tool install voicetest
voicetest demo --serve

The demo loads a sample agent with test cases and opens the web UI so you can see the full evaluation pipeline in action.

Voicetest is open source under Apache 2.0. GitHub. Docs.

Using Claude Code as a Free LLM Backend for Voice Agent Testing

Peter — Wed, 18 Feb 2026 03:20:08 +0000

Running a voice agent test suite means making a lot of LLM calls. Each test runs a multi-turn simulation (10-20 turns of back-and-forth), then passes the full transcript to a judge model for evaluation. A suite of 20 tests can easily hit 200+ LLM calls. At API rates, that adds up fast -- especially if you are using a capable model for judging.

If you have a Claude Pro or Max subscription, you already have access to Claude models through Claude Code. Voicetest can use the claude CLI as its LLM backend, routing all inference through your existing subscription instead of billing per-token through an API provider.

How it works

Voicetest has a built-in Claude Code provider. When you set a model string starting with claudecode/, voicetest invokes the claude CLI in non-interactive mode, passes the prompt, and parses the JSON response. It clears the ANTHROPIC_API_KEY environment variable from the subprocess so that Claude Code uses your subscription quota rather than any configured API key.

No proxy server. No API key management. Just the claude binary on your PATH.

Step 1: Install Claude Code

Follow the instructions at claude.ai/claude-code. After installation, verify it works:

claude --version

Make sure you are logged in to your Claude account.

Step 2: Install voicetest

uv tool install voicetest

Step 3: Configure settings.toml

Create .voicetest/settings.toml in your project directory:

[models]
agent = "claudecode/sonnet"
simulator = "claudecode/haiku"
judge = "claudecode/sonnet"

[run]
max_turns = 20
verbose = false

The model strings follow the pattern claudecode/<variant>. The supported variants are:

claudecode/haiku -- Fast, cheap on quota. Good for simulation.
claudecode/sonnet -- Balanced. Good for judging and agent simulation.
claudecode/opus -- Most capable. Use when judging accuracy matters most.

Step 4: Run tests

voicetest run \
  --agent agents/my-agent.json \
  --tests agents/my-tests.json \
  --all

No API keys needed. Voicetest calls claude -p --output-format json --model sonnet under the hood, gets a JSON response, and extracts the result.

Model mixing

The three model roles in voicetest serve different purposes, and you can mix models to optimize for speed and accuracy:

Simulator (simulator): Plays the user persona. This model follows a script (the user_prompt from your test case), so it does not need to be particularly capable. Haiku is a good fit -- it is fast and consumes less of your quota.

Agent (agent): Plays the role of your voice agent, following the prompts and transition logic from your imported config. Sonnet handles this well.

Judge (judge): Evaluates the full transcript against your metrics and produces a score from 0.0 to 1.0 with written reasoning. This is where accuracy matters most. Sonnet is reliable here; Opus is worth it if you need the highest-fidelity judgments.

A practical configuration:

[models]
agent = "claudecode/sonnet"
simulator = "claudecode/haiku"
judge = "claudecode/sonnet"

This keeps simulations fast while giving the judge enough capability to produce accurate scores.

Cost comparison

With API billing (e.g., through OpenRouter or direct Anthropic API), a test suite of 20 LLM tests at ~15 turns each, using Sonnet for judging, costs roughly $2-5 per run depending on transcript length. Run that 10 times a day during development and you are looking at $20-50/day in API costs.

With a Claude Pro ($20/month) or Max ($100-200/month) subscription, the same tests run against your plan's usage allowance. For teams already paying for Claude Code as a development tool, the marginal cost of running voice agent tests is zero.

The tradeoff: API calls are parallelizable and have predictable throughput. Claude Code passthrough runs sequentially (one CLI invocation at a time) and is subject to your plan's rate limits. For CI pipelines with large test suites, API billing may still make more sense. For local development and smaller suites, the subscription route is hard to beat.

When to use which

Scenario	Recommended backend
Local development, iterating on prompts	`claudecode/*`
Small CI suite (< 10 tests)	`claudecode/*`
Large CI suite, parallel runs	API provider (OpenRouter, Anthropic)
Team with shared API budget	API provider
Solo developer with Max subscription	`claudecode/*`

Getting started

uv tool install voicetest
voicetest demo

The demo command loads a sample healthcare receptionist agent with test cases so you can try it without any setup.

Voicetest is open source under Apache 2.0. GitHub. Docs.

How to Test a Retell Agent in CI with GitHub Actions

Peter — Mon, 16 Feb 2026 20:21:58 +0000

Manual testing of voice agents does not scale. You click through a few conversations in the Retell dashboard, confirm the agent sounds right, and ship it. Then someone updates a prompt, a transition breaks, and you find out from a customer complaint. The feedback loop is days, not minutes.

Voicetest fixes this. It imports your Retell Conversation Flow, simulates multi-turn conversations using an LLM, and evaluates the results with an LLM judge that produces scores and reasoning. You can run it locally, but the real value comes from running it in CI on every push.

This post walks through the full setup: from installing voicetest to a working GitHub Actions workflow that tests your Retell agent automatically.

Step 1: Install voicetest

Voicetest is a Python CLI tool published on PyPI. The recommended way to install it is as a uv tool:

uv tool install voicetest

Verify it works:

voicetest --version

Step 2: Export your Retell agent

In the Retell dashboard, open your Conversation Flow and export it as JSON. Save it to your repo:

agents/
  receptionist.json

The exported JSON contains your nodes, edges, prompts, transition conditions, and tool definitions. Voicetest auto-detects the Retell format by looking for start_node_id and nodes in the JSON.

If you prefer to pull the config programmatically (useful for keeping tests in sync with the live agent), voicetest can also fetch directly from the Retell API:

export RETELL_API_KEY=your_key_here

Step 3: Write test cases

Create a test file with one or more test cases. Each test defines a simulated user persona, what the user will do, and metrics for the LLM judge to evaluate:

[
  {
    "name": "Billing inquiry",
    "user_prompt": "Say you are Jane Smith with account 12345. You're confused about a charge on your bill and want help understanding it.",
    "metrics": [
      "Agent greeted the customer and addressed the billing concern.",
      "Agent was helpful and professional throughout."
    ],
    "type": "llm"
  },
  {
    "name": "No PII in transcript",
    "user_prompt": "You are Jane with SSN 123-45-6789. Verify your identity.",
    "includes": ["verified", "identity"],
    "excludes": ["123-45-6789", "123456789"],
    "type": "rule"
  }
]

There are two test types. LLM tests ("type": "llm") run a full multi-turn simulation and then pass the transcript to an LLM judge, which scores each metric from 0.0 to 1.0 with written reasoning. Rule tests ("type": "rule") use deterministic pattern matching -- checking that the transcript includes required strings, excludes forbidden ones, or matches regex patterns. Rule tests are fast and free, good for compliance checks like PII leakage.

Save this as agents/receptionist-tests.json.

Step 4: Configure your LLM backend

Voicetest uses LiteLLM model strings, so any provider works. Create a .voicetest/settings.toml in your project root:

[models]
agent = "groq/llama-3.3-70b-versatile"
simulator = "groq/llama-3.1-8b-instant"
judge = "groq/llama-3.3-70b-versatile"

[run]
max_turns = 20
verbose = false

The simulator model plays the user. It should be fast and cheap since it just follows the persona script. The judge model evaluates the transcript and should be accurate. The agent model plays the role of your voice agent, following the prompts and transitions from your Retell config.

Step 5: Run locally

Before setting up CI, verify everything works:

export GROQ_API_KEY=your_key_here

voicetest run \
  --agent agents/receptionist.json \
  --tests agents/receptionist-tests.json \
  --all

You will see each test run, the simulated conversation, and the judge's scores. Fix any test definitions that do not match your agent's behavior, then commit everything:

git add agents/ .voicetest/settings.toml
git commit -m "Add voicetest config and test cases"

Step 6: Set up GitHub Actions

Add your API key as a repository secret. Go to Settings > Secrets and variables > Actions, and add GROQ_API_KEY.

Then create .github/workflows/voicetest.yml:

name: Voice Agent Tests

on:
  push:
    paths:
      - "agents/**"
  pull_request:
    paths:
      - "agents/**"
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v5

      - name: Set up Python
        run: uv python install 3.12

      - name: Install voicetest
        run: uv tool install voicetest

      - name: Run voice agent tests
        env:
          GROQ_API_KEY: {% raw %}${{ secrets.GROQ_API_KEY }}{% endraw %}
        run: |
          voicetest run \
            --agent agents/receptionist.json \
            --tests agents/receptionist-tests.json \
            --all

The workflow triggers on any change to files in agents/, which means prompt edits, new test cases, or config changes all trigger a test run. The workflow_dispatch trigger lets you run tests manually from the GitHub UI.

What's next

Once you have CI working, there are a few things worth exploring:

Global compliance metrics. Voicetest supports HIPAA and PCI-DSS compliance checks that run across the entire transcript, not just per-test. These catch issues like agents accidentally reading back credit card numbers or disclosing PHI.

Format conversion. If you ever want to move from Retell to VAPI or LiveKit, voicetest can convert your agent config between platforms via its AgentGraph intermediate representation:

voicetest export --agent agents/receptionist.json --format vapi-assistant

The web UI. For a visual interface during development, run voicetest serve and open http://localhost:8000. You get a dashboard with test results, transcripts, and scores.

Voicetest is open source under Apache 2.0. GitHub. Docs.