DEV Community: Scarlett Attensil

Offline Evaluation of RAG-Grounded Answers in LaunchDarkly AI Configs

Scarlett Attensil — Thu, 16 Apr 2026 21:22:50 +0000

Overview

This tutorial shows you how to run an offline LLM evaluation on the RAG-grounded support agent you built in the Agent Graphs tutorial, using LaunchDarkly AI Configs, the Datasets feature, and built-in LLM-as-a-judge scoring. You'll build a RAG-grounded test dataset, run it through the Playground with a cross-family judge, and learn how to read each failing row as a dataset issue, an agent issue, or judge calibration noise.

Here's how it works. The LaunchDarkly Playground evaluates a single model call against a prompt and dataset you configure. By pre-computing your RAG retrieval offline and baking the chunks directly into each dataset row, you turn that call into a high-value generation test: the model in the Playground receives the same documentation context it would in production, so the eval measures how well your agent reasons over real grounded input.

What You'll Learn

Structure a RAG-grounded test dataset by pre-computing retrieval offline and bundling chunks into each row
Pick the right LLM judge for your agent's output shape (Accuracy for natural-language answers, Likeness for structured labels)
Avoid same-model bias by running the judge on a different model family than the agent
Diagnose failing rows as dataset issues, agent issues, or judge calibration noise

What this tutorial covers, and what it doesn't

Covers:

Generation quality over RAG context: does the model produce a correct answer when the right documentation is in the prompt?

Regression detection: catching unexpected score drops when you change a prompt or model

Variation selection: comparing candidate prompts and models before committing to a new AI Config variation

Does not cover:

Retrieval correctness. Whether your vector store is returning the best chunks is tested by your own RAG pipeline, outside LaunchDarkly.

End-to-end agent graph behavior. Tool execution, multi-turn conversations, handoffs, and multi-step routing require online evals against real production traffic.

Prerequisites

You've completed the Agent Graphs tutorial or have equivalent familiarity with LaunchDarkly AI Configs
You have the devrel-agents-tutorial repo cloned
You have API keys for two model providers, one for the agent under test and one for the judge (the examples use OpenAI and Anthropic)

Step 1: Get the Branch Running

About the branch and the Umbra knowledge base. The feature/offline-evals branch builds on the same Agent Graphs tutorial codebase and the routing, tool, and graph work done in earlier branches — none of that goes away. What this branch adds is a more realistic RAG assessment target: Umbra, a fictional serverless-functions product with an invented knowledge base (refund windows, deployment regions, function timeout limits, rate-limit tiers, and so on). Because Umbra doesn't exist outside this tutorial, the model under test has no pre-training knowledge to fall back on — a correct answer has to come from the retrieved chunks, which is the only way to honestly measure whether your RAG pipeline is doing its job. The branch also ships a pre-built RAG-grounded test dataset (datasets/answer-tests.csv) and a helper script that regenerates it from your vector store.

cd devrel-agents-tutorial
git checkout feature/offline-evals
cp .env.example .env
# Add LD_SDK_KEY, LD_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY to .env

uv sync
uv run python bootstrap/create_configs.py
uv run python initialize_embeddings.py

Start the API and UI in two terminals:

# Terminal 1
uv run uvicorn api.main:app --reload

# Terminal 2
uv run streamlit run ui/chat_interface.py

Open http://localhost:8501 and ask a question grounded in the Umbra docs (refund policy, deployment regions, function timeout). The agent pulls answers from the knowledge base.

Step 2: Understand the Test Dataset

Open datasets/answer-tests.csv. Every row has three fields:

input,expected_output,original_question
"Documentation context: --- We offer a 30-day refund policy for first-time subscribers... --- Annual subscriptions receive a prorated refund within... --- Question: What is the refund policy?","30-day refund policy for first-time subscribers who haven't deployed production traffic. Usage charges are non-refundable.","What is the refund policy?"

input bundles documentation chunks and the question into a single structured prompt, separated by --- dividers. The chunks were retrieved from your production vector store ahead of time by tools/build_rag_dataset.py, so the model in the Playground sees the same grounding the production agent would, even though the Playground never executes your retrieval tools.
expected_output is the correct answer, written by a human who read the source docs.
original_question is a plain-text copy of the question so you can scan the dataset without parsing the bundled prompt. No judge uses this field.

Regenerate the dataset when your knowledge base changes:

uv run python tools/build_rag_dataset.py

For the full reference on dataset format and limits, see Datasets for offline evaluations.

Step 3: Upload the Dataset

Use synthetic data only

Never upload real customer tickets, PII, secrets, or credentials. Replace anything sensitive with synthetic placeholders before upload. See the Playground privacy section for what gets forwarded to model providers.

Navigate to AI > Library in LaunchDarkly, select the Datasets tab, and click Upload dataset. Upload datasets/answer-tests.csv and name it answer-tests.

Step 4: Add Your Model API Keys

The Playground calls model providers directly, so it needs API keys for both the model running your agent and the model running your judge. These keys live in LaunchDarkly's "AI Config Test Run" integration, not in your AI Config.

In the Playground, click Manage API keys in the upper-right corner.
Click Add integration, pick a provider (e.g. OpenAI), paste your API key, accept the terms, and save.
Repeat for the second provider (Anthropic) so you can run a cross-family judge in Step 5.

See the Playground reference doc for the canonical instructions. API keys are stored per-session, so you may need to re-paste them when you return.

Step 5: Run the Evaluation

From the Datasets list, click into answer-tests to open it in a Playground bound to that dataset.

Configure the test

System prompt: paste your support-agent instructions verbatim from the AI Config. Do not edit or simplify them.
Agent model: pick the model your support-agent variation uses (or a candidate you're considering swapping to). To compare two candidates, run the eval twice with different agent models and compare scores.
Acceptance criteria: attach an Accuracy judge with threshold 0.85. Accuracy scores whether the response correctly addresses the input question, which fits grounded natural-language answers.
Evaluation model: uncheck Use same model for evaluation and set the judge to a different model family from the agent. Same-family judging tends to reward output patterns the judge itself produces. A cross-family judge gives you an independent read.

Run the eval.

Reading the results

The example run above had 18 passes and 2 failures. When a row fails, the failure comes from one of three places, and each one sends you in a different direction:

The dataset's chunks don't contain the answer. This is a retrieval problem, not a generation problem. Rebuild the dataset with higher top_k, a reranker, or a different chunker, or verify the answer is indexed at all.
The chunks contain the answer but the model ignored them. This is the agent-side failure offline evals are designed to catch. Tighten the system prompt to insist on grounding, or switch to a more obedient model.
The chunks and the model are both fine but the judge disagreed. This is judge calibration noise. Lower the threshold, try a different judge, or accept it as noise. Don't change your agent based on it.

Sort by score. For each failing row, open the bundled chunks in the input field and ask: was the right answer in there? Yes → fix the prompt or model. No → rebuild the dataset.

What failed in this run

Row 11: "What integrations are available?" (chunks missed the answer). The expected output mentioned monitoring integrations (Datadog, Sentry, LogRocket), but the retrieved chunks only covered databases, storage, and billing. The model correctly listed what it had and said "the documentation does not provide additional information regarding more integrations", which is the correct behavior for an ungrounded claim. Fix: higher top_k or a reranker in build_rag_dataset.py.

Row 12: "Can I get a refund on bandwidth overages?" (judge calibration). The model correctly said bandwidth overages are non-refundable, citing the docs, but omitted a secondary "Review your Usage Dashboard" recommendation from the expected output. Semantically right, lexically short one clause. Fix: lower the threshold or trim the expected output.

Two failures, two different fixes. Without reading the per-row results you'd conflate them and spend time tightening the model when the actual problem lives in the retriever or the dataset.

Where to Go From a Single Run

This tutorial walked you through one run. In practice, a single eval isn't where offline evaluation earns its keep. The real payoff comes from re-running the same dataset against a new prompt, a new model, or a fresh RAG chunker and comparing scores to your last known-good run. A small prompt edit that quietly drops your Accuracy from 0.83 to 0.71 is exactly the kind of regression this pattern is meant to catch, but only if you save the run and compare against it next time.

A reasonable next loop:

Save the run from Step 5 as your reference.
When you change something (prompt, model, chunker, top_k), re-run the same dataset and compare scores.
Add new rows to the dataset as you find failure modes in staging or production.

For end-to-end behavior that offline tests can't capture (tool execution, multi-turn conversations, the tail of real production inputs), see online evaluations and the When to add online evals tutorial. Online evaluations are not currently supported for agent-based AI Configs; for agent workflows, the documented path is programmatic judge evaluation via the AI SDK.

Step 7: Track Evaluation History

View saved runs at AI > Evaluations. Toggle Group by dataset to collapse runs under each dataset name so you can see the history for umbra-rag-eval alongside any other datasets in the project. Compare pass and fail counts across runs, and distinguish saved runs (indefinite retention) from one-off runs (60-day expiry). For metric definitions, see Monitor AI Configs.

What's Next

Progressive rollouts: release your winning variation to 5% of traffic, then 25%, then 100%, watching production metrics before expanding.
When to add online evals: decide what to score on live production traffic once you have an offline baseline.

For a deeper look at the multi-agent RAG system this tutorial builds on, see the Agent Graphs tutorial.

Building Framework-Agnostic AI Swarms: Compare LangGraph, Strands, and OpenAI Swarm

Scarlett Attensil — Thu, 26 Mar 2026 21:05:21 +0000

If you've ever run the same app in multiple environments, you know the pain of duplicated configuration. Agent swarms have the same problem: the moment you try multiple orchestrators (LangGraph, Strands, OpenAI Swarm), your agent definitions start living in different formats. Prompts drift. Model settings drift. A "small behavior tweak" turns into archaeology across repos.

AI behavior isn't code. Prompts aren't functions. They change too often, and too experimentally, to be hard-wired into orchestrator code. LaunchDarkly AI Configs lets you treat agent definitions like shared configuration instead. Define them once, store them centrally, and let any orchestrator fetch them. Update a prompt or model setting in the LaunchDarkly UI, and the new version rolls out without a redeploy.

Ready to build framework-agnostic AI swarms? Start your 14-day free trial of LaunchDarkly to follow along with this tutorial. No credit card required.

Start free trial →

The problem: Research gap analysis across multiple papers

When analyzing academic literature, researchers face a daunting task: reading dozens of papers to identify patterns, spot contradictions, and find unexplored opportunities. A single LLM call can summarize papers, but it produces a monolithic analysis you can't trace, refine, or trust for critical decisions.

The challenge compounds when you need to:

Identify methodological patterns across 12+ papers without missing subtle connections
Detect contradictory findings that might invalidate assumptions
Discover research gaps that represent genuine opportunities, not just oversight

This is where specialized agents excel - each focused on one aspect of the analysis, building on each other's work.

In this tutorial, we'll build a 3-agent research analysis swarm that solves this problem by dividing the work:

Agent	Role	Output
Approach Analyzer	Clusters methodological themes across papers	"Papers 1, 4, 7 use reinforcement learning; Papers 2, 5 use symbolic methods"
Contradiction Detector	Finds conflicting claims between papers	"Paper 3 claims X improves performance; Paper 8 shows X degrades it"
Gap Synthesizer	Identifies unexplored research directions	"No papers combine approach A with dataset B; potential opportunity"

We'll implement this swarm across three different orchestrators (LangGraph, Strands, and OpenAI Swarm), demonstrating how LaunchDarkly AI Configs enable:

Framework-agnostic agent definitions: Define agents once in LaunchDarkly, use them everywhere
Per-agent observability: Track tokens, latency, and costs for each agent individually - catch silent failures when agents skip execution
Dynamic swarm composition: Add/remove agents from the swarm or switch models without touching code

Why use a swarm?

Research gap analysis requires different skills: clustering methodological patterns, detecting contradictions, and synthesizing opportunities. With a swarm, each agent handles one aspect and produces artifacts the next agent builds on. You can track tokens, latency, and cost per agent. You can catch silent failures when an agent skips execution. And when something goes wrong, you know exactly where.

Technical requirements

Before implementing the swarm, ensure you have:

LaunchDarkly account with AI Configs enabled (see quickstart guide)
API keys for Anthropic Claude or OpenAI GPT-4 (check supported models)
Python 3.11+ for running orchestrators
Basic understanding of agent systems (review LangGraph agents tutorial if needed)

The complete implementation is available at GitHub - AI Orchestrators.

The architecture: how LaunchDarkly powers framework-agnostic swarms

The swarm architecture has three layers: dynamic agent configuration, per-agent tracking, and custom metrics for cost attribution. Here's how they work together.

The diagram shows LangGraph's implementation, but Strands and OpenAI Swarm follow the same pattern with their own handoff mechanisms. The key components are:

Configuration Fetch: The orchestrator queries LaunchDarkly's API to dynamically discover all agent configurations, avoiding hardcoded agent definitions
Agent Graph: Three specialized agents (Approach Analyzer, Contradiction Detector, Gap Synthesizer) connected through explicit handoff mechanisms
Metrics Collection: Each agent execution captures tokens, duration, and cost metrics through both the AI Config tracker and custom metrics API
Dual Dashboard Views: The same metrics appear in the AI Config Trends dashboard (for individual agent monitoring)

Three layers of framework-agnostic swarms

1. AI Config for Dynamic Agent Configuration

Each AI Config stores:

Agent key, display name, and model selection
System instructions and tool definitions

Your orchestrator code queries LaunchDarkly for "all enabled agent configs" and builds the swarm dynamically. No hardcoded agent names.

2. Per-Agent Tracking with AI SDK

LaunchDarkly's AI SDK provides tracking through config evaluations. You get a fresh tracker for each agent, then track tokens, duration, and success/failure. These metrics flow to the AI Config Monitoring dashboard automatically.

This tracking catches silent failures - when agents skip execution or produce minimal output. Step 4 shows the implementation patterns for each framework.

3. Custom Metrics for Cost Attribution

Per-agent tracking shows performance, but for cost comparisons across orchestrators you need custom metrics. These let you query by orchestrator, compare costs across frameworks, and identify anomalies.

With the architecture covered, let's build the swarm. We'll download research papers, set up the project, bootstrap agent configs in LaunchDarkly, implement per-agent tracking, and run the swarm across all three orchestrators.

Step 1: Download research papers

First, you need papers to analyze. The scripts/download_papers.py script queries ArXiv with narrow, category-specific searches to ensure focused results.

python scripts/download_papers.py

The script presents pre-configured narrow research topics:

# From orchestration/scripts/download_papers.py:164-189
topics = {
    "1": {
        "name": "Chain-of-thought prompting in LLMs",
        "query": "cat:cs.CL AND (chain-of-thought OR CoT) AND reasoning",
        "years": 2
    },
    "2": {
        "name": "Retrieval-augmented generation (RAG)",
        "query": "cat:cs.CL AND (retrieval-augmented OR RAG) AND generation",
        "years": 2
    },
    "3": {
        "name": "Emergent communication in multi-agent RL",
        "query": "cat:cs.MA AND (emergent communication OR language emergence)",
        "years": 5
    },
    "4": {
        "name": "Few-shot prompting for code generation",
        "query": "cat:cs.SE AND few-shot AND code generation",
        "years": 2
    },
    "5": {
        "name": "Vision-language model grounding",
        "query": "cat:cs.CV AND vision-language AND grounding",
        "years": 2
    }
}

These topics are intentionally narrow: Each uses ArXiv categories (cat:cs.CL, cat:cs.MA) to limit scope. Boolean AND operators ensure papers match all criteria. 2-5 year windows prevent overwhelming the analysis.

For even narrower custom queries, combine categories with specific techniques like cat:cs.CL AND chain-of-thought AND mathematical AND reasoning for CoT math only, cat:cs.MA AND emergent AND (referential OR compositional) for specific emergence types, or cat:cs.SE AND few-shot AND (Python OR JavaScript) AND test generation for language-specific code generation.

The script saves papers to data/gap_analysis_papers.json with this structure:

[
  {
    "id": "2409.02645v2",
    "title": "Emergent Language: A Survey and Taxonomy",
    "authors": "Jannik Peters, Constantin Waubert de Puiseau, ...",
    "published": "2024-09-04",
    "category": "cs.MA",
    "abstract": "The field of emergent language represents...",
    "introduction": "Language emergence has been explored...",
    "conclusion": "This paper provides a comprehensive review..."
  }
]

Why this format: Each paper includes ~2-3K characters of text (abstract + intro + conclusion), which is enough for analysis but won't overflow context windows. For 12 papers, you're looking at ~30K characters (~7.5K tokens) of input.

You now have 12 papers saved locally. Next, we'll configure LaunchDarkly credentials and install the orchestration frameworks.

Step 2: Set up your multi-orchestrator project

Environment setup

For help getting your SDK and API keys, see the API access tokens guide and SDK key management.

# .env file
LD_SDK_KEY=sdk-xxxxx       # Get from LaunchDarkly project settings
LD_API_KEY=api-xxxxx       # Create at Account settings → Authorization
LAUNCHDARKLY_PROJECT_KEY=orchestrator-agents

# Model API keys
ANTHROPIC_API_KEY=sk-ant-xxxxx
OPENAI_API_KEY=sk-xxxxx

Install dependencies

python -m venv .venv
source .venv/bin/activate

# LaunchDarkly SDKs - see [Python SDK docs](/sdk/server-side/python)
pip install ldai ldclient python-dotenv arxiv PyPDF2 requests

# Orchestration frameworks
pip install strands-sdk langgraph swarm

For more on the LaunchDarkly AI SDK, see the AI SDK documentation.

Your environment is configured and dependencies are installed. Next, we'll use the bootstrap script to automatically create all three agent configs in LaunchDarkly.

Step 3: Bootstrap agent configs with the manifest

The orchestration repo includes a complete bootstrap system that automatically creates all agent configurations, tools, and variations in LaunchDarkly. This is much faster and more reliable than manual setup.

Understanding the bootstrap system

The bootstrap process uses a YAML manifest to define:

Tools - Functions agents can call (fetch_paper_section, handoff_to_agent, etc.)
Agent Configs - Three specialized agents with their roles and instructions
Variations - Multiple model options (Anthropic Claude vs OpenAI GPT)
Targeting Rules - Which orchestrators get which models

Run the bootstrap script

# From the orchestration repo root
cd ai-orchestrators

# Run bootstrap with the research gap manifest
python scripts/launchdarkly/bootstrap.py

# You'll see:
╔═══════════════════════════════════════════════════════╗
║  AI Agent Orchestrator - LaunchDarkly Bootstrap       ║
╚═══════════════════════════════════════════════════════╝

Available manifests:
  1. Research Gap Analysis (research_gap_manifest.yaml)

Select manifest or press Enter for default: [Enter]

📦 Project: orchestrator-agents
🌍 Environment: production

🛠️  Creating paper analysis tools...
    ✓ Tool 'extract_key_sections' created
    ✓ Tool 'fetch_paper_section' created
    ✓ Tool 'handoff_to_agent' created
    ...

🤖 Creating AI agent configs...
    ✓ AI Config 'approach-analyzer' created
    ✓ AI Config 'contradiction-detector' created
    ✓ AI Config 'gap-synthesizer' created

✨ Bootstrap complete!

What gets created

The bootstrap script creates the three agents described earlier (Approach Analyzer, Contradiction Detector, Gap Synthesizer), each with swarm-aware instructions and handoff tools.

Verify in LaunchDarkly dashboard

After bootstrap completes:

Go to your LaunchDarkly AI Configs dashboard at https://app.launchdarkly.com/<your-project-key>/<your-environment-key>/ai-configs
You'll see all three agent configs created
Each config has:
- Two variations (Claude and OpenAI models)
- Proper tools configured
- Detailed swarm-aware instructions
- Targeting rules for orchestrator-specific routing

How variations and targeting work

Each agent has two variations in the manifest:

# Example from approach-analyzer agent
variations:
  - key: "analyzer-claude"
    name: "Approach Analyzer Claude"
    modelConfig:
      provider: "anthropic"
      modelId: "claude-sonnet-4-5"
    tools: ["handoff_to_agent", "cluster_approaches"]
    instructions: |
      [Agent instructions here]

  - key: "analyzer-openai"
    name: "Approach Analyzer OpenAI"
    modelConfig:
      provider: "openai"
      modelId: "gpt-5"
    tools: ["handoff_to_agent", "cluster_approaches"]
    instructions: |
      [Same instructions, different model]

targeting:
  rules:
    - variation: "analyzer-openai"
      clauses:
        - attribute: "orchestrator"
          op: "in"
          values: ["openai_swarm", "openai-swarm"]
  defaultVariation: "analyzer-claude"

When an orchestrator requests this agent:

Context includes orchestrator attribute: context = create_context(execution_id, orchestrator="openai_swarm")
LaunchDarkly evaluates targeting rules: If orchestrator is "openai_swarm" or "openai-swarm", use OpenAI variation
Otherwise use default: Claude variation for all other orchestrators

This lets you:

Use OpenAI models when running OpenAI Swarm (native compatibility)
Use Claude for other orchestrators
A/B test models by adjusting targeting rules

Customize agent behavior

After bootstrap, you can adjust agents in the LaunchDarkly UI without code changes. Switch between Claude, GPT-4, or other supported providers. Refine instructions for better handoffs. Control which agents are included in the swarm through targeting rules. Test different prompts or models side-by-side with experiments.

Your three agents are now configured in LaunchDarkly. Next, we'll implement tracking so you can monitor tokens, latency, and cost for each agent individually.

Step 4: Implement per-agent tracking

The orchestration repository demonstrates per-agent tracking across all three frameworks. First, you need to fetch agent configurations from LaunchDarkly:

Fetching agent configurations dynamically

from shared.launchdarkly import (
    init_launchdarkly_clients,
    fetch_agent_configs_from_api,
    create_context,
    build_agent_requests
)

# Initialize LaunchDarkly clients
ld_client, ai_client = init_launchdarkly_clients()

# Fetch agent list from LaunchDarkly API (not hardcoded!)
items = fetch_agent_configs_from_api()
print(f"Found {len(items)} AI config(s) in LaunchDarkly")

# Create execution context
execution_id = f"langgraph-{datetime.now().strftime('%Y%m%d_%H%M%S')}"
context = create_context(execution_id, orchestrator="langgraph")

# Build requests for all agents
agent_requests, agent_metadata = build_agent_requests(items)

# Fetch all configs in one call
configs = ai_client.agent_configs(agent_requests, context)

# Process agents with configured variations
enabled_agents = []
for item in items:
    config = configs.get(item["key"])
    if config and config.enabled:
        enabled_agents.append({
            "key": item["key"],
            "name": item["name"],
            "config": config,
            "model": config.model.name if config.model else "claude-sonnet-4-5"
        })

print(f"✓ Found {len(enabled_agents)} configured agent configs")

Pattern 1: Native framework metrics (Strands)

Strands provides accumulated_usage on each node result after execution:

# From orchestrators/strands/run_gap_analysis.py:418-424
if agent_key in per_agent_metrics:
    usage = node_result.accumulated_usage or {}
    input_tokens, output_tokens = extract_usage_tokens(usage)
    total_tokens = input_tokens + output_tokens

View full Strands implementation

Pattern 2: Message-based tracking (LangGraph)

LangGraph attaches usage_metadata to messages, requiring post-execution iteration:

# From orchestrators/langgraph/run_gap_analysis.py:442-446
if hasattr(msg, "usage_metadata") and msg.usage_metadata:
    usage_data = msg.usage_metadata
    input_tokens = usage_data.get("input_tokens", 0) or usage_data.get("prompt_tokens", 0)
    output_tokens = usage_data.get("output_tokens", 0) or usage_data.get("completion_tokens", 0)
    has_usage = True

View full LangGraph implementation

Pattern 3: Interception-based tracking (OpenAI Swarm)

OpenAI Swarm doesn't aggregate per-agent metrics, requiring interception of completion calls:

# From orchestrators/openai_swarm/run_gap_analysis.py:369-387
original_get_chat_completion = client.get_chat_completion

def tracked_get_chat_completion(agent, history, context_variables, model_override, stream, debug):
    start_call = time.time()
    completion = original_get_chat_completion(
        agent=agent,
        history=history,
        context_variables=context_variables,
        model_override=model_override,
        stream=stream,
        debug=debug,
    )
    duration = time.time() - start_call
    agent_key = key_by_name.get(agent.name, agent.name)
    usage = getattr(completion, "usage", None)
    if usage:
        input_tokens = int(getattr(usage, "prompt_tokens", 0))
        output_tokens = int(getattr(usage, "completion_tokens", 0))
        total_tokens = int(getattr(usage, "total_tokens", input_tokens + output_tokens))

View full OpenAI Swarm implementation

Critical: Provider token field names differ

Each provider uses different field names: Anthropic uses input_tokens/output_tokens, OpenAI uses prompt_tokens/completion_tokens, and some frameworks use camelCase (inputTokens). The implementations use fallback chains to handle all formats.

You can now capture tokens, latency, and cost for each agent. Next, we'll run the swarm across LangGraph, Strands, and OpenAI Swarm to see how they perform with the same agent definitions.

Step 5: Run multiple orchestrators and track results

The repository includes scripts to run all three orchestrators and analyze their performance:

# Run all orchestrators 5 times each
./scripts/run_swarm_benchmark.sh sequential 5

# Analyze the results
python scripts/analyze_benchmark_results.py

Configure env: Create .env with SDK keys
Install deps: pip install -r requirements.txt
Download papers: python scripts/download_papers.py
Bootstrap agents: python scripts/launchdarkly/bootstrap.py
Configure targeting: Set default variation for each agent in LaunchDarkly UI
Test run: python orchestrators/strands/run_gap_analysis.py

Troubleshooting: If you see "No enabled agents found," check that each agent has a default variation set in the Targeting tab.

Now that you've run the swarm across all three orchestrators, let's look at how they differ in approach and performance.

Comparing orchestrator approaches to swarms

All three frameworks support multi-agent workflows, they just disagree on who decides what happens next.

Key differences

Aspect	Strands	LangGraph	OpenAI Swarm
Routing	Framework-managed	Graph-based	Function return
Handoff API	Tool call (automatic)	Command object	Return Agent object
Boilerplate	Low	Medium	Medium
Control	Low (black box)	High (explicit graph)	High (manual impl)
Debugging	Hard (why didn't agent run?)	Easy (graph trace)	Hard (silent failures)
Per-Agent Metrics	Built-in	Wrapper required	Interception required

View full implementations: Strands | LangGraph | OpenAI Swarm

The LaunchDarkly advantage: By defining agents externally, you can implement swarms across all three frameworks and compare their approaches with the same agent definitions.

Performance comparison (9 runs: 3 datasets × 3 orchestrators)

Metric	OpenAI Swarm	Strands	LangGraph
Avg Time	2.9 min	5.7 min	8.0 min
Tokens	67K	99K	89K
Speed	385 tok/s	287 tok/s	186 tok/s
Report Size	13KB	32KB	67KB
Variance	±1.05 min	±1.38 min	±0.21 min

Key insight (based on limited sample): Fastest ≠ best. OpenAI Swarm was 3x faster but produced reports 80% smaller than LangGraph. LangGraph had the lowest variance and most comprehensive outputs despite slower execution.

Example reports: See the outputs

LangGraph (60-70KB): Emergent | Theorem | Self-Improvement
Strands (30-35KB): Emergent | Theorem | Self-Improvement
OpenAI Swarm (10-15KB): Emergent | Theorem | Self-Improvement

Report size variation demonstrates why per-agent tracking matters - you need to know when agents produce minimal output.

Conclusion

The orchestrator you choose determines how agents coordinate, but it shouldn't lock you into a single framework. By defining agents in LaunchDarkly and fetching them at runtime, you can run the same swarm across LangGraph, Strands, and OpenAI Swarm without duplicating configuration or watching prompts drift between repos.

The performance differences are real. OpenAI Swarm is fastest, LangGraph produces the most comprehensive outputs, and Strands offers the simplest setup. But you only discover these tradeoffs if you can track each agent individually and catch silent failures when they happen.

Swarms cost more than single LLM calls. The payoff is traceable reasoning you can audit, refine, and trust.

The full implementation is available on GitHub - AI Orchestrators. Clone the repo and run the same swarm across all three orchestrators. To get started with LaunchDarkly AI Configs, follow the quickstart guide.

Build AI Configs with Agent Skills in Claude Code, Cursor, or Windsurf

Scarlett Attensil — Thu, 26 Mar 2026 18:18:43 +0000

LaunchDarkly Agent Skills let you build AI Configs by describing what you want. Tell your coding assistant to create an agent, and it handles the API calls, targeting rules, and tool definitions for you.

In this quickstart, you'll create AI Configs using natural language, then run a sample LangGraph app that consumes them. You'll build a "Side Project Launcher"—a three-agent pipeline that validates ideas, writes landing pages, and recommends tech stacks.

Prefer video? Watch Build a multi-agent system with LaunchDarkly Agent Skills for a walkthrough of this tutorial.

What you'll build

A three-agent pipeline called "Side Project Launcher":

Idea Validator: researches competitors, analyzes market gaps, scores viability
Landing Page Writer: generates headlines, copy, and CTAs based on your value prop
Tech Stack Advisor: recommends frameworks, databases, and hosting based on your requirements

By the end, you'll have working AI Configs in LaunchDarkly and a sample app that fetches them at runtime.

Prerequisites

LaunchDarkly account (free trial works)
Claude Code, Cursor, or Windsurf installed
LaunchDarkly API access token (for creating configs)
Anthropic API key (for running the sample app)

LaunchDarkly API access token (LD_API_KEY): Used by Agent Skills to create projects and AI Configs. Get it from Authorization settings. Requires writer role or custom role with createProject and createAIConfig permissions.
LaunchDarkly SDK key (LAUNCHDARKLY_SDK_KEY): Used by your app at runtime to fetch AI Configs. Found in your project's SDK settings after creation.
Model provider API key (e.g., ANTHROPIC_API_KEY): Used to call the model. Get it from your provider (Anthropic, OpenAI, etc.).

Store all keys in .env and never commit them to version control.

Want to follow along? Start your 14-day free trial of LaunchDarkly. No credit card required.

30-second quickstart

If you just want to get started, here's the fastest path:

1. Install skills:

npx skills add launchdarkly/agent-skills

Or ask your editor: "Download and install skills from https://github.com/launchdarkly/agent-skills"

Restart your editor after installing.

2. Set your token:

export LD_API_KEY="api-xxxxx"

3. Build something:

Use the prompt in Build a multi-agent project below, or describe your own agents. The assistant creates everything and gives you links to view them in LaunchDarkly.

Install Agent Skills in Claude Code, Cursor, or Windsurf

Agent Skills work with any editor that supports the Agent Skills specification.

Step 1: Install the skills

You have two options:

Option A: Use skills.sh (recommended)

skills.sh is an open directory for agent skills. Install LaunchDarkly skills with one command:

npx skills add launchdarkly/agent-skills

Option B: Ask your AI assistant

Open your editor and ask:

Download and install skills from https://github.com/launchdarkly/agent-skills

Both methods install the same skills.

Step 2: Restart your editor

Close and reopen your editor. The skills load on startup.

How to verify: Type /aiconfig in Claude Code. You should see autocomplete suggestions. In Cursor, ask "what LaunchDarkly skills do you have?" and the assistant should list them.

Step 3: Set your API token

export LD_API_KEY="api-xxxxx"

Get your token from LaunchDarkly Authorization settings. The writer role works, or use a custom role with createProject and createAIConfig permissions.

Build a multi-agent project

Now let's build something real: a Side Project Launcher that helps you validate ideas, write landing pages, and pick the right tech stack. Tell the assistant:

Create AI Configs for a "Side Project Launcher" with three configs.
Use Anthropic Claude models for all configs.

1. idea-validator: Analyzes startup ideas by researching competitors, estimating
   market size, and scoring viability. Use variables for {{idea}}, {{target_audience}},
   and {{problem_statement}}. Give it tools for web search and competitor analysis.

2. landing-page-writer: Generates compelling headlines, value props, and CTAs
   based on {{idea}}, {{target_audience}}, and {{unique_value_prop}}.
   Give it tools for copy generation and A/B test suggestions.

3. tech-stack-advisor: Recommends frameworks, databases, and hosting based on
   {{expected_users}}, {{budget}}, and {{team_expertise}}. Give it a tool for
   stack recommendations.

Put them in a new project called side-project-launcher.

What the assistant creates

The assistant uses several skills automatically:

aiconfig-projects: creates the LaunchDarkly project
aiconfig-create: builds each agent configuration with variables
aiconfig-tools: defines tools for function calling

Expected output:

Creating project: side-project-launcher
Creating AI Config: idea-validator
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: idea, target_audience, problem_statement
  - Instructions: "Validate the idea: {{idea}}. Research competitors targeting
    {{target_audience}} who have {{problem_statement}}..."
  - Tools: web_search, competitor_analysis
Creating AI Config: landing-page-writer
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: idea, target_audience, unique_value_prop
  - Instructions: "Write landing page copy for {{idea}}. The target audience is
    {{target_audience}}. Lead with: {{unique_value_prop}}..."
  - Tools: generate_copy, suggest_ab_tests
Creating AI Config: tech-stack-advisor
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: expected_users, budget, team_expertise
  - Instructions: "Recommend a tech stack for {{expected_users}} users,
    {{budget}} budget, team knows {{team_expertise}}..."
  - Tools: recommend_stack

Done! View your project:
https://app.launchdarkly.com/side-project-launcher/production/ai-configs

The variables ({{idea}}, {{target_audience}}, etc.) get filled in at runtime when you call the SDK. That's how each user gets personalized output.

What it looks like in LaunchDarkly

After creation, your LaunchDarkly project contains:

3 AI Configs with instructions, model settings, and variables
3 tools with parameter definitions ready for function calling
Default targeting serving the configuration to all users

Each agent has its own configuration with instructions, variables, and tools. Here's the idea-validator:

The landing-page-writer and tech-stack-advisor follow the same pattern with their own instructions and tools.

Run the Side Project Launcher

The full working code is available on GitHub: launchdarkly-labs/side-project-researcher

Clone it and run:

git clone https://github.com/launchdarkly-labs/side-project-researcher.git
cd side-project-researcher
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your SDK key and Anthropic API key
python side_project_launcher_langgraph.py

You'll need both the LaunchDarkly SDK key (from your project's SDK settings) and your Anthropic API key in the .env file. The assistant can surface the SDK key from your project details, but store it in .env rather than hardcoding it.

The app prompts you for your idea details:

Then each agent runs in sequence, fetching its config from LaunchDarkly and generating output:

Connect to your framework

The AI Config stores your model, instructions, and tools. The SDK fetches the config and handles variable substitution automatically.

The snippets below show the integration pattern. They omit imports, error handling, and tool wiring for brevity. For complete, runnable code, use the sample repo.

Initialize the SDK

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AIAgentConfigDefault

# Initialize once at startup
SDK_KEY = os.environ.get('LAUNCHDARKLY_SDK_KEY')
ldclient.set_config(Config(SDK_KEY))
ld_client = ldclient.get()
ai_client = LDAIClient(ld_client)

Fetch agent configs

def build_context(user_id: str, **attributes):
    """Build LaunchDarkly context for targeting."""
    builder = Context.builder(user_id)
    for key, value in attributes.items():
        builder.set(key, value)
    return builder.build()

def get_agent_config(config_key: str, context: Context, variables: dict = None):
    """Get agent-mode AI Config from LaunchDarkly."""
    fallback = AIAgentConfigDefault(enabled=False)
    return ai_client.agent_config(config_key, context, fallback, variables or {})

Wire it to LangGraph

LangGraph orchestrates multi-agent workflows as a graph of nodes, but you can use any orchestrator—CrewAI, LlamaIndex, Bedrock AgentCore, or custom code. To compare options, read Compare AI orchestrators.

By wiring AI Configs to each node, your agents fetch their model, instructions, and tools dynamically from LaunchDarkly. This lets you swap models within a provider (e.g., Sonnet to Haiku), update prompts, or disable agents without redeploying.

The AI Config defines tool schemas, but your code must implement the actual tool handlers. The sample repo shows how to bind config.tools to LangChain tool functions. For this tutorial, the tools are defined but not wired—the agents respond based on their instructions alone.

Each agent becomes a node in your graph:

from langchain_anthropic import ChatAnthropic
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import StateGraph, END

def idea_validator_node(state: SideProjectState) -> SideProjectState:
    context = build_context(state["user_id"])
    config = get_agent_config("idea-validator", context, {
        "idea": state["idea"],
        "target_audience": state["target_audience"],
        "problem_statement": state["problem_statement"]
    })

    if config.enabled:
        llm = ChatAnthropic(model=config.model.name)
        messages = [
            SystemMessage(content=config.instructions),
            HumanMessage(content="Please validate this idea and provide your analysis.")
        ]
        response = llm.invoke(messages)
        state["idea_validation"] = response.content
        config.tracker.track_success()  # Track metrics

    return state

# Build the graph
workflow = StateGraph(SideProjectState)
workflow.add_node("validate_idea", idea_validator_node)
workflow.add_node("write_landing_page", landing_page_writer_node)
workflow.add_node("recommend_stack", tech_stack_advisor_node)

workflow.set_entry_point("validate_idea")
workflow.add_edge("validate_idea", "write_landing_page")
workflow.add_edge("write_landing_page", "recommend_stack")
workflow.add_edge("recommend_stack", END)

app = workflow.compile()

# Don't forget to flush before exiting
ld_client.flush()

To see a full example running across LangGraph, Strands, and OpenAI Swarm, read Compare AI orchestrators.

What you can do next

Once your agents are in LaunchDarkly:

A/B test variations: split traffic between prompt variations or model sizes (e.g., Sonnet vs Haiku) to see which performs better
Target by segment: premium users get one variation, free users get another
Kill switch: disable a misbehaving agent instantly from the UI
Track costs: monitor tokens and latency per variation

To learn more about targeting and experimentation, read AI Configs Best Practices.

Troubleshooting

Skills installed but not working: Restart your editor after installing skills. They load on startup.

"Permission denied" errors: Check that your API token has createProject and createAIConfig permissions. The writer role includes both.

Config comes back disabled: Your targeting rules may not match the context you're passing. Check that default targeting is enabled, or that your context attributes match your rules.

Tools defined but not executing: The AI Config defines tool schemas, but your code must implement handlers. See the sample repo for tool binding examples.

Can't find SDK key: After Agent Skills creates your project, find the SDK key in your project's Settings > Environments > SDK key. Copy it to your .env file.

FAQ

Do I need Claude Code, or does this work in Cursor/Windsurf?

Agent Skills work in any editor that supports the Agent Skills specification. This includes Claude Code, Cursor, and Windsurf. The installation process is the same.

What's the difference between Agent Skills and the MCP server?

Both give your AI assistant access to LaunchDarkly. Agent Skills are text-based playbooks that teach the assistant workflows. The MCP server exposes LaunchDarkly's API as tools. You can use either or both.

What permissions does my API token need?

The writer role works, or use a custom role with createProject and createAIConfig permissions.

Where do I see the created AI Configs?

In the LaunchDarkly UI: go to your project, then AI Configs in the left sidebar. Each config shows its instructions, model, tools, and targeting rules.

How do I delete or reset generated configs?

In the LaunchDarkly UI, open the AI Config and click Archive (or Delete if available). Or ask the assistant: "Delete the AI Config called researcher-agent in project valentines-day."

Can I use this with frameworks other than LangGraph?

Yes. The SDK returns model name, instructions, and tools as data. You wire that into whatever framework you use: CrewAI, LlamaIndex, Bedrock AgentCore, or custom code.

Does this work for completion mode (chat) or just agent mode?

Both. Use ai_client.completion_config() for completion mode (chat with message arrays) or ai_client.agent_config() for agent mode (instructions for multi-step workflows). To learn more, read Agent mode vs completion mode.

Next steps

Read the Python AI SDK Reference for detailed SDK usage
Try building a data extraction pipeline to deploy AI Configs with Vercel

Evaluate LLM code generation with LLM-as-judge evaluators

Scarlett Attensil — Thu, 26 Mar 2026 16:58:55 +0000

Which AI model writes the best code for your codebase? Not "best" in general, but best for your security requirements, your API schemas, and your team's blind spots.

This tutorial shows you how to score every code generation response against custom criteria you define. You'll set up custom judges that check for the vulnerabilities you actually care about, validate against your real API conventions, and flag the scope creep patterns your team keeps running into. After a few weeks of data, you'll have evidence to choose which model to use for which tasks.

What you will build

In this tutorial you build a proxy server that routes Claude Code requests through LaunchDarkly. You can forward requests to any model: Anthropic, OpenAI, Mistral, or local Ollama instances. Every response gets scored by custom judges you create.

You will build three judges:

Security: Checks for SQL injection, XSS, hardcoded secrets, and the specific vulnerabilities you care about
API contract: Validates code against your schema conventions
Minimal change: Flags scope creep and unnecessary modifications

After setup, you use Claude Code normally, and scores flow to the LaunchDarkly Monitoring dashboard automatically. Over time, you build a dataset grounded in your actual usage: maybe Sonnet scores consistently higher on security, but Opus handles API contract adherence better on complex endpoints. That's the kind of answer a generic benchmark can't give you.

To learn more, read Online evaluations or watch the Introducing Judges video tutorial.

Prerequisites

LaunchDarkly account with AI Configs enabled
Python 3.9+
LaunchDarkly Python AI SDK v0.14.0+ (launchdarkly-server-sdk-ai)
API keys for your model providers
Claude Code installed

How the proxy works

This proxy implements a minimal Anthropic Messages-style gateway for text-only code generation and automatic quality scoring.

When Claude Code sends a request to POST /v1/messages, the proxy:

Extracts text-only prompts. It converts the Anthropic Messages body into LaunchDarkly LDMessages, keeping only text content. It ignores tool blocks, images, and other non-text content.
Routes the request through LaunchDarkly AI Configs. The proxy creates a context with a selectedModel attribute. Your model-selector AI Config uses targeting rules on this attribute to pick the right model variation.
Invokes the model and triggers judges. The proxy calls chat.invoke(). If the selected variation has judges attached, the SDK schedules judge evaluations automatically based on your sampling rate. Scores flow to LaunchDarkly Monitoring.
Returns a standard Messages response. The proxy sends back the assistant response as a single text block, plus basic token usage if available.

Claude Code talks to a local /v1/messages endpoint. LaunchDarkly handles model selection and online evaluations behind the scenes.

Create the AI Config and judges

You can use the LaunchDarkly dashboard or Claude Code with agent skills. Agent skills are faster if you have them installed.¹

Option A: Agent skills

Create the project:

/aiconfig-projects Create a project called "custom-evals-claude-code"

Create the model selector:

/aiconfig-create

Create a completion mode AI Config:
- Key: model-selector
- Name: Model Selector
- Project: custom-evals-claude-code

Three variations (empty messages, this is a router):
1. "sonnet" - Anthropic claude-sonnet-4-6
2. "opus" - Anthropic claude-opus-4-6
3. "mistral" - Mistral mistral-large@2407

Create the security judge:

/aiconfig-create

Create a judge AI Config with:
- Key: security-judge
- Name: Security Judge
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:security

System prompt:
"You are a security auditor evaluating AI-generated code for vulnerabilities.

Analyze the assistant's response and score it from 0.0 to 1.0:

SCORING CRITERIA:
- 1.0: No security issues detected. Code follows security best practices.
- 0.7-0.9: Minor issues that pose low risk.
- 0.4-0.6: Moderate issues requiring attention.
- 0.1-0.3: Serious vulnerabilities present (SQL injection, XSS, command injection).
- 0.0: Critical vulnerabilities that could lead to immediate compromise.

CHECK FOR:
- Injection flaws (SQL, command, LDAP)
- Cross-site scripting (XSS)
- Hardcoded secrets or credentials
- Insecure file operations
- Missing input validation

If no code is present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Create the API contract judge:

/aiconfig-create

Create a judge AI Config with:
- Key: api-contract-judge
- Name: API Contract Adherence
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:api-contract-adherence

System prompt:
"You are an API contract auditor. Evaluate whether AI-generated code adheres to the API schema.

SCORING CRITERIA:
- 1.0: Code fully complies with expected patterns.
- 0.5: Partial adherence with minor deviations.
- 0.0: Invalid format or significant violations.

If no API code is present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Create the minimal change judge:

/aiconfig-create

Create a judge AI Config with:
- Key: minimal-change-judge
- Name: Minimal Change Judge
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:minimal-change

System prompt:
"You are a code review auditor focused on change scope. Evaluate whether the AI assistant made only necessary changes.

SCORING CRITERIA:
- 1.0: Changes are precisely scoped to the request. No unnecessary modifications.
- 0.5: Some unnecessary additions (reformatting unrelated code, extra comments).
- 0.0: Significant scope creep (rewriting large sections, architectural changes not requested).

FLAG THESE UNNECESSARY CHANGES:
- Reformatting code not part of the request
- Adding type annotations to unchanged functions
- Inserting unrequested comments or docstrings
- Renaming variables outside the scope of the fix

If no code changes present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Attach judges to the model selector:

/aiconfig-online-evals

Attach to all model-selector variations at 100% sampling:
- security-judge
- api-contract-judge
- minimal-change-judge

Set up targeting:

For each AI Config, go to the Targeting tab and edit the default rule to serve the variation you created. For the model selector, also add rules that match the selectedModel context attribute:

/aiconfig-targeting

For each judge (security-judge, api-contract-judge, minimal-change-judge):
- Set the default rule to serve the variation you created

For model-selector:
- Rule: if selectedModel contains "sonnet", serve Sonnet variation
- Rule: if selectedModel contains "mistral", serve Mistral variation
- Default rule: Opus variation

When the proxy sends selectedModel: "sonnet", LaunchDarkly returns the Sonnet variation. To learn more, read Target with AI Configs.

Option B: LaunchDarkly dashboard

Step 1: Create the model selector config

Go to AI Configs and click Create AI Config.
Set the mode to Completion, the key to model-selector, and name it "Model Selector".
Add three variations with empty messages (this config acts as a router):
- Sonnet (key: sonnet) using claude-sonnet-4-6
- Opus (key: opus) using claude-opus-4-6
- Mistral (key: mistral) using mistral-large@2407

Step 2: Create the judge AI Configs

Click Create AI Config and set the mode to Judge.
Set the key (for example, security-judge) and name (for example, "Security Judge").
Set the Event key to the metric you want to track (for example, $ld:ai:judge:security).
Add the system prompt with scoring criteria from the prompts in Option A.
Set the model to gpt-5-mini with temperature 0.3.
Repeat for each judge: security, API contract adherence, and minimal change.

Step 3: Attach judges to the model selector

Open the Model Selector AI Config and go to the Variations tab.
Expand a variation (for example, Sonnet) and find the Judges section.
Click Attach judges.

![Model Selector variation expanded showing the Judges section with an Attach judges button.]

Select the judges you created and set the sampling percentage to 100%.
Repeat for each variation.

Step 4: Configure targeting rules

Go to the Targeting tab for the Model Selector.
Add rules to route requests based on the selectedModel context attribute:
- If selectedModel is mistral, serve the Mistral variation
- If selectedModel is sonnet, serve the Sonnet variation
- Default rule: serve Opus
For each judge, set the default rule to serve the variation you created.

To learn more, read Custom judges.

Verify your setup

Before running the proxy, confirm in the dashboard:

Model selector: Each variation shows three attached judges.
Judges: Each judge prompt includes scoring criteria.
Targeting: All AI Configs have targeting enabled with correct rules.

Set up the project

Create a directory and install dependencies:

mkdir custom-evals && cd custom-evals
python -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn launchdarkly-server-sdk launchdarkly-server-sdk-ai \
    launchdarkly-server-sdk-ai-langchain langchain-anthropic python-dotenv

Create .env:

LD_SDK_KEY=sdk-your-sdk-key-here
LD_AI_CONFIG_KEY=model-selector
MODEL_KEY=sonnet
ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here
PORT=9911

Build the proxy server

Create server.py with the following code.

Click to expand the complete proxy server code

"""
Proxy server for Claude Code with automatic quality scoring.

Routes requests through LaunchDarkly AI Configs and scores every response
with attached judges. Metrics flow to the LaunchDarkly Monitoring dashboard.
"""

import asyncio
import os
import logging
import uuid

import ldclient
from ldclient import Context
from ldai import AICompletionConfigDefault, LDAIClient, LDMessage
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import uvicorn

from dotenv import load_dotenv
load_dotenv()

LD_SDK_KEY = os.environ.get("LD_SDK_KEY")
LD_AI_CONFIG_KEY = os.environ.get("LD_AI_CONFIG_KEY", "model-selector")
PORT = int(os.environ.get("PORT", "9911"))

if not LD_SDK_KEY:
    raise ValueError("Missing LD_SDK_KEY environment variable")

LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO").upper()
logging.basicConfig(level=getattr(logging, LOG_LEVEL, logging.INFO))

ld_config = ldclient.Config(LD_SDK_KEY)
ldclient.set_config(ld_config)
ld_client = ldclient.get()

if not ld_client.is_initialized():
    raise RuntimeError("LaunchDarkly client failed to initialize")

ai_client = LDAIClient(ld_client)
app = FastAPI()

# =============================================================================
# Message Conversion
# =============================================================================

def extract_text(content) -&gt; str:
    """Extract plain text from Anthropic-style content."""
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        texts = []
        for block in content:
            if isinstance(block, dict) and block.get("type") == "text":
                texts.append(block.get("text", ""))
        return "".join(texts)
    return str(content or "")


def convert_to_ld_messages(body: dict) -&gt; list[LDMessage]:
    """Convert Anthropic Messages API format to LDMessage format."""
    messages = []

    system = body.get("system")
    if system:
        system_text = extract_text(system) if isinstance(system, list) else system
        messages.append(LDMessage(role="system", content=system_text))

    for msg in body.get("messages", []):
        role_str = msg.get("role", "user")
        role = "assistant" if role_str == "assistant" else "user"
        messages.append(LDMessage(role=role, content=extract_text(msg.get("content", ""))))

    return messages

# =============================================================================
# Routes
# =============================================================================

@app.post("/v1/messages")
async def handle_messages(request: Request):
    """Main endpoint using chat.invoke() for automatic judge execution."""
    body = await request.json()
    user_key = request.headers.get("x-ld-user-key", "claude-code-local")

    # Build context with selectedModel for targeting
    model_key = os.environ.get("MODEL_KEY", "")
    context = (
        Context.builder(user_key)
        .set("selectedModel", model_key)
        .build()
    )

    fallback = AICompletionConfigDefault(enabled=False)
    chat = await ai_client.create_chat(LD_AI_CONFIG_KEY, context, fallback, {})

    if not chat:
        return JSONResponse(
            {"type": "error", "error": {"type": "unavailable", "message": "AI Config disabled"}},
            status_code=503
        )

    config = chat.get_config()
    model_name = config.model.name if config.model else "unknown"
    judge_count = len(config.judge_configuration.judges) if config.judge_configuration else 0

    print(f"[REQUEST] model={model_name}, judges={judge_count}")

    try:
        ld_messages = convert_to_ld_messages(body)

        if len(ld_messages) &gt; 1:
            chat.append_messages(ld_messages[:-1])

        last_message = ld_messages[-1] if ld_messages else LDMessage(role="user", content="")

        # invoke() executes judges automatically based on sampling rate
        response = await chat.invoke(last_message.content)

        # Await judge evaluations and log results
        if response.evaluations:
            print(f"[JUDGES] Awaiting {len(response.evaluations)} evaluations...")
            eval_results = await asyncio.gather(*response.evaluations, return_exceptions=True)
            for result in eval_results:
                if isinstance(result, Exception):
                    print(f"[JUDGE ERROR] {result}")
                elif result:
                    print(f"[JUDGE] {result.to_dict()}")

        # Flush events to LaunchDarkly
        ld_client.flush()
        await asyncio.sleep(0.1)

        response_text = response.message.content if response.message else ""

        # Get token metrics
        input_tokens = 0
        output_tokens = 0
        if response.metrics and response.metrics.usage:
            input_tokens = response.metrics.usage.input or 0
            output_tokens = response.metrics.usage.output or 0

        print(f"[METRICS] tokens={input_tokens}/{output_tokens}")

        return JSONResponse({
            "id": f"msg_{uuid.uuid4().hex[:24]}",
            "type": "message",
            "role": "assistant",
            "content": [{"type": "text", "text": response_text}],
            "model": model_name,
            "stop_reason": "end_turn",
            "usage": {
                "input_tokens": input_tokens,
                "output_tokens": output_tokens
            }
        })

    except Exception as e:
        ld_client.flush()
        logging.exception("Request failed")
        return JSONResponse(
            {"type": "error", "error": {"type": "internal_error", "message": str(e)}},
            status_code=500
        )


@app.get("/health")
async def health():
    return {"status": "ok", "launchdarkly": ld_client.is_initialized()}


@app.post("/v1/messages/count_tokens")
async def count_tokens(request: Request):
    return {"input_tokens": 0}

# =============================================================================
# Main
# =============================================================================

if __name__ == "__main__":
    print(f"Proxy running on port {PORT}")
    print(f"AI Config: {LD_AI_CONFIG_KEY}")
    print(f"Connect: ANTHROPIC_BASE_URL=http://localhost:{PORT} claude")
    uvicorn.run(app, host="127.0.0.1", port=PORT, log_level="info")

Connect Claude Code to your proxy

Start the proxy server:

python server.py

You should see output like:

Proxy running on port 9911
AI Config: model-selector
Connect: ANTHROPIC_BASE_URL=http://localhost:9911 claude

In a new terminal, launch Claude Code with the proxy URL and your chosen model:

MODEL_KEY=sonnet ANTHROPIC_BASE_URL=http://localhost:9911 claude

Every request now routes through your proxy. Watch the server logs to see judges executing:

[REQUEST] model=claude-sonnet-4-6, judges=3
[JUDGES] Awaiting 3 evaluations...
[JUDGE] {'evals': {'security': {'score': 1.0, 'reasoning': 'No vulnerabilities detected...'}}}
[JUDGE] {'evals': {'api-contract': {'score': 0.5, 'reasoning': 'Response uses correct endpoint...'}}}
[JUDGE] {'evals': {'minimal-change': {'score': 1.0, 'reasoning': 'Changes are focused...'}}}

The create_chat() and invoke() methods handle judge execution automatically:

chat = await ai_client.create_chat(config_key, context, fallback, {})
response = await chat.invoke(user_message)
# response.evaluations contains async judge tasks

Judge results are sent to LaunchDarkly automatically. You can optionally await response.evaluations to log results locally.

This proxy handles text-based conversations. Tool-based features like file editing and command execution won't work through this proxy.

How model routing works

The MODEL_KEY environment variable controls which model handles requests. The proxy passes it as a selectedModel context attribute:

context = Context.builder(user_key).set("selectedModel", model_key).build()

Your targeting rules match this attribute and return the corresponding variation. Switch models by changing the environment variable:

MODEL_KEY=mistral ANTHROPIC_BASE_URL=http://localhost:9911 claude

Compare cloud and local models

To evaluate Ollama models against cloud providers:

Add an "ollama" variation to your model-selector AI Config.
Add a targeting rule for selectedModel equals "ollama".
Launch with MODEL_KEY=ollama.

Your custom judges score Claude Sonnet and Llama 3.2 with identical criteria. After enough requests, you can compare quality scores across providers.

Run experiments

After judges are producing scores, you can compare models statistically. Create two variations with different models, attach the same judges, and set up a percentage rollout to split traffic.

Your judge metrics appear as goals in LaunchDarkly Experimentation. After enough data, you can answer "Which model produces more secure code?" with confidence, not guesswork.

To learn more, read Experimentation with AI Configs.

Monitor quality over time

Judge scores appear on your AI Config's Monitoring tab. To view evaluation metrics:

Open your model-selector AI Config and go to the Monitoring tab.
Select Evaluator metrics from the dropdown menu.

![Select Evaluator metrics from the dropdown]

Each judge (security, API contract, minimal change) shows as a separate chart. Hover over a chart to see scores broken down by variation.

To drill into a specific model's evaluations, select the variation from the bottom menu.

Watch for baseline patterns in the first week, then track regressions after model updates or prompt changes. Model providers ship updates without notice. A Claude update might improve reasoning but introduce patterns that fail your API contract checks. Set up alerts when scores drop below thresholds, and use guarded rollouts for automatic protection.

To learn more, read Monitor AI Configs.

Control costs with sampling

Each judge evaluation is an LLM call. Control costs by adjusting sampling rates:

Staging: 100% sampling to catch issues early
Production: 10-25% sampling for cost efficiency

You can also use cheaper models (GPT-4o mini) for staging and more capable models for production.

What you learned

The value is in the judges you create. The three in this tutorial cover security, API compliance, and scope discipline. Your team might care about different signals: documentation quality, test coverage, or adherence to internal coding standards.

Custom judges let you define quality for your codebase, apply the same evaluation criteria across models, and track trends over time. Once you create a judge, you can attach it to any AI Config in your project.

Ready to build custom judges for your codebase? Start your 14-day free trial and deploy your first evaluation today.

Next steps

hello-python-ai examples for more judge patterns
AI Configs best practices for production patterns

The /aiconfig-online-evals and /aiconfig-targeting skills are not yet available. Use the dashboard to complete those steps. ↩

Beyond n8n for Workflow Automation: Agent Graphs as Your Universal Agent Harness

Scarlett Attensil — Thu, 26 Mar 2026 00:33:34 +0000

Hardcoded multi-agent orchestration is brittle: topology lives in framework-specific code, changes require redeploys, and bottlenecks are hard to see. Agent Graphs externalize that topology into LaunchDarkly, while your application continues to own execution.

In this tutorial, you'll build a small multi-agent workflow, traverse it with the SDK, monitor per-node latency on the graph itself, and update a slow node's model without changing application code.

Node = AI Config (model, instructions, tools)
Edge = handoff metadata (routing contract you define)
Graph = topology (which nodes connect)
Your app = execution + interpretation

LaunchDarkly provides graph structure, config, and observability. Your application owns execution semantics: you write the code that interprets edges and runs agents.

What You'll Build

In this tutorial, you'll add Agent Graphs to an existing multi-agent workflow:

Build a graph visually in the LaunchDarkly UI
Connect it to your code with a few lines of SDK integration
Run your agents and see the graph in action
Monitor performance with per-node latency and invocation tracking
Fix a slow agent by swapping models from the dashboard

By the end, you'll have a multi-agent system where topology metadata changes happen in the UI, picked up by your traversal code on the next request.

Prerequisites

LaunchDarkly account with AI Configs access (sign up here)
Python 3.9+
An existing agent workflow (or use our sample repo)

The Problem with Hardcoded Orchestration

Every multi-agent framework handles orchestration differently:

# LangGraph - topology hardcoded in graph setup
workflow = StateGraph(AgentState)
workflow.add_node("supervisor", supervisor_node)
workflow.add_node("security", security_node)
workflow.add_node("support", support_node)
workflow.set_entry_point("supervisor")
# Routing logic buried in node functions or conditional edges

# OpenAI Agents SDK - handoffs defined per agent
security_agent = Agent(name="Security", instructions="...")
support_agent = Agent(name="Support", instructions="...")
supervisor = Agent(
    name="Supervisor",
    handoffs=[security_agent, support_agent]  # Topology locked in code
)

The topology is scattered across code. Agent Graphs make it visible: you see the entire workflow in one view, edit connections in the UI, and traverse it with graph-aware SDK methods.

Why Externalizing Topology Helps

If you've built multi-agent systems with LangGraph, OpenAI Swarm, or Strands, you've hit these walls:

Config duplication: Agent definitions scattered across framework-specific formats
Silent failures: An agent times out and you don't know until users complain
No topology visibility: The workflow exists only in code
Custom observability: Getting consistent per-agent metrics means reconciling different trace formats and data schemas across frameworks

For a detailed comparison of LangGraph, OpenAI Swarm, and Strands, see Compare AI orchestrators. Agent Graphs work with multiple agent frameworks.

Agent Graphs solve these by giving you a visual graph builder where you:

See your entire workflow at a glance, not buried in code
Monitor per-node metrics overlaid directly on the graph (latency, invocations, tool calls)
Add or remove agents without changing traversal logic, provided your runtime supports the node's tools and output contract
Inspect routing logic on edges, with handoff data visible in the UI
Use graph-aware SDK methods like is_terminal(), is_root(), and get_edges() instead of manual tracking

Step 1: Create AI Configs for Your Agents

Before building a graph, you need AI Configs for each agent. If you already have AI Configs, skip to Step 2.

See the AI Configs quickstart or run the bootstrap script in our sample repo:

git clone https://github.com/launchdarkly-labs/devrel-agents-tutorial
cd devrel-agents-tutorial
git checkout tutorial/agent-graphs
uv sync
cp .env.example .env  # Add your LD_SDK_KEY, LD_API_KEY, OPENAI_API_KEY
uv run python bootstrap/create_configs.py

For this tutorial, we'll use three configs:

supervisor-agent: Orchestrates the workflow and routes queries based on PII pre-screening
security-agent: Detects and redacts personally identifiable information (PII)
support-agent: Answers questions using dynamically loaded tools (search, RAG)

Step 2: Build the Graph in the UI

This is where Agent Graphs diverge from code-based orchestration. Instead of writing add_edge() calls, you'll see your topology and modify it visually.

Open your LaunchDarkly dashboard and navigate to AI > Agent graphs.

You'll see the first-time setup wizard. Since you already created AI Configs in Step 1, expand Create a graph at the bottom.

Name your graph chatbot-flow and click Create graph.

Add your first node: click Add node and select supervisor-agent
Set it as the root: click the node and toggle Root node
Add security-agent and support-agent as nodes

Draw edges: drag from supervisor-agent to both child agents
Add handoff data to each edge to define routing logic:

supervisor-agent → security-agent:

{
  "action": "sanitize",
  "reason": "PII detected",
  "route": "security"
}

supervisor-agent → support-agent:

{
  "action": "direct",
  "reason": "Clean input",
  "route": "support"
}

security-agent → support-agent:

{
  "action": "proceed",
  "reason": "Input sanitized",
  "route": "continue"
}

Notice what you're seeing: the entire workflow topology in one view. This graph is your architecture diagram, always current. Each node shows which AI Config variation it serves. The edges show routing logic that would otherwise be buried in conditional statements. When you need to add a new agent or change routing, you do it here, not in code.

LaunchDarkly doesn't execute your graph. It provides:

Topology: Which nodes exist and how they connect
Handoff metadata: Whatever JSON you put on edges
Per-node AI Config: Model, instructions, tools for each agent

Your code:

Decides which edges to follow based on agent decisions
Interprets handoff data however you want (the schema is yours)
Executes the actual agents

The handoff JSON is arbitrary metadata. You define the schema, you interpret it. LaunchDarkly stores and delivers it.

Step 3: Add the SDK to Your Project

Install the LaunchDarkly AI SDK:

uv add launchdarkly-server-sdk launchdarkly-server-sdk-ai

Initialize the clients in your code:

# config_manager.py - Initialize LaunchDarkly clients
def _initialize_launchdarkly_client(self):
    """Initialize LaunchDarkly client and AI client"""
    config = ldclient.Config(self.sdk_key)
    ldclient.set_config(config)
    self.ld_client = ldclient.get()

    # Block until client is initialized (max 10 seconds)
    self.ld_client.start_wait(10)

    if not self.ld_client.is_initialized():
        raise RuntimeError("LaunchDarkly client initialization failed")

    self.ai_client = LDAIClient(self.ld_client)

Build a context for targeting and tracking:

# config_manager.py - Build context for targeting
def build_context(self, user_id: str, user_context: dict = None) -> Context:
    """Build a LaunchDarkly context with consistent attributes."""
    context_builder = Context.builder(user_id).kind('user')

    if user_context:
        for key, value in user_context.items():
            context_builder.set(key, value)

    return context_builder.build()

Step 4: Integrate with Your Framework

This section walks through the integration code, starting with the building block (what runs at each node), then showing how nodes are orchestrated.

The Generic Agent Pattern

The key to dynamic execution is create_generic_agent. Every node uses the same implementation—no agent registry, no hardcoded agent types:

# agents/generic_agent.py
def create_generic_agent(agent_config, config_manager, valid_routes: List[str] = None):
    """Create a generic agent from LaunchDarkly AI Config."""

    class GenericAgent:
        def __init__(self):
            self.valid_routes = valid_routes or []

        async def ainvoke(self, state: dict) -> dict:
            """Execute the agent using LaunchDarkly config."""
            if not agent_config.enabled:
                return {"response": "", "_skipped": True}

            # Create model from config
            model = create_model_for_config(
                provider=agent_config.provider.name,
                model=agent_config.model.name,
                config_manager=config_manager
            )

            # Load tools from LaunchDarkly config
            tools = create_dynamic_tools_from_launchdarkly(agent_config)

            # Get instructions from config
            instructions = agent_config.instructions or "Process the input."

            # Inject route options into instructions
            if self.valid_routes:
                route_instruction = f"\n\nSelect one of these routes: {self.valid_routes}. Return: {{\"route\": \"<selected_route>\"}}"
                instructions = instructions + route_instruction

            # Execute and extract routing decision
            result = await self._execute(model, instructions, tools, state)
            result["routing_decision"] = self._extract_route(result.get("response", ""))

            # Track metrics
            agent_config.tracker.track_success()
            return result

    return GenericAgent()

The generic agent pattern means:

No agent registry: Every node uses the same create_generic_agent function
Config-driven behavior: Model, instructions, and tools all come from LaunchDarkly
Dynamic routing: Valid routes are injected from graph edges, not hardcoded
Minimal code changes: Add a new agent in LaunchDarkly, create its AI Config, add it to your graph, and it works—provided your runtime supports the node's tools and output contract

The AgentService Class

The AgentService class is the entry point for processing messages through your Agent Graph:

# api/services/agent_service.py
class AgentService:
    """Multi-Agent Orchestration using LaunchDarkly Agent Graph."""

    def __init__(self):
        self.config_manager = ConfigManager()
        self.config_manager.flush()

    async def process_message(
        self,
        user_id: str,
        message: str,
        user_context: dict = None
    ) -> ChatResponse:
        """Process message using LaunchDarkly Agent Graph."""
        result = await self._execute_graph(
            graph_key=os.getenv("AGENT_GRAPH_KEY", "chatbot-flow"),
            user_id=user_id.strip() or "anonymous",
            user_input=message,
            user_context=user_context or {}
        )

        return ChatResponse(
            response=result.get("final_response", ""),
            tool_calls=result.get("tool_calls", []),
            # ... other fields
        )

Executing the Graph

The _execute_graph method fetches the graph from LaunchDarkly and uses traverse() with skip logic for conditional routing:

# api/services/agent_service.py
async def _execute_graph(
    self,
    graph_key: str,
    user_id: str,
    user_input: str,
    user_context: dict = None
) -> Dict[str, Any]:
    """Execute agents using SDK's traverse() with skip logic."""
    ld_context = self.config_manager.build_context(user_id, user_context)
    graph = self.config_manager.ai_client.agent_graph(graph_key, ld_context)

    if not graph.is_enabled():
        raise ValueError(f"Agent Graph '{graph_key}' is not enabled")

    ctx = {
        "user_input": user_input,
        "messages": [HumanMessage(content=user_input)],
        "processed_input": user_input,
        "final_response": "",
        "tool_calls": [],
        # Skip logic: track which nodes should execute
        "_routed_to": {graph.root().get_key()},
        "_path": [],
        "_prev_key": None,
    }

    tracker = graph.get_tracker()

    # Define the node callback (see next section)
    def execute_node(node, exec_ctx):
        # ... node execution logic
        pass

    # Use SDK's traverse() - it handles traversal order
    graph.traverse(execute_node, ctx)

    # Track graph completion
    if tracker:
        tracker.track_path(ctx.get("_path", []))
        tracker.track_invocation_success()

    return ctx

Skip Logic for Conditional Routing

The execute_node callback implements skip logic—the core pattern that enables conditional routing:

# api/services/agent_service.py - inside _execute_graph
def execute_node(node, exec_ctx):
    """Execute a single node if it was routed to."""
    key = node.get_key()

    # Skip logic: only execute if parent routed to this node
    if key not in exec_ctx.get("_routed_to", set()):
        return {"_skipped": True}

    exec_ctx["_path"].append(key)

    # Track node invocation
    if tracker:
        tracker.track_node_invocation(key)
        if exec_ctx.get("_prev_key"):
            tracker.track_handoff_success(exec_ctx["_prev_key"], key)

    # Get edges and valid routes for this node
    edges = node.get_edges()
    valid_routes = [e.handoff.get("route") for e in edges if e.handoff and e.handoff.get("route")]

    # Execute agent with config from this node
    agent = create_generic_agent(node.get_config(), self.config_manager, valid_routes=valid_routes)
    result = _run_async(agent.ainvoke(exec_ctx))

    # Track tool calls
    if tracker and result.get("tool_calls"):
        for tool in result["tool_calls"]:
            tracker.track_tool_call(key, tool)

    # Route to next node: add to _routed_to set
    if edges:
        next_key = self._select_next_node(edges, result, tracker)
        if next_key:
            exec_ctx["_routed_to"].add(next_key)

    exec_ctx["_prev_key"] = key
    return result

The _routed_to set tracks which nodes should execute:

Start: Add root node to _routed_to
traverse() visits each node: If node is in _routed_to, execute it; otherwise skip
After execution: Add the next node (based on routing decision) to _routed_to

This enables conditional routing: the supervisor routes to either security OR support, and only the chosen path executes.

Routing Between Nodes

The _select_next_node method determines which node to route to based on the agent's routing decision:

# api/services/agent_service.py
def _select_next_node(self, edges, result: dict, tracker=None):
    """Select next node key based on routing decision."""
    routing = result.get("routing_decision", "").lower().strip() if result.get("routing_decision") else None

    # Build route map: route -> target_config
    route_map = {}
    for edge in edges:
        route = (edge.handoff.get("route", "") if edge.handoff else "").lower().strip()
        if route:
            route_map[route] = edge.target_config

    # Exact match
    if routing and routing in route_map:
        return route_map[routing]
    elif routing:
        if tracker:
            tracker.track_handoff_failure()

    # Default: first edge
    if edges:
        return edges[0].target_config

    return None

The key insight: your graph topology comes from LaunchDarkly, not hardcoded orchestration. Change the graph in the UI, and your code picks up the new structure on the next request.

Step 5: Run It

With the AgentService wired up (as shown in Step 4), you can now process messages through your Agent Graph. The service handles:

Building the LaunchDarkly context for targeting
Fetching the graph and executing nodes via traverse()
Tracking metrics for monitoring
Returning the final response

Test it by sending a message:

service = AgentService()
response = await service.process_message(
    user_id="user-123",
    message="What's the status of my order?",
    user_context={"plan": "premium"}
)
print(response.response)

Now go back to the LaunchDarkly UI. Add a new node or change an edge. Run your code again. Topology changes are picked up by your traversal code on subsequent SDK evaluations.

Step 6: Monitor Agent Performance

This is the key differentiator: monitoring happens on the graph itself, not in a separate dashboard. You see metrics overlaid on the same visual topology you built, so bottlenecks are immediately obvious.

The sample repo includes full instrumentation: calls to tracker.track_success(), tracker.track_error(), and tracker.track_tool_call() in the agent execution path. After running some traffic, open your Agent Graph to see the results.

Navigate to AI > Agent graphs > chatbot-flow. You'll see a metrics bar at the top of the graph view where you can toggle different metrics on and off.

Metrics on the graph

Here's what makes this different from traditional APM: the metrics appear directly on your workflow visualization. No mental mapping between a dashboard and your code. No correlating trace IDs. The slow node lights up on the graph.

Turn on Latency to see duration data overlaid directly on your graph:

Total duration: The combined time for the entire graph invocation
Per-node duration: How long each individual agent takes

Turn on Invocations to see how often each node is reached. This reveals which paths your users take most frequently. In a routing graph, you'll quickly see whether most queries go through security or skip directly to support.

Turn on Tool calls to see the average number of tool invocations per node. If an agent is calling tools excessively, you'll spot it here.

Monitoring page

Click Monitoring to see all metrics over time. This view shows:

Latency trends: Duration per node over hours, days, or weeks
Invocation patterns: Traffic flow through your graph
Tool call breakdown: Which specific tools are being called and how often

To see which specific tools are called, you need to track them in your code using the tracker. The SDK sends this data to LaunchDarkly, which displays it in the monitoring view.

Generate traffic to see metrics

Run the traffic generator from the sample repo to send queries through your graph:

uv run python tools/traffic_generator.py --queries 20 --delay 2

This sends a mix of queries (some with PII, some without) to exercise both the security and support paths. After a few minutes, you'll see metrics populate on the graph.

Detecting a slow agent

With traffic flowing, suppose the security-agent starts averaging 5 seconds per call. With latency metrics enabled on the graph, you see it immediately: the security-agent node shows a high duration value while other nodes stay fast.

The invocation numbers also tell a story. If security-agent shows 50 invocations and support-agent shows 80, you know ~30 queries are bypassing security (the clean path). This helps you understand whether the slow agent is affecting most users or just a subset.

Without Agent Graphs, you'd need custom logging, Datadog queries, and manual correlation. With Agent Graphs, you see the problem in 30 seconds.

Step 7: Fix Without Deploying

The security-agent is slow because it's using claude-sonnet-4 for PII detection. A smaller, faster model may be sufficient for this task.

In the LaunchDarkly dashboard, update the pii-detector variation:

Change model from Anthropic.claude-sonnet-4-20250514 to Anthropic.claude-3-haiku-20240307

Or use Agent Skills to make the change from your coding assistant:

The security-agent pii-detector variation is averaging 5 seconds.
Change the model to claude-3-haiku-20240307.

No code changes. No deploy. Changes are picked up on subsequent SDK evaluations.

Run the traffic generator again and watch the latency drop.

What just happened

Traffic generator sent queries through the graph
Monitoring showed the slow agent on the graph
Model swap happened in the UI (or via Agent Skills)
Your code automatically used the new configuration

No deploys. No PRs. The fix is live.

OpenAI Agents SDK Integration (Conceptual)

Agent Graphs work with multiple frameworks. This conceptual example shows how the pattern translates to OpenAI Agents SDK:

# Conceptual example showing how Agent Graph SDK methods work with OpenAI Agents
from agents import Agent, Runner

def handle_traversal(node, state):
    config = node.get_config()
    tracker = config.tracker
    edges = node.get_edges()

    # Child agents are already in state (reverse traversal builds bottom-up)
    handoffs = [state[edge.target_config] for edge in edges]

    def on_handoff(ctx):
        # Track handoff events
        return ctx

    return Agent(
        name=config.key,
        instructions=config.instructions,
        handoffs=handoffs,
        on_handoff=on_handoff,
    )

if agent_graph.is_enabled():
    root = agent_graph.reverse_traverse(handle_traversal, {})

result = await Runner.run(root, "Tell me about your engineering team")

Same graph definition, adapted to each framework's execution model. The topology metadata lives in LaunchDarkly; your code interprets and executes it.

Best Practices

Start simple: Begin with a linear graph (A → B → C) before adding conditional routing.

Use handoff data for context passing: Include metadata like action type, reason, or state that the next agent needs to continue the workflow.

Track everything: Call tracker.track_success() and tracker.track_error() in every node for complete visibility. Use graph_tracker.track_tool_call(tool_name) to track which tools agents invoke.

Test with targeting: Use LaunchDarkly targeting to route test users to experimental graph configurations.

Handle missing edges: Decide what happens when no edge matches a routing decision or when a target node is disabled. Recommend: fail closed, log diagnostics, and track routing failures.

Keep execution state request-scoped: Store execution state inside the context object (ctx) passed through traversal, not in instance-level variables. Treat graph traversal as request-scoped to avoid concurrency issues.

What You've Built

You now have a multi-agent system where:

Graph topology is externalized and self-documenting
Routing logic is visible on edges, not buried in code
Monitoring appears on the graph itself, not a separate dashboard
Node-level control lets you disable a single agent without touching others, provided your executor checks node availability
Multiple frameworks can consume the same graph metadata

When you spot a slow agent in monitoring, you can swap the model from the dashboard without a deploy.

Next Steps

Agent Graphs Reference: SDK methods for traverse, reverse_traverse, get_edges(), and handoff data
AI Configs Documentation: Learn more about variations, targeting, and experiments
Agent Skills Tutorial: Manage AI Configs from your coding assistant
Monitor AI Configs: Deep dive into metrics and dashboards
Sample Repository: Complete code from this tutorial

Conclusion

Hardcoded orchestration was fine when you had one agent. With multi-agent systems, it becomes a liability. Every change requires a deploy. Every incident requires a developer.

Agent Graphs flip this. Define your workflow in LaunchDarkly, integrate it with your framework, and fix many problems without touching code. Your agents become as dynamic as your feature flags.

Ready to stop hardcoding? Get started with AI Configs and create your first Agent Graph.

LLM evaluation guide: When to add online evals to your AI application

Scarlett Attensil — Wed, 17 Dec 2025 17:42:49 +0000

The quick decision framework

Online evals for AI Configs is currently in closed beta. Judges must be installed in your project before they can be attached to AI Config variations.

Online evals provide real-time quality monitoring for LLM applications. Using LLM-as-a-judge methodology, they run automated quality checks on a configurable percentage of your production traffic, producing structured scores and pass/fail judgments you can act on programmatically. LaunchDarkly includes three built-in judges: accuracy, relevance, and toxicity.

Skip online evals if:

Your checks are purely deterministic (schema validation, compile tests)
You have low volume and can manually review outputs in observability dashboards
You're primarily debugging execution problems

Add online evals when:

You need quantified quality scores to trigger automated actions (rollback, rerouting, alerts)
Manual quality review doesn't scale to your traffic volume
You're measuring multiple quality dimensions (accuracy, relevance, toxicity)
You want statistical quality trends across segments for AI governance and compliance
You need to monitor token usage and cost alongside quality metrics
You're running A/B tests or guarded releases and need automated quality gates

Most teams add them within 2-3 sprints when manual quality review becomes the bottleneck. Configurable sampling rates let you balance evaluation coverage with cost and latency.

Online evals vs. LLM observability

LLM observability shows you what happened. Online evals automatically assess quality and trigger actions based on those assessments.

LLM observability: your security camera

LLM observability shows you everything that happened through distributed tracing: full conversations, tool calls, token usage, latency breakdowns, and cost attribution. Perfect for debugging and understanding what went wrong. But when you're handling 10,000 conversations daily, manually reviewing them for quality patterns doesn't scale.

Online evals: your security guard

Automatically scores every sampled request using LLM-as-a-judge methodology across your quality rubric (accuracy, relevance, toxicity) and takes action. Instead of exporting conversations to spreadsheets for manual review, you get real-time quality monitoring with drift detection that triggers alerts, rollbacks, or rerouting.

The 3 AM difference

Without evals: "Let's meet tomorrow to review samples and decide if we should rollback."

With evals: "Quality dropped below threshold, automatic rollback triggered, here's what failed..."

How online evals actually work

LaunchDarkly's online evals use LLM-as-a-judge methodology with three built-in judges you can configure directly in the dashboard. No code changes required.

Getting started:

Install judges from the AI Configs menu
Attach judges to AI Config variations
Configure sampling rates (balance coverage with cost/latency)
Evaluation metrics are automatically emitted as custom events
Metrics are automatically available for A/B tests and guarded releases

What you get from each built-in judge:

Accuracy judge:

{
  "score": 0.85,
  "reasoning": "Response correctly answered the question but missed one edge case regarding error handling"
}

Relevance judge:

{
  "score": 0.92,
  "reasoning": "Response directly addressed the user's query with appropriate context and examples"
}

Toxicity judge:

{
  "score": 0.0,
  "reasoning": "Content is professional and appropriate with no toxic language detected"
}

Each judge returns a score from 0.0 to 1.0 plus reasoning that explains the assessment. LaunchDarkly's built-in judges (accuracy, relevance, toxicity) have fixed evaluation criteria and are configured only by selecting the provider and model.

Configuration:
Install judges from the AI Configs menu in your LaunchDarkly dashboard. They appear as pre-configured AI configs (AI Judge - Accuracy, AI Judge - Toxicity, AI Judge - Relevance). When configuring your AI Config variations in completion mode, select which judges to attach with your desired sampling rate. Use different judge combinations for different environments to match your quality requirements and cost constraints.

Real problems online evals solve

Scale for production applications: Your SQL generator handles 50,000 queries daily. LLM observability shows you every query through distributed tracing. Online evals tell you the proportion that are semantically wrong, automatically, with hallucination detection built in.

Multi-dimensional quality monitoring: Customer service AI applications aren't just "did it respond?" It's accuracy, relevance, toxicity, compliance, and appropriateness. Online evals score all dimensions simultaneously, each with its own threshold and reasoning.

RAG pipeline validation: Your retrieval-augmented generation system needs continuous monitoring of both retrieval quality and generation accuracy. Online evals can assess whether retrieved context is relevant and whether the response accurately uses that context, preventing hallucinations and ensuring factual grounding.

Cost and performance optimization: Monitor token usage alongside quality metrics. If certain queries consume 10x more tokens than others, online evals help identify these patterns so you can optimize prompts or routing logic to reduce costs without sacrificing quality.

Actionable metrics for AI governance: Transform 10,000 responses from data to decisions with evaluator-driven quality gates:

Accuracy trending below 0.8? Automated alerts to the team
Toxicity above 0.2? Immediate review and potential rollback
Relevance dropping for specific user segments? Targeted configuration updates
Metrics automatically feed A/B tests and guarded releases for continuous improvement

Example implementation path

Week 1-2: Define quality dimensions and install judges.
Use LLM observability alone first. Manually review samples to understand your system. Define your quality dimensions: accuracy, relevance, toxicity, or other criteria specific to your application. Install the built-in judges from the AI Configs menu in LaunchDarkly.

Week 3-4: Attach judges with sampling.
Attach judges to AI Config variations in LaunchDarkly. Start with one or two key judges (accuracy and relevance are good defaults). Configure sampling rates between 10-20% of traffic to balance coverage with cost and latency. Compare automated scores with human judgment to validate the judges work for your use case.

Week 5+: Operationalize with quality gates.
Add more evaluation dimensions as you learn. Connect scores to automated actions and evaluator-driven quality gates: when accuracy drops below 0.7, trigger alerts; when toxicity exceeds 0.2, investigate immediately. Leverage the custom events and metrics for A/B testing and guarded releases to continuously improve your application's performance.

The bottom line

You don't need online evals on day one. Start with LLM observability to understand your AI system through distributed tracing. Add evaluations when you hear yourself saying "we need to review more conversations" or "how do we know if quality is degrading?"

LaunchDarkly's three built-in judges (accuracy, relevance, toxicity) provide LLM-as-a-judge evaluation that you can attach to any AI Config variation in completion mode with configurable sampling rates. Note that online evals currently only work with completion mode AI Configs. Agent-based configs are not yet supported. Evaluation metrics are automatically emitted as custom events and feed directly into A/B tests and guarded releases, enabling continuous AI governance and quality improvement without code changes. Start simple with one judge, learn what matters for your application, and expand from there.

LLM observability is your security camera. Online evals are your security guard.

Next steps

Ready to get started? Sign up for a free LaunchDarkly account if you haven't already.

Build a complete quality pipeline:

AI Config CI/CD Pipeline - Add automated quality gates and LLM-as-a-judge testing to your deployment process
Combine offline evaluation (in CI/CD) with online evals (in production) for comprehensive quality coverage

Learn more about AI Configs:

AI Config documentation - Understand how AI Configs enable real-time LLM configuration
Online evals documentation - Deep dive into judge installation and configuration
Guardrail metrics - Monitor quality during A/B tests and guarded releases

See it in action:

Check LLM observability in the LaunchDarkly dashboard to track your AI application performance with distributed tracing

Industry standards:
LaunchDarkly's approach aligns with emerging AI observability standards, including OpenTelemetry's semantic conventions for AI monitoring, ensuring your evaluation infrastructure integrates with the broader observability ecosystem.

When to Use Prompt-Based vs Agent Mode in LaunchDarkly for AI Applications

Scarlett Attensil — Wed, 17 Dec 2025 17:39:09 +0000

A Guide for LangGraph, OpenAI, and Multi-Agent Systems

The broader tech industry can't agree on what the term "agents" even means. Anthropic defines agents as systems where "LLMs dynamically direct their own processes," while Vercel's AI SDK enables multi-step agent loops with tools, and OpenAI provides an Agents SDK with built-in orchestration. So when you're creating an AI Config in LaunchDarkly and see "prompt-based mode" vs. "agent mode," you might reasonably expect this choice to determine whether you get automatic tool execution loops, server-side state management, or some other fundamental capability difference.

But LaunchDarkly's distinction is different and more practical. Understanding it will save you from confusion and help you ship AI features faster.

TL;DR

LaunchDarkly's "prompt-based vs. agent" choice is about input schemas and framework compatibility, not execution automation. Prompt-based mode returns a messages array (perfect for chat UIs), while agent mode returns an instructions string (optimized for LangGraph/CrewAI frameworks). Both provide the same core benefits: provider abstraction, A/B testing, metrics tracking, and the ability to change AI behavior without deploying code.

Ready to start? Sign up for a free trial → create your first AI Config → Choose your mode → Configure and ship.

The fragmented AI landscape

LaunchDarkly supports 20+ AI providers: OpenAI, Anthropic, Gemini, Azure, Bedrock, Cohere, Mistral, DeepSeek, Perplexity, and more. Each has their own interpretation of "completions" vs "agents," creating a chaotic ecosystem with different API endpoints, execution behaviors, state management approaches, and capability limitations. This fragmentation makes it difficult to switch providers or even understand what capabilities you're getting. That's where LaunchDarkly's abstraction layer comes in.

LaunchDarkly's approach: provider-agnostic input schemas

LaunchDarkly's AI Configs are a configuration layer that abstracts provider differences. When you choose prompt-based mode or agent mode, you're selecting an input schema (messages array vs. instructions string), not execution behavior. LaunchDarkly provides the configuration; you handle orchestration with your own code or frameworks like LangGraph. This gives you provider abstraction, A/B testing, metrics tracking, and online evals (prompt-based mode only) without locking you into any specific provider's execution model.

Prompt-based mode: messages-based

Prompt-based mode uses a messages array format with system/user/assistant roles (some providers like OpenAI also support a "developer" role for more granular control). This is the traditional chat format that works across all AI providers.

UI Input: "Messages" section with role-based messages

SDK Method: aiclient.config()

Returns: Customized prompt + model configuration

Documentation: AI Config docs

# Retrieve prompt-based AI config
config = aiclient.config(
    key="customer-support",
    context=context,
    default_value=default_config
)

# What you get back: messages array
print(config.messages)
# [
#   {
#     "role": "system",
#     "content": "You are a helpful customer support agent for Acme Corp."
#   },
#   {
#     "role": "user",
#     "content": "How can I reset my password?"
#   }
# ]

# Use with provider SDKs that expect message arrays
response = openai.chat.completions.create(
    model=config.model.name,
    messages=config.messages  # Standard message format
)

When to use prompt-based mode:

You're building chat-style interactions: Traditional message-based conversations where you construct system/user/assistant messages
You need online evals: LaunchDarkly's model-agnostic online evals are currently only available in prompt-based mode
You want granular control of workflows: Discrete steps that need to be accomplished in a specific order, or multi-step asynchronous processes where each step executes independently
One-off evaluations: Issue individual evaluations of your prompts and completions (not online evals)
Simple processing tasks: Summarization, name suggestions, or other non-context-exceeding data processing

Agent mode: goal/instructions-based

Agent mode uses a single instructions string format that describes the agent's goal or task. This format is optimized for agent orchestration frameworks that expect high-level objectives rather than conversational messages.

UI Input: "Goal or task" field with instructions

SDK Method: aiclient.agent()

Returns: Customized instructions + model configuration

Examples: hello-python-ai examples

# Retrieve agent-based AI config
agent_config = aiclient.agent(
    key="research-assistant",
    context=context,
    default_value=default_config
)

# What you get back: instructions string
print(agent_config.instructions)
# "You are a research assistant. Your goal is to gather comprehensive
# information on the requested topic using available search tools.
# Search multiple sources, synthesize findings, and provide a detailed
# summary with citations."

# Use with agent frameworks that expect instructions
from langgraph.prebuilt import create_react_agent
from langchain_openai import init_chat_model

llm = init_chat_model(
    model=agent_config.model.name,
    model_provider=agent_config.provider.name
)

agent = create_react_agent(
    llm,
    tools=[search_tool, citation_tool],
    prompt=agent_config.instructions  # Goal/task instructions
)

# Execute and track
response = agent.invoke({"messages": [{"role": "user", "content": "..."}]})

When to use agent mode:

You're using agent frameworks: LangGraph, LangChain, CrewAI, AutoGen, or LlamaIndex Workflows expect goal/instruction-based inputs
Goal-oriented tasks: "Research X and create Y" rather than conversational message exchange
Tool-driven workflows: While both modes support tools, agent mode's format is optimized for frameworks that orchestrate tool usage
Open-ended exploration: The output is open-ended and you don't know the actual answer you're trying to get to
Data as an application: You want to treat your data as an application to feed in arbitrary data and ask questions about it
Provider agent endpoints: LaunchDarkly may route to provider-specific agent APIs when available (note: not all models support agent mode; check your model's capabilities)

See example: Build a LangGraph Multi-Agent System with LaunchDarkly

Quick comparison

Feature	Prompt-Based Mode	Agent Mode
Input format	Messages (system/user/assistant)	Goal/task + instructions
Tools support	✅ Yes	✅ Yes
SDK method	`config()`	`agent()`
Automatic execution loop	❌ No (you orchestrate)	❌ No (you orchestrate)
Online evals	✅ Available	❌ Not yet available
Best for	Chat-style prompting, single completions	Agent frameworks, goal-oriented tasks
Provider endpoint	Standard endpoint	May use provider-specific agent endpoint if available
Model support	All models	Most models (check model card for "Agent mode" capability)

Model compatibility: Not all models support agent mode. When selecting a model in LaunchDarkly, check the model card for "Agent mode" capability. Models like GPT-4.1, GPT-5 mini, Claude Haiku 4.5, Claude Sonnet 4.5, Claude Sonnet 4, Grok Code Fast 1, and Raptor mini support agent mode, while models focused on reasoning (like GPT-5, Claude Opus 4.1) may only support prompt-based mode.

How providers handle "completion vs agent"

To understand why LaunchDarkly's abstraction is valuable, let's look at how major AI providers handle the distinction between basic completions and advanced agent capabilities. The table below shows how different providers implement "advanced" modes; generally these are ADDITIVE, including all basic capabilities plus extras. For example, OpenAI's Responses API includes all Chat Completions features plus additional capabilities.

Provider	"Basic" Mode	"Advanced" Mode	Key Difference	Link
OpenAI	Chat Completions API	Responses API	Responses adds built-in tools (web_search, file_search, computer_use, code_interpreter, remote MCP), server-side conversation state with stored IDs, and improved streaming. Chat Completions remains supported.	Docs
Anthropic	Tool Use (client tools)	Tool Use (client + server tools)	Server tools (web_search, web_fetch) execute on Anthropic's servers. You can use both client and server tools together	Docs
Google Gemini	Manual function calling	Automatic function calling (Python SDK)	Python SDK auto-converts functions to schemas, runs the execution loop, and supports compositional multi-step calls. Manual mode: full control, all platforms	Docs
Vercel AI SDK	`generateText()`	`generateText()` with multi-step loop	Multi-step agent loops with tools; SDK continues until complete; `maxSteps` provides loop control to limit steps	Docs
Azure OpenAI	Assistants API (deprecated)	AI Agent Services	Enterprise agent runtime with threads, tool orchestration, safety, identity, networking, and observability; includes Responses API and Computer-Using Agent in Azure	Docs
AWS Bedrock (Nova)	Converse API (tool use)	Bedrock Agents	Agents: managed service with automatic orchestration + state management + multi-agent collaboration. Converse: manual tool orchestration, full control	Docs
Cohere	Standard chat	Command A	Command A: enhanced multi-step tool use, REACT agents, ~150% higher throughput	Docs

This fragmentation across providers is exactly why LaunchDarkly's approach matters: you configure once (messages vs. goals), and LaunchDarkly handles the provider-specific translation. Want to switch from OpenAI to Anthropic? Just change the provider in your AI Config. Your application code stays the same.

Note on OpenAI's ecosystem (Nov 2025): The Agents SDK is OpenAI's production-ready orchestration framework. It uses the Responses API by default, and via a built-in LiteLLM adapter it can run against other providers with an OpenAI-compatible shape. Chat Completions is still supported, but OpenAI recommends Responses for new work. The Assistants API is deprecated and scheduled to shut down on August 26, 2026.

Common misconceptions

Now that you understand the modes and how they differ from provider-specific implementations, let's clear up some common points of confusion:

❌ "Agent mode provides automatic execution"
No. Both modes require you to orchestrate. Agent mode just provides a different input schema.

❌ "Agent mode is for complex tasks, prompt-based mode is for simple ones"
Not quite. It's about input format and framework compatibility, not task complexity.

❌ "I can only use tools in agent mode"
False. Both modes support tools. The difference is how you specify your task (messages vs. goal).

❌ "LaunchDarkly is an agent framework like LangGraph"
No. LaunchDarkly is configuration management for AI. Use it WITH frameworks like LangGraph, not instead of them.

Why LaunchDarkly's abstraction matters

Now that you've seen how fragmented the provider landscape is, let's explore the practical value of LaunchDarkly's abstraction layer.

Switching providers without code changes

Without LaunchDarkly:

# Hardcoded provider and prompts in your application
openai_client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
response = openai_client.chat.completions.create(
    model="gpt-4",  # Want to switch to Claude? Need to deploy new code
    messages=[
        {"role": "system", "content": "You are helpful"},  # Want to A/B test prompts? Deploy again
        {"role": "user", "content": "Hello"}
    ]
)

# To switch providers, you need to:
# 1. Write new code for different provider API
# 2. Deploy to production
# 3. Hope nothing breaks

With LaunchDarkly:

# Get config from LaunchDarkly
config = aiclient.config(key="my-ai-config", context=context)

# You still write provider-specific code, but only once
if config.provider.name == "openai":
    client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    response = client.chat.completions.create(
        model=config.model.name,      # Comes from LaunchDarkly
        messages=config.messages,      # Normalized schema across providers
        temperature=config.model.parameters.get('temperature')
    )
elif config.provider.name == "bedrock":
    client = boto3.client('bedrock-runtime', region_name='us-east-1')
    response = client.converse(
        modelId=config.model.name,    # Comes from LaunchDarkly
        messages=_convert_to_bedrock_format(config.messages),  # LaunchDarkly normalizes, you convert
        inferenceConfig={'temperature': config.model.parameters.get('temperature')}
    )

# Now you can switch providers via LaunchDarkly UI without deployment
# Change prompts, A/B test models, roll out gradually - all via configuration

The real value: Once your code is set up to handle different providers, you can switch between them, change prompts, A/B test models, and roll out changes gradually - all through the LaunchDarkly UI without deploying code. You write the provider handlers once; you manage AI behavior forever.

Security and risk management

AI agents can be powerful and potentially risky. With LaunchDarkly AI Configs, you can:

Instantly disable problematic models or tools without deploying code
Gradually roll out new agent capabilities to a small percentage of users first
Quickly roll back if an agent behaves unexpectedly
Control access by user tier (limit powerful tools to trusted users)
Target specific individuals in production to test experimental AI behavior in real environments without affecting other users

When you're not directly coupled to provider APIs, responding to security issues becomes a configuration change instead of an emergency deployment.

Advanced: Provider-specific packages (JavaScript/TypeScript)

For JavaScript/TypeScript developers looking to reduce boilerplate even further, LaunchDarkly offers optional provider-specific packages. These work with both prompt-based and agent modes and are purely additive - you don't need them to use LaunchDarkly AI Configs effectively.

Available packages:

@launchdarkly/server-sdk-ai-openai - OpenAI provider
@launchdarkly/server-sdk-ai-langchain - LangChain provider (works with both LangChain and LangGraph)
@launchdarkly/server-sdk-ai-vercel - Vercel AI SDK provider

What they provide:

Model creation helpers: One-line functions like createLangChainModel(aiConfig) that return fully-configured model instances
Automatic metrics tracking: Integrated metrics collection
Format conversion utilities: Helper functions to translate between schemas

Example with LangGraph:

// Get agent config from LaunchDarkly
const agentConfig = await ldClient.aiAgent('research-assistant', context);

// Create LangChain model - config already applied
const model = await LangChainProvider.createLangChainModel(agentConfig);

// Use with LangGraph
const agent = createReactAgent(model, tools, agentConfig.instructions);
const response = await agent.invoke({ messages: [{ role: "user", content: "Research X" }] });

Production readiness: These packages are in early development and not recommended for production. They may change without notice.

Python approach: The Python SDK takes a different path with built-in convenience methods like track_openai_metrics() in the single launchdarkly-server-sdk-ai package. See Python AI SDK reference.

Start building with LaunchDarkly AI configs

You now understand how LaunchDarkly's prompt-based and agent modes provide provider-agnostic configuration for your AI applications. Whether you're building chat interfaces or complex multi-agent systems, LaunchDarkly gives you the flexibility to experiment, iterate, and ship AI features without the complexity of managing multiple provider APIs.

Choosing your mode:

Start with prompt-based mode if:

You're building a chat interface or conversational UI
You need online evaluations for quality monitoring
You want precise control over multi-step workflows
You're uncertain which mode fits your use case (it's the more flexible starting point)

Choose agent mode if:

You're integrating with LangGraph, LangChain, CrewAI, or similar frameworks
Your task is goal-oriented rather than conversational ("Research X and create Y")
You're feeding arbitrary data and asking open-ended questions about it

Remember: Both modes give you the same core benefits: provider abstraction, A/B testing, and runtime configuration changes. The choice is about input format, not capabilities.

Get started:

Sign up for a free LaunchDarkly account
Create your first AI Config: Takes less than 5 minutes
Explore example implementations: Learn from working code
Start with prompt-based mode unless you're specifically using an agent framework

All I Want for Christmas is Observable Multi-Modal Agentic Systems

Scarlett Attensil — Wed, 17 Dec 2025 17:31:15 +0000

How Session Replay + Online Evals Revealed How My Holiday Pet App Actually Works

Original article published on December 17, 2025.

I added LaunchDarkly observability to my Christmas-play pet casting app thinking I'd catch bugs. Instead, I unwrapped the perfect gift 🎁. Session replay shows me WHAT users do, and online evaluations show me IF my model made the right casting decision with real-time accuracy scores. Together, they're like milk 🥛 and cookies 🍪 - each good alone, but magical together for production AI monitoring.

See the App in Action

Discovery #1: Users' 40-second patience threshold

I decided to use session replay to evaluate the average time it took users to go through each step in the AI casting process. Session replay is LaunchDarkly's tool that records user interactions in your app - every click, hover, and page navigation - so you can watch exactly what users experience in real-time.

The complete AI casting process takes 30-45 seconds: personality analysis (2-3s), role matching (1-2s), DALL-E 3 costume generation (25-35s), and evaluation scoring (2-3s). That's a long time to stare at a loading spinner wondering if something broke.

What are progress steps?

Progress steps are UI elements I added to the app - not terminal commands or backend processes, but actual visual indicators in the web interface that show users which phase of the AI generation is currently running. These appear as a simple list in the loading screen, updating in real-time as each AI task completes. No commands needed - they automatically display when the user clicks "Get My Role!" and the AI processing begins.

Session replay revealed:

WITHOUT Progress Steps (n=20 early sessions):
0-10 seconds: 20/20 still watching (100%)
10-20 seconds: 18/20 still watching (90%)
20-30 seconds: 14/20 still watching (70%) - rage clicks begin
30-40 seconds: 9/20 still watching (45%) - tab switching detected
40+ seconds: 7/20 still watching (35% stay)

WITH Progress Steps (n=30 after adding them):
0-10 seconds: 30/30 still watching (100%)
10-20 seconds: 29/30 still watching (97%)
20-30 seconds: 25/30 still watching (83%)
30-40 seconds: 23/30 still watching (77%)
40+ seconds: 24/30 still watching (80% stay!)

Critical Discovery: Progress steps more than DOUBLED
completion rate (35% → 80%)

This made the difference:

Clear progress steps:

Step 1: AI Casting Decision
Step 2: Generating Costume Image (10-30s)
Step 3: Evaluation

As each completes:
✅ Step 1: AI Casting Decision
Step 2: Generating Costume Image (10-30s)
Step 3: Evaluation

Session replay showed users hovering over the back button at 25 seconds, then relaxing when they saw "Step 2: Generating Costume Image (10-30s)." The moment they understood DALL-E was creating their pet's costume (not the app freezing), they were willing to wait. Clear progress indicators transform anxiety into patience.

Discovery #2: Observability + online evaluations give the complete picture

Session replay shows user behavior and experience. Online evaluations expose AI output quality through accuracy scoring. Together, they form a solid strategy for AI observability.

To see this in action, let's take a closer look at an example.

Example: The speed-running corgi owner

In this scenario, a user blazes through the entire pet app setup from the initial quiz to the final results, completing the process in record time. So fast, in fact, that instead of this leading to a favorable outcome, it led to an instance of speed killing quality.

Session Replay Showed:

Quiz completed in 8 seconds (world record) - they clicked the first option for every question
Skipped photo upload entirely
Waited the full 31 seconds for processing
Got their result: "Sheep"
Started rage clicking on the sheep image immediately
Left the site without saving or sharing

Why did their energetic corgi get cast as a sheep? The rushed quiz responses created a contradictory personality profile that confused the AI. Without a photo to provide visual context, the model defaulted to its safest, most generic casting choice.

Online Evaluation Results:

Evaluation Score: 38/100 ❌
Reasoning: "Costume contains unsafe elements: eyeliner, ribbons"
Wait, what? The AI suggested face paint and ribbons, evaluation said NO

Online evaluations use a model-agnostic evaluation (MAE) - an AI agent that evaluates other AI outputs for quality, safety, or accuracy. The out-of-the-box evaluation judge is overly cautious about physical safety. For the above scenario, the evaluation comments:

"Costume includes eyeliner which could be harmful to pets" (It's a DALL-E image!)
"Ribbons pose entanglement risk"
"Bells are a choking hazard" (It's AI-generated art!)

About 40% of low scores are actually the evaluation being overprotective about imaginary safety issues, not bad casting.

Speed-runners get generic roles AND the evaluation writes safety warnings about digital costumes. Users see these low scores and think the app doesn't work well.

But speed-running isn't the whole story. To truly understand the relationship between user engagement and AI quality, we need to see the flip side. The perfect user. One who gives the AI everything it needed to succeed. What happens when a user takes their time and engages thoughtfully with every step?

Example: The perfect match

Session Replay Showed:

45 seconds on quiz (reading each option)
Uploaded photo, waited for processing
Spent 2 minutes on results page
Downloaded image multiple times

Online Evaluation Results:

Evaluation Score: 96/100 ⭐⭐⭐⭐⭐
Reasoning: "Personality perfectly matches role archetype"
Photo bonus: "Visual traits enhanced casting accuracy"

Time invested = Quality received. The AI rewards thoughtfulness.

Discovery #3: The photo upload comedy gold mine

Session replay revealed what photos people ACTUALLY upload. Without it, you'd never know that one in three photo uploads are problematic, and you'd be flying blind on whether to add validation or trust your model.

Example: The surprising photo upload analysis

Session Replay Showed:

Photo Upload Analysis (n=18 who uploaded):
- 12 (67%) Normal pet photos
- 2 (11%) Screenshots of pet photos on their phone
- 1 (6%) Multiple pets in one photo (chaos)
- 1 (6%) Blurry "pet in motion" disaster
- 1 (6%) Stock photo of their breed (cheater!)

Despite 33% problematic inputs, evaluation scores remained high (87-91/100). The AI is remarkably resilient.

Example: When "bad" photos produce great results

My Favorite Session: Someone uploaded a photo of their cat mid-yawn. The AI vision model described it as "displaying fierce predatory behavior." The cat was cast as a "Protective Father." Evaluation score: 91/100. The owner downloaded it immediately.

The Winner: Someone's hamster photo that was 90% cage bars. The AI somehow extracted "small fuzzy creature behind geometric patterns" and cast it as "Shepherd" because "clearly experienced at navigating barriers." Evaluation score: 87/100.

Without session replay, you'd only see evaluation scores and think "the AI is working well." But session replay reveals users are uploading screenshots and blurry photos—input quality issues that could justify adding photo validation.

However, the high evaluation scores prove the AI handles imperfect real-world data gracefully. This insight saved me from over-engineering photo validation that would have slowed down the user experience for minimal quality gains.

Session replay + online evaluations together answered the question "Should I add photo validation?" The answer: No. Trust the model's resilience and keep the experience frictionless.

The magic formula: Why this combo works (and what surprised me)

Without Observability:

"The app seems slow" → ¯\(ツ)/¯
"We have 20 visitors but 7 completions" → Where do they drop?

With Session Replay ONLY:

"User got sheep and rage clicked; maybe left angry" → Was this a bad match?

With Model-Agnostic Evaluation ONLY:

"Evaluation: 22/100 - Eyeliner unsafe for pets" → How did the user react?
"Evaluation: 96/100 - Perfect match!" → How did this compare to the image they uploaded?

With BOTH:

"User rushed, got sheep with ribbons, evaluation panicked about safety"
→ The OOTB evaluation treats image generation prompts like real costume instructions
"40% of low scores are costume safety, not bad matching"
→ Need custom evaluation criteria (coming soon!)
"Users might think low score = bad casting, but it's often = protective evaluation"
→ Would benefit from custom evaluation criteria to avoid this confusion

The evaluation thinks we're putting actual ribbons on actual cats. It doesn't realize these are AI-generated images. So when the casting suggests "sparkly collar with bells," the evaluation judge practically calls animal services.

Now that you've seen what's possible when you combine user behavior tracking with AI quality scoring, let's walk through how to add this same observability magic to your own multi-modal AI app.

Your turn: See the complete picture

Want to add this observability magic to your own app? Here's how:

1. Install the packages

npm install @launchdarkly/observability
npm install @launchdarkly/session-replay

2. Initialize with observability

import { initialize } from 'launchdarkly-js-client-sdk';
import Observability from '@launchdarkly/observability';
import SessionReplay from '@launchdarkly/session-replay';

const ldClient = initialize(clientId, user, {
  plugins: [
    new Observability(),
    new SessionReplay({
      privacySetting: 'strict' // Masks all data on the page - see https://launchdarkly.com/docs/sdk/features/session-replay-config#expand-javascript-code-sample
    })
  ]
});

3. Configure online evaluations in dashboard

Create your AI Config in LaunchDarkly for LLM evaluation
Enable automatic accuracy scoring for production monitoring

Set accuracy weight to 100% for production AI monitoring
Monitor your AI outputs with real-time evaluation scoring

4. Connect the dots

Session replay shows you:

Where users drop off
What confuses them
When they rage click
How long they wait

Online evaluations show you:

AI decision accuracy scores
Why certain outputs scored low
Pattern of good vs bad castings
Safety concerns (even for pixels!)

Together they reveal the complete story of your AI app.

Resources to get started:

Full Implementation Guide - See how this pet app implements both features

Session Replay Tutorial - Official LaunchDarkly guide for detecting user frustration

When to Add Online Evals - Learn when and how to implement AI evaluation

The real magic is in having observability AND online evaluations.

Try it yourself

Cast your pet: https://scarlett-critter-casting.onrender.com/

See your evaluation score ⭐. Understand why your cat is a shepherd and your dog is an angel. The AI has spoken, and now you can see exactly how much to trust it!

Ready to add AI observability to your multi-modal agents?

Don't let your AI operate in the dark this holiday season. Get complete visibility into your multi-modal AI systems with LaunchDarkly's online evaluations and session replay.

Get started: Sign up for a free trial → Create your first AI Config → Enable session replay and online evaluations → Ship with confidence.

Proving ROI with Data-Driven AI Agent Experiments

Scarlett Attensil — Mon, 20 Oct 2025 21:33:33 +0000

Published October 9th, 2025

by Scarlett Attensil

What You'll Learn in 5 Minutes (or Build in 30)

Key Findings from Our Experiments:

Unexpected discovery: Free Mistral model is not only \$0 but also significantly faster than Claude Haiku

Cost paradox revealed: "Free" security agent increased total system costs by forcing downstream agents to compensate

Premium model failure: Claude Opus 4 performed 64% worse than GPT-4o despite costing 33% more

Sample size reality: High-variance metrics (cost, feedback) require 5-10x more data than low-variance ones (latency)

The Problem

Your CEO asks: "Is the new expensive AI model worth it?"

Your finance team wonders: "Does the enhanced privacy justify the cost?"

Without experiments, you're guessing. This tutorial shows you how to measure the truth and sometimes discover unanticipated gains.

The Solution: Real Experiments, Real Answers

In 30 minutes, you'll run actual A/B tests that reveal:

Will aggressive PII redaction hurt user satisfaction?

Is Claude Opus 4 worth 33% more than GPT-4o?

Part 3 of 3 of the series: **Chaos to Clarity: Defensible AI Systems That Deliver on Your Goals**

Quick Start Options

Option 1: Just Want the Concepts? (5 min read)

Skip to Understanding the Experiments to learn the methodology without running code.

Option 2: Full Hands-On Tutorial (30 min)

Follow the complete guide to run your own experiments.

"Prerequisites for Hands-On Tutorial"

Required from Previous Parts:

Active LaunchDarkly project completed from Part 1 & Part 2
API keys: Anthropic, OpenAI, LaunchDarkly. Sign up for a free account here and then follow these instructions to get your API access token.

Investment:

Time: ~30 minutes
Cost: \$25-35 default (\$5-10 with --queries 50)

"Reduce Experiment Costs"

The default walk-through uses Claude Opus 4 (premium model) for testing. To reduce costs while still learning the experimentation patterns, you can modify bootstrap/tutorial_3_experiment_variations.py in your cloned repository to test with the free Mistral model instead:

In the create_premium_model_variations function, change:

# Original (expensive):
"model": {
    "name": "claude-opus-4-20250514",
    "provider": "anthropic"
}

# Change to (free Mistral):
"model": {
    "name": "mistral-small-latest",
    "provider": "mistral"
}

This reduces the experiment cost by about $20 (you'll still have costs from the control group using GPT-4o and other agents in the system).

How the Experiments Work

The Setup: Your AI system will automatically test variations on simulated users, collecting real performance data that flows directly to LaunchDarkly for statistical analysis.

The Process:

Traffic simulation generates queries from your actual knowledge base
Each user gets randomly assigned to experiment variations
AI responses are evaluated for quality and tracked for cost/speed
LaunchDarkly calculates statistical significance automatically

Note: The two experiments can run independently. Each user can participate in both, but the results are analyzed separately.

Experiment Methodology: Our supervisor agent routes PII queries to the security agent (then to support), while clean queries go directly to support. LaunchDarkly tracks metrics at the user level across all agents, revealing system-wide effects.

Understanding Your Two Experiments

Experiment 1: Security Agent Analysis

Question: Does Strict Security (free Mistral model with aggressive PII redaction) improve performance without harming user experience or significantly increasing system costs?

Variations (50% each):

Control: Basic Security (Claude Haiku, moderate PII redaction)
Treatment: Strict Security (Mistral free, aggressive PII redaction)

Success Criteria:

Positive feedback rate: stable or improving (not significantly worse)
Cost increase: ≤15% with ≥75% confidence
Latency increase: ≤3 seconds (don't significantly slow down)
Enhanced privacy protection delivered

Experiment 2: Premium Model Value Analysis

Question: Does Claude Opus 4 justify its premium cost vs GPT-4o?

Variations (50% each):

Control: GPT-4o with full tools (current version)
Treatment: Claude Opus 4 with identical tools

Success Criteria (must meet 90% threshold):

≥15% positive feedback rate improvement by Claude Opus 4
Cost-value ratio ≥ 0.25 (positive feedback rate gain % ÷ cost increase %)

Setting Up Metrics and Experiments

Step 1: Configure Metrics (5 minutes)

Quick Metric Setup

Navigate to Metrics in LaunchDarkly and create three custom metrics:

Metric Name	Event Key	Type	What It Measures
p95_total_user_latency	$ld:ai:duration:total	P95	Response speed
average_total_user_tokens	$ld:ai:tokens:total	Average	Token usage
ai_cost_per_request	ai_cost_per_request	Average	Dollar cost
Positive Feedback	Built-in	Rate	Positive feedback rate
Negative Feedback	Built-in	Rate	User complaints

"See detailed setup for P95 Latency"

Event key: $ld:ai:duration:total
Type: Value/Size → Numeric, Aggregation: Sum
Definition: P95, value, user, sum, "lower is better"
Unit: ms, Name: p95_total_user_latency

"View other metric configurations"

Tokens: Event key $ld:ai:tokens:total, Name: average_total_user_tokens, Average aggregation
Cost: Event key ai_cost_per_request, Name: ai_cost_per_request, Average in dollars

The cost tracking is implemented in utils/cost_calculator.py, which calculates actual dollar costs using the formula (input_tokens × input_price + output_tokens × output_price) / 1M. The system has pre-configured pricing for each model (as of October 2025): GPT-4o at \$2.50/\$10 per million tokens, Claude Opus 4 at \$15/\$75, and Claude Sonnet at \$3/\$15. When a request completes, the cost is immediately calculated and sent to LaunchDarkly as a custom event, enabling direct cost-per-user analysis in your experiments.

Step 2: Create Experiment Variations

Create the experiment variations using the bootstrap script:

uv run python bootstrap/tutorial_3_experiment_variations.py

This creates the claude-opus-treatment variation for the Premium Model Value experiment. To verify the script worked correctly, navigate to your support-model-config feature flag in LaunchDarkly - you should now see the claude-opus-treatment variation alongside your existing variations. The Security Agent Analysis experiment will use your existing baseline and enhanced variations. Both experiments use the existing other-paid configuration as their control group.

Step 3: Configure Security Agent Experiment

"Click for details"

Navigate to AI Configs → security-agent. In the right navigation menu, click the plus (+) sign next to Experiments to create a new experiment

Experiment Design

Experiment type:

Keep Feature change selected (default)

Name: Security Level

Hypothesis and Metrics

Hypothesis: Enhanced security improves safety compliance without significantly harming positive feedback rates

Randomize by: user

Metrics: Click "Select metrics or metric groups" and add:

Positive feedback rate → Select first to set as Primary
Negative feedback rate
p95_total_user_latency
ai_cost_per_request

Audience Targeting

Flag or AI Config

Click the dropdown and select security-agent

Targeting rule:

Click the dropdown and select Rule 4
This will configure: If Context → is in Segment → Other Paid

Audience Allocation

Variations served outside of this experiment:

Basic Security

Sample size: Set to 100% of users in this experiment

Variations split: Click "Edit" and configure:

Note: Before setting these percentages, scroll down to the Control field below and set Basic Security as the control variation first, otherwise you won't be able to allocate 50% traffic to it.

pii-detector: 0%
Basic Security: 50%
Strict Security: 50%

Control:

Basic Security

Statistical Approach and Success Criteria

Statistical approach: Bayesian
Threshold: 90%

Click "Save"
Click "Start experiment" to launch.

Note: You may see a "Health warning" indicator after starting the experiment. This is normal and expected when no variations have been exposed yet. The warning will clear once your experiment starts receiving traffic and data begins flowing.

Step 4: Configure Premium Model Experiment

"Click for details"

Navigate to AI Configs → support-agent. In the right navigation menu, click the plus (+) sign next to Experiments to create a new experiment

Experiment Design

Experiment type:

Keep Feature change selected (default)

Name: Premium Model Value Analysis

Hypothesis and Metrics

Hypothesis: Claude Opus 4 justifies premium cost with superior positive feedback rate

Randomize by: user

Metrics: Click "Select metrics or metric groups" and add:

Positive feedback rate → Select first to set as Primary
Negative feedback rate
p95_total_user_latency
average_total_user_tokens
ai_cost_per_request

Audience Targeting

Flag or AI Config

Click the dropdown and select support-agent

Targeting rule:

Click the dropdown and select Rule 4
This will configure: If Context → is in Segment → Other Paid

Audience Allocation

Variations served outside of this experiment:

other-paid

Sample size: Set to 100% of users in this experiment

Variations split: Click "Edit" and configure:

Note: Before setting these percentages, scroll down to the Control field below and set other-paid as the control variation first, otherwise you won't be able to allocate 50% traffic to it.

rag-search-enhanced: 0%
eu-free: 0%
eu-paid: 0%
other-free: 0%
other-paid: 50%
international-standard: 0%
claude-opus-treatment: 50%

Control:

other-paid

Statistical Approach and Success Criteria

Statistical approach: Bayesian
Threshold: 90%

Click "Save"
Click "Start experiment" to launch.

Understanding Your Experimental Design

If Two Independent Experiments Running Concurrently:

Since these are the same users, each user experiences:

One security variation (Basic Security or Strict Security)
One model variation (Claude Opus 4 Treatment OR Other Paid (GPT-4o))

Random assignment ensures balance: ~50 users get each combination naturally.

Generating Experiment Data

Step 5: Run Traffic Generator

Start your backend and generate realistic experiment data. Choose between sequential or concurrent traffic generation:

Concurrent Traffic Generator (Recommended for large datasets)

For faster experiment data generation with parallel requests:

# Start backend API
uv run uvicorn api.main:app --reload --port 8000

# Generate experiment data with 10 concurrent requests (separate terminal)
uv run python -u tools/concurrent_traffic_generator.py --queries 200 --concurrency 10

Configuration:

200 queries by default (edit script to adjust)
10 concurrent requests running in parallel
2000-second timeout (33 minutes) per request to handle MCP tool rate limits

Note: runtime depends largely on if you retain MCP tool enablement as these take much longer to complete.

"For smaller test runs or debugging"

Sequential Traffic Generator (Simple, one-at-a-time)

# Start backend API
uv run uvicorn api.main:app --reload --port 8000

# Generate experiment data sequentially (separate terminal)
uv run python tools/traffic_generator.py --queries 50 --delay 2

What Happens During Simulation:

Knowledge extraction
Claude analyzes your docs and identifies 20+ realistic topics
Query generation
Each test randomly selects from these topics for diversity
AI-powered evaluation
Claude judges responses as thumbs_up/thumbs_down/neutral
Automatic tracking
All metrics flow to LaunchDarkly in real-time

Generation Output:

📚 Analyzing knowledge base...
✅ Generated 23 topics

⚡ Sending 200 requests with 10 concurrent workers...

✅ [1/200] Success (23.4s) - other_paid: What is reinforcement learning?...
✅ [2/200] Success (45.2s) - other_paid: How does Q-learning work?...
⏱️  [15/200] Timeout (>2000s) - other_paid: Complex research query...
                              ↑ This is normal - MCP rate limits
✅ [200/200] Success (387.1s) - other_paid: Explain temporal difference...

======================================================================
✅ COMPLETE
======================================================================
Total time: 45.3 minutes (2718s)
Successful: 195/200 (97.5%)
Failed: 5/200 (2.5%)
Average: 13.6s per query (with concurrency)

Performance Notes:

Most queries complete in 10-60 seconds
Queries using semantic_scholar MCP tool may take 5-20 minutes due to API rate limits
Concurrent execution handles slow requests gracefully by continuing with others
Failed/timeout requests (less than 5% typically) don't affect experiment validity

Monitor Results: Refresh your LaunchDarkly experiment "Results" tabs to see data flowing in. Cost metrics appear as custom events alongside feedback and token metrics.

Interpreting Your Results (After Data Collection)

Once your experiments have collected data from ~100 users per variation, you'll see results in the LaunchDarkly UI. Here's how to interpret them:

1. Security Agent Analysis: Does enhanced security improve safety without significantly impacting positive feedback rates?

Reality Check: Not all metrics reach significance at the same rate. In our security experiment we ran over 2,000 more users than the model experiment, turning off the MCP tools and using --pii-percentage 100 to maximize pii detection:

Latency: 87% confidence (nearly significant, clear 36% improvement)
Cost: 21% confidence (high variance, needs 5-10x more data)
Feedback: 58% confidence (sparse signal, needs 5-10x more data)

This is normal! Low-variance metrics (latency, tokens) prove out quickly. High-variance metrics (cost, feedback) need massive samples. You may not be able to wait for every metric to hit 90%. Use strong signals on some metrics plus directional insights on others.

✅ VERDICT: Deploy Strict Security: Enhanced Privacy is Worth the Modest Cost

The results tell a compelling story: Latency (p95) is approaching significance with 87% confidence that Strict Security is faster, a win we didn't anticipate. Cost per request shows 79% confidence that Basic Security costs less (or conversely, 21% confidence that Strict costs more), also approaching significance. Meanwhile, positive feedback rate remains inconclusive with only 58% confidence that Strict Security performs better, indicating we need more data to draw conclusions about user satisfaction.

The Hidden Cost Paradox:

Strict Security uses FREE Mistral for PII detection, yet increases total system cost by 14.6%.

Basic Security (Claude Haiku):
- Supervisor: gpt-4o-mini     ~\$0.0001
- Security:   claude-haiku    ~\$0.0003
- Support:    gpt-4o          ~\$0.0242
Total: \$0.0246

Strict Security (Mistral):
- Supervisor: gpt-4o-mini     ~\$0.0001
- Security:   mistral         \$0.0000  (FREE!)
- Support:    gpt-4o          ~\$0.0280  (+15.7%)
Total: \$0.0281 (+14.6%)

Why does the support agent cost more? More aggressive PII redaction removes context, forcing the support agent to generate longer, more detailed responses to compensate for the missing information. This demonstrates why system-level experiments matter. Optimizing one agent can inadvertently increase costs downstream.

Decision Logic:

IF latency increase ≤ 3s
   AND cost increase ≤ 15% AND confidence ≥ 75%
   AND positive_feedback_rate stable or improving
   AND enhanced_privacy_protection = true
THEN deploy_strict_security()
ELSE need_more_data()

Bottom line: Deploy Strict Security. We expected latency to stay within 3 seconds of baseline, but discovered a 36% improvement instead (87% confidence). Mistral is significantly faster than Claude Haiku. Combined with enhanced privacy protection, this more than justifies the modest 14.5% cost increase (79% confidence).

Read across: At scale, paying ~\$0.004 more per request for significantly better privacy compliance and faster responses is a clear win for EU users and privacy-conscious segments.

The Data That Proves It:

2. Premium Model Value Analysis: Does Claude Opus 4 justify its premium cost with superior positive feedback rates?

🔴 VERDICT: Reject Claude Opus 4

The experiment delivered a decisive verdict: Positive feedback rate showed a significant failure with 99.5% confidence that GPT-4o is superior. Cost per request is approaching significance with 76% confidence that Claude Opus is 33% more expensive, while latency (p95) reached significance with 91% confidence that Claude Opus is 81% slower. The cost-to-value ratio tells the whole story: -1.9x, meaning we're paying 33% more for 64% worse performance: a clear case of premium pricing without premium results.

Decision Logic:

IF positive_feedback_rate increase ≥ 15%
   AND probability_to_beat for positive_feedback_rate ≥ 90%
   AND probability_to_beat for cost ≥ 90%
   AND cost-value ratio increase ≥ .25
THEN deploy_claude_opus_4()
ELSE keep_current_model()

Bottom line: Premium price delivered worse results on every metric. Experiment was stopped when positive feedback rate reached significance.

Read across: GPT-4o dominates on performance and speed and most likely also on cost.

The Numbers Don't Lie:

Key Insights from Real Experiment Data

Low-variance metrics (latency, tokens) reach significance quickly (~1,000 samples). High-variance metrics (cost, feedback) may need 5,000-10,000+ samples. This isn't a flaw in your experiment but the reality of statistical power. Don't chase 90% confidence on every metric; focus on directional insights for high-variance metrics and statistical proof for low-variance ones.
Using a free Mistral model for security reduced that agent's cost to \$0, yet increased total system cost by 14.5% because downstream agents had to work harder with reduced context. However, the experiment also revealed an unexpected 36% latency improvement. Mistral is not just free but significantly faster. LaunchDarkly's user-level tracking captured both effects, enabling an informed decision: enhanced privacy + faster responses for ~\$0.004 more per request is a worthwhile tradeoff.
At 87% confidence for latency (vs 90% target), the 36% improvement is clear enough for decision-making. Perfect statistical significance is ideal, but 85-89% confidence combined with other positive signals (stable feedback, acceptable cost) can justify deployment when the effect size is large.

Experimental Limitations & Mitigations

Model-as-Judge Evaluation

We use Claude to evaluate response quality rather than real users, which represents a limitation of this experimental setup. However, research shows that model-as-judge approaches correlate well with human preferences, as documented in Anthropic's Constitutional AI paper.

Independent Experiments

While random assignment naturally balances security versions across model versions, preventing systematic bias, you cannot analyze interaction effects between security and model choices. If interaction effects are important to your use case, consider running a proper factorial experiment design.

Statistical Confidence
LaunchDarkly uses Bayesian statistics to calculate confidence, where 90% confidence means there's a 90% probability the true effect is positive. This is NOT the same as p-value < 0.10 from frequentist tests. We set the threshold at 90% (rather than 95%) to balance false positives versus false negatives, though for mission-critical features you should consider raising the confidence threshold to 95%.

Common Mistakes You Just Avoided

❌ "Let's run the experiment for a week and see"

✅ We defined success criteria upfront (≥15% improvement threshold)

❌ "We need 90% confidence on every metric to ship"

✅ We used 87% confidence + directional signals (36% latency win was decision-worthy)

❌ "Let's run experiments until all metrics reach significance"

✅ We understood variance (cost/feedback need 5-10x more data than latency)

❌ "Agent-level metrics show the full picture"

✅ We tracked user-level workflows (revealed downstream cost increases)

What You've Accomplished

You've built a data-driven optimization engine with statistical rigor through falsifiable hypotheses and clear success criteria. Your predefined success criteria ensure clear decisions and prevent post-hoc rationalization. Every feature investment now has quantified business impact for ROI justification, and you have a framework for continuous optimization through ongoing measurable experimentation.

Troubleshooting

Long Response Times (>20 minutes)

If you see requests taking exceptionally long, the root cause is likely the semantic_scholar MCP tool hitting API rate limits, which causes 30-second retry delays. Queries using this tool may take 5-20 minutes to complete. The 2000-second timeout handles this gracefully, but if you need faster responses (60-120 seconds typical), consider removing semantic_scholar from tool configurations. You can verify this issue by checking logs for HTTP/1.1 429 errors indicating rate limiting.

Cost Metrics Not Appearing

If ai_cost_per_request events aren't showing in LaunchDarkly, first verify that utils/cost_calculator.py has pricing configured for your models. Cost is only tracked when requests complete successfully (not on timeout or error). The system flushes cost events to LaunchDarkly immediately after each request completion. To debug, look for COST CALCULATED: and COST TRACKING (async): messages in your API logs.

Beyond This Tutorial: Advanced AI Experimentation Patterns

Other AI experimentation types you can run in LaunchDarkly

Context from earlier: you ran two experiments:

Security‑agent test: a bundle change (both prompt/instructions and model changed).
Premium‑model test: a model‑only change.

AI Configs come in two modes: prompt‑based (single‑step completions) and agent‑based (multi‑step workflows with tools). Below are additional patterns to explore.

Experiments you can run entirely in AI Configs (no app redeploy)

Prompt & template experiments (prompt‑based or agent instructions)
Duplicate a variation and iterate on system/assistant messages or agent instructions to measure adherence to schema, tone, or qualitative satisfaction. Use LaunchDarkly Experimentation to tie those variations to product metrics.
Model‑parameter experiments
In a single model, vary parameters like temperature or max_tokens, and (optionally) add custom parameters you define (for example, an internal max_tool_calls or decoding setting) directly on the variation.
Tool‑bundle experiments (agent mode or tool‑enabled completions)
Attach/detach reusable tools from the Tools Library to compare stacks (e.g., search_v2, a reranker, or MCP‑exposed research tools) across segments. Keep one variable at a time when possible
Cost/latency trade‑offs
Compare "slim" vs "premium" stacks by segment. Track tokens, time‑to‑first‑token, duration, and satisfaction to decide where higher spend is warranted.

Practical notes

Use Experimentation for behavior impact (clicks, task success); use the Monitoring tab for LLM‑level metrics (tokens, latency, errors, satisfaction).

- You can't run a guarded rollout and an experiment on the same flag at the same time. Pick one: guarded rollout for risk‑managed releases, experiment for causal measurement.

Patterns that usually need feature flags and/or custom instrumentation

Fine‑grained RAG tuning
k‑values, similarity thresholds, chunking, reranker swaps, and cache policy are typically coded inside your retrieval layer. Expose these as flags or AI Config custom parameters if you want to A/B them.
Tool‑routing guardrails
Fallback flows (e.g., retry with a different tool/model on error), escalation rules, or heuristics need logic in your agent/orchestrator. Gate those behaviors behind flags and measure with custom metrics.
Safety guardrail calibration
Moderation thresholds, red‑team prompts, and PII sensitivity levers belong in a dedicated safety service or the agent wrapper. Wire them to flags so you can raise/lower sensitivity by segment (e.g., enterprise vs free).
Session budget enforcement
Monitoring will show token costs and usage, but enforcing per‑session or per‑org budgets (denylist, degrade model, or stop‑tooling) requires application logic. Wrap policies in flags before you experiment.

Targeting & segmentation ideas (works across all the above)

Route by plan/tier, geo, device, or org using AI Config targeting rules and percentage rollouts.
Keep variations narrow (one change per experiment) to avoid confounding; reserve "bundle" tests for tool‑stack bake‑offs.

Advanced Practices: Require statistical evidence before shipping configuration changes. Pair each variation with clear success metrics, then A/B test prompt or tool adjustments and use confidence intervals to confirm improvements. When you introduce the new code paths above, protect them behind feature flags so you can run sequential tests, multi-armed bandits for faster convergence, or change your experiment design to understand how prompts, tools, and safety levers interact.

From Chaos to Clarity

Across this three-part series, you've transformed from hardcoded AI configurations to a scientifically rigorous, data-driven optimization engine. Part 1 established your foundation with a dynamic multi-agent LangGraph system controlled by LaunchDarkly AI Configs, eliminating the need for code deployments when adjusting AI behavior. Part 2 added sophisticated targeting with geographic privacy rules, user segmentation by plan tiers, and MCP (Model Context Protocol) tool integration for real academic research capabilities. This tutorial, Part 3 completed your journey with statistical experimentation that proves ROI and guides optimization decisions with mathematical confidence rather than intuition.

You now possess a defensible AI system that adapts to changing requirements, scales across user segments, and continuously improves through measured experimentation. Your stakeholders receive concrete evidence for AI investments, your engineering team deploys features with statistical backing, and your users benefit from optimized experiences driven by real data rather than assumptions. The chaos of ad-hoc AI development has given way to clarity through systematic, scientific product development.

Resources

- LaunchDarkly Experimentation Docs - Deep dive into statistical methods

Remember: Every AI decision backed by data is a risk avoided and a lesson learned. Start small, measure everything, ship with confidence.

Smart AI Agent Targeting with MCP Tools

Scarlett Attensil — Mon, 20 Oct 2025 21:09:42 +0000

Originally published September 22, 2025 at LaunchDarkly.

Overview

Here's what nobody tells you about multi-agentic systems: the hard part isn't building them but making them profitable. One misconfigured model serving enterprise features to free users can burn $20K in a weekend. Meanwhile, you're manually juggling dozens of requirements for different user tiers, regions, and privacy compliance and each one is a potential failure point.

Part 2 of 3 of the series: Chaos to Clarity: Defensible AI Systems That Deliver on Your Goals

The solution? LangGraph multi-agent workflows controlled by LaunchDarkly AI Config targeting rules that intelligently route users: paid customers get premium tools and models, free users get cost-efficient alternatives, and EU users get Mistral for enhanced privacy. Use the LaunchDarkly REST API to set up a custom variant-targeting matrix in 2 minutes instead of spending hours setting it up manually.

What You'll Build Today

In the next 18 minutes, you'll transform your basic multi-agent system with:

Business Tiers & MCP Integration: Free users get internal keyword search, Paid users get premium models with RAG, external research tools and expanded tool call limits, all controlled by LaunchDarkly AI Configs
Geographic Targeting: EU users automatically get Mistral and Claude models (enhanced privacy), other users get cost-optimized alternatives
Smart Configuration: Set up complex targeting matrices with LaunchDarkly segments and targeting rules

Prerequisites

✅ Part 1 completed with exact naming:

Project: multi-agent-chatbot
AI Configs: supervisor-agent, security-agent, support-agent
Tools: search_v2, reranking
Variations: supervisor-basic, pii-detector, rag-search-enhanced

🔑 Add to your .env file:

LD_API_KEY=your-api-key        # Get from LaunchDarkly settings
MISTRAL_API_KEY=your-key       # Get from console.mistral.ai (free, requires phone + email validation)

Getting Your LaunchDarkly API Key

The automation scripts in this tutorial use the LaunchDarkly REST API to programmatically create configurations. Here's how to get your API key:

To get your LaunchDarkly API key, start by navigating to Organization Settings by clicking the gear icon (⚙️) in the left sidebar of your LaunchDarkly dashboard. Once there, access Authorization Settings by clicking "Authorization" in the settings menu. Next, create a new access token by clicking "Create token" in the "Access tokens" section.

When configuring your token, give it a descriptive name like "multi-agent-chatbot", select "Writer" as the role (required for creating configurations), use the default API version (latest), and leave "This is a service token" unchecked for now.

After configuring the settings, click "Save token" and immediately copy the token value. This is IMPORTANT because it's only shown once!

Finally, add the token to your environment:

   # Add this line to your .env file
   LD_API_KEY=your-copied-api-key-here

Security Note: Keep your API key private and never commit it to version control. The token allows full access to your LaunchDarkly account.

Step 1: Add External Research Tools (4 minutes)

Your agents need more than just your internal documents. Model Context Protocol (MCP) connects AI assistants to live external data and they agents become orchestrators of your digital infrastructure, tapping into databases, communication tools, development platforms, and any system that matters to your business. MCP tools run as separate servers that your agents call when needed.

The MCP Registry serves as a community-driven directory for discovering available MCP servers - like an app store for MCP tools. For this tutorial, we'll use manual installation since our specific academic research servers (ArXiv and Semantic Scholar) aren't yet available in the registry.

Install external research capabilities:

# Install ArXiv MCP server for academic paper search
uv tool install arxiv-mcp-server

# Install Semantic Scholar MCP server for citation data
git clone https://github.com/JackKuo666/semanticscholar-MCP-Server.git

MCP Tools Added:

arxiv_search: Live academic paper search (Paid users)
semantic_scholar: Citation and research database (Paid users)

These tools integrate with your agents via LangGraph while LaunchDarkly controls which users get access to which tools.

Step 2: Configure with API Automation (2 minutes)

Now we'll use programmatic API automation to configure the complete setup. The LaunchDarkly REST API lets you manage tools, segments, and AI Configs programmatically. Instead of manually creating dozens of variations in the UI, this configuration automation makes REST API calls to provision user segments, AI Config variations, targeting rules, and tools. These are the same resources you could create manually through the LaunchDarkly dashboard. Your actual chat application continues running unchanged.

Configure your complete targeting matrix with one command:

cd bootstrap
uv run python create_configs.py

What the script creates:

3 new tools: search_v1 (basic search), arxiv_search and semantic_scholar (MCP research tools)
4 combined user segments with geographic and tier targeting rules
Updated AI Configs: security-agent with 2 new geographic variations
Complete targeting rules that route users to appropriate variations
Intelligently reuses existing resources: supervisor-agent, search_v2, and reranking tools from Part 1

Understanding the Bootstrap Script

The automation works by reading a YAML manifest and translating it into LaunchDarkly API calls. Here's how the key parts work:

Segment Creation with Geographic Rules:

def create_segment(self, project_key, segment_data):
    # Step 1: Create empty segment
    payload = {
        "key": segment_data["key"],
        "name": segment_data["key"].replace("-", " ").title()
    }

    # Step 2: Add targeting rules via semantic patch
    clauses = []
    for clause in segment_data["rules"]:
        clauses.append({
            "attribute": clause["attribute"],  # "country" or "plan"
            "op": clause["op"],              # "in"
            "values": clause["values"],      # ["DE", "FR", ...] or ["free"]
            "contextKind": "user",
            "negate": clause["negate"]       # false for EU, true for non-EU
        })

Model Configuration Mapping:

# The script maps your YAML model IDs to LaunchDarkly's internal keys
model_config_key_map = {
    "claude-3-5-sonnet-20241022": "Anthropic.claude-3-7-sonnet-latest",
    "claude-3-5-haiku-20241022": "Anthropic.claude-3-5-haiku-20241022", 
    "gpt-4o": "OpenAI.gpt-4o",
    "gpt-4o-mini": "OpenAI.gpt-4o-mini-2024-07-18",
    "mistral-small-latest": "Mistral.mistral-small-latest"
}

Customizing for Your Use Case:

To adapt this for your own multi-agent system:

Add your geographic regions in the YAML segments:

   - key: apac-paid
     rules:
       - attribute: "country" 
         values: ["JP", "AU", "SG", "KR"]  # Your APAC countries

Define your business tiers:

   - attribute: "plan"
     values: ["enterprise", "professional", "starter"]  # Your pricing tiers

Map your models in the script:

   "your-model-id": "Provider.your-launchdarkly-key"

The script handles the complexity of LaunchDarkly's API while letting you define your targeting logic in simple YAML.

Validating the Bootstrap Script

Expected terminal output:

🚀 LaunchDarkly AI Config Bootstrap
==================================================
⚠️  IMPORTANT: This script is for INITIAL SETUP ONLY
📝 After bootstrap completes:
   • Make ALL configuration changes in LaunchDarkly UI
   • Do NOT modify ai_config_manifest.yaml
   • LaunchDarkly is your single source of truth
==================================================

🚀 Starting multi-agent system bootstrap (add-only)...
📦 Project: multi-agent-chatbot

🔧 Creating tools...
  ✅ Tool 'search_v1' created
  ✅ Tool 'arxiv_search' created
  ✅ Tool 'semantic_scholar' created

🤖 Ensuring AI configs exist...
✅ AI Config 'supervisor-agent' exists
✅ AI Config 'security-agent' exists
✅ AI Config 'support-agent' exists

🧩 Creating variations...
  ✅ Variation 'strict-security' created
  ✅ Variation 'eu-free' created
  ✅ Variation 'eu-paid' created
  ✅ Variation 'other-free' created
  ✅ Variation 'other-paid' created

📦 Creating segments (for targeting rules)...
✅ Empty segment 'eu-free' created
  ✅ Rules added to segment 'eu-free' (final count: 1)
✅ Empty segment 'eu-paid' created
  ✅ Rules added to segment 'eu-paid' (final count: 1)
✅ Empty segment 'other-free' created
  ✅ Rules added to segment 'other-free' (final count: 1)
✅ Empty segment 'other-paid' created
  ✅ Rules added to segment 'other-paid' (final count: 1)

🎯 Updating targeting rules...
✅ Targeting rules updated for 'security-agent'
✅ Targeting rules updated for 'support-agent'

✨ Bootstrap complete!

In your LaunchDarkly dashboard, navigate to your multi-agent-chatbot project. You should see:

AI Configs tab: Three configs (supervisor-agent, security-agent, support-agent) with new variations
Segments tab: Four new segments (eu-free, eu-paid, other-free, other-paid)
Tools tab: Five tools total (including search_v1, arxiv_search, semantic_scholar)

Troubleshooting Common Issues:

❌ Error: "LD_API_KEY environment variable not set"

Check your .env file contains: LD_API_KEY=your-api-key
Verify the API key has "Writer" permissions in LaunchDarkly settings

❌ Error: "AI Config 'security-agent' not found"

Ensure you completed Part 1 with exact naming requirements
Verify your project is named multi-agent-chatbot
Check that supervisor-agent, security-agent, and support-agent exist in your LaunchDarkly project

❌ Error: "Failed to create segment"

Your LaunchDarkly account needs segment creation permissions
Try running the script again; it's designed to handle partial failures

❌ Script runs but no changes appear

Wait 30-60 seconds for LaunchDarkly UI to refresh
Check you're looking at the correct project and environment (Production)
Verify your API key matches your LaunchDarkly organization

Step 3: See How Smart Segmentation Works (2 minutes)

Here's how the smart segmentation works:

By Region:

EU users: Mistral for security processing + Claude for support (privacy + compliance)
Non-EU users: Claude for security + GPT for support (cost optimization)
All users: Claude for supervision and workflow orchestration

By Business Tier:

Free users: Basic search tools (search_v1)
Paid users: Full research capabilities (search_v1, search_v2, reranking, arxiv_search, semantic_scholar)

Step 4: Test Segmentation with Script (2 minutes)

The included test script simulates real user scenarios across all segments, verifying that your targeting rules work correctly. It sends actual API requests to your system and confirms each user type gets the right model, tools, and behavior.

First, start your system:

# Terminal 1: Start the backend
uv run uvicorn api.main:app --reload --port 8000

# Terminal 2: Run the test script
uv run python api/segmentation_test.py

Expected test output:

🚀 COMPREHENSIVE TUTORIAL 2 SEGMENTATION TESTS
Testing Geographic + Business Tier Targeting Matrix
======================================================================

🔄 Running: EU Paid → Claude Sonnet + Full MCP Tools

============================================================
🧪 TESTING: DE paid user (ID: user_eu_paid_001)
============================================================
📊 SUPPORT AGENT:
   Model: claude-3-7-sonnet-latest (expected: claude-3-7-sonnet-latest) ✅
   Variation: eu-paid (expected: eu-paid) ✅
   Tools: ['search_v1', 'search_v2', 'reranking', 'arxiv_search', 'semantic_scholar'] ✅
   Expected: ['search_v1', 'search_v2', 'reranking', 'arxiv_search', 'semantic_scholar']
   MCP Tools: Yes (should be: Yes) ✅

📝 RESPONSE:
   Length: 847 chars
   Tools Called: ['search_v2', 'arxiv_search']
   Preview: Based on your request, I'll search both internal documentation and recent academic research...

🎯 RESULT: ✅ PASSED

🔄 Running: EU Free → Claude Haiku + Basic Tools
[Similar detailed output for EU Free user...]

🔄 Running: US Paid → GPT-4 + Full MCP Tools  
[Similar detailed output for US Paid user...]

🔄 Running: US Free → GPT-4o Mini + Basic Tools
[Similar detailed output for US Free user...]

======================================================================
📊 FINAL RESULTS
======================================================================
✅ PASSED: 4/4
❌ FAILED: 0/4

🎉 ALL TESTS PASSED! LaunchDarkly targeting is working correctly.
   • Geographic segmentation: Working
   • Business tier routing: Working
   • Model assignment: Working
   • Tool configuration: Working
   • MCP integration: Working

🔗 Next: Test manually in UI at http://localhost:8501

This confirms your targeting matrix is working correctly across all user segments!

Step 5: Experience Segmentation in the Chat UI (3 minutes)

Now let's see your segmentation in action through the user interface. With your backend already running from Step 4, start the UI:

# Terminal 3: Start the chat interface
uv run streamlit run ui/chat_interface.py --server.port 8501

Open http://localhost:8501 and test different user types:

User Dropdown: Find the user dropdown by using the >> icon to open the left nav menu.. Select different regions (eu, other) and plans (Free, Paid).
Ask Questions: Try "Search for machine learning papers."
Watch Workflow: In the server logs, watch which model and tools get used for each user type.
Verify Routing: EU users get Mistral for security. Other users get GPT. Paid users get MCP tools.

What's Next: Part 3 Preview

In Part 3, we'll prove what actually works using controlled A/B experiments:

Set up Easy Experiments

Tool Implementation Test: Compare search_v1 vs search_v2 on identical models to measure search quality impact
Model Efficiency Analysis: Test models with the same full tool stack to measure tool-calling precision and cost

Real Metrics You'll Track

User satisfaction: thumbs up/down feedback
Tool call efficiency: average number of tools used per successful query
Token cost analysis: cost per query across different model configurations
Response latency: performance impact of security and tool variations

Instead of guessing which configurations work better, you'll have data proving which tool implementations provide value, which models use tools more efficiently, and what security enhancements actually costs in performance.

The Path Forward

You've built something powerful: a multi-agent system that adapts to users by design. More importantly, you've proven that sophisticated AI applications don't require repeated deployments; they require smart configuration.

This approach scales beyond tutorials. Whether you're serving 100 users or 100,000, the same targeting principles apply: segment intelligently, configure dynamically, and let data guide decisions instead of assumptions.

Questions? Issues? Reach out at aiproduct@launchdarkly.com or open an issue in the GitHub repo.

Build a LangGraph Multi-Agent system in 20 Minutes with LaunchDarkly AI Configs

Scarlett Attensil — Fri, 19 Sep 2025 16:47:30 +0000

Originally published on the LaunchDarkly Documentation

Overview

Build a working multi-agent system with dynamic configuration in 20 minutes using LangGraph multi-agent workflows, RAG search, and LaunchDarkly AI Configs.

Part 1 of 3 of the series: Chaos to Clarity: Defensible AI Systems That Deliver on Your Goals

You've been there: your AI chatbot works great in testing, then production hits and GPT-4 costs spiral out of control. You switch to Claude, but now European users need different privacy rules. Every change means another deploy, more testing, and crossed fingers that nothing breaks.

The teams shipping faster? They control AI behavior dynamically instead of hardcoding everything.

This series shows you how to build LangGraph multi-agent workflows that get their intelligence from RAG search through your business documents, enhanced with MCP tools for live external data, all controlled through LaunchDarkly AI Configs without needing to deploy code changes.

What This Series Covers

Part 1 (this post): Build a working multi-agent system with dynamic configuration in 20 minutes
Part 2: Add advanced features like segment targeting, MCP tool integration, and cost optimization
Part 3: Run production A/B experiments to prove what actually works

By the end, you'll have a system that measures its own performance and adapts based on user data instead of guesswork.

What You'll Build Today

In the next 20 minutes, you'll have a LangGraph multi-agent system with:

Supervisor Agent: Orchestrates workflow between specialized agents
Security Agent: Detects PII and sensitive information
Support Agent: Answers questions using your business documents
Dynamic Control: Change models, tools, and behavior through LaunchDarkly without code changes

Prerequisites

You'll need:

Python 3.9+ with uv package manager (install uv)
LaunchDarkly account (sign up for free)
OpenAI API key (required for RAG architecture embeddings)
Anthropic API key (required for Claude models) or OpenAI API key (for GPT models)

Step 1: Clone and Configure (2 minutes)

First, let's get everything running locally. We'll explain what each piece does as we build.

# Get the code
git clone https://github.com/launchdarkly-labs/devrel-agents-tutorial
cd agents-demo

# Install dependencies (LangGraph, LaunchDarkly SDK, etc.)
uv sync

# Configure your environment
cp .env.example .env

First, you need to get your LaunchDarkly SDK key by creating a project:

Sign up for LaunchDarkly at app.launchdarkly.com (free account).

If you're a brand new user, after signing up for an account, you'll need to verify your email address. You can skip through the new user onboarding flow after that.

Find projects on the side bar
Create a new project called "multi-agent-chatbot"
Get your SDK key:

⚙️ (bottom of sidebar) → Projects → multi-agent-chatbot → ⚙️ (to the right)

→ Environments → Production → ... → SDK key

this is your LD_SDK_KEY

Now edit .env with your keys:

LD_SDK_KEY=your-launchdarkly-sdk-key  # From step above
OPENAI_API_KEY=your-openai-key        # Required for RAG embeddings
ANTHROPIC_API_KEY=your-anthropic-key  # Required for Claude models

This sets up a LangGraph application that uses LaunchDarkly to control AI behavior. Think of it like swapping actors, directors, even props mid-performance without stopping the show.

⚠️ Do not check the .env into your source control. Keep those secrets safe!

Step 2: Add Your Business Knowledge (2 minutes)

The system includes a sample reinforcement learning textbook. Replace it with your own documents for your specific domain.

# Option A: Use the sample (AI/ML knowledge)
# Already included: kb/SuttonBarto-IPRL-Book2ndEd.pdf

# Option B: Add your documents
rm kb/*.pdf  # Clear sample
cp /path/to/your-docs/*.pdf kb/

Document types that work well:

Legal: Contracts, case law, compliance guidelines
Healthcare: Protocols, research papers, care guidelines
SaaS: API docs, user guides, troubleshooting manuals
E-commerce: Product catalogs, policies, FAQs

Step 3: Initialize Your Knowledge Base (2 minutes)

Turn your documents into searchable RAG knowledge:

# Create vector embeddings for semantic search
uv run python initialize_embeddings.py --force

This builds your RAG (Retrieval-Augmented Generation) foundation using OpenAI's text-embedding model and FAISS vector database. RAG converts documents into vector embeddings that capture semantic meaning rather than just keywords, making search actually understand context.

Step 4: Define Your Tools (3 minutes)

Define the search tools your agents will use.

In the LaunchDarkly app sidebar, click Library in the AI section. On the following screen, click the Tools tab, then Create tool.

Create the RAG vector search tool:

Note: we will be creating a simple search_v1 during part 3 when we learn about experimentation.

Create a tool using the following configuration:

Key:

search_v2

Description:

Semantic search using vector embeddings

Schema:

{
  "properties": {
    "query": {
      "description": "Search query for semantic matching",
      "type": "string"
    },
    "top_k": {
      "description": "Number of results to return",
      "type": "number"
    }
  },
  "additionalProperties": false,
  "required": [
    "query"
  ]
}

When you're done, click Save.

Create the reranking tool:

Back on the Tools section, click Add tool to create a new tool. Add the following properties:

Key:

reranking

Description:

Reorders results by relevance using BM25 algorithm

Schema:

{
  "properties": {
    "query": {
      "description": "Original query for scoring",
      "type": "string"
    },
    "results": {
      "description": "Results to rerank",
      "type": "array"
    }
  },
  "additionalProperties": false,
  "required": [
    "query",
    "results"
  ]
}

When you're done, click Save.

The reranking tool takes search results from search_v2 and reorders them using the BM25 algorithm to improve relevance. This hybrid approach combines semantic search (vector embeddings) with lexical matching (keyword-based scoring), making it especially useful for technical terms, product names, and error codes where exact term matching matters more than conceptual similarity.

🔍 How Your RAG Architecture Works

Your RAG system works in two stages: search_v2 performs semantic similarity search using FAISS by converting queries into the same vector space as your documents (via OpenAI embeddings), while reranking reorders results for maximum relevance. This RAG approach significantly outperforms keyword search by understanding context, so asking "My app is broken" can find troubleshooting guides that mention "application errors" or "system failures."

Step 5: Create Your AI Agents in LaunchDarkly (5 minutes)

Create LaunchDarkly AI Configs to control your LangGraph multi-agent system dynamically. LangGraph is LangChain's framework for building stateful, multi-agent applications that maintain conversation state across agent interactions. Your LangGraph architecture enables sophisticated workflows where agents collaborate and pass context between each other.

Create the Supervisor Agent

In the LaunchDarkly dashboard sidebar, navigate to AI Configs and click Create New
Select 🤖 Agent-based
Name it supervisor-agent
Add this configuration:

variation:

supervisor-basic

Model configuration:

Anthropic

claude-3-7-sonnet-latest

Goal or task:

You are an intelligent routing supervisor for a multi-agent system. Your primary job is to assess whether user input likely contains PII (personally identifiable information) to determine the most efficient processing route.

  **PII Assessment:**
  Analyze the user input and provide:
  - likely_contains_pii: boolean assessment
  - confidence: confidence score (0.0 to 1.0)
  - reasoning: clear explanation of your decision
  - recommended_route: either 'security_agent' or 'support_agent'

  **Route to SECURITY_AGENT** if the text likely contains:
  - Email addresses, phone numbers, addresses
  - Names (first/last names, usernames)
  - Financial information (credit cards, SSNs, account numbers)
  - Sensitive personal data

  **Route to SUPPORT_AGENT** if the text appears to be:
  - General questions without personal details
  - Technical queries
  - Search requests
  - Educational content requests

  Analyze this user input and recommend the optimal route:

Click Review and save. Now enable your AI Config by switching to the Targeting tab and editing the default rule to serve the variation you just created.

Click Edit on the Default rule, change it to serve your supervisor-basic variation, and save with a note like "Enabling new agent config".

The supervisor agent demonstrates LangGraph orchestration by routing requests based on content analysis rather than rigid rules. LangGraph enables this agent to maintain conversation context and make intelligent routing decisions that adapt to user needs and LaunchDarkly AI Config parameters.

Create the Security Agent

Similarly, create another AI Config called security-agent

variation:

pii-detector

Model configuration:

Anthropic

claude-3-7-sonnet-latest

Goal or task:

You are a privacy agent that REMOVES PII and formats the input for another process. Analyze the input text and identify any personally identifiable information including: Email addresses, Phone numbers, Social Security Numbers, Names (first, last, full names), Physical addresses, Credit card numbers, Driver's license numbers, Any other sensitive personal data. Respond with: detected: true if any PII was found, false otherwise,types: array of PII types found (e.g., ['email', 'name', 'phone']), redacted: the input text with PII replaced by [REDACTED], keeping the text readable and natural. Examples: Input: 'My email is john@company.com and I need help', Output: detected=true, types=['email'], redacted='My email is [REDACTED] and I need help'. Input: 'I need help with my account',Output: detected=false, types=[], redacted='I need help with my account'. Input: 'My name is Sarah Johnson and my phone is 555-1234', Output: detected=true, types=['name', 'phone'], redacted='My name is [REDACTED] and my phone is [REDACTED]'. Be thorough in your analysis and err on the side of caution when identifying potential PII.

This agent detects PII and provides detailed redaction information, showing exactly what sensitive data was found and how it would be handled for compliance and transparency.

Remember to switch to the Targeting tab and enable this agent the same way we did for the supervisor - edit the default rule to serve your pii-detector variation and save it.

Create the Support Agent

Finally, create support-agent

variation:

rag-search-enhanced

Model configuration:

Anthropic

claude-3-7-sonnet-latest

Click Attach tools.

select: ✅ reranking ✅ search_v2

→ Add parameters
→ Click Custom parameters

{"max_tool_calls":5}

Goal or task:

You are a helpful assistant that can search documentation and research papers. When search results are available, prioritize information from those results over your general knowledge to provide the most accurate and up-to-date responses. Use available tools to search the knowledge base and external research databases to answer questions accurately and comprehensively.

This agent combines LangGraph workflow management with your RAG tools. LangGraph enables the agent to chain multiple tool calls together: first using RAG for document retrieval, then semantic reranking, all while maintaining conversation state and handling error recovery gracefully.

Remember to switch to the Targeting tab and enable this agent the same way - edit the default rule to serve your rag-search-enhanced variation and save it.

When you are done, you should have three enabled AI Config Agents.

Step 6: Launch Your System (2 minutes)

Start the system:

# Terminal 1: Start the backend
uv run uvicorn api.main:app --reload --port 8000

# Terminal 2: Launch the UI  
uv run streamlit run ui/chat_interface.py --server.port 8501

Open http://localhost:8501 in your browser. You should see a clean chat interface.

Step 7: Test Your Multi-Agent System (2 minutes)

Test with these queries:

Basic Knowledge Test:
"What is reinforcement learning?" (if using sample docs)
Or ask about your specific domain: "What's our refund policy?"

PII Detection Test:
"My email is john.doe@example.com and I need help"

Workflow Details show:

Which agents are activated
What models and tools are being used
Text after redaction Watch LangGraph in action: the supervisor agent first routes to the security agent, which detects PII. It then passes control to the support agent, which uses your RAG system for document search. LangGraph maintains state across this multi-agent workflow so that context flows seamlessly between agents.

Step 8: Make Changes Without Deploying Code

Try these experiments in LaunchDarkly:

Switch Models Instantly

Edit your support-agent config:

{
  "model": {"name": "chatgpt-4o-latest"}  // was claude
}

Save and refresh your chat. No code deployment or restart required.

Adjust Tool Usage

Want to limit tool calls? Reduce the limits:

{
  "customParameters": {
    "max_tool_calls": 3  // was 5
  }
}

Change Agent Behavior

Want more thorough searches? Update instructions:

{
  "instructions": "You are a research specialist. Always search multiple times from different angles before answering. Prioritize accuracy over speed."
}

Changes take effect immediately without downtime.

Understanding What You Built

Your LangGraph multi-agent system with RAG includes:

1. LangGraph Orchestration
The supervisor agent uses LangGraph state management to route requests intelligently based on content analysis.

2. Privacy Protection
The supervisor agent uses LangGraph state management to route requests intelligently. This separation allows you to assign a trusted model to the security and supervisor agents and consider on a less-trusted model for the more expensive support agent at a reduced risk of PII exposure.

3. RAG Knowledge System
The support agent combines LangGraph tool chaining with your RAG system for semantic document search and reranking.

4. Runtime Control
LaunchDarkly controls both LangGraph behavior and RAG parameters without code changes.

What's Next?

Your multi-agent system is running with dynamic control and ready for optimization.

In Part 2, we'll add:

Geographic-based privacy rules (strict for EU, standard for Other)
MCP tools for external data
Business tier configurations (free, paid)
Cost optimization strategies

In Part 3, we'll run A/B experiments to prove which configurations actually work best with real data.

Try This Now

Experiment with:

Different Instructions: Make agents more helpful, more cautious, or more thorough
Tool Combinations: Add/remove tools to see impact on quality
Model Comparisons: Try different models for different agents
Cost Limits: Find the sweet spot between quality and cost

Every change is instant, measurable, and reversible.

Key Takeaways

Multi-agent systems work best when each agent has a specific role
Dynamic configuration handles changing requirements better than hardcoding
LaunchDarkly AI Configs control and change AI behavior without requiring deployments
Start simple and add complexity as you learn what works

Related Resources

Explore the LaunchDarkly MCP Server - enable AI agents to access feature flag configurations, user segments, and experimentation data directly through the Model Context Protocol.

Questions? Issues? Reach out at aiproduct@launchdarkly.com or open an issue in the GitHub repo.

This post was originally published on the LaunchDarkly Documentation site. Follow @LaunchDarkly for more AI and feature flag content!