DEV Community: LaunchDarkly

AI Experimentation Best Practices: From Evaluation to Safe Production Rollouts

Scarlett Attensil — Tue, 02 Jun 2026 17:09:35 +0000

Introduction

Artificial intelligence tools, particularly large language models (LLMs), are not like traditional software. AI is probabilistic, so the same instructions and inputs can produce different results, especially when using non-zero temperature or other sampling methods, and those results can shift as your context changes. That unpredictability brings real risks because models can miss the mark, invent facts, or generate unfair or unsafe outputs. They can also incur unexpected costs and slow down under heavy loads, and they must constantly adapt to evolving policies and ethical guidelines.

AI experimentation means iteratively testing data, algorithms, prompts, models, and parameters to optimize model performance and validate hypotheses. You need a clear, repeatable way to try ideas, compare prompts and models, validate how your system finds and uses information, and do safety checks before changes reach real users. Experimentation is not just a nice-to-have; it is essential for shipping AI responsibly, optimizing resource efficiency, reducing costs, and accelerating innovation through rapid, evidence-based iteration cycles.

Throughout this guide, we distinguish evaluation from experimentation. Evaluation means offline benchmarking and scoring, including test sets, human or AI judges, and quality metrics. Experimentation means controlled production changes that affect real users through A/B tests, staged rollouts, or other release strategies. Evaluation tells you whether a variant clears a quality bar; experimentation tells you whether it beats the baseline in production, with statistical confidence and guardrails.

In this article, we cover the core ideas and practical steps for AI experimentation: how to plan a test, evaluate changes, run controlled trials with real users, choose metrics that actually matter to your product, and roll out changes safely. By the end, you will have a process that moves from initial concept to monitored, controlled production release.

AI Experimentation Best Practices

Best Practice	Description
Use experimentation to manage uncertainty	AI outputs can shift over time. Structured experimentation helps teams measure, compare, and validate changes before they reach users. LaunchDarkly AgentControl experiments and release options help turn unpredictability into a controlled process for improvement.
Build trust through evidence, not intuition	Without experimentation, teams rely on gut feeling. Controlled tests provide measurable evidence of what works. Use LaunchDarkly Experimentation, metrics, and AgentControl monitoring to make confident, data-driven decisions.
Detect and reduce hidden risks early	Experimentation surfaces hallucinations, bias, latency regressions, and safety failures before they affect broad audiences. Online evaluations and guarded rollouts help teams detect regressions and pause or roll back unsafe changes.
Enable continuous improvement	AI systems evolve as data, models, and contexts change. Config variations, config targeting, and progressive rollouts give teams a repeatable way to adapt while controlling exposure.
Design experiments with statistical power and variance in mind	Collect multiple observations per variant to account for nondeterminism. Use confidence intervals and statistical significance tests rather than single-run comparisons. Define a minimum detectable effect (MDE), guardrail metrics, and a decision rule before launch. LaunchDarkly experiments and metrics support this evidence-based workflow.
Support responsible and compliant AI	Experimentation frameworks help teams evaluate whether updates align with ethical standards, privacy requirements, and evolving policies. LaunchDarkly role-based access control, approvals, and audit logs help make responsible AI development a built-in process.
Keep track of cost and latency	Track per-session spend and speed, set budgets and max token limits, optimize prompts and context, use caching or streaming where appropriate, and monitor TTFT, p95/p99 latency, retries, and spend. AgentControl monitoring and autogenerated AI metrics help surface cost, latency, and token usage by variation.
Conduct controlled testing with real users	Run A/B tests, sticky cohorts, staged rollouts, or interleaving strategies. Measure satisfaction, task completion, latency, cost, and business impact. Use targeting rules, percentage rollouts, and guarded rollouts to control exposure and rollback thresholds.
Perform evaluation	Define metrics for truthfulness, user experience, reliability, safety, cost, and speed. Test in layers and expand only when stable. Evaluation tells you whether a system meets a bar, while experimentation determines which variant should be trusted in production. LaunchDarkly online evaluations, datasets, and judges support layered AI evaluation workflows.
Use retrieval evaluation for RAG	Evaluate model quality by measuring recall@k, precision@k, citation accuracy, unsupported claim rate, cost, and latency. After offline quality assessment, use live or shadow traffic for controlled experiments that optimize retrievers, chunking, ranking, or reranking. LaunchDarkly AgentControl experiments and monitoring help compare these changes safely.
Ensure proper governance and safety for AI experimentation	Pre-register your experiment plan, including hypothesis, primary metric, MDE, guardrails, and rollback rules. Version prompts, models, and configurations. LaunchDarkly config management, approvals, and audit logs help preserve compliance, safety, and auditability.

Note: Testing different chunking or embedding models usually requires building and validating separate vector indexes, and sometimes separate databases, because embeddings are tied to the index schema. Swapping these at inference time requires architectural planning, reindexing, and migration.

Why AI Needs Experimentation

Traditional software works like a calculator: same input, same output. AI is more like a conversational assistant that can be helpful and creative but sometimes surprising. Since AI is not fully predictable and small changes in wording can shift results, you cannot judge the quality of an AI feature from a single right answer.

AI features are pipelines with many moving parts: models that may update, prompts that steer behavior, tools and APIs that can fail, and knowledge sources that drift as content changes. All of these can affect accuracy, safety, speed, and cost. A one-time test will not catch issues that appear under real traffic.

That is why experimentation is essential. It gives teams a structured way to observe, measure, and improve AI behavior as conditions change. Through continuous testing, you can detect drift, uncover hidden risks, and build confidence that your system performs reliably and responsibly.

LaunchDarkly helps teams operationalize this workflow with AgentControl, configs, config variations, config targeting, monitoring, and online evaluations.

The Hierarchy of Levers: Where to Focus Your Optimization Efforts

In practice, AI experimentation levers should be optimized in order of impact and reversibility:

System message
Examples
Output format
Context
Retries and fallbacks
Models and parameters

This order matters because many high-impact changes can be made without retraining or rebuilding your system. With AgentControl config variations, teams can version and compare these changes while controlling exposure through targeting.

System Message Variations

The system message is one of the most powerful levers in shaping an AI model’s behavior. It defines the model’s role, tone, and boundaries, setting the personality and guardrails for how it responds.

Small changes here can dramatically affect safety and reliability. Tightening tone or adding an out-of-scope clause can prevent speculative or unsafe content. However, overly rigid instructions can make responses sound robotic or unhelpful.

Experiment with several system-message variations and test how they perform across normal, edge, and adversarial scenarios. The goal is not only to find one prompt that works, but to understand how tone and framing influence quality, cost, safety, and latency. Store and compare these variants with AgentControl config variations and monitor results with config performance monitoring.

Choosing the Right Number of Examples

Compare zero-shot, one-shot, and few-shot examples, typically 3-5 examples. Mix common cases and edge cases, include “do” and “don’t” examples, and show the exact output format. Short examples teach patterns, but they also add tokens and delay. Measure accuracy, format adherence, generalization, latency, and cost with autogenerated AI metrics.

Output Format

Choose between free text, structured templates, or native structured outputs. Structured outputs are easier to parse and validate but can constrain creativity or break on truncation. Always validate responses, handle partial outputs gracefully, and keep templates simple. During testing, a temporary explain field can help diagnose why one variation performs better than another.

Context Window Size

Your experiment should test the cost-benefit tradeoff between precise context and extended context. Increasing context often increases cost and latency without improving output quality. Use AgentControl monitoring to compare variation-level latency and token usage before promoting a longer-context variant.

Retries With Backoff

Use one or two attempts for temporary errors such as rate limits, timeouts, or server overload. Add exponential backoff and jitter. Log error rates, latency, and cost. Ensure idempotency, cap retries, enforce timeouts, and offer a polite fallback when limits are hit. For production rollout, pair retry changes with guarded rollouts so latency and error regressions can halt expansion.

Fallback Chain

Route to a backup model or provider in the event of failures or slowness. Keep prompts and formats aligned so the backup model understands the same prompt structure and returns responses in the same format. Preserve conversation state, verify required features on the fallback, and log reasons for routing. LaunchDarkly config targeting can help route different cohorts to different model or provider variations.

The Expansion Rule

Experimentation should scale based on evidence, not enthusiasm. Once your pilot shows strong performance, expand the rollout to broader audiences. Scale only when metrics justify it: success rates are high, failure rates are low, safety checks pass, and time or cost remains acceptable. Use percentage rollouts, progressive rollouts, or guarded rollouts to expand with controlled risk.

Models and Parameters

Models and parameters are the tuning panel for an AI system: the set of dials you use when you want more accuracy, fewer hallucinations, faster responses, or lower cost.

Start with the right model for the job. Use a more capable model for complex reasoning or planning and a smaller, faster model for routine tasks. Match the model’s strength to the complexity and stakes of the task rather than defaulting to the largest model. Lock down the exact model version when possible so results stay reproducible as the model evolves. Version pinning reduces variability, but it does not eliminate drift. Upstream model behavior and real-world inputs can still change, so production experiments and ongoing holdbacks remain necessary.

AgentControl lets teams manage model selection, prompt content, provider configuration, and generation parameters with configs, variations, and AI model configurations.

Temperature

Temperature controls how adventurous or conservative a model’s output is. It is the primary generation setting most users adjust.

Keep it low, around 0-0.3, for code, structured formats, or safety-critical tasks.
Use higher values, around 0.7-1.0, for creativity or brainstorming.
Stay in the middle for everyday conversations.

Other sampling parameters, such as top_p or top_k, also influence output diversity, but temperature usually has the largest and most predictable effect, so it is often the first parameter worth tuning.

Retrieval and Search

Do not rely only on keywords because meaning matters. Semantic search helps the model understand intent. Hybrid search, combining semantic and keyword search, often works best for short queries or exact names. Choose an embedding model that fits your language and domain, and keep its version fixed.

A graph database models relationships and traversals, such as “how is X connected to Y?” A vector database or vector-enabled datastore is optimized for similarity search over embeddings to support retrieval in RAG pipelines. When testing retrieval changes, use online evaluations and AgentControl experiments to compare quality, latency, and cost.

Chunking and Metadata

Split documents into natural sections with slight overlaps. Sliding windows help for long text. Add metadata to improve filtering and relevance. When experimenting, start with a baseline and change one variable at a time: temperature, chunk size, top_k, reranking, or search type. Evaluate offline using a labeled dataset from your domain, then use controlled rollout strategies such as percentage rollouts or guarded rollouts before broad exposure.

Tool and Function Management

Tools are the hands and eyes of your AI. They turn abstract intelligence into real-world action. However, giving an AI system too many tools at once can create reliability, safety, and cost problems. A focused, well-defined toolset keeps the system efficient and predictable.

When experimenting with tools, start small. Give the AI only the tools it truly needs, then expand based on evidence. Simulate tool behavior with mock or historical data before allowing live writes or sensitive operations. Monitor error rates, latency, and cost. Use circuit breakers, fallback paths, and kill switches to keep the system stable when a tool fails.

LaunchDarkly AgentControl tools, agents, feature flags, and release controls can help teams expose new tool behavior gradually and roll back unsafe changes quickly.

Cost and Latency

Managing cost and latency in AI systems is like tuning a race car: you want speed and performance, but you cannot afford to burn all your fuel in one lap. The trick is knowing where your money and time actually go: input tokens, output tokens, model rates, tool usage, retries, retrieval, and post-processing.

Experiment design also affects cost. Multi-armed bandit approaches can reduce spend by shifting traffic away from losing variants early, while long, fixed-horizon A/B tests can waste budget after a clear loser emerges. Track cost per successful answer rather than cost per call so you know which variants are efficient and useful.

Several habits help:

Match the model to the job: Use smaller models for routine tasks and larger models for complex reasoning.
Set clear budgets: Cap tokens, cost, and retries per session.
Cache and reuse: Avoid paying twice for the same retrieval or generated output.
Retry wisely: Validate inputs early and use exponential backoff to avoid waste.
Measure what matters: Track cost per successful answer, not just cost per request.
Watch latency signals: Monitor time to first token, p95/p99 latency, and error rates.

LaunchDarkly Monitoring and autogenerated AgentControl metrics help teams compare token usage, duration, and variation-level performance.

Experimentation Before User Exposure

Before any major AI update reaches real users, it deserves a proper dress rehearsal. Catching issues early prevents bad experiences, unnecessary costs, and reputational damage.

Start by building a test set that mirrors real-world scenarios: genuine examples, synthetic edge cases, and adversarial prompts. If you are working with RAG, make sure answers link back to sources so you can evaluate grounding and citation quality. Use an AI judge or evaluation rubric to score correctness, completeness, clarity, safety, and faithfulness. LaunchDarkly datasets, judges, offline evaluations, and online evaluations support this progression from offline testing to production measurement.

Best practices include:

Set clear thresholds: Define what “good enough” means before the test begins.
Shadow test safely: Run the new model alongside the current one on real traffic while hiding results from users.
Control costs: Sample requests, cache results, and limit verbosity.
Protect fairness and privacy: Compare variants across quality, reliability, cost, and speed while respecting data boundaries.

Once the new model shows stable performance, no quality drops, no latency spikes, and no cost overruns, move to a canary rollout with rollback ready. Use percentage rollouts for fixed exposure, progressive rollouts for scheduled expansion, and guarded rollouts when you want metric-based monitoring and rollback.

Controlled Testing With Real Users

Testing with real users is where theory meets reality. The goal is to gather insight while keeping risk low and user experience intact.

A practical way to do this is A/B testing. By assigning users to consistent cohorts, you can compare different versions of your AI system under real conditions. This supports statistical decision-making, such as confidence intervals and significance testing, rather than anecdotal wins.

To make tests meaningful:

Keep traffic splits representative: Cover different user segments, regions, and use cases with targeting rules.
Tag everything: Include version, prompt, model, parameters, and settings in every request so outcomes are traceable.
Measure real impact: Track satisfaction, edits, retries, task completion, conversion, revenue lift, latency, and cost with LaunchDarkly metrics.

When rolling out updates, start with an internal beta, then gradually expand to 1%, 5%, 10%, and beyond. Watch quality, latency, safety, and failure rates closely. If something goes wrong, roll back and investigate. Creating guarded rollouts gives teams a structured way to tie rollout expansion to live metrics.

Not all AI experiments have a fixed end date. Many teams run ongoing control groups, holdbacks, or adaptive allocation strategies that monitor performance as models, data, and user behavior change. Even then, explicit guardrails and rollback thresholds are essential so optimization never trades off safety, latency, or cost.

Evaluation

Evaluation is not just checking whether the model runs. It is understanding how well the system serves users, how reliable it is under real conditions, and whether it delivers value within operational limits. A strong evaluation framework balances quality, cost, safety, and performance.

Layer evaluations in stages: offline tests, shadow testing, limited rollout, and broader production experiments. Set clear targets for quality, reliability, and cost. Instrument everything so you can explain wins and diagnose regressions. Expand only when metrics hold steady.

Quality and Accuracy

Start with the basics: Does the model tell the truth? Validate answers against known ground truth using offline tests and side-by-side reviews. AI judges provide scalable signals, but they should be calibrated against human review and used primarily for relative comparison between variants rather than absolute truth. LaunchDarkly judges and online evaluations help automate this scoring.

User Experience

Even a technically accurate model fails if it frustrates users. Focus on fast, helpful first responses and fewer handoffs to humans. Measure satisfaction, task completion, rewrite rates, time to first token, and time to useful answer.

Reliability

Reliability means tools behave predictably. Check that outputs match expected formats and that retries or timeouts are rare. Track error rates, schema validity, and success ratios. Define service-level objectives and trigger rollback if failures exceed limits. LaunchDarkly guarded rollouts can connect metric regressions to automated release decisions.

Cost and Speed

Every token, retrieval, and retry has a price. Break down latency and cost by stage to identify where resources are spent. Use smaller or cached models for routine tasks, stream responses where appropriate, and tighten prompts to reduce waste.

Observability

You cannot improve what you cannot see. Log prompts, parameters, model versions, config versions, and tool calls while masking personal data. Feed this data into dashboards that track cost, speed, quality, and safety. Use AgentControl monitoring, metrics, and observability integrations to detect drift and regressions.

Evaluating Retrieval Quality

Great answers depend on great context. Assess retrievers, rerankers, and generators separately and together:

Recall@k shows whether the right documents appear.
Precision@k shows whether retrieved documents are relevant.
nDCG and MRR show how well relevant documents are ranked.
Attributable accuracy connects correct answers to supporting evidence.
Unsupported claim rate flags hallucinations.
Citation correctness, freshness, cost, and latency show whether retrieval adds value.

Use offline QA sets with labeled passages and slice results by topic, query type, and language. Add confidence gating so the system can admit uncertainty instead of fabricating answers.

Observability for Retrieval

Instrument retrieval just like generation. Log query details, index versions, retrieved document IDs, ranking scores, and latency. Use dashboards to visualize recall, accuracy, and latency percentiles. Before rolling out a new index, use canary or shadow testing and control exposure with AgentControl targeting.

Governance and Safety in AI Experimentation

Governance and safety keep AI experimentation trustworthy. The goal is to find measurable improvement while protecting users, respecting constraints, and keeping experiments reproducible.

Security and Access Control

Before any experiment touches real data or users, define who can change what and how. Limit who can modify prompts, deploy models, access production logs, or adjust rollout rules. Use separate environments for development, staging, and production. LaunchDarkly supports these practices through role-based access control, approvals, audit logs, and environments.

Safety Guardrails

Set hard limits that experiments cannot violate. Use content filters, rate limits, token budgets, circuit breakers, and quality thresholds. Define rollback conditions for error rates, latency spikes, toxicity, unsupported claims, or cost overruns. Release policies and guarded rollouts help standardize these controls.

Reproducibility and Compliance

Strong governance means being able to prove what happened. Fix random seeds or sampling settings where supported. Version dataset snapshots, model IDs, prompt templates, guardrails, and configuration files. Store experiment plans, analysis rules, inputs, outputs, parameters, and results. LaunchDarkly config management and config version comparison help preserve reproducibility.

Rollback and Kill Switches

No matter how careful you are, things can go wrong. Keep the previous version ready. Test rollback procedures regularly. Use kill switches that can immediately halt an experiment if safety or quality issues emerge. LaunchDarkly feature flags, guarded rollouts, and guarded rollout management support fast mitigation.

Ongoing Monitoring

Governance does not stop at launch. Continue tracking model performance, user behavior, data distributions, latency, cost, and safety. Periodically rerun safety and quality checks as the system evolves. Maintain a documented process for investigating failures, notifying stakeholders, and implementing fixes.

How LaunchDarkly Helps With AI Experimentation

Where many AI tools stop at evaluation, LaunchDarkly helps enable production experimentation with traffic allocation, metrics, statistical comparison, and controlled release workflows. AI experimentation needs an operational layer that manages prompts, models, parameters, cohorts, traffic allocation, and rollouts safely. Building that layer yourself can quickly become complex.

LaunchDarkly provides:

Instant updates without deployments: Change prompts, swap models, or adjust parameters through AgentControl without redeploying application code.
Safe, gradual rollouts: Test new models on a small percentage of users with percentage rollouts, progressive rollouts, and guarded rollouts.
Centralized control with governance: Use approvals, audit logs, and role-based access control.
Built-in experimentation framework: Run experiments and AgentControl experiments comparing models, prompts, or parameters.
Separation of concerns: Developers can focus on building features while cross-functional teams safely participate in experimentation workflows through controlled configuration changes.

LaunchDarkly feature flags and AgentControl let you treat AI components as dynamic configurations rather than static code, giving you the speed and safety needed for continuous experimentation at scale. The following example shows how to switch between two model configurations with AgentControl.

Example: Switching Between AI Model Variations With AgentControl

In the LaunchDarkly dashboard, open AI, select AgentControl, create a config for the AI workflow, and define variations for each model you want to compare. For implementation details, start with the AgentControl quickstart, then review Create configs, Create and manage config variations, Config targeting, and the Python AI SDK reference.

After setting up config variations, use targeting to control which model variation is served and define a safe default.

Note: This example is simplified for illustration. Production implementations should externalize secrets, define explicit fallbacks, enforce timeouts, and include error handling and guardrails.

Install Dependencies

# Install required dependencies.
# In a notebook, you can run these commands with a leading !.
# In a terminal, run them without the leading !.

!pip install launchdarkly-server-sdk
!pip install launchdarkly-server-sdk-ai
!pip install openai

Import Dependencies

import os

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault
from openai import OpenAI

Set Up Clients

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
openai_api_key = os.getenv("OPENAI_API_KEY")

if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

if not openai_api_key:
    raise RuntimeError("Missing OPENAI_API_KEY")

ldclient.set_config(Config(ld_sdk_key))

ld_client = ldclient.get()
ai_client = LDAIClient(ld_client)
openai_client = OpenAI(api_key=openai_api_key)

Create Evaluation Contexts

# Context 1: control group user.
context_user_a = (
    Context.builder("user-alpha-001")
    .kind("user")
    .set("firstName", "Alice")
    .set("lastName", "Anderson")
    .set("email", "alice@example.com")
    .set("experimentGroup", "control")
    .build()
)

# Context 2: treatment group user.
context_user_b = (
    Context.builder("user-beta-002")
    .kind("user")
    .set("firstName", "Bob")
    .set("lastName", "Baker")
    .set("email", "bob@example.com")
    .set("experimentGroup", "treatment")
    .build()
)

Run the Same Query Against Two Config Variations

fallback_value = AICompletionConfigDefault(enabled=False)
user_query = "Write a detailed essay on NASA"


def run_configured_completion(context, label):
    config, tracker = ai_client.completion_config(
        "ai-experimentation",
        context,
        fallback_value,
        {"user_query": user_query}
    )

    if not config.enabled:
        raise RuntimeError(f"AI config is disabled for {label}")

    messages = [message.to_dict() for message in config.messages]
    messages.append({"role": "user", "content": user_query})

    completion = tracker.track_openai_metrics(
        lambda: openai_client.chat.completions.create(
            model=config.model.name,
            messages=messages,
            temperature=config.model.parameters.get("temperature", 0.7),
            max_tokens=config.model.parameters.get("maxTokens", 800)
        )
    )

    print(f"{label}")
    print("-" * 50)
    print(f"Model: {config.model.name}")
    print(f"Response:\n{completion.choices[0].message.content}")
    print()

    return config.model.name, completion.choices[0].message.content


model_a, response_a = run_configured_completion(context_user_a, "User A: Control")
model_b, response_b = run_configured_completion(context_user_b, "User B: Treatment")

print("=" * 50)
print("Comparison")
print("=" * 50)
print(f"User A got: {model_a}")
print(f"User B got: {model_b}")

Using the code above, one user can receive a baseline model variation while another receives an experimental model variation, without requiring a redeploy. This makes it easier to compare quality, latency, and cost under controlled conditions. The AI SDK can also report metrics to LaunchDarkly, which you can review in AgentControl monitoring and use in AgentControl experiments.

Final Thoughts

Experimentation should be part of everyday AI work: a habit, not a one-off project. Keep iterating, version your data, and let real numbers guide decisions instead of hunches. Treat every AI change like a hypothesis. Every hypothesis should map to a traffic allocation strategy, decision rule, and rollback condition.

Change one thing at a time. Start offline, move to shadow testing, then gradually expand through controlled rollouts while tracking quality, cost, latency, safety, and user outcomes. LaunchDarkly AgentControl, config variations, online evaluations, experiments, and guarded rollouts make this process practical by keeping prompts, models, parameters, metrics, and release controls versioned, targetable, measurable, and reversible.

MLOps Lifecycle: Stages, Workflow, and Best Practices

Scarlett Attensil — Tue, 02 Jun 2026 16:48:26 +0000

A machine learning model that performs well on day one will not remain stable by default. Performance can degrade over time due to data drift, changes in user behavior, evolving feature sets, or updates to upstream systems. These changes rarely cause immediate failure, but they reduce reliability and make model behavior harder to understand.

The core issue is not model quality, but a lack of coordination across the lifecycle. Decisions made early in the lifecycle affect every stage that follows. When stages operate in isolation, traceability breaks down. For example, code versioning may capture model changes, but not dataset lineage, feature definitions, or runtime behavior.

MLOps addresses this by treating machine learning as a continuous, end-to-end lifecycle. It connects data, features, training, deployment, monitoring, and governance into a single operating model. Each stage introduces its own assumptions and dependencies, from training and validation to deployment, monitoring, and governance.

Summary of key MLOps lifecycle concepts

Stage	Activities and Outputs
Data Ingestion and Labeling	Collect raw data (logs, databases, APIs, and sensors), annotate or label it if necessary, and clean it. The output will be versioned datasets or snapshots.
Feature Engineering	Take raw data and transform it into features (e.g., normalization, encoding, and aggregation) and register these features in a feature store.
Model Training and Experimentation	Perform training jobs and hyperparameter tuning. The output of this stage will be trained model artifacts like weights and checkpoints.
Validation and Testing	Test new models against holdout or test data. The output will be accuracy, loss, fairness metrics, and validation reports.
Packaging and CI/CD	Package the model into a deployable artifact or container and push it to a model registry or a container registry.
Deployment and Rollout	Deploy the model to production (REST endpoint, batch service, etc.). Manage traffic with canary releases and/or blue-green deployments. For LLM applications, Configs extends these capabilities to prompt versioning and model provider management.
Monitoring and Observability	Monitor system health: latency, error rates, etc. Monitor machine learning health, including elements like prediction quality and data drift.
Feedback and Retraining	Collect new labeled data and initiate the process of retraining the model. Schedule retraining runs using the newly collected data.
Governance and Approval	Conduct human-in-the-loop reviews and compliance checks before deploying the model. Maintain documentation of the models (e.g., model cards and data sheets), and implement automated policy checks.

The following diagram shows how the major MLOps lifecycle stages connect in practice, from data ingestion through deployment, monitoring, and retraining, along with the operational outputs produced at each step.

Data ingestion and preparation

Data as a first-class production artifact

Most ML systems do not make data ingestion a control boundary, instead treating it as a background process. Initially, everything looks good, but then issues creep in, such as missing columns, silent null propagation, schema changes, late arrival of upstream data, or unknown outliers. There is no catastrophic failure, just gradual degradation of model performance, making it hard to debug and identify exactly what original data was used.

Data ingestion should be a first-class citizen in the MLOps workflow. It is essential to establish reproducibility, compliance, and reliability for models. Determinism and measurable data quality should be achieved.

Ingestion as a control layer

Data ingestion must control the entry of all data being routed and validated. Data should be collected either in batches or streams before undergoing deterministic data cleansing transformations. Before any data is saved, schema requirements should be validated. In addition, each ingestion point should create a snapshot or version for future reference. Data lineage and quality metrics should be recorded at every stage along the processing route, so if validation fails, training on that data stops completely.

In MLOps, one key operational choice is whether a system should be fail-closed or fail-open. Fail-closed systems stop processing as soon as an anomaly is detected, maximizing safety. Fail-open systems continue processing with fallback logic, maximizing availability. The decision should depend on business risk, not the default implementation.

The pseudocode below shows a simplified ingestion control flow: load raw data, validate its schema, apply deterministic transformations, measure drift, and then store the resulting dataset version and metadata for downstream training.

raw_data = load_from_source(config["data"]["source"])

validate_schema(raw_data, config["data"]["schema"])

cleaned = apply_transformations(
    raw_data,
    null_strategy=config["data"]["null_handling"],
    outlier_strategy=config["data"]["outlier_policy"],
)

drift_score = compute_drift(cleaned)

if drift_score > config["data"]["drift_threshold"]:
    alert("Distribution shift detected")

dataset_version = snapshot_dataset(cleaned)
store_metadata(dataset_version, drift_score)

For high-risk ML workflows such as regulated decisions, fraud detection, or safety-sensitive systems, ingestion pipelines should usually fail closed. In lower-risk cases, teams may choose fail-open behavior with explicit fallback logic, but that should be a conscious business decision rather than an implicit default.

Deterministic validation signals

Deterministic validation means data checks that always produce the same pass/fail outcome for the same data based on predefined rules. If a required column disappears, a null rate exceeds an allowed threshold, or a distribution shift crosses a defined limit, the pipeline should respond predictably every time. These checks are often the first reliable sign of upstream data problems, such as schema changes, silent null propagation, or newly introduced categorical values.

In addition to checking whether columns exist, validating data effectively should include the following aspects:

Determining null counts and validating other attribute values
Validating that attribute values fall into the correct range
Limiting the number of categories available for categorical attributes
Measuring distributional shifts in an attribute through either a PSI or KS test
Measuring the number of duplicate records before any data goes into your model at all

Operational validation heuristics

In practice, ingestion validation is implemented as a set of operational heuristics that help teams interpret failures quickly. The signal itself matters, but so does what it usually implies operationally, because that determines whether the right response is to stop the pipeline, investigate upstream systems, or trigger a fallback path.

Signal	Interpretation
Missing required column	Usually indicates that an upstream schema or API contract changed and downstream transformations may no longer be valid
Null rate > threshold	Often suggests corrupted source records, partial extraction failures, or broken joins in the upstream pipeline
Distribution drift > threshold	May indicate a change in user behavior, source population, collection logic, or rollout conditions
High duplicate rate	Often points to replayed ingestion jobs, duplicate event delivery, or broken deduplication logic
Unseen categories	Can break encoders or produce invalid feature mappings if serving logic was built against a fixed category set

Data versioning and lineage

Having immutable dataset snapshots is critically important to ensure reproducible results. To allow reproducible training runs, each training run must reference the dataset version ID, schema hash, transformation configuration, and associated quality metrics. Without versioning, retraining becomes non-deterministic.

In regulated environments, ingestion needs to automatically enforce PII masking, field-level anonymization, and retention tagging. These controls should be enforced automatically as part of the ingestion pipeline rather than handled through ad hoc manual review because manual compliance steps are hard to audit and easy to bypass under delivery pressure.

Configuration and feature flag controls

In mature ML systems, ingestion rules should be controlled through external configuration rather than hard-coded into pipeline logic. This allows teams to adjust schema strictness, null-handling rules, drift thresholds, and anonymization behavior without redeploying the pipeline. The YAML below shows one way to define those ingestion policies declaratively.

data:
  source: "s3://raw/customer_data"
  schema: "schemas/customer_v3.yaml"
  null_handling: "impute_median"
  outlier_policy: "clip_99_percentile"
  drift_threshold: 0.1

validation:
  enforce_strict_schema: true
  max_null_rate: 0.05

Feature flags can control behaviors such as strict schema validation, drift blocking, and auto-anonymization. This enables the gradual introduction of more stringent validation, with instant rollback if the rules block production unexpectedly.

Ingestion-level operating metrics

The ingestion stage should expose a small set of operating metrics so teams can tell whether data is arriving on time, passing validation, and staying within expected quality bounds. These are stage-specific signals used to manage data intake, not a replacement for the broader production monitoring discussed later in the article.

Data intake needs to be measurable.

Key metrics:

Batch success rate
Ingestion latency
Drift score per batch
Null rate per critical feature
Rejected batch percentage
Schema violation count

Because ingestion is the first control boundary in the lifecycle, failures and drift detected here often surface before model-level symptoms appear in production. When ingestion is declarative, versioned, validated, and measurable, downstream training and deployment become far more reproducible.

Outputs

Versioned dataset snapshots
Validation reports and schema versions
Recorded data quality metrics
Metadata required for reproducibility

Feature engineering

Feature engineering is the lifecycle stage where raw, validated data is converted into the model inputs used during training and inference. In MLOps, this stage matters because feature definitions must remain consistent across offline training and online serving. If the transformation logic differs between those environments, the model may behave well in evaluation but degrade in production due to training-serving skew.

Defining the feature contract before transformation

With robust ML systems, feature definitions serve as the single source of truth; the transformation code simply implements them. The use of a feature-first approach helps make transformations deterministic, reducing the risk of training-serving skew. This consistency must extend across both offline feature stores used for training and backtesting, and online feature stores used for real-time inference. Aligning these environments helps prevent silent feature drift, invalid values, or data corruption in production.

Deterministic feature transformations

Feature transformations should be deterministic: The same input should produce the same output when the same feature definition and configuration are applied. This is what allows training, backtesting, and live inference to remain aligned. Tools such as Pandas, Spark, or feature platforms such as Feast can be used to implement that logic.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# Example: Scaling numeric features
scaler = StandardScaler()
scaled_features = scaler.fit_transform(df[["age", "income"]])

# Example: Encoding categorical features
try:
    encoder = OneHotEncoder(sparse_output=False)
except TypeError:
    # scikit-learn < 1.2 uses the sparse parameter name.
    encoder = OneHotEncoder(sparse=False)

encoded_features = encoder.fit_transform(df[["gender", "region"]])

Unit tests and train-serving consistency

Unit tests help verify both transformation correctness and train-serving consistency. In practice, that means confirming that the same feature logic used during training is also used when live requests are processed in production.

import pandas as pd
from sklearn.preprocessing import StandardScaler


def test_feature_scaling():
    df_test = pd.DataFrame({"age": [20, 40]})
    scaler = StandardScaler().fit(df_test)
    transformed = scaler.transform(df_test)

    assert transformed[0][0] < transformed[1][0]  # Scaling check

Ensure that the same transformation logic is applied during both training and serving to prevent training-serving skew. Automate feature value validation before training, which can include range and null checks.

Monitoring feature distributions

Teams usually encode feature-level validation rules separately from transformation code so they can check whether important features remain within expected bounds over time. The example below shows a simple configuration for monitoring a few feature ranges.

feature_monitoring:
  features:
    - age
    - income
    - purchase_count
  validations:
    age: [0, 120]
    income: [0, 1000000]
    purchase_count: [0, 1000]

Feature registry and versioning

Store feature definitions and pipelines in a feature registry to ensure consistency.

{
  "feature_set": "customer_features",
  "version": "v1",
  "features": ["age", "income", "purchase_count"],
  "validation_status": "passed"
}

Use Git or a feature registry to track all changes. Versioned feature pipelines support reproducibility across both training and production.

Outputs

Feature transformation pipelines
Generated feature tables or vectors
Versioned feature definitions in a registry

Model training and experimentation

Once feature sets are available, the next stage is to train candidate models and record the context needed to reproduce and compare those runs later.

Careful automation of training and experiment tracking helps improve reproducibility, consistency, and the ability to compare different models with each other at different times.

Automating model training

Whenever possible, the training process should be automated. This includes scheduling regular training runs, running hyperparameter sweeps, and retraining models when new data becomes available. Automated pipelines save time and reduce human error, especially when managing multiple models or experimenting with different parameters.

Tracking experiments

Every model training run should be tracked to ensure reproducibility and facilitate later comparisons. This means logging the hyperparameters used, such as learning rate and number of trees, dataset snapshots, code versions, and training and validation metrics.

For example, this can be done using MLflow in Python:

import mlflow

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_param("num_trees", 100)

    # Training code goes here
    model.fit(X_train, y_train)

    # Log evaluation metrics
    accuracy = model.score(X_val, y_val)
    mlflow.log_metric("val_accuracy", accuracy)

    # Save the trained model
    mlflow.log_artifact("model.pkl")

This method tracks all of an experiment, and you can repeat the model or compare it with any other run later.

Controls and best practices

To prevent problems during training, configure an early stopping rule and define a limit for the total number of training epochs to avoid runaway training. You should also perform integration tests after loading the trained model using sample inputs. Each trained model should be saved as a versioned artifact in your chosen artifact service, such as S3 or the MLflow Model Registry. Finally, seed random number generators to ensure deterministic training and log the seed. These practices help maintain consistency, reproducibility, and reliability across training runs.

Outputs

Trained model artifacts (pickle, ONNX, TensorFlow SavedModel)
Training logs and experiment metadata
Hyperparameter and dataset configuration snapshots

Validation, testing, and evaluation

Model evaluation starts with offline assessments using a holdout test dataset. In this stage, model performance is measured using task-appropriate measures. For classification, those measures may include accuracy, precision, recall, F1 score, ROC curve, and confusion matrix. For regression, common measures include RMSE, MAE, or R-squared. It is also necessary to evaluate domain-specific business metrics, such as conversion lift, cost of errors, or revenue impact, to ensure the deployed model provides business value in addition to statistical performance.

While offline assessment provides important deployment guidance, automated checks against predetermined thresholds or recorded baselines should be part of gated validation before promotion. These checks should validate fairness or bias issues and use unit tests to confirm that known inputs return expected outputs. If a required threshold is violated, the pipeline should fail and prevent the model from being promoted to production.

To maintain reliability, automate checks that compare metrics against defined thresholds or baselines. For example, the pipeline should fail if a model's accuracy falls below the previous version. The pipeline should also fail if a fairness metric for a protected group is violated. Include unit tests to confirm that the model produces correct predictions on known inputs. Only models that pass all validation checks should advance to deployment.

Outputs

Validation reports and evaluation metrics
Metric visualizations (confusion matrices, ROC curves)
Automated test logs and validation summaries

Packaging and CI/CD

Once a machine learning model is validated, it should be packaged for deployment. This usually includes creating a container image, such as a Docker image, that includes the model and all code required to execute it. You can upload your model to a managed service like MLflow or Amazon S3.

Packaging is about more than reproducing something; it is also about controlling its promotion. When a model artifact has been validated, it should have proper versioning, registry storage, and associated promotion paths, such as staging and production, supported by defined approval and traceability workflows. The purpose of packaging is to ensure that the deployable unit is exactly the one that was validated, with its runtime dependencies, metadata, and configuration captured in a controlled and versioned form. Following a promotion path lowers release risk and makes rollback to a previous version easier if a problem occurs.

If you are using a continuous integration / continuous deployment (CI/CD) system like Jenkins, GitHub Actions, or Azure DevOps, deployment can usually be automated through the CI/CD pipeline. Typical steps involve retrieving the model from its storage location, building the Docker container image, running basic tests, and pushing the model image to a registry. Each image should contain a version tag that identifies which model version was deployed.

To maintain safety and reliability, the CI/CD pipeline should run automated checks, including code validation, test requests to the container, and Docker image security scans. Always use fixed version tags rather than latest to avoid accidental overwrites. If any test fails, the pipeline should stop immediately to prevent a faulty model from being deployed.

Proper packaging combined with automated CI/CD makes model deployment easier, safer, and more consistent.

Deployment and runtime controls

Deployment is the stage where a validated model is exposed to production traffic through an endpoint, batch workflow, or embedded application path. The operational goal is not just to make the model reachable but to release it in a way that limits user risk, supports rollback, and preserves observability during change.

One common runtime control is a feature flag, which is a configurable switch that changes application behavior without requiring a redeploy. In ML systems, feature flags can be used to route users between model versions, limit exposure to selected cohorts, or revert quickly to a known-safe model when problems appear. Tools such as LaunchDarkly provide this kind of runtime control.

Deployment strategies are designed to minimize exposure of new models, whereas guardrails are designed to minimize risk. You can also control which users see the new model by using feature flags in tools like LaunchDarkly. One way to implement feature flags is by wrapping your inference code with a toggle that allows you to use the new model or fall back to the old model:

from ldclient import Context

context = Context.builder(user_id).kind("user").build()

if ld_client.variation("new-model-enabled", context, False):
    prediction = new_model.predict(features)
else:
    prediction = old_model.predict(features)

This approach supports gradual rollouts; you can start by directing a small percentage of real traffic to the new model and increasing exposure only if metrics remain strong.

Always have a rollback plan. Monitor the canary release closely, and if errors rise or latency spikes, revert the feature flag and redeploy the previous model.

To maintain reliability, track latency and error rates for unusual patterns. Conduct integration tests in a staging environment before promoting a model to production. Log every deployment event, and to prevent user impact, trigger alerts or automated rollbacks if any service-level agreements (SLAs) are breached.

Outputs

Running model endpoints (Kubernetes deployments or cloud inference services)
Feature flag configurations controlling rollout
Traffic routing and rollout policies

Monitoring and observability

Unlike the ingestion-level operating metrics discussed earlier, this stage focuses on production-wide monitoring of the live ML system after deployment, including both infrastructure behavior and model behavior under real traffic.

Once a model is deployed in production, it is essential to continuously monitor both the system and the model, which allows for early detection of issues and ensures that the model continues to perform as expected.

Observing system and model metrics

Monitoring should include both infrastructure and model metrics.

Infrastructure metrics monitor the system's health and performance. Here are some examples.

Metric	Purpose
CPU and GPU usage	Ensure that compute resources are not overloaded
Memory consumption	Avoid memory bottlenecks that could slow down inference
Throughput	Track the number of requests the system handles per second
Latency	Monitor response times to maintain consistent performance

Model metrics track the model's performance in production.

Metric	Purpose
Prediction distributions	Detect unusual patterns or shifts in model outputs
Live accuracy	Measure accuracy on recently labeled data to catch performance drops
Error rates	Monitor mispredictions or failures to quickly identify anomalies

Comparing these metrics against training baselines helps you detect data drift. For example, changes in input feature distributions can be measured using KL divergence or the population stability index.

Concept drift should also be tracked; this occurs when a model's performance declines over time without code changes. Unexpected shifts in feature correlations or drops in model quality are strong indicators that something in the data or environment has changed.

Real-time dashboards and alerts

A key tool for monitoring is a real-time dashboard that displays prediction histograms, feature drift charts, and alert counts. Dashboards facilitate quick problem detection as issues arise, providing automated alerts when thresholds are exceeded and sending alerts through channels such as email or text for different severity levels. Minor drifts may generate a helpdesk ticket, while major anomalies may page the on-call technician.

Explainability and logging

For business-critical models, explainability tools can help users understand predictions and investigate why a model may be failing or drifting. All logs and metrics should be preserved and correlated, ideally within dashboards or monitoring systems, so that any issue can be quickly traced, diagnosed, and made actionable.

Feedback loop and retraining

A well-developed machine learning system continues to evolve after deployment. Production usage generates feedback in the form of new data, user corrections, and observed model performance, which can be used to retrain and improve the model over time. Examples include user corrections, newly added labeled examples for retraining, or additional incoming data generated through real usage.

There are numerous options for initiating retraining. Some teams use a scheduled approach, such as retraining every month. Others use automated triggers when data drift exceeds an established threshold or when model performance drops below acceptable levels.

Once retraining is triggered, the same data processing pathway used to develop the original model should be used with the newly input data: develop a new model, conduct validation, and deploy the new model to replace the original only if it passes all relevant validation. Before any model replacement, compare the new model and existing model using a common dataset.

All retraining activities should be carefully documented, including the dataset version, model configuration, and performance metrics. This helps ensure full traceability and reproducibility.

Controlled retraining workflow

Retraining should be triggered by explicit conditions, such as scheduled cadence, measured drift, or degraded production performance. Each run should record the dataset version, feature set version, model configuration, evaluation results, and release decision. Before fully switching to a new model, deploy it in shadow mode. In this setup, both the old and new models run side by side on the same inputs, and their outputs are compared without impacting real users. This helps identify unexpected differences early.

Business metrics should also be evaluated. For example, a small A/B test can confirm whether the new model improves conversion rates, reduces errors, or lowers operational costs. If the new model performs worse than the current one, immediately revert to the old model and investigate the issue. Deployment should not proceed if performance declines.

All retraining cycles should be recorded clearly with the following information: what changed, the reason for retraining, how improvements were measured, and who authorized the release. Maintaining this record makes audits easier and improves transparency.

Each retraining cycle produces important outputs, including updated training datasets, newly trained model artifacts, and retraining and evaluation reports. All of these artifacts should be securely stored and versioned so they can be reviewed, audited, or reproduced in the future.

Closed-loop learning

A well-integrated feedback loop links monitoring, validation, deployment, and retraining together. If a negative trend occurs or performance deviates from expectations, data retrieval can be triggered automatically. Once recent data has been processed, the updated model can replace the existing deployed model with confidence.

Output

Updated training datasets
Newly trained model artifacts
Retraining and evaluation reports

Governance and approval

When machine learning systems operate at scale, governance becomes essential. It is not enough for a model to function correctly from a technical standpoint; it must also be reviewed, documented, and formally approved before reaching users.

Strong governance frameworks establish a clear delineation of role expectations. For example, a data scientist may develop and train a model, while an ML engineer is responsible for deployment. A governance or compliance officer may check documentation and approve the release. After passing technical testing, models must complete formal review processes that include reviewing the model card, data documentation, bias analysis, and performance reports before receiving final approval.

Many organizations separate their environments into development, testing, and production. Models are promoted step by step, with each stage requiring sign-off from the appropriate team. This structured process helps ensure that no model reaches production without proper oversight and review.

Policy and compliance controls

Governance should not depend solely on manual reviews. Wherever possible, it should be reinforced through automation.

Policy as code involves defining governance rules directly in code. For example, the pipeline can automatically verify that the model card includes all required fields, performance metrics meet predefined thresholds, and bias evaluations have been completed.

The following YAML snippet defines policies for a model.

model_policy:
  required_model_card_fields:
    - model_owner
    - intended_use
  min_auc: 0.85
  max_bias_diff: 0.05

on_failure: block_promotion

If any of these requirements are not satisfied, the pipeline should fail automatically.

All approvals and deployments must be recorded in an audit log. Model artifacts should be securely stored and protected with signatures or checksums to prevent tampering. In regulated environments, compliance reviews must occur before a model is allowed to serve real users. Only models that pass every governance check should be permitted to reach end users.

Outputs

Model approval records
Audit logs of model releases
Governance and compliance reports

How LaunchDarkly supports the MLOps lifecycle

LaunchDarkly can act as a runtime control plane for MLOps, helping to enable safer releases, faster iteration, and measurable improvements in production. Its capabilities map directly to several lifecycle stages covered in this article.

A key practice in ML systems is the separation between deployment and release. With LaunchDarkly, teams can ship models or prompt changes behind feature flags and release them only when confidence is established. This means a new model version can be deployed to production infrastructure without any user seeing it until the flag is toggled on.

For safe model rollouts, LaunchDarkly supports progressive delivery and canary releases. It allows teams to expose a new model version to as little as 1% of traffic and scale up gradually to 100%. Rollouts can also be targeted to specific cohorts such as internal users, particular regions, or individual tenants, giving teams fine-grained control over who experiences the new behavior.

Feature flags enable dynamic control of ML functionality at runtime. A single flag can switch between Model A and Model B without redeployment, and it can provide the ability to revert to an earlier version when issues arise. Multivariate flags also allow teams to live-tune parameters such as confidence thresholds, temperature settings, top-p settings, and scoring cutoffs without changing code.

The ability to quickly roll back is crucial for minimizing risk when something goes wrong. LaunchDarkly includes kill switches, which stop access to a risky model or prompt immediately without redeploying. This capability can be important during time-critical incidents.

For online experimentation, LaunchDarkly supports A/B testing on real production traffic. Teams can compare model or prompt variants and measure their impact on quality metrics, latency, and cost before committing to a full rollout. The example below walks through this in detail.

For GenAI and LLM applications, LaunchDarkly offers Configs, which manage prompts, model selection, temperature, and other parameters as versioned configurations. Configs provide:

Prompt and model updates without redeployment
Built-in metrics tracking (tokens, latency, cost per variation)
Online Evaluations for automated quality scoring
Variable substitution for dynamic prompts ({{user_tier}}, {{[context](https://launchdarkly.com/docs/home/flags/contexts)}})

To learn more, read the AgentControl documentation. Prompt and model updates can be rolled out progressively and safely, just like any other feature change.

Guardrails and governance are also built in. Guarded rollouts automatically pause or roll back changes when monitored metrics regress. Approval workflows, role-based access control, and audit logging can support compliance and traceability practices often required by regulated environments.

Example: A/B testing ML models with LaunchDarkly

The following example demonstrates how LaunchDarkly feature flags can be used to A/B test two ML model versions during deployment. A string-type feature flag named model-version is created with two variations, model-a and model-b, and a 50%/50% rollout. Each incoming inference request is routed to one of two models based on the flag evaluation for that user.

In the LaunchDarkly dashboard, create a feature flag with the following configurations:

Name: model-version
Flag type: string
Variation 1: model-a (Logistic Regression)
Variation 2: model-b (Random Forest)
Default rule: 50%/50% rollout

Both models are trained on the same dataset (Iris), so the only variable in the A/B test is the model architecture. The code is shown below.

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data,
    iris.target,
    test_size=0.4,
    random_state=42,
)

# Model A: Logistic Regression (baseline)
model_a = LogisticRegression(max_iter=200, random_state=42)
model_a.fit(X_train, y_train)

# Model B: Random Forest (challenger)
model_b = RandomForestClassifier(n_estimators=50, random_state=42)
model_b.fit(X_train, y_train)

For each incoming request, the LaunchDarkly SDK evaluates the flag and returns the assigned variant. The application routes the request to the corresponding model, as shown below.

import ldclient
from ldclient import Context
from ldclient.config import Config

ldclient.set_config(Config("sdk-YOUR-KEY"))
client = ldclient.get()

# For each inference request
context = Context.builder(user_key).kind("user").build()
variant = client.variation("model-version", context, "model-a")

if variant == "model-b":
    prediction = model_b.predict(sample)
else:
    prediction = model_a.predict(sample)

After simulating 200 inference requests split across both variants, you can compare accuracy and latency to determine which model to promote.

After routing requests between variants, you need to aggregate the resulting outcomes so the two models can be compared on shared evaluation metrics such as accuracy and latency. To analyze A/B test results, navigate to the experiment's Results tab in LaunchDarkly. The Results tab provides:

Visualization options: Probability density, relative difference, and arm averages graphs
Statistical analysis: Probability to be best, expected loss, and confidence intervals
Filtering: Slice results by metric, variation, or user attributes
PDF export: Download results for stakeholder review

import numpy as np

summary = df.groupby("variant").agg(
    total_requests=("correct", "count"),
    correct_predictions=("correct", "sum"),
    accuracy=("correct", "mean"),
    avg_latency_ms=("latency_ms", "mean"),
    p95_latency_ms=("latency_ms", lambda x: np.percentile(x, 95)),
)

Based on the results, an automated decision-making process determines whether the challenger should replace the baseline. If Model B outperforms Model A by a defined threshold, the flag default rule is updated to serve Model B to all users. If it underperforms, traffic stays on Model A.

LaunchDarkly Guarded Rollouts automate this decision-making. Configure a metric threshold, such as accuracy must not regress by more than 1%, and LaunchDarkly automatically:

Pauses the rollout if the metric degrades
Rolls back to the baseline if the threshold is breached
Continues the progressive rollout if metrics remain healthy

No custom code is required for promotion or rollback logic.

To learn more, read the Guarded Rollouts documentation.

ACCURACY_THRESHOLD = 0.01  # Model B must beat Model A by at least 1%

acc_a = summary.loc["model-a", "accuracy"]
acc_b = summary.loc["model-b", "accuracy"]
lift = acc_b - acc_a

if lift >= ACCURACY_THRESHOLD:
    # Promote: update LaunchDarkly flag to serve model-b to 100%
    print("Promote Model B to full traffic.")
elif lift > -ACCURACY_THRESHOLD:
    print("No significant difference. Collect more data.")
else:
    # Rollback: keep model-a as default
    print("Keep Model A. Model B underperforms.")

Taken together, this example shows how runtime flags can separate model deployment from model release, support controlled experimentation on live traffic, and shorten rollback time when a challenger underperforms. In that sense, LaunchDarkly fits into the deployment and release-control layer of the broader MLOps lifecycle.

In a live environment, automatic rollback of flag changes is possible with LaunchDarkly Guarded Rollouts. This enables automatic rollback when one of the monitored metrics regresses.

Extending to LLM applications

The same progressive rollout principles apply to LLM applications, but with additional configuration dimensions. While traditional ML models require only version routing, LLMs need prompt management, temperature tuning, and provider selection. LaunchDarkly Configs can handle these requirements through percentage rollouts, instant rollback, and automated monitoring.

Best practices for managing the MLOps lifecycle

A mature MLOps lifecycle connects data ingestion, feature engineering, training, deployment, and monitoring into a continuous operational loop. The objective is to make machine learning systems reliable and repeatable rather than experimental.

Version and track everything: Data, code, models, and configurations should all be managed as versioned artifacts, enabling clear traceability and reproducibility of results.
Automate validation and testing: Data schemas, feature transformations, and model outputs should all be validated through automated checks. Any change in code or configuration should automatically trigger validation within the CI/CD pipeline.
Use feature flags and gradual rollouts: Teams should use feature flags and gradual rollouts to release new models incrementally rather than all at once. By toggling a new model behind a feature flag and gradually moving traffic over, you can monitor performance and make adjustments before fully replacing the previous version.
Implement continuous monitoring: Continuous monitoring of system performance is crucial. Track key metrics such as data drift, model accuracy, system health, and infrastructure in real time. Establish alerts so issues are identified and addressed early, preventing negative impact on users.

Build governance into the pipeline: Design the system with governance built in by adding approval workflows, documentation requirements, and audit logging directly into the pipeline. Include model cards and model lineage records so you have traceable evidence of the decisions made.

Conclusion

Structuring machine learning systems in this fashion can improve reliability, transparency, and compliance. Each step of a model's lifecycle yields clearly defined artifacts, and quality standards are enforced through automation via your pipelines. Over time, consistency between the original intent of the model, user data, and production performance creates a strong feedback loop.

AI Pipeline: Preventing Drift in Production Systems

Scarlett Attensil — Tue, 02 Jun 2026 16:32:09 +0000

A common failure pattern in a retrieval-augmented generation (RAG) system is a progressive decline in performance. This decline, which can be difficult for users to detect initially, often begins with a reduction in retrieval relevance. Over time, it may lead to longer response times and increasingly inaccurate, incomplete, or less helpful responses. This gradual degradation of the system's performance creates a challenging user experience.

Production failures often stem from uncoordinated changes, with operators adjusting retrieval settings, reranking methods, or model routing without a shared change process. Without explicit versioning and ownership, it becomes difficult to trace which change caused a regression or who made it.

This article argues that production AI pipelines, particularly RAG systems, must be designed around explicit control of change. The system must treat retrieval and prompting, evaluation, and model selection as controllable elements that people running the system must be able to modify through visible changes during active system use. The goal is not to introduce new techniques but to show how existing, well-understood methods can be composed into a production system that remains stable, measurable, and adaptable over time.

Summary of core AI pipeline design considerations

Stage	Core Focus	Why It Matters	Relevant AgentControl Config Features
Problem Definition	Defining use case, retrieval scope, and measurable KPIs	Unenforceable or missing baselines make it impossible to detect degradation later.	AgentControl config variations and tools for retrieval depth, reranker selection, and instant switching without redeployment
Knowledge Grounding and Retrieval	Chunking, embeddings, retrieval, GraphRAG, and reranking	Uncontrolled changes to retrieval parameters are a primary source of grounding failures.	AgentControl config variations for retrieval parameters (top-k, graph hops); model-specific index routing
Model Selection and Orchestration	Routing across embedding models, rerankers, and LLMs	Hard-coded models make every experiment a redeployment and every failure a production incident.	AgentControl config variations bundle model, prompt, and parameters atomically; percentage rollouts for A/B testing
Prompt Engineering and Configuration Management	Versioned, parameterized prompt templates	Ungoverned prompt changes are one of the fastest ways to break a functioning pipeline.	AgentControl config for prompt versioning with variable substitution and segment-based targeting
Evaluation and Guardrails	Grounding accuracy, safety filters, and gating logic	Changes without evaluation gates allow regressions to reach users undetected.	AgentControl Online Evaluations for accuracy, relevance, and toxicity scoring; guarded rollouts for automatic rollback
Experimentation and Feature Flags	Controlled variant testing under live traffic	Without bounded exposure, pipeline variables interact in ways that are difficult to diagnose or reverse.	AgentControl config variations with percentage rollouts; traffic allocation by segment or context
Deployment and Rollout	Separating code deployment from behavioral rollout	Releasing behavior changes to all users at once amplifies the impacted scope of any regression.	AgentControl config targeting with percentage rollouts; progressive exposure with instant rollback
Cost and Latency Optimization	Complexity-based routing, caching, and batching	Routing all requests through the highest-capability path increases cost without proportional quality gain.	AgentControl config targeting rules for model tier routing; context-based cost optimization
Monitoring and Observability	Tracking retrieval drift, grounding accuracy, and latency	Early detection of issues. RAG systems degrade gradually: Regressions often only become visible after affecting end users.	AgentControl monitoring for per-variation metrics
Feedback and Iteration	Structured collection of user signals and error traces	Continuous improvement loops. Ad hoc iteration based on intuition rather than signals leads to unpredictable system behavior.	AgentControl configs with monitoring signals

Note that “hit rate” refers to the proportion of user queries for which the retrieval layer successfully returns at least one relevant document that is subsequently used in the generated response.

In practice, this architecture benefits multiple roles across the AI team. Engineers can test retrieval and model changes safely under controlled exposure, product teams can iterate on prompts through versioned configurations, and operations teams gain faster response to production regressions through automated rollback and monitoring signals.

Note: For demo purposes, you can set up a reference implementation of a configuration-driven RAG pipeline in a single executable environment, while in practice, production usually operates with numerous services.

The following diagram shows how these stages from the above table connect in a production RAG pipeline, each independently configurable under live traffic conditions.

Changes such as modifying retrieval depth or enabling a reranker can be exposed to a subset of users, evaluated against grounding and latency thresholds, and automatically rolled back if regressions are detected, essentially keeping iteration safe without freezing the system.

Production reliability depends on three disciplines: explicit versioning of prompts and models, continuous evaluation signals, and enforced rollback logic. Together, these prevent uncontrolled drift while enabling safe iteration.

Problem definition: Making quality measurable before you optimize

The quality of a production RAG pipeline is strongly influenced by how clearly the problem is defined before implementation. The problem definition directly constrains downstream design choices such as retrieval scope, evaluation metrics, latency budgets, and acceptable trade-offs across the system.

Start by identifying the primary use case and pairing it with measurable KPIs: retrieval hit rate, reranker lift, citation accuracy, latency budgets, and hallucination or grounding error rates. These should be treated as configurable ranges rather than fixed standards, since acceptable thresholds differ across domains and use cases.

While initial requirements may live in planning tools like Jira or Confluence, AgentControl configs elevate key parameters to operational controls, making retrieval thresholds, quality gates, and rollback triggers runtime-configurable rather than static specifications. Unlike hardcoded thresholds buried in application code, AgentControl configs surface these parameters in a dashboard where they can be adjusted, monitored, and rolled back by anyone with access, not just engineers with deployment permissions.

For example, teams may externalize the thresholds for retrieval quality, reranker effectiveness, and latency as configuration flags that control enforcement and rollback. In Python, these can be combined as shown below:

# NOTE:
# Teams can externalize evaluation thresholds as runtime
# configuration so enforcement logic is adjustable without
# redeployment.

import logging
import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault

ldclient.set_config(Config("YOUR_SDK_KEY"))
ai_client = LDAIClient(ldclient.get())

context = (
    Context.builder("user-123")
    .kind("user")
    .set("environment", "production")
    .build()
)

fallback_value = AICompletionConfigDefault(enabled=False)

config, tracker = ai_client.completion_config(
    "rag-eval-config",
    context,
    fallback_value,
    {"query": user_query}
)

if config.enabled:
    custom = config.model._custom if hasattr(config.model, "_custom") else {}

    min_retrieval_hit = float(custom.get("min_retrieval_hit_rate", 0.82))
    min_reranker_lift = float(custom.get("min_reranker_lift", 0.12))
    max_latency_ms = int(custom.get("max_retrieval_latency_ms", 90))

    if measured_hit_rate < min_retrieval_hit or measured_latency > max_latency_ms:
        logging.warning("Threshold violation detected; using fallback path.")
        return fallback_response()

# Continue normal pipeline execution

In this model, thresholds are active policies integrated into evaluation gates and rollout controls. Changes can be exposed incrementally, measured against live metrics, and automatically rolled back when performance degrades. By treating configuration as an operational control surface rather than static settings, the pipeline remains adaptable without sacrificing production stability.

Knowledge grounding: Designing retrieval as a configurable system

In production RAG workflows, unregulated changes to retrieval methods and performance parameters—such as chunking strategies, embedding models, graph traversal depth, or reranker settings—often degrade retrieval quality. This degradation then propagates downstream, manifesting as grounding failures during generation. To minimize this risk, the retrieval layer should be designed as a thoroughly parameterized system, encompassing chunking methods (fixed or semantic), embedding model selection, retrieval depth (top-k), GraphRAG traversal depth, reranker configuration, and context window limits.

A layered retrieval approach might consist of vector-based retrieval of unstructured data, optional graph-based expansion for the improvement of relational context, and reranking for precision. A control-plane system governs the parameters exposed by each pipeline layer, making them observable, configurable, and safe to experiment with without modifying application code. In LaunchDarkly, AgentControl configs provide this control layer, storing retrieval configuration as versioned variations that can be tested incrementally and rolled back instantly. Retrieval quality remains adjustable at runtime, and the retrieval quality is not dependent on the speed of the assessment of the variants (e.g., chunk size or hop depth), since the assessment can be done with live traffic, provided that changes are gated and evaluated incrementally.

Parameters such as top-k, graph-hop depth, reranker toggles, embedding model selection, and fallback behavior are governed through configuration, enabling safe iteration without redeployment. AgentControl config targeting enables instant fallback by switching which variation is served with no redeployment required. If a new retrieval strategy degrades quality, revert to the baseline variation in seconds. This means existing vector stores such as Pinecone, Weaviate, FAISS for embeddings, and Neo4j for knowledge graphs are able to continue being used.

For instance, it is possible to construct a multi-layer pipeline: RAG on internal documents, GraphRAG via Neo4j for structured data, and a cross-encoder reranker. The transitions between these layers can be made configurable, allowing reranking to be enabled or disabled, embedding strategies to be adjusted, and routing logic to evolve while preserving production quality.

In practice, retrieval parameters can also be managed as part of a versioned AI configuration, allowing chunking, retrieval depth, graph expansion, reranking, and index selection to evolve together under controlled rollout.

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault

ldclient.set_config(Config("YOUR_SDK_KEY"))
ai_client = LDAIClient(ldclient.get())

context = (
    Context.builder("user-123")
    .kind("user")
    .set("tier", "premium")
    .set("environment", "production")
    .build()
)

fallback = AICompletionConfigDefault(enabled=False)

config, tracker = ai_client.completion_config(
    "rag-retrieval-config",
    context,
    fallback,
    {"query": user_query}
)

if config.enabled:
    custom = config.model._custom if hasattr(config.model, "_custom") else {}

    chunk_size = int(custom.get("chunk_size", 350))
    retrieval_top_k = int(custom.get("retrieval_top_k", 10))
    enable_graph_rag = bool(custom.get("enable_graph_rag", False))
    graph_hops = int(custom.get("graph_hops", 2))
    enable_reranker = bool(custom.get("enable_reranker", True))
    embedding_model = custom.get("embedding_model", "e5-base")
    reranker_model = custom.get("reranker_model", "cross-encoder")

    # Note: Switching embedding models requires separate precomputed indexes
    # per model. The configuration should control both the embedding model
    # and the index being queried.

    # Note: Increasing retrieval_top_k sends more retrieved content downstream,
    # which can increase token usage, cost, and context-window pressure.

    chunks = chunk(text, size=chunk_size)
    embeddings = embed(chunks, model=embedding_model)
    vec_results = vector_store.retrieve(embeddings, top_k=retrieval_top_k)

    graph_results = []
    if enable_graph_rag:
        graph_results = neo4j_query(hops=graph_hops, node_type="Document")

    # In production, graph expansion should apply relevance filtering
    # or weighted merging to avoid flooding the context window.
    results = merge(vec_results, graph_results)

    if enable_reranker:
        results = rerank(results, model=reranker_model)

# Apply context limits/fallbacks as needed
final_context = trim_to_context_budget(results)

In this model, retrieval behavior becomes a controlled surface rather than a static implementation detail. Teams can enable or disable GraphRAG, adjust traversal depth, swap embedding models, or toggle rerankers safely while monitoring grounding accuracy and latency. This configuration-driven approach keeps retrieval flexible without sacrificing production stability.

Model selection and orchestration

Hard-coding embedding models, rerankers, or LLMs directly into orchestration logic is a common anti-pattern in production AI systems that is easy to trace. The case becomes even more difficult if the embedding model, reranker, or chat model is hard-coded, for then every experiment becomes a redeployment. This will not only slow down the learning process but also increase the risk at the same time.

Model selection should be treated as a routing process rather than a one-time selection decision. AgentControl configs bundle model, prompt, temperature, and max_tokens as a single versioned configuration. When you switch variations, all parameters change atomically, reducing the risk of mismatched model/prompt combinations that can occur when using separate flags for each parameter. In practice, this means that each request is dynamically routed to a model variant based on configuration, traffic allocation, or runtime signals rather than binding the pipeline to a single hard-coded model. The pipeline is asking for “an embedding model” or “a chat model” all the time. Model selection is governed through configuration rather than hard-coded API calls.

In LaunchDarkly, AgentControl config bundles the model, prompt, temperature, and max_tokens as a single versioned variation. When a variation changes, these parameters update atomically, eliminating the risk of mismatched configurations and allowing traffic allocation and fallback behavior to be controlled safely. Fallbacks can be triggered by concrete conditions such as degradation in grounding accuracy, violations of latency budgets, elevated error rates, or failed evaluation checks, allowing the pipeline to revert to a known-stable model automatically.

The following simplified example illustrates how model routing can be externalized through configuration. Rather than binding the pipeline to a specific chat model, the active variant is selected at runtime based on a configuration flag, enabling controlled experimentation and safe fallback behavior.

# Illustrative example showing configuration-driven model routing
# using LaunchDarkly AgentControl config.

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault

ldclient.set_config(Config("YOUR_SDK_KEY"))
ai_client = LDAIClient(ldclient.get())

# Evaluation context used for targeting and experiments
context = (
    Context.builder("user-123")
    .set("environment", "production")
    .build()
)

fallback = AICompletionConfigDefault(enabled=False)

# Retrieve the full AI configuration (model, prompt, parameters)
config, tracker = ai_client.completion_config(
    "chat-config",
    context,
    fallback,
    {"context": retrieved_docs}
)

if config.enabled:
    completion = tracker.track_openai_metrics(
        lambda: openai_client.chat.completions.create(
            model=config.model.name,
            messages=[msg.to_dict() for msg in config.messages],
            temperature=config.model.parameters.get("temperature", 0.7),
            max_tokens=config.model.parameters.get("maxTokens", 4096)
        )
    )

AgentControl configs separate model selection from application code entirely. The pipeline requests a configuration, and AgentControl config returns the complete model setup based on targeting rules, enabling A/B tests, gradual rollouts, and instant rollback without code changes.

Controlled experiments can be conducted behind this one banner. For example, a Mistral or LLaMA-based deployment can be given just 5% of the total traffic while the baseline continues to be unaffected. Experimentation primitives such as traffic allocation, targeting, and instant kill switches support safer operation in production when combined with proper evaluation signals, monitoring, and rollback discipline, but they do not replace sound system design or operational oversight.

Prompt engineering and configuration management

Prompts are not constant resources; they move with changes in requirements, the evolution of data, and the appearance of edge cases. A lack of governance, coupled with changing prompts, is one of the quickest methods to cause the uprooting of a perfectly functioning pipeline.

AgentControl configs store prompts as versioned configurations in LaunchDarkly rather than in application code. Prompts support variable substitution such as {{context}} and {{user_tier}}, and the template structure, variable slots, and active prompt variants can all be versioned and selected at runtime. This allows teams to test prompt variants, compare outcomes, and restore previous versions when needed. The following simplified example shows how a prompt variant might be selected through configuration at runtime.

from ldai.client import LDAIClient, AICompletionConfigDefault

ai_client = LDAIClient(ldclient.get())
fallback = AICompletionConfigDefault(enabled=False)

config, tracker = ai_client.completion_config(
    "rag-assistant-config",
    context,
    fallback,
    {"context": grounded_context, "user_tier": "premium"}  # Variable substitution
)

# Prompt is stored in LaunchDarkly, not in code.
# Variables like {{context}} and {{user_tier}} are substituted automatically.
if config.enabled:
    messages = [msg.to_dict() for msg in config.messages]

This method allows for structured testing, selecting specific users to expose to the new feature, and quickly going back to the previous version, thus harmonizing prompt iteration with the deployment discipline already established for code.

Evaluation and guardrails

During evaluation, configuration values remain in effect, but the focus shifts from the configuration itself to measurable attributes of system behavior, such as grounding quality, latency, and safety-related metrics. Changes in retrieval, prompts, or models should be governed by both objective metrics and qualitative evaluation signals.

Objectively speaking, the correctness of grounding, the time taken, and the accuracy of citations are among the measures applied. Relevance and helpfulness are typically assessed through LLM-as-judge patterns, an approach popularized by tools such as OpenAI Evals and Patronus.

AgentControl includes built-in Online Evaluations that allow teams to attach judges for metrics such as accuracy, relevance, and toxicity to any variation. Sampling rates can be configured, and the resulting scores appear in the Monitoring dashboard alongside operational metrics such as latency and cost. These signals should be regarded as indicators of relative change rather than absolute truths. AgentControl displays them per variation, making it easy to compare whether a variant actually outperforms the baseline without building custom analytics. When used together through Guarded releases, they drive gating decisions automatically, pausing rollout exposure or triggering rollback when quality thresholds are violated without requiring manual intervention.

Safety evaluation typically focuses on detecting risks related to personally identifiable information (PII), toxicity, and compliance violations. Deterministic detectors such as Presidio are often used alongside probabilistic classifiers and cloud DLP services to reduce false negatives. In addition, evaluation systems can attach automated judges to monitor safety signals. For example, AgentControl Online Evaluations can apply toxicity judges to sampled responses and surface the results in monitoring dashboards.

The following simplified example illustrates how evaluation signals can be computed by the application and emitted as events to support configuration-driven gating decisions. In this pattern, scoring logic remains inside the application, while promotion or rollback behavior is governed through configurable rules.

def evaluate_and_gate(variant_id, output, context):
    scores = compute_eval_scores(output)
    pii_risk = run_pii_pipeline(output)

    # AI SDK provides automatic tracking of tokens, duration, and errors.
    completion = tracker.track_openai_metrics(
        lambda: openai_client.chat.completions.create(
            model=config.model.name,
            messages=[msg.to_dict() for msg in config.messages]
        )
    )

    # Optional: track additional custom metrics.
    ld_client.track(
        "rag_quality_metrics",
        context,
        data={
            "variant": variant_id,
            "grounding_accuracy": scores["grounding_accuracy"],
            "hallucination_rate": scores["hallucination_rate"],
            "latency_ms": scores["latency_ms"],
            "pii_risk": pii_risk.level,
        }
    )

    # Configuration-driven gating.
    if scores["grounding_accuracy"] < ld_client.variation(
        "min_grounding_accuracy",
        context,
        0.88
    ):
        if ld_client.variation("enable_rollback", context, True):
            return fallback_response(context)

    return output

Evaluation signals are computed by the application and emitted as events to support configuration-driven gating decisions. When grounding accuracy falls below the configured threshold, guarded rollouts automatically pause the variant and restore the baseline, without requiring manual intervention.

Experimentation and feature flags

As soon as evaluation and guardrails are implemented, experimentation ceases to be treated as such and is instead fully integrated within the system's daily cycle. The state of the pipeline at this stage is not “trying out methods and praying for the best” but an incessant, subtle, and well-managed learning process.

In practice, RAG-based experimentation is rarely isolated to a single variable. Adjustments in one area often influence others. For example, increasing retrieval depth changes the volume of context supplied to the model, graph traversal affects which documents are visible, rerankers modify relevance ordering, prompt changes alter tone and structure, and switching models impacts latency and cost. These dimensions interact, which makes controlled experimentation and careful gating essential.

AgentControl configs make these interactions explicit and controllable. Each variation represents a complete configuration—model, prompt, parameters, and tools—that can be tested against others under controlled traffic allocation, allowing multiple variables to evolve under bounded exposure. Traffic allocation and evaluation thresholds are managed through AgentControl configs, while Guardian-guarded rollouts enforce rollback conditions automatically when metrics indicate regression. The pipeline decides at runtime which choices to make instead of sending out a new deployment every time there is an idea to be tested. The code remains unchanged; only the behavior changes.

Each configuration change becomes a small, bounded experiment with a clearly defined blast radius and rollback path. Every meaningful decision in the pipeline is externalized to AgentControl configs. Multiple variations evolve safely under controlled exposure, with built-in metrics showing which performs better, no custom instrumentation required. From the application's perspective, this process is straightforward: On each request, it simply retrieves the active configuration and executes accordingly.

The following simplified example demonstrates how multiple pipeline parameters can be externalized as configuration variables. Rather than hard-coding retrieval depth, graph traversal limits, reranker activation, prompt versions, or model variants, these values are resolved at runtime, enabling controlled experimentation and gradual rollout. In this pattern, retrieval depth, graph expansion, reranking behavior, and model selection are resolved together as part of a versioned AI configuration rather than managed as unrelated flags.

from ldclient import Context
from ldclient.config import Config
import ldclient
from ldai.client import LDAIClient, AICompletionConfigDefault

ldclient.set_config(Config("YOUR_SDK_KEY"))
ai_client = LDAIClient(ldclient.get())

context = (
    Context.builder("user-123")
    .set("environment", "production")
    .build()
)

fallback = AICompletionConfigDefault(enabled=False)

config, tracker = ai_client.completion_config(
    "rag-pipeline-config",
    context,
    fallback,
    {"query": user_query}
)

if config.enabled:
    custom = config.model._custom if hasattr(config.model, "_custom") else {}
    top_k = int(custom.get("retrieval_top_k", 8))
    graph_hops = int(custom.get("graph_hops", 1))
    enable_reranker = bool(custom.get("enable_reranker", True))
    model_variant = config.model.name

In this pattern, experimentation occurs by adjusting configuration values and traffic allocation rather than modifying orchestration logic.

Key Insight: No redeployment is required to adjust retrieval depth, switch prompt variants, or test new models.

Configuration determines runtime behavior, while evaluation metrics determine whether those changes persist. For example, a config with two variations might include:

Variation A (Baseline): GPT-4o-mini, temperature 0.3, concise prompt
Variation B (Experimental): Claude 3 Haiku, temperature 0.5, detailed prompt with citations

Use percentage rollouts to send 10% of traffic to Variation B, then compare token cost, latency, and quality metrics in the LaunchDarkly dashboard before promoting.

The diagram below illustrates how an experiment progresses from limited exposure to promotion or rollback based on measurable thresholds.

At this stage, experimentation becomes a routine, low-risk operational activity rather than an ad hoc process with uncertain production impact. Variants are promoted only when metrics validate improvement; otherwise, rollback restores the baseline automatically. Decisions are governed by thresholds and enforced by configuration logic, not by informal coordination or manual caution.

In practice, this shifts how teams manage risk. Engineers can test ideas earlier and under real traffic, while product teams receive measurable feedback instead of speculation. When regressions occur, as they inevitably will, the system absorbs them predictably through rollback mechanisms rather than escalating into production incidents.

This is not experimentation for its own sake. It is controlled exposure, enforced by configuration and measurable thresholds rather than personal discipline alone.

Deployment and rollout

In AI pipelines, notably RAG systems, deployment and rollout are the key operational milestones that support controlled execution and minimize the risk of production disruptions. Deployment refers to changes in code or infrastructure. Rollout, by contrast, refers to the controlled exposure of new behavior in production, such as introducing new models or configuration variants gradually to reduce risk.

Separating deployment from rollout allows teams to validate behavior changes incrementally under real traffic conditions. Take, for instance, a new LLM version rollout: Start with a small percentage of traffic and track grounding correctness and latency. If metrics remain healthy, exposure can be increased. If issues emerge, rollback is immediate and configuration-driven.

LaunchDarkly controls these rollouts through AgentControl Configs percentage-based targeting and segment rules, with Guardian guarded rollouts that automatically pause or roll back based on quality signals. Exposure is increased only when metrics confirm the new variation is safe. Actual deployment is handled by infrastructure tools such as Kubernetes or Docker, while LaunchDarkly acts as the rollout control layer, managing fallback routes and ensuring configuration consistency across services. For example, retrieval config updates might be first shown to beta users and later on rolled out based on performance.

Here is a Python example that uses the LaunchDarkly SDK to manage the rollouts dynamically in your pipeline.

import os
import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

ldclient.set_config(Config(ld_sdk_key))
ld_client = ldclient.get()
ai_client = LDAIClient(ld_client)

def handle_rag_request(user_context_data, query, variables):
    context = (
        Context.builder(user_context_data["user_id"])
        .set("segment", user_context_data.get("segment", "unknown"))
        .build()
    )

    fallback = AICompletionConfigDefault(enabled=False)

    config, tracker = ai_client.completion_config(
        "llm-rollout-config",
        context,
        fallback,
        variables
    )

    if config.enabled:
        # Model, prompt, and parameters are versioned together.
        response = tracker.track_openai_metrics(
            lambda: generate_response(config)
        )
        log_performance_metrics(response)
        return response

    return fallback_response(query)

# Close ld_client in your application's shutdown hook, not here.

This setup allows rollout percentages, targeting rules, and cohort segmentation to be adjusted directly through configuration. Exposure can be increased incrementally under live traffic, while evaluation metrics determine whether promotion continues or rollback is triggered.

Separating deployment from rollout ensures that behavioral changes are introduced gradually and reversibly, reducing production risk while maintaining iteration speed.

Cost and latency optimization

Complex queries usually call for the use of models of higher capability to satisfy the quality requirement, which usually means longer latency and more cost. On the other hand, many requests can be handled with lower-cost paths when quality signals remain within acceptable limits.

This is fundamentally about dynamic optimization. Possible strategies include routing queries based on complexity, caching frequently accessed results, adjusting retrieval depth, and batching requests where appropriate. These techniques aim to balance accuracy, latency, and cost rather than optimizing any one dimension in isolation.

In practice, routing policies and cost controls can be governed through AgentControl configs targeting rules. Requests can be routed based on attributes such as user tier, region, or query characteristics. For example, complex queries may be sent to a larger model such as GPT-4 while simple lookups are routed to a smaller model, and different retrieval or reranking strategies can be applied to different traffic segments, all without modifying application code.

A complexity classifier provides a practical example of cost-aware routing in production AI systems. Rather than routing all requests to the most expensive model and deepest retrieval path, the system evaluates incoming queries and dynamically selects an appropriate tier.

Routing decisions are not hard-coded. Instead, configuration flags determine tier selection and threshold limits. This allows routing behavior to evolve safely under live traffic without redeployment. In practice, complexity classifiers should be treated as heuristics and continuously calibrated using production metrics.

Below is an illustrative example of cost- and latency-aware routing controlled through configuration.

# NOTE:
# Illustrative example showing cost- and latency-aware routing via configuration.
# The complexity heuristic below is a placeholder and must be adapted
# to your domain, metrics, and production requirements.

import os
import ldclient
from ldclient import Context
from ldclient.config import Config

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

ldclient.set_config(Config(ld_sdk_key))
ld_client = ldclient.get()

def query_complexity(query):
    # Placeholder heuristic for illustration only.
    # Production systems usually consider richer signals such as:
    # - keyword density or semantic difficulty
    # - question structure and reasoning depth
    # - historical user interaction patterns
    # - observed quality or latency metrics
    # Any routing heuristic should be validated against production
    # evaluation metrics before broad rollout.
    return len(query) / 100.0

def handle_optimized_request(user_context_data, query):
    context = (
        Context.builder(user_context_data["user_id"])
        .set("region", user_context_data["region"])
        .build()
    )

    # Configuration-controlled routing decision.
    tier = ld_client.variation("llm_tier", context, "auto")

    if tier == "auto":
        complexity_threshold = ld_client.variation("complexity_threshold", context, 0.4)
        tier = "small" if query_complexity(query) < complexity_threshold else "large"

    # Select model based on resolved tier.
    model = "small_llm" if tier == "small" else "large_llm"

    response = llm_call(model, query)

    # Optional escalation path if quality signals fall below tolerance
    # (for example retrying with a larger model if grounding or confidence checks fail).

    log_optimization_metrics(response)

    return response

In many production deployments, routing decisions can also be managed directly through AgentControl config targeting rules. For example, different model tiers may be served based on user subscription level, geographic region, or environment without requiring custom routing logic in application code.

This pattern allows teams to adjust routing tiers, thresholds, and fallback behavior dynamically. Lightweight requests can be served at lower cost and latency, while complex queries are automatically escalated to higher-capability paths. Performance and quality remain observable and controllable through configuration.

The diagram below illustrates how routing decisions move between lightweight and high-capability paths based on configuration and runtime signals.

The lightweight path (small model with shallow retrieval) and the high-capability path (large model with deep retrieval) converge before caching, batching, and final response generation. This ensures that cost optimization does not fragment the delivery pipeline and that observability remains consistent across tiers.

Monitoring and observability

In every production RAG pipeline, performance monitoring is a must-do process that helps you discover issues before they get out of control. To do this, it is necessary to monitor the most important metrics, which include retrieval drift, grounding accuracy, hallucination rates, and related reliability signals. Latency spikes and the overall health of the API will be monitored as well. In practice, these metrics should be tracked over time and analyzed across percentiles (e.g., p95, p99) rather than relying solely on averages.

These observability signals will then be incorporated into your control plane, for instance, LaunchDarkly, to automate actions such as rollbacks or switching to safe-mode configurations whenever things go wrong. Observability provides the signals; AgentControl configs enforce the decisions. When grounding accuracy drops below the threshold, AgentControl configs can automatically shift traffic back to the stable variation without waiting for manual intervention.

Why invest in continuous monitoring? Because RAG systems are inherently dynamic: models evolve, underlying data shifts, and user behavior changes across contexts. These factors can gradually degrade performance in ways that are not immediately visible. Continuous monitoring is necessary because regressions often become apparent only after affecting end users. Early detection of hallucinations or accuracy drops allows issues to be addressed proactively, improving reliability without constant manual intervention.

Telemetry data can be collected through vendor-neutral observability frameworks such as OpenTelemetry or through an organization's internal monitoring infrastructure. AgentControl configs also provide a built-in monitoring dashboard that surfaces variation-level metrics automatically, allowing teams to compare model and prompt performance across experiments without additional instrumentation. The dashboard reports metrics such as token usage, cost, latency, and quality scores for each configuration variant. These metrics can then be used by rollout controls to suspend, promote, or roll back configurations when thresholds fall outside acceptable ranges.

For example, if the performance of the grounding deteriorates, LaunchDarkly will immediately unmask the experimental model and pull the reranker back to reliable settings, depending solely on the live data. The intervention should be driven by predefined thresholds and rules defined per metric rather than by a single global criterion, and not by unpredictable human interference.

Shown below is a Python snippet that demonstrates feeding metrics into LaunchDarkly for decision-making purposes.

# NOTE:
# Illustrative example showing how observability signals can be
# used to drive configuration-based rollback decisions.
# Telemetry is collected by external monitoring systems
# and evaluated against configuration thresholds.

import os
import ldclient
from ldclient import Context
from ldclient.config import Config

def get_current_metrics():
    # Placeholder for real telemetry collected through
    # observability systems such as OpenTelemetry.
    return {"grounding_accuracy": 0.85, "hallucination_rate": 0.05}

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

ldclient.set_config(Config(ld_sdk_key))
ld_client = ldclient.get()

def monitor_and_adjust(context_key):
    context = Context.builder(context_key).build()
    metrics = get_current_metrics()

    accuracy_threshold = ld_client.variation("accuracy_threshold", context, 0.90)

    if metrics["grounding_accuracy"] < accuracy_threshold:
        # Configuration-driven rollback decision.
        if ld_client.variation("enable_rollback", context, True):
            revert_to_stable_config()

    # Metrics are typically emitted to observability systems
    # and evaluated alongside AgentControl monitoring dashboards.

When AI requests are executed through the AI SDK, operational metrics such as token usage, cost, latency, and success rates are captured automatically using the track_openai_metrics() instrumentation. These signals appear in the AgentControl monitoring dashboard alongside evaluation scores and configuration variations, enabling teams to compare model and prompt performance without building custom analytics pipelines.

Using observability data to inform LaunchDarkly controls makes your pipeline adaptive. Teams get traceability, faster incident response, and safer production iteration by clearly separating metric computation from configuration enforcement.

To help you visualize, here's a flowchart of the observability pipeline.

Observability metrics feed directly into configuration-driven gating rules, enabling automatic rollback or promotion without manual intervention. The control plane does not compute metrics itself; it enforces decisions based on thresholds defined in configuration.

Feedback and iteration

Closing the RAG pipeline loop involves activating structured user feedback, such as ratings, error logs, and usage traces, and feeding these signals back into retrieval, prompt, or model adjustments. AgentControl configs then control the release of refined configurations. New prompt variations can be tested on a small percentage of traffic, evaluated against satisfaction metrics, and promoted only when signals confirm improvement, ensuring that they are tested under limited exposure before broader rollout. This promotes a never-ending process of evolution with closely-knit feedback loops, allowing you to quickly iterate while still having a solid production environment, without requiring broad, unmanaged production changes.

In practice, these feedback signals can be sourced from RLHF-derived signals, user feedback systems, satisfaction metrics, or built-in telemetry. LaunchDarkly coordinates the rollout of the updates, managing the exposure or reversions depending on the feedback received.

Feedback signals such as lower satisfaction scores, increased clarification requests, or higher error rates indicate that users struggle with the new prompt format. The new prompt variant can be kept in evaluation-only mode, allowing time to refine and retest through online evaluations, automated or semi-automated assessments run on live or shadow traffic, before any wider rollout.

Here's a Python snippet to integrate feedback signals with LaunchDarkly for adaptive rollouts.

# NOTE:
# Illustrative example showing how aggregated feedback signals
# can influence configuration-driven rollout decisions.
# Feedback signals are computed by the application and evaluated
# against thresholds managed through configuration.

import os
import ldclient
from ldclient import Context
from ldclient.config import Config

def get_user_feedback():
    # Placeholder for aggregated feedback signals.
    return {"satisfaction_score": 0.75, "error_rate": 0.10}

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

ldclient.set_config(Config(ld_sdk_key))
ld_client = ldclient.get()

def process_feedback_and_iterate(context_key, tracker):
    context = Context.builder(context_key).build()
    feedback = get_user_feedback()

    satisfaction_threshold = ld_client.variation(
        "satisfaction_threshold",
        context,
        0.80
    )

    if feedback["satisfaction_score"] > satisfaction_threshold:
        # Promote variant only when feedback trends meet acceptance criteria.
        if ld_client.variation("promote_variant", context, True):
            rollout_updated_variant()
    else:
        # Keep variant in evaluation mode or rollback.
        revert_to_baseline()

    # Track user feedback with AI SDK.
    if feedback["satisfaction_score"] > satisfaction_threshold:
        tracker.track_feedback({"kind": "positive"})
    else:
        tracker.track_feedback({"kind": "negative"})

The method used here makes the iterations both data-driven and lower-risk. Teams blend the feedback with the same setup and launch discipline as other pipeline modifications, thus preventing ad hoc decision-making and ensuring that the system behaves predictably even while it is being continuously evolved.

Last thoughts

Production AI systems rarely fail because a model is imperfect; they fail because change is unmanaged. In RAG pipelines, even small adjustments to retrieval depth, reranking logic, prompt structure, or model routing can compound quickly under real traffic. When those decisions are embedded directly in code, iteration becomes slow, risky, and difficult to reverse.

The goal is not to freeze behavior but to externalize it. AgentControl configs provide that external control surface: versioned configurations, percentage rollouts, automatic metrics, and instant rollback. When configuration, experimentation, evaluation, and rollout are treated as first-class architectural concerns, change becomes measurable and reversible. Teams can introduce improvements incrementally, observe their impact under real conditions, and roll back regressions without destabilizing the system.

Unlike general-purpose feature flag workflows, AgentControl configs are designed specifically for runtime AI configuration, combining prompt versioning, automatic metrics tracking, and online evaluations in a single control surface. Unlike broader MLOps platforms, it focuses on operational behavior in production rather than training pipeline management.

The more dynamic and compositional AI systems become, the more valuable controlled change becomes. Production RAG is not about finding a perfect configuration. It is about building a system that can evolve safely, intentionally, and continuously. AgentControl configs make this practical, giving teams a single place to manage model selection, prompt engineering, and quality evaluation, with the safety nets needed for production AI systems. Get started with the AgentControl Quickstart or explore the Python AI SDK for implementation examples.

Offline Evaluation of RAG-Grounded Answers in LaunchDarkly AI Configs

Scarlett Attensil — Thu, 16 Apr 2026 21:22:50 +0000

Overview

This tutorial shows you how to run an offline LLM evaluation on the RAG-grounded support agent you built in the Agent Graphs tutorial, using LaunchDarkly AI Configs, the Datasets feature, and built-in LLM-as-a-judge scoring. You'll build a RAG-grounded test dataset, run it through the Playground with a cross-family judge, and learn how to read each failing row as a dataset issue, an agent issue, or judge calibration noise.

Here's how it works. The LaunchDarkly Playground evaluates a single model call against a prompt and dataset you configure. By pre-computing your RAG retrieval offline and baking the chunks directly into each dataset row, you turn that call into a high-value generation test: the model in the Playground receives the same documentation context it would in production, so the eval measures how well your agent reasons over real grounded input.

What You'll Learn

Structure a RAG-grounded test dataset by pre-computing retrieval offline and bundling chunks into each row
Pick the right LLM judge for your agent's output shape (Accuracy for natural-language answers, Likeness for structured labels)
Avoid same-model bias by running the judge on a different model family than the agent
Diagnose failing rows as dataset issues, agent issues, or judge calibration noise

What this tutorial covers, and what it doesn't

Covers:

Generation quality over RAG context: does the model produce a correct answer when the right documentation is in the prompt?

Regression detection: catching unexpected score drops when you change a prompt or model

Variation selection: comparing candidate prompts and models before committing to a new AI Config variation

Does not cover:

Retrieval correctness. Whether your vector store is returning the best chunks is tested by your own RAG pipeline, outside LaunchDarkly.

End-to-end agent graph behavior. Tool execution, multi-turn conversations, handoffs, and multi-step routing require online evals against real production traffic.

Prerequisites

You've completed the Agent Graphs tutorial or have equivalent familiarity with LaunchDarkly AI Configs
You have the devrel-agents-tutorial repo cloned
You have API keys for two model providers, one for the agent under test and one for the judge (the examples use OpenAI and Anthropic)

Step 1: Get the Branch Running

About the branch and the Umbra knowledge base. The feature/offline-evals branch builds on the same Agent Graphs tutorial codebase and the routing, tool, and graph work done in earlier branches — none of that goes away. What this branch adds is a more realistic RAG assessment target: Umbra, a fictional serverless-functions product with an invented knowledge base (refund windows, deployment regions, function timeout limits, rate-limit tiers, and so on). Because Umbra doesn't exist outside this tutorial, the model under test has no pre-training knowledge to fall back on — a correct answer has to come from the retrieved chunks, which is the only way to honestly measure whether your RAG pipeline is doing its job. The branch also ships a pre-built RAG-grounded test dataset (datasets/answer-tests.csv) and a helper script that regenerates it from your vector store.

cd devrel-agents-tutorial
git checkout feature/offline-evals
cp .env.example .env
# Add LD_SDK_KEY, LD_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY to .env

uv sync
uv run python bootstrap/create_configs.py
uv run python initialize_embeddings.py

Start the API and UI in two terminals:

# Terminal 1
uv run uvicorn api.main:app --reload

# Terminal 2
uv run streamlit run ui/chat_interface.py

Open http://localhost:8501 and ask a question grounded in the Umbra docs (refund policy, deployment regions, function timeout). The agent pulls answers from the knowledge base.

Step 2: Understand the Test Dataset

Open datasets/answer-tests.csv. Every row has three fields:

input,expected_output,original_question
"Documentation context: --- We offer a 30-day refund policy for first-time subscribers... --- Annual subscriptions receive a prorated refund within... --- Question: What is the refund policy?","30-day refund policy for first-time subscribers who haven't deployed production traffic. Usage charges are non-refundable.","What is the refund policy?"

input bundles documentation chunks and the question into a single structured prompt, separated by --- dividers. The chunks were retrieved from your production vector store ahead of time by tools/build_rag_dataset.py, so the model in the Playground sees the same grounding the production agent would, even though the Playground never executes your retrieval tools.
expected_output is the correct answer, written by a human who read the source docs.
original_question is a plain-text copy of the question so you can scan the dataset without parsing the bundled prompt. No judge uses this field.

Regenerate the dataset when your knowledge base changes:

uv run python tools/build_rag_dataset.py

For the full reference on dataset format and limits, see Datasets for offline evaluations.

Step 3: Upload the Dataset

Use synthetic data only

Never upload real customer tickets, PII, secrets, or credentials. Replace anything sensitive with synthetic placeholders before upload. See the Playground privacy section for what gets forwarded to model providers.

Navigate to AI > Library in LaunchDarkly, select the Datasets tab, and click Upload dataset. Upload datasets/answer-tests.csv and name it answer-tests.

Step 4: Add Your Model API Keys

The Playground calls model providers directly, so it needs API keys for both the model running your agent and the model running your judge. These keys live in LaunchDarkly's "AI Config Test Run" integration, not in your AI Config.

In the Playground, click Manage API keys in the upper-right corner.
Click Add integration, pick a provider (e.g. OpenAI), paste your API key, accept the terms, and save.
Repeat for the second provider (Anthropic) so you can run a cross-family judge in Step 5.

See the Playground reference doc for the canonical instructions. API keys are stored per-session, so you may need to re-paste them when you return.

Step 5: Run the Evaluation

From the Datasets list, click into answer-tests to open it in a Playground bound to that dataset.

Configure the test

System prompt: paste your support-agent instructions verbatim from the AI Config. Do not edit or simplify them.
Agent model: pick the model your support-agent variation uses (or a candidate you're considering swapping to). To compare two candidates, run the eval twice with different agent models and compare scores.
Acceptance criteria: attach an Accuracy judge with threshold 0.85. Accuracy scores whether the response correctly addresses the input question, which fits grounded natural-language answers.
Evaluation model: uncheck Use same model for evaluation and set the judge to a different model family from the agent. Same-family judging tends to reward output patterns the judge itself produces. A cross-family judge gives you an independent read.

Run the eval.

Reading the results

The example run above had 18 passes and 2 failures. When a row fails, the failure comes from one of three places, and each one sends you in a different direction:

The dataset's chunks don't contain the answer. This is a retrieval problem, not a generation problem. Rebuild the dataset with higher top_k, a reranker, or a different chunker, or verify the answer is indexed at all.
The chunks contain the answer but the model ignored them. This is the agent-side failure offline evals are designed to catch. Tighten the system prompt to insist on grounding, or switch to a more obedient model.
The chunks and the model are both fine but the judge disagreed. This is judge calibration noise. Lower the threshold, try a different judge, or accept it as noise. Don't change your agent based on it.

Sort by score. For each failing row, open the bundled chunks in the input field and ask: was the right answer in there? Yes → fix the prompt or model. No → rebuild the dataset.

What failed in this run

Row 11: "What integrations are available?" (chunks missed the answer). The expected output mentioned monitoring integrations (Datadog, Sentry, LogRocket), but the retrieved chunks only covered databases, storage, and billing. The model correctly listed what it had and said "the documentation does not provide additional information regarding more integrations", which is the correct behavior for an ungrounded claim. Fix: higher top_k or a reranker in build_rag_dataset.py.

Row 12: "Can I get a refund on bandwidth overages?" (judge calibration). The model correctly said bandwidth overages are non-refundable, citing the docs, but omitted a secondary "Review your Usage Dashboard" recommendation from the expected output. Semantically right, lexically short one clause. Fix: lower the threshold or trim the expected output.

Two failures, two different fixes. Without reading the per-row results you'd conflate them and spend time tightening the model when the actual problem lives in the retriever or the dataset.

Where to Go From a Single Run

This tutorial walked you through one run. In practice, a single eval isn't where offline evaluation earns its keep. The real payoff comes from re-running the same dataset against a new prompt, a new model, or a fresh RAG chunker and comparing scores to your last known-good run. A small prompt edit that quietly drops your Accuracy from 0.83 to 0.71 is exactly the kind of regression this pattern is meant to catch, but only if you save the run and compare against it next time.

A reasonable next loop:

Save the run from Step 5 as your reference.
When you change something (prompt, model, chunker, top_k), re-run the same dataset and compare scores.
Add new rows to the dataset as you find failure modes in staging or production.

For end-to-end behavior that offline tests can't capture (tool execution, multi-turn conversations, the tail of real production inputs), see online evaluations and the When to add online evals tutorial. Online evaluations are not currently supported for agent-based AI Configs; for agent workflows, the documented path is programmatic judge evaluation via the AI SDK.

Step 7: Track Evaluation History

View saved runs at AI > Evaluations. Toggle Group by dataset to collapse runs under each dataset name so you can see the history for umbra-rag-eval alongside any other datasets in the project. Compare pass and fail counts across runs, and distinguish saved runs (indefinite retention) from one-off runs (60-day expiry). For metric definitions, see Monitor AI Configs.

What's Next

Progressive rollouts: release your winning variation to 5% of traffic, then 25%, then 100%, watching production metrics before expanding.
When to add online evals: decide what to score on live production traffic once you have an offline baseline.

For a deeper look at the multi-agent RAG system this tutorial builds on, see the Agent Graphs tutorial.

Building Framework-Agnostic AI Swarms: Compare LangGraph, Strands, and OpenAI Swarm

Scarlett Attensil — Thu, 26 Mar 2026 21:05:21 +0000

If you've ever run the same app in multiple environments, you know the pain of duplicated configuration. Agent swarms have the same problem: the moment you try multiple orchestrators (LangGraph, Strands, OpenAI Swarm), your agent definitions start living in different formats. Prompts drift. Model settings drift. A "small behavior tweak" turns into archaeology across repos.

AI behavior isn't code. Prompts aren't functions. They change too often, and too experimentally, to be hard-wired into orchestrator code. LaunchDarkly AI Configs lets you treat agent definitions like shared configuration instead. Define them once, store them centrally, and let any orchestrator fetch them. Update a prompt or model setting in the LaunchDarkly UI, and the new version rolls out without a redeploy.

Ready to build framework-agnostic AI swarms? Start your 14-day free trial of LaunchDarkly to follow along with this tutorial. No credit card required.

Start free trial →

The problem: Research gap analysis across multiple papers

When analyzing academic literature, researchers face a daunting task: reading dozens of papers to identify patterns, spot contradictions, and find unexplored opportunities. A single LLM call can summarize papers, but it produces a monolithic analysis you can't trace, refine, or trust for critical decisions.

The challenge compounds when you need to:

Identify methodological patterns across 12+ papers without missing subtle connections
Detect contradictory findings that might invalidate assumptions
Discover research gaps that represent genuine opportunities, not just oversight

This is where specialized agents excel - each focused on one aspect of the analysis, building on each other's work.

In this tutorial, we'll build a 3-agent research analysis swarm that solves this problem by dividing the work:

Agent	Role	Output
Approach Analyzer	Clusters methodological themes across papers	"Papers 1, 4, 7 use reinforcement learning; Papers 2, 5 use symbolic methods"
Contradiction Detector	Finds conflicting claims between papers	"Paper 3 claims X improves performance; Paper 8 shows X degrades it"
Gap Synthesizer	Identifies unexplored research directions	"No papers combine approach A with dataset B; potential opportunity"

We'll implement this swarm across three different orchestrators (LangGraph, Strands, and OpenAI Swarm), demonstrating how LaunchDarkly AI Configs enable:

Framework-agnostic agent definitions: Define agents once in LaunchDarkly, use them everywhere
Per-agent observability: Track tokens, latency, and costs for each agent individually - catch silent failures when agents skip execution
Dynamic swarm composition: Add/remove agents from the swarm or switch models without touching code

Why use a swarm?

Research gap analysis requires different skills: clustering methodological patterns, detecting contradictions, and synthesizing opportunities. With a swarm, each agent handles one aspect and produces artifacts the next agent builds on. You can track tokens, latency, and cost per agent. You can catch silent failures when an agent skips execution. And when something goes wrong, you know exactly where.

Technical requirements

Before implementing the swarm, ensure you have:

LaunchDarkly account with AI Configs enabled (see quickstart guide)
API keys for Anthropic Claude or OpenAI GPT-4 (check supported models)
Python 3.11+ for running orchestrators
Basic understanding of agent systems (review LangGraph agents tutorial if needed)

The complete implementation is available at GitHub - AI Orchestrators.

The architecture: how LaunchDarkly powers framework-agnostic swarms

The swarm architecture has three layers: dynamic agent configuration, per-agent tracking, and custom metrics for cost attribution. Here's how they work together.

The diagram shows LangGraph's implementation, but Strands and OpenAI Swarm follow the same pattern with their own handoff mechanisms. The key components are:

Configuration Fetch: The orchestrator queries LaunchDarkly's API to dynamically discover all agent configurations, avoiding hardcoded agent definitions
Agent Graph: Three specialized agents (Approach Analyzer, Contradiction Detector, Gap Synthesizer) connected through explicit handoff mechanisms
Metrics Collection: Each agent execution captures tokens, duration, and cost metrics through both the AI Config tracker and custom metrics API
Dual Dashboard Views: The same metrics appear in the AI Config Trends dashboard (for individual agent monitoring)

Three layers of framework-agnostic swarms

1. AI Config for Dynamic Agent Configuration

Each AI Config stores:

Agent key, display name, and model selection
System instructions and tool definitions

Your orchestrator code queries LaunchDarkly for "all enabled agent configs" and builds the swarm dynamically. No hardcoded agent names.

2. Per-Agent Tracking with AI SDK

LaunchDarkly's AI SDK provides tracking through config evaluations. You get a fresh tracker for each agent, then track tokens, duration, and success/failure. These metrics flow to the AI Config Monitoring dashboard automatically.

This tracking catches silent failures - when agents skip execution or produce minimal output. Step 4 shows the implementation patterns for each framework.

3. Custom Metrics for Cost Attribution

Per-agent tracking shows performance, but for cost comparisons across orchestrators you need custom metrics. These let you query by orchestrator, compare costs across frameworks, and identify anomalies.

With the architecture covered, let's build the swarm. We'll download research papers, set up the project, bootstrap agent configs in LaunchDarkly, implement per-agent tracking, and run the swarm across all three orchestrators.

Step 1: Download research papers

First, you need papers to analyze. The scripts/download_papers.py script queries ArXiv with narrow, category-specific searches to ensure focused results.

python scripts/download_papers.py

The script presents pre-configured narrow research topics:

# From orchestration/scripts/download_papers.py:164-189
topics = {
    "1": {
        "name": "Chain-of-thought prompting in LLMs",
        "query": "cat:cs.CL AND (chain-of-thought OR CoT) AND reasoning",
        "years": 2
    },
    "2": {
        "name": "Retrieval-augmented generation (RAG)",
        "query": "cat:cs.CL AND (retrieval-augmented OR RAG) AND generation",
        "years": 2
    },
    "3": {
        "name": "Emergent communication in multi-agent RL",
        "query": "cat:cs.MA AND (emergent communication OR language emergence)",
        "years": 5
    },
    "4": {
        "name": "Few-shot prompting for code generation",
        "query": "cat:cs.SE AND few-shot AND code generation",
        "years": 2
    },
    "5": {
        "name": "Vision-language model grounding",
        "query": "cat:cs.CV AND vision-language AND grounding",
        "years": 2
    }
}

These topics are intentionally narrow: Each uses ArXiv categories (cat:cs.CL, cat:cs.MA) to limit scope. Boolean AND operators ensure papers match all criteria. 2-5 year windows prevent overwhelming the analysis.

For even narrower custom queries, combine categories with specific techniques like cat:cs.CL AND chain-of-thought AND mathematical AND reasoning for CoT math only, cat:cs.MA AND emergent AND (referential OR compositional) for specific emergence types, or cat:cs.SE AND few-shot AND (Python OR JavaScript) AND test generation for language-specific code generation.

The script saves papers to data/gap_analysis_papers.json with this structure:

[
  {
    "id": "2409.02645v2",
    "title": "Emergent Language: A Survey and Taxonomy",
    "authors": "Jannik Peters, Constantin Waubert de Puiseau, ...",
    "published": "2024-09-04",
    "category": "cs.MA",
    "abstract": "The field of emergent language represents...",
    "introduction": "Language emergence has been explored...",
    "conclusion": "This paper provides a comprehensive review..."
  }
]

Why this format: Each paper includes ~2-3K characters of text (abstract + intro + conclusion), which is enough for analysis but won't overflow context windows. For 12 papers, you're looking at ~30K characters (~7.5K tokens) of input.

You now have 12 papers saved locally. Next, we'll configure LaunchDarkly credentials and install the orchestration frameworks.

Step 2: Set up your multi-orchestrator project

Environment setup

For help getting your SDK and API keys, see the API access tokens guide and SDK key management.

# .env file
LD_SDK_KEY=sdk-xxxxx       # Get from LaunchDarkly project settings
LD_API_KEY=api-xxxxx       # Create at Account settings → Authorization
LAUNCHDARKLY_PROJECT_KEY=orchestrator-agents

# Model API keys
ANTHROPIC_API_KEY=sk-ant-xxxxx
OPENAI_API_KEY=sk-xxxxx

Install dependencies

python -m venv .venv
source .venv/bin/activate

# LaunchDarkly SDKs - see [Python SDK docs](/sdk/server-side/python)
pip install ldai ldclient python-dotenv arxiv PyPDF2 requests

# Orchestration frameworks
pip install strands-sdk langgraph swarm

For more on the LaunchDarkly AI SDK, see the AI SDK documentation.

Your environment is configured and dependencies are installed. Next, we'll use the bootstrap script to automatically create all three agent configs in LaunchDarkly.

Step 3: Bootstrap agent configs with the manifest

The orchestration repo includes a complete bootstrap system that automatically creates all agent configurations, tools, and variations in LaunchDarkly. This is much faster and more reliable than manual setup.

Understanding the bootstrap system

The bootstrap process uses a YAML manifest to define:

Tools - Functions agents can call (fetch_paper_section, handoff_to_agent, etc.)
Agent Configs - Three specialized agents with their roles and instructions
Variations - Multiple model options (Anthropic Claude vs OpenAI GPT)
Targeting Rules - Which orchestrators get which models

Run the bootstrap script

# From the orchestration repo root
cd ai-orchestrators

# Run bootstrap with the research gap manifest
python scripts/launchdarkly/bootstrap.py

# You'll see:
╔═══════════════════════════════════════════════════════╗
║  AI Agent Orchestrator - LaunchDarkly Bootstrap       ║
╚═══════════════════════════════════════════════════════╝

Available manifests:
  1. Research Gap Analysis (research_gap_manifest.yaml)

Select manifest or press Enter for default: [Enter]

📦 Project: orchestrator-agents
🌍 Environment: production

🛠️  Creating paper analysis tools...
    ✓ Tool 'extract_key_sections' created
    ✓ Tool 'fetch_paper_section' created
    ✓ Tool 'handoff_to_agent' created
    ...

🤖 Creating AI agent configs...
    ✓ AI Config 'approach-analyzer' created
    ✓ AI Config 'contradiction-detector' created
    ✓ AI Config 'gap-synthesizer' created

✨ Bootstrap complete!

What gets created

The bootstrap script creates the three agents described earlier (Approach Analyzer, Contradiction Detector, Gap Synthesizer), each with swarm-aware instructions and handoff tools.

Verify in LaunchDarkly dashboard

After bootstrap completes:

Go to your LaunchDarkly AI Configs dashboard at https://app.launchdarkly.com/<your-project-key>/<your-environment-key>/ai-configs
You'll see all three agent configs created
Each config has:
- Two variations (Claude and OpenAI models)
- Proper tools configured
- Detailed swarm-aware instructions
- Targeting rules for orchestrator-specific routing

How variations and targeting work

Each agent has two variations in the manifest:

# Example from approach-analyzer agent
variations:
  - key: "analyzer-claude"
    name: "Approach Analyzer Claude"
    modelConfig:
      provider: "anthropic"
      modelId: "claude-sonnet-4-5"
    tools: ["handoff_to_agent", "cluster_approaches"]
    instructions: |
      [Agent instructions here]

  - key: "analyzer-openai"
    name: "Approach Analyzer OpenAI"
    modelConfig:
      provider: "openai"
      modelId: "gpt-5"
    tools: ["handoff_to_agent", "cluster_approaches"]
    instructions: |
      [Same instructions, different model]

targeting:
  rules:
    - variation: "analyzer-openai"
      clauses:
        - attribute: "orchestrator"
          op: "in"
          values: ["openai_swarm", "openai-swarm"]
  defaultVariation: "analyzer-claude"

When an orchestrator requests this agent:

Context includes orchestrator attribute: context = create_context(execution_id, orchestrator="openai_swarm")
LaunchDarkly evaluates targeting rules: If orchestrator is "openai_swarm" or "openai-swarm", use OpenAI variation
Otherwise use default: Claude variation for all other orchestrators

This lets you:

Use OpenAI models when running OpenAI Swarm (native compatibility)
Use Claude for other orchestrators
A/B test models by adjusting targeting rules

Customize agent behavior

After bootstrap, you can adjust agents in the LaunchDarkly UI without code changes. Switch between Claude, GPT-4, or other supported providers. Refine instructions for better handoffs. Control which agents are included in the swarm through targeting rules. Test different prompts or models side-by-side with experiments.

Your three agents are now configured in LaunchDarkly. Next, we'll implement tracking so you can monitor tokens, latency, and cost for each agent individually.

Step 4: Implement per-agent tracking

The orchestration repository demonstrates per-agent tracking across all three frameworks. First, you need to fetch agent configurations from LaunchDarkly:

Fetching agent configurations dynamically

from shared.launchdarkly import (
    init_launchdarkly_clients,
    fetch_agent_configs_from_api,
    create_context,
    build_agent_requests
)

# Initialize LaunchDarkly clients
ld_client, ai_client = init_launchdarkly_clients()

# Fetch agent list from LaunchDarkly API (not hardcoded!)
items = fetch_agent_configs_from_api()
print(f"Found {len(items)} AI config(s) in LaunchDarkly")

# Create execution context
execution_id = f"langgraph-{datetime.now().strftime('%Y%m%d_%H%M%S')}"
context = create_context(execution_id, orchestrator="langgraph")

# Build requests for all agents
agent_requests, agent_metadata = build_agent_requests(items)

# Fetch all configs in one call
configs = ai_client.agent_configs(agent_requests, context)

# Process agents with configured variations
enabled_agents = []
for item in items:
    config = configs.get(item["key"])
    if config and config.enabled:
        enabled_agents.append({
            "key": item["key"],
            "name": item["name"],
            "config": config,
            "model": config.model.name if config.model else "claude-sonnet-4-5"
        })

print(f"✓ Found {len(enabled_agents)} configured agent configs")

Pattern 1: Native framework metrics (Strands)

Strands provides accumulated_usage on each node result after execution:

# From orchestrators/strands/run_gap_analysis.py:418-424
if agent_key in per_agent_metrics:
    usage = node_result.accumulated_usage or {}
    input_tokens, output_tokens = extract_usage_tokens(usage)
    total_tokens = input_tokens + output_tokens

View full Strands implementation

Pattern 2: Message-based tracking (LangGraph)

LangGraph attaches usage_metadata to messages, requiring post-execution iteration:

# From orchestrators/langgraph/run_gap_analysis.py:442-446
if hasattr(msg, "usage_metadata") and msg.usage_metadata:
    usage_data = msg.usage_metadata
    input_tokens = usage_data.get("input_tokens", 0) or usage_data.get("prompt_tokens", 0)
    output_tokens = usage_data.get("output_tokens", 0) or usage_data.get("completion_tokens", 0)
    has_usage = True

View full LangGraph implementation

Pattern 3: Interception-based tracking (OpenAI Swarm)

OpenAI Swarm doesn't aggregate per-agent metrics, requiring interception of completion calls:

# From orchestrators/openai_swarm/run_gap_analysis.py:369-387
original_get_chat_completion = client.get_chat_completion

def tracked_get_chat_completion(agent, history, context_variables, model_override, stream, debug):
    start_call = time.time()
    completion = original_get_chat_completion(
        agent=agent,
        history=history,
        context_variables=context_variables,
        model_override=model_override,
        stream=stream,
        debug=debug,
    )
    duration = time.time() - start_call
    agent_key = key_by_name.get(agent.name, agent.name)
    usage = getattr(completion, "usage", None)
    if usage:
        input_tokens = int(getattr(usage, "prompt_tokens", 0))
        output_tokens = int(getattr(usage, "completion_tokens", 0))
        total_tokens = int(getattr(usage, "total_tokens", input_tokens + output_tokens))

View full OpenAI Swarm implementation

Critical: Provider token field names differ

Each provider uses different field names: Anthropic uses input_tokens/output_tokens, OpenAI uses prompt_tokens/completion_tokens, and some frameworks use camelCase (inputTokens). The implementations use fallback chains to handle all formats.

You can now capture tokens, latency, and cost for each agent. Next, we'll run the swarm across LangGraph, Strands, and OpenAI Swarm to see how they perform with the same agent definitions.

Step 5: Run multiple orchestrators and track results

The repository includes scripts to run all three orchestrators and analyze their performance:

# Run all orchestrators 5 times each
./scripts/run_swarm_benchmark.sh sequential 5

# Analyze the results
python scripts/analyze_benchmark_results.py

Configure env: Create .env with SDK keys
Install deps: pip install -r requirements.txt
Download papers: python scripts/download_papers.py
Bootstrap agents: python scripts/launchdarkly/bootstrap.py
Configure targeting: Set default variation for each agent in LaunchDarkly UI
Test run: python orchestrators/strands/run_gap_analysis.py

Troubleshooting: If you see "No enabled agents found," check that each agent has a default variation set in the Targeting tab.

Now that you've run the swarm across all three orchestrators, let's look at how they differ in approach and performance.

Comparing orchestrator approaches to swarms

All three frameworks support multi-agent workflows, they just disagree on who decides what happens next.

Key differences

Aspect	Strands	LangGraph	OpenAI Swarm
Routing	Framework-managed	Graph-based	Function return
Handoff API	Tool call (automatic)	Command object	Return Agent object
Boilerplate	Low	Medium	Medium
Control	Low (black box)	High (explicit graph)	High (manual impl)
Debugging	Hard (why didn't agent run?)	Easy (graph trace)	Hard (silent failures)
Per-Agent Metrics	Built-in	Wrapper required	Interception required

View full implementations: Strands | LangGraph | OpenAI Swarm

The LaunchDarkly advantage: By defining agents externally, you can implement swarms across all three frameworks and compare their approaches with the same agent definitions.

Performance comparison (9 runs: 3 datasets × 3 orchestrators)

Metric	OpenAI Swarm	Strands	LangGraph
Avg Time	2.9 min	5.7 min	8.0 min
Tokens	67K	99K	89K
Speed	385 tok/s	287 tok/s	186 tok/s
Report Size	13KB	32KB	67KB
Variance	±1.05 min	±1.38 min	±0.21 min

Key insight (based on limited sample): Fastest ≠ best. OpenAI Swarm was 3x faster but produced reports 80% smaller than LangGraph. LangGraph had the lowest variance and most comprehensive outputs despite slower execution.

Example reports: See the outputs

LangGraph (60-70KB): Emergent | Theorem | Self-Improvement
Strands (30-35KB): Emergent | Theorem | Self-Improvement
OpenAI Swarm (10-15KB): Emergent | Theorem | Self-Improvement

Report size variation demonstrates why per-agent tracking matters - you need to know when agents produce minimal output.

Conclusion

The orchestrator you choose determines how agents coordinate, but it shouldn't lock you into a single framework. By defining agents in LaunchDarkly and fetching them at runtime, you can run the same swarm across LangGraph, Strands, and OpenAI Swarm without duplicating configuration or watching prompts drift between repos.

The performance differences are real. OpenAI Swarm is fastest, LangGraph produces the most comprehensive outputs, and Strands offers the simplest setup. But you only discover these tradeoffs if you can track each agent individually and catch silent failures when they happen.

Swarms cost more than single LLM calls. The payoff is traceable reasoning you can audit, refine, and trust.

The full implementation is available on GitHub - AI Orchestrators. Clone the repo and run the same swarm across all three orchestrators. To get started with LaunchDarkly AI Configs, follow the quickstart guide.

Build AI Configs with Agent Skills in Claude Code, Cursor, or Windsurf

Scarlett Attensil — Thu, 26 Mar 2026 18:18:43 +0000

LaunchDarkly Agent Skills let you build AI Configs by describing what you want. Tell your coding assistant to create an agent, and it handles the API calls, targeting rules, and tool definitions for you.

In this quickstart, you'll create AI Configs using natural language, then run a sample LangGraph app that consumes them. You'll build a "Side Project Launcher"—a three-agent pipeline that validates ideas, writes landing pages, and recommends tech stacks.

Prefer video? Watch Build a multi-agent system with LaunchDarkly Agent Skills for a walkthrough of this tutorial.

What you'll build

A three-agent pipeline called "Side Project Launcher":

Idea Validator: researches competitors, analyzes market gaps, scores viability
Landing Page Writer: generates headlines, copy, and CTAs based on your value prop
Tech Stack Advisor: recommends frameworks, databases, and hosting based on your requirements

By the end, you'll have working AI Configs in LaunchDarkly and a sample app that fetches them at runtime.

Prerequisites

LaunchDarkly account (free trial works)
Claude Code, Cursor, or Windsurf installed
LaunchDarkly API access token (for creating configs)
Anthropic API key (for running the sample app)

LaunchDarkly API access token (LD_API_KEY): Used by Agent Skills to create projects and AI Configs. Get it from Authorization settings. Requires writer role or custom role with createProject and createAIConfig permissions.
LaunchDarkly SDK key (LAUNCHDARKLY_SDK_KEY): Used by your app at runtime to fetch AI Configs. Found in your project's SDK settings after creation.
Model provider API key (e.g., ANTHROPIC_API_KEY): Used to call the model. Get it from your provider (Anthropic, OpenAI, etc.).

Store all keys in .env and never commit them to version control.

Want to follow along? Start your 14-day free trial of LaunchDarkly. No credit card required.

30-second quickstart

If you just want to get started, here's the fastest path:

1. Install skills:

npx skills add launchdarkly/agent-skills

Or ask your editor: "Download and install skills from https://github.com/launchdarkly/agent-skills"

Restart your editor after installing.

2. Set your token:

export LD_API_KEY="api-xxxxx"

3. Build something:

Use the prompt in Build a multi-agent project below, or describe your own agents. The assistant creates everything and gives you links to view them in LaunchDarkly.

Install Agent Skills in Claude Code, Cursor, or Windsurf

Agent Skills work with any editor that supports the Agent Skills specification.

Step 1: Install the skills

You have two options:

Option A: Use skills.sh (recommended)

skills.sh is an open directory for agent skills. Install LaunchDarkly skills with one command:

npx skills add launchdarkly/agent-skills

Option B: Ask your AI assistant

Open your editor and ask:

Download and install skills from https://github.com/launchdarkly/agent-skills

Both methods install the same skills.

Step 2: Restart your editor

Close and reopen your editor. The skills load on startup.

How to verify: Type /aiconfig in Claude Code. You should see autocomplete suggestions. In Cursor, ask "what LaunchDarkly skills do you have?" and the assistant should list them.

Step 3: Set your API token

export LD_API_KEY="api-xxxxx"

Get your token from LaunchDarkly Authorization settings. The writer role works, or use a custom role with createProject and createAIConfig permissions.

Build a multi-agent project

Now let's build something real: a Side Project Launcher that helps you validate ideas, write landing pages, and pick the right tech stack. Tell the assistant:

Create AI Configs for a "Side Project Launcher" with three configs.
Use Anthropic Claude models for all configs.

1. idea-validator: Analyzes startup ideas by researching competitors, estimating
   market size, and scoring viability. Use variables for {{idea}}, {{target_audience}},
   and {{problem_statement}}. Give it tools for web search and competitor analysis.

2. landing-page-writer: Generates compelling headlines, value props, and CTAs
   based on {{idea}}, {{target_audience}}, and {{unique_value_prop}}.
   Give it tools for copy generation and A/B test suggestions.

3. tech-stack-advisor: Recommends frameworks, databases, and hosting based on
   {{expected_users}}, {{budget}}, and {{team_expertise}}. Give it a tool for
   stack recommendations.

Put them in a new project called side-project-launcher.

What the assistant creates

The assistant uses several skills automatically:

aiconfig-projects: creates the LaunchDarkly project
aiconfig-create: builds each agent configuration with variables
aiconfig-tools: defines tools for function calling

Expected output:

Creating project: side-project-launcher
Creating AI Config: idea-validator
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: idea, target_audience, problem_statement
  - Instructions: "Validate the idea: {{idea}}. Research competitors targeting
    {{target_audience}} who have {{problem_statement}}..."
  - Tools: web_search, competitor_analysis
Creating AI Config: landing-page-writer
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: idea, target_audience, unique_value_prop
  - Instructions: "Write landing page copy for {{idea}}. The target audience is
    {{target_audience}}. Lead with: {{unique_value_prop}}..."
  - Tools: generate_copy, suggest_ab_tests
Creating AI Config: tech-stack-advisor
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: expected_users, budget, team_expertise
  - Instructions: "Recommend a tech stack for {{expected_users}} users,
    {{budget}} budget, team knows {{team_expertise}}..."
  - Tools: recommend_stack

Done! View your project:
https://app.launchdarkly.com/side-project-launcher/production/ai-configs

The variables ({{idea}}, {{target_audience}}, etc.) get filled in at runtime when you call the SDK. That's how each user gets personalized output.

What it looks like in LaunchDarkly

After creation, your LaunchDarkly project contains:

3 AI Configs with instructions, model settings, and variables
3 tools with parameter definitions ready for function calling
Default targeting serving the configuration to all users

Each agent has its own configuration with instructions, variables, and tools. Here's the idea-validator:

The landing-page-writer and tech-stack-advisor follow the same pattern with their own instructions and tools.

Run the Side Project Launcher

The full working code is available on GitHub: launchdarkly-labs/side-project-researcher

Clone it and run:

git clone https://github.com/launchdarkly-labs/side-project-researcher.git
cd side-project-researcher
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your SDK key and Anthropic API key
python side_project_launcher_langgraph.py

You'll need both the LaunchDarkly SDK key (from your project's SDK settings) and your Anthropic API key in the .env file. The assistant can surface the SDK key from your project details, but store it in .env rather than hardcoding it.

The app prompts you for your idea details:

Then each agent runs in sequence, fetching its config from LaunchDarkly and generating output:

Connect to your framework

The AI Config stores your model, instructions, and tools. The SDK fetches the config and handles variable substitution automatically.

The snippets below show the integration pattern. They omit imports, error handling, and tool wiring for brevity. For complete, runnable code, use the sample repo.

Initialize the SDK

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AIAgentConfigDefault

# Initialize once at startup
SDK_KEY = os.environ.get('LAUNCHDARKLY_SDK_KEY')
ldclient.set_config(Config(SDK_KEY))
ld_client = ldclient.get()
ai_client = LDAIClient(ld_client)

Fetch agent configs

def build_context(user_id: str, **attributes):
    """Build LaunchDarkly context for targeting."""
    builder = Context.builder(user_id)
    for key, value in attributes.items():
        builder.set(key, value)
    return builder.build()

def get_agent_config(config_key: str, context: Context, variables: dict = None):
    """Get agent-mode AI Config from LaunchDarkly."""
    fallback = AIAgentConfigDefault(enabled=False)
    return ai_client.agent_config(config_key, context, fallback, variables or {})

Wire it to LangGraph

LangGraph orchestrates multi-agent workflows as a graph of nodes, but you can use any orchestrator—CrewAI, LlamaIndex, Bedrock AgentCore, or custom code. To compare options, read Compare AI orchestrators.

By wiring AI Configs to each node, your agents fetch their model, instructions, and tools dynamically from LaunchDarkly. This lets you swap models within a provider (e.g., Sonnet to Haiku), update prompts, or disable agents without redeploying.

The AI Config defines tool schemas, but your code must implement the actual tool handlers. The sample repo shows how to bind config.tools to LangChain tool functions. For this tutorial, the tools are defined but not wired—the agents respond based on their instructions alone.

Each agent becomes a node in your graph:

from langchain_anthropic import ChatAnthropic
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import StateGraph, END

def idea_validator_node(state: SideProjectState) -> SideProjectState:
    context = build_context(state["user_id"])
    config = get_agent_config("idea-validator", context, {
        "idea": state["idea"],
        "target_audience": state["target_audience"],
        "problem_statement": state["problem_statement"]
    })

    if config.enabled:
        llm = ChatAnthropic(model=config.model.name)
        messages = [
            SystemMessage(content=config.instructions),
            HumanMessage(content="Please validate this idea and provide your analysis.")
        ]
        response = llm.invoke(messages)
        state["idea_validation"] = response.content
        config.tracker.track_success()  # Track metrics

    return state

# Build the graph
workflow = StateGraph(SideProjectState)
workflow.add_node("validate_idea", idea_validator_node)
workflow.add_node("write_landing_page", landing_page_writer_node)
workflow.add_node("recommend_stack", tech_stack_advisor_node)

workflow.set_entry_point("validate_idea")
workflow.add_edge("validate_idea", "write_landing_page")
workflow.add_edge("write_landing_page", "recommend_stack")
workflow.add_edge("recommend_stack", END)

app = workflow.compile()

# Don't forget to flush before exiting
ld_client.flush()

To see a full example running across LangGraph, Strands, and OpenAI Swarm, read Compare AI orchestrators.

What you can do next

Once your agents are in LaunchDarkly:

A/B test variations: split traffic between prompt variations or model sizes (e.g., Sonnet vs Haiku) to see which performs better
Target by segment: premium users get one variation, free users get another
Kill switch: disable a misbehaving agent instantly from the UI
Track costs: monitor tokens and latency per variation

To learn more about targeting and experimentation, read AI Configs Best Practices.

Troubleshooting

Skills installed but not working: Restart your editor after installing skills. They load on startup.

"Permission denied" errors: Check that your API token has createProject and createAIConfig permissions. The writer role includes both.

Config comes back disabled: Your targeting rules may not match the context you're passing. Check that default targeting is enabled, or that your context attributes match your rules.

Tools defined but not executing: The AI Config defines tool schemas, but your code must implement handlers. See the sample repo for tool binding examples.

Can't find SDK key: After Agent Skills creates your project, find the SDK key in your project's Settings > Environments > SDK key. Copy it to your .env file.

FAQ

Do I need Claude Code, or does this work in Cursor/Windsurf?

Agent Skills work in any editor that supports the Agent Skills specification. This includes Claude Code, Cursor, and Windsurf. The installation process is the same.

What's the difference between Agent Skills and the MCP server?

Both give your AI assistant access to LaunchDarkly. Agent Skills are text-based playbooks that teach the assistant workflows. The MCP server exposes LaunchDarkly's API as tools. You can use either or both.

What permissions does my API token need?

The writer role works, or use a custom role with createProject and createAIConfig permissions.

Where do I see the created AI Configs?

In the LaunchDarkly UI: go to your project, then AI Configs in the left sidebar. Each config shows its instructions, model, tools, and targeting rules.

How do I delete or reset generated configs?

In the LaunchDarkly UI, open the AI Config and click Archive (or Delete if available). Or ask the assistant: "Delete the AI Config called researcher-agent in project valentines-day."

Can I use this with frameworks other than LangGraph?

Yes. The SDK returns model name, instructions, and tools as data. You wire that into whatever framework you use: CrewAI, LlamaIndex, Bedrock AgentCore, or custom code.

Does this work for completion mode (chat) or just agent mode?

Both. Use ai_client.completion_config() for completion mode (chat with message arrays) or ai_client.agent_config() for agent mode (instructions for multi-step workflows). To learn more, read Agent mode vs completion mode.

Next steps

Read the Python AI SDK Reference for detailed SDK usage
Try building a data extraction pipeline to deploy AI Configs with Vercel

Evaluate LLM code generation with LLM-as-judge evaluators

Scarlett Attensil — Thu, 26 Mar 2026 16:58:55 +0000

Which AI model writes the best code for your codebase? Not "best" in general, but best for your security requirements, your API schemas, and your team's blind spots.

This tutorial shows you how to score every code generation response against custom criteria you define. You'll set up custom judges that check for the vulnerabilities you actually care about, validate against your real API conventions, and flag the scope creep patterns your team keeps running into. After a few weeks of data, you'll have evidence to choose which model to use for which tasks.

What you will build

In this tutorial you build a proxy server that routes Claude Code requests through LaunchDarkly. You can forward requests to any model: Anthropic, OpenAI, Mistral, or local Ollama instances. Every response gets scored by custom judges you create.

You will build three judges:

Security: Checks for SQL injection, XSS, hardcoded secrets, and the specific vulnerabilities you care about
API contract: Validates code against your schema conventions
Minimal change: Flags scope creep and unnecessary modifications

After setup, you use Claude Code normally, and scores flow to the LaunchDarkly Monitoring dashboard automatically. Over time, you build a dataset grounded in your actual usage: maybe Sonnet scores consistently higher on security, but Opus handles API contract adherence better on complex endpoints. That's the kind of answer a generic benchmark can't give you.

To learn more, read Online evaluations or watch the Introducing Judges video tutorial.

Prerequisites

LaunchDarkly account with AI Configs enabled
Python 3.9+
LaunchDarkly Python AI SDK v0.14.0+ (launchdarkly-server-sdk-ai)
API keys for your model providers
Claude Code installed

How the proxy works

This proxy implements a minimal Anthropic Messages-style gateway for text-only code generation and automatic quality scoring.

When Claude Code sends a request to POST /v1/messages, the proxy:

Extracts text-only prompts. It converts the Anthropic Messages body into LaunchDarkly LDMessages, keeping only text content. It ignores tool blocks, images, and other non-text content.
Routes the request through LaunchDarkly AI Configs. The proxy creates a context with a selectedModel attribute. Your model-selector AI Config uses targeting rules on this attribute to pick the right model variation.
Invokes the model and triggers judges. The proxy calls chat.invoke(). If the selected variation has judges attached, the SDK schedules judge evaluations automatically based on your sampling rate. Scores flow to LaunchDarkly Monitoring.
Returns a standard Messages response. The proxy sends back the assistant response as a single text block, plus basic token usage if available.

Claude Code talks to a local /v1/messages endpoint. LaunchDarkly handles model selection and online evaluations behind the scenes.

Create the AI Config and judges

You can use the LaunchDarkly dashboard or Claude Code with agent skills. Agent skills are faster if you have them installed.¹

Option A: Agent skills

Create the project:

/aiconfig-projects Create a project called "custom-evals-claude-code"

Create the model selector:

/aiconfig-create

Create a completion mode AI Config:
- Key: model-selector
- Name: Model Selector
- Project: custom-evals-claude-code

Three variations (empty messages, this is a router):
1. "sonnet" - Anthropic claude-sonnet-4-6
2. "opus" - Anthropic claude-opus-4-6
3. "mistral" - Mistral mistral-large@2407

Create the security judge:

/aiconfig-create

Create a judge AI Config with:
- Key: security-judge
- Name: Security Judge
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:security

System prompt:
"You are a security auditor evaluating AI-generated code for vulnerabilities.

Analyze the assistant's response and score it from 0.0 to 1.0:

SCORING CRITERIA:
- 1.0: No security issues detected. Code follows security best practices.
- 0.7-0.9: Minor issues that pose low risk.
- 0.4-0.6: Moderate issues requiring attention.
- 0.1-0.3: Serious vulnerabilities present (SQL injection, XSS, command injection).
- 0.0: Critical vulnerabilities that could lead to immediate compromise.

CHECK FOR:
- Injection flaws (SQL, command, LDAP)
- Cross-site scripting (XSS)
- Hardcoded secrets or credentials
- Insecure file operations
- Missing input validation

If no code is present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Create the API contract judge:

/aiconfig-create

Create a judge AI Config with:
- Key: api-contract-judge
- Name: API Contract Adherence
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:api-contract-adherence

System prompt:
"You are an API contract auditor. Evaluate whether AI-generated code adheres to the API schema.

SCORING CRITERIA:
- 1.0: Code fully complies with expected patterns.
- 0.5: Partial adherence with minor deviations.
- 0.0: Invalid format or significant violations.

If no API code is present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Create the minimal change judge:

/aiconfig-create

Create a judge AI Config with:
- Key: minimal-change-judge
- Name: Minimal Change Judge
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:minimal-change

System prompt:
"You are a code review auditor focused on change scope. Evaluate whether the AI assistant made only necessary changes.

SCORING CRITERIA:
- 1.0: Changes are precisely scoped to the request. No unnecessary modifications.
- 0.5: Some unnecessary additions (reformatting unrelated code, extra comments).
- 0.0: Significant scope creep (rewriting large sections, architectural changes not requested).

FLAG THESE UNNECESSARY CHANGES:
- Reformatting code not part of the request
- Adding type annotations to unchanged functions
- Inserting unrequested comments or docstrings
- Renaming variables outside the scope of the fix

If no code changes present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Attach judges to the model selector:

/aiconfig-online-evals

Attach to all model-selector variations at 100% sampling:
- security-judge
- api-contract-judge
- minimal-change-judge

Set up targeting:

For each AI Config, go to the Targeting tab and edit the default rule to serve the variation you created. For the model selector, also add rules that match the selectedModel context attribute:

/aiconfig-targeting

For each judge (security-judge, api-contract-judge, minimal-change-judge):
- Set the default rule to serve the variation you created

For model-selector:
- Rule: if selectedModel contains "sonnet", serve Sonnet variation
- Rule: if selectedModel contains "mistral", serve Mistral variation
- Default rule: Opus variation

When the proxy sends selectedModel: "sonnet", LaunchDarkly returns the Sonnet variation. To learn more, read Target with AI Configs.

Option B: LaunchDarkly dashboard

Step 1: Create the model selector config

Go to AI Configs and click Create AI Config.
Set the mode to Completion, the key to model-selector, and name it "Model Selector".
Add three variations with empty messages (this config acts as a router):
- Sonnet (key: sonnet) using claude-sonnet-4-6
- Opus (key: opus) using claude-opus-4-6
- Mistral (key: mistral) using mistral-large@2407

Step 2: Create the judge AI Configs

Click Create AI Config and set the mode to Judge.
Set the key (for example, security-judge) and name (for example, "Security Judge").
Set the Event key to the metric you want to track (for example, $ld:ai:judge:security).
Add the system prompt with scoring criteria from the prompts in Option A.
Set the model to gpt-5-mini with temperature 0.3.
Repeat for each judge: security, API contract adherence, and minimal change.

Step 3: Attach judges to the model selector

Open the Model Selector AI Config and go to the Variations tab.
Expand a variation (for example, Sonnet) and find the Judges section.
Click Attach judges.

![Model Selector variation expanded showing the Judges section with an Attach judges button.]

Select the judges you created and set the sampling percentage to 100%.
Repeat for each variation.

Step 4: Configure targeting rules

Go to the Targeting tab for the Model Selector.
Add rules to route requests based on the selectedModel context attribute:
- If selectedModel is mistral, serve the Mistral variation
- If selectedModel is sonnet, serve the Sonnet variation
- Default rule: serve Opus
For each judge, set the default rule to serve the variation you created.

To learn more, read Custom judges.

Verify your setup

Before running the proxy, confirm in the dashboard:

Model selector: Each variation shows three attached judges.
Judges: Each judge prompt includes scoring criteria.
Targeting: All AI Configs have targeting enabled with correct rules.

Set up the project

Create a directory and install dependencies:

mkdir custom-evals && cd custom-evals
python -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn launchdarkly-server-sdk launchdarkly-server-sdk-ai \
    launchdarkly-server-sdk-ai-langchain langchain-anthropic python-dotenv

Create .env:

LD_SDK_KEY=sdk-your-sdk-key-here
LD_AI_CONFIG_KEY=model-selector
MODEL_KEY=sonnet
ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here
PORT=9911

Build the proxy server

Create server.py with the following code.

Click to expand the complete proxy server code

"""
Proxy server for Claude Code with automatic quality scoring.

Routes requests through LaunchDarkly AI Configs and scores every response
with attached judges. Metrics flow to the LaunchDarkly Monitoring dashboard.
"""

import asyncio
import os
import logging
import uuid

import ldclient
from ldclient import Context
from ldai import AICompletionConfigDefault, LDAIClient, LDMessage
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import uvicorn

from dotenv import load_dotenv
load_dotenv()

LD_SDK_KEY = os.environ.get("LD_SDK_KEY")
LD_AI_CONFIG_KEY = os.environ.get("LD_AI_CONFIG_KEY", "model-selector")
PORT = int(os.environ.get("PORT", "9911"))

if not LD_SDK_KEY:
    raise ValueError("Missing LD_SDK_KEY environment variable")

LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO").upper()
logging.basicConfig(level=getattr(logging, LOG_LEVEL, logging.INFO))

ld_config = ldclient.Config(LD_SDK_KEY)
ldclient.set_config(ld_config)
ld_client = ldclient.get()

if not ld_client.is_initialized():
    raise RuntimeError("LaunchDarkly client failed to initialize")

ai_client = LDAIClient(ld_client)
app = FastAPI()

# =============================================================================
# Message Conversion
# =============================================================================

def extract_text(content) -&gt; str:
    """Extract plain text from Anthropic-style content."""
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        texts = []
        for block in content:
            if isinstance(block, dict) and block.get("type") == "text":
                texts.append(block.get("text", ""))
        return "".join(texts)
    return str(content or "")


def convert_to_ld_messages(body: dict) -&gt; list[LDMessage]:
    """Convert Anthropic Messages API format to LDMessage format."""
    messages = []

    system = body.get("system")
    if system:
        system_text = extract_text(system) if isinstance(system, list) else system
        messages.append(LDMessage(role="system", content=system_text))

    for msg in body.get("messages", []):
        role_str = msg.get("role", "user")
        role = "assistant" if role_str == "assistant" else "user"
        messages.append(LDMessage(role=role, content=extract_text(msg.get("content", ""))))

    return messages

# =============================================================================
# Routes
# =============================================================================

@app.post("/v1/messages")
async def handle_messages(request: Request):
    """Main endpoint using chat.invoke() for automatic judge execution."""
    body = await request.json()
    user_key = request.headers.get("x-ld-user-key", "claude-code-local")

    # Build context with selectedModel for targeting
    model_key = os.environ.get("MODEL_KEY", "")
    context = (
        Context.builder(user_key)
        .set("selectedModel", model_key)
        .build()
    )

    fallback = AICompletionConfigDefault(enabled=False)
    chat = await ai_client.create_chat(LD_AI_CONFIG_KEY, context, fallback, {})

    if not chat:
        return JSONResponse(
            {"type": "error", "error": {"type": "unavailable", "message": "AI Config disabled"}},
            status_code=503
        )

    config = chat.get_config()
    model_name = config.model.name if config.model else "unknown"
    judge_count = len(config.judge_configuration.judges) if config.judge_configuration else 0

    print(f"[REQUEST] model={model_name}, judges={judge_count}")

    try:
        ld_messages = convert_to_ld_messages(body)

        if len(ld_messages) &gt; 1:
            chat.append_messages(ld_messages[:-1])

        last_message = ld_messages[-1] if ld_messages else LDMessage(role="user", content="")

        # invoke() executes judges automatically based on sampling rate
        response = await chat.invoke(last_message.content)

        # Await judge evaluations and log results
        if response.evaluations:
            print(f"[JUDGES] Awaiting {len(response.evaluations)} evaluations...")
            eval_results = await asyncio.gather(*response.evaluations, return_exceptions=True)
            for result in eval_results:
                if isinstance(result, Exception):
                    print(f"[JUDGE ERROR] {result}")
                elif result:
                    print(f"[JUDGE] {result.to_dict()}")

        # Flush events to LaunchDarkly
        ld_client.flush()
        await asyncio.sleep(0.1)

        response_text = response.message.content if response.message else ""

        # Get token metrics
        input_tokens = 0
        output_tokens = 0
        if response.metrics and response.metrics.usage:
            input_tokens = response.metrics.usage.input or 0
            output_tokens = response.metrics.usage.output or 0

        print(f"[METRICS] tokens={input_tokens}/{output_tokens}")

        return JSONResponse({
            "id": f"msg_{uuid.uuid4().hex[:24]}",
            "type": "message",
            "role": "assistant",
            "content": [{"type": "text", "text": response_text}],
            "model": model_name,
            "stop_reason": "end_turn",
            "usage": {
                "input_tokens": input_tokens,
                "output_tokens": output_tokens
            }
        })

    except Exception as e:
        ld_client.flush()
        logging.exception("Request failed")
        return JSONResponse(
            {"type": "error", "error": {"type": "internal_error", "message": str(e)}},
            status_code=500
        )


@app.get("/health")
async def health():
    return {"status": "ok", "launchdarkly": ld_client.is_initialized()}


@app.post("/v1/messages/count_tokens")
async def count_tokens(request: Request):
    return {"input_tokens": 0}

# =============================================================================
# Main
# =============================================================================

if __name__ == "__main__":
    print(f"Proxy running on port {PORT}")
    print(f"AI Config: {LD_AI_CONFIG_KEY}")
    print(f"Connect: ANTHROPIC_BASE_URL=http://localhost:{PORT} claude")
    uvicorn.run(app, host="127.0.0.1", port=PORT, log_level="info")

Connect Claude Code to your proxy

Start the proxy server:

python server.py

You should see output like:

Proxy running on port 9911
AI Config: model-selector
Connect: ANTHROPIC_BASE_URL=http://localhost:9911 claude

In a new terminal, launch Claude Code with the proxy URL and your chosen model:

MODEL_KEY=sonnet ANTHROPIC_BASE_URL=http://localhost:9911 claude

Every request now routes through your proxy. Watch the server logs to see judges executing:

[REQUEST] model=claude-sonnet-4-6, judges=3
[JUDGES] Awaiting 3 evaluations...
[JUDGE] {'evals': {'security': {'score': 1.0, 'reasoning': 'No vulnerabilities detected...'}}}
[JUDGE] {'evals': {'api-contract': {'score': 0.5, 'reasoning': 'Response uses correct endpoint...'}}}
[JUDGE] {'evals': {'minimal-change': {'score': 1.0, 'reasoning': 'Changes are focused...'}}}

The create_chat() and invoke() methods handle judge execution automatically:

chat = await ai_client.create_chat(config_key, context, fallback, {})
response = await chat.invoke(user_message)
# response.evaluations contains async judge tasks

Judge results are sent to LaunchDarkly automatically. You can optionally await response.evaluations to log results locally.

This proxy handles text-based conversations. Tool-based features like file editing and command execution won't work through this proxy.

How model routing works

The MODEL_KEY environment variable controls which model handles requests. The proxy passes it as a selectedModel context attribute:

context = Context.builder(user_key).set("selectedModel", model_key).build()

Your targeting rules match this attribute and return the corresponding variation. Switch models by changing the environment variable:

MODEL_KEY=mistral ANTHROPIC_BASE_URL=http://localhost:9911 claude

Compare cloud and local models

To evaluate Ollama models against cloud providers:

Add an "ollama" variation to your model-selector AI Config.
Add a targeting rule for selectedModel equals "ollama".
Launch with MODEL_KEY=ollama.

Your custom judges score Claude Sonnet and Llama 3.2 with identical criteria. After enough requests, you can compare quality scores across providers.

Run experiments

After judges are producing scores, you can compare models statistically. Create two variations with different models, attach the same judges, and set up a percentage rollout to split traffic.

Your judge metrics appear as goals in LaunchDarkly Experimentation. After enough data, you can answer "Which model produces more secure code?" with confidence, not guesswork.

To learn more, read Experimentation with AI Configs.

Monitor quality over time

Judge scores appear on your AI Config's Monitoring tab. To view evaluation metrics:

Open your model-selector AI Config and go to the Monitoring tab.
Select Evaluator metrics from the dropdown menu.

![Select Evaluator metrics from the dropdown]

Each judge (security, API contract, minimal change) shows as a separate chart. Hover over a chart to see scores broken down by variation.

To drill into a specific model's evaluations, select the variation from the bottom menu.

Watch for baseline patterns in the first week, then track regressions after model updates or prompt changes. Model providers ship updates without notice. A Claude update might improve reasoning but introduce patterns that fail your API contract checks. Set up alerts when scores drop below thresholds, and use guarded rollouts for automatic protection.

To learn more, read Monitor AI Configs.

Control costs with sampling

Each judge evaluation is an LLM call. Control costs by adjusting sampling rates:

Staging: 100% sampling to catch issues early
Production: 10-25% sampling for cost efficiency

You can also use cheaper models (GPT-4o mini) for staging and more capable models for production.

What you learned

The value is in the judges you create. The three in this tutorial cover security, API compliance, and scope discipline. Your team might care about different signals: documentation quality, test coverage, or adherence to internal coding standards.

Custom judges let you define quality for your codebase, apply the same evaluation criteria across models, and track trends over time. Once you create a judge, you can attach it to any AI Config in your project.

Ready to build custom judges for your codebase? Start your 14-day free trial and deploy your first evaluation today.

Next steps

hello-python-ai examples for more judge patterns
AI Configs best practices for production patterns

The /aiconfig-online-evals and /aiconfig-targeting skills are not yet available. Use the dashboard to complete those steps. ↩

Beyond n8n for Workflow Automation: Agent Graphs as Your Universal Agent Harness

Scarlett Attensil — Thu, 26 Mar 2026 00:33:34 +0000

Hardcoded multi-agent orchestration is brittle: topology lives in framework-specific code, changes require redeploys, and bottlenecks are hard to see. Agent Graphs externalize that topology into LaunchDarkly, while your application continues to own execution.

In this tutorial, you'll build a small multi-agent workflow, traverse it with the SDK, monitor per-node latency on the graph itself, and update a slow node's model without changing application code.

Node = AI Config (model, instructions, tools)
Edge = handoff metadata (routing contract you define)
Graph = topology (which nodes connect)
Your app = execution + interpretation

LaunchDarkly provides graph structure, config, and observability. Your application owns execution semantics: you write the code that interprets edges and runs agents.

What You'll Build

In this tutorial, you'll add Agent Graphs to an existing multi-agent workflow:

Build a graph visually in the LaunchDarkly UI
Connect it to your code with a few lines of SDK integration
Run your agents and see the graph in action
Monitor performance with per-node latency and invocation tracking
Fix a slow agent by swapping models from the dashboard

By the end, you'll have a multi-agent system where topology metadata changes happen in the UI, picked up by your traversal code on the next request.

Prerequisites

LaunchDarkly account with AI Configs access (sign up here)
Python 3.9+
An existing agent workflow (or use our sample repo)

The Problem with Hardcoded Orchestration

Every multi-agent framework handles orchestration differently:

# LangGraph - topology hardcoded in graph setup
workflow = StateGraph(AgentState)
workflow.add_node("supervisor", supervisor_node)
workflow.add_node("security", security_node)
workflow.add_node("support", support_node)
workflow.set_entry_point("supervisor")
# Routing logic buried in node functions or conditional edges

# OpenAI Agents SDK - handoffs defined per agent
security_agent = Agent(name="Security", instructions="...")
support_agent = Agent(name="Support", instructions="...")
supervisor = Agent(
    name="Supervisor",
    handoffs=[security_agent, support_agent]  # Topology locked in code
)

The topology is scattered across code. Agent Graphs make it visible: you see the entire workflow in one view, edit connections in the UI, and traverse it with graph-aware SDK methods.

Why Externalizing Topology Helps

If you've built multi-agent systems with LangGraph, OpenAI Swarm, or Strands, you've hit these walls:

Config duplication: Agent definitions scattered across framework-specific formats
Silent failures: An agent times out and you don't know until users complain
No topology visibility: The workflow exists only in code
Custom observability: Getting consistent per-agent metrics means reconciling different trace formats and data schemas across frameworks

For a detailed comparison of LangGraph, OpenAI Swarm, and Strands, see Compare AI orchestrators. Agent Graphs work with multiple agent frameworks.

Agent Graphs solve these by giving you a visual graph builder where you:

See your entire workflow at a glance, not buried in code
Monitor per-node metrics overlaid directly on the graph (latency, invocations, tool calls)
Add or remove agents without changing traversal logic, provided your runtime supports the node's tools and output contract
Inspect routing logic on edges, with handoff data visible in the UI
Use graph-aware SDK methods like is_terminal(), is_root(), and get_edges() instead of manual tracking

Step 1: Create AI Configs for Your Agents

Before building a graph, you need AI Configs for each agent. If you already have AI Configs, skip to Step 2.

See the AI Configs quickstart or run the bootstrap script in our sample repo:

git clone https://github.com/launchdarkly-labs/devrel-agents-tutorial
cd devrel-agents-tutorial
git checkout tutorial/agent-graphs
uv sync
cp .env.example .env  # Add your LD_SDK_KEY, LD_API_KEY, OPENAI_API_KEY
uv run python bootstrap/create_configs.py

For this tutorial, we'll use three configs:

supervisor-agent: Orchestrates the workflow and routes queries based on PII pre-screening
security-agent: Detects and redacts personally identifiable information (PII)
support-agent: Answers questions using dynamically loaded tools (search, RAG)

Step 2: Build the Graph in the UI

This is where Agent Graphs diverge from code-based orchestration. Instead of writing add_edge() calls, you'll see your topology and modify it visually.

Open your LaunchDarkly dashboard and navigate to AI > Agent graphs.

You'll see the first-time setup wizard. Since you already created AI Configs in Step 1, expand Create a graph at the bottom.

Name your graph chatbot-flow and click Create graph.

Add your first node: click Add node and select supervisor-agent
Set it as the root: click the node and toggle Root node
Add security-agent and support-agent as nodes

Draw edges: drag from supervisor-agent to both child agents
Add handoff data to each edge to define routing logic:

supervisor-agent → security-agent:

{
  "action": "sanitize",
  "reason": "PII detected",
  "route": "security"
}

supervisor-agent → support-agent:

{
  "action": "direct",
  "reason": "Clean input",
  "route": "support"
}

security-agent → support-agent:

{
  "action": "proceed",
  "reason": "Input sanitized",
  "route": "continue"
}

Notice what you're seeing: the entire workflow topology in one view. This graph is your architecture diagram, always current. Each node shows which AI Config variation it serves. The edges show routing logic that would otherwise be buried in conditional statements. When you need to add a new agent or change routing, you do it here, not in code.

LaunchDarkly doesn't execute your graph. It provides:

Topology: Which nodes exist and how they connect
Handoff metadata: Whatever JSON you put on edges
Per-node AI Config: Model, instructions, tools for each agent

Your code:

Decides which edges to follow based on agent decisions
Interprets handoff data however you want (the schema is yours)
Executes the actual agents

The handoff JSON is arbitrary metadata. You define the schema, you interpret it. LaunchDarkly stores and delivers it.

Step 3: Add the SDK to Your Project

Install the LaunchDarkly AI SDK:

uv add launchdarkly-server-sdk launchdarkly-server-sdk-ai

Initialize the clients in your code:

# config_manager.py - Initialize LaunchDarkly clients
def _initialize_launchdarkly_client(self):
    """Initialize LaunchDarkly client and AI client"""
    config = ldclient.Config(self.sdk_key)
    ldclient.set_config(config)
    self.ld_client = ldclient.get()

    # Block until client is initialized (max 10 seconds)
    self.ld_client.start_wait(10)

    if not self.ld_client.is_initialized():
        raise RuntimeError("LaunchDarkly client initialization failed")

    self.ai_client = LDAIClient(self.ld_client)

Build a context for targeting and tracking:

# config_manager.py - Build context for targeting
def build_context(self, user_id: str, user_context: dict = None) -> Context:
    """Build a LaunchDarkly context with consistent attributes."""
    context_builder = Context.builder(user_id).kind('user')

    if user_context:
        for key, value in user_context.items():
            context_builder.set(key, value)

    return context_builder.build()

Step 4: Integrate with Your Framework

This section walks through the integration code, starting with the building block (what runs at each node), then showing how nodes are orchestrated.

The Generic Agent Pattern

The key to dynamic execution is create_generic_agent. Every node uses the same implementation—no agent registry, no hardcoded agent types:

# agents/generic_agent.py
def create_generic_agent(agent_config, config_manager, valid_routes: List[str] = None):
    """Create a generic agent from LaunchDarkly AI Config."""

    class GenericAgent:
        def __init__(self):
            self.valid_routes = valid_routes or []

        async def ainvoke(self, state: dict) -> dict:
            """Execute the agent using LaunchDarkly config."""
            if not agent_config.enabled:
                return {"response": "", "_skipped": True}

            # Create model from config
            model = create_model_for_config(
                provider=agent_config.provider.name,
                model=agent_config.model.name,
                config_manager=config_manager
            )

            # Load tools from LaunchDarkly config
            tools = create_dynamic_tools_from_launchdarkly(agent_config)

            # Get instructions from config
            instructions = agent_config.instructions or "Process the input."

            # Inject route options into instructions
            if self.valid_routes:
                route_instruction = f"\n\nSelect one of these routes: {self.valid_routes}. Return: {{\"route\": \"<selected_route>\"}}"
                instructions = instructions + route_instruction

            # Execute and extract routing decision
            result = await self._execute(model, instructions, tools, state)
            result["routing_decision"] = self._extract_route(result.get("response", ""))

            # Track metrics
            agent_config.tracker.track_success()
            return result

    return GenericAgent()

The generic agent pattern means:

No agent registry: Every node uses the same create_generic_agent function
Config-driven behavior: Model, instructions, and tools all come from LaunchDarkly
Dynamic routing: Valid routes are injected from graph edges, not hardcoded
Minimal code changes: Add a new agent in LaunchDarkly, create its AI Config, add it to your graph, and it works—provided your runtime supports the node's tools and output contract

The AgentService Class

The AgentService class is the entry point for processing messages through your Agent Graph:

# api/services/agent_service.py
class AgentService:
    """Multi-Agent Orchestration using LaunchDarkly Agent Graph."""

    def __init__(self):
        self.config_manager = ConfigManager()
        self.config_manager.flush()

    async def process_message(
        self,
        user_id: str,
        message: str,
        user_context: dict = None
    ) -> ChatResponse:
        """Process message using LaunchDarkly Agent Graph."""
        result = await self._execute_graph(
            graph_key=os.getenv("AGENT_GRAPH_KEY", "chatbot-flow"),
            user_id=user_id.strip() or "anonymous",
            user_input=message,
            user_context=user_context or {}
        )

        return ChatResponse(
            response=result.get("final_response", ""),
            tool_calls=result.get("tool_calls", []),
            # ... other fields
        )

Executing the Graph

The _execute_graph method fetches the graph from LaunchDarkly and uses traverse() with skip logic for conditional routing:

# api/services/agent_service.py
async def _execute_graph(
    self,
    graph_key: str,
    user_id: str,
    user_input: str,
    user_context: dict = None
) -> Dict[str, Any]:
    """Execute agents using SDK's traverse() with skip logic."""
    ld_context = self.config_manager.build_context(user_id, user_context)
    graph = self.config_manager.ai_client.agent_graph(graph_key, ld_context)

    if not graph.is_enabled():
        raise ValueError(f"Agent Graph '{graph_key}' is not enabled")

    ctx = {
        "user_input": user_input,
        "messages": [HumanMessage(content=user_input)],
        "processed_input": user_input,
        "final_response": "",
        "tool_calls": [],
        # Skip logic: track which nodes should execute
        "_routed_to": {graph.root().get_key()},
        "_path": [],
        "_prev_key": None,
    }

    tracker = graph.get_tracker()

    # Define the node callback (see next section)
    def execute_node(node, exec_ctx):
        # ... node execution logic
        pass

    # Use SDK's traverse() - it handles traversal order
    graph.traverse(execute_node, ctx)

    # Track graph completion
    if tracker:
        tracker.track_path(ctx.get("_path", []))
        tracker.track_invocation_success()

    return ctx

Skip Logic for Conditional Routing

The execute_node callback implements skip logic—the core pattern that enables conditional routing:

# api/services/agent_service.py - inside _execute_graph
def execute_node(node, exec_ctx):
    """Execute a single node if it was routed to."""
    key = node.get_key()

    # Skip logic: only execute if parent routed to this node
    if key not in exec_ctx.get("_routed_to", set()):
        return {"_skipped": True}

    exec_ctx["_path"].append(key)

    # Track node invocation
    if tracker:
        tracker.track_node_invocation(key)
        if exec_ctx.get("_prev_key"):
            tracker.track_handoff_success(exec_ctx["_prev_key"], key)

    # Get edges and valid routes for this node
    edges = node.get_edges()
    valid_routes = [e.handoff.get("route") for e in edges if e.handoff and e.handoff.get("route")]

    # Execute agent with config from this node
    agent = create_generic_agent(node.get_config(), self.config_manager, valid_routes=valid_routes)
    result = _run_async(agent.ainvoke(exec_ctx))

    # Track tool calls
    if tracker and result.get("tool_calls"):
        for tool in result["tool_calls"]:
            tracker.track_tool_call(key, tool)

    # Route to next node: add to _routed_to set
    if edges:
        next_key = self._select_next_node(edges, result, tracker)
        if next_key:
            exec_ctx["_routed_to"].add(next_key)

    exec_ctx["_prev_key"] = key
    return result

The _routed_to set tracks which nodes should execute:

Start: Add root node to _routed_to
traverse() visits each node: If node is in _routed_to, execute it; otherwise skip
After execution: Add the next node (based on routing decision) to _routed_to

This enables conditional routing: the supervisor routes to either security OR support, and only the chosen path executes.

Routing Between Nodes

The _select_next_node method determines which node to route to based on the agent's routing decision:

# api/services/agent_service.py
def _select_next_node(self, edges, result: dict, tracker=None):
    """Select next node key based on routing decision."""
    routing = result.get("routing_decision", "").lower().strip() if result.get("routing_decision") else None

    # Build route map: route -> target_config
    route_map = {}
    for edge in edges:
        route = (edge.handoff.get("route", "") if edge.handoff else "").lower().strip()
        if route:
            route_map[route] = edge.target_config

    # Exact match
    if routing and routing in route_map:
        return route_map[routing]
    elif routing:
        if tracker:
            tracker.track_handoff_failure()

    # Default: first edge
    if edges:
        return edges[0].target_config

    return None

The key insight: your graph topology comes from LaunchDarkly, not hardcoded orchestration. Change the graph in the UI, and your code picks up the new structure on the next request.

Step 5: Run It

With the AgentService wired up (as shown in Step 4), you can now process messages through your Agent Graph. The service handles:

Building the LaunchDarkly context for targeting
Fetching the graph and executing nodes via traverse()
Tracking metrics for monitoring
Returning the final response

Test it by sending a message:

service = AgentService()
response = await service.process_message(
    user_id="user-123",
    message="What's the status of my order?",
    user_context={"plan": "premium"}
)
print(response.response)

Now go back to the LaunchDarkly UI. Add a new node or change an edge. Run your code again. Topology changes are picked up by your traversal code on subsequent SDK evaluations.

Step 6: Monitor Agent Performance

This is the key differentiator: monitoring happens on the graph itself, not in a separate dashboard. You see metrics overlaid on the same visual topology you built, so bottlenecks are immediately obvious.

The sample repo includes full instrumentation: calls to tracker.track_success(), tracker.track_error(), and tracker.track_tool_call() in the agent execution path. After running some traffic, open your Agent Graph to see the results.

Navigate to AI > Agent graphs > chatbot-flow. You'll see a metrics bar at the top of the graph view where you can toggle different metrics on and off.

Metrics on the graph

Here's what makes this different from traditional APM: the metrics appear directly on your workflow visualization. No mental mapping between a dashboard and your code. No correlating trace IDs. The slow node lights up on the graph.

Turn on Latency to see duration data overlaid directly on your graph:

Total duration: The combined time for the entire graph invocation
Per-node duration: How long each individual agent takes

Turn on Invocations to see how often each node is reached. This reveals which paths your users take most frequently. In a routing graph, you'll quickly see whether most queries go through security or skip directly to support.

Turn on Tool calls to see the average number of tool invocations per node. If an agent is calling tools excessively, you'll spot it here.

Monitoring page

Click Monitoring to see all metrics over time. This view shows:

Latency trends: Duration per node over hours, days, or weeks
Invocation patterns: Traffic flow through your graph
Tool call breakdown: Which specific tools are being called and how often

To see which specific tools are called, you need to track them in your code using the tracker. The SDK sends this data to LaunchDarkly, which displays it in the monitoring view.

Generate traffic to see metrics

Run the traffic generator from the sample repo to send queries through your graph:

uv run python tools/traffic_generator.py --queries 20 --delay 2

This sends a mix of queries (some with PII, some without) to exercise both the security and support paths. After a few minutes, you'll see metrics populate on the graph.

Detecting a slow agent

With traffic flowing, suppose the security-agent starts averaging 5 seconds per call. With latency metrics enabled on the graph, you see it immediately: the security-agent node shows a high duration value while other nodes stay fast.

The invocation numbers also tell a story. If security-agent shows 50 invocations and support-agent shows 80, you know ~30 queries are bypassing security (the clean path). This helps you understand whether the slow agent is affecting most users or just a subset.

Without Agent Graphs, you'd need custom logging, Datadog queries, and manual correlation. With Agent Graphs, you see the problem in 30 seconds.

Step 7: Fix Without Deploying

The security-agent is slow because it's using claude-sonnet-4 for PII detection. A smaller, faster model may be sufficient for this task.

In the LaunchDarkly dashboard, update the pii-detector variation:

Change model from Anthropic.claude-sonnet-4-20250514 to Anthropic.claude-3-haiku-20240307

Or use Agent Skills to make the change from your coding assistant:

The security-agent pii-detector variation is averaging 5 seconds.
Change the model to claude-3-haiku-20240307.

No code changes. No deploy. Changes are picked up on subsequent SDK evaluations.

Run the traffic generator again and watch the latency drop.

What just happened

Traffic generator sent queries through the graph
Monitoring showed the slow agent on the graph
Model swap happened in the UI (or via Agent Skills)
Your code automatically used the new configuration

No deploys. No PRs. The fix is live.

OpenAI Agents SDK Integration (Conceptual)

Agent Graphs work with multiple frameworks. This conceptual example shows how the pattern translates to OpenAI Agents SDK:

# Conceptual example showing how Agent Graph SDK methods work with OpenAI Agents
from agents import Agent, Runner

def handle_traversal(node, state):
    config = node.get_config()
    tracker = config.tracker
    edges = node.get_edges()

    # Child agents are already in state (reverse traversal builds bottom-up)
    handoffs = [state[edge.target_config] for edge in edges]

    def on_handoff(ctx):
        # Track handoff events
        return ctx

    return Agent(
        name=config.key,
        instructions=config.instructions,
        handoffs=handoffs,
        on_handoff=on_handoff,
    )

if agent_graph.is_enabled():
    root = agent_graph.reverse_traverse(handle_traversal, {})

result = await Runner.run(root, "Tell me about your engineering team")

Same graph definition, adapted to each framework's execution model. The topology metadata lives in LaunchDarkly; your code interprets and executes it.

Best Practices

Start simple: Begin with a linear graph (A → B → C) before adding conditional routing.

Use handoff data for context passing: Include metadata like action type, reason, or state that the next agent needs to continue the workflow.

Track everything: Call tracker.track_success() and tracker.track_error() in every node for complete visibility. Use graph_tracker.track_tool_call(tool_name) to track which tools agents invoke.

Test with targeting: Use LaunchDarkly targeting to route test users to experimental graph configurations.

Handle missing edges: Decide what happens when no edge matches a routing decision or when a target node is disabled. Recommend: fail closed, log diagnostics, and track routing failures.

Keep execution state request-scoped: Store execution state inside the context object (ctx) passed through traversal, not in instance-level variables. Treat graph traversal as request-scoped to avoid concurrency issues.

What You've Built

You now have a multi-agent system where:

Graph topology is externalized and self-documenting
Routing logic is visible on edges, not buried in code
Monitoring appears on the graph itself, not a separate dashboard
Node-level control lets you disable a single agent without touching others, provided your executor checks node availability
Multiple frameworks can consume the same graph metadata

When you spot a slow agent in monitoring, you can swap the model from the dashboard without a deploy.

Next Steps

Agent Graphs Reference: SDK methods for traverse, reverse_traverse, get_edges(), and handoff data
AI Configs Documentation: Learn more about variations, targeting, and experiments
Agent Skills Tutorial: Manage AI Configs from your coding assistant
Monitor AI Configs: Deep dive into metrics and dashboards
Sample Repository: Complete code from this tutorial

Conclusion

Hardcoded orchestration was fine when you had one agent. With multi-agent systems, it becomes a liability. Every change requires a deploy. Every incident requires a developer.

Agent Graphs flip this. Define your workflow in LaunchDarkly, integrate it with your framework, and fix many problems without touching code. Your agents become as dynamic as your feature flags.

Ready to stop hardcoding? Get started with AI Configs and create your first Agent Graph.

LLM evaluation guide: When to add online evals to your AI application

Scarlett Attensil — Wed, 17 Dec 2025 17:42:49 +0000

The quick decision framework

Online evals for AI Configs is currently in closed beta. Judges must be installed in your project before they can be attached to AI Config variations.

Online evals provide real-time quality monitoring for LLM applications. Using LLM-as-a-judge methodology, they run automated quality checks on a configurable percentage of your production traffic, producing structured scores and pass/fail judgments you can act on programmatically. LaunchDarkly includes three built-in judges: accuracy, relevance, and toxicity.

Skip online evals if:

Your checks are purely deterministic (schema validation, compile tests)
You have low volume and can manually review outputs in observability dashboards
You're primarily debugging execution problems

Add online evals when:

You need quantified quality scores to trigger automated actions (rollback, rerouting, alerts)
Manual quality review doesn't scale to your traffic volume
You're measuring multiple quality dimensions (accuracy, relevance, toxicity)
You want statistical quality trends across segments for AI governance and compliance
You need to monitor token usage and cost alongside quality metrics
You're running A/B tests or guarded releases and need automated quality gates

Most teams add them within 2-3 sprints when manual quality review becomes the bottleneck. Configurable sampling rates let you balance evaluation coverage with cost and latency.

Online evals vs. LLM observability

LLM observability shows you what happened. Online evals automatically assess quality and trigger actions based on those assessments.

LLM observability: your security camera

LLM observability shows you everything that happened through distributed tracing: full conversations, tool calls, token usage, latency breakdowns, and cost attribution. Perfect for debugging and understanding what went wrong. But when you're handling 10,000 conversations daily, manually reviewing them for quality patterns doesn't scale.

Online evals: your security guard

Automatically scores every sampled request using LLM-as-a-judge methodology across your quality rubric (accuracy, relevance, toxicity) and takes action. Instead of exporting conversations to spreadsheets for manual review, you get real-time quality monitoring with drift detection that triggers alerts, rollbacks, or rerouting.

The 3 AM difference

Without evals: "Let's meet tomorrow to review samples and decide if we should rollback."

With evals: "Quality dropped below threshold, automatic rollback triggered, here's what failed..."

How online evals actually work

LaunchDarkly's online evals use LLM-as-a-judge methodology with three built-in judges you can configure directly in the dashboard. No code changes required.

Getting started:

Install judges from the AI Configs menu
Attach judges to AI Config variations
Configure sampling rates (balance coverage with cost/latency)
Evaluation metrics are automatically emitted as custom events
Metrics are automatically available for A/B tests and guarded releases

What you get from each built-in judge:

Accuracy judge:

{
  "score": 0.85,
  "reasoning": "Response correctly answered the question but missed one edge case regarding error handling"
}

Relevance judge:

{
  "score": 0.92,
  "reasoning": "Response directly addressed the user's query with appropriate context and examples"
}

Toxicity judge:

{
  "score": 0.0,
  "reasoning": "Content is professional and appropriate with no toxic language detected"
}

Each judge returns a score from 0.0 to 1.0 plus reasoning that explains the assessment. LaunchDarkly's built-in judges (accuracy, relevance, toxicity) have fixed evaluation criteria and are configured only by selecting the provider and model.

Configuration:
Install judges from the AI Configs menu in your LaunchDarkly dashboard. They appear as pre-configured AI configs (AI Judge - Accuracy, AI Judge - Toxicity, AI Judge - Relevance). When configuring your AI Config variations in completion mode, select which judges to attach with your desired sampling rate. Use different judge combinations for different environments to match your quality requirements and cost constraints.

Real problems online evals solve

Scale for production applications: Your SQL generator handles 50,000 queries daily. LLM observability shows you every query through distributed tracing. Online evals tell you the proportion that are semantically wrong, automatically, with hallucination detection built in.

Multi-dimensional quality monitoring: Customer service AI applications aren't just "did it respond?" It's accuracy, relevance, toxicity, compliance, and appropriateness. Online evals score all dimensions simultaneously, each with its own threshold and reasoning.

RAG pipeline validation: Your retrieval-augmented generation system needs continuous monitoring of both retrieval quality and generation accuracy. Online evals can assess whether retrieved context is relevant and whether the response accurately uses that context, preventing hallucinations and ensuring factual grounding.

Cost and performance optimization: Monitor token usage alongside quality metrics. If certain queries consume 10x more tokens than others, online evals help identify these patterns so you can optimize prompts or routing logic to reduce costs without sacrificing quality.

Actionable metrics for AI governance: Transform 10,000 responses from data to decisions with evaluator-driven quality gates:

Accuracy trending below 0.8? Automated alerts to the team
Toxicity above 0.2? Immediate review and potential rollback
Relevance dropping for specific user segments? Targeted configuration updates
Metrics automatically feed A/B tests and guarded releases for continuous improvement

Example implementation path

Week 1-2: Define quality dimensions and install judges.
Use LLM observability alone first. Manually review samples to understand your system. Define your quality dimensions: accuracy, relevance, toxicity, or other criteria specific to your application. Install the built-in judges from the AI Configs menu in LaunchDarkly.

Week 3-4: Attach judges with sampling.
Attach judges to AI Config variations in LaunchDarkly. Start with one or two key judges (accuracy and relevance are good defaults). Configure sampling rates between 10-20% of traffic to balance coverage with cost and latency. Compare automated scores with human judgment to validate the judges work for your use case.

Week 5+: Operationalize with quality gates.
Add more evaluation dimensions as you learn. Connect scores to automated actions and evaluator-driven quality gates: when accuracy drops below 0.7, trigger alerts; when toxicity exceeds 0.2, investigate immediately. Leverage the custom events and metrics for A/B testing and guarded releases to continuously improve your application's performance.

The bottom line

You don't need online evals on day one. Start with LLM observability to understand your AI system through distributed tracing. Add evaluations when you hear yourself saying "we need to review more conversations" or "how do we know if quality is degrading?"

LaunchDarkly's three built-in judges (accuracy, relevance, toxicity) provide LLM-as-a-judge evaluation that you can attach to any AI Config variation in completion mode with configurable sampling rates. Note that online evals currently only work with completion mode AI Configs. Agent-based configs are not yet supported. Evaluation metrics are automatically emitted as custom events and feed directly into A/B tests and guarded releases, enabling continuous AI governance and quality improvement without code changes. Start simple with one judge, learn what matters for your application, and expand from there.

LLM observability is your security camera. Online evals are your security guard.

Next steps

Ready to get started? Sign up for a free LaunchDarkly account if you haven't already.

Build a complete quality pipeline:

AI Config CI/CD Pipeline - Add automated quality gates and LLM-as-a-judge testing to your deployment process
Combine offline evaluation (in CI/CD) with online evals (in production) for comprehensive quality coverage

Learn more about AI Configs:

AI Config documentation - Understand how AI Configs enable real-time LLM configuration
Online evals documentation - Deep dive into judge installation and configuration
Guardrail metrics - Monitor quality during A/B tests and guarded releases

See it in action:

Check LLM observability in the LaunchDarkly dashboard to track your AI application performance with distributed tracing

Industry standards:
LaunchDarkly's approach aligns with emerging AI observability standards, including OpenTelemetry's semantic conventions for AI monitoring, ensuring your evaluation infrastructure integrates with the broader observability ecosystem.

When to Use Prompt-Based vs Agent Mode in LaunchDarkly for AI Applications

Scarlett Attensil — Wed, 17 Dec 2025 17:39:09 +0000

A Guide for LangGraph, OpenAI, and Multi-Agent Systems

The broader tech industry can't agree on what the term "agents" even means. Anthropic defines agents as systems where "LLMs dynamically direct their own processes," while Vercel's AI SDK enables multi-step agent loops with tools, and OpenAI provides an Agents SDK with built-in orchestration. So when you're creating an AI Config in LaunchDarkly and see "prompt-based mode" vs. "agent mode," you might reasonably expect this choice to determine whether you get automatic tool execution loops, server-side state management, or some other fundamental capability difference.

But LaunchDarkly's distinction is different and more practical. Understanding it will save you from confusion and help you ship AI features faster.

TL;DR

LaunchDarkly's "prompt-based vs. agent" choice is about input schemas and framework compatibility, not execution automation. Prompt-based mode returns a messages array (perfect for chat UIs), while agent mode returns an instructions string (optimized for LangGraph/CrewAI frameworks). Both provide the same core benefits: provider abstraction, A/B testing, metrics tracking, and the ability to change AI behavior without deploying code.

Ready to start? Sign up for a free trial → create your first AI Config → Choose your mode → Configure and ship.

The fragmented AI landscape

LaunchDarkly supports 20+ AI providers: OpenAI, Anthropic, Gemini, Azure, Bedrock, Cohere, Mistral, DeepSeek, Perplexity, and more. Each has their own interpretation of "completions" vs "agents," creating a chaotic ecosystem with different API endpoints, execution behaviors, state management approaches, and capability limitations. This fragmentation makes it difficult to switch providers or even understand what capabilities you're getting. That's where LaunchDarkly's abstraction layer comes in.

LaunchDarkly's approach: provider-agnostic input schemas

LaunchDarkly's AI Configs are a configuration layer that abstracts provider differences. When you choose prompt-based mode or agent mode, you're selecting an input schema (messages array vs. instructions string), not execution behavior. LaunchDarkly provides the configuration; you handle orchestration with your own code or frameworks like LangGraph. This gives you provider abstraction, A/B testing, metrics tracking, and online evals (prompt-based mode only) without locking you into any specific provider's execution model.

Prompt-based mode: messages-based

Prompt-based mode uses a messages array format with system/user/assistant roles (some providers like OpenAI also support a "developer" role for more granular control). This is the traditional chat format that works across all AI providers.

UI Input: "Messages" section with role-based messages

SDK Method: aiclient.config()

Returns: Customized prompt + model configuration

Documentation: AI Config docs

# Retrieve prompt-based AI config
config = aiclient.config(
    key="customer-support",
    context=context,
    default_value=default_config
)

# What you get back: messages array
print(config.messages)
# [
#   {
#     "role": "system",
#     "content": "You are a helpful customer support agent for Acme Corp."
#   },
#   {
#     "role": "user",
#     "content": "How can I reset my password?"
#   }
# ]

# Use with provider SDKs that expect message arrays
response = openai.chat.completions.create(
    model=config.model.name,
    messages=config.messages  # Standard message format
)

When to use prompt-based mode:

You're building chat-style interactions: Traditional message-based conversations where you construct system/user/assistant messages
You need online evals: LaunchDarkly's model-agnostic online evals are currently only available in prompt-based mode
You want granular control of workflows: Discrete steps that need to be accomplished in a specific order, or multi-step asynchronous processes where each step executes independently
One-off evaluations: Issue individual evaluations of your prompts and completions (not online evals)
Simple processing tasks: Summarization, name suggestions, or other non-context-exceeding data processing

Agent mode: goal/instructions-based

Agent mode uses a single instructions string format that describes the agent's goal or task. This format is optimized for agent orchestration frameworks that expect high-level objectives rather than conversational messages.

UI Input: "Goal or task" field with instructions

SDK Method: aiclient.agent()

Returns: Customized instructions + model configuration

Examples: hello-python-ai examples

# Retrieve agent-based AI config
agent_config = aiclient.agent(
    key="research-assistant",
    context=context,
    default_value=default_config
)

# What you get back: instructions string
print(agent_config.instructions)
# "You are a research assistant. Your goal is to gather comprehensive
# information on the requested topic using available search tools.
# Search multiple sources, synthesize findings, and provide a detailed
# summary with citations."

# Use with agent frameworks that expect instructions
from langgraph.prebuilt import create_react_agent
from langchain_openai import init_chat_model

llm = init_chat_model(
    model=agent_config.model.name,
    model_provider=agent_config.provider.name
)

agent = create_react_agent(
    llm,
    tools=[search_tool, citation_tool],
    prompt=agent_config.instructions  # Goal/task instructions
)

# Execute and track
response = agent.invoke({"messages": [{"role": "user", "content": "..."}]})

When to use agent mode:

You're using agent frameworks: LangGraph, LangChain, CrewAI, AutoGen, or LlamaIndex Workflows expect goal/instruction-based inputs
Goal-oriented tasks: "Research X and create Y" rather than conversational message exchange
Tool-driven workflows: While both modes support tools, agent mode's format is optimized for frameworks that orchestrate tool usage
Open-ended exploration: The output is open-ended and you don't know the actual answer you're trying to get to
Data as an application: You want to treat your data as an application to feed in arbitrary data and ask questions about it
Provider agent endpoints: LaunchDarkly may route to provider-specific agent APIs when available (note: not all models support agent mode; check your model's capabilities)

See example: Build a LangGraph Multi-Agent System with LaunchDarkly

Quick comparison

Feature	Prompt-Based Mode	Agent Mode
Input format	Messages (system/user/assistant)	Goal/task + instructions
Tools support	✅ Yes	✅ Yes
SDK method	`config()`	`agent()`
Automatic execution loop	❌ No (you orchestrate)	❌ No (you orchestrate)
Online evals	✅ Available	❌ Not yet available
Best for	Chat-style prompting, single completions	Agent frameworks, goal-oriented tasks
Provider endpoint	Standard endpoint	May use provider-specific agent endpoint if available
Model support	All models	Most models (check model card for "Agent mode" capability)

Model compatibility: Not all models support agent mode. When selecting a model in LaunchDarkly, check the model card for "Agent mode" capability. Models like GPT-4.1, GPT-5 mini, Claude Haiku 4.5, Claude Sonnet 4.5, Claude Sonnet 4, Grok Code Fast 1, and Raptor mini support agent mode, while models focused on reasoning (like GPT-5, Claude Opus 4.1) may only support prompt-based mode.

How providers handle "completion vs agent"

To understand why LaunchDarkly's abstraction is valuable, let's look at how major AI providers handle the distinction between basic completions and advanced agent capabilities. The table below shows how different providers implement "advanced" modes; generally these are ADDITIVE, including all basic capabilities plus extras. For example, OpenAI's Responses API includes all Chat Completions features plus additional capabilities.

Provider	"Basic" Mode	"Advanced" Mode	Key Difference	Link
OpenAI	Chat Completions API	Responses API	Responses adds built-in tools (web_search, file_search, computer_use, code_interpreter, remote MCP), server-side conversation state with stored IDs, and improved streaming. Chat Completions remains supported.	Docs
Anthropic	Tool Use (client tools)	Tool Use (client + server tools)	Server tools (web_search, web_fetch) execute on Anthropic's servers. You can use both client and server tools together	Docs
Google Gemini	Manual function calling	Automatic function calling (Python SDK)	Python SDK auto-converts functions to schemas, runs the execution loop, and supports compositional multi-step calls. Manual mode: full control, all platforms	Docs
Vercel AI SDK	`generateText()`	`generateText()` with multi-step loop	Multi-step agent loops with tools; SDK continues until complete; `maxSteps` provides loop control to limit steps	Docs
Azure OpenAI	Assistants API (deprecated)	AI Agent Services	Enterprise agent runtime with threads, tool orchestration, safety, identity, networking, and observability; includes Responses API and Computer-Using Agent in Azure	Docs
AWS Bedrock (Nova)	Converse API (tool use)	Bedrock Agents	Agents: managed service with automatic orchestration + state management + multi-agent collaboration. Converse: manual tool orchestration, full control	Docs
Cohere	Standard chat	Command A	Command A: enhanced multi-step tool use, REACT agents, ~150% higher throughput	Docs

This fragmentation across providers is exactly why LaunchDarkly's approach matters: you configure once (messages vs. goals), and LaunchDarkly handles the provider-specific translation. Want to switch from OpenAI to Anthropic? Just change the provider in your AI Config. Your application code stays the same.

Note on OpenAI's ecosystem (Nov 2025): The Agents SDK is OpenAI's production-ready orchestration framework. It uses the Responses API by default, and via a built-in LiteLLM adapter it can run against other providers with an OpenAI-compatible shape. Chat Completions is still supported, but OpenAI recommends Responses for new work. The Assistants API is deprecated and scheduled to shut down on August 26, 2026.

Common misconceptions

Now that you understand the modes and how they differ from provider-specific implementations, let's clear up some common points of confusion:

❌ "Agent mode provides automatic execution"
No. Both modes require you to orchestrate. Agent mode just provides a different input schema.

❌ "Agent mode is for complex tasks, prompt-based mode is for simple ones"
Not quite. It's about input format and framework compatibility, not task complexity.

❌ "I can only use tools in agent mode"
False. Both modes support tools. The difference is how you specify your task (messages vs. goal).

❌ "LaunchDarkly is an agent framework like LangGraph"
No. LaunchDarkly is configuration management for AI. Use it WITH frameworks like LangGraph, not instead of them.

Why LaunchDarkly's abstraction matters

Now that you've seen how fragmented the provider landscape is, let's explore the practical value of LaunchDarkly's abstraction layer.

Switching providers without code changes

Without LaunchDarkly:

# Hardcoded provider and prompts in your application
openai_client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
response = openai_client.chat.completions.create(
    model="gpt-4",  # Want to switch to Claude? Need to deploy new code
    messages=[
        {"role": "system", "content": "You are helpful"},  # Want to A/B test prompts? Deploy again
        {"role": "user", "content": "Hello"}
    ]
)

# To switch providers, you need to:
# 1. Write new code for different provider API
# 2. Deploy to production
# 3. Hope nothing breaks

With LaunchDarkly:

# Get config from LaunchDarkly
config = aiclient.config(key="my-ai-config", context=context)

# You still write provider-specific code, but only once
if config.provider.name == "openai":
    client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    response = client.chat.completions.create(
        model=config.model.name,      # Comes from LaunchDarkly
        messages=config.messages,      # Normalized schema across providers
        temperature=config.model.parameters.get('temperature')
    )
elif config.provider.name == "bedrock":
    client = boto3.client('bedrock-runtime', region_name='us-east-1')
    response = client.converse(
        modelId=config.model.name,    # Comes from LaunchDarkly
        messages=_convert_to_bedrock_format(config.messages),  # LaunchDarkly normalizes, you convert
        inferenceConfig={'temperature': config.model.parameters.get('temperature')}
    )

# Now you can switch providers via LaunchDarkly UI without deployment
# Change prompts, A/B test models, roll out gradually - all via configuration

The real value: Once your code is set up to handle different providers, you can switch between them, change prompts, A/B test models, and roll out changes gradually - all through the LaunchDarkly UI without deploying code. You write the provider handlers once; you manage AI behavior forever.

Security and risk management

AI agents can be powerful and potentially risky. With LaunchDarkly AI Configs, you can:

Instantly disable problematic models or tools without deploying code
Gradually roll out new agent capabilities to a small percentage of users first
Quickly roll back if an agent behaves unexpectedly
Control access by user tier (limit powerful tools to trusted users)
Target specific individuals in production to test experimental AI behavior in real environments without affecting other users

When you're not directly coupled to provider APIs, responding to security issues becomes a configuration change instead of an emergency deployment.

Advanced: Provider-specific packages (JavaScript/TypeScript)

For JavaScript/TypeScript developers looking to reduce boilerplate even further, LaunchDarkly offers optional provider-specific packages. These work with both prompt-based and agent modes and are purely additive - you don't need them to use LaunchDarkly AI Configs effectively.

Available packages:

@launchdarkly/server-sdk-ai-openai - OpenAI provider
@launchdarkly/server-sdk-ai-langchain - LangChain provider (works with both LangChain and LangGraph)
@launchdarkly/server-sdk-ai-vercel - Vercel AI SDK provider

What they provide:

Model creation helpers: One-line functions like createLangChainModel(aiConfig) that return fully-configured model instances
Automatic metrics tracking: Integrated metrics collection
Format conversion utilities: Helper functions to translate between schemas

Example with LangGraph:

// Get agent config from LaunchDarkly
const agentConfig = await ldClient.aiAgent('research-assistant', context);

// Create LangChain model - config already applied
const model = await LangChainProvider.createLangChainModel(agentConfig);

// Use with LangGraph
const agent = createReactAgent(model, tools, agentConfig.instructions);
const response = await agent.invoke({ messages: [{ role: "user", content: "Research X" }] });

Production readiness: These packages are in early development and not recommended for production. They may change without notice.

Python approach: The Python SDK takes a different path with built-in convenience methods like track_openai_metrics() in the single launchdarkly-server-sdk-ai package. See Python AI SDK reference.

Start building with LaunchDarkly AI configs

You now understand how LaunchDarkly's prompt-based and agent modes provide provider-agnostic configuration for your AI applications. Whether you're building chat interfaces or complex multi-agent systems, LaunchDarkly gives you the flexibility to experiment, iterate, and ship AI features without the complexity of managing multiple provider APIs.

Choosing your mode:

Start with prompt-based mode if:

You're building a chat interface or conversational UI
You need online evaluations for quality monitoring
You want precise control over multi-step workflows
You're uncertain which mode fits your use case (it's the more flexible starting point)

Choose agent mode if:

You're integrating with LangGraph, LangChain, CrewAI, or similar frameworks
Your task is goal-oriented rather than conversational ("Research X and create Y")
You're feeding arbitrary data and asking open-ended questions about it

Remember: Both modes give you the same core benefits: provider abstraction, A/B testing, and runtime configuration changes. The choice is about input format, not capabilities.

Get started:

Sign up for a free LaunchDarkly account
Create your first AI Config: Takes less than 5 minutes
Explore example implementations: Learn from working code
Start with prompt-based mode unless you're specifically using an agent framework

All I Want for Christmas is Observable Multi-Modal Agentic Systems

Scarlett Attensil — Wed, 17 Dec 2025 17:31:15 +0000

How Session Replay + Online Evals Revealed How My Holiday Pet App Actually Works

Original article published on December 17, 2025.

I added LaunchDarkly observability to my Christmas-play pet casting app thinking I'd catch bugs. Instead, I unwrapped the perfect gift 🎁. Session replay shows me WHAT users do, and online evaluations show me IF my model made the right casting decision with real-time accuracy scores. Together, they're like milk 🥛 and cookies 🍪 - each good alone, but magical together for production AI monitoring.

See the App in Action

Discovery #1: Users' 40-second patience threshold

I decided to use session replay to evaluate the average time it took users to go through each step in the AI casting process. Session replay is LaunchDarkly's tool that records user interactions in your app - every click, hover, and page navigation - so you can watch exactly what users experience in real-time.

The complete AI casting process takes 30-45 seconds: personality analysis (2-3s), role matching (1-2s), DALL-E 3 costume generation (25-35s), and evaluation scoring (2-3s). That's a long time to stare at a loading spinner wondering if something broke.

What are progress steps?

Progress steps are UI elements I added to the app - not terminal commands or backend processes, but actual visual indicators in the web interface that show users which phase of the AI generation is currently running. These appear as a simple list in the loading screen, updating in real-time as each AI task completes. No commands needed - they automatically display when the user clicks "Get My Role!" and the AI processing begins.

Session replay revealed:

WITHOUT Progress Steps (n=20 early sessions):
0-10 seconds: 20/20 still watching (100%)
10-20 seconds: 18/20 still watching (90%)
20-30 seconds: 14/20 still watching (70%) - rage clicks begin
30-40 seconds: 9/20 still watching (45%) - tab switching detected
40+ seconds: 7/20 still watching (35% stay)

WITH Progress Steps (n=30 after adding them):
0-10 seconds: 30/30 still watching (100%)
10-20 seconds: 29/30 still watching (97%)
20-30 seconds: 25/30 still watching (83%)
30-40 seconds: 23/30 still watching (77%)
40+ seconds: 24/30 still watching (80% stay!)

Critical Discovery: Progress steps more than DOUBLED
completion rate (35% → 80%)

This made the difference:

Clear progress steps:

Step 1: AI Casting Decision
Step 2: Generating Costume Image (10-30s)
Step 3: Evaluation

As each completes:
✅ Step 1: AI Casting Decision
Step 2: Generating Costume Image (10-30s)
Step 3: Evaluation

Session replay showed users hovering over the back button at 25 seconds, then relaxing when they saw "Step 2: Generating Costume Image (10-30s)." The moment they understood DALL-E was creating their pet's costume (not the app freezing), they were willing to wait. Clear progress indicators transform anxiety into patience.

Discovery #2: Observability + online evaluations give the complete picture

Session replay shows user behavior and experience. Online evaluations expose AI output quality through accuracy scoring. Together, they form a solid strategy for AI observability.

To see this in action, let's take a closer look at an example.

Example: The speed-running corgi owner

In this scenario, a user blazes through the entire pet app setup from the initial quiz to the final results, completing the process in record time. So fast, in fact, that instead of this leading to a favorable outcome, it led to an instance of speed killing quality.

Session Replay Showed:

Quiz completed in 8 seconds (world record) - they clicked the first option for every question
Skipped photo upload entirely
Waited the full 31 seconds for processing
Got their result: "Sheep"
Started rage clicking on the sheep image immediately
Left the site without saving or sharing

Why did their energetic corgi get cast as a sheep? The rushed quiz responses created a contradictory personality profile that confused the AI. Without a photo to provide visual context, the model defaulted to its safest, most generic casting choice.

Online Evaluation Results:

Evaluation Score: 38/100 ❌
Reasoning: "Costume contains unsafe elements: eyeliner, ribbons"
Wait, what? The AI suggested face paint and ribbons, evaluation said NO

Online evaluations use a model-agnostic evaluation (MAE) - an AI agent that evaluates other AI outputs for quality, safety, or accuracy. The out-of-the-box evaluation judge is overly cautious about physical safety. For the above scenario, the evaluation comments:

"Costume includes eyeliner which could be harmful to pets" (It's a DALL-E image!)
"Ribbons pose entanglement risk"
"Bells are a choking hazard" (It's AI-generated art!)

About 40% of low scores are actually the evaluation being overprotective about imaginary safety issues, not bad casting.

Speed-runners get generic roles AND the evaluation writes safety warnings about digital costumes. Users see these low scores and think the app doesn't work well.

But speed-running isn't the whole story. To truly understand the relationship between user engagement and AI quality, we need to see the flip side. The perfect user. One who gives the AI everything it needed to succeed. What happens when a user takes their time and engages thoughtfully with every step?

Example: The perfect match

Session Replay Showed:

45 seconds on quiz (reading each option)
Uploaded photo, waited for processing
Spent 2 minutes on results page
Downloaded image multiple times

Online Evaluation Results:

Evaluation Score: 96/100 ⭐⭐⭐⭐⭐
Reasoning: "Personality perfectly matches role archetype"
Photo bonus: "Visual traits enhanced casting accuracy"

Time invested = Quality received. The AI rewards thoughtfulness.

Discovery #3: The photo upload comedy gold mine

Session replay revealed what photos people ACTUALLY upload. Without it, you'd never know that one in three photo uploads are problematic, and you'd be flying blind on whether to add validation or trust your model.

Example: The surprising photo upload analysis

Session Replay Showed:

Photo Upload Analysis (n=18 who uploaded):
- 12 (67%) Normal pet photos
- 2 (11%) Screenshots of pet photos on their phone
- 1 (6%) Multiple pets in one photo (chaos)
- 1 (6%) Blurry "pet in motion" disaster
- 1 (6%) Stock photo of their breed (cheater!)

Despite 33% problematic inputs, evaluation scores remained high (87-91/100). The AI is remarkably resilient.

Example: When "bad" photos produce great results

My Favorite Session: Someone uploaded a photo of their cat mid-yawn. The AI vision model described it as "displaying fierce predatory behavior." The cat was cast as a "Protective Father." Evaluation score: 91/100. The owner downloaded it immediately.

The Winner: Someone's hamster photo that was 90% cage bars. The AI somehow extracted "small fuzzy creature behind geometric patterns" and cast it as "Shepherd" because "clearly experienced at navigating barriers." Evaluation score: 87/100.

Without session replay, you'd only see evaluation scores and think "the AI is working well." But session replay reveals users are uploading screenshots and blurry photos—input quality issues that could justify adding photo validation.

However, the high evaluation scores prove the AI handles imperfect real-world data gracefully. This insight saved me from over-engineering photo validation that would have slowed down the user experience for minimal quality gains.

Session replay + online evaluations together answered the question "Should I add photo validation?" The answer: No. Trust the model's resilience and keep the experience frictionless.

The magic formula: Why this combo works (and what surprised me)

Without Observability:

"The app seems slow" → ¯\(ツ)/¯
"We have 20 visitors but 7 completions" → Where do they drop?

With Session Replay ONLY:

"User got sheep and rage clicked; maybe left angry" → Was this a bad match?

With Model-Agnostic Evaluation ONLY:

"Evaluation: 22/100 - Eyeliner unsafe for pets" → How did the user react?
"Evaluation: 96/100 - Perfect match!" → How did this compare to the image they uploaded?

With BOTH:

"User rushed, got sheep with ribbons, evaluation panicked about safety"
→ The OOTB evaluation treats image generation prompts like real costume instructions
"40% of low scores are costume safety, not bad matching"
→ Need custom evaluation criteria (coming soon!)
"Users might think low score = bad casting, but it's often = protective evaluation"
→ Would benefit from custom evaluation criteria to avoid this confusion

The evaluation thinks we're putting actual ribbons on actual cats. It doesn't realize these are AI-generated images. So when the casting suggests "sparkly collar with bells," the evaluation judge practically calls animal services.

Now that you've seen what's possible when you combine user behavior tracking with AI quality scoring, let's walk through how to add this same observability magic to your own multi-modal AI app.

Your turn: See the complete picture

Want to add this observability magic to your own app? Here's how:

1. Install the packages

npm install @launchdarkly/observability
npm install @launchdarkly/session-replay

2. Initialize with observability

import { initialize } from 'launchdarkly-js-client-sdk';
import Observability from '@launchdarkly/observability';
import SessionReplay from '@launchdarkly/session-replay';

const ldClient = initialize(clientId, user, {
  plugins: [
    new Observability(),
    new SessionReplay({
      privacySetting: 'strict' // Masks all data on the page - see https://launchdarkly.com/docs/sdk/features/session-replay-config#expand-javascript-code-sample
    })
  ]
});

3. Configure online evaluations in dashboard

Create your AI Config in LaunchDarkly for LLM evaluation
Enable automatic accuracy scoring for production monitoring

Set accuracy weight to 100% for production AI monitoring
Monitor your AI outputs with real-time evaluation scoring

4. Connect the dots

Session replay shows you:

Where users drop off
What confuses them
When they rage click
How long they wait

Online evaluations show you:

AI decision accuracy scores
Why certain outputs scored low
Pattern of good vs bad castings
Safety concerns (even for pixels!)

Together they reveal the complete story of your AI app.

Resources to get started:

Full Implementation Guide - See how this pet app implements both features

Session Replay Tutorial - Official LaunchDarkly guide for detecting user frustration

When to Add Online Evals - Learn when and how to implement AI evaluation

The real magic is in having observability AND online evaluations.

Try it yourself

Cast your pet: https://scarlett-critter-casting.onrender.com/

See your evaluation score ⭐. Understand why your cat is a shepherd and your dog is an angel. The AI has spoken, and now you can see exactly how much to trust it!

Ready to add AI observability to your multi-modal agents?

Don't let your AI operate in the dark this holiday season. Get complete visibility into your multi-modal AI systems with LaunchDarkly's online evaluations and session replay.

Get started: Sign up for a free trial → Create your first AI Config → Enable session replay and online evaluations → Ship with confidence.

Day 7 | 🎄✨The Rockefeller tree in NYC: SLOs that actually drive decisions

Alexis Roberson — Wed, 17 Dec 2025 04:47:02 +0000

Originally published in the LaunchDarkly Docs

Most Subject Level Objectives or SLOs sit in dashboards gathering dust. Seeing that SLOS are performance targets that can be measured, they're extremely important in understanding the quality of a service or system. You define them, measure them, but when your conditions are not met, there’s no followup.

The biggest drawback is that SLOs are created to add value but if they’re never reinforced, it’s impossible to drive decisions, influence roadmaps, or help during incidents.

When it comes to defining SLOs, many folks often start at the top of the funnel by picking general metrics to measure, but in order to create SLOs that work, it’s important to understand how the roots impact the leaves.

In this post, we'll cover the pitfalls leading to out of sync SLOs with a few tips and tricks to ensure what you measure produces business value. You'll also see an example of how to set SLOs in realtime for a flag evaluation feature that you can implement in your own planning process.

But first, we'll explore a tree metaphor to recap key observability components and how they're influence expands from the roots all the way to the leaves. What if the popular Rockefeller tree in NYC represented the relationship between telemetry data and SLOs?

Understanding the Observability Tree

SLOs would essentially be the leaves on the Response branch as shown in the above image, meaning they are visible, measurable targets that everyone can see. The leaves would not be possible without the support of the trunk.

This is your telemetry data or the traces, logs, events you collect from your system. The data you collect acts as the foundation and support for the branches and leaves.

The roots represent the things you cannot see but are still vital to overall health of your system. This is the hard part of understanding system behavior, debugging unknown unknowns, and making data-driven decisions.

Most teams skip the roots entirely. They define SLOs using only the trunk (logs, traces, events), measuring things that can already be measured. However, the business outcomes and user behaviors are buried in what would be defined as the roots. Sometimes you have to dig through the soil to ensure your SLOs don't end up technically accurate yet strategically ineffective.

What Makes an SLO Decision-Worthy

So what makes a good SLO? The goal of a SLO is to bridge engineering and business needs to support a high quality user experience. And a good SLO depends on three things, business clarity or asking the right questions, measurability, or can these components actually be measured, and actionable targets, or the game plan for when things go wrong.

First, you need business clarity, which are the roots of the previously mentioned observability tree. This means articulating why something matters in concrete terms like dollars, users, retention and also avoiding vague statements like "uptime is important." For instance, if I were measuring the impact of downtime on a checkout feature, I could establish the SLO scope with “each minute of checkout downtime costs us $12,000 in lost revenue based on our average transaction volume.” It is essential to be able to explain the business impact in one clear sentence.

Second, you need measurability. This is like the trunk of the tree. Your SLO must connect to your golden signals such as latency, traffic, errors, saturation. This is where a lot of aspirational SLOs fall apart. Upper management might want to measure user happiness, but how can engineering translate this into actual metrics? Try to express the business impact in one clear sentence. If that's difficult, it’s usually a sign the problem definition needs a bit more shaping before defining the SLO.

Third, you need actionable targets, which represent the leaves on the observability tree. This is where most SLOs fail even when they get the first two right. There's a number, maybe even a threshold, but no clear action plan. What happens when you miss it? Who gets paged? What gets paused? Decision-worthy SLOs specify exactly what happens at different levels of degradation, and more importantly, they give everyone the confidence to make decisions based on those levels.

Building production resilient SLOs: LaunchDarkly’s Flag evaluation example

We can apply these same principles of building a production-worthy SLO using LaunchDarkly’s flag evaluation feature.

The flag evaluation feature in the monitoring tab is an extension of observability where it tracks how often each flag variation is served to different contexts over time, and highlights flag changes that might affect evaluation patterns.

Now, let’s build a SLO.

Step 1: Start with the business question

What would be impacted if the flag evaluations monitoring feature broke? Customers use these charts to understand rollout progress, debug targeting issues, and verify that their flags are working as expected. If evaluation data is delayed or missing, they can't trust what they're seeing. They might roll back a working feature thinking it's broken, or fail to catch a real problem because the charts show stale data. This undermines confidence in the platform and increases support load.

Step 2: Translate to user experience terms

What does "working well" look like? When a customer makes a flag change and checks the monitoring tab, they see updated evaluation counts within a couple minutes. The charts load quickly (under 3 seconds). The data is accurate meaning evaluation counts match what's actually happening in their application. If there's a delay, we tell them explicitly rather than showing stale data as if it's current.

Step 3: Connect to telemetry

We track several golden signals for this feature.

Data pipeline latency: time from evaluation event to appearing in charts.
Chart load time: how long it takes to render the monitoring page.
Data accuracy: comparing our recorded evaluations against a known sample.
Error rate: failed queries or chart rendering errors.

For the sake of this example will set arbitrary numbers for these signals. Let’s say you had a median pipeline latency of 45 seconds, p95 at 2 minutes, p99 at 5 minutes. And a chart load time averages 1.2 seconds. Data accuracy is 99.7 percent (some evaluations drop due to sampling) and error rate is 0.3 percent.

Using this data, we can set the target.

Step 4: Set the target

Based on that data, here's our SLO: 98 percent of flag evaluation events will appear in monitoring charts within 3 minutes, with chart load times under 3 seconds at p95.

Why these numbers? Customer research shows they expect "near real-time" monitoring, which they define as 2-3 minutes. Anything longer feels like stale data. Three seconds for chart loading is the threshold where users perceive delay and start questioning if something's broken.

We chose 98 percent instead of 99.9 percent because some evaluation events get sampled out intentionally for cost reasons, and occasional data pipeline delays from third-party dependencies are acceptable.

Now that we have our targets, we can use those thresholds to set conditional responses based on alerts or indicators.

Step 5: Define operational responses

Responses for Green, Red, or Yellow indicators in production:

If Green (>98%, <3 min, <3 sec load), continue normal operations.
If Yellow (95-98%, or 3-5 min, or 3-5 sec load), alert on-call, investigate within 4 hours.
If Red (<95%, or >5 min, or >5 sec load), page immediately, update status page if widespread.

Step 6: Drive decisions

Now the SLO becomes your decision-making framework. When engineering proposes adding a new feature like "evaluations by SDK" breakdown, the first question is: "Will this keep us within our 3-second chart load SLO?" If the answer is no, we either optimize the implementation or push back on the feature.

Infrastructure changes get evaluated the same way. Before migrating the data pipeline to a new system, we load tests against both our latency and accuracy targets. If the migration risks our SLO, we either fix the architecture or delay the migration. Another way I've seen SLOs used is planning future work. ie. if a team knows they are in the yellow this month, they may avoid picking up other risky work.

The SLO transforms from a monitoring target into a decision filter, helping to determine what gets shipped and what doesn’t.

Bringing it all together

Great SLOs aren't just leaves you pluck and add to dashboards. They're connected to everything below them from the trunk of solid telemetry to the roots of understanding what actually matters to your business and users. If you skip those foundational layers, your SLOs become technically accurate but strategically useless.

Start with the roots. Ask what would be impacted if this feature were to break. Work your way up through user experience and technical measurement. Build SLOs that bridge engineering and business with clear thresholds and clear consequences. And finally, make them specific enough to drive real decisions.