DEV Community: Scarlett Attensil

Benchmark LangGraph, Strands, OpenAI Agents, and Google ADK on the same agent graph

Scarlett Attensil — Tue, 23 Jun 2026 19:03:00 +0000

Agent framework debates are mostly vibes. One engineer swears LangGraph is faster, another prefers the OpenAI Agents SDK, someone wants Google ADK because it feels future-proof. The team picks one, wires the workflow into its SDK, and the choice is welded in. Changing frameworks later means tearing out the wiring for one SDK and rebuilding the workflow on another, an expensive rewrite few teams take on.

This tutorial makes that decision reversible and then settles it with data. You put the agent graph in LaunchDarkly and run four frameworks (LangGraph, Strands, OpenAI Agents SDK, and Google ADK) over the same topology, with the model pinned so the framework is the only variable. A LaunchDarkly experiment ranks them on graph latency and token use, with an LLM judge guarding quality. The results table tells you which framework runs your graph fastest without degrading it.

This tutorial is the sequel to Compare AI orchestrators, which ran the same workflow across frameworks but kept the topology in each framework's code. Here, the topology, routing, models, prompts, tools, and judge all live in LaunchDarkly, and each framework supplies only two functions.

The experiment results do more than set a benchmark. The flag that splits experiment traffic also routes production. When one framework wins, you don't rewrite the app; you change the flag to serve the winner. In a single loop, LaunchDarkly does three jobs: the graph definition, the experiment split, and the runtime control that ships the winner.

The workload is a research-gap analysis over a set of arXiv papers. Two readers, approach-analyzer and contradiction-detector, read the same papers in parallel and fan in to gap-synthesizer, which writes the report.

Prerequisites

A LaunchDarkly account with AgentControl access, and your environment's SDK key
Python 3.11+ and uv
An ANTHROPIC_API_KEY for the pinned model. OPENAI_API_KEY and GOOGLE_API_KEY are only needed if you run the optional native-model bake-off in Step 9
The companion repo: ai-orchestrators on branch tutorial/graph-experiments

The experiment design

The comparison is controlled: same graph, same model, same papers, same judge, with the framework as the only variable. Mechanically it runs in four stages:

Bootstrap. manifest.yaml creates the node configs, graph, orchestrator flag, and judge in LaunchDarkly.
Route. On each request, the app evaluates the orchestrator flag to pick a framework: langgraph, strands, openai-agents, or google-adk.
Run. The dispatcher runs the shared graph as a directed acyclic graph (DAG). The two readers run concurrently and fan in to the synthesizer.
Measure. Each run records how long the graph took, how many tokens it used, and whether the report passed the quality judge.

The shape looks like this:

                        ┌──▶  approach-analyzer  ───────┐
   intake (papers) ─────┤                               ├──▶  gap-synthesizer  ──▶  report
                        └──▶  contradiction-detector  ──┘

Step 1: Create the graph, flag, and judge

Everything starts from one file, config/graph_experiment_manifest.yaml. It declares the fetch_paper tool, four node configs (intake plus the three agents, pinned to claude-sonnet-4-5), the graph, the orchestrator flag, and the judge.

First, clone the companion repo and install its dependencies with uv:

git clone https://github.com/launchdarkly-labs/ai-orchestrators
cd ai-orchestrators
git checkout tutorial/graph-experiments
uv sync

Next, set up a LaunchDarkly project. The bootstrap doesn't create one, so create it with the LaunchDarkly MCP server, the projects agent skill, or the UI. Name it graph-experiments to match the value in .env.example, so the defaults work without edits. When it exists, copy its key into LD_PROJECT_KEY and its production environment SDK key into LD_SDK_KEY in .env. The runners and experiment harness use that SDK key to evaluate the flag and graph. The bootstrap also reads LD_API_KEY from .env to create the resources.

Copy the example file to create your .env:

cp .env.example .env   # then set LD_PROJECT_KEY, LD_SDK_KEY, and LD_API_KEY in .env

With the keys in place, run the bootstrap:

uv run python scripts/launchdarkly/bootstrap.py config/graph_experiment_manifest.yaml

This creates all four node configs, the research-gap-graph, the orchestrator flag (created off), and the gap-quality-judge attached to the gap-synthesizer node (its synthesizer-claude variation, set to 100% sampling). The judge scores the final report against the source papers, so it can verify grounding and citations. A judge can only check based on the information it has, so we give it the papers, not only an upstream agent's analysis.

When the graph ships, it is incomplete by design. The bootstrap creates the contradiction-detector config but wires only intake to approach-analyzer to gap-synthesizer, leaving the detector out. You'll add it in Step 5 to complete the parallel fan-in.

When it finishes, the bootstrap prints a link to your new agent graph. Open it and review the topology before moving on. The graph shows a straight line from intake to approach-analyzer to gap-synthesizer, with contradiction-detector created but not yet wired in.

The flag is intentionally set to off

Until the experiment is live, ld.variation("orchestrator", …) falls back to the code default "langgraph", so every request routes to LangGraph. That behavior is correct for this stage. You'll force specific frameworks in Step 4, and the experiment takes over in Step 7.

Step 2: The dispatcher runs the graph

The dispatcher is the heart of the project, and it's the same code for every framework. It reads the graph as a DAG, runs the entry nodes concurrently, hands every node the papers as ground truth, and connects the readers at the fan-in node. The only framework-specific pieces are build_agent and invoke, which are passed in as arguments.

The whole process is about 100 lines, built on the agent graph traversal methods in the SDK. The complete dispatcher.py is in the companion repo.

Traversal methods or the managed run

The SDK gives you two ways to run an agent graph. This dispatcher uses the traversal methods, the lower-level API you walk yourself: agent_graph(), reverse_traverse(), the node and edge accessors, and the graph tracker. The SDK also offers a fully managed create_agent_graph(...).run() that handles orchestration and collects metrics for you with no traversal code, which we recommend when you're on a supported framework, such as LangGraph or the OpenAI Agents SDK, and don't need to inspect the run.

We use the traversal methods here because in a bake-off the walk itself is a controlled variable: one traversal with identical semantics for every framework (the managed runner covers two of the four today), concurrent execution of the independent readers under our own scheduling, and control over the exact input each agent and the judge receive. Same SDK, lower-level surface.

The dispatcher carries the design in four parts: it builds the execution plan from the graph's edges, composes each node's input, runs every ready node concurrently each round, and records the graph's metrics once per run.

First, the dispatcher builds the execution plan from the graph's edges, so the topology you draw in LaunchDarkly runs:

for key, node in nodes.items():
    for edge in node.get_edges():
        target = edge.target_config
        if target in nodes:
            succ[key].append(target)
            preds[target].append(key)

Next, every node receives the source papers and any upstream analyses, so each agent and the judge work directly from the source material rather than a summary handed down a chain:

def compose_input(user_input, predecessor_outputs):
    parts = [f"=== SOURCE PAPERS ===\n{user_input}"]
    for key, out in predecessor_outputs:
        if out and out.strip():
            parts.append(f"=== {key} ===\n{out}")
    return "\n\n".join(parts)

Then each round runs every node whose predecessors have finished, concurrently, so the two readers fan out and fan in with no special casing:

ready = [k for k in pending if all(p in done for p in preds[k])]
results = await asyncio.gather(*(run_node(k) for k in ready))

Finally, the dispatcher records the graph's metrics on each run, including the end-to-end latency the experiment ranks on:

graph_tracker.track_duration(int((time.monotonic() - start) * 1000))
graph_tracker.track_total_tokens(TokenUsage(input=totals["in"], output=totals["out"], total=totals["in"] + totals["out"]))
graph_tracker.track_path(path)
graph_tracker.track_invocation_success()

The dispatcher reads the topology at runtime, so reshaping the workflow in the UI, adding a node, or redrawing an edge takes effect on the next request with no code change. You'll do exactly that in Step 5.

Step 3: Each framework is a thin adapter

Each framework implements build_agent(node_key, config, instructions) and async invoke(agent, input_text, tracker). Everything dynamic still comes from the LaunchDarkly node config: the model, the attached tools, and the instructions.

LangGraph has a LaunchDarkly companion package, so its runner is only a few lines. The companion handles model creation, tool binding, and token tracking, so the adapter holds no framework plumbing of its own:

def build_agent(node_key, config, instructions):
    llm = create_langchain_model(config)
    tools = build_tools(config, TOOL_REGISTRY)  # binds only this node's attached tools
    return create_react_agent(llm, tools, prompt=instructions)

async def invoke(agent, input_text, tracker):
    result = await tracker.track_metrics_of_async(
        lambda res: LDAIMetrics(success=True, tokens=sum_token_usage_from_messages(res.get("messages", []))),
        lambda: agent.ainvoke({"messages": [{"role": "user", "content": input_text}]}),
    )
    messages = result.get("messages", [])
    for message in messages:
        for name in get_tool_calls_from_response(message):
            tracker.track_tool_call(name)
    text = _content_to_text(messages[-1].content) if messages else ""
    return text, sum_token_usage_from_messages(messages)

Strands has no companion package, so its runner builds the model with a small provider-aware factory and binds tools with Strands' native @tool. The contract is identical:

def build_agent(node_key, config, instructions):
    return Agent(
        name=node_key,
        model=_create_strands_model(config),
        system_prompt=instructions or "Process the input and respond.",
        tools=_bind_tools(config),
        callback_handler=None,
    )

OpenAI Agents and Google ADK round out the four. For the comparison to stay fair, all four have to run the same model, but these two SDKs default to their own vendors' models. LiteLLM, a thin adapter, lets them call any provider, so we point both at the pinned claude-sonnet-4-5 and keep the model identical across all four orchestrators. No OpenAI or Google servers are involved. Instead, LiteLLM translates the request format in process and the call goes straight to Anthropic with your key.

Google ADK is fully companion-free, and OpenAI Agents uses the ldai_openai companion for token and tool-call telemetry even though it builds the model through LiteLLM. This experiment pins one model across all four frameworks, so every framework here runs Claude. Pointing each framework at its own vendor's default model instead is a separate, optional exercise, the native-model bake-off in Step 9. The tool callables live in TOOL_REGISTRY, a plain {name: callable} map that each framework binds its own way.

One tracking API, any framework

LaunchDarkly records tokens and latency through one framework-agnostic tracker. You provide a TokenUsage and call track_*, and the metrics flow the same way regardless of orchestrator. For LangGraph and OpenAI Agents, the companion helpers (ldai_langchain, ldai_openai) populate it automatically. For anything else, you read the framework's own usage and pass it along. Every orchestrator emits identical metrics, so you can compare them directly.

You can also use framework-specific tutorials

If you want a framework-specific starting point, Build a LangGraph multi-agent system walks the LangGraph path from scratch, and Migrate a hardcoded LangGraph agent to AgentControl shows how to externalize an existing agent's config and prompts.

Step 4: Smoke test the graph

Before you run any experiment, confirm the bootstrapped graph runs end to end. First, run one framework:

uv run python orchestrators/verify_run.py langgraph

It prints the path it took and the first part of the report. On the graph as it shipped, the path is intake -> approach-analyzer -> gap-synthesizer: intake runs its short pass, approach-analyzer reads the papers, and gap-synthesizer writes the report. There's no contradiction-detector yet, and no error. The metrics land in the AgentControl UI under the graph you created.

Step 5: Add the parallel fan-in in the UI

Here's the payoff of keeping the topology in LaunchDarkly: you finish building the workflow in the UI, with no redeploy, and the running app picks up the new shape on its next request. The contradiction-detector config already exists, with its fetch_paper tool attached. You wire it into the graph to add the second reader and form the parallel fan-in.

To complete the graph:

Click Agents in the LaunchDarkly sidebar.
Click Agent graphs.
Select research-gap-graph.
Add the contradiction-detector node.
Draw an edge from intake to contradiction-detector, then another from contradiction-detector to gap-synthesizer.
Click Save.

You add no routing logic: the edge itself is the route, because routing is structural.

Re-run the smoke test:

uv run python orchestrators/verify_run.py langgraph

The path now includes contradiction-detector, and because approach-analyzer and contradiction-detector run concurrently, their order can vary. You completed a multi-agent workflow from the UI, and the config you wired in already had its tool attached.

You finished a multi-agent workflow from the UI, mid-development, and the dispatcher ran the new shape on the next request. No redeploy, no code change: the graph you draw is the graph that runs.

Step 6: Smoke test all four frameworks

Before you collect experiment data, make sure all four frameworks can run the completed graph. One command runs all four in sequence:

uv run python orchestrators/verify_run.py all

It runs each framework against the completed graph and ends with a pass/fail summary, one line per framework, exiting non-zero if any framework failed, so it works as a gate. Each framework prints the path it took and a preview of its report, then a final summary collects the results. A successful run looks like this:

▶ Running 'langgraph' over 2 papers on graph 'research-gap-graph'...
  ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer
▶ Running 'strands' over 2 papers on graph 'research-gap-graph'...
  ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer
▶ Running 'openai-agents' over 2 papers on graph 'research-gap-graph'...
  ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer
▶ Running 'google-adk' over 2 papers on graph 'research-gap-graph'...
  ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer
=== smoke summary ===
  ✓ langgraph
  ✓ strands
  ✓ openai-agents
  ✓ google-adk

If a framework fails, its line shows an ✗ instead of a ✓ and the command exits non-zero. All four smoke-test against the pinned Claude model. ANTHROPIC_API_KEY is the only model key you need, because OpenAI Agents and Google ADK reach Claude through LiteLLM. The OpenAI Agents SDK turns on tracing by default and looks for OPENAI_API_KEY to export traces, so the openai-agents run may print a harmless tracing warning when that key is absent. It doesn't affect the run.

Step 7: Run it through the experiment

Now you can use a LaunchDarkly experiment to rank the four frameworks on real traffic, on the same graph, with the model held constant. Because the model is fixed, the comparison is operational: which orchestrator delivers the model's quality fastest, with the least token overhead. The bootstrap already created the flag, the judge, and the graph.

These metrics are measured on each request, so do a one-time setup first:

Make the request context kind available for experiments.
Set the analysis unit of graph latency, tokens, and the judge metric to request.

Then create the experiment in the UI:

Create an experiment with the orchestrator flag as the treatment.
Set the primary metric to Graph latency ($ld:ai:graph:duration:total, the time for a complete graph execution).
Add tokens and $ld:ai:judge:gap-quality as secondary metrics.
Set the audience to 100% and the randomization unit to request. Each run is a single request, there are no users in this workflow, and request is the unit LaunchDarkly measures AI and graph metrics by.
Turn on the orchestrator flag, which the bootstrap created set to off, so it serves the experiment's variations.
Start an experiment iteration.

We rank on latency and tokens because, with the model and the graph held constant, those are the things that genuinely differ: a framework can move quality only by degrading the plumbing, like a truncated report or a broken tool call. So $ld:ai:judge:gap-quality stays a guardrail that catches a framework "winning" by cutting corners, not part of the ranking. Swap the model, prompt, or tools later instead of the framework, and that same judge becomes your primary metric.

Then drive traffic. The flag assigns each run one framework at random:

uv run python scripts/run_experiment.py --runs-per-category 6

That's six runs over each of the six shipped topics, 36 in total. Assignment is random, so it usually fills all four variations, though it isn't guaranteed. Each run analyzes the topic's entire paper set, because gap analysis needs every paper to find real gaps.

Open the experiment in LaunchDarkly: latency per variation, with tokens and $ld:ai:judge:gap-quality alongside. The winner is the framework with the best latency and lowest token use that doesn't let quality slip. Because the model is pinned, cost is a fixed multiple of tokens, so the token column is also the cost ranking; for actual dollar figures, read them from Insights.

Because the experiment holds everything but the framework constant, most of these bars land close, often within a few percent, which is by design.

In our run, Strands won on speed: it ran the graph fastest, with quality holding at the guardrail. If you optimize for speed and quality holds, that makes Strands the orchestrator to ship for this workload. Six topics and one randomized split isn't a large sample, so confirm the lead with more topics before you standardize on it. You can do that in Step 9.

Step 8: Ship the winner with runtime control

The experiment gave you data. The reason to run it in LaunchDarkly, rather than a one-off script, is that acting on that data takes no deploy: the orchestrator flag that was the experiment treatment is also your production router.

When a variation wins, stop the iteration and set the flag's default to that framework. Every request routes to it on the next evaluation, with no redeploy.

Then automate what you don't want to babysit. An adaptive trigger watches a guardrail and changes a flag on its own when production drifts past it. The orchestrator you shipped is operational and won't degrade by itself, so point the trigger at the model flag from Step 9: it fails over to a backup model when your primary provider has a bad day, the same guardrail driving a different flag. That closes the loop: experiment to find the winner, runtime control to ship it, and automation to keep it healthy.

Step 9: Extend the experiment

Tighten the bands by adding more topics. Confidence comes from more distinct topics, not more runs over the same few. Download one with a title-phrase (ti:) query, and the harness picks it up automatically on the next run:

uv run python scripts/download_papers.py --query 'ti:"LLM-as-a-judge"'

Make quality the headline by flipping a config, not a flag. The framework lives in the orchestrator flag because it is app-level routing, not a property of any agent. The model, the prompt, and the tool set are different: they live in the node configs, so you experiment on the config itself. Add a second variation to a node, such as gap-synthesizer with a stronger model or a tightened prompt, and run an experiment with that config as the treatment and its variations as the arms. Pin the framework by setting the orchestrator flag to one value and leave the graph alone, so the config is the only thing moving. The judge attached to the synthesizer already emits $ld:ai:judge:gap-quality, so quality is the primary metric with no new instrumentation. Now it genuinely moves, because a different model or prompt reasons differently about the same papers.

Experiment on the graph shape with a graph-key flag. The dispatcher takes the graph key as an argument, so the shape is another value you can put behind a flag:

graph_key = ld.variation("graph_shape", context, "research-gap-graph")
result = await execute_graph(ai_client, graph_key, context, user_input, build_agent, invoke)

Build two graphs with different keys: for example, a linear research-gap-graph-linear (intake to approach-analyzer to gap-synthesizer) against the parallel research-gap-graph, or one with an added critic node against one without. Make a multivariate graph_shape flag whose variations are those graph keys, evaluate it exactly as the app evaluates orchestrator, and set it as the experiment treatment with the framework and model held constant. You are measuring whether the extra structure earns its latency and quality, and because the dispatcher runs whatever shape the key resolves to, no runner or dispatcher code changes. You build the judge once, and it is the guardrail for the framework bake-off and the headline metric for every model, prompt, tool, and shape you test next.

Run a native-model bake-off. This experiment holds the model constant so the framework is the only variable. To compare each framework on its own default model instead, build separate node configs per framework. This is the optional bake-off the prerequisites mention. It's a follow-up beyond this walkthrough, and the only part that needs OPENAI_API_KEY and GOOGLE_API_KEY.

Whatever you flip, follow three rules:

Change one variable at a time (the framework, the model, or the shape), never two. If you change more than one, you can't attribute the win.
Keep the quality guardrail on every run, because the fastest variant is often the one that quietly truncated its report or dropped a tool call.
Earn confidence with distinct inputs, not repeats: a tight band around three repeated topics is still a tight band around the wrong number.

To learn more about judge design, read When to add online evals and Evaluating with LLM-as-judge evaluators. To add a pre-production regression layer, read Offline evaluation of RAG-grounded answers.

Recap and next steps

Framework choice doesn't have to be a one-way door. Put the topology in a LaunchDarkly agent graph, have each framework supply only build_agent and invoke, and let one experiment settle a question that usually gets answered by whoever argues hardest: pin the model, let the judge guard quality, and pick the orchestrator that delivers it fastest, with evidence in hand.

Then keep going, because the framework is only the first swappable component. The same flag, experiment, and judge machinery compares models, prompts, tools, and whole graph shapes the same way, so "which is better" stops being a debate and becomes a measurement. And because the experiment and the runtime control are one flag, you never stop at a finding: you ship it, ramp it with a progressive rollout, and let an adaptive trigger hold the line in production while the AI iteration loop for reliable agents keeps the next change shipping behind eval gates.

The complete code is in the sample repo. Get started with AgentControl, point the four frameworks at a graph your team actually runs, and settle the next framework argument with a number instead of a hunch.

AI Experimentation Best Practices: From Evaluation to Safe Production Rollouts

Scarlett Attensil — Tue, 02 Jun 2026 17:09:35 +0000

Introduction

Artificial intelligence tools, particularly large language models (LLMs), are not like traditional software. AI is probabilistic, so the same instructions and inputs can produce different results, especially when using non-zero temperature or other sampling methods, and those results can shift as your context changes. That unpredictability brings real risks because models can miss the mark, invent facts, or generate unfair or unsafe outputs. They can also incur unexpected costs and slow down under heavy loads, and they must constantly adapt to evolving policies and ethical guidelines.

AI experimentation means iteratively testing data, algorithms, prompts, models, and parameters to optimize model performance and validate hypotheses. You need a clear, repeatable way to try ideas, compare prompts and models, validate how your system finds and uses information, and do safety checks before changes reach real users. Experimentation is not just a nice-to-have; it is essential for shipping AI responsibly, optimizing resource efficiency, reducing costs, and accelerating innovation through rapid, evidence-based iteration cycles.

Throughout this guide, we distinguish evaluation from experimentation. Evaluation means offline benchmarking and scoring, including test sets, human or AI judges, and quality metrics. Experimentation means controlled production changes that affect real users through A/B tests, staged rollouts, or other release strategies. Evaluation tells you whether a variant clears a quality bar; experimentation tells you whether it beats the baseline in production, with statistical confidence and guardrails.

In this article, we cover the core ideas and practical steps for AI experimentation: how to plan a test, evaluate changes, run controlled trials with real users, choose metrics that actually matter to your product, and roll out changes safely. By the end, you will have a process that moves from initial concept to monitored, controlled production release.

AI Experimentation Best Practices

Best Practice	Description
Use experimentation to manage uncertainty	AI outputs can shift over time. Structured experimentation helps teams measure, compare, and validate changes before they reach users. LaunchDarkly AgentControl experiments and release options help turn unpredictability into a controlled process for improvement.
Build trust through evidence, not intuition	Without experimentation, teams rely on gut feeling. Controlled tests provide measurable evidence of what works. Use LaunchDarkly Experimentation, metrics, and AgentControl monitoring to make confident, data-driven decisions.
Detect and reduce hidden risks early	Experimentation surfaces hallucinations, bias, latency regressions, and safety failures before they affect broad audiences. Online evaluations and guarded rollouts help teams detect regressions and pause or roll back unsafe changes.
Enable continuous improvement	AI systems evolve as data, models, and contexts change. Config variations, config targeting, and progressive rollouts give teams a repeatable way to adapt while controlling exposure.
Design experiments with statistical power and variance in mind	Collect multiple observations per variant to account for nondeterminism. Use confidence intervals and statistical significance tests rather than single-run comparisons. Define a minimum detectable effect (MDE), guardrail metrics, and a decision rule before launch. LaunchDarkly experiments and metrics support this evidence-based workflow.
Support responsible and compliant AI	Experimentation frameworks help teams evaluate whether updates align with ethical standards, privacy requirements, and evolving policies. LaunchDarkly role-based access control, approvals, and audit logs help make responsible AI development a built-in process.
Keep track of cost and latency	Track per-session spend and speed, set budgets and max token limits, optimize prompts and context, use caching or streaming where appropriate, and monitor TTFT, p95/p99 latency, retries, and spend. AgentControl monitoring and autogenerated AI metrics help surface cost, latency, and token usage by variation.
Conduct controlled testing with real users	Run A/B tests, sticky cohorts, staged rollouts, or interleaving strategies. Measure satisfaction, task completion, latency, cost, and business impact. Use targeting rules, percentage rollouts, and guarded rollouts to control exposure and rollback thresholds.
Perform evaluation	Define metrics for truthfulness, user experience, reliability, safety, cost, and speed. Test in layers and expand only when stable. Evaluation tells you whether a system meets a bar, while experimentation determines which variant should be trusted in production. LaunchDarkly online evaluations, datasets, and judges support layered AI evaluation workflows.
Use retrieval evaluation for RAG	Evaluate model quality by measuring recall@k, precision@k, citation accuracy, unsupported claim rate, cost, and latency. After offline quality assessment, use live or shadow traffic for controlled experiments that optimize retrievers, chunking, ranking, or reranking. LaunchDarkly AgentControl experiments and monitoring help compare these changes safely.
Ensure proper governance and safety for AI experimentation	Pre-register your experiment plan, including hypothesis, primary metric, MDE, guardrails, and rollback rules. Version prompts, models, and configurations. LaunchDarkly config management, approvals, and audit logs help preserve compliance, safety, and auditability.

Note: Testing different chunking or embedding models usually requires building and validating separate vector indexes, and sometimes separate databases, because embeddings are tied to the index schema. Swapping these at inference time requires architectural planning, reindexing, and migration.

Why AI Needs Experimentation

Traditional software works like a calculator: same input, same output. AI is more like a conversational assistant that can be helpful and creative but sometimes surprising. Since AI is not fully predictable and small changes in wording can shift results, you cannot judge the quality of an AI feature from a single right answer.

AI features are pipelines with many moving parts: models that may update, prompts that steer behavior, tools and APIs that can fail, and knowledge sources that drift as content changes. All of these can affect accuracy, safety, speed, and cost. A one-time test will not catch issues that appear under real traffic.

That is why experimentation is essential. It gives teams a structured way to observe, measure, and improve AI behavior as conditions change. Through continuous testing, you can detect drift, uncover hidden risks, and build confidence that your system performs reliably and responsibly.

LaunchDarkly helps teams operationalize this workflow with AgentControl, configs, config variations, config targeting, monitoring, and online evaluations.

The Hierarchy of Levers: Where to Focus Your Optimization Efforts

In practice, AI experimentation levers should be optimized in order of impact and reversibility:

System message
Examples
Output format
Context
Retries and fallbacks
Models and parameters

This order matters because many high-impact changes can be made without retraining or rebuilding your system. With AgentControl config variations, teams can version and compare these changes while controlling exposure through targeting.

System Message Variations

The system message is one of the most powerful levers in shaping an AI model’s behavior. It defines the model’s role, tone, and boundaries, setting the personality and guardrails for how it responds.

Small changes here can dramatically affect safety and reliability. Tightening tone or adding an out-of-scope clause can prevent speculative or unsafe content. However, overly rigid instructions can make responses sound robotic or unhelpful.

Experiment with several system-message variations and test how they perform across normal, edge, and adversarial scenarios. The goal is not only to find one prompt that works, but to understand how tone and framing influence quality, cost, safety, and latency. Store and compare these variants with AgentControl config variations and monitor results with config performance monitoring.

Choosing the Right Number of Examples

Compare zero-shot, one-shot, and few-shot examples, typically 3-5 examples. Mix common cases and edge cases, include “do” and “don’t” examples, and show the exact output format. Short examples teach patterns, but they also add tokens and delay. Measure accuracy, format adherence, generalization, latency, and cost with autogenerated AI metrics.

Output Format

Choose between free text, structured templates, or native structured outputs. Structured outputs are easier to parse and validate but can constrain creativity or break on truncation. Always validate responses, handle partial outputs gracefully, and keep templates simple. During testing, a temporary explain field can help diagnose why one variation performs better than another.

Context Window Size

Your experiment should test the cost-benefit tradeoff between precise context and extended context. Increasing context often increases cost and latency without improving output quality. Use AgentControl monitoring to compare variation-level latency and token usage before promoting a longer-context variant.

Retries With Backoff

Use one or two attempts for temporary errors such as rate limits, timeouts, or server overload. Add exponential backoff and jitter. Log error rates, latency, and cost. Ensure idempotency, cap retries, enforce timeouts, and offer a polite fallback when limits are hit. For production rollout, pair retry changes with guarded rollouts so latency and error regressions can halt expansion.

Fallback Chain

Route to a backup model or provider in the event of failures or slowness. Keep prompts and formats aligned so the backup model understands the same prompt structure and returns responses in the same format. Preserve conversation state, verify required features on the fallback, and log reasons for routing. LaunchDarkly config targeting can help route different cohorts to different model or provider variations.

The Expansion Rule

Experimentation should scale based on evidence, not enthusiasm. Once your pilot shows strong performance, expand the rollout to broader audiences. Scale only when metrics justify it: success rates are high, failure rates are low, safety checks pass, and time or cost remains acceptable. Use percentage rollouts, progressive rollouts, or guarded rollouts to expand with controlled risk.

Models and Parameters

Models and parameters are the tuning panel for an AI system: the set of dials you use when you want more accuracy, fewer hallucinations, faster responses, or lower cost.

Start with the right model for the job. Use a more capable model for complex reasoning or planning and a smaller, faster model for routine tasks. Match the model’s strength to the complexity and stakes of the task rather than defaulting to the largest model. Lock down the exact model version when possible so results stay reproducible as the model evolves. Version pinning reduces variability, but it does not eliminate drift. Upstream model behavior and real-world inputs can still change, so production experiments and ongoing holdbacks remain necessary.

AgentControl lets teams manage model selection, prompt content, provider configuration, and generation parameters with configs, variations, and AI model configurations.

Temperature

Temperature controls how adventurous or conservative a model’s output is. It is the primary generation setting most users adjust.

Keep it low, around 0-0.3, for code, structured formats, or safety-critical tasks.
Use higher values, around 0.7-1.0, for creativity or brainstorming.
Stay in the middle for everyday conversations.

Other sampling parameters, such as top_p or top_k, also influence output diversity, but temperature usually has the largest and most predictable effect, so it is often the first parameter worth tuning.

Retrieval and Search

Do not rely only on keywords because meaning matters. Semantic search helps the model understand intent. Hybrid search, combining semantic and keyword search, often works best for short queries or exact names. Choose an embedding model that fits your language and domain, and keep its version fixed.

A graph database models relationships and traversals, such as “how is X connected to Y?” A vector database or vector-enabled datastore is optimized for similarity search over embeddings to support retrieval in RAG pipelines. When testing retrieval changes, use online evaluations and AgentControl experiments to compare quality, latency, and cost.

Chunking and Metadata

Split documents into natural sections with slight overlaps. Sliding windows help for long text. Add metadata to improve filtering and relevance. When experimenting, start with a baseline and change one variable at a time: temperature, chunk size, top_k, reranking, or search type. Evaluate offline using a labeled dataset from your domain, then use controlled rollout strategies such as percentage rollouts or guarded rollouts before broad exposure.

Tool and Function Management

Tools are the hands and eyes of your AI. They turn abstract intelligence into real-world action. However, giving an AI system too many tools at once can create reliability, safety, and cost problems. A focused, well-defined toolset keeps the system efficient and predictable.

When experimenting with tools, start small. Give the AI only the tools it truly needs, then expand based on evidence. Simulate tool behavior with mock or historical data before allowing live writes or sensitive operations. Monitor error rates, latency, and cost. Use circuit breakers, fallback paths, and kill switches to keep the system stable when a tool fails.

LaunchDarkly AgentControl tools, agents, feature flags, and release controls can help teams expose new tool behavior gradually and roll back unsafe changes quickly.

Cost and Latency

Managing cost and latency in AI systems is like tuning a race car: you want speed and performance, but you cannot afford to burn all your fuel in one lap. The trick is knowing where your money and time actually go: input tokens, output tokens, model rates, tool usage, retries, retrieval, and post-processing.

Experiment design also affects cost. Multi-armed bandit approaches can reduce spend by shifting traffic away from losing variants early, while long, fixed-horizon A/B tests can waste budget after a clear loser emerges. Track cost per successful answer rather than cost per call so you know which variants are efficient and useful.

Several habits help:

Match the model to the job: Use smaller models for routine tasks and larger models for complex reasoning.
Set clear budgets: Cap tokens, cost, and retries per session.
Cache and reuse: Avoid paying twice for the same retrieval or generated output.
Retry wisely: Validate inputs early and use exponential backoff to avoid waste.
Measure what matters: Track cost per successful answer, not just cost per request.
Watch latency signals: Monitor time to first token, p95/p99 latency, and error rates.

LaunchDarkly Monitoring and autogenerated AgentControl metrics help teams compare token usage, duration, and variation-level performance.

Experimentation Before User Exposure

Before any major AI update reaches real users, it deserves a proper dress rehearsal. Catching issues early prevents bad experiences, unnecessary costs, and reputational damage.

Start by building a test set that mirrors real-world scenarios: genuine examples, synthetic edge cases, and adversarial prompts. If you are working with RAG, make sure answers link back to sources so you can evaluate grounding and citation quality. Use an AI judge or evaluation rubric to score correctness, completeness, clarity, safety, and faithfulness. LaunchDarkly datasets, judges, offline evaluations, and online evaluations support this progression from offline testing to production measurement.

Best practices include:

Set clear thresholds: Define what “good enough” means before the test begins.
Shadow test safely: Run the new model alongside the current one on real traffic while hiding results from users.
Control costs: Sample requests, cache results, and limit verbosity.
Protect fairness and privacy: Compare variants across quality, reliability, cost, and speed while respecting data boundaries.

Once the new model shows stable performance, no quality drops, no latency spikes, and no cost overruns, move to a canary rollout with rollback ready. Use percentage rollouts for fixed exposure, progressive rollouts for scheduled expansion, and guarded rollouts when you want metric-based monitoring and rollback.

Controlled Testing With Real Users

Testing with real users is where theory meets reality. The goal is to gather insight while keeping risk low and user experience intact.

A practical way to do this is A/B testing. By assigning users to consistent cohorts, you can compare different versions of your AI system under real conditions. This supports statistical decision-making, such as confidence intervals and significance testing, rather than anecdotal wins.

To make tests meaningful:

Keep traffic splits representative: Cover different user segments, regions, and use cases with targeting rules.
Tag everything: Include version, prompt, model, parameters, and settings in every request so outcomes are traceable.
Measure real impact: Track satisfaction, edits, retries, task completion, conversion, revenue lift, latency, and cost with LaunchDarkly metrics.

When rolling out updates, start with an internal beta, then gradually expand to 1%, 5%, 10%, and beyond. Watch quality, latency, safety, and failure rates closely. If something goes wrong, roll back and investigate. Creating guarded rollouts gives teams a structured way to tie rollout expansion to live metrics.

Not all AI experiments have a fixed end date. Many teams run ongoing control groups, holdbacks, or adaptive allocation strategies that monitor performance as models, data, and user behavior change. Even then, explicit guardrails and rollback thresholds are essential so optimization never trades off safety, latency, or cost.

Evaluation

Evaluation is not just checking whether the model runs. It is understanding how well the system serves users, how reliable it is under real conditions, and whether it delivers value within operational limits. A strong evaluation framework balances quality, cost, safety, and performance.

Layer evaluations in stages: offline tests, shadow testing, limited rollout, and broader production experiments. Set clear targets for quality, reliability, and cost. Instrument everything so you can explain wins and diagnose regressions. Expand only when metrics hold steady.

Quality and Accuracy

Start with the basics: Does the model tell the truth? Validate answers against known ground truth using offline tests and side-by-side reviews. AI judges provide scalable signals, but they should be calibrated against human review and used primarily for relative comparison between variants rather than absolute truth. LaunchDarkly judges and online evaluations help automate this scoring.

User Experience

Even a technically accurate model fails if it frustrates users. Focus on fast, helpful first responses and fewer handoffs to humans. Measure satisfaction, task completion, rewrite rates, time to first token, and time to useful answer.

Reliability

Reliability means tools behave predictably. Check that outputs match expected formats and that retries or timeouts are rare. Track error rates, schema validity, and success ratios. Define service-level objectives and trigger rollback if failures exceed limits. LaunchDarkly guarded rollouts can connect metric regressions to automated release decisions.

Cost and Speed

Every token, retrieval, and retry has a price. Break down latency and cost by stage to identify where resources are spent. Use smaller or cached models for routine tasks, stream responses where appropriate, and tighten prompts to reduce waste.

Observability

You cannot improve what you cannot see. Log prompts, parameters, model versions, config versions, and tool calls while masking personal data. Feed this data into dashboards that track cost, speed, quality, and safety. Use AgentControl monitoring, metrics, and observability integrations to detect drift and regressions.

Evaluating Retrieval Quality

Great answers depend on great context. Assess retrievers, rerankers, and generators separately and together:

Recall@k shows whether the right documents appear.
Precision@k shows whether retrieved documents are relevant.
nDCG and MRR show how well relevant documents are ranked.
Attributable accuracy connects correct answers to supporting evidence.
Unsupported claim rate flags hallucinations.
Citation correctness, freshness, cost, and latency show whether retrieval adds value.

Use offline QA sets with labeled passages and slice results by topic, query type, and language. Add confidence gating so the system can admit uncertainty instead of fabricating answers.

Observability for Retrieval

Instrument retrieval just like generation. Log query details, index versions, retrieved document IDs, ranking scores, and latency. Use dashboards to visualize recall, accuracy, and latency percentiles. Before rolling out a new index, use canary or shadow testing and control exposure with AgentControl targeting.

Governance and Safety in AI Experimentation

Governance and safety keep AI experimentation trustworthy. The goal is to find measurable improvement while protecting users, respecting constraints, and keeping experiments reproducible.

Security and Access Control

Before any experiment touches real data or users, define who can change what and how. Limit who can modify prompts, deploy models, access production logs, or adjust rollout rules. Use separate environments for development, staging, and production. LaunchDarkly supports these practices through role-based access control, approvals, audit logs, and environments.

Safety Guardrails

Set hard limits that experiments cannot violate. Use content filters, rate limits, token budgets, circuit breakers, and quality thresholds. Define rollback conditions for error rates, latency spikes, toxicity, unsupported claims, or cost overruns. Release policies and guarded rollouts help standardize these controls.

Reproducibility and Compliance

Strong governance means being able to prove what happened. Fix random seeds or sampling settings where supported. Version dataset snapshots, model IDs, prompt templates, guardrails, and configuration files. Store experiment plans, analysis rules, inputs, outputs, parameters, and results. LaunchDarkly config management and config version comparison help preserve reproducibility.

Rollback and Kill Switches

No matter how careful you are, things can go wrong. Keep the previous version ready. Test rollback procedures regularly. Use kill switches that can immediately halt an experiment if safety or quality issues emerge. LaunchDarkly feature flags, guarded rollouts, and guarded rollout management support fast mitigation.

Ongoing Monitoring

Governance does not stop at launch. Continue tracking model performance, user behavior, data distributions, latency, cost, and safety. Periodically rerun safety and quality checks as the system evolves. Maintain a documented process for investigating failures, notifying stakeholders, and implementing fixes.

How LaunchDarkly Helps With AI Experimentation

Where many AI tools stop at evaluation, LaunchDarkly helps enable production experimentation with traffic allocation, metrics, statistical comparison, and controlled release workflows. AI experimentation needs an operational layer that manages prompts, models, parameters, cohorts, traffic allocation, and rollouts safely. Building that layer yourself can quickly become complex.

LaunchDarkly provides:

Instant updates without deployments: Change prompts, swap models, or adjust parameters through AgentControl without redeploying application code.
Safe, gradual rollouts: Test new models on a small percentage of users with percentage rollouts, progressive rollouts, and guarded rollouts.
Centralized control with governance: Use approvals, audit logs, and role-based access control.
Built-in experimentation framework: Run experiments and AgentControl experiments comparing models, prompts, or parameters.
Separation of concerns: Developers can focus on building features while cross-functional teams safely participate in experimentation workflows through controlled configuration changes.

LaunchDarkly feature flags and AgentControl let you treat AI components as dynamic configurations rather than static code, giving you the speed and safety needed for continuous experimentation at scale. The following example shows how to switch between two model configurations with AgentControl.

Example: Switching Between AI Model Variations With AgentControl

In the LaunchDarkly dashboard, open AI, select AgentControl, create a config for the AI workflow, and define variations for each model you want to compare. For implementation details, start with the AgentControl quickstart, then review Create configs, Create and manage config variations, Config targeting, and the Python AI SDK reference.

After setting up config variations, use targeting to control which model variation is served and define a safe default.

Note: This example is simplified for illustration. Production implementations should externalize secrets, define explicit fallbacks, enforce timeouts, and include error handling and guardrails.

Install Dependencies

# Install required dependencies.
# In a notebook, you can run these commands with a leading !.
# In a terminal, run them without the leading !.

!pip install launchdarkly-server-sdk
!pip install launchdarkly-server-sdk-ai
!pip install openai

Import Dependencies

import os

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault
from openai import OpenAI

Set Up Clients

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
openai_api_key = os.getenv("OPENAI_API_KEY")

if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

if not openai_api_key:
    raise RuntimeError("Missing OPENAI_API_KEY")

ldclient.set_config(Config(ld_sdk_key))

ld_client = ldclient.get()
ai_client = LDAIClient(ld_client)
openai_client = OpenAI(api_key=openai_api_key)

Create Evaluation Contexts

# Context 1: control group user.
context_user_a = (
    Context.builder("user-alpha-001")
    .kind("user")
    .set("firstName", "Alice")
    .set("lastName", "Anderson")
    .set("email", "alice@example.com")
    .set("experimentGroup", "control")
    .build()
)

# Context 2: treatment group user.
context_user_b = (
    Context.builder("user-beta-002")
    .kind("user")
    .set("firstName", "Bob")
    .set("lastName", "Baker")
    .set("email", "bob@example.com")
    .set("experimentGroup", "treatment")
    .build()
)

Run the Same Query Against Two Config Variations

fallback_value = AICompletionConfigDefault(enabled=False)
user_query = "Write a detailed essay on NASA"


def run_configured_completion(context, label):
    config, tracker = ai_client.completion_config(
        "ai-experimentation",
        context,
        fallback_value,
        {"user_query": user_query}
    )

    if not config.enabled:
        raise RuntimeError(f"AI config is disabled for {label}")

    messages = [message.to_dict() for message in config.messages]
    messages.append({"role": "user", "content": user_query})

    completion = tracker.track_openai_metrics(
        lambda: openai_client.chat.completions.create(
            model=config.model.name,
            messages=messages,
            temperature=config.model.parameters.get("temperature", 0.7),
            max_tokens=config.model.parameters.get("maxTokens", 800)
        )
    )

    print(f"{label}")
    print("-" * 50)
    print(f"Model: {config.model.name}")
    print(f"Response:\n{completion.choices[0].message.content}")
    print()

    return config.model.name, completion.choices[0].message.content


model_a, response_a = run_configured_completion(context_user_a, "User A: Control")
model_b, response_b = run_configured_completion(context_user_b, "User B: Treatment")

print("=" * 50)
print("Comparison")
print("=" * 50)
print(f"User A got: {model_a}")
print(f"User B got: {model_b}")

Using the code above, one user can receive a baseline model variation while another receives an experimental model variation, without requiring a redeploy. This makes it easier to compare quality, latency, and cost under controlled conditions. The AI SDK can also report metrics to LaunchDarkly, which you can review in AgentControl monitoring and use in AgentControl experiments.

Final Thoughts

Experimentation should be part of everyday AI work: a habit, not a one-off project. Keep iterating, version your data, and let real numbers guide decisions instead of hunches. Treat every AI change like a hypothesis. Every hypothesis should map to a traffic allocation strategy, decision rule, and rollback condition.

Change one thing at a time. Start offline, move to shadow testing, then gradually expand through controlled rollouts while tracking quality, cost, latency, safety, and user outcomes. LaunchDarkly AgentControl, config variations, online evaluations, experiments, and guarded rollouts make this process practical by keeping prompts, models, parameters, metrics, and release controls versioned, targetable, measurable, and reversible.

MLOps Lifecycle: Stages, Workflow, and Best Practices

Scarlett Attensil — Tue, 02 Jun 2026 16:48:26 +0000

A machine learning model that performs well on day one will not remain stable by default. Performance can degrade over time due to data drift, changes in user behavior, evolving feature sets, or updates to upstream systems. These changes rarely cause immediate failure, but they reduce reliability and make model behavior harder to understand.

The core issue is not model quality, but a lack of coordination across the lifecycle. Decisions made early in the lifecycle affect every stage that follows. When stages operate in isolation, traceability breaks down. For example, code versioning may capture model changes, but not dataset lineage, feature definitions, or runtime behavior.

MLOps addresses this by treating machine learning as a continuous, end-to-end lifecycle. It connects data, features, training, deployment, monitoring, and governance into a single operating model. Each stage introduces its own assumptions and dependencies, from training and validation to deployment, monitoring, and governance.

Summary of key MLOps lifecycle concepts

Stage	Activities and Outputs
Data Ingestion and Labeling	Collect raw data (logs, databases, APIs, and sensors), annotate or label it if necessary, and clean it. The output will be versioned datasets or snapshots.
Feature Engineering	Take raw data and transform it into features (e.g., normalization, encoding, and aggregation) and register these features in a feature store.
Model Training and Experimentation	Perform training jobs and hyperparameter tuning. The output of this stage will be trained model artifacts like weights and checkpoints.
Validation and Testing	Test new models against holdout or test data. The output will be accuracy, loss, fairness metrics, and validation reports.
Packaging and CI/CD	Package the model into a deployable artifact or container and push it to a model registry or a container registry.
Deployment and Rollout	Deploy the model to production (REST endpoint, batch service, etc.). Manage traffic with canary releases and/or blue-green deployments. For LLM applications, Configs extends these capabilities to prompt versioning and model provider management.
Monitoring and Observability	Monitor system health: latency, error rates, etc. Monitor machine learning health, including elements like prediction quality and data drift.
Feedback and Retraining	Collect new labeled data and initiate the process of retraining the model. Schedule retraining runs using the newly collected data.
Governance and Approval	Conduct human-in-the-loop reviews and compliance checks before deploying the model. Maintain documentation of the models (e.g., model cards and data sheets), and implement automated policy checks.

The following diagram shows how the major MLOps lifecycle stages connect in practice, from data ingestion through deployment, monitoring, and retraining, along with the operational outputs produced at each step.

Data ingestion and preparation

Data as a first-class production artifact

Most ML systems do not make data ingestion a control boundary, instead treating it as a background process. Initially, everything looks good, but then issues creep in, such as missing columns, silent null propagation, schema changes, late arrival of upstream data, or unknown outliers. There is no catastrophic failure, just gradual degradation of model performance, making it hard to debug and identify exactly what original data was used.

Data ingestion should be a first-class citizen in the MLOps workflow. It is essential to establish reproducibility, compliance, and reliability for models. Determinism and measurable data quality should be achieved.

Ingestion as a control layer

Data ingestion must control the entry of all data being routed and validated. Data should be collected either in batches or streams before undergoing deterministic data cleansing transformations. Before any data is saved, schema requirements should be validated. In addition, each ingestion point should create a snapshot or version for future reference. Data lineage and quality metrics should be recorded at every stage along the processing route, so if validation fails, training on that data stops completely.

In MLOps, one key operational choice is whether a system should be fail-closed or fail-open. Fail-closed systems stop processing as soon as an anomaly is detected, maximizing safety. Fail-open systems continue processing with fallback logic, maximizing availability. The decision should depend on business risk, not the default implementation.

The pseudocode below shows a simplified ingestion control flow: load raw data, validate its schema, apply deterministic transformations, measure drift, and then store the resulting dataset version and metadata for downstream training.

raw_data = load_from_source(config["data"]["source"])

validate_schema(raw_data, config["data"]["schema"])

cleaned = apply_transformations(
    raw_data,
    null_strategy=config["data"]["null_handling"],
    outlier_strategy=config["data"]["outlier_policy"],
)

drift_score = compute_drift(cleaned)

if drift_score > config["data"]["drift_threshold"]:
    alert("Distribution shift detected")

dataset_version = snapshot_dataset(cleaned)
store_metadata(dataset_version, drift_score)

For high-risk ML workflows such as regulated decisions, fraud detection, or safety-sensitive systems, ingestion pipelines should usually fail closed. In lower-risk cases, teams may choose fail-open behavior with explicit fallback logic, but that should be a conscious business decision rather than an implicit default.

Deterministic validation signals

Deterministic validation means data checks that always produce the same pass/fail outcome for the same data based on predefined rules. If a required column disappears, a null rate exceeds an allowed threshold, or a distribution shift crosses a defined limit, the pipeline should respond predictably every time. These checks are often the first reliable sign of upstream data problems, such as schema changes, silent null propagation, or newly introduced categorical values.

In addition to checking whether columns exist, validating data effectively should include the following aspects:

Determining null counts and validating other attribute values
Validating that attribute values fall into the correct range
Limiting the number of categories available for categorical attributes
Measuring distributional shifts in an attribute through either a PSI or KS test
Measuring the number of duplicate records before any data goes into your model at all

Operational validation heuristics

In practice, ingestion validation is implemented as a set of operational heuristics that help teams interpret failures quickly. The signal itself matters, but so does what it usually implies operationally, because that determines whether the right response is to stop the pipeline, investigate upstream systems, or trigger a fallback path.

Signal	Interpretation
Missing required column	Usually indicates that an upstream schema or API contract changed and downstream transformations may no longer be valid
Null rate > threshold	Often suggests corrupted source records, partial extraction failures, or broken joins in the upstream pipeline
Distribution drift > threshold	May indicate a change in user behavior, source population, collection logic, or rollout conditions
High duplicate rate	Often points to replayed ingestion jobs, duplicate event delivery, or broken deduplication logic
Unseen categories	Can break encoders or produce invalid feature mappings if serving logic was built against a fixed category set

Data versioning and lineage

Having immutable dataset snapshots is critically important to ensure reproducible results. To allow reproducible training runs, each training run must reference the dataset version ID, schema hash, transformation configuration, and associated quality metrics. Without versioning, retraining becomes non-deterministic.

In regulated environments, ingestion needs to automatically enforce PII masking, field-level anonymization, and retention tagging. These controls should be enforced automatically as part of the ingestion pipeline rather than handled through ad hoc manual review because manual compliance steps are hard to audit and easy to bypass under delivery pressure.

Configuration and feature flag controls

In mature ML systems, ingestion rules should be controlled through external configuration rather than hard-coded into pipeline logic. This allows teams to adjust schema strictness, null-handling rules, drift thresholds, and anonymization behavior without redeploying the pipeline. The YAML below shows one way to define those ingestion policies declaratively.

data:
  source: "s3://raw/customer_data"
  schema: "schemas/customer_v3.yaml"
  null_handling: "impute_median"
  outlier_policy: "clip_99_percentile"
  drift_threshold: 0.1

validation:
  enforce_strict_schema: true
  max_null_rate: 0.05

Feature flags can control behaviors such as strict schema validation, drift blocking, and auto-anonymization. This enables the gradual introduction of more stringent validation, with instant rollback if the rules block production unexpectedly.

Ingestion-level operating metrics

The ingestion stage should expose a small set of operating metrics so teams can tell whether data is arriving on time, passing validation, and staying within expected quality bounds. These are stage-specific signals used to manage data intake, not a replacement for the broader production monitoring discussed later in the article.

Data intake needs to be measurable.

Key metrics:

Batch success rate
Ingestion latency
Drift score per batch
Null rate per critical feature
Rejected batch percentage
Schema violation count

Because ingestion is the first control boundary in the lifecycle, failures and drift detected here often surface before model-level symptoms appear in production. When ingestion is declarative, versioned, validated, and measurable, downstream training and deployment become far more reproducible.

Outputs

Versioned dataset snapshots
Validation reports and schema versions
Recorded data quality metrics
Metadata required for reproducibility

Feature engineering

Feature engineering is the lifecycle stage where raw, validated data is converted into the model inputs used during training and inference. In MLOps, this stage matters because feature definitions must remain consistent across offline training and online serving. If the transformation logic differs between those environments, the model may behave well in evaluation but degrade in production due to training-serving skew.

Defining the feature contract before transformation

With robust ML systems, feature definitions serve as the single source of truth; the transformation code simply implements them. The use of a feature-first approach helps make transformations deterministic, reducing the risk of training-serving skew. This consistency must extend across both offline feature stores used for training and backtesting, and online feature stores used for real-time inference. Aligning these environments helps prevent silent feature drift, invalid values, or data corruption in production.

Deterministic feature transformations

Feature transformations should be deterministic: The same input should produce the same output when the same feature definition and configuration are applied. This is what allows training, backtesting, and live inference to remain aligned. Tools such as Pandas, Spark, or feature platforms such as Feast can be used to implement that logic.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder, StandardScaler

# Example: Scaling numeric features
scaler = StandardScaler()
scaled_features = scaler.fit_transform(df[["age", "income"]])

# Example: Encoding categorical features
try:
    encoder = OneHotEncoder(sparse_output=False)
except TypeError:
    # scikit-learn < 1.2 uses the sparse parameter name.
    encoder = OneHotEncoder(sparse=False)

encoded_features = encoder.fit_transform(df[["gender", "region"]])

Unit tests and train-serving consistency

Unit tests help verify both transformation correctness and train-serving consistency. In practice, that means confirming that the same feature logic used during training is also used when live requests are processed in production.

import pandas as pd
from sklearn.preprocessing import StandardScaler


def test_feature_scaling():
    df_test = pd.DataFrame({"age": [20, 40]})
    scaler = StandardScaler().fit(df_test)
    transformed = scaler.transform(df_test)

    assert transformed[0][0] < transformed[1][0]  # Scaling check

Ensure that the same transformation logic is applied during both training and serving to prevent training-serving skew. Automate feature value validation before training, which can include range and null checks.

Monitoring feature distributions

Teams usually encode feature-level validation rules separately from transformation code so they can check whether important features remain within expected bounds over time. The example below shows a simple configuration for monitoring a few feature ranges.

feature_monitoring:
  features:
    - age
    - income
    - purchase_count
  validations:
    age: [0, 120]
    income: [0, 1000000]
    purchase_count: [0, 1000]

Feature registry and versioning

Store feature definitions and pipelines in a feature registry to ensure consistency.

{
  "feature_set": "customer_features",
  "version": "v1",
  "features": ["age", "income", "purchase_count"],
  "validation_status": "passed"
}

Use Git or a feature registry to track all changes. Versioned feature pipelines support reproducibility across both training and production.

Outputs

Feature transformation pipelines
Generated feature tables or vectors
Versioned feature definitions in a registry

Model training and experimentation

Once feature sets are available, the next stage is to train candidate models and record the context needed to reproduce and compare those runs later.

Careful automation of training and experiment tracking helps improve reproducibility, consistency, and the ability to compare different models with each other at different times.

Automating model training

Whenever possible, the training process should be automated. This includes scheduling regular training runs, running hyperparameter sweeps, and retraining models when new data becomes available. Automated pipelines save time and reduce human error, especially when managing multiple models or experimenting with different parameters.

Tracking experiments

Every model training run should be tracked to ensure reproducibility and facilitate later comparisons. This means logging the hyperparameters used, such as learning rate and number of trees, dataset snapshots, code versions, and training and validation metrics.

For example, this can be done using MLflow in Python:

import mlflow

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_param("num_trees", 100)

    # Training code goes here
    model.fit(X_train, y_train)

    # Log evaluation metrics
    accuracy = model.score(X_val, y_val)
    mlflow.log_metric("val_accuracy", accuracy)

    # Save the trained model
    mlflow.log_artifact("model.pkl")

This method tracks all of an experiment, and you can repeat the model or compare it with any other run later.

Controls and best practices

To prevent problems during training, configure an early stopping rule and define a limit for the total number of training epochs to avoid runaway training. You should also perform integration tests after loading the trained model using sample inputs. Each trained model should be saved as a versioned artifact in your chosen artifact service, such as S3 or the MLflow Model Registry. Finally, seed random number generators to ensure deterministic training and log the seed. These practices help maintain consistency, reproducibility, and reliability across training runs.

Outputs

Trained model artifacts (pickle, ONNX, TensorFlow SavedModel)
Training logs and experiment metadata
Hyperparameter and dataset configuration snapshots

Validation, testing, and evaluation

Model evaluation starts with offline assessments using a holdout test dataset. In this stage, model performance is measured using task-appropriate measures. For classification, those measures may include accuracy, precision, recall, F1 score, ROC curve, and confusion matrix. For regression, common measures include RMSE, MAE, or R-squared. It is also necessary to evaluate domain-specific business metrics, such as conversion lift, cost of errors, or revenue impact, to ensure the deployed model provides business value in addition to statistical performance.

While offline assessment provides important deployment guidance, automated checks against predetermined thresholds or recorded baselines should be part of gated validation before promotion. These checks should validate fairness or bias issues and use unit tests to confirm that known inputs return expected outputs. If a required threshold is violated, the pipeline should fail and prevent the model from being promoted to production.

To maintain reliability, automate checks that compare metrics against defined thresholds or baselines. For example, the pipeline should fail if a model's accuracy falls below the previous version. The pipeline should also fail if a fairness metric for a protected group is violated. Include unit tests to confirm that the model produces correct predictions on known inputs. Only models that pass all validation checks should advance to deployment.

Outputs

Validation reports and evaluation metrics
Metric visualizations (confusion matrices, ROC curves)
Automated test logs and validation summaries

Packaging and CI/CD

Once a machine learning model is validated, it should be packaged for deployment. This usually includes creating a container image, such as a Docker image, that includes the model and all code required to execute it. You can upload your model to a managed service like MLflow or Amazon S3.

Packaging is about more than reproducing something; it is also about controlling its promotion. When a model artifact has been validated, it should have proper versioning, registry storage, and associated promotion paths, such as staging and production, supported by defined approval and traceability workflows. The purpose of packaging is to ensure that the deployable unit is exactly the one that was validated, with its runtime dependencies, metadata, and configuration captured in a controlled and versioned form. Following a promotion path lowers release risk and makes rollback to a previous version easier if a problem occurs.

If you are using a continuous integration / continuous deployment (CI/CD) system like Jenkins, GitHub Actions, or Azure DevOps, deployment can usually be automated through the CI/CD pipeline. Typical steps involve retrieving the model from its storage location, building the Docker container image, running basic tests, and pushing the model image to a registry. Each image should contain a version tag that identifies which model version was deployed.

To maintain safety and reliability, the CI/CD pipeline should run automated checks, including code validation, test requests to the container, and Docker image security scans. Always use fixed version tags rather than latest to avoid accidental overwrites. If any test fails, the pipeline should stop immediately to prevent a faulty model from being deployed.

Proper packaging combined with automated CI/CD makes model deployment easier, safer, and more consistent.

Deployment and runtime controls

Deployment is the stage where a validated model is exposed to production traffic through an endpoint, batch workflow, or embedded application path. The operational goal is not just to make the model reachable but to release it in a way that limits user risk, supports rollback, and preserves observability during change.

One common runtime control is a feature flag, which is a configurable switch that changes application behavior without requiring a redeploy. In ML systems, feature flags can be used to route users between model versions, limit exposure to selected cohorts, or revert quickly to a known-safe model when problems appear. Tools such as LaunchDarkly provide this kind of runtime control.

Deployment strategies are designed to minimize exposure of new models, whereas guardrails are designed to minimize risk. You can also control which users see the new model by using feature flags in tools like LaunchDarkly. One way to implement feature flags is by wrapping your inference code with a toggle that allows you to use the new model or fall back to the old model:

from ldclient import Context

context = Context.builder(user_id).kind("user").build()

if ld_client.variation("new-model-enabled", context, False):
    prediction = new_model.predict(features)
else:
    prediction = old_model.predict(features)

This approach supports gradual rollouts; you can start by directing a small percentage of real traffic to the new model and increasing exposure only if metrics remain strong.

Always have a rollback plan. Monitor the canary release closely, and if errors rise or latency spikes, revert the feature flag and redeploy the previous model.

To maintain reliability, track latency and error rates for unusual patterns. Conduct integration tests in a staging environment before promoting a model to production. Log every deployment event, and to prevent user impact, trigger alerts or automated rollbacks if any service-level agreements (SLAs) are breached.

Outputs

Running model endpoints (Kubernetes deployments or cloud inference services)
Feature flag configurations controlling rollout
Traffic routing and rollout policies

Monitoring and observability

Unlike the ingestion-level operating metrics discussed earlier, this stage focuses on production-wide monitoring of the live ML system after deployment, including both infrastructure behavior and model behavior under real traffic.

Once a model is deployed in production, it is essential to continuously monitor both the system and the model, which allows for early detection of issues and ensures that the model continues to perform as expected.

Observing system and model metrics

Monitoring should include both infrastructure and model metrics.

Infrastructure metrics monitor the system's health and performance. Here are some examples.

Metric	Purpose
CPU and GPU usage	Ensure that compute resources are not overloaded
Memory consumption	Avoid memory bottlenecks that could slow down inference
Throughput	Track the number of requests the system handles per second
Latency	Monitor response times to maintain consistent performance

Model metrics track the model's performance in production.

Metric	Purpose
Prediction distributions	Detect unusual patterns or shifts in model outputs
Live accuracy	Measure accuracy on recently labeled data to catch performance drops
Error rates	Monitor mispredictions or failures to quickly identify anomalies

Comparing these metrics against training baselines helps you detect data drift. For example, changes in input feature distributions can be measured using KL divergence or the population stability index.

Concept drift should also be tracked; this occurs when a model's performance declines over time without code changes. Unexpected shifts in feature correlations or drops in model quality are strong indicators that something in the data or environment has changed.

Real-time dashboards and alerts

A key tool for monitoring is a real-time dashboard that displays prediction histograms, feature drift charts, and alert counts. Dashboards facilitate quick problem detection as issues arise, providing automated alerts when thresholds are exceeded and sending alerts through channels such as email or text for different severity levels. Minor drifts may generate a helpdesk ticket, while major anomalies may page the on-call technician.

Explainability and logging

For business-critical models, explainability tools can help users understand predictions and investigate why a model may be failing or drifting. All logs and metrics should be preserved and correlated, ideally within dashboards or monitoring systems, so that any issue can be quickly traced, diagnosed, and made actionable.

Feedback loop and retraining

A well-developed machine learning system continues to evolve after deployment. Production usage generates feedback in the form of new data, user corrections, and observed model performance, which can be used to retrain and improve the model over time. Examples include user corrections, newly added labeled examples for retraining, or additional incoming data generated through real usage.

There are numerous options for initiating retraining. Some teams use a scheduled approach, such as retraining every month. Others use automated triggers when data drift exceeds an established threshold or when model performance drops below acceptable levels.

Once retraining is triggered, the same data processing pathway used to develop the original model should be used with the newly input data: develop a new model, conduct validation, and deploy the new model to replace the original only if it passes all relevant validation. Before any model replacement, compare the new model and existing model using a common dataset.

All retraining activities should be carefully documented, including the dataset version, model configuration, and performance metrics. This helps ensure full traceability and reproducibility.

Controlled retraining workflow

Retraining should be triggered by explicit conditions, such as scheduled cadence, measured drift, or degraded production performance. Each run should record the dataset version, feature set version, model configuration, evaluation results, and release decision. Before fully switching to a new model, deploy it in shadow mode. In this setup, both the old and new models run side by side on the same inputs, and their outputs are compared without impacting real users. This helps identify unexpected differences early.

Business metrics should also be evaluated. For example, a small A/B test can confirm whether the new model improves conversion rates, reduces errors, or lowers operational costs. If the new model performs worse than the current one, immediately revert to the old model and investigate the issue. Deployment should not proceed if performance declines.

All retraining cycles should be recorded clearly with the following information: what changed, the reason for retraining, how improvements were measured, and who authorized the release. Maintaining this record makes audits easier and improves transparency.

Each retraining cycle produces important outputs, including updated training datasets, newly trained model artifacts, and retraining and evaluation reports. All of these artifacts should be securely stored and versioned so they can be reviewed, audited, or reproduced in the future.

Closed-loop learning

A well-integrated feedback loop links monitoring, validation, deployment, and retraining together. If a negative trend occurs or performance deviates from expectations, data retrieval can be triggered automatically. Once recent data has been processed, the updated model can replace the existing deployed model with confidence.

Output

Updated training datasets
Newly trained model artifacts
Retraining and evaluation reports

Governance and approval

When machine learning systems operate at scale, governance becomes essential. It is not enough for a model to function correctly from a technical standpoint; it must also be reviewed, documented, and formally approved before reaching users.

Strong governance frameworks establish a clear delineation of role expectations. For example, a data scientist may develop and train a model, while an ML engineer is responsible for deployment. A governance or compliance officer may check documentation and approve the release. After passing technical testing, models must complete formal review processes that include reviewing the model card, data documentation, bias analysis, and performance reports before receiving final approval.

Many organizations separate their environments into development, testing, and production. Models are promoted step by step, with each stage requiring sign-off from the appropriate team. This structured process helps ensure that no model reaches production without proper oversight and review.

Policy and compliance controls

Governance should not depend solely on manual reviews. Wherever possible, it should be reinforced through automation.

Policy as code involves defining governance rules directly in code. For example, the pipeline can automatically verify that the model card includes all required fields, performance metrics meet predefined thresholds, and bias evaluations have been completed.

The following YAML snippet defines policies for a model.

model_policy:
  required_model_card_fields:
    - model_owner
    - intended_use
  min_auc: 0.85
  max_bias_diff: 0.05

on_failure: block_promotion

If any of these requirements are not satisfied, the pipeline should fail automatically.

All approvals and deployments must be recorded in an audit log. Model artifacts should be securely stored and protected with signatures or checksums to prevent tampering. In regulated environments, compliance reviews must occur before a model is allowed to serve real users. Only models that pass every governance check should be permitted to reach end users.

Outputs

Model approval records
Audit logs of model releases
Governance and compliance reports

How LaunchDarkly supports the MLOps lifecycle

LaunchDarkly can act as a runtime control plane for MLOps, helping to enable safer releases, faster iteration, and measurable improvements in production. Its capabilities map directly to several lifecycle stages covered in this article.

A key practice in ML systems is the separation between deployment and release. With LaunchDarkly, teams can ship models or prompt changes behind feature flags and release them only when confidence is established. This means a new model version can be deployed to production infrastructure without any user seeing it until the flag is toggled on.

For safe model rollouts, LaunchDarkly supports progressive delivery and canary releases. It allows teams to expose a new model version to as little as 1% of traffic and scale up gradually to 100%. Rollouts can also be targeted to specific cohorts such as internal users, particular regions, or individual tenants, giving teams fine-grained control over who experiences the new behavior.

Feature flags enable dynamic control of ML functionality at runtime. A single flag can switch between Model A and Model B without redeployment, and it can provide the ability to revert to an earlier version when issues arise. Multivariate flags also allow teams to live-tune parameters such as confidence thresholds, temperature settings, top-p settings, and scoring cutoffs without changing code.

The ability to quickly roll back is crucial for minimizing risk when something goes wrong. LaunchDarkly includes kill switches, which stop access to a risky model or prompt immediately without redeploying. This capability can be important during time-critical incidents.

For online experimentation, LaunchDarkly supports A/B testing on real production traffic. Teams can compare model or prompt variants and measure their impact on quality metrics, latency, and cost before committing to a full rollout. The example below walks through this in detail.

For GenAI and LLM applications, LaunchDarkly offers Configs, which manage prompts, model selection, temperature, and other parameters as versioned configurations. Configs provide:

Prompt and model updates without redeployment
Built-in metrics tracking (tokens, latency, cost per variation)
Online Evaluations for automated quality scoring
Variable substitution for dynamic prompts ({{user_tier}}, {{[context](https://launchdarkly.com/docs/home/flags/contexts)}})

To learn more, read the AgentControl documentation. Prompt and model updates can be rolled out progressively and safely, just like any other feature change.

Guardrails and governance are also built in. Guarded rollouts automatically pause or roll back changes when monitored metrics regress. Approval workflows, role-based access control, and audit logging can support compliance and traceability practices often required by regulated environments.

Example: A/B testing ML models with LaunchDarkly

The following example demonstrates how LaunchDarkly feature flags can be used to A/B test two ML model versions during deployment. A string-type feature flag named model-version is created with two variations, model-a and model-b, and a 50%/50% rollout. Each incoming inference request is routed to one of two models based on the flag evaluation for that user.

In the LaunchDarkly dashboard, create a feature flag with the following configurations:

Name: model-version
Flag type: string
Variation 1: model-a (Logistic Regression)
Variation 2: model-b (Random Forest)
Default rule: 50%/50% rollout

Both models are trained on the same dataset (Iris), so the only variable in the A/B test is the model architecture. The code is shown below.

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data,
    iris.target,
    test_size=0.4,
    random_state=42,
)

# Model A: Logistic Regression (baseline)
model_a = LogisticRegression(max_iter=200, random_state=42)
model_a.fit(X_train, y_train)

# Model B: Random Forest (challenger)
model_b = RandomForestClassifier(n_estimators=50, random_state=42)
model_b.fit(X_train, y_train)

For each incoming request, the LaunchDarkly SDK evaluates the flag and returns the assigned variant. The application routes the request to the corresponding model, as shown below.

import ldclient
from ldclient import Context
from ldclient.config import Config

ldclient.set_config(Config("sdk-YOUR-KEY"))
client = ldclient.get()

# For each inference request
context = Context.builder(user_key).kind("user").build()
variant = client.variation("model-version", context, "model-a")

if variant == "model-b":
    prediction = model_b.predict(sample)
else:
    prediction = model_a.predict(sample)

After simulating 200 inference requests split across both variants, you can compare accuracy and latency to determine which model to promote.

After routing requests between variants, you need to aggregate the resulting outcomes so the two models can be compared on shared evaluation metrics such as accuracy and latency. To analyze A/B test results, navigate to the experiment's Results tab in LaunchDarkly. The Results tab provides:

Visualization options: Probability density, relative difference, and arm averages graphs
Statistical analysis: Probability to be best, expected loss, and confidence intervals
Filtering: Slice results by metric, variation, or user attributes
PDF export: Download results for stakeholder review

import numpy as np

summary = df.groupby("variant").agg(
    total_requests=("correct", "count"),
    correct_predictions=("correct", "sum"),
    accuracy=("correct", "mean"),
    avg_latency_ms=("latency_ms", "mean"),
    p95_latency_ms=("latency_ms", lambda x: np.percentile(x, 95)),
)

Based on the results, an automated decision-making process determines whether the challenger should replace the baseline. If Model B outperforms Model A by a defined threshold, the flag default rule is updated to serve Model B to all users. If it underperforms, traffic stays on Model A.

LaunchDarkly Guarded Rollouts automate this decision-making. Configure a metric threshold, such as accuracy must not regress by more than 1%, and LaunchDarkly automatically:

Pauses the rollout if the metric degrades
Rolls back to the baseline if the threshold is breached
Continues the progressive rollout if metrics remain healthy

No custom code is required for promotion or rollback logic.

To learn more, read the Guarded Rollouts documentation.

ACCURACY_THRESHOLD = 0.01  # Model B must beat Model A by at least 1%

acc_a = summary.loc["model-a", "accuracy"]
acc_b = summary.loc["model-b", "accuracy"]
lift = acc_b - acc_a

if lift >= ACCURACY_THRESHOLD:
    # Promote: update LaunchDarkly flag to serve model-b to 100%
    print("Promote Model B to full traffic.")
elif lift > -ACCURACY_THRESHOLD:
    print("No significant difference. Collect more data.")
else:
    # Rollback: keep model-a as default
    print("Keep Model A. Model B underperforms.")

Taken together, this example shows how runtime flags can separate model deployment from model release, support controlled experimentation on live traffic, and shorten rollback time when a challenger underperforms. In that sense, LaunchDarkly fits into the deployment and release-control layer of the broader MLOps lifecycle.

In a live environment, automatic rollback of flag changes is possible with LaunchDarkly Guarded Rollouts. This enables automatic rollback when one of the monitored metrics regresses.

Extending to LLM applications

The same progressive rollout principles apply to LLM applications, but with additional configuration dimensions. While traditional ML models require only version routing, LLMs need prompt management, temperature tuning, and provider selection. LaunchDarkly Configs can handle these requirements through percentage rollouts, instant rollback, and automated monitoring.

Best practices for managing the MLOps lifecycle

A mature MLOps lifecycle connects data ingestion, feature engineering, training, deployment, and monitoring into a continuous operational loop. The objective is to make machine learning systems reliable and repeatable rather than experimental.

Version and track everything: Data, code, models, and configurations should all be managed as versioned artifacts, enabling clear traceability and reproducibility of results.
Automate validation and testing: Data schemas, feature transformations, and model outputs should all be validated through automated checks. Any change in code or configuration should automatically trigger validation within the CI/CD pipeline.
Use feature flags and gradual rollouts: Teams should use feature flags and gradual rollouts to release new models incrementally rather than all at once. By toggling a new model behind a feature flag and gradually moving traffic over, you can monitor performance and make adjustments before fully replacing the previous version.
Implement continuous monitoring: Continuous monitoring of system performance is crucial. Track key metrics such as data drift, model accuracy, system health, and infrastructure in real time. Establish alerts so issues are identified and addressed early, preventing negative impact on users.

Build governance into the pipeline: Design the system with governance built in by adding approval workflows, documentation requirements, and audit logging directly into the pipeline. Include model cards and model lineage records so you have traceable evidence of the decisions made.

Conclusion

Structuring machine learning systems in this fashion can improve reliability, transparency, and compliance. Each step of a model's lifecycle yields clearly defined artifacts, and quality standards are enforced through automation via your pipelines. Over time, consistency between the original intent of the model, user data, and production performance creates a strong feedback loop.

AI Pipeline: Preventing Drift in Production Systems

Scarlett Attensil — Tue, 02 Jun 2026 16:32:09 +0000

A common failure pattern in a retrieval-augmented generation (RAG) system is a progressive decline in performance. This decline, which can be difficult for users to detect initially, often begins with a reduction in retrieval relevance. Over time, it may lead to longer response times and increasingly inaccurate, incomplete, or less helpful responses. This gradual degradation of the system's performance creates a challenging user experience.

Production failures often stem from uncoordinated changes, with operators adjusting retrieval settings, reranking methods, or model routing without a shared change process. Without explicit versioning and ownership, it becomes difficult to trace which change caused a regression or who made it.

This article argues that production AI pipelines, particularly RAG systems, must be designed around explicit control of change. The system must treat retrieval and prompting, evaluation, and model selection as controllable elements that people running the system must be able to modify through visible changes during active system use. The goal is not to introduce new techniques but to show how existing, well-understood methods can be composed into a production system that remains stable, measurable, and adaptable over time.

Summary of core AI pipeline design considerations

Stage	Core Focus	Why It Matters	Relevant AgentControl Config Features
Problem Definition	Defining use case, retrieval scope, and measurable KPIs	Unenforceable or missing baselines make it impossible to detect degradation later.	AgentControl config variations and tools for retrieval depth, reranker selection, and instant switching without redeployment
Knowledge Grounding and Retrieval	Chunking, embeddings, retrieval, GraphRAG, and reranking	Uncontrolled changes to retrieval parameters are a primary source of grounding failures.	AgentControl config variations for retrieval parameters (top-k, graph hops); model-specific index routing
Model Selection and Orchestration	Routing across embedding models, rerankers, and LLMs	Hard-coded models make every experiment a redeployment and every failure a production incident.	AgentControl config variations bundle model, prompt, and parameters atomically; percentage rollouts for A/B testing
Prompt Engineering and Configuration Management	Versioned, parameterized prompt templates	Ungoverned prompt changes are one of the fastest ways to break a functioning pipeline.	AgentControl config for prompt versioning with variable substitution and segment-based targeting
Evaluation and Guardrails	Grounding accuracy, safety filters, and gating logic	Changes without evaluation gates allow regressions to reach users undetected.	AgentControl Online Evaluations for accuracy, relevance, and toxicity scoring; guarded rollouts for automatic rollback
Experimentation and Feature Flags	Controlled variant testing under live traffic	Without bounded exposure, pipeline variables interact in ways that are difficult to diagnose or reverse.	AgentControl config variations with percentage rollouts; traffic allocation by segment or context
Deployment and Rollout	Separating code deployment from behavioral rollout	Releasing behavior changes to all users at once amplifies the impacted scope of any regression.	AgentControl config targeting with percentage rollouts; progressive exposure with instant rollback
Cost and Latency Optimization	Complexity-based routing, caching, and batching	Routing all requests through the highest-capability path increases cost without proportional quality gain.	AgentControl config targeting rules for model tier routing; context-based cost optimization
Monitoring and Observability	Tracking retrieval drift, grounding accuracy, and latency	Early detection of issues. RAG systems degrade gradually: Regressions often only become visible after affecting end users.	AgentControl monitoring for per-variation metrics
Feedback and Iteration	Structured collection of user signals and error traces	Continuous improvement loops. Ad hoc iteration based on intuition rather than signals leads to unpredictable system behavior.	AgentControl configs with monitoring signals

Note that “hit rate” refers to the proportion of user queries for which the retrieval layer successfully returns at least one relevant document that is subsequently used in the generated response.

In practice, this architecture benefits multiple roles across the AI team. Engineers can test retrieval and model changes safely under controlled exposure, product teams can iterate on prompts through versioned configurations, and operations teams gain faster response to production regressions through automated rollback and monitoring signals.

Note: For demo purposes, you can set up a reference implementation of a configuration-driven RAG pipeline in a single executable environment, while in practice, production usually operates with numerous services.

The following diagram shows how these stages from the above table connect in a production RAG pipeline, each independently configurable under live traffic conditions.

Changes such as modifying retrieval depth or enabling a reranker can be exposed to a subset of users, evaluated against grounding and latency thresholds, and automatically rolled back if regressions are detected, essentially keeping iteration safe without freezing the system.

Production reliability depends on three disciplines: explicit versioning of prompts and models, continuous evaluation signals, and enforced rollback logic. Together, these prevent uncontrolled drift while enabling safe iteration.

Problem definition: Making quality measurable before you optimize

The quality of a production RAG pipeline is strongly influenced by how clearly the problem is defined before implementation. The problem definition directly constrains downstream design choices such as retrieval scope, evaluation metrics, latency budgets, and acceptable trade-offs across the system.

Start by identifying the primary use case and pairing it with measurable KPIs: retrieval hit rate, reranker lift, citation accuracy, latency budgets, and hallucination or grounding error rates. These should be treated as configurable ranges rather than fixed standards, since acceptable thresholds differ across domains and use cases.

While initial requirements may live in planning tools like Jira or Confluence, AgentControl configs elevate key parameters to operational controls, making retrieval thresholds, quality gates, and rollback triggers runtime-configurable rather than static specifications. Unlike hardcoded thresholds buried in application code, AgentControl configs surface these parameters in a dashboard where they can be adjusted, monitored, and rolled back by anyone with access, not just engineers with deployment permissions.

For example, teams may externalize the thresholds for retrieval quality, reranker effectiveness, and latency as configuration flags that control enforcement and rollback. In Python, these can be combined as shown below:

# NOTE:
# Teams can externalize evaluation thresholds as runtime
# configuration so enforcement logic is adjustable without
# redeployment.

import logging
import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault

ldclient.set_config(Config("YOUR_SDK_KEY"))
ai_client = LDAIClient(ldclient.get())

context = (
    Context.builder("user-123")
    .kind("user")
    .set("environment", "production")
    .build()
)

fallback_value = AICompletionConfigDefault(enabled=False)

config, tracker = ai_client.completion_config(
    "rag-eval-config",
    context,
    fallback_value,
    {"query": user_query}
)

if config.enabled:
    custom = config.model._custom if hasattr(config.model, "_custom") else {}

    min_retrieval_hit = float(custom.get("min_retrieval_hit_rate", 0.82))
    min_reranker_lift = float(custom.get("min_reranker_lift", 0.12))
    max_latency_ms = int(custom.get("max_retrieval_latency_ms", 90))

    if measured_hit_rate < min_retrieval_hit or measured_latency > max_latency_ms:
        logging.warning("Threshold violation detected; using fallback path.")
        return fallback_response()

# Continue normal pipeline execution

In this model, thresholds are active policies integrated into evaluation gates and rollout controls. Changes can be exposed incrementally, measured against live metrics, and automatically rolled back when performance degrades. By treating configuration as an operational control surface rather than static settings, the pipeline remains adaptable without sacrificing production stability.

Knowledge grounding: Designing retrieval as a configurable system

In production RAG workflows, unregulated changes to retrieval methods and performance parameters—such as chunking strategies, embedding models, graph traversal depth, or reranker settings—often degrade retrieval quality. This degradation then propagates downstream, manifesting as grounding failures during generation. To minimize this risk, the retrieval layer should be designed as a thoroughly parameterized system, encompassing chunking methods (fixed or semantic), embedding model selection, retrieval depth (top-k), GraphRAG traversal depth, reranker configuration, and context window limits.

A layered retrieval approach might consist of vector-based retrieval of unstructured data, optional graph-based expansion for the improvement of relational context, and reranking for precision. A control-plane system governs the parameters exposed by each pipeline layer, making them observable, configurable, and safe to experiment with without modifying application code. In LaunchDarkly, AgentControl configs provide this control layer, storing retrieval configuration as versioned variations that can be tested incrementally and rolled back instantly. Retrieval quality remains adjustable at runtime, and the retrieval quality is not dependent on the speed of the assessment of the variants (e.g., chunk size or hop depth), since the assessment can be done with live traffic, provided that changes are gated and evaluated incrementally.

Parameters such as top-k, graph-hop depth, reranker toggles, embedding model selection, and fallback behavior are governed through configuration, enabling safe iteration without redeployment. AgentControl config targeting enables instant fallback by switching which variation is served with no redeployment required. If a new retrieval strategy degrades quality, revert to the baseline variation in seconds. This means existing vector stores such as Pinecone, Weaviate, FAISS for embeddings, and Neo4j for knowledge graphs are able to continue being used.

For instance, it is possible to construct a multi-layer pipeline: RAG on internal documents, GraphRAG via Neo4j for structured data, and a cross-encoder reranker. The transitions between these layers can be made configurable, allowing reranking to be enabled or disabled, embedding strategies to be adjusted, and routing logic to evolve while preserving production quality.

In practice, retrieval parameters can also be managed as part of a versioned AI configuration, allowing chunking, retrieval depth, graph expansion, reranking, and index selection to evolve together under controlled rollout.

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault

ldclient.set_config(Config("YOUR_SDK_KEY"))
ai_client = LDAIClient(ldclient.get())

context = (
    Context.builder("user-123")
    .kind("user")
    .set("tier", "premium")
    .set("environment", "production")
    .build()
)

fallback = AICompletionConfigDefault(enabled=False)

config, tracker = ai_client.completion_config(
    "rag-retrieval-config",
    context,
    fallback,
    {"query": user_query}
)

if config.enabled:
    custom = config.model._custom if hasattr(config.model, "_custom") else {}

    chunk_size = int(custom.get("chunk_size", 350))
    retrieval_top_k = int(custom.get("retrieval_top_k", 10))
    enable_graph_rag = bool(custom.get("enable_graph_rag", False))
    graph_hops = int(custom.get("graph_hops", 2))
    enable_reranker = bool(custom.get("enable_reranker", True))
    embedding_model = custom.get("embedding_model", "e5-base")
    reranker_model = custom.get("reranker_model", "cross-encoder")

    # Note: Switching embedding models requires separate precomputed indexes
    # per model. The configuration should control both the embedding model
    # and the index being queried.

    # Note: Increasing retrieval_top_k sends more retrieved content downstream,
    # which can increase token usage, cost, and context-window pressure.

    chunks = chunk(text, size=chunk_size)
    embeddings = embed(chunks, model=embedding_model)
    vec_results = vector_store.retrieve(embeddings, top_k=retrieval_top_k)

    graph_results = []
    if enable_graph_rag:
        graph_results = neo4j_query(hops=graph_hops, node_type="Document")

    # In production, graph expansion should apply relevance filtering
    # or weighted merging to avoid flooding the context window.
    results = merge(vec_results, graph_results)

    if enable_reranker:
        results = rerank(results, model=reranker_model)

# Apply context limits/fallbacks as needed
final_context = trim_to_context_budget(results)

In this model, retrieval behavior becomes a controlled surface rather than a static implementation detail. Teams can enable or disable GraphRAG, adjust traversal depth, swap embedding models, or toggle rerankers safely while monitoring grounding accuracy and latency. This configuration-driven approach keeps retrieval flexible without sacrificing production stability.

Model selection and orchestration

Hard-coding embedding models, rerankers, or LLMs directly into orchestration logic is a common anti-pattern in production AI systems that is easy to trace. The case becomes even more difficult if the embedding model, reranker, or chat model is hard-coded, for then every experiment becomes a redeployment. This will not only slow down the learning process but also increase the risk at the same time.

Model selection should be treated as a routing process rather than a one-time selection decision. AgentControl configs bundle model, prompt, temperature, and max_tokens as a single versioned configuration. When you switch variations, all parameters change atomically, reducing the risk of mismatched model/prompt combinations that can occur when using separate flags for each parameter. In practice, this means that each request is dynamically routed to a model variant based on configuration, traffic allocation, or runtime signals rather than binding the pipeline to a single hard-coded model. The pipeline is asking for “an embedding model” or “a chat model” all the time. Model selection is governed through configuration rather than hard-coded API calls.

In LaunchDarkly, AgentControl config bundles the model, prompt, temperature, and max_tokens as a single versioned variation. When a variation changes, these parameters update atomically, eliminating the risk of mismatched configurations and allowing traffic allocation and fallback behavior to be controlled safely. Fallbacks can be triggered by concrete conditions such as degradation in grounding accuracy, violations of latency budgets, elevated error rates, or failed evaluation checks, allowing the pipeline to revert to a known-stable model automatically.

The following simplified example illustrates how model routing can be externalized through configuration. Rather than binding the pipeline to a specific chat model, the active variant is selected at runtime based on a configuration flag, enabling controlled experimentation and safe fallback behavior.

# Illustrative example showing configuration-driven model routing
# using LaunchDarkly AgentControl config.

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault

ldclient.set_config(Config("YOUR_SDK_KEY"))
ai_client = LDAIClient(ldclient.get())

# Evaluation context used for targeting and experiments
context = (
    Context.builder("user-123")
    .set("environment", "production")
    .build()
)

fallback = AICompletionConfigDefault(enabled=False)

# Retrieve the full AI configuration (model, prompt, parameters)
config, tracker = ai_client.completion_config(
    "chat-config",
    context,
    fallback,
    {"context": retrieved_docs}
)

if config.enabled:
    completion = tracker.track_openai_metrics(
        lambda: openai_client.chat.completions.create(
            model=config.model.name,
            messages=[msg.to_dict() for msg in config.messages],
            temperature=config.model.parameters.get("temperature", 0.7),
            max_tokens=config.model.parameters.get("maxTokens", 4096)
        )
    )

AgentControl configs separate model selection from application code entirely. The pipeline requests a configuration, and AgentControl config returns the complete model setup based on targeting rules, enabling A/B tests, gradual rollouts, and instant rollback without code changes.

Controlled experiments can be conducted behind this one banner. For example, a Mistral or LLaMA-based deployment can be given just 5% of the total traffic while the baseline continues to be unaffected. Experimentation primitives such as traffic allocation, targeting, and instant kill switches support safer operation in production when combined with proper evaluation signals, monitoring, and rollback discipline, but they do not replace sound system design or operational oversight.

Prompt engineering and configuration management

Prompts are not constant resources; they move with changes in requirements, the evolution of data, and the appearance of edge cases. A lack of governance, coupled with changing prompts, is one of the quickest methods to cause the uprooting of a perfectly functioning pipeline.

AgentControl configs store prompts as versioned configurations in LaunchDarkly rather than in application code. Prompts support variable substitution such as {{context}} and {{user_tier}}, and the template structure, variable slots, and active prompt variants can all be versioned and selected at runtime. This allows teams to test prompt variants, compare outcomes, and restore previous versions when needed. The following simplified example shows how a prompt variant might be selected through configuration at runtime.

from ldai.client import LDAIClient, AICompletionConfigDefault

ai_client = LDAIClient(ldclient.get())
fallback = AICompletionConfigDefault(enabled=False)

config, tracker = ai_client.completion_config(
    "rag-assistant-config",
    context,
    fallback,
    {"context": grounded_context, "user_tier": "premium"}  # Variable substitution
)

# Prompt is stored in LaunchDarkly, not in code.
# Variables like {{context}} and {{user_tier}} are substituted automatically.
if config.enabled:
    messages = [msg.to_dict() for msg in config.messages]

This method allows for structured testing, selecting specific users to expose to the new feature, and quickly going back to the previous version, thus harmonizing prompt iteration with the deployment discipline already established for code.

Evaluation and guardrails

During evaluation, configuration values remain in effect, but the focus shifts from the configuration itself to measurable attributes of system behavior, such as grounding quality, latency, and safety-related metrics. Changes in retrieval, prompts, or models should be governed by both objective metrics and qualitative evaluation signals.

Objectively speaking, the correctness of grounding, the time taken, and the accuracy of citations are among the measures applied. Relevance and helpfulness are typically assessed through LLM-as-judge patterns, an approach popularized by tools such as OpenAI Evals and Patronus.

AgentControl includes built-in Online Evaluations that allow teams to attach judges for metrics such as accuracy, relevance, and toxicity to any variation. Sampling rates can be configured, and the resulting scores appear in the Monitoring dashboard alongside operational metrics such as latency and cost. These signals should be regarded as indicators of relative change rather than absolute truths. AgentControl displays them per variation, making it easy to compare whether a variant actually outperforms the baseline without building custom analytics. When used together through Guarded releases, they drive gating decisions automatically, pausing rollout exposure or triggering rollback when quality thresholds are violated without requiring manual intervention.

Safety evaluation typically focuses on detecting risks related to personally identifiable information (PII), toxicity, and compliance violations. Deterministic detectors such as Presidio are often used alongside probabilistic classifiers and cloud DLP services to reduce false negatives. In addition, evaluation systems can attach automated judges to monitor safety signals. For example, AgentControl Online Evaluations can apply toxicity judges to sampled responses and surface the results in monitoring dashboards.

The following simplified example illustrates how evaluation signals can be computed by the application and emitted as events to support configuration-driven gating decisions. In this pattern, scoring logic remains inside the application, while promotion or rollback behavior is governed through configurable rules.

def evaluate_and_gate(variant_id, output, context):
    scores = compute_eval_scores(output)
    pii_risk = run_pii_pipeline(output)

    # AI SDK provides automatic tracking of tokens, duration, and errors.
    completion = tracker.track_openai_metrics(
        lambda: openai_client.chat.completions.create(
            model=config.model.name,
            messages=[msg.to_dict() for msg in config.messages]
        )
    )

    # Optional: track additional custom metrics.
    ld_client.track(
        "rag_quality_metrics",
        context,
        data={
            "variant": variant_id,
            "grounding_accuracy": scores["grounding_accuracy"],
            "hallucination_rate": scores["hallucination_rate"],
            "latency_ms": scores["latency_ms"],
            "pii_risk": pii_risk.level,
        }
    )

    # Configuration-driven gating.
    if scores["grounding_accuracy"] < ld_client.variation(
        "min_grounding_accuracy",
        context,
        0.88
    ):
        if ld_client.variation("enable_rollback", context, True):
            return fallback_response(context)

    return output

Evaluation signals are computed by the application and emitted as events to support configuration-driven gating decisions. When grounding accuracy falls below the configured threshold, guarded rollouts automatically pause the variant and restore the baseline, without requiring manual intervention.

Experimentation and feature flags

As soon as evaluation and guardrails are implemented, experimentation ceases to be treated as such and is instead fully integrated within the system's daily cycle. The state of the pipeline at this stage is not “trying out methods and praying for the best” but an incessant, subtle, and well-managed learning process.

In practice, RAG-based experimentation is rarely isolated to a single variable. Adjustments in one area often influence others. For example, increasing retrieval depth changes the volume of context supplied to the model, graph traversal affects which documents are visible, rerankers modify relevance ordering, prompt changes alter tone and structure, and switching models impacts latency and cost. These dimensions interact, which makes controlled experimentation and careful gating essential.

AgentControl configs make these interactions explicit and controllable. Each variation represents a complete configuration—model, prompt, parameters, and tools—that can be tested against others under controlled traffic allocation, allowing multiple variables to evolve under bounded exposure. Traffic allocation and evaluation thresholds are managed through AgentControl configs, while Guardian-guarded rollouts enforce rollback conditions automatically when metrics indicate regression. The pipeline decides at runtime which choices to make instead of sending out a new deployment every time there is an idea to be tested. The code remains unchanged; only the behavior changes.

Each configuration change becomes a small, bounded experiment with a clearly defined blast radius and rollback path. Every meaningful decision in the pipeline is externalized to AgentControl configs. Multiple variations evolve safely under controlled exposure, with built-in metrics showing which performs better, no custom instrumentation required. From the application's perspective, this process is straightforward: On each request, it simply retrieves the active configuration and executes accordingly.

The following simplified example demonstrates how multiple pipeline parameters can be externalized as configuration variables. Rather than hard-coding retrieval depth, graph traversal limits, reranker activation, prompt versions, or model variants, these values are resolved at runtime, enabling controlled experimentation and gradual rollout. In this pattern, retrieval depth, graph expansion, reranking behavior, and model selection are resolved together as part of a versioned AI configuration rather than managed as unrelated flags.

from ldclient import Context
from ldclient.config import Config
import ldclient
from ldai.client import LDAIClient, AICompletionConfigDefault

ldclient.set_config(Config("YOUR_SDK_KEY"))
ai_client = LDAIClient(ldclient.get())

context = (
    Context.builder("user-123")
    .set("environment", "production")
    .build()
)

fallback = AICompletionConfigDefault(enabled=False)

config, tracker = ai_client.completion_config(
    "rag-pipeline-config",
    context,
    fallback,
    {"query": user_query}
)

if config.enabled:
    custom = config.model._custom if hasattr(config.model, "_custom") else {}
    top_k = int(custom.get("retrieval_top_k", 8))
    graph_hops = int(custom.get("graph_hops", 1))
    enable_reranker = bool(custom.get("enable_reranker", True))
    model_variant = config.model.name

In this pattern, experimentation occurs by adjusting configuration values and traffic allocation rather than modifying orchestration logic.

Key Insight: No redeployment is required to adjust retrieval depth, switch prompt variants, or test new models.

Configuration determines runtime behavior, while evaluation metrics determine whether those changes persist. For example, a config with two variations might include:

Variation A (Baseline): GPT-4o-mini, temperature 0.3, concise prompt
Variation B (Experimental): Claude 3 Haiku, temperature 0.5, detailed prompt with citations

Use percentage rollouts to send 10% of traffic to Variation B, then compare token cost, latency, and quality metrics in the LaunchDarkly dashboard before promoting.

The diagram below illustrates how an experiment progresses from limited exposure to promotion or rollback based on measurable thresholds.

At this stage, experimentation becomes a routine, low-risk operational activity rather than an ad hoc process with uncertain production impact. Variants are promoted only when metrics validate improvement; otherwise, rollback restores the baseline automatically. Decisions are governed by thresholds and enforced by configuration logic, not by informal coordination or manual caution.

In practice, this shifts how teams manage risk. Engineers can test ideas earlier and under real traffic, while product teams receive measurable feedback instead of speculation. When regressions occur, as they inevitably will, the system absorbs them predictably through rollback mechanisms rather than escalating into production incidents.

This is not experimentation for its own sake. It is controlled exposure, enforced by configuration and measurable thresholds rather than personal discipline alone.

Deployment and rollout

In AI pipelines, notably RAG systems, deployment and rollout are the key operational milestones that support controlled execution and minimize the risk of production disruptions. Deployment refers to changes in code or infrastructure. Rollout, by contrast, refers to the controlled exposure of new behavior in production, such as introducing new models or configuration variants gradually to reduce risk.

Separating deployment from rollout allows teams to validate behavior changes incrementally under real traffic conditions. Take, for instance, a new LLM version rollout: Start with a small percentage of traffic and track grounding correctness and latency. If metrics remain healthy, exposure can be increased. If issues emerge, rollback is immediate and configuration-driven.

LaunchDarkly controls these rollouts through AgentControl Configs percentage-based targeting and segment rules, with Guardian guarded rollouts that automatically pause or roll back based on quality signals. Exposure is increased only when metrics confirm the new variation is safe. Actual deployment is handled by infrastructure tools such as Kubernetes or Docker, while LaunchDarkly acts as the rollout control layer, managing fallback routes and ensuring configuration consistency across services. For example, retrieval config updates might be first shown to beta users and later on rolled out based on performance.

Here is a Python example that uses the LaunchDarkly SDK to manage the rollouts dynamically in your pipeline.

import os
import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AICompletionConfigDefault

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

ldclient.set_config(Config(ld_sdk_key))
ld_client = ldclient.get()
ai_client = LDAIClient(ld_client)

def handle_rag_request(user_context_data, query, variables):
    context = (
        Context.builder(user_context_data["user_id"])
        .set("segment", user_context_data.get("segment", "unknown"))
        .build()
    )

    fallback = AICompletionConfigDefault(enabled=False)

    config, tracker = ai_client.completion_config(
        "llm-rollout-config",
        context,
        fallback,
        variables
    )

    if config.enabled:
        # Model, prompt, and parameters are versioned together.
        response = tracker.track_openai_metrics(
            lambda: generate_response(config)
        )
        log_performance_metrics(response)
        return response

    return fallback_response(query)

# Close ld_client in your application's shutdown hook, not here.

This setup allows rollout percentages, targeting rules, and cohort segmentation to be adjusted directly through configuration. Exposure can be increased incrementally under live traffic, while evaluation metrics determine whether promotion continues or rollback is triggered.

Separating deployment from rollout ensures that behavioral changes are introduced gradually and reversibly, reducing production risk while maintaining iteration speed.

Cost and latency optimization

Complex queries usually call for the use of models of higher capability to satisfy the quality requirement, which usually means longer latency and more cost. On the other hand, many requests can be handled with lower-cost paths when quality signals remain within acceptable limits.

This is fundamentally about dynamic optimization. Possible strategies include routing queries based on complexity, caching frequently accessed results, adjusting retrieval depth, and batching requests where appropriate. These techniques aim to balance accuracy, latency, and cost rather than optimizing any one dimension in isolation.

In practice, routing policies and cost controls can be governed through AgentControl configs targeting rules. Requests can be routed based on attributes such as user tier, region, or query characteristics. For example, complex queries may be sent to a larger model such as GPT-4 while simple lookups are routed to a smaller model, and different retrieval or reranking strategies can be applied to different traffic segments, all without modifying application code.

A complexity classifier provides a practical example of cost-aware routing in production AI systems. Rather than routing all requests to the most expensive model and deepest retrieval path, the system evaluates incoming queries and dynamically selects an appropriate tier.

Routing decisions are not hard-coded. Instead, configuration flags determine tier selection and threshold limits. This allows routing behavior to evolve safely under live traffic without redeployment. In practice, complexity classifiers should be treated as heuristics and continuously calibrated using production metrics.

Below is an illustrative example of cost- and latency-aware routing controlled through configuration.

# NOTE:
# Illustrative example showing cost- and latency-aware routing via configuration.
# The complexity heuristic below is a placeholder and must be adapted
# to your domain, metrics, and production requirements.

import os
import ldclient
from ldclient import Context
from ldclient.config import Config

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

ldclient.set_config(Config(ld_sdk_key))
ld_client = ldclient.get()

def query_complexity(query):
    # Placeholder heuristic for illustration only.
    # Production systems usually consider richer signals such as:
    # - keyword density or semantic difficulty
    # - question structure and reasoning depth
    # - historical user interaction patterns
    # - observed quality or latency metrics
    # Any routing heuristic should be validated against production
    # evaluation metrics before broad rollout.
    return len(query) / 100.0

def handle_optimized_request(user_context_data, query):
    context = (
        Context.builder(user_context_data["user_id"])
        .set("region", user_context_data["region"])
        .build()
    )

    # Configuration-controlled routing decision.
    tier = ld_client.variation("llm_tier", context, "auto")

    if tier == "auto":
        complexity_threshold = ld_client.variation("complexity_threshold", context, 0.4)
        tier = "small" if query_complexity(query) < complexity_threshold else "large"

    # Select model based on resolved tier.
    model = "small_llm" if tier == "small" else "large_llm"

    response = llm_call(model, query)

    # Optional escalation path if quality signals fall below tolerance
    # (for example retrying with a larger model if grounding or confidence checks fail).

    log_optimization_metrics(response)

    return response

In many production deployments, routing decisions can also be managed directly through AgentControl config targeting rules. For example, different model tiers may be served based on user subscription level, geographic region, or environment without requiring custom routing logic in application code.

This pattern allows teams to adjust routing tiers, thresholds, and fallback behavior dynamically. Lightweight requests can be served at lower cost and latency, while complex queries are automatically escalated to higher-capability paths. Performance and quality remain observable and controllable through configuration.

The diagram below illustrates how routing decisions move between lightweight and high-capability paths based on configuration and runtime signals.

The lightweight path (small model with shallow retrieval) and the high-capability path (large model with deep retrieval) converge before caching, batching, and final response generation. This ensures that cost optimization does not fragment the delivery pipeline and that observability remains consistent across tiers.

Monitoring and observability

In every production RAG pipeline, performance monitoring is a must-do process that helps you discover issues before they get out of control. To do this, it is necessary to monitor the most important metrics, which include retrieval drift, grounding accuracy, hallucination rates, and related reliability signals. Latency spikes and the overall health of the API will be monitored as well. In practice, these metrics should be tracked over time and analyzed across percentiles (e.g., p95, p99) rather than relying solely on averages.

These observability signals will then be incorporated into your control plane, for instance, LaunchDarkly, to automate actions such as rollbacks or switching to safe-mode configurations whenever things go wrong. Observability provides the signals; AgentControl configs enforce the decisions. When grounding accuracy drops below the threshold, AgentControl configs can automatically shift traffic back to the stable variation without waiting for manual intervention.

Why invest in continuous monitoring? Because RAG systems are inherently dynamic: models evolve, underlying data shifts, and user behavior changes across contexts. These factors can gradually degrade performance in ways that are not immediately visible. Continuous monitoring is necessary because regressions often become apparent only after affecting end users. Early detection of hallucinations or accuracy drops allows issues to be addressed proactively, improving reliability without constant manual intervention.

Telemetry data can be collected through vendor-neutral observability frameworks such as OpenTelemetry or through an organization's internal monitoring infrastructure. AgentControl configs also provide a built-in monitoring dashboard that surfaces variation-level metrics automatically, allowing teams to compare model and prompt performance across experiments without additional instrumentation. The dashboard reports metrics such as token usage, cost, latency, and quality scores for each configuration variant. These metrics can then be used by rollout controls to suspend, promote, or roll back configurations when thresholds fall outside acceptable ranges.

For example, if the performance of the grounding deteriorates, LaunchDarkly will immediately unmask the experimental model and pull the reranker back to reliable settings, depending solely on the live data. The intervention should be driven by predefined thresholds and rules defined per metric rather than by a single global criterion, and not by unpredictable human interference.

Shown below is a Python snippet that demonstrates feeding metrics into LaunchDarkly for decision-making purposes.

# NOTE:
# Illustrative example showing how observability signals can be
# used to drive configuration-based rollback decisions.
# Telemetry is collected by external monitoring systems
# and evaluated against configuration thresholds.

import os
import ldclient
from ldclient import Context
from ldclient.config import Config

def get_current_metrics():
    # Placeholder for real telemetry collected through
    # observability systems such as OpenTelemetry.
    return {"grounding_accuracy": 0.85, "hallucination_rate": 0.05}

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

ldclient.set_config(Config(ld_sdk_key))
ld_client = ldclient.get()

def monitor_and_adjust(context_key):
    context = Context.builder(context_key).build()
    metrics = get_current_metrics()

    accuracy_threshold = ld_client.variation("accuracy_threshold", context, 0.90)

    if metrics["grounding_accuracy"] < accuracy_threshold:
        # Configuration-driven rollback decision.
        if ld_client.variation("enable_rollback", context, True):
            revert_to_stable_config()

    # Metrics are typically emitted to observability systems
    # and evaluated alongside AgentControl monitoring dashboards.

When AI requests are executed through the AI SDK, operational metrics such as token usage, cost, latency, and success rates are captured automatically using the track_openai_metrics() instrumentation. These signals appear in the AgentControl monitoring dashboard alongside evaluation scores and configuration variations, enabling teams to compare model and prompt performance without building custom analytics pipelines.

Using observability data to inform LaunchDarkly controls makes your pipeline adaptive. Teams get traceability, faster incident response, and safer production iteration by clearly separating metric computation from configuration enforcement.

To help you visualize, here's a flowchart of the observability pipeline.

Observability metrics feed directly into configuration-driven gating rules, enabling automatic rollback or promotion without manual intervention. The control plane does not compute metrics itself; it enforces decisions based on thresholds defined in configuration.

Feedback and iteration

Closing the RAG pipeline loop involves activating structured user feedback, such as ratings, error logs, and usage traces, and feeding these signals back into retrieval, prompt, or model adjustments. AgentControl configs then control the release of refined configurations. New prompt variations can be tested on a small percentage of traffic, evaluated against satisfaction metrics, and promoted only when signals confirm improvement, ensuring that they are tested under limited exposure before broader rollout. This promotes a never-ending process of evolution with closely-knit feedback loops, allowing you to quickly iterate while still having a solid production environment, without requiring broad, unmanaged production changes.

In practice, these feedback signals can be sourced from RLHF-derived signals, user feedback systems, satisfaction metrics, or built-in telemetry. LaunchDarkly coordinates the rollout of the updates, managing the exposure or reversions depending on the feedback received.

Feedback signals such as lower satisfaction scores, increased clarification requests, or higher error rates indicate that users struggle with the new prompt format. The new prompt variant can be kept in evaluation-only mode, allowing time to refine and retest through online evaluations, automated or semi-automated assessments run on live or shadow traffic, before any wider rollout.

Here's a Python snippet to integrate feedback signals with LaunchDarkly for adaptive rollouts.

# NOTE:
# Illustrative example showing how aggregated feedback signals
# can influence configuration-driven rollout decisions.
# Feedback signals are computed by the application and evaluated
# against thresholds managed through configuration.

import os
import ldclient
from ldclient import Context
from ldclient.config import Config

def get_user_feedback():
    # Placeholder for aggregated feedback signals.
    return {"satisfaction_score": 0.75, "error_rate": 0.10}

ld_sdk_key = os.getenv("LAUNCHDARKLY_SDK_KEY")
if not ld_sdk_key:
    raise RuntimeError("Missing LAUNCHDARKLY_SDK_KEY")

ldclient.set_config(Config(ld_sdk_key))
ld_client = ldclient.get()

def process_feedback_and_iterate(context_key, tracker):
    context = Context.builder(context_key).build()
    feedback = get_user_feedback()

    satisfaction_threshold = ld_client.variation(
        "satisfaction_threshold",
        context,
        0.80
    )

    if feedback["satisfaction_score"] > satisfaction_threshold:
        # Promote variant only when feedback trends meet acceptance criteria.
        if ld_client.variation("promote_variant", context, True):
            rollout_updated_variant()
    else:
        # Keep variant in evaluation mode or rollback.
        revert_to_baseline()

    # Track user feedback with AI SDK.
    if feedback["satisfaction_score"] > satisfaction_threshold:
        tracker.track_feedback({"kind": "positive"})
    else:
        tracker.track_feedback({"kind": "negative"})

The method used here makes the iterations both data-driven and lower-risk. Teams blend the feedback with the same setup and launch discipline as other pipeline modifications, thus preventing ad hoc decision-making and ensuring that the system behaves predictably even while it is being continuously evolved.

Last thoughts

Production AI systems rarely fail because a model is imperfect; they fail because change is unmanaged. In RAG pipelines, even small adjustments to retrieval depth, reranking logic, prompt structure, or model routing can compound quickly under real traffic. When those decisions are embedded directly in code, iteration becomes slow, risky, and difficult to reverse.

The goal is not to freeze behavior but to externalize it. AgentControl configs provide that external control surface: versioned configurations, percentage rollouts, automatic metrics, and instant rollback. When configuration, experimentation, evaluation, and rollout are treated as first-class architectural concerns, change becomes measurable and reversible. Teams can introduce improvements incrementally, observe their impact under real conditions, and roll back regressions without destabilizing the system.

Unlike general-purpose feature flag workflows, AgentControl configs are designed specifically for runtime AI configuration, combining prompt versioning, automatic metrics tracking, and online evaluations in a single control surface. Unlike broader MLOps platforms, it focuses on operational behavior in production rather than training pipeline management.

The more dynamic and compositional AI systems become, the more valuable controlled change becomes. Production RAG is not about finding a perfect configuration. It is about building a system that can evolve safely, intentionally, and continuously. AgentControl configs make this practical, giving teams a single place to manage model selection, prompt engineering, and quality evaluation, with the safety nets needed for production AI systems. Get started with the AgentControl Quickstart or explore the Python AI SDK for implementation examples.

The standard advice on handling bias in tech is wrong

Scarlett Attensil — Tue, 19 May 2026 18:19:19 +0000

Every woman who has worked in tech long enough has gotten the same three pieces of advice. Confront the behavior in the moment. Build an allyship network. Escalate to HR if it crosses a line.

I have tried all three. The first one cost me a working relationship I needed. The second one created a debt I never asked for. The third one put my name on a list and changed nothing about the person who did the thing.

This post is about what I do instead.

To be clear about scope: I am a Senior Developer Educator who has worked across data science, AI/ML, and developer education at multiple companies. My experience covers technical IC work and content-focused roles. If you are a manager of a 200-person org, the calculus may be different. If you are early career and still building a reputation, parts of this will not apply yet.

Here is what I have learned.

Confronting in the moment makes you the problem

The advice goes like this. Someone interrupts you, takes credit for your work, or makes a comment that lands wrong. You name it on the spot. "I was still speaking." "That was my idea from the sync last Tuesday." Direct, calm, professional.

The intent is good. The math does not work.

Confrontation in the moment costs you the room. Whatever you were trying to accomplish in that meeting is now derailed. You become the person who made the meeting awkward. The other party almost never says "you are right, I apologize." They explain, they soften, they reframe. The meeting moves on. You have spent your credibility to teach a lesson nobody asked to learn.

Worse, it lodges in the calibration discussion. Six months later, when someone is evaluating you for a stretch project, the data point that surfaces is "she can be difficult in meetings." Not the original behavior. Your response to it.

I am not saying never push back. I am saying the price is real and it falls on you, not on the person who caused the problem. Spend that currency on the things that matter. The interruption in a five-person room where the work product is at stake is worth a response. The interruption in a 30-person all-hands is not.

Allyship asks create the wrong dependency

The advice here goes like this. Find a senior man who will amplify your voice in meetings, vouch for your ideas, interrupt the interrupter on your behalf. Allies are powerful. Find them.

I have had good allies. They have made real differences. The problem is the structure.

When your visibility in a room depends on a man being in the room and choosing to use his voice for you, you have outsourced your authority. The ally goes on vacation. The ally moves to a different team. The ally has his own promotion cycle and is suddenly less available. Your visibility goes with him.

There is also a quieter cost. Performative allyship is its own ecosystem. Some allies advocate in ways that benefit their own brand more than they shift the dynamic. They make a visible show of speaking up, the room notices the ally, and you become a supporting character in someone else's narrative arc. Your idea becomes the thing the ally championed.

The version of this that works is mutual. You have a peer or near-peer who knows your work cold, and you do the same for them, and you happen to mention each other's contributions because you have actually been in the trenches together. That is not allyship as it is sold. That is normal professional credit, exchanged between people who respect each other.

HR works for the company

This one is the hardest to say out loud, because it sounds cynical and people want to believe HR is on their side.

HR exists to manage legal and reputational risk for the company. When that risk aligns with protecting you, they will protect you. When it does not, they will manage you. Both things can be true of the same HR person on the same day.

When you escalate to HR, three things happen at once. You go on a list. The other party gets coached, usually lightly. Your manager finds out, even if you were told it would be confidential, because your manager has to be looped in for any next-round decisions that involve you. The dynamic with your manager is permanently different from that moment on. They are now managing both you and a known issue, and they are evaluating which one is easier to make go away.

This does not mean never go to HR. It means go to HR with clear eyes. Use it when the behavior is severe enough that you would leave over it anyway, or when you need a paper trail because you have already decided to sue. Do not use it as the first or second line of response. The cost-benefit only works at the extremes.

What I actually do

The replacement playbook is less dramatic. It is also more effective, at least for me.

Document everything for yourself

I keep a private file. Date, what happened, who was there, my response if any. I do not write it for HR or for a future case. I write it so that I can see patterns.

Gender dynamics are gaslighting-prone. The behavior is often plausibly deniable in any single instance. You start to wonder if you are imagining it. The file makes that question answerable. 12 interruptions across four meetings with the same person is a pattern. One is a moment.

The file also helps you decide. When I review it monthly, I notice which incidents still feel hot and which ones I have moved past. The ones still hot get attention. The ones I have moved past taught me they were not worth the calories.

Build influence outside the orbit

The most powerful move I know is to make your reputation depend on people who are not in the dynamic. Skip-levels. Adjacent teams. External community. Public work.

If your visibility comes from your manager and your immediate peers, then any bias in that small group has outsized weight. If your visibility also comes from a talk you gave at a conference, a tutorial that is the top result for a search query, a working relationship with a VP two levels up, that small group's opinion is one data point among many.

This is also the highest-value move you can make for your career independent of any bias. The bias problem and the career problem have the same solution.

Choose what to ignore on purpose

Most micro-incidents are not worth a response. Picking your battles is not surrender. It is portfolio management.

The principle I use: respond to things that have ongoing operational cost, ignore things that are purely social. Someone taking credit for your work in front of your skip-level has operational cost. Someone explaining your own field to you at a happy hour does not. The first shapes future decisions about your work. The second is a bad time you can walk away from.

The energy you save here is what you spend on the work that compounds. Shipping the thing. Writing the post. Giving the talk. The work is what creates room to maneuver.

Leave well, not loud

Sometimes the answer is to go.

When that is the answer, leaving well matters more than leaving loud. A loud exit feels righteous. It also makes every future hiring manager nervous. Leaving well means quiet, on your timeline, with the relationships you want to keep intact. Tell the people who matter to you that you are going. Do not tell the people who caused the problem anything they have not earned.

The best revenge, if you want to think of it that way, is to show up two years later with a better title at a better company and have your work cited in the room you left. The loud exit forecloses on that future. The quiet exit keeps it open.

The underlying trade

The standard advice positions the woman experiencing bias as the agent of correction. Confront him. Recruit allies for yourself. Escalate up the chain. All of it is work, all of it is risky, all of it is uncompensated.

The replacement is to stop trying to fix the dynamic and start building standing that makes the dynamic less relevant. None of this fixes the structural problem. The structural problem will outlast any individual woman's career strategy. What this does is keep you operating at full capacity inside an imperfect system, while the slow work of fixing the system happens at the pace it happens.

That is the trade I have made. Your trade may look different.

The second-order cost of a(nother) layoff year.

Scarlett Attensil — Mon, 18 May 2026 23:14:46 +0000

Layoff years compress organizations. Scopes overlap. Lanes that were clear during a growth year start running into each other. Most of the resulting friction works itself out. Some of it doesn't.

than 300 tech layoff events. Meta is reportedly preparing to cut around 8,000 employees, Oracle reportedly sent 6 a.m. layoff notices in a restructuring that could affect up to 30,000 roles, and Microsoft has launched what GeekWire reports is the first voluntary retirement program in its 51-year history. Coinbase, Block, Cisco, and others have framed recent cuts around AI, flatter teams, and reallocating resources toward higher-growth work.

A market this tight raises the stakes on every working relationship. In a year where leadership is openly asking who can do whose job, that overlap becomes a resource question. When one party works to resolve it collaboratively and the other works to win it, the cost lands on whoever is doing the accommodating.

The thing to know up front is that the asymmetry doesn't close by trying harder on your side. The other party isn't trying to find common ground. They're building a case. Engaging more just gives them better material.

What works is borrowing from infrastructure.

In systems work, there's a concept called blast radius. It describes how far damage spreads when something breaks. A blast radius of one means a single service goes down. A blast radius of everything means a misconfigured deploy takes the whole company offline. The job is to design things so failures stay small.

The same logic applies when someone in your org has set you in their sights. You aren't trying to repair anything. You're trying to keep the damage local.

Shrink the surface

In security, attack surface is the sum of every place a system can be reached, queried, or pulled into. The first move is to make yours smaller without making the change obvious.

This can mean cutting things you used to think were good for your career. Impromptu coffees. The drop-in DM that turns into a thirty-minute therapy session about middle management. Pulled-aside-after-the-meeting conversations. None of it should be bad on its own but all of it can be edited and reframed to fit their negative narrative.

Your early drafts need to stop going into shared docs. Personal stuff should stop showing up in team channels. The hobbies, the weekend plans, the half-joke about a manager. All of it is data, and the safer working assumption is that anything shared at work could end up in front of someone you haven't met.

Make everything written

This single move does more work than anything else on this list.

Slack threads. Doc comments. Ticket updates. Email when it has to be. Anything but the hallway conversation and the unscheduled call.

The first reason is the obvious one. If things escalate later, you'll need a record, and the record will not exist if you talked it out.

The second reason matters more. A lot of what gets said in person doesn't survive being typed. You can almost feel the person editing themselves once they know they're on the record.

When someone pushes for a quick call or wants to grab you for five minutes, redirect politely. "Happy to look once you send it over." "Can you drop the question in the ticket so I can give it real attention." Then go back to whatever you were doing.

This isn't being difficult. It's insisting on the baseline professional communication you'd ask of any other cross-team colleague. The fact that this one finds it inconvenient is sort of the point.

Side effect worth knowing: if they refuse to put anything in writing, that's information too. Anyone who only wants to engage off the record is telling you what kind of conversation they want to be having.

Stop arguing with the framing

Problematic colleagues like to offer you stories about who you are. Sometimes flattering, sometimes critical. The thread is that the framing always shrinks your scope, or anchors your work to theirs.

You don't have to argue with it. You also don't have to accept it. A useful script for these moments: acknowledge what you heard, restate what you're actually doing, move on.

"Got it. I'm still planning to ship the launch post next week. Happy to coordinate if there's overlap."

That's the whole thing. No defense. No correction. No explanation of why their version of your role is wrong. The argument isn't winnable anyway. They aren't actually offering a view of your job. They're offering a smaller version of you, and the only correct response to a smaller version of you is to keep being the size you are.

Protect what compounds

The real cost of one of these dynamics isn't the conflicts. It's what they do to your attention. The counter is to protect the parts of your work that compound.

This matters more right now than it did two years ago. Coinbase cut roughly 14 percent of its workforce and called the resulting team "lean, fast, and AI-native." Block cut nearly half its staff, and Jack Dorsey told the company that smaller, flatter teams were the future. When that's the prevailing logic across the industry, the people who survive are the ones whose work has obvious, legible, compounding leverage. You want to be expensive to lose, and you want the case for keeping you to be visible without anyone having to fight for you in a room you aren't in.

A useful weekly check: would the next thing you're shipping still matter if this person weren't in your org. When the answer is no for too many weeks in a row, the dynamic has captured you, and it's time to reset.

Keep the file

Somewhere on a personal machine, not anything synced to work, keep a plain text document with dates, what happened, who was there, what was said or written. No commentary. No theories. Just the receipts.

You'll likely never need it. You'll be glad you have it.

The file isn't a place for venting. Venting goes to friends, a therapist, or a journal you'd be okay losing. The file's job is to make a clear factual case if it ever has to be made to a manager or HR. Trying to reconstruct six months of incidents from memory while stressed and being asked pointed questions is a particular kind of awful. The file is insurance against that. It costs maybe four minutes a week.

Escalate carefully or not at all

Most of these dynamics don't need escalation. They burn out. The colleague gets bored, the team reorgs, or more likely, the company has bigger problems. The default move is to outlast the thing, not confront it.

The threshold for taking it up the chain is concrete harm. Credit getting taken in a way that hits a perf review. Material misrepresentation of you to leadership. Sustained interference with shipping.

When escalation does happen, lead with impact on the work. "This is blocking shipping X by Y" gets traction. "This is making me uncomfortable" doesn't, even though both can be true at the same time. Managers will move on the first. Many will quietly let the second die.

What to actually do this week

Three things, if any of this is hitting.

Pull the receipts on the last two months. Even if nothing is currently on fire. Especially if nothing is currently on fire, because the moment you'll wish you had this information is the moment you're least able to gather it.

Close one informal channel. The unscheduled call. The "can I grab you" Slack. The hallway side-meeting. Just one. The point isn't to wall off the relationship in a day. The point is to start changing the geometry.

And look at what's shipping next. If it's a thing decided in reaction to something happening internally, swap it. Ship the thing that grows your own visibility instead. The colleague has a finite amount of energy for this. So do you. Spend yours on the work that compounds.

There's no clean ending here. The breakthrough conversation doesn't happen. The colleague doesn't suddenly see what you've been seeing. What happens, with your attention back on the work, is that the work gets better. Doors open in rooms no one inside your team controls. Turns out, it was always running on access to you. By the time you notice it's gone quiet, you're somewhere else entirely.

That's the win.

Claude Code is the engine, Cursor is the cockpit

Scarlett Attensil — Sat, 16 May 2026 22:30:39 +0000

My daily workflow looks nothing like it did a year ago. A lot has landed in Claude Code recently. Skills replaced custom slash commands, subagents and plugins followed, and four days ago /goal shipped, which lets an agent run autonomously for hours or days against a completion condition. Cursor 3 came out with an agents-first interface and a cheap in-house model. MCP went from "interesting protocol" to how every tool plugs into every other tool.

The net effect is that I now run Claude Code as the primary engine and treat Cursor as the cockpit. Cursor is where I drop in when I want a GUI, design mode, or a real-time view of a diff landing. Most everything else, including the long-running stuff, lives in Claude Code.

Five patterns that stuck.

Pattern 1: Claude Code is the engine, Cursor is the cockpit

The biggest mindset shift was treating the two tools as different categories of thing rather than competitors.

Claude Code runs in the terminal with Opus 4.7 behind it, and it's built around the idea that you can describe an outcome and let it work. /goal and skills make it possible to run multi-hour tasks against verifiable end conditions, then codify the workflows you do over and over. That's the engine.

Cursor is where I go when I specifically need to see something. The new agents window is where I manage parallel work, but for the tight inline loops I still flip back to the editor window, where Tab predictions and Composer beat anything else. Design mode lives there too, which is where I end up when I'm chasing a UI tweak. With Composer 2 sitting at 50 cents per million input tokens, Cursor 3 also makes a cheap surface for parallel cloud agents on smaller tasks. The cloud agents are the ones I'll fire off before a meeting, expecting a PR to be waiting when I get back. That's the cockpit.

My rough heuristic now:

One-sentence tasks I can walk away from for an hour go to Claude Code, usually with /goal.
Anything I want to watch happen in my editor goes to Cursor.
Repeatable workflows I've done more than twice get codified as a Claude Code skill, so it doesn't matter which surface I call them from.

A bonus trick that applies in both tools: plan with a smart slow model, execute with a fast cheap one. In Cursor 3, plan in plan mode with Opus 4.7, then switch to Composer 2 Fast to build. In Claude Code, draft the plan with the default Opus model and hand the execution off via /goal so a smaller verifier model does the per-turn checking. Saves real money and almost never costs you quality.

Pattern 2: One conventions file, two readers

Both tools support project-level instruction files, and as of Cursor 3 they both support skills as well. Claude Code reads CLAUDE.md at the repo root plus nested ones for subdirectories, with skills in .claude/skills/. Cursor reads from .cursor/rules/ and pulls skills from .cursor/skills/. Same primitives, two surfaces.

The mistake I made early was treating these as separate. Different rules in each file, drift between them, and inevitably one tool would produce code that violated rules the other knew about.

The fix is to write conventions once in a canonical file, then have both CLAUDE.md and your top-level Cursor rule reference it. Mine live at docs/conventions.md. One source of truth, two readers.

What goes in there: naming conventions, error handling patterns, the testing approach you actually use, libraries you prefer alongside libraries you've banned, PR structure expectations. And the one most people forget, feature flag conventions. If your team uses LaunchDarkly, this is the place to write down which flag types map to which use cases, your naming conventions, and the standard rollout cadence. Both agents will then default to writing flag-gated code correctly without you reminding them every time.

You can use these files for tone, too. A line still earning its keep at the bottom of my CLAUDE.md:

When you finish a non-trivial task, summarize what you did in the voice of a tired senior engineer who has seen it all.

Commit the file. Your teammates' agents inherit your conventions, and your sense of humor, for free.

Pattern 3: `/goal` for anything bigger than one conversation

This is the new one. Claude Code shipped /goal on May 12, 2026. Codex shipped its equivalent the same week. If you're not using it yet, this is the upgrade with the biggest leverage in years.

The premise is that you give Claude Code a completion condition instead of a list of steps. After each turn, a small verifier model checks whether the condition holds. If it doesn't, Claude takes another turn. If it does, the goal clears and control comes back to you. In practice this means handing off tasks that would have eaten a whole afternoon of back-and-forth, and getting a PR when you wake up.

What makes a good goal:

One measurable end state. "All TypeScript errors resolved, all tests passing, PR opened against main" is a goal. "Make the auth flow better" is a wish.
A roadmap or PRD to drive against. The pattern working best for me right now is to hand it a docs/roadmap.md with a checklist of tasks, then write the goal as "every task on the roadmap is checked off and verified." The verifier has something concrete to check.
Explicit constraints for anything that must not change. /goal accepts up to 4,000 characters in the condition, so there's room to write "do not touch payments/" and "do not modify the public API surface."

A LaunchDarkly-shaped example that has earned its keep on my machine:

/goal Walk the release-checkout-redesign flag from 0% to 100% using
our standard rollout cadence in docs/conventions.md. After each ramp,
check error rate from the metrics MCP server. Stop and return to me
if any ramp shows error rate above baseline + 10%, or when the flag
is at 100% with three consecutive green ramps. Do not modify any code,
only the flag targeting.

That's a task I used to babysit for half a day. Now I run it while writing the next feature in Cursor.

Pattern 4: Skills for the workflows you do more than twice

The single most underused feature in Claude Code is skills. They replaced what used to be custom slash commands, they auto-invoke when the description matches, and they cost almost nothing in tokens until they actually load.

A skill is a folder at .claude/skills/<name>/ with a SKILL.md inside. The file has YAML frontmatter with a name and description, and a markdown body underneath. That's it. Supporting scripts, example templates, helper files are optional and all live in the same folder.

The mental flip that helped me: anything I've copy-pasted into Claude Code twice should be a skill. Not eventually. Today.

Skills that have earned their place in my repos:

pr-prep. Runs the linter, runs the test suite, drafts a PR description against the convention in CLAUDE.md, and stops short of pushing.
flag-rollout. The LaunchDarkly rollout playbook. Reads the cadence from docs/conventions.md, has access to the LaunchDarkly MCP server and the metrics MCP server, and walks a flag through staged ramps with verification between each. Same shape as the /goal example above, but invocable as /flag-rollout without retyping the condition.
test-writer. Writes tests strictly using the conventions in the repo's testing doc. Vitest for unit and integration, Playwright for E2E, no Jest, no surprise mocks.
project-context. A tiny per-repo skill that tells the agent which Jira board, which LaunchDarkly project, and which feature flag prefix this codebase maps to. Sounds trivial. It killed the entire class of "wait, which board are we looking at" questions every agent used to ask me.

Each one is maybe 30 lines of Markdown. The compounding effect shows up within a week.

One more distinction worth pinning down, because it tripped me up early. Rules versus skills. Rules are the what. They go into every conversation as ambient context. "We use TypeScript. Vitest for tests. No new ORMs without a discussion." Skills are the how. They're loaded on demand when the description matches. "Walk a LaunchDarkly flag through staged ramps with verification" is a skill. "Always wrap network calls in our retry helper" is a rule. Keep rules brief and ambient. Push the heavy procedural stuff into skills, where it costs nothing in tokens until the agent actually pulls it in.

A quick note on skills versus subagents, because the difference matters. A skill is inline. Its instructions load into the current conversation and shape how the main agent behaves. A subagent is a separate, isolated context that the main agent delegates to and gets a summary back from. Use skills for "do it this way." Use subagents for "go do this heavy thing and come back when you're done." Most people want a skill 90 percent of the time.

Pattern 5: MCP servers, configured once, available everywhere

MCP went from "interesting protocol" in 2025 to the standard way every tool talks to every other tool by May 2026. Both Claude Code and Cursor have first-class MCP support with effectively identical configuration semantics, so a server you set up in one is a five-minute job to mirror in the other.

The LaunchDarkly MCP server is a good example of why this matters. Same server. Same credentials. Same flag set. Accessible from either surface. In practice that means I can ask the agent to "wrap this checkout component in a flag called release-checkout-redesign, default off" while I'm in Cursor writing the component, then switch to Claude Code in the terminal and run the /flag-rollout skill against that same flag. One mental model across two tools.

A few things I've learned that aren't obvious from the docs:

Resist the urge to connect every server you've heard of. Both tools handle large MCP rosters better than they used to, but every active tool eats a slice of your context budget. Keep four to six servers active and rotate as projects change.
Configure the same set in both tools. It removes the "wait, which one has the GitHub MCP" friction and lets you move work between them without thinking.
MCP servers compose with skills. A skill can specify which MCP tools it's allowed to use via allowed-tools in the frontmatter. Scoping a skill to only the LaunchDarkly MCP server makes it physically incapable of touching anything else, which is much safer than relying on the model to behave.

Conclusion

The May 2026 stack, as I run it:

Claude Code as the engine, for /goal-driven long tasks, skills, and anything else that should run without supervision.
Cursor as the cockpit, for visual work, design mode, tight inline edit loops, and cheap parallel cloud agents.
A shared conventions file both tools read from, so they code the same way.
/goal for anything bigger than one conversation.
Skills for anything you've done more than twice.
MCP servers mirrored across both tools, kept to a tight set you actually use.

The shift this year is that the highest-leverage work isn't editor tricks anymore. It's setting up the agents to run without you and getting them to do the same thing the same way every time. The editor matters less every quarter. The skill library and the goals you can hand off are what compound.

If you've got skills or MCP setups I should steal, send them my way. You can find me on LinkedIn.

If You Can Survive a Toddler, You Can Ship LLMs in Production

Scarlett Attensil — Thu, 14 May 2026 17:43:09 +0000

A few years back I was running a time-series pipeline that scored incoming product reviews on a 1-10 scale. The scorer was an LLM. Reviews rolled in continuously, ratings flowed into a dashboard the product team checked every Monday morning. Everything ran clean for months. Then one Monday the chart had a step in it.

Reviews from the prior week averaged 6.4. The current week averaged 7.6. Same product. Same customers. The reviews themselves, when I went back to read them, looked indistinguishable from what we had been getting all year.

The model had changed. The provider had pushed a quiet update to the weights, and the LLM that gave us 6.4-equivalent scores last week was now giving 7.6-equivalent scores for the same content. Every historical comparison in that dashboard was silently invalid. The cleanup took a week. The harder conversation was about how much of our reporting had been real in the first place.

That kind of failure is the default behavior of LLMs in production. Trying to engineer it away with tighter parameters or pinned versions is a losing fight. The job is to design for it. I learned the lesson twice. Once from the reviews pipeline. Once from raising two kids.

What parents of small children already know about non-determinism

If you have lived through the toddler years, you have run this experiment a few hundred times without calling it one. The lunch you packed all last week, the one that came home empty every day, suddenly gets pushed off the table on Tuesday with full commitment. The bedtime story that worked for six straight nights stops working on the seventh. The nap routine the babysitter swore was solid breaks the moment you start calling it a "rule."

Experienced parents eventually stop trying to force determinism on the kid. Patterns and trends still matter. But you stop expecting any individual input to produce any individual output, and you build a system that absorbs the variance instead of fighting it. This is the same shift production AI engineers make, usually after their first calibration regression.

The LLM-as-judge can drift too

The reviews pipeline taught me that the judge can be the most fragile thing in the system. The model being evaluated can drift. The model doing the evaluating can drift too. Without something stable to anchor against, you cannot tell which one moved.

The pattern that works is a small held-out set of inputs with known, human-validated scores, and the habit of re-running it on a regular cadence. Call it the calibration set. 20 to 50 examples is plenty. You re-score the calibration set first. If the average jumps from 6.4 to 7.6 with no other changes, you know the judge moved, not the data. Without that anchor, the same diagnosis takes weeks of reading individual reviews and arguing about what changed.

This is where offline evaluations in AgentControl earn their keep. You upload your calibration set as a dataset, point a judge at it, and re-run on a cadence or before any variation change. The discipline I had to learn the hard way: keep the judge anchored, keep its inputs comparable, watch the distribution rather than any single response, becomes a property of the configuration instead of a script someone has to remember to run.

The parenting version is the pencil marks on a doorframe. The doorframe does not move. Every few months you put the kid against it, shoes off and back to the wall. If the line jumps three inches and you realize the kid is wearing sneakers, you take the shoes off and measure again before believing any of it. The doorframe is your held-out set. The shoes-off rule is the discipline that keeps re-runs comparable.

Temperature zero is a comfort blanket

Setting temperature to zero feels like it should make a model deterministic. Within a single model version it mostly does. The catch is that determinism within a version buys you nothing across versions.

My reviews pipeline had been running at temperature zero the whole time. The provider's swap underneath did not care. The judge changed, and greedy sampling kept producing the new, shifted scores with the same false confidence as before. Temperature zero compresses the variance you can see during testing, which makes you feel safer. It does nothing about the variance that actually breaks production. Design as if the model can produce a different valid output every time, because eventually it will.

Build the fallback before the happy path

The last move gets cut for time most often, which is why it matters. Before you ship the model that does the new thing well, ship the path that runs when the model does the new thing badly, slowly, or not at all.

For an LLM endpoint that usually looks like a cached response for known-bad inputs, a secondary model behind the primary that can take traffic when the primary fails an inline check, a circuit breaker that routes around the model entirely if error rates cross a threshold, and a logging path that captures failure cases instead of returning a stack trace to the user. None of this is exotic. All of it assumes the model will misbehave at some point and defines what good behavior looks like when it does.

The stronger version of the same idea is making the fallback adaptive instead of static. A static fallback still needs a human to notice something is wrong and pull the lever. An adaptive system watches the production signal itself and switches over without anyone in the loop. This is what configuration-driven LLM tooling is built for. With AgentControl by LaunchDarkly, model variations live as configuration rather than code, traffic shifts between them without a deploy, and a guarded rollout can tie an online evaluation score, or any AgentControl metric you care about, directly to whether a variation advances, pauses, or reverts. When the judge sees scores regress past a threshold, the rollout reverses itself. The fallback stops being a piece of code someone wrote in case of trouble. It becomes the architecture, watching itself.

Parents already operate this way. The grocery store meltdown will happen. The school will call at 11am about a "low-grade fever." You have a snack pre-stuffed in your bag and a backup babysitter on text. The fallback is the architecture. The happy path is the bonus.

What changed for me

After the reviews pipeline incident, my work changed. I started logging the model version returned in every API response. I built a calibration set. I stopped trusting any single eval run as a verdict. The boring fallback path now ships before the impressive demo path.

None of this is harder than the alternative. It is mostly a matter of accepting at the architecture stage what every parent of a small kid already knows. The interesting unit of measurement is the distribution, not the sample.`

Offline Evaluation of RAG-Grounded Answers in LaunchDarkly AI Configs

Scarlett Attensil — Thu, 16 Apr 2026 21:22:50 +0000

Overview

This tutorial shows you how to run an offline LLM evaluation on the RAG-grounded support agent you built in the Agent Graphs tutorial, using LaunchDarkly AI Configs, the Datasets feature, and built-in LLM-as-a-judge scoring. You'll build a RAG-grounded test dataset, run it through the Playground with a cross-family judge, and learn how to read each failing row as a dataset issue, an agent issue, or judge calibration noise.

Here's how it works. The LaunchDarkly Playground evaluates a single model call against a prompt and dataset you configure. By pre-computing your RAG retrieval offline and baking the chunks directly into each dataset row, you turn that call into a high-value generation test: the model in the Playground receives the same documentation context it would in production, so the eval measures how well your agent reasons over real grounded input.

What You'll Learn

Structure a RAG-grounded test dataset by pre-computing retrieval offline and bundling chunks into each row
Pick the right LLM judge for your agent's output shape (Accuracy for natural-language answers, Likeness for structured labels)
Avoid same-model bias by running the judge on a different model family than the agent
Diagnose failing rows as dataset issues, agent issues, or judge calibration noise

What this tutorial covers, and what it doesn't

Covers:

Generation quality over RAG context: does the model produce a correct answer when the right documentation is in the prompt?

Regression detection: catching unexpected score drops when you change a prompt or model

Variation selection: comparing candidate prompts and models before committing to a new AI Config variation

Does not cover:

Retrieval correctness. Whether your vector store is returning the best chunks is tested by your own RAG pipeline, outside LaunchDarkly.

End-to-end agent graph behavior. Tool execution, multi-turn conversations, handoffs, and multi-step routing require online evals against real production traffic.

Prerequisites

You've completed the Agent Graphs tutorial or have equivalent familiarity with LaunchDarkly AI Configs
You have the devrel-agents-tutorial repo cloned
You have API keys for two model providers, one for the agent under test and one for the judge (the examples use OpenAI and Anthropic)

Step 1: Get the Branch Running

About the branch and the Umbra knowledge base. The feature/offline-evals branch builds on the same Agent Graphs tutorial codebase and the routing, tool, and graph work done in earlier branches — none of that goes away. What this branch adds is a more realistic RAG assessment target: Umbra, a fictional serverless-functions product with an invented knowledge base (refund windows, deployment regions, function timeout limits, rate-limit tiers, and so on). Because Umbra doesn't exist outside this tutorial, the model under test has no pre-training knowledge to fall back on — a correct answer has to come from the retrieved chunks, which is the only way to honestly measure whether your RAG pipeline is doing its job. The branch also ships a pre-built RAG-grounded test dataset (datasets/answer-tests.csv) and a helper script that regenerates it from your vector store.

cd devrel-agents-tutorial
git checkout feature/offline-evals
cp .env.example .env
# Add LD_SDK_KEY, LD_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY to .env

uv sync
uv run python bootstrap/create_configs.py
uv run python initialize_embeddings.py

Start the API and UI in two terminals:

# Terminal 1
uv run uvicorn api.main:app --reload

# Terminal 2
uv run streamlit run ui/chat_interface.py

Open http://localhost:8501 and ask a question grounded in the Umbra docs (refund policy, deployment regions, function timeout). The agent pulls answers from the knowledge base.

Step 2: Understand the Test Dataset

Open datasets/answer-tests.csv. Every row has three fields:

input,expected_output,original_question
"Documentation context: --- We offer a 30-day refund policy for first-time subscribers... --- Annual subscriptions receive a prorated refund within... --- Question: What is the refund policy?","30-day refund policy for first-time subscribers who haven't deployed production traffic. Usage charges are non-refundable.","What is the refund policy?"

input bundles documentation chunks and the question into a single structured prompt, separated by --- dividers. The chunks were retrieved from your production vector store ahead of time by tools/build_rag_dataset.py, so the model in the Playground sees the same grounding the production agent would, even though the Playground never executes your retrieval tools.
expected_output is the correct answer, written by a human who read the source docs.
original_question is a plain-text copy of the question so you can scan the dataset without parsing the bundled prompt. No judge uses this field.

Regenerate the dataset when your knowledge base changes:

uv run python tools/build_rag_dataset.py

For the full reference on dataset format and limits, see Datasets for offline evaluations.

Step 3: Upload the Dataset

Use synthetic data only

Never upload real customer tickets, PII, secrets, or credentials. Replace anything sensitive with synthetic placeholders before upload. See the Playground privacy section for what gets forwarded to model providers.

Navigate to AI > Library in LaunchDarkly, select the Datasets tab, and click Upload dataset. Upload datasets/answer-tests.csv and name it answer-tests.

Step 4: Add Your Model API Keys

The Playground calls model providers directly, so it needs API keys for both the model running your agent and the model running your judge. These keys live in LaunchDarkly's "AI Config Test Run" integration, not in your AI Config.

In the Playground, click Manage API keys in the upper-right corner.
Click Add integration, pick a provider (e.g. OpenAI), paste your API key, accept the terms, and save.
Repeat for the second provider (Anthropic) so you can run a cross-family judge in Step 5.

See the Playground reference doc for the canonical instructions. API keys are stored per-session, so you may need to re-paste them when you return.

Step 5: Run the Evaluation

From the Datasets list, click into answer-tests to open it in a Playground bound to that dataset.

Configure the test

System prompt: paste your support-agent instructions verbatim from the AI Config. Do not edit or simplify them.
Agent model: pick the model your support-agent variation uses (or a candidate you're considering swapping to). To compare two candidates, run the eval twice with different agent models and compare scores.
Acceptance criteria: attach an Accuracy judge with threshold 0.85. Accuracy scores whether the response correctly addresses the input question, which fits grounded natural-language answers.
Evaluation model: uncheck Use same model for evaluation and set the judge to a different model family from the agent. Same-family judging tends to reward output patterns the judge itself produces. A cross-family judge gives you an independent read.

Run the eval.

Reading the results

The example run above had 18 passes and 2 failures. When a row fails, the failure comes from one of three places, and each one sends you in a different direction:

The dataset's chunks don't contain the answer. This is a retrieval problem, not a generation problem. Rebuild the dataset with higher top_k, a reranker, or a different chunker, or verify the answer is indexed at all.
The chunks contain the answer but the model ignored them. This is the agent-side failure offline evals are designed to catch. Tighten the system prompt to insist on grounding, or switch to a more obedient model.
The chunks and the model are both fine but the judge disagreed. This is judge calibration noise. Lower the threshold, try a different judge, or accept it as noise. Don't change your agent based on it.

Sort by score. For each failing row, open the bundled chunks in the input field and ask: was the right answer in there? Yes → fix the prompt or model. No → rebuild the dataset.

What failed in this run

Row 11: "What integrations are available?" (chunks missed the answer). The expected output mentioned monitoring integrations (Datadog, Sentry, LogRocket), but the retrieved chunks only covered databases, storage, and billing. The model correctly listed what it had and said "the documentation does not provide additional information regarding more integrations", which is the correct behavior for an ungrounded claim. Fix: higher top_k or a reranker in build_rag_dataset.py.

Row 12: "Can I get a refund on bandwidth overages?" (judge calibration). The model correctly said bandwidth overages are non-refundable, citing the docs, but omitted a secondary "Review your Usage Dashboard" recommendation from the expected output. Semantically right, lexically short one clause. Fix: lower the threshold or trim the expected output.

Two failures, two different fixes. Without reading the per-row results you'd conflate them and spend time tightening the model when the actual problem lives in the retriever or the dataset.

Where to Go From a Single Run

This tutorial walked you through one run. In practice, a single eval isn't where offline evaluation earns its keep. The real payoff comes from re-running the same dataset against a new prompt, a new model, or a fresh RAG chunker and comparing scores to your last known-good run. A small prompt edit that quietly drops your Accuracy from 0.83 to 0.71 is exactly the kind of regression this pattern is meant to catch, but only if you save the run and compare against it next time.

A reasonable next loop:

Save the run from Step 5 as your reference.
When you change something (prompt, model, chunker, top_k), re-run the same dataset and compare scores.
Add new rows to the dataset as you find failure modes in staging or production.

For end-to-end behavior that offline tests can't capture (tool execution, multi-turn conversations, the tail of real production inputs), see online evaluations and the When to add online evals tutorial. Online evaluations are not currently supported for agent-based AI Configs; for agent workflows, the documented path is programmatic judge evaluation via the AI SDK.

Step 7: Track Evaluation History

View saved runs at AI > Evaluations. Toggle Group by dataset to collapse runs under each dataset name so you can see the history for umbra-rag-eval alongside any other datasets in the project. Compare pass and fail counts across runs, and distinguish saved runs (indefinite retention) from one-off runs (60-day expiry). For metric definitions, see Monitor AI Configs.

What's Next

Progressive rollouts: release your winning variation to 5% of traffic, then 25%, then 100%, watching production metrics before expanding.
When to add online evals: decide what to score on live production traffic once you have an offline baseline.

For a deeper look at the multi-agent RAG system this tutorial builds on, see the Agent Graphs tutorial.

Building Framework-Agnostic AI Swarms: Compare LangGraph, Strands, and OpenAI Swarm

Scarlett Attensil — Thu, 26 Mar 2026 21:05:21 +0000

If you've ever run the same app in multiple environments, you know the pain of duplicated configuration. Agent swarms have the same problem: the moment you try multiple orchestrators (LangGraph, Strands, OpenAI Swarm), your agent definitions start living in different formats. Prompts drift. Model settings drift. A "small behavior tweak" turns into archaeology across repos.

AI behavior isn't code. Prompts aren't functions. They change too often, and too experimentally, to be hard-wired into orchestrator code. LaunchDarkly AI Configs lets you treat agent definitions like shared configuration instead. Define them once, store them centrally, and let any orchestrator fetch them. Update a prompt or model setting in the LaunchDarkly UI, and the new version rolls out without a redeploy.

Ready to build framework-agnostic AI swarms? Start your 14-day free trial of LaunchDarkly to follow along with this tutorial. No credit card required.

Start free trial →

The problem: Research gap analysis across multiple papers

When analyzing academic literature, researchers face a daunting task: reading dozens of papers to identify patterns, spot contradictions, and find unexplored opportunities. A single LLM call can summarize papers, but it produces a monolithic analysis you can't trace, refine, or trust for critical decisions.

The challenge compounds when you need to:

Identify methodological patterns across 12+ papers without missing subtle connections
Detect contradictory findings that might invalidate assumptions
Discover research gaps that represent genuine opportunities, not just oversight

This is where specialized agents excel - each focused on one aspect of the analysis, building on each other's work.

In this tutorial, we'll build a 3-agent research analysis swarm that solves this problem by dividing the work:

Agent	Role	Output
Approach Analyzer	Clusters methodological themes across papers	"Papers 1, 4, 7 use reinforcement learning; Papers 2, 5 use symbolic methods"
Contradiction Detector	Finds conflicting claims between papers	"Paper 3 claims X improves performance; Paper 8 shows X degrades it"
Gap Synthesizer	Identifies unexplored research directions	"No papers combine approach A with dataset B; potential opportunity"

We'll implement this swarm across three different orchestrators (LangGraph, Strands, and OpenAI Swarm), demonstrating how LaunchDarkly AI Configs enable:

Framework-agnostic agent definitions: Define agents once in LaunchDarkly, use them everywhere
Per-agent observability: Track tokens, latency, and costs for each agent individually - catch silent failures when agents skip execution
Dynamic swarm composition: Add/remove agents from the swarm or switch models without touching code

Why use a swarm?

Research gap analysis requires different skills: clustering methodological patterns, detecting contradictions, and synthesizing opportunities. With a swarm, each agent handles one aspect and produces artifacts the next agent builds on. You can track tokens, latency, and cost per agent. You can catch silent failures when an agent skips execution. And when something goes wrong, you know exactly where.

Technical requirements

Before implementing the swarm, ensure you have:

LaunchDarkly account with AI Configs enabled (see quickstart guide)
API keys for Anthropic Claude or OpenAI GPT-4 (check supported models)
Python 3.11+ for running orchestrators
Basic understanding of agent systems (review LangGraph agents tutorial if needed)

The complete implementation is available at GitHub - AI Orchestrators.

The architecture: how LaunchDarkly powers framework-agnostic swarms

The swarm architecture has three layers: dynamic agent configuration, per-agent tracking, and custom metrics for cost attribution. Here's how they work together.

The diagram shows LangGraph's implementation, but Strands and OpenAI Swarm follow the same pattern with their own handoff mechanisms. The key components are:

Configuration Fetch: The orchestrator queries LaunchDarkly's API to dynamically discover all agent configurations, avoiding hardcoded agent definitions
Agent Graph: Three specialized agents (Approach Analyzer, Contradiction Detector, Gap Synthesizer) connected through explicit handoff mechanisms
Metrics Collection: Each agent execution captures tokens, duration, and cost metrics through both the AI Config tracker and custom metrics API
Dual Dashboard Views: The same metrics appear in the AI Config Trends dashboard (for individual agent monitoring)

Three layers of framework-agnostic swarms

1. AI Config for Dynamic Agent Configuration

Each AI Config stores:

Agent key, display name, and model selection
System instructions and tool definitions

Your orchestrator code queries LaunchDarkly for "all enabled agent configs" and builds the swarm dynamically. No hardcoded agent names.

2. Per-Agent Tracking with AI SDK

LaunchDarkly's AI SDK provides tracking through config evaluations. You get a fresh tracker for each agent, then track tokens, duration, and success/failure. These metrics flow to the AI Config Monitoring dashboard automatically.

This tracking catches silent failures - when agents skip execution or produce minimal output. Step 4 shows the implementation patterns for each framework.

3. Custom Metrics for Cost Attribution

Per-agent tracking shows performance, but for cost comparisons across orchestrators you need custom metrics. These let you query by orchestrator, compare costs across frameworks, and identify anomalies.

With the architecture covered, let's build the swarm. We'll download research papers, set up the project, bootstrap agent configs in LaunchDarkly, implement per-agent tracking, and run the swarm across all three orchestrators.

Step 1: Download research papers

First, you need papers to analyze. The scripts/download_papers.py script queries ArXiv with narrow, category-specific searches to ensure focused results.

python scripts/download_papers.py

The script presents pre-configured narrow research topics:

# From orchestration/scripts/download_papers.py:164-189
topics = {
    "1": {
        "name": "Chain-of-thought prompting in LLMs",
        "query": "cat:cs.CL AND (chain-of-thought OR CoT) AND reasoning",
        "years": 2
    },
    "2": {
        "name": "Retrieval-augmented generation (RAG)",
        "query": "cat:cs.CL AND (retrieval-augmented OR RAG) AND generation",
        "years": 2
    },
    "3": {
        "name": "Emergent communication in multi-agent RL",
        "query": "cat:cs.MA AND (emergent communication OR language emergence)",
        "years": 5
    },
    "4": {
        "name": "Few-shot prompting for code generation",
        "query": "cat:cs.SE AND few-shot AND code generation",
        "years": 2
    },
    "5": {
        "name": "Vision-language model grounding",
        "query": "cat:cs.CV AND vision-language AND grounding",
        "years": 2
    }
}

These topics are intentionally narrow: Each uses ArXiv categories (cat:cs.CL, cat:cs.MA) to limit scope. Boolean AND operators ensure papers match all criteria. 2-5 year windows prevent overwhelming the analysis.

For even narrower custom queries, combine categories with specific techniques like cat:cs.CL AND chain-of-thought AND mathematical AND reasoning for CoT math only, cat:cs.MA AND emergent AND (referential OR compositional) for specific emergence types, or cat:cs.SE AND few-shot AND (Python OR JavaScript) AND test generation for language-specific code generation.

The script saves papers to data/gap_analysis_papers.json with this structure:

[
  {
    "id": "2409.02645v2",
    "title": "Emergent Language: A Survey and Taxonomy",
    "authors": "Jannik Peters, Constantin Waubert de Puiseau, ...",
    "published": "2024-09-04",
    "category": "cs.MA",
    "abstract": "The field of emergent language represents...",
    "introduction": "Language emergence has been explored...",
    "conclusion": "This paper provides a comprehensive review..."
  }
]

Why this format: Each paper includes ~2-3K characters of text (abstract + intro + conclusion), which is enough for analysis but won't overflow context windows. For 12 papers, you're looking at ~30K characters (~7.5K tokens) of input.

You now have 12 papers saved locally. Next, we'll configure LaunchDarkly credentials and install the orchestration frameworks.

Step 2: Set up your multi-orchestrator project

Environment setup

For help getting your SDK and API keys, see the API access tokens guide and SDK key management.

# .env file
LD_SDK_KEY=sdk-xxxxx       # Get from LaunchDarkly project settings
LD_API_KEY=api-xxxxx       # Create at Account settings → Authorization
LAUNCHDARKLY_PROJECT_KEY=orchestrator-agents

# Model API keys
ANTHROPIC_API_KEY=sk-ant-xxxxx
OPENAI_API_KEY=sk-xxxxx

Install dependencies

python -m venv .venv
source .venv/bin/activate

# LaunchDarkly SDKs - see [Python SDK docs](/sdk/server-side/python)
pip install ldai ldclient python-dotenv arxiv PyPDF2 requests

# Orchestration frameworks
pip install strands-sdk langgraph swarm

For more on the LaunchDarkly AI SDK, see the AI SDK documentation.

Your environment is configured and dependencies are installed. Next, we'll use the bootstrap script to automatically create all three agent configs in LaunchDarkly.

Step 3: Bootstrap agent configs with the manifest

The orchestration repo includes a complete bootstrap system that automatically creates all agent configurations, tools, and variations in LaunchDarkly. This is much faster and more reliable than manual setup.

Understanding the bootstrap system

The bootstrap process uses a YAML manifest to define:

Tools - Functions agents can call (fetch_paper_section, handoff_to_agent, etc.)
Agent Configs - Three specialized agents with their roles and instructions
Variations - Multiple model options (Anthropic Claude vs OpenAI GPT)
Targeting Rules - Which orchestrators get which models

Run the bootstrap script

# From the orchestration repo root
cd ai-orchestrators

# Run bootstrap with the research gap manifest
python scripts/launchdarkly/bootstrap.py

# You'll see:
╔═══════════════════════════════════════════════════════╗
║  AI Agent Orchestrator - LaunchDarkly Bootstrap       ║
╚═══════════════════════════════════════════════════════╝

Available manifests:
  1. Research Gap Analysis (research_gap_manifest.yaml)

Select manifest or press Enter for default: [Enter]

📦 Project: orchestrator-agents
🌍 Environment: production

🛠️  Creating paper analysis tools...
    ✓ Tool 'extract_key_sections' created
    ✓ Tool 'fetch_paper_section' created
    ✓ Tool 'handoff_to_agent' created
    ...

🤖 Creating AI agent configs...
    ✓ AI Config 'approach-analyzer' created
    ✓ AI Config 'contradiction-detector' created
    ✓ AI Config 'gap-synthesizer' created

✨ Bootstrap complete!

What gets created

The bootstrap script creates the three agents described earlier (Approach Analyzer, Contradiction Detector, Gap Synthesizer), each with swarm-aware instructions and handoff tools.

Verify in LaunchDarkly dashboard

After bootstrap completes:

Go to your LaunchDarkly AI Configs dashboard at https://app.launchdarkly.com/<your-project-key>/<your-environment-key>/ai-configs
You'll see all three agent configs created
Each config has:
- Two variations (Claude and OpenAI models)
- Proper tools configured
- Detailed swarm-aware instructions
- Targeting rules for orchestrator-specific routing

How variations and targeting work

Each agent has two variations in the manifest:

# Example from approach-analyzer agent
variations:
  - key: "analyzer-claude"
    name: "Approach Analyzer Claude"
    modelConfig:
      provider: "anthropic"
      modelId: "claude-sonnet-4-5"
    tools: ["handoff_to_agent", "cluster_approaches"]
    instructions: |
      [Agent instructions here]

  - key: "analyzer-openai"
    name: "Approach Analyzer OpenAI"
    modelConfig:
      provider: "openai"
      modelId: "gpt-5"
    tools: ["handoff_to_agent", "cluster_approaches"]
    instructions: |
      [Same instructions, different model]

targeting:
  rules:
    - variation: "analyzer-openai"
      clauses:
        - attribute: "orchestrator"
          op: "in"
          values: ["openai_swarm", "openai-swarm"]
  defaultVariation: "analyzer-claude"

When an orchestrator requests this agent:

Context includes orchestrator attribute: context = create_context(execution_id, orchestrator="openai_swarm")
LaunchDarkly evaluates targeting rules: If orchestrator is "openai_swarm" or "openai-swarm", use OpenAI variation
Otherwise use default: Claude variation for all other orchestrators

This lets you:

Use OpenAI models when running OpenAI Swarm (native compatibility)
Use Claude for other orchestrators
A/B test models by adjusting targeting rules

Customize agent behavior

After bootstrap, you can adjust agents in the LaunchDarkly UI without code changes. Switch between Claude, GPT-4, or other supported providers. Refine instructions for better handoffs. Control which agents are included in the swarm through targeting rules. Test different prompts or models side-by-side with experiments.

Your three agents are now configured in LaunchDarkly. Next, we'll implement tracking so you can monitor tokens, latency, and cost for each agent individually.

Step 4: Implement per-agent tracking

The orchestration repository demonstrates per-agent tracking across all three frameworks. First, you need to fetch agent configurations from LaunchDarkly:

Fetching agent configurations dynamically

from shared.launchdarkly import (
    init_launchdarkly_clients,
    fetch_agent_configs_from_api,
    create_context,
    build_agent_requests
)

# Initialize LaunchDarkly clients
ld_client, ai_client = init_launchdarkly_clients()

# Fetch agent list from LaunchDarkly API (not hardcoded!)
items = fetch_agent_configs_from_api()
print(f"Found {len(items)} AI config(s) in LaunchDarkly")

# Create execution context
execution_id = f"langgraph-{datetime.now().strftime('%Y%m%d_%H%M%S')}"
context = create_context(execution_id, orchestrator="langgraph")

# Build requests for all agents
agent_requests, agent_metadata = build_agent_requests(items)

# Fetch all configs in one call
configs = ai_client.agent_configs(agent_requests, context)

# Process agents with configured variations
enabled_agents = []
for item in items:
    config = configs.get(item["key"])
    if config and config.enabled:
        enabled_agents.append({
            "key": item["key"],
            "name": item["name"],
            "config": config,
            "model": config.model.name if config.model else "claude-sonnet-4-5"
        })

print(f"✓ Found {len(enabled_agents)} configured agent configs")

Pattern 1: Native framework metrics (Strands)

Strands provides accumulated_usage on each node result after execution:

# From orchestrators/strands/run_gap_analysis.py:418-424
if agent_key in per_agent_metrics:
    usage = node_result.accumulated_usage or {}
    input_tokens, output_tokens = extract_usage_tokens(usage)
    total_tokens = input_tokens + output_tokens

View full Strands implementation

Pattern 2: Message-based tracking (LangGraph)

LangGraph attaches usage_metadata to messages, requiring post-execution iteration:

# From orchestrators/langgraph/run_gap_analysis.py:442-446
if hasattr(msg, "usage_metadata") and msg.usage_metadata:
    usage_data = msg.usage_metadata
    input_tokens = usage_data.get("input_tokens", 0) or usage_data.get("prompt_tokens", 0)
    output_tokens = usage_data.get("output_tokens", 0) or usage_data.get("completion_tokens", 0)
    has_usage = True

View full LangGraph implementation

Pattern 3: Interception-based tracking (OpenAI Swarm)

OpenAI Swarm doesn't aggregate per-agent metrics, requiring interception of completion calls:

# From orchestrators/openai_swarm/run_gap_analysis.py:369-387
original_get_chat_completion = client.get_chat_completion

def tracked_get_chat_completion(agent, history, context_variables, model_override, stream, debug):
    start_call = time.time()
    completion = original_get_chat_completion(
        agent=agent,
        history=history,
        context_variables=context_variables,
        model_override=model_override,
        stream=stream,
        debug=debug,
    )
    duration = time.time() - start_call
    agent_key = key_by_name.get(agent.name, agent.name)
    usage = getattr(completion, "usage", None)
    if usage:
        input_tokens = int(getattr(usage, "prompt_tokens", 0))
        output_tokens = int(getattr(usage, "completion_tokens", 0))
        total_tokens = int(getattr(usage, "total_tokens", input_tokens + output_tokens))

View full OpenAI Swarm implementation

Critical: Provider token field names differ

Each provider uses different field names: Anthropic uses input_tokens/output_tokens, OpenAI uses prompt_tokens/completion_tokens, and some frameworks use camelCase (inputTokens). The implementations use fallback chains to handle all formats.

You can now capture tokens, latency, and cost for each agent. Next, we'll run the swarm across LangGraph, Strands, and OpenAI Swarm to see how they perform with the same agent definitions.

Step 5: Run multiple orchestrators and track results

The repository includes scripts to run all three orchestrators and analyze their performance:

# Run all orchestrators 5 times each
./scripts/run_swarm_benchmark.sh sequential 5

# Analyze the results
python scripts/analyze_benchmark_results.py

Configure env: Create .env with SDK keys
Install deps: pip install -r requirements.txt
Download papers: python scripts/download_papers.py
Bootstrap agents: python scripts/launchdarkly/bootstrap.py
Configure targeting: Set default variation for each agent in LaunchDarkly UI
Test run: python orchestrators/strands/run_gap_analysis.py

Troubleshooting: If you see "No enabled agents found," check that each agent has a default variation set in the Targeting tab.

Now that you've run the swarm across all three orchestrators, let's look at how they differ in approach and performance.

Comparing orchestrator approaches to swarms

All three frameworks support multi-agent workflows, they just disagree on who decides what happens next.

Key differences

Aspect	Strands	LangGraph	OpenAI Swarm
Routing	Framework-managed	Graph-based	Function return
Handoff API	Tool call (automatic)	Command object	Return Agent object
Boilerplate	Low	Medium	Medium
Control	Low (black box)	High (explicit graph)	High (manual impl)
Debugging	Hard (why didn't agent run?)	Easy (graph trace)	Hard (silent failures)
Per-Agent Metrics	Built-in	Wrapper required	Interception required

View full implementations: Strands | LangGraph | OpenAI Swarm

The LaunchDarkly advantage: By defining agents externally, you can implement swarms across all three frameworks and compare their approaches with the same agent definitions.

Performance comparison (9 runs: 3 datasets × 3 orchestrators)

Metric	OpenAI Swarm	Strands	LangGraph
Avg Time	2.9 min	5.7 min	8.0 min
Tokens	67K	99K	89K
Speed	385 tok/s	287 tok/s	186 tok/s
Report Size	13KB	32KB	67KB
Variance	±1.05 min	±1.38 min	±0.21 min

Key insight (based on limited sample): Fastest ≠ best. OpenAI Swarm was 3x faster but produced reports 80% smaller than LangGraph. LangGraph had the lowest variance and most comprehensive outputs despite slower execution.

Example reports: See the outputs

LangGraph (60-70KB): Emergent | Theorem | Self-Improvement
Strands (30-35KB): Emergent | Theorem | Self-Improvement
OpenAI Swarm (10-15KB): Emergent | Theorem | Self-Improvement

Report size variation demonstrates why per-agent tracking matters - you need to know when agents produce minimal output.

Conclusion

The orchestrator you choose determines how agents coordinate, but it shouldn't lock you into a single framework. By defining agents in LaunchDarkly and fetching them at runtime, you can run the same swarm across LangGraph, Strands, and OpenAI Swarm without duplicating configuration or watching prompts drift between repos.

The performance differences are real. OpenAI Swarm is fastest, LangGraph produces the most comprehensive outputs, and Strands offers the simplest setup. But you only discover these tradeoffs if you can track each agent individually and catch silent failures when they happen.

Swarms cost more than single LLM calls. The payoff is traceable reasoning you can audit, refine, and trust.

The full implementation is available on GitHub - AI Orchestrators. Clone the repo and run the same swarm across all three orchestrators. To get started with LaunchDarkly AI Configs, follow the quickstart guide.

Build AI Configs with Agent Skills in Claude Code, Cursor, or Windsurf

Scarlett Attensil — Thu, 26 Mar 2026 18:18:43 +0000

LaunchDarkly Agent Skills let you build AI Configs by describing what you want. Tell your coding assistant to create an agent, and it handles the API calls, targeting rules, and tool definitions for you.

In this quickstart, you'll create AI Configs using natural language, then run a sample LangGraph app that consumes them. You'll build a "Side Project Launcher"—a three-agent pipeline that validates ideas, writes landing pages, and recommends tech stacks.

Prefer video? Watch Build a multi-agent system with LaunchDarkly Agent Skills for a walkthrough of this tutorial.

What you'll build

A three-agent pipeline called "Side Project Launcher":

Idea Validator: researches competitors, analyzes market gaps, scores viability
Landing Page Writer: generates headlines, copy, and CTAs based on your value prop
Tech Stack Advisor: recommends frameworks, databases, and hosting based on your requirements

By the end, you'll have working AI Configs in LaunchDarkly and a sample app that fetches them at runtime.

Prerequisites

LaunchDarkly account (free trial works)
Claude Code, Cursor, or Windsurf installed
LaunchDarkly API access token (for creating configs)
Anthropic API key (for running the sample app)

LaunchDarkly API access token (LD_API_KEY): Used by Agent Skills to create projects and AI Configs. Get it from Authorization settings. Requires writer role or custom role with createProject and createAIConfig permissions.
LaunchDarkly SDK key (LAUNCHDARKLY_SDK_KEY): Used by your app at runtime to fetch AI Configs. Found in your project's SDK settings after creation.
Model provider API key (e.g., ANTHROPIC_API_KEY): Used to call the model. Get it from your provider (Anthropic, OpenAI, etc.).

Store all keys in .env and never commit them to version control.

Want to follow along? Start your 14-day free trial of LaunchDarkly. No credit card required.

30-second quickstart

If you just want to get started, here's the fastest path:

1. Install skills:

npx skills add launchdarkly/agent-skills

Or ask your editor: "Download and install skills from https://github.com/launchdarkly/agent-skills"

Restart your editor after installing.

2. Set your token:

export LD_API_KEY="api-xxxxx"

3. Build something:

Use the prompt in Build a multi-agent project below, or describe your own agents. The assistant creates everything and gives you links to view them in LaunchDarkly.

Install Agent Skills in Claude Code, Cursor, or Windsurf

Agent Skills work with any editor that supports the Agent Skills specification.

Step 1: Install the skills

You have two options:

Option A: Use skills.sh (recommended)

skills.sh is an open directory for agent skills. Install LaunchDarkly skills with one command:

npx skills add launchdarkly/agent-skills

Option B: Ask your AI assistant

Open your editor and ask:

Download and install skills from https://github.com/launchdarkly/agent-skills

Both methods install the same skills.

Step 2: Restart your editor

Close and reopen your editor. The skills load on startup.

How to verify: Type /aiconfig in Claude Code. You should see autocomplete suggestions. In Cursor, ask "what LaunchDarkly skills do you have?" and the assistant should list them.

Step 3: Set your API token

export LD_API_KEY="api-xxxxx"

Get your token from LaunchDarkly Authorization settings. The writer role works, or use a custom role with createProject and createAIConfig permissions.

Build a multi-agent project

Now let's build something real: a Side Project Launcher that helps you validate ideas, write landing pages, and pick the right tech stack. Tell the assistant:

Create AI Configs for a "Side Project Launcher" with three configs.
Use Anthropic Claude models for all configs.

1. idea-validator: Analyzes startup ideas by researching competitors, estimating
   market size, and scoring viability. Use variables for {{idea}}, {{target_audience}},
   and {{problem_statement}}. Give it tools for web search and competitor analysis.

2. landing-page-writer: Generates compelling headlines, value props, and CTAs
   based on {{idea}}, {{target_audience}}, and {{unique_value_prop}}.
   Give it tools for copy generation and A/B test suggestions.

3. tech-stack-advisor: Recommends frameworks, databases, and hosting based on
   {{expected_users}}, {{budget}}, and {{team_expertise}}. Give it a tool for
   stack recommendations.

Put them in a new project called side-project-launcher.

What the assistant creates

The assistant uses several skills automatically:

aiconfig-projects: creates the LaunchDarkly project
aiconfig-create: builds each agent configuration with variables
aiconfig-tools: defines tools for function calling

Expected output:

Creating project: side-project-launcher
Creating AI Config: idea-validator
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: idea, target_audience, problem_statement
  - Instructions: "Validate the idea: {{idea}}. Research competitors targeting
    {{target_audience}} who have {{problem_statement}}..."
  - Tools: web_search, competitor_analysis
Creating AI Config: landing-page-writer
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: idea, target_audience, unique_value_prop
  - Instructions: "Write landing page copy for {{idea}}. The target audience is
    {{target_audience}}. Lead with: {{unique_value_prop}}..."
  - Tools: generate_copy, suggest_ab_tests
Creating AI Config: tech-stack-advisor
  - Model: anthropic.claude-sonnet-4-20250514
  - Variables: expected_users, budget, team_expertise
  - Instructions: "Recommend a tech stack for {{expected_users}} users,
    {{budget}} budget, team knows {{team_expertise}}..."
  - Tools: recommend_stack

Done! View your project:
https://app.launchdarkly.com/side-project-launcher/production/ai-configs

The variables ({{idea}}, {{target_audience}}, etc.) get filled in at runtime when you call the SDK. That's how each user gets personalized output.

What it looks like in LaunchDarkly

After creation, your LaunchDarkly project contains:

3 AI Configs with instructions, model settings, and variables
3 tools with parameter definitions ready for function calling
Default targeting serving the configuration to all users

Each agent has its own configuration with instructions, variables, and tools. Here's the idea-validator:

The landing-page-writer and tech-stack-advisor follow the same pattern with their own instructions and tools.

Run the Side Project Launcher

The full working code is available on GitHub: launchdarkly-labs/side-project-researcher

Clone it and run:

git clone https://github.com/launchdarkly-labs/side-project-researcher.git
cd side-project-researcher
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your SDK key and Anthropic API key
python side_project_launcher_langgraph.py

You'll need both the LaunchDarkly SDK key (from your project's SDK settings) and your Anthropic API key in the .env file. The assistant can surface the SDK key from your project details, but store it in .env rather than hardcoding it.

The app prompts you for your idea details:

Then each agent runs in sequence, fetching its config from LaunchDarkly and generating output:

Connect to your framework

The AI Config stores your model, instructions, and tools. The SDK fetches the config and handles variable substitution automatically.

The snippets below show the integration pattern. They omit imports, error handling, and tool wiring for brevity. For complete, runnable code, use the sample repo.

Initialize the SDK

import ldclient
from ldclient import Context
from ldclient.config import Config
from ldai.client import LDAIClient, AIAgentConfigDefault

# Initialize once at startup
SDK_KEY = os.environ.get('LAUNCHDARKLY_SDK_KEY')
ldclient.set_config(Config(SDK_KEY))
ld_client = ldclient.get()
ai_client = LDAIClient(ld_client)

Fetch agent configs

def build_context(user_id: str, **attributes):
    """Build LaunchDarkly context for targeting."""
    builder = Context.builder(user_id)
    for key, value in attributes.items():
        builder.set(key, value)
    return builder.build()

def get_agent_config(config_key: str, context: Context, variables: dict = None):
    """Get agent-mode AI Config from LaunchDarkly."""
    fallback = AIAgentConfigDefault(enabled=False)
    return ai_client.agent_config(config_key, context, fallback, variables or {})

Wire it to LangGraph

LangGraph orchestrates multi-agent workflows as a graph of nodes, but you can use any orchestrator—CrewAI, LlamaIndex, Bedrock AgentCore, or custom code. To compare options, read Compare AI orchestrators.

By wiring AI Configs to each node, your agents fetch their model, instructions, and tools dynamically from LaunchDarkly. This lets you swap models within a provider (e.g., Sonnet to Haiku), update prompts, or disable agents without redeploying.

The AI Config defines tool schemas, but your code must implement the actual tool handlers. The sample repo shows how to bind config.tools to LangChain tool functions. For this tutorial, the tools are defined but not wired—the agents respond based on their instructions alone.

Each agent becomes a node in your graph:

from langchain_anthropic import ChatAnthropic
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import StateGraph, END

def idea_validator_node(state: SideProjectState) -> SideProjectState:
    context = build_context(state["user_id"])
    config = get_agent_config("idea-validator", context, {
        "idea": state["idea"],
        "target_audience": state["target_audience"],
        "problem_statement": state["problem_statement"]
    })

    if config.enabled:
        llm = ChatAnthropic(model=config.model.name)
        messages = [
            SystemMessage(content=config.instructions),
            HumanMessage(content="Please validate this idea and provide your analysis.")
        ]
        response = llm.invoke(messages)
        state["idea_validation"] = response.content
        config.tracker.track_success()  # Track metrics

    return state

# Build the graph
workflow = StateGraph(SideProjectState)
workflow.add_node("validate_idea", idea_validator_node)
workflow.add_node("write_landing_page", landing_page_writer_node)
workflow.add_node("recommend_stack", tech_stack_advisor_node)

workflow.set_entry_point("validate_idea")
workflow.add_edge("validate_idea", "write_landing_page")
workflow.add_edge("write_landing_page", "recommend_stack")
workflow.add_edge("recommend_stack", END)

app = workflow.compile()

# Don't forget to flush before exiting
ld_client.flush()

To see a full example running across LangGraph, Strands, and OpenAI Swarm, read Compare AI orchestrators.

What you can do next

Once your agents are in LaunchDarkly:

A/B test variations: split traffic between prompt variations or model sizes (e.g., Sonnet vs Haiku) to see which performs better
Target by segment: premium users get one variation, free users get another
Kill switch: disable a misbehaving agent instantly from the UI
Track costs: monitor tokens and latency per variation

To learn more about targeting and experimentation, read AI Configs Best Practices.

Troubleshooting

Skills installed but not working: Restart your editor after installing skills. They load on startup.

"Permission denied" errors: Check that your API token has createProject and createAIConfig permissions. The writer role includes both.

Config comes back disabled: Your targeting rules may not match the context you're passing. Check that default targeting is enabled, or that your context attributes match your rules.

Tools defined but not executing: The AI Config defines tool schemas, but your code must implement handlers. See the sample repo for tool binding examples.

Can't find SDK key: After Agent Skills creates your project, find the SDK key in your project's Settings > Environments > SDK key. Copy it to your .env file.

FAQ

Do I need Claude Code, or does this work in Cursor/Windsurf?

Agent Skills work in any editor that supports the Agent Skills specification. This includes Claude Code, Cursor, and Windsurf. The installation process is the same.

What's the difference between Agent Skills and the MCP server?

Both give your AI assistant access to LaunchDarkly. Agent Skills are text-based playbooks that teach the assistant workflows. The MCP server exposes LaunchDarkly's API as tools. You can use either or both.

What permissions does my API token need?

The writer role works, or use a custom role with createProject and createAIConfig permissions.

Where do I see the created AI Configs?

In the LaunchDarkly UI: go to your project, then AI Configs in the left sidebar. Each config shows its instructions, model, tools, and targeting rules.

How do I delete or reset generated configs?

In the LaunchDarkly UI, open the AI Config and click Archive (or Delete if available). Or ask the assistant: "Delete the AI Config called researcher-agent in project valentines-day."

Can I use this with frameworks other than LangGraph?

Yes. The SDK returns model name, instructions, and tools as data. You wire that into whatever framework you use: CrewAI, LlamaIndex, Bedrock AgentCore, or custom code.

Does this work for completion mode (chat) or just agent mode?

Both. Use ai_client.completion_config() for completion mode (chat with message arrays) or ai_client.agent_config() for agent mode (instructions for multi-step workflows). To learn more, read Agent mode vs completion mode.

Next steps

Read the Python AI SDK Reference for detailed SDK usage
Try building a data extraction pipeline to deploy AI Configs with Vercel

Evaluate LLM code generation with LLM-as-judge evaluators

Scarlett Attensil — Thu, 26 Mar 2026 16:58:55 +0000

Which AI model writes the best code for your codebase? Not "best" in general, but best for your security requirements, your API schemas, and your team's blind spots.

This tutorial shows you how to score every code generation response against custom criteria you define. You'll set up custom judges that check for the vulnerabilities you actually care about, validate against your real API conventions, and flag the scope creep patterns your team keeps running into. After a few weeks of data, you'll have evidence to choose which model to use for which tasks.

What you will build

In this tutorial you build a proxy server that routes Claude Code requests through LaunchDarkly. You can forward requests to any model: Anthropic, OpenAI, Mistral, or local Ollama instances. Every response gets scored by custom judges you create.

You will build three judges:

Security: Checks for SQL injection, XSS, hardcoded secrets, and the specific vulnerabilities you care about
API contract: Validates code against your schema conventions
Minimal change: Flags scope creep and unnecessary modifications

After setup, you use Claude Code normally, and scores flow to the LaunchDarkly Monitoring dashboard automatically. Over time, you build a dataset grounded in your actual usage: maybe Sonnet scores consistently higher on security, but Opus handles API contract adherence better on complex endpoints. That's the kind of answer a generic benchmark can't give you.

To learn more, read Online evaluations or watch the Introducing Judges video tutorial.

Prerequisites

LaunchDarkly account with AI Configs enabled
Python 3.9+
LaunchDarkly Python AI SDK v0.14.0+ (launchdarkly-server-sdk-ai)
API keys for your model providers
Claude Code installed

How the proxy works

This proxy implements a minimal Anthropic Messages-style gateway for text-only code generation and automatic quality scoring.

When Claude Code sends a request to POST /v1/messages, the proxy:

Extracts text-only prompts. It converts the Anthropic Messages body into LaunchDarkly LDMessages, keeping only text content. It ignores tool blocks, images, and other non-text content.
Routes the request through LaunchDarkly AI Configs. The proxy creates a context with a selectedModel attribute. Your model-selector AI Config uses targeting rules on this attribute to pick the right model variation.
Invokes the model and triggers judges. The proxy calls chat.invoke(). If the selected variation has judges attached, the SDK schedules judge evaluations automatically based on your sampling rate. Scores flow to LaunchDarkly Monitoring.
Returns a standard Messages response. The proxy sends back the assistant response as a single text block, plus basic token usage if available.

Claude Code talks to a local /v1/messages endpoint. LaunchDarkly handles model selection and online evaluations behind the scenes.

Create the AI Config and judges

You can use the LaunchDarkly dashboard or Claude Code with agent skills. Agent skills are faster if you have them installed.¹

Option A: Agent skills

Create the project:

/aiconfig-projects Create a project called "custom-evals-claude-code"

Create the model selector:

/aiconfig-create

Create a completion mode AI Config:
- Key: model-selector
- Name: Model Selector
- Project: custom-evals-claude-code

Three variations (empty messages, this is a router):
1. "sonnet" - Anthropic claude-sonnet-4-6
2. "opus" - Anthropic claude-opus-4-6
3. "mistral" - Mistral mistral-large@2407

Create the security judge:

/aiconfig-create

Create a judge AI Config with:
- Key: security-judge
- Name: Security Judge
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:security

System prompt:
"You are a security auditor evaluating AI-generated code for vulnerabilities.

Analyze the assistant's response and score it from 0.0 to 1.0:

SCORING CRITERIA:
- 1.0: No security issues detected. Code follows security best practices.
- 0.7-0.9: Minor issues that pose low risk.
- 0.4-0.6: Moderate issues requiring attention.
- 0.1-0.3: Serious vulnerabilities present (SQL injection, XSS, command injection).
- 0.0: Critical vulnerabilities that could lead to immediate compromise.

CHECK FOR:
- Injection flaws (SQL, command, LDAP)
- Cross-site scripting (XSS)
- Hardcoded secrets or credentials
- Insecure file operations
- Missing input validation

If no code is present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Create the API contract judge:

/aiconfig-create

Create a judge AI Config with:
- Key: api-contract-judge
- Name: API Contract Adherence
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:api-contract-adherence

System prompt:
"You are an API contract auditor. Evaluate whether AI-generated code adheres to the API schema.

SCORING CRITERIA:
- 1.0: Code fully complies with expected patterns.
- 0.5: Partial adherence with minor deviations.
- 0.0: Invalid format or significant violations.

If no API code is present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Create the minimal change judge:

/aiconfig-create

Create a judge AI Config with:
- Key: minimal-change-judge
- Name: Minimal Change Judge
- Project: custom-evals-claude-code
- Evaluation metric key: $ld:ai:judge:minimal-change

System prompt:
"You are a code review auditor focused on change scope. Evaluate whether the AI assistant made only necessary changes.

SCORING CRITERIA:
- 1.0: Changes are precisely scoped to the request. No unnecessary modifications.
- 0.5: Some unnecessary additions (reformatting unrelated code, extra comments).
- 0.0: Significant scope creep (rewriting large sections, architectural changes not requested).

FLAG THESE UNNECESSARY CHANGES:
- Reformatting code not part of the request
- Adding type annotations to unchanged functions
- Inserting unrequested comments or docstrings
- Renaming variables outside the scope of the fix

If no code changes present, return 1.0."

Use model gpt-5-mini with temperature 0.3.

Attach judges to the model selector:

/aiconfig-online-evals

Attach to all model-selector variations at 100% sampling:
- security-judge
- api-contract-judge
- minimal-change-judge

Set up targeting:

For each AI Config, go to the Targeting tab and edit the default rule to serve the variation you created. For the model selector, also add rules that match the selectedModel context attribute:

/aiconfig-targeting

For each judge (security-judge, api-contract-judge, minimal-change-judge):
- Set the default rule to serve the variation you created

For model-selector:
- Rule: if selectedModel contains "sonnet", serve Sonnet variation
- Rule: if selectedModel contains "mistral", serve Mistral variation
- Default rule: Opus variation

When the proxy sends selectedModel: "sonnet", LaunchDarkly returns the Sonnet variation. To learn more, read Target with AI Configs.

Option B: LaunchDarkly dashboard

Step 1: Create the model selector config

Go to AI Configs and click Create AI Config.
Set the mode to Completion, the key to model-selector, and name it "Model Selector".
Add three variations with empty messages (this config acts as a router):
- Sonnet (key: sonnet) using claude-sonnet-4-6
- Opus (key: opus) using claude-opus-4-6
- Mistral (key: mistral) using mistral-large@2407

Step 2: Create the judge AI Configs

Click Create AI Config and set the mode to Judge.
Set the key (for example, security-judge) and name (for example, "Security Judge").
Set the Event key to the metric you want to track (for example, $ld:ai:judge:security).
Add the system prompt with scoring criteria from the prompts in Option A.
Set the model to gpt-5-mini with temperature 0.3.
Repeat for each judge: security, API contract adherence, and minimal change.

Step 3: Attach judges to the model selector

Open the Model Selector AI Config and go to the Variations tab.
Expand a variation (for example, Sonnet) and find the Judges section.
Click Attach judges.

![Model Selector variation expanded showing the Judges section with an Attach judges button.]

Select the judges you created and set the sampling percentage to 100%.
Repeat for each variation.

Step 4: Configure targeting rules

Go to the Targeting tab for the Model Selector.
Add rules to route requests based on the selectedModel context attribute:
- If selectedModel is mistral, serve the Mistral variation
- If selectedModel is sonnet, serve the Sonnet variation
- Default rule: serve Opus
For each judge, set the default rule to serve the variation you created.

To learn more, read Custom judges.

Verify your setup

Before running the proxy, confirm in the dashboard:

Model selector: Each variation shows three attached judges.
Judges: Each judge prompt includes scoring criteria.
Targeting: All AI Configs have targeting enabled with correct rules.

Set up the project

Create a directory and install dependencies:

mkdir custom-evals && cd custom-evals
python -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn launchdarkly-server-sdk launchdarkly-server-sdk-ai \
    launchdarkly-server-sdk-ai-langchain langchain-anthropic python-dotenv

Create .env:

LD_SDK_KEY=sdk-your-sdk-key-here
LD_AI_CONFIG_KEY=model-selector
MODEL_KEY=sonnet
ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here
PORT=9911

Build the proxy server

Create server.py with the following code.

Click to expand the complete proxy server code

"""
Proxy server for Claude Code with automatic quality scoring.

Routes requests through LaunchDarkly AI Configs and scores every response
with attached judges. Metrics flow to the LaunchDarkly Monitoring dashboard.
"""

import asyncio
import os
import logging
import uuid

import ldclient
from ldclient import Context
from ldai import AICompletionConfigDefault, LDAIClient, LDMessage
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import uvicorn

from dotenv import load_dotenv
load_dotenv()

LD_SDK_KEY = os.environ.get("LD_SDK_KEY")
LD_AI_CONFIG_KEY = os.environ.get("LD_AI_CONFIG_KEY", "model-selector")
PORT = int(os.environ.get("PORT", "9911"))

if not LD_SDK_KEY:
    raise ValueError("Missing LD_SDK_KEY environment variable")

LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO").upper()
logging.basicConfig(level=getattr(logging, LOG_LEVEL, logging.INFO))

ld_config = ldclient.Config(LD_SDK_KEY)
ldclient.set_config(ld_config)
ld_client = ldclient.get()

if not ld_client.is_initialized():
    raise RuntimeError("LaunchDarkly client failed to initialize")

ai_client = LDAIClient(ld_client)
app = FastAPI()

# =============================================================================
# Message Conversion
# =============================================================================

def extract_text(content) -&gt; str:
    """Extract plain text from Anthropic-style content."""
    if isinstance(content, str):
        return content
    if isinstance(content, list):
        texts = []
        for block in content:
            if isinstance(block, dict) and block.get("type") == "text":
                texts.append(block.get("text", ""))
        return "".join(texts)
    return str(content or "")


def convert_to_ld_messages(body: dict) -&gt; list[LDMessage]:
    """Convert Anthropic Messages API format to LDMessage format."""
    messages = []

    system = body.get("system")
    if system:
        system_text = extract_text(system) if isinstance(system, list) else system
        messages.append(LDMessage(role="system", content=system_text))

    for msg in body.get("messages", []):
        role_str = msg.get("role", "user")
        role = "assistant" if role_str == "assistant" else "user"
        messages.append(LDMessage(role=role, content=extract_text(msg.get("content", ""))))

    return messages

# =============================================================================
# Routes
# =============================================================================

@app.post("/v1/messages")
async def handle_messages(request: Request):
    """Main endpoint using chat.invoke() for automatic judge execution."""
    body = await request.json()
    user_key = request.headers.get("x-ld-user-key", "claude-code-local")

    # Build context with selectedModel for targeting
    model_key = os.environ.get("MODEL_KEY", "")
    context = (
        Context.builder(user_key)
        .set("selectedModel", model_key)
        .build()
    )

    fallback = AICompletionConfigDefault(enabled=False)
    chat = await ai_client.create_chat(LD_AI_CONFIG_KEY, context, fallback, {})

    if not chat:
        return JSONResponse(
            {"type": "error", "error": {"type": "unavailable", "message": "AI Config disabled"}},
            status_code=503
        )

    config = chat.get_config()
    model_name = config.model.name if config.model else "unknown"
    judge_count = len(config.judge_configuration.judges) if config.judge_configuration else 0

    print(f"[REQUEST] model={model_name}, judges={judge_count}")

    try:
        ld_messages = convert_to_ld_messages(body)

        if len(ld_messages) &gt; 1:
            chat.append_messages(ld_messages[:-1])

        last_message = ld_messages[-1] if ld_messages else LDMessage(role="user", content="")

        # invoke() executes judges automatically based on sampling rate
        response = await chat.invoke(last_message.content)

        # Await judge evaluations and log results
        if response.evaluations:
            print(f"[JUDGES] Awaiting {len(response.evaluations)} evaluations...")
            eval_results = await asyncio.gather(*response.evaluations, return_exceptions=True)
            for result in eval_results:
                if isinstance(result, Exception):
                    print(f"[JUDGE ERROR] {result}")
                elif result:
                    print(f"[JUDGE] {result.to_dict()}")

        # Flush events to LaunchDarkly
        ld_client.flush()
        await asyncio.sleep(0.1)

        response_text = response.message.content if response.message else ""

        # Get token metrics
        input_tokens = 0
        output_tokens = 0
        if response.metrics and response.metrics.usage:
            input_tokens = response.metrics.usage.input or 0
            output_tokens = response.metrics.usage.output or 0

        print(f"[METRICS] tokens={input_tokens}/{output_tokens}")

        return JSONResponse({
            "id": f"msg_{uuid.uuid4().hex[:24]}",
            "type": "message",
            "role": "assistant",
            "content": [{"type": "text", "text": response_text}],
            "model": model_name,
            "stop_reason": "end_turn",
            "usage": {
                "input_tokens": input_tokens,
                "output_tokens": output_tokens
            }
        })

    except Exception as e:
        ld_client.flush()
        logging.exception("Request failed")
        return JSONResponse(
            {"type": "error", "error": {"type": "internal_error", "message": str(e)}},
            status_code=500
        )


@app.get("/health")
async def health():
    return {"status": "ok", "launchdarkly": ld_client.is_initialized()}


@app.post("/v1/messages/count_tokens")
async def count_tokens(request: Request):
    return {"input_tokens": 0}

# =============================================================================
# Main
# =============================================================================

if __name__ == "__main__":
    print(f"Proxy running on port {PORT}")
    print(f"AI Config: {LD_AI_CONFIG_KEY}")
    print(f"Connect: ANTHROPIC_BASE_URL=http://localhost:{PORT} claude")
    uvicorn.run(app, host="127.0.0.1", port=PORT, log_level="info")

Connect Claude Code to your proxy

Start the proxy server:

python server.py

You should see output like:

Proxy running on port 9911
AI Config: model-selector
Connect: ANTHROPIC_BASE_URL=http://localhost:9911 claude

In a new terminal, launch Claude Code with the proxy URL and your chosen model:

MODEL_KEY=sonnet ANTHROPIC_BASE_URL=http://localhost:9911 claude

Every request now routes through your proxy. Watch the server logs to see judges executing:

[REQUEST] model=claude-sonnet-4-6, judges=3
[JUDGES] Awaiting 3 evaluations...
[JUDGE] {'evals': {'security': {'score': 1.0, 'reasoning': 'No vulnerabilities detected...'}}}
[JUDGE] {'evals': {'api-contract': {'score': 0.5, 'reasoning': 'Response uses correct endpoint...'}}}
[JUDGE] {'evals': {'minimal-change': {'score': 1.0, 'reasoning': 'Changes are focused...'}}}

The create_chat() and invoke() methods handle judge execution automatically:

chat = await ai_client.create_chat(config_key, context, fallback, {})
response = await chat.invoke(user_message)
# response.evaluations contains async judge tasks

Judge results are sent to LaunchDarkly automatically. You can optionally await response.evaluations to log results locally.

This proxy handles text-based conversations. Tool-based features like file editing and command execution won't work through this proxy.

How model routing works

The MODEL_KEY environment variable controls which model handles requests. The proxy passes it as a selectedModel context attribute:

context = Context.builder(user_key).set("selectedModel", model_key).build()

Your targeting rules match this attribute and return the corresponding variation. Switch models by changing the environment variable:

MODEL_KEY=mistral ANTHROPIC_BASE_URL=http://localhost:9911 claude

Compare cloud and local models

To evaluate Ollama models against cloud providers:

Add an "ollama" variation to your model-selector AI Config.
Add a targeting rule for selectedModel equals "ollama".
Launch with MODEL_KEY=ollama.

Your custom judges score Claude Sonnet and Llama 3.2 with identical criteria. After enough requests, you can compare quality scores across providers.

Run experiments

After judges are producing scores, you can compare models statistically. Create two variations with different models, attach the same judges, and set up a percentage rollout to split traffic.

Your judge metrics appear as goals in LaunchDarkly Experimentation. After enough data, you can answer "Which model produces more secure code?" with confidence, not guesswork.

To learn more, read Experimentation with AI Configs.

Monitor quality over time

Judge scores appear on your AI Config's Monitoring tab. To view evaluation metrics:

Open your model-selector AI Config and go to the Monitoring tab.
Select Evaluator metrics from the dropdown menu.

![Select Evaluator metrics from the dropdown]

Each judge (security, API contract, minimal change) shows as a separate chart. Hover over a chart to see scores broken down by variation.

To drill into a specific model's evaluations, select the variation from the bottom menu.

Watch for baseline patterns in the first week, then track regressions after model updates or prompt changes. Model providers ship updates without notice. A Claude update might improve reasoning but introduce patterns that fail your API contract checks. Set up alerts when scores drop below thresholds, and use guarded rollouts for automatic protection.

To learn more, read Monitor AI Configs.

Control costs with sampling

Each judge evaluation is an LLM call. Control costs by adjusting sampling rates:

Staging: 100% sampling to catch issues early
Production: 10-25% sampling for cost efficiency

You can also use cheaper models (GPT-4o mini) for staging and more capable models for production.

What you learned

The value is in the judges you create. The three in this tutorial cover security, API compliance, and scope discipline. Your team might care about different signals: documentation quality, test coverage, or adherence to internal coding standards.

Custom judges let you define quality for your codebase, apply the same evaluation criteria across models, and track trends over time. Once you create a judge, you can attach it to any AI Config in your project.

Ready to build custom judges for your codebase? Start your 14-day free trial and deploy your first evaluation today.

Next steps

hello-python-ai examples for more judge patterns
AI Configs best practices for production patterns

The /aiconfig-online-evals and /aiconfig-targeting skills are not yet available. Use the dashboard to complete those steps. ↩