DEV Community: Alexis Roberson

Wait, I was today years old when I realized you can post on dev like its Twitter. Well hello good people of the internets

Alexis Roberson — Sun, 27 Sep 2026 06:05:16 +0000

Don't put your agent guardrails in the system prompt

Alexis Roberson — Sat, 26 Sep 2026 22:44:02 +0000

Coding agents write, refactor, and open pull requests at a pace humans can't match. The instinct is to stuff safety into the system prompt: "never push to main," "don't touch secrets," "always run the tests." That feels like control. It isn't.

A system prompt is advice to a model. Models are trained to follow instructions, and they are also trained to be helpful. When those goals collide, helpful often wins. A determined user, a long thread, or a cleverly worded task can dilute or override the rules you thought were hard-coded. Even without adversarial pressure, prompt-only guardrails drift. Every time you tune tone, add a workflow, or paste more context, you change the same surface that was supposed to enforce policy.

If your safety depends on the model remembering to behave, you don't have a control plane. You have a suggestion.

Why prompt-only guardrails fail in practice

Three failure modes show up again and again.

First, prompts are soft. They sit in the same channel as user messages and tool output. There is no separate enforcement path. A policy that lives only in text can be negotiated, forgotten, or reinterpreted.

Second, prompts are hard to operate. You can't gradually expose a new rule to 5% of agents, measure whether it reduces bad tool calls, and roll it back in one click. Changing a prompt usually means redeploying config for everyone at once. That is the opposite of how mature teams ship risky behavior.

Third, prompts don't produce evidence. When something goes wrong, you need to know which control failed. A buried sentence in a 2,000-token system message rarely leaves a clear audit trail. External controls can log a deny, a blocked tool call, or a failed policy check. An ignored instruction usually just looks like "the agent did a weird thing."

What "outside the prompt" actually means

External guardrails live in the layers around the model, not inside its instructions. Think in four buckets.

Permissions

Decide what the agent is allowed to invoke before it speaks. Tool allowlists, deny lists, and scoped credentials belong here. If the agent cannot call git push --force or read production secrets, you don't need the prompt to politely refuse.

Scope

Limit where the agent may act: which repos, branches, environments, and file paths. Narrow blast radius by default. Broaden it deliberately.

Release

Treat agent-generated change like any other risky change: expose a small slice first, watch live signals, and reverse when those signals degrade, before the blast radius grows. Deploy behind a switch. Expand when signals look healthy. Pull it back without a rebuild when they don't. The agent can still move fast; you decide who feels the impact.

Measurement

Watch quality, cost, error rate, and review outcomes. When a new agent config or tool policy regresses those signals, pause or revert. Judgment without measurement is optimism.

The system prompt still has a job. Use it for style, priorities, and how to reason about the task. Put the hard limits where a process—not a paragraph can enforce them.

A practical control stack for coding agents

You don't need a perfect architecture on day one. A useful starter set looks like this:

Separate policy from personality. Keep the system prompt focused on how the agent works. Move "must / must not" rules into config that your runtime or CI checks independently.
Default to least privilege on tools. Start with read-only where you can. Require explicit elevation for write, merge, deploy, and secret access. Log every elevation.
Gate high-impact actions. Before the agent merges, deletes, or talks to production, run a policy check or human approval step that the model cannot skip by rephrasing the request.
Ship agent output behind progressive release. Whether the change is a feature, a refactor, or a prompt-driven config, release it the way you release anything else that might hurt customers: small, observable, reversible.
Instrument the boring failures. Track blocked tool calls, policy denials, failed tests after agent PRs, and rollback events. Those numbers tell you whether your guardrails are real.

Teams that already wrap risky human-written code in flags and progressive delivery already know this pattern. Coding agents just raise the volume. The same control infrastructure that makes fast shipping safe for people makes fast shipping safe for agents.

What to try this week

Pick one coding agent workflow you already trust enough to use daily. Map every irreversible action it can take. For each one, ask: does a prompt ask it not to, or does something outside the model prevent it? Move the top three irreversible actions behind an external check. Leave the prompt alone for tone and task framing.

You will learn more from three hard denials you can measure than from thirty soft rules the model might follow.

Closing

System prompts are guidance. Guardrails are infrastructure. If you want coding agents to move quickly without inheriting silent failure modes, put the limits where you can observe them, change them, and reverse them...outside the prompt.

Building the Agent Flinch with Jev and LaunchDarkly

Alexis Roberson — Mon, 21 Sep 2026 17:12:16 +0000

Two things happened over the weekend. I got access to Jev, and I came across a post from Biswa G Singh, co-founder of ZeonAI Labs, about a coding agent that deleted his team's dev vector database.

The diagnosis behind the delete was right. A bad import had left a local test cluster on the wrong schema, so wiping the collection and re-importing was a reasonable repair.

The port the agent used had been a tunnel to the shared dev environment for the entire session. Minutes earlier it had run read-only queries through that same port, so it filed the number away as local and never checked again.

What stayed with me was his point that a tired human at 11pm still hesitates before typing DELETE against something that might be shared. The agent had the same information and none of the hesitation.

He called that missing pause the flinch, and with Jev in hand I wanted to see what building one takes under the assumption that reasoning is cheap and accountability isn't.

His team made three changes the same day:

No destructive operation runs without a human confirming the target.
Shared environments never sit inside an agent's working session.
Read-only is enforced by access control rather than by an instruction in a prompt.

What Jev is

Jev is TypeSafe's System One model. Instead of writing a paragraph, it answers typed questions over a state you hand it and returns calibrated confidence with each answer.

A noul is a yes-or-no question that comes back as a probability, and a choice is a pick from a fixed list that comes back with a confidence score. That shape is what makes it usable as a judge in front of a tool call, since code can compare a number to a threshold and cannot do much with prose.

I called it through the TypeSafe SDK and pinned the version. The model is also reachable through Vercel AI Gateway and OpenRouter.

The check I ended up with has three layers, and none of them relies solely on the other two.

Layer	What it does	Where it's configured
Deterministic check	Resolves host and port to an environment, classifies the command, applies policy	`environments.yaml` plus the `flinch-policy` flag
Jev check	Scores the same state with typed yes-or-no answers and one choice	The `flinch-jev` flag
Human confirmation	Requires the resolved environment name as the approval token	A host-only argument, `--i-approve-target`

Reproducing the incident

In order to reproduce the same issue, I added semantic search to an interior design agent I'm working on called Decora. It allows users to search a furniture catalog by vibe, pin pieces to a board, and ask for a plan.

I stored the catalog in a vector database, the same class of system the incident hit. Changing the embedding model changes the vector size, and changing the vector size means dropping the collection and rebuilding it. That rebuild is the legitimate maintenance task where a confident agent can aim at the wrong cluster.

The setup is two Qdrant containers on one laptop, both from a single Docker Compose file. local-test on 6333 is allowed to be broken, and shared-dev on 6335 holds the 69-point catalog the running app searches.

A socat tunnel forwards 6334 to shared-dev, so the dangerous port sits next to the safe one and looks just as local. The agent that talks to Decora through MCP holds a read-only key on shared-dev, so the chat path cannot delete anything.

The coding-agent path that fixes schema is the one that can still hold a full-access key and a URL that happens to end in 6334.

A registry file maps every address, including the tunnel, to a name and a sensitivity, and it tags every command the agent can run as read-only, write, or destructive. Every layer that follows reads this file:

# environments.yaml (excerpt)
environments:
  - name: local-test
    hosts: ["localhost:6333", "127.0.0.1:6333"]
    sensitivity: local
  - name: shared-dev
    hosts: ["localhost:6335", "127.0.0.1:6335"]
    sensitivity: shared
  - name: shared-dev-tunnel
    hosts: ["localhost:6334", "127.0.0.1:6334"]
    sensitivity: shared
    notes: "Looks local. Forwards to shared-dev."

tools:
  - name: qdrant.collections.get
    kind: read-only
    match: "GET /collections"
  - name: qdrant.collection.delete
    kind: destructive
    match: "DELETE /collections/{name}"

To recreate the broken state, seed local-test with a collection at the wrong vector size. The seed script refuses any environment other than local-test:

python scripts/seed_broken_local_test.py --environment local-test
# target=local-test url=http://localhost:6333 size=768

The reproduction script aims the same two calls at the tunnel, a GET on 6334 followed by a DELETE on that same port, while the stated intent is still "fix the broken schema on local-test." It always runs the flinch check now.

The unguarded fact is the identity it prints first. Intended local-test, actual shared-dev-tunnel, would_destroy_shared true. Default is a dry run, so the DELETE is never sent.

python scripts/repro_flinch_baseline.py

{
  "intended_target": "local-test",
  "actual_environment": "shared-dev-tunnel",
  "actual_url": "http://localhost:6334",
  "sensitivity": "shared",
  "would_destroy_shared": true
}

The diagnosis of a broken schema is sound, and a collection delete is the right repair when the target is local-test. The miss is identity. A live --execute against that port, if the check ever allowed it, would wipe shared-dev.

Step 1: Create the policy flag

With the gap between reasoning and accountability proven, the policy that closes it needs a home outside the code, and LaunchDarkly plays that part for the rest of this build.

It holds the policy as a JSON flag, so the check tightens or loosens without a deploy. It holds the judge's model pin and question text the same way, so the wording can change without touching the app.

Custom metrics and a guarded rollout decide when the check moves from watching to enforcing. Each check also writes a flinch.decision log. That is the control layer around the agent, and the flag is only the first piece of it.

I created a multivariate JSON flag, flinch-policy, in the existing decor-agent-ai project. I did that from a chat connected to the LaunchDarkly MCP server. The prompts that created the flags live in flinch-mcp-prompts.md. What matters here is the three variations.

fail-closed is the off variation and the SDK default.

{
  "mode": "live",
  "on_ld_error": "confirm",
  "destructive": {
    "local": "confirm",
    "shared": "block"
  }
}

shadow computes the pause and does not enforce it.

{
  "mode": "shadow",
  "on_ld_error": "confirm",
  "destructive": {
    "local": "allow",
    "shared": "confirm"
  }
}

live is the first time enforce is true. Shared destructive work is still confirm, never allow.

{
  "mode": "live",
  "on_ld_error": "confirm",
  "destructive": {
    "local": "allow",
    "shared": "confirm"
  }
}

Production targeting is on. The default rule started on shadow.

The three variations differ in two places. mode is shadow or live, and destructive.shared is confirm or block. No variation is allowed to serve allow for shared destructive work, and the code treats a variation that tries as confirm.

If LaunchDarkly is unreachable or the JSON is garbage, the in-code default is fail-closed, so the app does not fail open.

# app/flags.py
FLINCH_POLICY_DEFAULT = {
    "mode": "live",
    "on_ld_error": "confirm",
    "destructive": {"local": "confirm", "shared": "block"},
}

Step 2: Pin the Jev model in its own flag

The second flag is a config bag for the Jev check, flinch-jev, and it pins the model to jev-1.13.0. Same MCP chat, second prompt, same appendix.

Production serves pinned. A preview variation exists so jev-latest can be tried against a test context. The default rule and the off variation both serve pinned, so production never sees a floating model.

{
  "model": "jev-1.13.0",
  "choice_confidence_min": 0.72,
  "questions": {}
}

I added the TypeSafe key to the environment file.

# .env
TYPESAFE_API_KEY=your-key-here

The app does not call Jev until step 6. Wiring the key first means the later step is a code change and nothing else.

Step 3: Define the Jev questions

The questions live in the pinned variation of flinch-jev, so the wording can change without a deploy. I added them in the LaunchDarkly UI by editing the variation's JSON, and this is the object that fills the empty questions field from step 2:

{
  "destructive": {
    "type": "noul",
    "instructions": "Is `command` a destructive action against stored data or cluster schema (delete collection, recreate collection, drop volume, wipe and re-import)? Read-only GET, search, and list are not destructive."
  },
  "target_shared": {
    "type": "noul",
    "instructions": "Could `resolved_target` be a shared or non-local environment? Treat sensitivity `shared`, hosts that are tunnels, and ports that forward to shared-dev as shared. `environment.notes` may say it looks local."
  },
  "intent_match": {
    "type": "noul",
    "instructions": "Does `resolved_target` match `stated_intent`? If the agent said local-test or localhost test cluster but `environment.name` is shared-dev or shared-dev-tunnel, or `recent_uses` show this same host was used without re-checking, the target does not match."
  },
  "action": {
    "type": "choice",
    "instructions": "Choose the next step for this tool call. Code will also apply policy; pick the option that fits this state.",
    "criteria": {
      "proceed": "Read-only, or a write that clearly targets the intended local environment with no shared/tunnel ambiguity.",
      "confirm_with_human": "Destructive or shared-target, or intent is ambiguous, or the same port was already used and now the action is stronger. A person should approve the target.",
      "block": "Destructive action on a shared or tunnel target, or the command would wipe data the agent did not name as the intended target."
    }
  }
}

All four go out in one request over the same state. A noul near 0.5 means Jev is unsure, and a choice confidence under 0.72 is treated as confirm_with_human even when the pick is proceed.

Step 4: Build the state for each check

Both checks read the same named JSON, built by the host before the command runs. It carries the exact command, the stated intent, the resolved target, the registry row, and the recent uses of that host in this session.

For the DELETE in the reproduction, the state looks like this:

{
  "command": "DELETE http://localhost:6334/collections/decor_catalog",
  "stated_intent": "Fix the broken decor_catalog schema on local-test",
  "resolved_target": {
    "host": "localhost:6334",
    "url": "http://localhost:6334",
    "collection": "decor_catalog"
  },
  "environment": {
    "name": "shared-dev-tunnel",
    "sensitivity": "shared",
    "hosts": ["localhost:6334", "127.0.0.1:6334"],
    "notes": "Looks local. Forwards to shared-dev."
  },
  "sensitivity": "shared",
  "recent_uses": [
    {
      "command": "GET http://localhost:6334/collections/decor_catalog",
      "environment": "shared-dev-tunnel",
      "kind": "read-only"
    }
  ]
}

The recent_uses field is the incident in miniature. The earlier GET on the same port is what made the agent confident, and here it is written down where a judge can see it.

Step 5: Add the deterministic check

The first layer is ordinary Python that never calls Jev and never talks to Qdrant. It resolves the address to a registry row, classifies the command from the tools list, reads flinch-policy, and returns allow, confirm, or block:

# app/flinch_check.py
def decide(state: dict, policy: dict | None = None) -> dict:
    sensitivity = state.get("sensitivity") or "shared"
    policy = policy or get_flinch_policy(
        str(state.get("context_key") or "flinch-baseline"),
        qdrant_sensitivity=sensitivity,
    )
    kind = classify_command(str(state.get("command") or ""))
    mode = policy.get("mode") or "live"
    if kind == "read-only":
        decision, reason = "allow", "read-only command"
    elif kind == "write":
        decision = "confirm" if sensitivity == "shared" else "allow"
        reason = f"write on {sensitivity} target"
    else:
        destructive = policy.get("destructive") or {}
        decision = destructive.get(sensitivity) or "confirm"
        if sensitivity == "shared" and decision == "allow":
            decision = "confirm"
        reason = f"destructive action on {sensitivity} target"
    if mode == "off":
        return {"layer": "deterministic", "mode": mode, "kind": kind,
                "sensitivity": sensitivity, "decision": "allow",
                "enforce": False, "reason": "policy mode off"}
    return {"layer": "deterministic", "mode": mode, "kind": kind,
            "sensitivity": sensitivity, "decision": decision,
            "enforce": mode == "live", "reason": reason}

The check lives in app/flinch_check.py. The MCP server stays model-free and never imports it.

Decora's chat harness runs decide() in front of every tools/call, including host-side persist that the model skipped. Search and other read-only catalog calls go through. Writes to the in-memory project store are tagged local, so they are not treated as shared-dev.

The reproduction script is the coding-agent stand-in for Qdrant DELETE, and it runs the same function before 6334. Jev still sits on that destructive path, not on every sofa search.

An agent that is very sure 6334 is local-test still has to get past a row that says otherwise.

Step 6: Add the Jev check

Jev sits on the same state after the deterministic check, not instead of it. The backend loads the model, threshold, and questions from flinch-jev at runtime, calls TypeSafe, and maps the answer onto a suggestion the host can use:

# app/jev_check.py
class TypeSafeJev:
    def evaluate(self, state: dict, config: dict) -> dict:
        from typesafe_sdk import TypeSafeClient

        model = (config.get("model") or "jev-1.13.0").strip()
        questions = config.get("questions") or {}
        if not questions:
            raise ValueError("flinch-jev questions are empty")
        key = get_settings().typesafe_api_key.strip()
        with TypeSafeClient(api_key=key or None, model=model) as client:
            response = client.system_one(state=state, questions=questions, model=model)
        nouls = {name: float(answer.noul) for name, answer in response.nouls.items()}
        picked = response.choices["action"]
        return {
            "source": "typesafe",
            "model": response.model or model,
            "nouls": nouls,
            "action": picked.choice,
            "confidence": float(picked.confidence),
            "suggested": _suggest(picked.choice, float(picked.confidence), config),
        }


def _suggest(action, confidence, config) -> str:
    threshold = config.get("choice_confidence_min", 0.72)
    if action not in {"proceed", "confirm_with_human", "block"}:
        return "confirm_with_human"
    if confidence is not None and confidence < float(threshold):
        return "confirm_with_human"
    return action

Typed output means Jev cannot return an invalid action. It can still pick the wrong valid one, which is why it is never the only safeguard.

The host keeps the last word, mapping policy plus scores onto allow, confirm, or block, and it logs the model field from every response so a version change shows up in the data.

If Jev is down or the confidence comes back empty, the policy still runs, and destructive or shared work stays on confirm.

The TypeSafe bill for this whole day, including the 40-key rollout loop later on, was 63 requests, 57,066 tokens, and $0.0021. A single flinch is one of those requests.

Step 7: Build the human confirmation step

One pushback in the comments on Singh's post is the design brief for this step. A human confirming the target receives the same information the agent had, an address that looks right, and approves it for the same reason the agent ran it.

The fix is to resolve the address to an identity before asking, since a person flinches at a named shared cluster and not at a localhost port.

When the decision is confirm, the process does not accept a yes. It prints the command, the resolved environment, the URL, the sensitivity, the deterministic verdict, and the Jev scores, and then it asks for the environment name itself:

flinch confirmation (approve the TARGET, not the action)
command: DELETE http://localhost:6334/collections/decor_catalog
stated_intent: Fix the broken decor_catalog schema on local-test
resolved_name: shared-dev-tunnel
hosts: ['localhost:6334', '127.0.0.1:6334']
url: http://localhost:6334
collection: decor_catalog
sensitivity: shared
deterministic: confirm (destructive action on shared target)
jev_model: jev-1.13.0 source=typesafe
jev_nouls: {"destructive": 0.98, "intent_match": 0.12, "target_shared": 0.95}
jev_action: block suggested=confirm_with_human confidence=0.31
type this environment name to approve: shared-dev-tunnel

Approval is a host argument, not a tool the model can call. Typing the name you meant fails when it isn't the name the registry resolved:

python scripts/repro_flinch_baseline.py --i-approve-target local-test
python scripts/repro_flinch_baseline.py --i-approve-target shared-dev-tunnel

typed: "local-test"
target: "shared-dev-tunnel"
approved: false
approval refused: typed name does not match resolved target

typed: "shared-dev-tunnel"
target: "shared-dev-tunnel"
approved: true
outcome: approved_dry_run
phase_b skipped (dry-run). Pass --execute to wipe shared-dev via 6334.

local-test is refused because the resolved name is shared-dev-tunnel. The matching name is accepted, logged with who and when, and the DELETE still does not go out.

A yes proves nothing. Naming the environment proves the person read the resolution.

Step 8: Wire up telemetry

Every check writes one decision row and sends it two places. Four custom metrics feed the rollout in the next step, and LaunchDarkly Observability keeps the full record.

They are plain custom count metrics in the same project, and the create-metric tool on the MCP server takes these definitions as they are:

[
  {"key": "flinch-check",    "name": "Flinch check",    "kind": "custom", "eventKey": "flinch_check",    "successCriteria": "HigherThanBaseline", "tags": ["flinch", "agent-guardrail"]},
  {"key": "flinch-block",    "name": "Flinch block",    "kind": "custom", "eventKey": "flinch_block",    "successCriteria": "LowerThanBaseline",  "tags": ["flinch", "agent-guardrail"]},
  {"key": "flinch-escalate", "name": "Flinch escalate", "kind": "custom", "eventKey": "flinch_escalate", "successCriteria": "LowerThanBaseline",  "tags": ["flinch", "agent-guardrail"]},
  {"key": "flinch-override", "name": "Flinch override", "kind": "custom", "eventKey": "flinch_override", "successCriteria": "LowerThanBaseline",  "tags": ["flinch", "agent-guardrail"]}
]

Metric	Fires when
`flinch_check`	Every check, with environment, kind, outcome, Jev model, and whether code and Jev disagreed
`flinch_block`	The outcome is `blocked`, or `would_block` in shadow
`flinch_escalate`	The deterministic decision is confirm, or Jev suggests `confirm_with_human`
`flinch_override`	A human approved the target by name

The Observability side is a flinch.check span and a flinch.decision log on the decor-agent service. Each row carries the command, the resolved environment, the deterministic decision, the Jev model and nouls, and the disagreement bit.

A clever agent can always tell a coherent story about why 6334 was fine. The useful artifact says which environment it actually had, which policy it evaluated, and whether anything stopped it.

Production logs for service decor-agent, filtered to flinch.decision. These are the dry-run checks from the guarded-rollout traffic, including the refused and accepted target names. No DELETE was sent.

flinch.check ran 114 times in this window, P50 about 180µs, with no errors. /all and /bulk are Qdrant collection reads from those same dry-runs, not extra deletes.

Step 9: Roll out shadow to live

Both layers can now say confirm or block, which raises the question of when they are allowed to. A flag that can only be on or off makes a poor flinch, because you want to see the pause fire on the command before it is allowed to stop anything.

Shadow writes the same decision record live would have written, and live is the first time enforce is true.

Moving between them is a guarded rollout on flinch-policy in production, not a one-step flip of the default rule. On the flag's Targeting page, under the default rule, I chose Serve, then Guarded rollout, with these settings:

Control is shadow, treatment is live, and the off variation stays fail-closed
Target by user
Stages of 10 percent for 15 minutes, then 50 percent for 15 minutes. A single stage cannot exceed 50 percent. If a stage stays healthy, LaunchDarkly can promote toward 100. A laptop demo does not supply that sample
flinch-escalate as a rollback-and-notify metric, since the tunnel DELETE resolves to confirm, not block
flinch-override monitored as well. A human typing the tunnel name on purpose is a different event from the check catching a miss

The tunnel DELETE in this scenario resolves to confirm, not block, so hanging rollback on flinch-block alone leaves you staring at an empty chart. Instrument the no that fires.

The first guarded release on this flag reverted to shadow for insufficient sample size. That is Guardian doing its job, not the flinch failing.

I started a second rollout and sent distinct context keys through the reproduction while it was still on the first stage.

for i in $(seq 1 40); do
  python scripts/repro_flinch_baseline.py --context-key "flinch-user-$i"
done

Most keys stayed on shadow with enforce false. A few landed on live with enforce true. Both still resolved the DELETE to confirm, and neither sent it.

First stage of the second guarded rollout, 10% live / 90% shadow, four unique users on live. Escalate and override did not have enough sample to call a regression. The release was in progress. It had not promoted, paused, or rolled back yet. Completing Guardian would take far more distinct users than this loop.

What the gate is for

Agents are already good at the part this incident got right. They can read a 768-dimension collection sitting next to a 384-dimension one and propose a rebuild.

The expensive part is showing, after the fact, that the plan was aimed at the cluster you thought it was. That proof also has to cover a judge you can version scoring the same state, and the human who said yes naming the target instead of the action.

Execution will keep getting cheaper. The flinch is the part that has to stay yours.

My First Conference Talk at APIWorld 2026 (A Reflection)

Alexis Roberson — Sat, 05 Sep 2026 21:21:21 +0000

I have to come clean.

Speaking at APIWorld this past week wasn’t actually my first talk acceptance. I had a talk accepted a few years ago, but the conference itself was ultimately cancelled due to the lack of sponsors. But I’d be lying if I didn’t admit to being a slight bit relieved at the time. I was prepared to deliver the best talk I could regardless of the circumstance but I battled so heavily with belonging that the thought of getting on stage to share my opinion terrified me.

Fast forward to two days ago, I finally took the stage after mainly speaking at and hosting company meetups over the years. This time was different. To me, it wasn’t about belonging. That wasn’t the headliner in my mind. It wasn’t about feeling worthy either. It was about sharing about this thing I built and how it helped me see the correlation between two approaches to deploying AI into production. Two approaches that are more complimentary to each other than I think a lot of folks realize.

But the stage wasn’t the preparation, it was the fruit of everything that happened off stage. The months building the project, writing bad CFPs, getting feedback (thanks Nnenna Ndukwe), going back to the drawing board, and writing a CFP for a session I, myself would actually want to attend.

Here’s a few things I learned from delivering my first talk:

1. Your talk can be innovative without being inauthentic.

There will be so much temptation to find a trend and build a CFP or talk around it, but that wasn’t working for me.

The goal of conferences is to bring curious minds together from far and wide to strategize on where we’re going, being honest about where we are, and using where we’ve been to inform the others.

With that said, it is far greater to speak about what you’re excited about and if it just so happens to align with an industry great, but don’t force it.

Which leads to the next point, do study trends and build with tools and technologies like MCP, agentic best practices, etc.

So trends become a way of doing, not hype, and you’re solving real problems and when the time comes to write a CFP for it, you’re speaking from experience.

2. The gap between awareness and adoption is still real

Technology is moving fast, adoption at the top doesn’t always trickle down to the masses as quickly.

There are folks who need to know where to start just as much as those intermediate who want the latest and greatest.

And constantly making this bridge accessible through giving new perspectives of already established things like the impact of LLMs on APIs for instance is a start.

3. Stay witty my friends

I aspire to allow my wit to shine more as I speak in these spaces. This may seem like an odd point here, but go with me.

Name a memorable conference talk you watched that didn’t have even a sprinkle of wit in it.

All day everyday, we’re discussing some weighty stuff, highly technical with an undertone of the chaos happening in the world.

So folks come into these talks with all this on their mind and sometimes it can keep them from the enjoyment of learning.

In our desire, to learn something to land a good job or look favorable in our industry, I don’t want us to lose that.

So wit is a way to loosen everyone up and say hey just enjoy the moment, learn something and let’s have a good time. Never underestimate the power of wit to prepare folks for learning not just listening to another talk.

A note on speaker preparation:

Write the script, practice it until, in the words of the late great Prince, “It seeps into the floorboards” and I’ll add let it seep it into the walls from you practicing, reciting aloud tweaking, so it’s airtight. Record yourself saying the script, then play it back as you navigate between slides.

So write the script, practice it, then on stage let it breathe, meaning you’ll have set with the material long enough to see a slide and know the point you’re trying to make and convey the message in your own words, ie speak extemporaneously.

And if none of this makes sense, I’ll just say my account was hacked and we’ll pretend this never happened.

Kapeesh?

Check out my slides and project here.
https://github.com/arober39/decor-agent

Connect with me on socials!
https://canva.link/spos9942r0u2nhy

How I Built an AI Content Factory That Sounds Like Me

Alexis Roberson — Fri, 07 Aug 2026 18:46:03 +0000

I used to spend hours rewriting AI-drafted video scripts that sounded nothing like me. At best, I might finish one or two that were just okay, but most of the work was still on my shoulders. Now, with my new system, I can get 15 scripts done in one session. They match my writing style, my voice, and my company's knowledge. Instead of full rewrites, each script just needs a quick review. I built the system in about a week, and the difference showed up before the week was out. The first project was an internal video series to teach people about the software factory and LaunchDarkly, and I barely had to edit those scripts. For the first time, the AI handles most of the work.

My recent projects have mostly involved agentic software delivery. The software factory is where all of it was heading, and my company spent months preparing to help customers build their own. We had internal material, public documentation, and ongoing conversations I wanted to add clarity to. So my challenge wasn't just learning, it was learning while producing content at the same time. Essentially, devrel.

By the end, I had built what I like to call my own personal content factory. At first, it was separate from the software factory it describes, but over time, the line between them blurred. My approach is to let AI handle the bulk of the work, while I step in for the important decisions. This is the only way I've found to make AI content sound like me. The AI creates the drafts, and I step in at three key points: checking the voice, the facts, and the overall feel.

The quality ceiling is set before the first draft.

I didn't just ask for a video script about a topic. Instead, I gave Claude access to every source I had—public docs, internal notes, and product requirements. It used research agents to read everything at once and came back with clear, organized notes I could use.

I set two important rules for this step. First, every part of the research was labeled as either public-safe or internal-only right from the start. When your sources mix private details with public information, you need to set those boundaries before turning anything into content. External scripts only use the public-safe material, and that rule is clear from the beginning. Second, if there's no specific detail, there's no script. Skipping the research means you get scripts that sound confident but say nothing. Real details are what make a technical video worth watching, and those only come from solid sources.

Curriculum before scripts

Before writing scripts, I had Claude turn the research into a curriculum with modules, learning goals, dependencies, and self-check questions at the end of each section. This step became the backbone for the whole series—defining episode boundaries, showing how concepts build, and preventing accidental repetition.

It's also the simplest and most valuable part of the whole system. If you take away just one step, make it this one—it only costs some text.

Drafts arrive sounding like a polished stranger.

Once the curriculum was ready, Claude drafted every episode. The scripts were well-structured, accurate, and sounded polished, though not quite like me. This is where most people stop, but with some patience and effort, it's worth pushing through.

When I first asked Claude to match my style, it just added my catchphrases to its usual writing. That wasn't enough. What really worked was having it break down my writing at the sentence level—looking at my punctuation, my rhythm, and how I connect ideas. Your voice comes from sentence structure, not just word choice. So I rewrote one script myself, making sure every sentence sounded like something I'd actually say. (Reading it out loud helps; you'll hear things your eyes miss.) That finished script became the example the system uses before drafting anything new.

Once I had a finished script to use as a gold standard, I could start automating the rest of the process.

The detour that became the greatest ramp-up, binging my own series

I still didn't fully understand the topic I was writing about. So I had Claude turn the draft scripts into avatar videos using HeyGen. I connected the two systems so the script went straight to the video renderer, no extra steps needed. Then I binge-watched a course on my own subject, presented by an avatar, using my own research.

This was the best way I've ever learned new material. Reading makes you an editor, judging each sentence. Watching turns you into a learner, experiencing the teaching. Ideas I'd read several times finally stuck after just one viewing. Each episode was also a screen test. If a video lost my attention, it meant I'd written something confusing—feedback I wouldn't have caught by just reading. I was my own test audience. The scripts that passed became the ones a real presenter used.

I wouldn't do this by default again because avatar rendering gets expensive—$200 in a single day for a couple dozen videos, mostly for my own learning. Next time, I'd only render videos destined for real content and stick to text for self-study.

Polish tools optimize for the page, not for meaning.

Once a script felt like it matched my style, I ran it through Grammarly. First, I checked for cohesion, then used the AI detector, rewriter, and Humanizer tools, usually with the Precisionist voice and sometimes the Naturalist setting.

Once, Grammarly changed my phrase "this step becomes free" to "this step becomes unnecessary," which completely reversed my point. These tools make writing smoother, but they don't always get the meaning right. After each edit, reread any sentence that states a fact, and pay attention to the conjunctions it adds. Words like "since" or "therefore" can suggest a cause-and-effect you never meant.

That's the second checkpoint. Nothing gets published until the facts are correct.

Finished work becomes the standard, and nothing derives from drafts.

Every finished script became the new voice standard, so future drafts matched my rhythm, transitions, and even where I put question marks. We wrote these rules into a living style guide. Each completed script made the next drafts better. Fine-tuning wasn't about changing prompts, but about locking in finished scripts until the standard matched my voice. This happened within a week. By the back half of the series, editing took just a few minutes instead of full rewrites.

One rule guides everything after this: LinkedIn clips, one-liners, and short videos are only made from finished, human-approved scripts, never from new drafts. The voice is already set in the finished work, so repurposing keeps it. Creating new content from scratch brings back that AI-generated sound.

That's the third checkpoint. Taste is built into the process, not just decided at the end.

One surprise from running everything in a single, ongoing session was that the AI became a better continuity editor than I am. When I rewrote my series introduction late in the process, Claude caught every reference it broke—episodes that pointed to things that no longer existed. Over 14 episodes, that's a level of consistency no human could keep up with.

The parallel I can't unsee

Research starts with clear sensitivity boundaries, and planning happens before anything is built. The AI produces more than any person could alone. Human judgment comes in at specific checkpoints, not everywhere. Only finished work sets the standard for future work—nothing new is based on unfinished drafts.

Our customers' software factories work the same way. Automation does most of the work, human judgment is built into the process, and nothing is released unless a person stands behind it. I set out to make content about this world and ended up proving its value by using it myself. The factory sets the pace, and the checkpoints ensure reliability. Both are needed.

The AI didn't replace my writing. I see it as a co-intelligence that speeds up the process between drafts. It never decides what gets published—that's still my job.

OpenTelemetry for LLM Applications: A Practical Guide with LaunchDarkly and Langfuse

Alexis Roberson — Wed, 04 Mar 2026 03:11:34 +0000

Originally published in the LaunchDarkly Docs

LLM applications have a telemetry problem. Unlike traditional software where you can trace a bug to a specific line of code or a failed API call, LLM failures are a bit more nuanced. A response that's slightly off, a prompt that worked yesterday but not today, or a model swap can quietly degrade your user experience. OpenTelemetry gives you a structured way to pull back the curtain by capturing token usage, model metadata, latency, and agent responses so you truly know what's happening inside your application.

This tutorial walks you through instrumenting a real LLM application with OTel spans, capturing the right attributes, and fanning out those traces simultaneously to Langfuse and LaunchDarkly's Guarded Releases. Both are LLM observability tools, but they give you different lenses on the same trace data. Langfuse is purpose-built for prompt debugging and cost analysis — surfacing prompt content, completions, and per-agent token usage.

LaunchDarkly connects that same trace data to the specific model variant that was active during a request, giving you flag-correlated observability with automated rollback if a variant starts degrading your users' experience. One OTel collector, two complementary views, no custom integrations required.

Guarded releases are LaunchDarkly's observability solution that encompasses application performance thresholds, release auto-remediation, and release monitoring, along with error monitoring and session replay.

The WorkLunch App

In order to see the full process of instrumenting an LLM application, I added a new feature in an app called WorkLunch where users were able to create/join office communities and swap lunches based on preference. Now they're also able to improve the description field of their lunch post to make it more appealing to potential swappers and receive recommendations for compatible swaps.

So in the initial description you may write, "Grilled cheese sandwich", then click the AI Suggest button. The app replaces it with, "Golden, buttery grilled cheese with perfectly melted cheese sandwiched between crispy white bread. This comfort food classic is grilled to perfection with a satisfying crunch on the outside and gooey, cheesy goodness on the inside. Simple, delicious, and guaranteed to hit the spot!"

Now which lunch post are you more than likely to click on?

This subtle addition takes the app from a fun, simple lunch swap experience to a viable LLM application that still requires the same visibility and observability of traditional systems. OpenTelemetry allows you to extract the necessary data like token count, model name, agent responses, etc in order to properly debug system failures.

Multi-Agent Architecture

The WorkLunch backend uses 3 agents to rewrite the lunch post description and find good lunch swaps.

The orchestrator coordinates the other two agents. It receives the user's request and the model type, calls the description agent first, then passes the generated description into the match agent. It acts as the parent span that ties the whole chain together.
The description agent takes the user's sparse lunch post input and calls Claude to generate an appealing 2-3 sentence description.
The match agent takes the user's lunch post (including the description just generated) plus a list of other active posts in the community, and uses AI to suggest 2-3 posts that would make good swaps.

These features are controlled by two feature flags, one for enabling the AI suggest feature and the other to control which model version the app uses. Every layer gets its own OTel span, creating a trace tree that shows the full request lifecycle.

Prerequisites

Before you start, you'll need the following installed locally and accounts set up with the services below.

Docker Desktop
Node.js / Npm
An Anthropic account (or openai)
A LaunchDarkly account (sdk key and access token)
A Langfuse account (public and secret key)
A Supabase account (database configuration)

Environment variables

Once you're all setup, clone the WorkLunch repo. Copy the example .env file and fill in your values:

cp .env.example .env

Your .env should contain:

# .env

# Supabase (required for the app)
EXPO_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
EXPO_PUBLIC_SUPABASE_ANON_KEY=your-anon-key

# LaunchDarkly client-side (required for feature flags in the frontend)
EXPO_PUBLIC_LAUNCHDARKLY_SDK_KEY=mob-your-mobile-key
EXPO_PUBLIC_LAUNCHDARKLY_CLIENT_SIDE_ID=your-client-side-id

# AI Backend URL (where docker compose runs the Python backend)
EXPO_PUBLIC_AI_BACKEND_URL=http://localhost:8000

# --- Docker Compose vars (used by the backend + otel-collector) ---

# Anthropic API key for Claude
ANTHROPIC_API_KEY=sk-ant-your-key-here

# LaunchDarkly server-side SDK key (starts with sdk-, NOT mob-)
LD_SDK_KEY=sdk-your-key-here

# Langfuse auth — Base64 of "public_key:secret_key" (keep on one line)
LANGFUSE_AUTH_HEADER=your-base64-encoded-string

Supabase setup

Create a new Supabase project and grab your Project URL and Anon key from Dashboard → Settings → API
Run the migration files in supabase/migrations/ to create the database schema. Execute them in order in the Supabase Dashboard → SQL Editor:

supabase/migrations/

20240101000000_initial_schema.sql        ← tables: profiles, spaces, posts, proposals, messages, trades
20240101000001_rls_policies.sql          ← row-level security policies
20240101000002_storage_setup.sql         ← storage bucket for post photos
20240101000003_disable_email_confirmation.sql  ← simplifies local dev auth
20240205000000_fix_space_memberships_rls_recursion.sql
20240205000001_spaces_delete_policy.sql
20240206000000_space_creator_as_admin.sql
20240206100000_delete_space_rpc.sql

Be sure to run each file in order as later migrations depend on tables and policies from earlier ones.

LaunchDarkly setup

Create two feature flags in your new LaunchDarkly worklunch project:

ai-suggest-enabled — Boolean flag, client-side. Gates visibility of the AI Suggest button in the frontend. Set it to true for users you want to test with.
llm-model-variant — String flag, server-side. Controls which Claude model the backend uses. Set the default value to claude-sonnet-4-20250514. Add a variation for claude-haiku-4-5-20251001 if you want to experiment with a faster/cheaper model.

Langfuse setup

Create a new project in Langfuse (note whether your project URL starts with us.cloud.langfuse.com or cloud.langfuse.com — this determines your region)
Go to Project Settings → API Keys and create a new key pair
Generate your Base64 auth header and place it inside your .env file:

echo -n "pk-lf-your-public-key:sk-lf-your-secret-key" | base64

Quick Start

Once your .env is configured:

# Install frontend dependencies
npm install

# Start the OTel Collector + Python backend
docker compose up --build

# In a separate terminal, start the Expo dev server
npm run web

Verify traces are flowing by checking the collector logs:

docker compose logs -f otel-collector

You should see spans with gen_ai.* attributes and feature_flag events printed by the debug exporter.

Now, let's take a look at how each agent is instrumented to send spans to LaunchDarkly.

Step 1: Instrument your LLM application

Initialize the Tracer and Application

The FastAPI app sets up OTel, LaunchDarkly, CORS, and auto-instrumentation in a single lifespan handler:

# backend/app/main.py
from contextlib import asynccontextmanager

import ldclient
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from app.config import settings
from app.routers.suggest import router as suggest_router

def setup_otel() -> None:
    """Configure OpenTelemetry with OTLP gRPC exporter."""
    provider = TracerProvider()
    provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(endpoint=settings.OTEL_EXPORTER_ENDPOINT, insecure=True)
        )
    )
    trace.set_tracer_provider(provider)

def setup_launchdarkly() -> None:
    """Initialize LaunchDarkly server SDK."""
    config = ldclient.Config(settings.LD_SDK_KEY)
    ldclient.set_config(config)

@asynccontextmanager
async def lifespan(app: FastAPI):
    setup_otel()
    setup_launchdarkly()
    yield
    ldclient.get().close()
    provider = trace.get_tracer_provider()
    if hasattr(provider, "shutdown"):
        provider.shutdown()

app = FastAPI(title="WorkLunch AI Backend", lifespan=lifespan)

# CORS
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Instrument FastAPI with OTel
FastAPIInstrumentor.instrument_app(app)

# Routes
app.include_router(suggest_router, prefix="/api/v1")

@app.get("/health")
async def health():
    return {"status": "ok"}

The Route: Flag evaluation + feature flag span event

The FastAPI route is where the LaunchDarkly flag gets evaluated. The feature_flag span event on this span is what LaunchDarkly's observability layer looks for when correlating traces with flag evaluations.

# backend/app/routers/suggest.py
import ldclient
from fastapi import APIRouter
from opentelemetry import trace

from app.agents import orchestrator
from app.models import SuggestRequest, SuggestResponse

router = APIRouter()
tracer = trace.get_tracer("worklunch.routers.suggest")

DEFAULT_MODEL = "claude-sonnet-4-20250514"


@router.post("/suggest", response_model=SuggestResponse)
async def suggest(request: SuggestRequest) -> SuggestResponse:
    with tracer.start_as_current_span("suggest.endpoint") as span:
        # Evaluate the model variant flag
        ld_client = ldclient.get()
        context = ldclient.Context.builder("worklunch-backend").kind("service").build()
        model = ld_client.variation("llm-model-variant", context, DEFAULT_MODEL)

        # Emit the feature_flag span event — this is what LD correlates with
        span.add_event(
            "feature_flag",
            {
                "feature_flag.key": "llm-model-variant",
                "feature_flag.provider.name": "LaunchDarkly",
                "feature_flag.variant": str(model),
            },
        )
        span.set_attribute("gen_ai.request.model", model)

        # The flag-controlled model flows into the orchestrator
        description, matched_posts = await orchestrator.run(request, model)

    return SuggestResponse(
        suggested_description=description,
        matched_posts=matched_posts,
    )

The Orchestrator: Parent span for the agent chain

The orchestrator creates a parent span and calls each sub-agent sequentially. Because the sub-agent spans are created while the orchestrator span is active, OTel automatically nests them as children.

# backend/app/agents/orchestrator.py
from opentelemetry import trace

from app.agents.description_agent import generate_description
from app.agents.match_agent import find_matches
from app.models import MatchedPost, SuggestRequest

tracer = trace.get_tracer("worklunch.orchestrator")

async def run(
    request: SuggestRequest, model: str
) -> tuple[str, list[MatchedPost]]:
    with tracer.start_as_current_span("orchestrator.run") as span:
        span.set_attribute("orchestrator.model", model)
        span.set_attribute("orchestrator.title", request.title)
        span.set_attribute("orchestrator.active_posts_count", len(request.active_posts))

        # Step 1: Generate description
        description = await generate_description(request, model)

        # Step 2: Find matches using the generated description
        matched_posts = await find_matches(
            title=request.title,
            description=description,
            category=request.category,
            dietary_preferences=request.dietary_preferences,
            active_posts=request.active_posts,
            model=model,
        )

        span.set_attribute("orchestrator.matches_found", len(matched_posts))

    return description, matched_posts

The Description Agent: LLM Call with genAI semantic conventions

This is where the OTel GenAI Semantic Conventions come in. The conventions define a standard schema for LLM spans — gen_ai.system, gen_ai.request.model, gen_ai.usage.*, and prompt/completion content as span events.

# backend/app/agents/description_agent.py
import json

import anthropic
from opentelemetry import trace

from app.config import settings
from app.models import SuggestRequest

tracer = trace.get_tracer("worklunch.agents.description")

async def generate_description(request: SuggestRequest, model: str) -> str:
    client = anthropic.Anthropic(api_key=settings.ANTHROPIC_API_KEY)

    system_prompt = (
        "You are a helpful assistant that writes appealing, concise lunch descriptions "
        "for a lunch-swapping app. Given a title and optional details, write a 2-3 sentence "
        "description that makes the lunch sound appetizing and highlights what makes it special. "
        "Mention any dietary info naturally if provided. Keep it friendly and casual."
    )

    user_content_parts = [f"Lunch title: {request.title}"]
    if request.description:
        user_content_parts.append(f"Current description: {request.description}")
    if request.category:
        user_content_parts.append(f"Category: {request.category}")
    if request.dietary_preferences:
        user_content_parts.append(f"Dietary preferences: {request.dietary_preferences}")
    if request.allergies:
        user_content_parts.append(f"Allergies to note: {request.allergies}")

    user_content = "\n".join(user_content_parts)
    messages = [{"role": "user", "content": user_content}]

    with tracer.start_as_current_span("description_agent.generate") as span:
        # GenAI semantic conventions — provider and request attributes
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.max_tokens", 256)
        span.set_attribute("gen_ai.request.temperature", 0.7)

        # Log prompt as a span event (keeps large payloads out of the attribute index)
        span.add_event(
            "gen_ai.content.prompt",
            {"gen_ai.prompt": json.dumps(messages)},
        )

        response = client.messages.create(
            model=model,
            max_tokens=256,
            temperature=0.7,
            system=system_prompt,
            messages=messages,
        )

        result = response.content[0].text

        # Response attributes — model identity, finish reason, token usage
        span.set_attribute("gen_ai.response.model", response.model)
        span.set_attribute(
            "gen_ai.response.finish_reasons", [response.stop_reason or "end_turn"]
        )
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)

        # Log completion as a span event
        span.add_event(
            "gen_ai.content.completion",
            {"gen_ai.completion": result},
        )

    return result

The Match Agent: Structured JSON output from an LLM

The match agent follows the same GenAI span pattern but with different parameters (lower temperature for more deterministic output, higher token budget for JSON) and post-processing to parse structured JSON from the LLM response.

# backend/app/agents/match_agent.py
import json

import anthropic
from opentelemetry import trace

from app.config import settings
from app.models import ActivePost, MatchedPost

tracer = trace.get_tracer("worklunch.agents.match")

async def find_matches(
    title: str,
    description: str,
    category: str | None,
    dietary_preferences: str | None,
    active_posts: list[ActivePost],
    model: str,
) -> list[MatchedPost]:
    if not active_posts:
        return []

    client = anthropic.Anthropic(api_key=settings.ANTHROPIC_API_KEY)

    system_prompt = (
        "You are a lunch-matching assistant. Given a user's lunch post and a list of "
        "active posts from other users, suggest 2-3 posts that would make good swaps. "
        "Consider complementary flavors, dietary compatibility, and variety. "
        "Respond with valid JSON only — an array of objects with keys: "
        '"post_id", "title", "reason". Keep reasons to one short sentence.'
    )

    posts_text = "\n".join(
        f"- ID: {p.id}, Title: {p.title}, Description: {p.description}, "
        f"Category: {p.category}, By: {p.user_name}"
        for p in active_posts
    )

    user_parts = [
        f"My lunch: {title}",
        f"Description: {description}",
    ]
    if category:
        user_parts.append(f"Category: {category}")
    if dietary_preferences:
        user_parts.append(f"My dietary preferences: {dietary_preferences}")
    user_parts.append(f"\nAvailable posts to match with:\n{posts_text}")

    user_content = "\n".join(user_parts)
    messages = [{"role": "user", "content": user_content}]

    with tracer.start_as_current_span("match_agent.find") as span:
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.max_tokens", 512)
        span.set_attribute("gen_ai.request.temperature", 0.3)

        span.add_event(
            "gen_ai.content.prompt",
            {"gen_ai.prompt": json.dumps(messages)},
        )

        response = client.messages.create(
            model=model,
            max_tokens=512,
            temperature=0.3,
            system=system_prompt,
            messages=messages,
        )

        raw = response.content[0].text

        span.set_attribute("gen_ai.response.model", response.model)
        span.set_attribute(
            "gen_ai.response.finish_reasons", [response.stop_reason or "end_turn"]
        )
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)

        span.add_event(
            "gen_ai.content.completion",
            {"gen_ai.completion": raw},
        )

    # Parse the JSON response
    try:
        cleaned = raw.strip()
        if cleaned.startswith("```

"):
            cleaned = cleaned.split("\n", 1)[1]
            cleaned = cleaned.rsplit("

```", 1)[0]
        matches_data = json.loads(cleaned)
        return [MatchedPost(**m) for m in matches_data[:3]]
    except (json.JSONDecodeError, KeyError, TypeError):
        return []

For each of these agents, Langfuse receives the full trace including prompt/completion content for debugging. LaunchDarkly receives the same trace and correlates the feature_flag event with the HTTP span for experimentation metrics.

Step 2: Configure the OTel collector

This is where the fan-out happens. The collector receives traces over OTLP and exports them to both backends simultaneously. Two pipelines from the same receiver is the key: you configure one receivers block and reference it in multiple pipelines — no duplication of ingestion, no changes needed in application code.

otel-collector-config.yaml

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

  # Stamp traces with the LD project identifier so the endpoint
  # knows which project they belong to
  resource/launchdarkly:
    attributes:
      - key: launchdarkly.project_id
        value: "${env:LD_SDK_KEY}"
        action: upsert

exporters:
  # Langfuse — LLM-specific traces with full prompt content
  otlphttp/langfuse:
    endpoint: https://us.cloud.langfuse.com/api/public/otel
    headers:
      Authorization: "Basic ${env:LANGFUSE_AUTH_HEADER}"

  # LaunchDarkly — flag-correlated observability
  # No auth header needed; identification is via the
  # launchdarkly.project_id resource attribute
  otlphttp/launchdarkly:
    endpoint: https://otel.observability.app.launchdarkly.com

  # Debug exporter for local development
  debug:
    verbosity: detailed

service:
  pipelines:
    # Pipeline 1: Full LLM traces to Langfuse (includes prompt content)
    traces/llm-observability:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/langfuse]

    # Pipeline 2: Flag-correlated traces to LaunchDarkly
    traces/feature-flags:
      receivers: [otlp]
      processors: [resource/launchdarkly, batch]
      exporters: [otlphttp/launchdarkly]

    # Pipeline 3: Debug output for development
    traces/debug:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

Now if we rerun our application, we should see LaunchDarkly Traces capturing the Otel spans.

docker compose up --build
npm run web # in a separate terminal

How LaunchDarkly processes OTel traces

LaunchDarkly receives traces for logging and converts OTel span data into events for use with Experimentation and Guarded Rollouts. The process works like this:

Your application emits a span that covers an HTTP request (or LLM call). This span carries standard HTTP attributes: http.response.status_code, http.route, latency derived from span duration.
On that same span (or a parent span in the same trace), you've emitted a feature_flag span event with feature_flag.key and feature_flag.variant.
LaunchDarkly's collector ingests the trace and looks for HTTP spans that overlap with spans containing at least one feature_flag event. When it finds a match, it produces a metric event associating the flag variant with the observed latency and error rate (5xx status codes).
Those metric events flow into Experimentation, where they become the outcome metrics for your flag-controlled A/B test — for example, comparing claude-sonnet-4-20250514 vs claude-haiku-4-5-20251001 on p95 latency and error rate without writing a single line of custom metric instrumentation.

Every span in the agent chain is nested under a single trace. The collector fans out that trace to both backends simultaneously. Langfuse gets the full LLM details for prompt debugging and cost analysis. LaunchDarkly gets the flag-correlated signal it needs for automated rollout decisions.

Key attributes from gen_ai trace spans

Step 3: Trigger a guarded rollout

With traces flowing into LaunchDarkly and span events carrying your flag evaluations, you can now configure a Guarded Rollout that automatically rolls back the AI Suggest feature if token costs spike or response truncation increases as you increase the percentage of users who see it.

In the LaunchDarkly UI, navigate to your flag (ai-suggest-enabled), under Default rule click Edit and select Guarded Rollout.

You'll need to create two new custom metrics to attach to the guarded rollout. The first will is the AI tokens total rollout metric. This will measure cost-per-request as a gate for releasing the feature to a wider audience and alert if average tokens per request exceeds your baseline by more than 25%. And the second is AI completion truncated error metric, which will catch prompt truncation before users notice degraded output quality. This metric will halt the rollout if the rate climbs above your control baseline.

For ai.tokens.total:

Event kind: Custom
Event key: ai.tokens.total
What do you want to measure?: Value / Size (Numeric) — you're passing the actual token count as the magnitude
Metric name: AI tokens total

For ai.completion.truncated:

Event kind: Custom
Event key: ai.completion.truncated
What do you want to measure?: Occurrence (Binary) — you're tracking whether truncation happened at least once, not how many times
Metric name: AI completion truncated

Select the two newly created metrics.

Set the threshold to 25 percent for 1 week.

Click Save.

LaunchDarkly will now monitor both metrics against the ai-suggest-enabled flag and trigger an automatic rollback if either threshold is breached.

What You've Built

At this point you have a fully instrumented LLM application where every layer of the stack tells a story. The FastAPI route evaluates a LaunchDarkly flag and stamps the result onto the trace. The orchestrator creates a parent span that ties the entire agent chain together. Each agent makes a Claude API call and records exactly what was sent, what came back, and how many tokens it cost. The OTel Collector fans all of that out to two backends simultaneously without a single line of application code changing between them.

Langfuse gives you the LLM-specific view: prompt content, completions, token usage, and latency per agent so you can debug why a description came out wrong or whether the match agent is consistently burning more tokens than expected. LaunchDarkly gives you the experimentation view: which model variant was active during a given request, how latency and error rates compare between claude-sonnet-4-20250514 and claude-haiku, and the automated safety net to roll back if a new variant starts degrading your users' experience. Both tools are consuming the same trace. Neither required a custom integration.

Conclusion

LLM applications fail in ways that traditional monitoring wasn't designed to catch. OpenTelemetry gives you the standard schema to do that, and the collector architecture gives you the flexibility to route that signal wherever it's most useful.

If you're building anything with LLMs in production, start here. Instrument at the agent level, follow the GenAI semantic conventions, and build your observability pipeline before you need it.

The full source code for the WorkLunch app is available here. Clone it, swap in your API keys, and you'll have a working multi-agent trace pipeline running locally in under ten minutes.

Additional resources

Detection to Resolution: Real world debugging with rage clicks and session replay

Alexis Roberson — Wed, 04 Mar 2026 03:00:54 +0000

Originally published in the LaunchDarkly Docs

Part 3 of 3: Rage Click Detection with LaunchDarkly

Overview

In Part 1, we set up rage click detection using LaunchDarkly's Session Replay. In Part 2, we connected those frustration signals to guarded releases for automated rollback protection.

Now it's time to put it all together. In this final installment, we'll walk through real-world debugging scenarios using our WorkLunch app—a cross-platform application built with React Native and Expo where coworkers can swap lunches.

These scenarios demonstrate how the integrated system of feature flags, session replay, and guarded releases can transform the way you diagnose and fix production issues.

All code for this blog post can be found here.

The Debugging Workflow: An Observability Loop

Before diving into scenarios, let's understand the complete workflow we've built:

This workflow enables you to:

Detect frustration signals automatically as they happen.
Alert when thresholds are breached during rollouts.
Investigate with full session context, not just error logs.
Fix with confidence knowing the exact user experience.
Verify the fix works by monitoring the new rollout.

When a guarded release detects a spike in rage clicks, it automatically correlates the frustration with specific flag variations:

The alert tells you exactly which feature caused the problem: "Rage clicks increased 10x for users on join-community-redesign: true" It’s important to note that rage clicks aren’t the only measure for frustration signals around user experience.

When items are out of place, confusing or navigation is not consistent across mobile and web, it can cause users to rage scroll. Unlike rage clicks that can be measured and tracked, rage scrolls are a little more nuanced, which will result in relying heavily on session replay and drops and traffic to diagnose.

Just as llm observability tools monitor model outputs for hallucinations or drift, rage click detection monitors the human layer, catching UX failures that no server-side metric would reveal.

Using feature flags, session replay and guarded releases, let’s take a closer look at both scenarios to debug in a real-world worklunch app.

Scenario 1: Form Validation Frustration

The Setup

Your team ships a new "Create Community" button redesign and places the feature behind a feature flag called (inline-form-validation).

After the feature flag is toggled on, you notice users on this form are rage clicking the Create Community button. Your metrics show community creation conversions dropped 15 percent since enabling the flag, but there are no errors in the logs.

Thankfully, you connected your feature flag with a Guarded Release and can use it to debug the issue.

Investigation Steps

Step 1: Search for sessions with rage clicks

Since we attached a Guarded Rollout to your feature flag using a metric for detecting rage clicks, we can navigate to the LaunchDarkly’s Session page and filter for those specific sessions.

In the LaunchDarkly Sessions tab search bar, apply this filter:

has_rage_clicks=true featureflag.key=inline-form-validation

This query filters for sessions with rage clicks and where the inline-form-validation flag is enabled.

Step 2: Watch for user behavior patterns

In the session replay, you see:

User fills out the form completely.
User clicks Create Community.
User sees a loading spinner and nothing else.
User clicks button repeatedly.
User abandon forms altogether.

Step 3: Identify the UX failure
The community was being created successfully meaning the API returned a 200 and the new community existed. The new feature was intended to simply change the button color (baby blue → purple), but it broke the code that allows the success card to be shown.

The Root Cause

When the flag is off, the code does the right thing after a successful create:

Success → Set createdSpace → Show verification card (name, join code, "Back to community list")
User taps Back → Return to community list

When the flag is on, the success path was missing:

Success → Nothing. The if (!useInlineValidation) block runs only for the old flow, so when the flag is on, setCreatedSpace is never called.
The user stays on the form with no confirmation, no join code, and no way to know the create worked.

// Only the control (flag OFF) got success UI; the treatment (flag ON) did not
if (!useInlineValidation) {
  setCreatedSpace({ name: space.name, join_code: space.join_code });
}

The Fix

To fix the issue, you can either rollback to previous working code or fix the issue and toggle flag back on.

Option A – Roll back the flag: Set inline-form-validation to off in LaunchDarkly. Users are no longer on the new feature; they get the existing, working code path and see the verification card after Create Community.

Option B – Fix the new feature so it no longer breaks the success card: The new feature (flag on) breaks the code that shows the success card. Fix it by adding the success handling to the flag-on path. Also, when the flag is on, also call setCreatedSpace after a successful create so the success card is shown again.

Resolution

With the root cause identified from Session Replay, you update the code and resolve the rollout:

Roll back inline-form-validation to false so users get the verification card and can return to the list (instant, no deployment needed).
Fix the flag-on success path so it shows the verification card (or redirects) after create.
Deploy and re-enable the flag with the guarded rollout (rage click metrics) still attached, then monitor to confirm rage clicks return to baseline.

Time to resolution: 30 minutes to an hour (compared to potentially days of user complaints and support tickets).

Scenario 2: The Infinite Scroll Frustration

The Setup

Your team ships a feature flag (new-filter-location) that moves the existing Category and Sort controls on the "Lunches for Swap" feed from the top of the list to the bottom.

Everything works fine until you notice a drop in traffic.

Since you see no uptick in rage clicks for your Guarded Release, you know it must be another user signal you’re missing that could indicate user frustration. So you decide to investigate further by placing yourself in the user's shoes using session replay.

Investigation Steps

Step 1: Filter by flag variation

feature_flag.key=new-filter-location feature_flag.result.value=true

This query will filter for sessions where the feature flag new-filter-location is set to true.

In session replay, you see:

users scroll down the feed, then back up, then down again.
Minimal item taps.
Constantly moving through the list without making selections.
User navigates to a different page.

The pattern suggests they're looking for something rather than browsing items.

Step 3: Identify UX Failure
Replay shows the issue: “Category” and “Sort” are not at the top where users expect them. With the flag on, the controls were moved to below every post in the feed.

The Root Cause

The feature flag moves the existing filter bar from the top to the bottom of the list (ListFooterComponent when the flag is on). Same controls, different position.

// Flag ON: same filter bar rendered as ListFooterComponent — breaks user expectation
ListFooterComponent={
  useFiltersAtBottom ? (
    <View style={styles.filterBar}>...</View>
  ) : null
}

The Fix

Step 1 – Roll back the flag: Set new-filter-location to false. Filters return to the top (original position); rage scrolls drop.

Step 2 – Fix the experiment: Don't move the filters to the bottom. Keep the filter bar at the top regardless of the flag, or remove the flag and leave filters in their original place.

// Fix: always show filter bar at top (original place)
{/* Filters always above the list */}
<View style={styles.filterBar}>
  <TouchableOpacity style={styles.filterButton} onPress={openCategoryFilter}>
    <Text style={styles.filterButtonText}>Category: {categoryLabel}</Text>
  </TouchableOpacity>
  <TouchableOpacity style={styles.filterButton} onPress={openSortMenu}>
    <Text style={styles.filterButtonText}>Sort: {sortOrder === 'newest' ? 'Newest' : 'Oldest'}</Text>
  </TouchableOpacity>
</View>
<FlatList ... ListFooterComponent={null} />

Resolution

Roll back new-filter-location to false so filters are at the top again
Fix keep the filter bar at the top.
Deploy and re-enable the flag with monitoring.

Building your debugging playbook

Based on these scenarios, here's a systematic approach to rage click debugging:

Step 1: Triage with Filters
| What to look for | Search query |
| --- | --- |
| All frustrated users | has_rage_clicks=true |
| Specific page issues | has_rage_clicks=true visited-url="*/spaces/create*" |
| Browser-specific | has_rage_clicks=true browser_name=Chrome |
| High frustration | has_rage_clicks=true active_length>180s |

Step 2: Identify the Pattern
When reviewing session replays, look for:

Visual feedback gaps (Did the UI acknowledge the click?)
Loading states (Is there a spinner? Does it ever resolve?)
Error visibility (If there's an error, can the user see it?)
Touch target issues (On mobile, are elements too small or overlapping?)
Timing problems (Does the click happen before the element is ready?)

Step 3: Correlate with Technical Data
Session replay shows you the user's experience. Pair it with:

Network tab (API response codes and payloads.)
Console errors (JavaScript exceptions.)
Feature flag state (Which variation was the user seeing?)
Timing (When in the session did frustration peak?)

Step 4: Fix and Verify

Roll back using your feature flag (instant, no deployment needed).
Fix the root cause in code.
Re-deploy with the guarded rollout active.
Monitor rage click metrics to confirm the fix worked.

Whether you're shipping traditional features or AI-powered experiences, this playbook reflects a devops for ai mindset: continuous monitoring, fast rollback, and data-driven debugging that closes the gap between deployment and user impact.

Bringing it all together

The combination of rage click detection, session replay, and guarded releases creates something powerful: observability that starts with the human experience.

Traditional monitoring asks: "Is the system healthy?"

This approach asks: "Are users successful?"

When you can detect frustration in real-time, watch exactly what users experienced, and roll back problematic features instantly, you fundamentally change how fast you can ship with confidence.

This workflow fits into the broader ai deployment lifecycle, treating user frustration signals as first-class deployment metrics, just like error rates or latency. By embedding rage click detection into your ai lifecycle management strategy, every release becomes a feedback loop that improves both the product and the process.

The next time your users are frustrated, you'll know exactly what went wrong, which users were affected, and why. And the best part is you'll fix it before most users even notice.

Additional Resources

Day 7 | 🎄✨The Rockefeller tree in NYC: SLOs that actually drive decisions

Alexis Roberson — Wed, 17 Dec 2025 04:47:02 +0000

Originally published in the LaunchDarkly Docs

Most Subject Level Objectives or SLOs sit in dashboards gathering dust. Seeing that SLOS are performance targets that can be measured, they're extremely important in understanding the quality of a service or system. You define them, measure them, but when your conditions are not met, there’s no followup.

The biggest drawback is that SLOs are created to add value but if they’re never reinforced, it’s impossible to drive decisions, influence roadmaps, or help during incidents.

When it comes to defining SLOs, many folks often start at the top of the funnel by picking general metrics to measure, but in order to create SLOs that work, it’s important to understand how the roots impact the leaves.

In this post, we'll cover the pitfalls leading to out of sync SLOs with a few tips and tricks to ensure what you measure produces business value. You'll also see an example of how to set SLOs in realtime for a flag evaluation feature that you can implement in your own planning process.

But first, we'll explore a tree metaphor to recap key observability components and how they're influence expands from the roots all the way to the leaves. What if the popular Rockefeller tree in NYC represented the relationship between telemetry data and SLOs?

Understanding the Observability Tree

SLOs would essentially be the leaves on the Response branch as shown in the above image, meaning they are visible, measurable targets that everyone can see. The leaves would not be possible without the support of the trunk.

This is your telemetry data or the traces, logs, events you collect from your system. The data you collect acts as the foundation and support for the branches and leaves.

The roots represent the things you cannot see but are still vital to overall health of your system. This is the hard part of understanding system behavior, debugging unknown unknowns, and making data-driven decisions.

Most teams skip the roots entirely. They define SLOs using only the trunk (logs, traces, events), measuring things that can already be measured. However, the business outcomes and user behaviors are buried in what would be defined as the roots. Sometimes you have to dig through the soil to ensure your SLOs don't end up technically accurate yet strategically ineffective.

What Makes an SLO Decision-Worthy

So what makes a good SLO? The goal of a SLO is to bridge engineering and business needs to support a high quality user experience. And a good SLO depends on three things, business clarity or asking the right questions, measurability, or can these components actually be measured, and actionable targets, or the game plan for when things go wrong.

First, you need business clarity, which are the roots of the previously mentioned observability tree. This means articulating why something matters in concrete terms like dollars, users, retention and also avoiding vague statements like "uptime is important." For instance, if I were measuring the impact of downtime on a checkout feature, I could establish the SLO scope with “each minute of checkout downtime costs us $12,000 in lost revenue based on our average transaction volume.” It is essential to be able to explain the business impact in one clear sentence.

Second, you need measurability. This is like the trunk of the tree. Your SLO must connect to your golden signals such as latency, traffic, errors, saturation. This is where a lot of aspirational SLOs fall apart. Upper management might want to measure user happiness, but how can engineering translate this into actual metrics? Try to express the business impact in one clear sentence. If that's difficult, it’s usually a sign the problem definition needs a bit more shaping before defining the SLO.

Third, you need actionable targets, which represent the leaves on the observability tree. This is where most SLOs fail even when they get the first two right. There's a number, maybe even a threshold, but no clear action plan. What happens when you miss it? Who gets paged? What gets paused? Decision-worthy SLOs specify exactly what happens at different levels of degradation, and more importantly, they give everyone the confidence to make decisions based on those levels.

Building production resilient SLOs: LaunchDarkly’s Flag evaluation example

We can apply these same principles of building a production-worthy SLO using LaunchDarkly’s flag evaluation feature.

The flag evaluation feature in the monitoring tab is an extension of observability where it tracks how often each flag variation is served to different contexts over time, and highlights flag changes that might affect evaluation patterns.

Now, let’s build a SLO.

Step 1: Start with the business question

What would be impacted if the flag evaluations monitoring feature broke? Customers use these charts to understand rollout progress, debug targeting issues, and verify that their flags are working as expected. If evaluation data is delayed or missing, they can't trust what they're seeing. They might roll back a working feature thinking it's broken, or fail to catch a real problem because the charts show stale data. This undermines confidence in the platform and increases support load.

Step 2: Translate to user experience terms

What does "working well" look like? When a customer makes a flag change and checks the monitoring tab, they see updated evaluation counts within a couple minutes. The charts load quickly (under 3 seconds). The data is accurate meaning evaluation counts match what's actually happening in their application. If there's a delay, we tell them explicitly rather than showing stale data as if it's current.

Step 3: Connect to telemetry

We track several golden signals for this feature.

Data pipeline latency: time from evaluation event to appearing in charts.
Chart load time: how long it takes to render the monitoring page.
Data accuracy: comparing our recorded evaluations against a known sample.
Error rate: failed queries or chart rendering errors.

For the sake of this example will set arbitrary numbers for these signals. Let’s say you had a median pipeline latency of 45 seconds, p95 at 2 minutes, p99 at 5 minutes. And a chart load time averages 1.2 seconds. Data accuracy is 99.7 percent (some evaluations drop due to sampling) and error rate is 0.3 percent.

Using this data, we can set the target.

Step 4: Set the target

Based on that data, here's our SLO: 98 percent of flag evaluation events will appear in monitoring charts within 3 minutes, with chart load times under 3 seconds at p95.

Why these numbers? Customer research shows they expect "near real-time" monitoring, which they define as 2-3 minutes. Anything longer feels like stale data. Three seconds for chart loading is the threshold where users perceive delay and start questioning if something's broken.

We chose 98 percent instead of 99.9 percent because some evaluation events get sampled out intentionally for cost reasons, and occasional data pipeline delays from third-party dependencies are acceptable.

Now that we have our targets, we can use those thresholds to set conditional responses based on alerts or indicators.

Step 5: Define operational responses

Responses for Green, Red, or Yellow indicators in production:

If Green (>98%, <3 min, <3 sec load), continue normal operations.
If Yellow (95-98%, or 3-5 min, or 3-5 sec load), alert on-call, investigate within 4 hours.
If Red (<95%, or >5 min, or >5 sec load), page immediately, update status page if widespread.

Step 6: Drive decisions

Now the SLO becomes your decision-making framework. When engineering proposes adding a new feature like "evaluations by SDK" breakdown, the first question is: "Will this keep us within our 3-second chart load SLO?" If the answer is no, we either optimize the implementation or push back on the feature.

Infrastructure changes get evaluated the same way. Before migrating the data pipeline to a new system, we load tests against both our latency and accuracy targets. If the migration risks our SLO, we either fix the architecture or delay the migration. Another way I've seen SLOs used is planning future work. ie. if a team knows they are in the yellow this month, they may avoid picking up other risky work.

The SLO transforms from a monitoring target into a decision filter, helping to determine what gets shipped and what doesn’t.

Bringing it all together

Great SLOs aren't just leaves you pluck and add to dashboards. They're connected to everything below them from the trunk of solid telemetry to the roots of understanding what actually matters to your business and users. If you skip those foundational layers, your SLOs become technically accurate but strategically useless.

Start with the roots. Ask what would be impacted if this feature were to break. Work your way up through user experience and technical measurement. Build SLOs that bridge engineering and business with clear thresholds and clear consequences. And finally, make them specific enough to drive real decisions.

Day 6 | 💸 The famous green character that stole your cloud budget: the cardinality problem

Alexis Roberson — Tue, 16 Dec 2025 01:17:54 +0000

Originally published in the LaunchDarkly Docs

Every December, engineering teams unwrap the same unwanted gift: their annual observability bill. And every year, it's bigger than the last.

You know the pattern. Services multiply. Traffic grows. Someone discovers OpenTelemetry and suddenly every microservice is emitting 50 spans per request instead of 5. Then January rolls around and your observability platform sends an invoice that's 30% higher than last quarter.

Your VP of Engineering wants to know why.

You could blame it on the famous green character who hates Christmas, or you could join other teams who are getting serious about cost-efficient observability. That is, collecting telemetry data based on value, not volume.

Why "collect everything" no longer works

The old playbook was simple: instrument everything, store it all, figure out what you need later. Storage was cheap enough. Queries were fast enough. No need to overthink it.

Then, three things happened:

OpenTelemetry went mainstream. Teams migrated from vendor agents to OTel and began adding spans for everything. This added more visibility, but with 10x the data.
AI observability tools arrived. Platforms started using LLMs to analyze traces and suggest root causes. Powerful, but also expensive to run against terabytes of unfiltered trace data.
CFOs started asking questions. "Our_traffic grew 15% but observability costs grew 40%. Explain."

To stop instrumenting wouldn't be an option and also you want to make informed decisions, but still the biggest culprit, hiding in your telemetry stack is cardinality.

Cardinality will eat your budget

Cardinality is the observability villain. It sneaks in quietly, one innocent-looking label at a time, and before you know it, it's stolen your entire cloud budget. What is cardinality? It's just the number of unique time series your metrics generate, but it's also the main driver of observability costs that nobody sees coming.

Low cardinality: http_requests_total tracked by method and status_code. Maybe 20 unique combinations. Fairly manageable.

High cardinality: Same counter, but now you've added user_id, request_id, and session_token as labels. By simply adding these labels, you’ve just created millions of unique time series. Each one needs storage, indexing, and query compute. This will compound your bill faster than you can say deck the halls, except you wouldn’t be able to deck the halls, you’d be paying off your usage bill.

Stopping the Green character: set cardinality budgets

Most teams don't set limits on how many time series a service can create even though they should., but you can.

Start by auditing what you're currently generating. Look for metrics with >100K unique time series, or labels that include UUIDs, request IDs, or email addresses. These are your problem children.

Then set budgets. Give each service a limit, like 50K time series max. Assign team quotas so the checkout team knows they get 200K total across all their services. Create attribute allowlists that define exactly which labels are allowed in production. Yes, this feels restrictive at first. Your developers will complain. They'll argue that they need that user_id label for debugging. And sometimes they're right. But forcing that conversation up front means they have to justify the cost, not just add labels reflexively.

Finally, enforce budgets through linters that flag high-cardinality attributes in code review, CI checks that fail if estimates get too high, and dashboards that alert when cardinality spikes. This isn't about being restrictive. It's about being intentional. If you're adding a label, you should know why and what it'll cost.

Cardinality budgets solve the metrics problem, but what about traces? That's where sampling comes in.

Sampling: without the guilt

Not all sampling strategies are created equal, and picking the right one depends on what you're trying to protect.

Head-based sampling is pretty strict. You decide whether to keep a trace at the very start of a request, before you know if it'll be interesting. Fast checkout gets dropped. Slow checkout that timeout also gets dropped, because the decision happened too early. Not great.

Tail-based sampling is smarter. Wait until the trace completes, then decide based on what actually happened. Keep errors, high latency, or specific user cohorts. Sample down the boring stuff. This costs more (you have to buffer complete traces) but you keep what matters.

Probabilistic sampling is the middle ground. Keep 10% of everything, regardless of content. Predictable cost reduction, but you'll still lose some critical events. Works fine for stable services where trends matter more than individual traces.

Now rule-based sampling is where things get interesting, and honestly where most teams should be spending their energy. The idea is dead simple: different traffic deserves different sampling rates. You keep 100% of traces during feature rollouts because you actually care about every request when you're validating a new flow.

If you're using LaunchDarkly for progressive rollouts, you can tie sampling rates directly to flag evaluations. 100% sampling for users in the new variant, 10% for the control group. Your main API endpoints can run at 50% since they're stable and high-volume. Internal health checks that just verify the service is alive need maybe 5%, or even less. I've seen teams go down to 1% for health checks and never miss it.

The key is that you're making these decisions based on the actual value of the signal, not just applying a blanket rate across everything. Adjust based on context: feature flags, experiments, specific endpoints, user cohorts, whatever makes sense for your system.

Sampling isn't about compromising visibility. It's about amplifying signals. The noisy 90% of traces you're storing never get looked at anyway.

Once you've decided what to keep, you still need to decide how long to keep it and at what resolution.

Downsample vs. Discard: know when to do which

Not all data reduction is the same, and mixing up downsampling with discarding is how teams accidentally delete data they actually need.

Downsample when you need historical context but not full precision. SLO burn rates don't need second-by-second granularity so you can downsample to 1-minute intervals and still catch every trend. An additional practice is to keep high-res data for a week, then downsample to hourly for long-term retention.

Discard when the data is redundant or has served its purpose. For instance, debug spans from a canary that passed three days ago can be deleted. Or if you captured an error in both a trace and a log, you can pick one source of truth and drop the duplicate.

The rule of thumb here is If you'll never query it, don't store it. If you might need it for trends in six months, downsample it. If you need it immediately when something breaks, keep it at full resolution with an aggressive retention policy.

What this actually looks like

Cost-efficient observability isn't about cutting capabilities. It's about cutting waste.

Start by auditing your cardinality. Find the metrics generating hundreds of thousands of time series because someone added user_id as a label. Then, set budgets like 50K per service, 200K per team and enforce them through linters and CI checks. Create ways to encourage developers to justify high-cardinality labels before they ship, not after the bill arrives.

Then you’ll be ready to tackle sampling. Drop the blanket 10% probabilistic rate and switch to rule-based sampling tied to actual value. Keep 100% of traces during feature rollouts. Sample stable endpoints at 10%. Go as low as 1% for health checks. If you're running feature flags, tie sampling to flag evaluations so you capture what matters and discard what doesn't.

Finally, clean up retention, downsample SLO metrics to 1-minute intervals, discard debug spans from canaries that passed days ago and delete duplicate error data.

This not only leads to lower bills, but also cleaner dashboards, faster queries, fewer noisy alerts, and teams that spend less time swimming through telemetry and more time fixing actual problems.

Observability ROI isn't measured in data volume. It's measured in how fast you detect and resolve issues.

The teams figuring this out in 2025 aren't collecting everything. They're collecting what matters.

Day 3 | 🔔 Jingle All the Way to Zero-Config Observability

Alexis Roberson — Wed, 10 Dec 2025 19:10:26 +0000

Originally published in the LaunchDarkly Docs

For years, auto-instrumentation promised effortless observability but kept falling short. You'd still end up manually adding spans to business logic, hunting down missing metadata, or trying to piece together how a feature rollout was affecting customers.

That finally shifted in 2025. With OTel auto-instrumentation maturing and LaunchDarkly adding built-in OTel support to server-side SDKs, teams started getting feature flag context baked into their traces without writing instrumentation code. The zero-config promise actually started delivering.

Auto-instrumentation has always had a blind spot: it shows you what happened, but not why. You'd see a latency spike, but had no idea which feature flag was active, which users hit it, or what experiment was running.

Without that context, you're doing detective work. Digging through logs, matching up timestamps, guessing at what caused what. Manual instrumentation helped, but you paid for it in engineering time, inconsistent coverage, and mounting technical debt.

Auto-instrumentation that actually knows about your features

The game changed when OTel auto-instrumentation actually got good. Instead of just capturing basic HTTP calls, it now handles:

Framework-level request tracing.
Automatic context propagation across services.
Runtime metadata and environment details.
Errors and exceptions without manual try-catch blocks.

LaunchDarkly takes this further by injecting flag evaluation data straight into OTel spans. Every time you evaluate a flag, you automatically get the flag key, user context, which variation served, and the targeting rule that fired. That data feeds into your existing OTel pipeline, so your traces finally show which features were active and who was affected - not just database queries and API calls.

So how do you actually set this up?

To get started with Otel trace hooks and feature flag data, simply add the hooks to your LaunchDarkly client config.

import ldclient
from ldclient import Config
from ldotel.tracing import Hook

config = Config('YOUR_SDK_KEY', hooks=[Hook()])
ldclient.set_config(config=config)

client = ldclient.get()

This flows into your existing OpenTelemetry pipeline, enriching every trace with feature-aware context.

The TracingHook automatically decorates your OpenTelemetry spans with flag evaluation events. When your application evaluates flags during a request, those evaluations become part of the trace along with the full context about what was evaluated and for whom.

You can also configure your OpenTelemetry collector or exporter to point to LaunchDarkly's OTLP endpoint, and you're done.

For HTTP:

https://otel.observability.app.launchdarkly.com:4318

For gRPC:

https://otel.observability.app.launchdarkly.com:4317

This feature is also available in .Net, Go, Java, Node.JS and Ruby.

Auto-instrumentation handles the rest, HTTP spans, database calls, framework-level tracing, error capture, and now, feature flag context.

What auto-instrumentation unlocks

When you ship a new feature variant, you immediately see how it performs per cohort. If there's a latency spike in the "new-checkout-flow" variation, you'll know within minutes before it affects user experience.

That same visibility matters during incidents. When an outage hits, filter traces by flag evaluation to see which features were active when errors occurred. The trace shows you whether it was the new recommendation engine, the optimized query path, or something else entirely.

This is especially powerful for experimentation. LaunchDarkly processes your OTel traces into metrics automatically, so when you run an A/B test, you get latency, error rate, and throughput calculated per variation without extra config. The same telemetry powering your dashboards powers your experiments.

The best part of this setup is that it scales without additional work. As teams ship more features behind flags, the telemetry gets more valuable without getting more expensive to maintain. New services inherit feature-aware tracing just by initializing the SDK.

When to add custom spans

Zero-config doesn't mean never-config. You'll still want custom spans for:

Business logic milestones. If you need to measure time-to-recommendation or search-to-purchase, custom spans make that explicit.
ML pipeline stages. Feature extraction, model inference, and post-processing often warrant their own spans for detailed performance analysis.
Cross-service boundaries. Queue producers, stream processors, and async workers may need manual context propagation and span creation.
Experiment-specific KPIs. If your A/B test measures "items added to cart" or "video completion rate," you'll instrument those as custom metrics.

The important part is you're writing these spans to capture business value, not to patch holes in your instrumentation.

Delivering real value

Combining mature auto-instrumentation with feature-aware enrichment changes how teams approach observability. It's no longer a separate investment that competes with feature development. It's a byproduct of how you ship features.

When you evaluate a flag, you get telemetry. When you roll out a feature, you get performance data segmented by variation. When you run an experiment, you get metrics derived from production traces. The instrumentation you would have written manually is now embedded in the tools you already use.

Observability stops being something you retrofit after launch and becomes something you inherit by default. Which means teams spend less time debugging instrumentation gaps and more time acting on insights.

That's the promise of zero-config, finally delivered.

Ready to try it? Explore LaunchDarkly's OpenTelemetry integration documentation or sign up for a free trial account.

Enable Observability or Experimentation in your LaunchDarkly dashboard and start seeing feature-aware telemetry from your existing traces.

Day 2 | 🎅 He knows if you have been bad or good... But what if he gets it wrong?

Alexis Roberson — Tue, 09 Dec 2025 20:21:47 +0000

Originally published in the LaunchDarkly Docs

"He knows if you've been bad or good..."

As kids, we accepted the magic. As engineers in 2025, we need to understand the mechanism. So let's imagine Santa's "naughty or nice" system as a modern AI architecture running at scale. What would it take to make it observable when things go wrong?

The architecture: Santa's distributed AI system

Santa's operation would need three layers. The input layer handles behavioral data from 2 billion children on a point system. "Shared toys with siblings" gets +10 points, "Threw tantrum at store" loses 5.

The processing layer runs multiple AI agents working together. A Data Agent collects and organizes behavioral events. A Context Agent retrieves relevant history: letters to Santa, past behavior, family situation. A Judgment Agent analyzes everything and makes the Nice/Naughty determination. And a Gift Agent recommends appropriate presents based on the decision.

The integration layer connects to MCP servers for Toy Inventory, Gift Preferences, Delivery Routes, and Budget Tracking.

It's elegant. It scales. And when it breaks, it's a nightmare to debug.

The Problem: A good child on the Naughty List

It's Christmas Eve at 11:47 PM.

A parent calls, furious. Emma, age 7, has been a model child all year. She should be getting the bicycle she asked for. Instead, the system says: Naughty List - No Gift.

You pull up the logs:

Emma's judgment: 421 NICE points vs 189 NAUGHTY points
Gift Agent tries to check bicycle inventory → TIMEOUT
Gift Agent retries → TIMEOUT  
Gift Agent retries again → TIMEOUT
Gift Agent checks inventory again → Count changed
Gift Agent reasoning: "Inventory uncertain, cannot fulfill request"
Gift Agent defaults to: NAUGHTY LIST

Emma wasn't naughty. The Toy Inventory MCP was overloaded from Christmas Eve traffic. But the agent's reasoning chain interpreted three timeouts as "this child's request cannot be fulfilled" and failed to the worst possible default.

With traditional APIs, you'd find the bug on line 47, fix it, and deploy. With AI agents, it's not that simple. The agent decided to interpret timeouts that way. You didn't code that logic. The LLM's 70 billion parameters did.

This is the core challenge of AI observability: You're debugging decisions, not code.

Why AI Systems are hard to debug

Black box reasoning and reproducibility go hand in hand. With traditional debugging, you step through the code and find the exact line that caused the problem. With AI agents, you only see inputs and outputs. The agent received three timeouts and decided to default to NAUGHTY_LIST. Why? Neural network reasoning you can't inspect.

And even if you could inspect it, you couldn't reliably reproduce it. Run Emma's case in test four times and you might get:

Run 1: NICE LIST, gift = bicycle ✓
Run 2: NICE LIST, gift = video game ✓
Run 3: NICE LIST, gift = art supplies ✓
Run 4: NAUGHTY LIST, no gift ✗

Temperature settings and sampling introduce randomness. Same input, different results every time. Traditional logs show you what happened. AI observability needs to show you why, and in a way you can actually verify.

Then there's the question of quality. Consider this child:

Refused to eat vegetables (10 times) but helped put away dishes
Yelled at siblings (3 times) but defended a classmate from a bully
Skipped homework (5 times) but cared for a sick puppy

Is this child naughty or nice? The answer depends on context, values, and interpretation. Your agent returns NICE (312 points), gift = books about empathy. A traditional API would return 200 OK and call it success. For an AI agent, you need to ask: Did it judge correctly?

Costs can spiral out of control. Mrs. Claus (Santa's CFO) sees the API bill jump from 5,000 in Week 1 to 890,000 on December 24th. What happened? One kid didn't write a letter. They wrote a 15,000-word philosophical essay. Instead of flagging it, the agent processed every last word, burning through 53,500 tokens for a single child. At scale, this bankrupts the workshop.

And failures cascade in unexpected ways. The Gift Agent doesn't just fail when it hits a timeout. It reasons through failure. It interpreted three timeouts as "system is unreliable," then saw the inventory count change and concluded "inventory is volatile, cannot guarantee fulfillment." Each interpretation fed into the next, creating a chain of reasoning that led to: "Better to disappoint than make a promise I can't keep. Default to NAUGHTY_LIST."

With traditional code, you debug line by line. With AI agents, you need to debug the entire reasoning chain. Not just what APIs were called, but why the agent called them and how it interpreted each result.

What Santa Actually Needs

The answer isn't to throw out traditional observability, but to build on top of it. Think of it as three layers.

This is exactly what we've built at LaunchDarkly. Our platform combines AI observability, online evaluations, and feature management to help you understand, measure, and control AI agent behavior in production. Let's walk through how each layer works.

Start with the fundamentals. You still need distributed tracing across your agent network, latency breakdowns showing where time is spent, token usage per request, cost attribution by agent, and tool call success rates for your MCP servers. When the Toy Inventory MCP goes down, you need to see it immediately. When costs spike, you need alerts. This isn't optional. It's table stakes for running any production system.

For Santa's workshop, this means tracing requests across Data Agent → Context Agent → Judgment Agent → Gift Agent, monitoring MCP server health, tracking token consumption per child evaluation, and alerting when costs spike unexpectedly. It’s important to note, LaunchDarkly's AI observability captures all of this out of the box, providing full visibility into your agent's infrastructure performance and resource consumption.

Then add semantic observability. This is where AI diverges from traditional systems. You need to capture the reasoning, not just the results. For every decision, log the complete prompt, retrieved context, tool calls and their results, the agent's reasoning chain, and confidence scores.

When Emma lands on the Naughty List, you can replay the entire decision. The Gift Agent received three timeouts from the Toy Inventory MCP, interpreted "inventory uncertain" as "cannot fulfill request," and defaulted to NAUGHTY_LIST as the "safe" outcome. Now you understand why it happened. And more importantly, you realize this isn't a bug in your code. It's a reasoning pattern the model developed. Reasoning patterns require different fixes than code bugs.

LaunchDarkly's trace viewer lets you inspect every step of the agent's decision-making process, from the initial prompt to the final output, including all tool calls and the reasoning behind each step.

Finally, use online evals. Where observability shows what happened, online evals automatically assess quality and take action. Using the LLM-as-a-judge approach, you score every sampled decision. One AI judges another's work:

{
  "accuracy": {
    "score": 0.3,
    "reasoning": "Timeouts should trigger retry logic, not default to 
      worst-case outcome. System error conflated with behavioral judgment."
  },
  "fairness": {
    "score": 0.4,
    "reasoning": "Similar timeout patterns resulted in NICE determination 
      for other children. Inconsistent failure handling."
  }
}

This changes the conversation from vague to specific.

Without evals: "Let's meet tomorrow to review Emma's case and decide if we should rollback."

With evals: "Accuracy dropped below 0.7 for the 'timeout cascade defaults to NAUGHTY' pattern. Automatic rollback triggered. Here are the 23 affected cases."

LaunchDarkly's online evaluations run continuously in production, automatically scoring your agent's decisions and alerting you when quality degrades. You can define custom evaluation criteria tailored to your use case and set thresholds that trigger automatic actions.

This is where feature management and experimentation come in. Feature flags paired with guarded rollouts let you control deployments and roll back bad ones. Experimentation lets you A/B test different approaches. With AI agents, you're doing the same thing, but instead of testing button colors or checkout flows, you're testing prompt variations, model versions, and reasoning strategies. When your evals detect accuracy has dropped below threshold, you automatically roll back to the previous agent configuration.

Use feature flags to control which model version, prompt template, or reasoning strategy your agents use and seamlessly roll back when something goes wrong. Our experimentation platform lets you A/B test different agent configurations and measure which performs better on your custom metrics. Check out our guide on feature flagging AI applications.

You're not just observing decisions. You're evaluating quality in real-time and taking action.

Debugging Emma: all three layers in action

Traditional observability shows the Toy Inventory MCP experienced three timeouts that triggered retry logic. Token usage remained average. From an infrastructure perspective, nothing looked catastrophic.

Semantic observability reveals where the reasoning went wrong. The Gift Agent interpreted the timeouts as "inventory uncertain" and made the leap to "cannot fulfill requests." Rather than recognizing this as a temporary system issue, it treated the timeouts as a data problem and defaulted to NAUGHTY_LIST.

Online evals reveal this isn't just a one-off problem with Emma, but a pattern happening across multiple cases. The accuracy judge flagged this decision at 0.3, well below acceptable thresholds. Querying for similar low-accuracy decisions reveals 23 other cases where timeout cascades resulted in NAUGHTY_LIST defaults.

Each layer tells part of the story. Together, they give you everything you need to fix it before more parents call.

With LaunchDarkly, all three layers work together in a single platform. You can trace the infrastructure issue, inspect the reasoning chain, evaluate the decision quality, and automatically roll back to a safer configuration, all within minutes of Emma's case being flagged.

Conclusion

Every AI agent system faces these exact challenges. Customer service agents making support decisions. Code assistants suggesting fixes. Content moderators judging appropriateness. Recommendation engines personalizing experiences. They all struggle with the same problems.

Traditional observability tools weren't built for this. AI systems make decisions, and decisions need different observability than code.

Santa's system says "He knows if you've been bad or good." But how he knows matters. Because when Emma gets coal instead of a bicycle due to a timeout cascade at 11:47 PM on Christmas Eve, you need to understand what happened, find similar cases, measure if it's systematic, fix it without breaking other cases, and ensure it doesn't happen again.

You can't do that with traditional observability alone. AI agents aren't APIs. They're decision-makers. Which means you need to observe them differently.

LaunchDarkly provides the complete platform for building reliable AI agent systems: observability to understand what's happening, online evaluations to measure quality, and feature management to control and iterate safely. Whether you're building Santa's naughty-or-nice system or a production AI application, you need all three layers working together.

Ready to make your AI agents more reliable? Start with our AI quickstart guide to see how LaunchDarkly can help you ship AI agents with confidence.

Day 1 | 🎄 Observability under the Tree: What Changed in 2025

Alexis Roberson — Mon, 08 Dec 2025 20:21:33 +0000

Originally published in the LaunchDarkly Docs

Sometimes you need distance to see what really changed. And that’s exactly how 2025 felt in regards to observability. While the focus was rightfully on AI agents and MCP servers, the observability space was experiencing its own maturation. And if you feel you missed this shift, no need to panic, Santa left some gifts under the tree. Those gifts being a universal standard for instrumentation, practical platform consolidation, and AI-native capabilities that finally deliver on the hype.

Gift #1: OpenTelemetry or the foundation nobody wanted to build

For years, observability felt like the Wild West. Every vendor had their own way of doing things, their own agents, instrumentation libraries. If you wanted to switch tools, you basically had to re-instrument your entire application. It was exhausting, but nobody wanted to be the first to blink (so to speak).

OpenTelemetry had been building toward something for years, but 2025, in my opinion, was when it reached a tipping point. After gradual adoption since the early 2020s, it crossed from "early adopters only" to "the default choice." The specs had stabilized, the major players finally committed, and the tooling matured enough that adoption became low-risk rather than bleeding-edge.

The change was subtle at first. A cloud provider here, a major vendor there, all quietly releasing native OTel support. Then it accelerated. Suddenly the question wasn't "should we adopt OpenTelemetry?" but "why haven't we adopted it yet?" Teams started instrumenting once and sending data anywhere they wanted. The instrumentation lock-in that had defined observability for years started to ease. Now, you still had considerations around data migration and historical context, but the path forward got clearer.

Even more, here's what made this interesting: standardization didn't just reduce vendor lock-in. It unlocked something bigger, and that is tool scrutinization by reducing scope and being more selective about what tool goes in your toolbox.

Gift #2: Platform Consolidation after the dust has settled

With a common data standard gaining real traction, something practical happened: the tooling chaos started to make less sense.

Think about the typical observability stack from a few years ago. Metrics in one place, logs in another, traces somewhere else. APM here, synthetic monitoring there, and probably two or three other specialized tools for edge cases. Each with its own interface, its own query language, its own billing model, and its own person on the team who understood how it worked.

2025 was the year teams started asking "why are we doing this to ourselves?"

Platform consolidation had been promised for years. However, this time felt different. With OTel providing a standard data layer, platforms could actually focus on the analysis and visualization layer instead of fighting about ingestion formats. They could genuinely unify telemetry data in ways that reduced cognitive load.

It is becoming common practice for teams to cut their tool count from eight or nine down to two or three, and actually improve coverage in the process. It’s important to note that the cost story varied with some teams seeing real savings from eliminating redundant ingestion and storage and others mainly reducing operational overhead rather than raw dollar spend. At the end of the day the consistent theme was relief.

With tool consolidation, came the freedom to innovate in the world of query execution, where instead of building complex queries to retrieve specific data, teams could interact with data sets using natural language.

Gift #3 Natural language query execution

I’m no stranger to learning and mastering query languages like the Elasticsearch Query DSL and though optimized for improving latency and data processing, there still remained a continual learning curve.

2025 brought natural language interfaces that crossed the "actually useful" threshold. Though not perfect as complex queries still benefit from knowing the underlying language, it has proven genuinely helpful for common questions. You could ask questions like:

"Show me API latency for checkout over the last hour broken down by region."

And get back related data without memorizing aggregation functions or time window syntax. The true value is removing the barrier of entry, so less technical team members like product managers and business-focused teams can also understand the type of performance issues that were generally understood by DevOps or SREs. In other words, the observability data that had been locked behind specialized knowledge became accessible to more people who needed it.

But accessibility alone doesn't solve the deeper problem, which is knowing what to look for in the first place to understand real business impact.

Gift #4 Aligned Business KPIs with measurable metrics

Historically, observability was about system health. Is the service up? What's the error rate? How's latency looking? All important questions, but that unfortunately rarely answered the question of why any of this should matter to other teams. The shift toward business outcomes had been building, but 2025 is when it moved from buzzword to practical framework.

Tools made it easier to connect technical metrics to business impact. That latency spike? You could now configure dashboards and alerts that showed the correlation with conversion drops and revenue impact, rather than just knowing the service was slow. This error rate? You could track it against transaction volume and calculate lost revenue. This performance degradation?

You could filter by customer segment and see who was actually affected. Going as far as detecting rage clicks on a page affecting user experience.

The setup still required work like defining what business metrics mattered, instrumenting those alongside technical metrics, building the correlation logic. But the platforms made it possible without custom data pipelines and analytics infrastructure.

This changed prioritization too. Instead of treating every alert as equally urgent, teams could focus on what actually moved the needle. The 99th percentile latency issue that only affected a low-value endpoint? Still worth fixing eventually, but not at 2 AM. The intermittent error affecting checkout? All hands on deck.

The business-outcome framing made observability feel less like a tax and more like a strategic capability. But it also created a new opportunity that begged the question: how can AI truly be used to not just automate processes but make real decisions around data interpretation and root cause analysis with the intent of self healing.

Gift #5 AI-Native observability, when the promises started delivering

Which brings us to AI, the talk of the town that has finally started delivering in meaningful ways in 2025.

Not the "AI-powered" marketing speak that got slapped on every dashboard, but real AI-native observability that changed how teams work. Anomaly detection that's actually tuned for your environment and doesn't require a data science degree to configure. Root cause analysis that connects the dots across services, traces, logs, and business metrics without requiring manual correlation. Predictive insights that warn you about capacity issues before they become incidents. All built into dashboards like Vega AI for thorough investigations.

The capabilities themselves weren't entirely new as statistical anomaly detection has existed for years. What changed was the integration, the usability, and the accuracy. The systems got good enough that you could trust them to surface real issues without drowning you in false positives.

All the previous observability gifts created the perfect foundation for AI to enter the chat. From OTel's standardized data formats creating consistent structure leading to more reliable pattern recognition. To the consolidated platforms that provide a unified view making natural language interfaces even more powerful because you can ask not just "what happened" but "why did this happen and what should I do about it." By the time you add connections between metrics and business impact, you already have a full observability system primed and ready for AI.

Bonus: What LaunchDarkly left in your stockings this year

While the industry was catching up on the observability fundamentals, LaunchDarkly was actively building toward the same vision, with one crucial difference: observability unified with feature flag context. Here's what landed in 2025 that aligns with these broader trends.

Vega AI: Making AI-Native Observability Real

Speaking of AI capabilities that actually deliver, LaunchDarkly introduced Vega, an AI-powered debugging companion that goes beyond generic copilots. Vega combines two integrated capabilities: an AI debugging assistant that summarizes errors, identifies root causes, and suggests code fixes with GitHub integration, and a natural language search assistant that lets you query observability data without memorizing syntax. The key difference? Context. Vega understands feature flags alongside telemetry data, connecting what changed in your release with what broke in production, enabling faster triage and smarter fixes.

OpenTelemetry Support: Embracing the Standard

In the spirit of OTel's tipping point, LaunchDarkly's C++ Server SDK added native OpenTelemetry tracing support. Teams can now enrich their distributed traces with LaunchDarkly context, sending observability data to power Guardian while maintaining the flexibility to route telemetry wherever they need. It's exactly the kind of integration that becomes possible when standardization reduces the friction.

Session Replay for Mobile: Closing the Visibility Gap

Web session replay has been table stakes, but mobile remains a black box. LaunchDarkly's new native iOS and Android SDKs for Session Replay now capture UI interactions, gestures, and network events tied directly to feature flag evaluations. When an issue occurs, you can see exactly what your users experienced and which flag variation they encountered, bringing the same level of observability to mobile that web teams have enjoyed.

Regression Debugging: Automated Error Detection

The Guarded Releases capability now includes regression debugging that automatically surfaces frontend errors tied to new feature flag variations without manual instrumentation. When a rollout triggers a regression, relevant errors appear directly in the monitoring view, with one-click access to deeper analysis in the observability view. It's the kind of integration that makes detecting issues faster and connecting dots easier.

LLM Observability: AI-Specific Telemetry

As GenAI applications moved to production, teams needed visibility beyond standard metrics. LaunchDarkly's LLM observability tracks not just performance metrics like latency and error rates, but semantic details including prompts, token usage, and responses. It addresses the unique challenge of monitoring AI features where understanding quality matters as much as performance.

Guardrail Metrics and Feature Flag Monitoring: Business Impact by Default

Two updates reinforced the shift toward business-outcome observability. Guardrail metrics let teams define default trusted metrics that automatically apply to any guarded rollout, ensuring consistent measurement across releases. Feature Flag Monitoring provides a unified view showing errors, web vitals, and the impact of feature changes in one place, with flag change annotations overlaid to see correlation. You ship a feature, you immediately see how it's performing, all with the context of which flags are active.

Together, these updates represent LaunchDarkly's commitment to the same principles driving industry-wide change: standardized instrumentation, consolidated platforms, natural language accessibility, business-outcome alignment, and AI-native capabilities. The difference is in the integration, where feature flags provide the missing context that helps teams understand not just what happened, but what they changed that caused it to happen.

What next?

When I look at these changes together, I see something more significant than individual feature releases or product launches. The observability landscape of 2025 represents a meaningful shift in what's practical and accessible.

It's more accessible seeing that more people on the team can explore data and ask questions without specialized training. It's more focused because we have better tools for measuring what actually matters for business outcomes, not just what's easy to instrument. It's more intelligent in that AI handles more of the grunt work of correlation and pattern matching. And it's more practical because costs are more manageable and the tool sprawl that was slowly crushing teams can hopefully start to recede.

As we head into 2026, I'm not thinking about what new features will launch. I'm thinking about what becomes possible when these foundations are in place. When observability isn't a specialized skill but a shared capability. When understanding system behavior is directly tied to business strategy. When the friction between having a question and getting an answer continues to decrease.

That's the real gift 2025 left us. Not just better tools, but a better way of working. One that's still evolving, but noticeably more mature than where we were.

Join us for Day 2 of the Holiday Campaign where we'll discuss the reality of observability for AI agents.