I used Jaeger trace debugging on my agent and found out the LLM wasn’t the problem at all

A few weeks ago I hit one of those agent bugs that makes everyone immediately blame the model.

Webhook comes in. Payload gets cleaned up. Retrieval runs. GPT-5 or Claude gets called. A tool fires. Final answer is weird.

And because the weird part is visible at the model output, the team says the usual thing:

“The LLM got confused.”

I’ve started to really dislike that sentence.

Once I wired the whole path into Jaeger using OpenTelemetry over OTLP, it became obvious the model was mostly innocent. The bad behavior started before the LLM span.

The actual problem was a broken workflow branch upstream.

That one trace saved me from another week of pointless prompt tweaking.

The bug looked like an LLM problem

From logs, the failure looked random.

Most runs were fine.
A few runs were not.
The bad runs looked like the model had made a dumb decision.

But once I looked at the trace, the sequence was obvious:

1. Webhook payload arrived with inconsistent fields
2. Preprocessing normalized one field and silently damaged another
3. Retrieval got malformed input
4. Retrieval returned weak context
5. GPT-5 answered the question it actually received
6. Tool execution continued with false confidence

That is not “the model got confused.”

That is a pipeline bug wearing an LLM costume.

Why traces helped more than logs

Logs told me isolated facts.

Traces showed me the shape of the run.

That matters a lot for agent systems.

When you can see webhook intake, preprocessing, retrieval, model call, and tool execution in one timeline, you stop asking:

Why did GPT-5 do something weird?

And start asking:

Where did this run become wrong?

That question is way more useful.

Jaeger is easier to use than people remember

I had outdated assumptions about Jaeger.

I still thought of it as this weird separate tracing stack with custom plumbing.

That’s not really the situation anymore.

Jaeger v2 fits nicely into an OpenTelemetry setup. If you’re already emitting OTLP, Jaeger feels much less like a side quest.

The endpoints I cared about were:

• OTLP gRPC: 4317
• OTLP HTTP: 4318
• Jaeger UI: 16686

That means you can instrument your app once and push traces through standard OTLP instead of doing something bespoke.

Local Jaeger run command

This is the basic container command I used while testing:

docker run --rm \
  --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 5778:5778 \
  -p 9411:9411 \
  -v /path/to/local/config.yaml:/jaeger/config.yaml \
  jaegertracing/jaeger:2.21.0 \
  --config /jaeger/config.yaml

Once that was running, the architecture diagram stopped being theoretical.

I had receipts.

The span layout I actually wanted

For agent debugging, I wanted a trace that looked roughly like this:

webhook.receive
  └── preprocess.normalize
      └── retrieval.query
          └── llm.chat.completion
              └── tool.execute

If one of those spans starts getting slow, malformed, or inconsistent, you can usually see where the run bent out of shape.

That’s the part logs tend to hide.

A practical OpenTelemetry setup

If you’re instrumenting a Python or Node service that orchestrates agents, the important thing is to keep trace context across every step.

Example: OTLP exporter config

export OTEL_SERVICE_NAME=agent-worker
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Example: useful span attributes for agent runs

I’d strongly recommend attaching attributes like these:

• workflow_id
• tenant
• channel
• model
• tool_name
• retrieval_index
• provider

Those attributes turn a trace from a spaghetti screenshot into something you can actually search.

The collector is where debugging gets good

Jaeger is where I looked at traces.

OpenTelemetry Collector is where I made the traces useful.

That distinction matters.

The collector is the control point where you can filter, enrich, and shape telemetry before it lands in Jaeger.

For example:

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - resource.attributes["service.name"] == "service1"

  attributes/accountid:
    actions:
      - key: account_id
        value: 2245
        action: insert

For agent systems, I’d go further and insert metadata that helps explain a run:

processors:
  attributes/agent:
    actions:
      - key: workflow_id
        value: wf_123
        action: insert
      - key: channel
        value: slack
        action: insert
      - key: model
        value: gpt-5
        action: insert
      - key: tool_name
        value: crm_lookup
        action: insert

That kind of enrichment is boring until you need it.
Then it becomes the difference between “I think the bug is somewhere in the pipeline” and “the Slack branch passed malformed retrieval input at 14:03:11.”

Service Performance Monitoring is sneakily useful for agents

One thing I liked more than expected was Jaeger’s Service Performance Monitoring view.

It derives RED metrics from spans and surfaces latency percentiles like P95, P75, and P50.

There’s also an Impact metric based on latency × request rate.

That’s useful for agent systems because it tells you where to start.

If preprocessing has the highest impact, don’t spend your afternoon staring at LLM spans.
If retrieval latency suddenly blows up at P95, that’s probably where the run started deforming.

That workflow is simple and effective:

1. Find the ugly service
2. Jump into the traces
3. Inspect the exact bad run

That order is much better than starting from the model output and guessing backward.

This gets very real in n8n and OpenClaw

This clicked for me when I stopped thinking about toy apps and started thinking about real automation stacks.

n8n

If you’re self-hosting n8n, you already have a decent operational surface:

• /healthz
• /healthz/readiness
• /metrics

And n8n also supports tracing executions with OpenTelemetry.

That combination is strong.

Health endpoints tell you whether the worker is alive.
Metrics tell you whether the box is under pressure.
Traces tell you whether the delay started at webhook intake, workflow execution, retrieval, or the model call.

If your workflow “hangs on AI,” that might be an LLM provider issue.
But it might also be queue delay, bad branching, or retrieval timing out before the model is even called.

Those are completely different incidents.

OpenClaw

OpenClaw is another good example because it spans a messy real-world path:

• Discord
• Slack
• Teams
• Telegram
• WhatsApp
• iMessage

That means a single user interaction can cross channel ingress, session lookup, routing, retrieval, model invocation, and tool execution.

If all you have is model logs, you’re blind.

If you have end-to-end traces, you can see whether the slow part was:

• Discord webhook intake
• a bad session branch
• retrieval latency
• a stuck tool call
• provider-side model latency

That’s the difference between debugging and superstition.

The easiest way to sabotage tracing: sampling

This is the part that bites teams constantly.

A lot of OpenTelemetry SDK setups start effectively at 100% sampling while you’re local or early in development.

Then someone turns on remote sampling and assumes they still have enough visibility.

Not necessarily.

A default probabilistic strategy like 0.001 means 0.1% of traces.

If your weird branch shows up once every few hundred runs, you can absolutely sample away the one trace that explains the bug.

That’s how teams end up saying:

We added tracing and it didn’t help.

No, you added tracing and then threw most of it away.

Check the sampling strategy

curl "jaeger-collector:14268/api/sampling?service=foobar"

Example response:

{"strategyType":"PROBABILISTIC","probabilisticSampling":{"samplingRate":0.001}}

My opinion: when you’re debugging weird branching behavior, bias toward fidelity first. Worry about cost and volume after you can actually see the problem.

Traffic spikes can make observability lie to you too

There’s another failure mode that’s easy to miss.

If traffic spikes during the exact incident you care about, collectors can drop traces if downstream storage can’t keep up.

That matters if you’re running high-volume automations, always-on agents, or bursty webhook workflows.

If your n8n workers suddenly get slammed or your multi-channel agent starts ingesting traffic across Slack, Discord, and WhatsApp at the same time, the exact traces you need can disappear.

The practical answer is boring but solid:

• buffer properly
• understand queue behavior
• consider Kafka between collectors and storage if volume justifies it

This is also why “open source tracing is free” is not really true in production.

The software can be free.
Reliable high-fidelity observability is not.

What Jaeger is good at, and what it isn’t

Here’s the split I wish I had earlier:

Option	What it’s best at
Jaeger	Open-source distributed tracing backend, OTLP ingest on 4317/4318, strong for tracing request flow and service latency
OpenTelemetry Collector	Pipeline control point for receivers, processors, exporters, filtering, enrichment, and routing
n8n monitoring stack	Health endpoints, metrics, and execution tracing for webhook-driven automations

My blunt take:

Jaeger is excellent for distributed tracing, but it is not LLM-native observability.

If you need prompt versioning, eval workflows, or token-level cost analytics, you may want a specialized GenAI observability tool too.

I would not try to force Jaeger to be everything.

But for answering this question:

Where did this agent run actually go wrong?

Jaeger is extremely good.

Why this matters more when you pay per token

One side effect of bad observability is that teams often “debug” by rerunning workflows over and over.

More retries.
More prompt variants.
More model calls.
More tool loops.

If you’re paying per token, that gets expensive fast.

That’s one reason I think teams building agents and automations should care a lot about predictable infrastructure costs.

If your workflows run 24/7 in n8n, Make, Zapier, OpenClaw, or custom agent stacks, debugging gets a lot less stressful when compute cost doesn’t spike every time you investigate a weird branch.

That’s also why products like Standard Compute are interesting for agent-heavy workloads: you keep the OpenAI-compatible API shape, but avoid turning every debugging session and every retry storm into a billing event.

For teams doing real automation volume, that matters more than people admit.

The main thing I learned

The biggest lesson was not “use tracing.”

It was this:

Bad agent behavior often starts before the LLM span, and traces show that faster than prompt tweaking ever will.

Now when an agent acts weird, I don’t start with the model.

I start here:

1. webhook intake
2. preprocessing
3. retrieval
4. model call
5. tool execution

Most of the time, the answer is not mystical.
It’s just annoying.

A branch condition was wrong.
A field got normalized badly.
A retrieval query got mangled.
A tool retried in the wrong place.

The model was the witness, not the criminal.

If you’re building agents in n8n, running multi-channel workflows in OpenClaw, or shipping your own orchestration layer, that’s the practical takeaway:

Make the run visible end to end, or you’ll keep fixing the loudest step instead of the first broken one.