The most useful thing my home cluster does is answer "which bin goes out tonight?" from my phone. I text a bot, and a few seconds later it tells me the organic container goes out Monday. No calendar, no municipal website, no app that wants my postcode and an email address.
Underneath that boring convenience, three agents talked to each other, made two model calls, and emitted 278 OpenTelemetry spans. I had no idea about any of it until I turned on tracing, and then I found out the hard way that Langfuse's free tier bills every single one of those spans.
This is how agent-to-agent delegation actually looks when you instrument it, and how to keep the trace volume from eating a free tier in a week.
The architecture
There is one entry point. A Telegram message hits an n8n workflow, which forwards it to a router agent. The router does not answer anything itself — it holds the other agents as tools and delegates over A2A.
flowchart TD
A["Telegram message — which bin this week?"] --> B["n8n workflow"]
B -->|"A2A message/send"| C["home-router agent"]
C -->|"delegates"| D["afval-agent — waste calendar"]
C -->|"delegates"| E["k8s-agent — cluster questions"]
C -->|"delegates"| F["github-agent — repos and PRs"]
D -->|"next_collection"| G["MCP server — municipal waste API"]
C --> H["OTLP spans"]
D --> H
H --> I["OTel Collector — filter processor"]
I -->|"survivors only"| J[("Langfuse Cloud EU")]
The routing rule lives in the agent's system message, not in the workflow. Adding a specialist means adding an entry to a list, not editing a chain of if nodes:
delegates:
- name: afval-agent
namespace: kagent
- name: k8s-agent
namespace: kagent
- name: github-agent
namespace: kagent
That is the whole appeal of A2A as a pattern. The router is a dumb switchboard with opinions about who handles what, and each specialist stays ignorant of the others.
What a delegation actually returns
Here is the part worth internalising before you instrument anything. A2A is not a function call that returns a string. It returns a task with a full history, and the delegation shows up inside that history as a tool call.
Ask "which container do I put out at the end of this week?" and the chain runs three levels deep. The router picks a specialist:
{
"kind": "data",
"data": {
"id": "call_468731",
"name": "kagent__NS__afval_agent",
"args": { "request": "Which container do I put out at the end of this week?" }
},
"metadata": {
"kagent_subagent_session_id": "7553862b-2a92-45a1-96a3-4afda9107119",
"kagent_type": "function_call"
}
}
The specialist then calls its own MCP tool, which is where the actual municipal data enters:
{ "name": "next_collection", "args": {} }
And the answer comes back in the language I asked in, naming the container and the weekday:
De GFT-container wordt aanstaande maandag 17 augustus opgehaald.
The container names are not invented by the model — they are a lookup table over the municipality's waste stream IDs, which is the only reason the answer is trustworthy:
const names = {
112: 'GFT (groente, fruit, tuinafval)', // organic
113: 'PMD (plastic, metaal, drankkartons)', // plastic, metal, cartons
87: 'Papier en karton', // paper and cardboard
101: 'Restafval', // general waste
};
That table is the whole reason this is useful rather than a party trick. The model decides which question is being asked and phrases the reply; it never decides which bin goes out. The system prompt makes that explicit — "never guess a date or container, if a tool call fails, say so plainly" — because an agent that confidently invents a collection day is worse than no agent at all.
Three things matter in that exchange. The specialist gets its own subagent_session_id, so it is a separate conversation with its own context. The tool call is where fact meets inference — everything above it is language handling, everything below it is municipal data. And the token usage comes back per hop: the router spent 592 prompt tokens deciding where to send the question, the specialist spent 514 answering it, and the router spent 797 more summarising the answer back to me.
Three model calls to find out it is the green one on Monday. That is the real cost shape of a routing architecture, and you cannot see it without tracing.
Turning on tracing
kagent speaks OTLP natively, so there is no Python to write and no SDK to wire in. It is Helm values:
- otel:
tracing:
enabled: true
exporter:
otlp:
endpoint: "https://cloud.langfuse.com/api/public/otel/v1/traces"
protocol: "http/protobuf"
insecure: false
timeout: 15000
Langfuse authenticates with a Basic header, and the chart has no field for OTLP headers — so it goes in as a raw environment variable through envFrom, from a secret created out of band:
AUTH=$(echo -n "pk-lf-xxx:sk-lf-xxx" | base64)
kubectl create secret generic langfuse-otel -n kagent \
--from-literal=OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Basic ${AUTH},x-langfuse-ingestion-version=4"
That worked on the second try. The first attempt put the controller into CrashLoopBackOff:
failed to initialize tracing
error: create span exporter: invalid OTLP protocol - should be one of ['grpc', 'http/protobuf']
http is not a valid value. It has to be http/protobuf, spelled exactly like that. Worth knowing because the upstream tracing docs demonstrate gRPC on port 4317, and Langfuse does not implement OTLP over gRPC at all — so you have to switch protocols, and the obvious spelling is the wrong one.
The bill arrives
Tracing worked. The first real trace looked like this:
TRACE: POST /api/a2a/kagent/home-router/ 278 observations $0.007371
├─ gemini.generate_content gemini-3.1-flash-lite in=775 out=38
├─ call_llm gemini-3.1-flash-lite in=775 out=38
└─ generate_content gemini-3.1-flash-lite in=775 out=38
Cost attribution, token counts per hop, the full delegation tree. Exactly what I wanted — and 278 spans for one question about rubbish.
Langfuse's free tier is 50,000 units a month, where a unit is any tracing data point: the trace, every observation, every score. Not 50,000 requests. At 278 spans per question that is roughly 180 questions a month before ingestion stops, and there is no overage on the free tier — it just cuts off.
Counting what those 278 spans actually were made the problem obvious:
| Spans | What they were |
|---|---|
| 34 | EventQueue.dequeue_event |
| 20 |
enqueue_event / task_done / close
|
| 19 | bare POST HTTP client spans |
| 3 | actual model calls |
Over 90% was the A2A framework narrating its own internal event queue. Useful if you are debugging the framework. Useless if you are debugging your agents.
Filtering at the collector
The fix is an OpenTelemetry Collector between the agents and Langfuse, running the filter processor. Conditions are ORed, and a matching span is dropped:
processors:
filter/drop-a2a-noise:
error_mode: ignore
traces:
span:
- 'IsMatch(name, "^a2a\\.server\\.events\\.event_queue\\.EventQueue\\..*")'
- 'IsMatch(name, "^a2a\\.server\\.events\\.in_memory_queue_manager\\..*")'
- 'name == "POST"'
- 'name == "GET"'
- 'name == "POST / http send"'
- 'IsMatch(name, "^POST /api/tasks$")'
- 'IsMatch(name, "^POST /api/sessions/.*/events$")'
Every line is one class of noise measured from a real trace, not a guess. What is deliberately not matched: call_llm, generate_content, and EventConsumer.agent_task_callback — the model calls and the span that marks a delegation hop.
The exporter needs one detail that is easy to get wrong:
exporters:
otlphttp/langfuse:
endpoint: "https://cloud.langfuse.com/api/public/otel"
headers:
Authorization: "${env:LANGFUSE_AUTH_HEADER}"
x-langfuse-ingestion-version: "4"
That is the OTLP base URL, not the /v1/traces path. otlphttp appends the signal path itself, so giving it the full path produces /v1/traces/v1/traces and a silent 404.
The agents then point at the collector instead of at Langfuse, over plaintext inside the cluster:
- otel:
tracing:
enabled: true
exporter:
otlp:
endpoint: "http://otel-collector-opentelemetry-collector.kagent.svc.cluster.local:4318/v1/traces"
protocol: "http/protobuf"
insecure: true
Nice side effect: the Langfuse credentials now live in exactly one pod. Nothing else in the cluster can talk to the SaaS.
The result
| Spans per question | Questions/month on 50k units | |
|---|---|---|
| Before | 278 | ~180 |
| After | 59 | ~845 |
79% reduction, with cost and token attribution fully intact — 12 generations still reporting in/out per call, and the A2A delegation still visible as a tree rather than a pile of disconnected traces.
Field notes
- A "unit" is not a request, and this is the whole trap. Every pricing page for LLM observability quotes a number that sounds like request volume. On a multi-agent system it is closer to span volume, which for A2A frameworks is one to two orders of magnitude higher. Trace one real request and count before you estimate anything.
-
Distributed context propagation is the difference between a trace and a pile of traces. kagent handles it, so delegations nest correctly. If you build agents yourself,
inject/extractthe trace context across the hop or you get N disconnected root traces and no way to reconstruct who called whom. - Filter at the collector, not at the source. Turning down instrumentation in the agents means losing spans you cannot get back when you actually need to debug. A collector filter is a config change you can relax for an afternoon and tighten again, with the full firehose still available on the way in.
- Watch out for the transition window after a rollout. I saw a 229-span trace right after switching endpoints and briefly thought the filter was broken. It was agent pods that had not rotated yet. Wait for the rollout to finish before you judge the numbers.
- The free tier is the right call here, but know what you are sending. Every Telegram message and every agent response goes to a third party for 30 days. Langfuse supports client-side masking — use it if your agents touch anything from a secrets manager.
Closing
The user-facing feature is a text message that says the green container goes out Monday. Everything above it — the router, the delegation, the traces, the filter processor — exists so that when it eventually answers wrong, I can open a single trace and see which agent made the bad call and what it cost. Agent systems are not hard to build any more; they are hard to see into. Two Helm values and seven filter conditions is a cheap price for the difference.
Top comments (0)