DEV Community

Cover image for Making agent contention visible: 6 parallel agents, one API key, and OpenTelemetry span links
Kamal Nayan
Kamal Nayan

Posted on

Making agent contention visible: 6 parallel agents, one API key, and OpenTelemetry span links

#ai #observability #opentelemetry #devops

Run six AI agents in parallel and they collide: two write the same file, four repeat the same API call, everyone queues behind one shared key. A per-agent trace can never show it, because the fact that explains the bug lives in a different agent's trace that nobody thinks to compare.

Here is the failure that started this. Six agents finish "successfully" in SigNoz. The refactorer's diff is on disk, but half its lines are gone because the dep-upgrader wrote the same file a beat later. Every individual trace is green. We built our own version of that failure and could not see it either, so we wrote SwarmScope.

This is a build note for the Agents of SigNoz track. Four things I did not expect: a blank OTEL_* env var that broke our exporter, why one big trace for 6 concurrent agents is worse than 6 linked traces, the one span attribute that made cross-agent contention computable, and a metric-temporality gotcha that returns HTTP 200 with no data.

One honest disclaimer up front: LLM calls in the demo are simulated from a static price table. asyncio.sleep for latency, tokens from a range, cost from a per-model multiplier. Everything else (asyncio concurrency, on-disk I/O, lock contention, OTel spans and metrics) is real.

What we built

An OTel SDK, a contention detector, and a Warden that provisions SigNoz artifacts through the SigNoz MCP server. The demo runs 6 agents with distinct roles (refactorer, tester, doc-writer, dep-upgrader, linter, security-scanner) against a small on-disk workspace and one shared "API key". The detector reports four collision kinds: write_write, read_write, duplicate_work, lease_starvation.

flowchart LR
  A[6 agents] -- spans, metrics --> C[OTel Collector :4318]
  C --> S[SigNoz UI :8080]
  W[Warden] -- JSON-RPC --> M[SigNoz MCP :8000]
  M -- create dashboards / alerts / views --> S
  W -- reads metrics via MCP --> M
  W -- writes control.json --> A
Enter fullscreen mode Exit fullscreen mode

Things I would tell my past self

1. Blank OTEL_* env vars silently break the exporter

The first "no data in SigNoz" hunt ended here:

requests.exceptions.MissingSchema:
Invalid URL '/v1/traces': No scheme supplied. Perhaps you meant https:///v1/traces?
Enter fullscreen mode Exit fullscreen mode

An agent harness two shells up had exported OTEL_EXPORTER_OTLP_ENDPOINT= (empty string, not unset). The exporter's default resolution kept the empty string, so the URL became "" + "/v1/traces". Fix, from swarmscope/sdk/tracing.py:

def _env(name: str) -> str | None:
    """os.getenv, but treats blank strings as unset."""
    value = os.getenv(name)
    return value.strip() if value and value.strip() else None
Enter fullscreen mode Exit fullscreen mode

Every OTEL_* variable in the SDK goes through this. Treat empty strings as unset everywhere.

2. One giant trace with 6 concurrent agents is worse than 6 linked traces

Our first version made each agent a child of a swarm.run root. The flamegraph looked like a barcode: six parallel bars starting near t=0, all with children, and the eye kept reading vertical position as parent-child order (it is not, siblings just stack).

OpenTelemetry has a primitive for exactly this: span links associate a span with spans in another trace without making it a child. Each agent root becomes its own trace, linked back to the run:

@contextmanager
def agent(agent_id: str, role: str):
    parent_ctx = _run_span_context.get()
    links = [Link(parent_ctx, {"swarmscope.link": "swarm_root"})] if parent_ctx else []
    span = tracer.start_span(
        SPAN_AGENT,
        context=trace.set_span_in_context(trace.INVALID_SPAN),
        attributes={A_RUN_ID: rid, A_AGENT_ID: agent_id, A_AGENT_ROLE: role},
        links=links,
    )
Enter fullscreen mode Exit fullscreen mode

trace.set_span_in_context(trace.INVALID_SPAN) is the bit that detaches the span from any ambient parent. Each agent gets a readable flamegraph, and swarm.run_id on every span keeps the fleet correlatable with one filter.

SigNoz traces explorer listing six independent agent traces filtered by a single swarm.run_id
Six agents, six traces, all starting within the same second and joined by span links.

3. Contention is uncomputable without swarm.resource_key on every tool span

Vanilla OTel spans cannot answer "did two agents fight over the same thing", because the spans have no shared name for the thing. We picked one attribute and made it mandatory:

with tool_call(
    "write_file",
    resource_key=f"file:{target}",        # or apikey:openai-main, row:orders:42
    resource_op="write",                  # read | write
    args={"path": target},
    ledger=self.ledger,
):
    ...
Enter fullscreen mode Exit fullscreen mode

Downstream is a sweep line over spans grouped by (run_id, resource_key). Overlap where one op is write gives write_write or read_write. The same (tool, args_hash) from two agents inside 60s gives duplicate_work. lease_wait_ms > 500 gives lease_starvation. That is O(n log n + k), in swarmscope/detect/engine.py. A live in-process ContentionLedger twin runs during the swarm so the dashboard lights up in real time.

SigNoz trace detail showing a swarm.agent span with the span link and the swarm.resource_key attribute expanded
The attribute that makes contention computable: the detector groups tool spans by swarm.resource_key.

4. SigNoz returns HTTP 200 with aggregations: null on the wrong temporality

This is the debugging detail I wish I had found in the docs. Query a metric with the wrong temporality and the server returns 200 OK with no error, no warning, empty result. The OTel Python SDK exports counters as cumulative, so a Delta query silently returns nothing:

temporality = "Delta"        -> results[0].aggregations = null
temporality = "Cumulative"   -> series kind=write_write, values 8, 132
Enter fullscreen mode Exit fullscreen mode

Same query, only that field changed. The fix: omit temporality from the builder query and the MCP tool auto-fetches it from the metric's metadata. The v5 response shape, for anyone parsing it by hand:

data.data.results[].aggregations[].series[].labels[].{key:{name}, value}
data.data.results[].aggregations[].series[].values[].{timestamp, value}
Enter fullscreen mode Exit fullscreen mode

SigNoz metrics explorer querying swarm.contention.collisions grouped by kind
These counters are Cumulative. Querying Delta returns an empty result with no error, so let the metric metadata pick the temporality.

Self-hosting SigNoz with Foundry (and a Docker Desktop trap)

install.sh and deploy/docker-compose are deprecated as of v0.130.0. foundryctl is the supported path now. The whole install is one file:

# deploy/casting.yaml
apiVersion: v1alpha1
kind: Installation
metadata: { name: signoz }
spec:
  deployment: { flavor: compose, mode: docker }
  mcp: { spec: { enabled: true } }
Enter fullscreen mode Exit fullscreen mode

foundryctl apply -f deploy/casting.yaml brings up SigNoz UI on :8080, OTLP HTTP on :4318, MCP on :8000/mcp.

Docker Desktop on macOS refused to mount our first working directory:

error while creating mount source path '/Users/.../Desktop/...':
mkdir /Users/.../Desktop: operation not permitted
Enter fullscreen mode Exit fullscreen mode

~/Desktop is protected by macOS TCC even when Docker has Full Disk Access. Moving the checkout to ~/swarmscope fixed it.

Auth chain in order

  1. POST /api/v1/register with email, password, name. Creates the first admin.
  2. POST /api/v2/sessions/email_password with email, password, and orgID in the body. A missing orgID returns a failure that reads like bad credentials.
  3. POST /api/v1/service_accounts with name. Lowercase and hyphens only, underscores or camelCase are rejected.
  4. POST /api/v1/service_accounts/{id}/keys for the key.
  5. Attach the role via POST /api/v1/service_account_roles with {"serviceAccountId": ..., "roleId": ...}. The nested /service_accounts/{id}/roles route wants a different shape and returned invalid uuid for ours.

Subsequent requests use the SIGNOZ-API-KEY: <key> header.

Metrics, MCP, and the healing loop

Seven custom metrics, names fixed in swarmscope/sdk/attrs.py:

metric what it answers
swarm.agents.active agents alive right now
swarm.contention.collisions contention trend, by kind and resource_key
swarm.resource.wait_ms p95 wait per resource
swarm.tool.calls tool call rate with a duplicate label
swarm.cost.usd simulated spend, per model per agent
swarm.tokens tokens by model and input/output
swarm.remediation.actions what the Warden did about it

Dashboards, alerts, and saved Query Builder views are provisioned through the SigNoz MCP server. Write tools we call: signoz_create_dashboard, signoz_create_alert, signoz_create_view, signoz_create_notification_channel. Read tools (signoz_execute_builder_query, signoz_aggregate_traces) close the loop: the Warden queries collision rates back out, picks an action (leases, banning duplicates, capping concurrency), writes .swarmscope/control.json, and agents pick it up on the next task. Each healing action is a swarm.remediation span, so trace and log line up in the Logs tab.

SigNoz's hosted AI assistant (Noz) is Cloud-only, so we could not point it at our self-hosted install. Fair limitation.

SwarmScope contention dashboard in SigNoz showing collisions over time broken down by kind
Contention over time by kind, on a dashboard the Warden provisioned for itself through the MCP server.

Before and after, same workload

metric chaos run-7528eeba2361 guarded run-c73d441d0ac2
tool calls 24 21
tokens (simulated) 7,531 5,665
cost USD (simulated) 0.0388 0.0290
collisions 88 6
write_write 66 0
read_write 9 0
duplicate_work 13 6
worst contended resource apikey:openai-main (63) none
duplicate calls prevented n/a 3

Side-by-side SigNoz view of the chaos run and the guarded run on the same workload
Same workload, chaos on the left, guarded on the right: 88 collisions drop to 6.

88 to 6 collisions, roughly 25% less simulated cost, same workload. Leases kill write_write and read_write on the hot files. Identical (tool, args_hash) pairs across agents are short-circuited.

Reproduce this 

git clone https://github.com/kamalbuilds/swarmscope && cd swarmscope
foundryctl apply -f deploy/casting.yaml            # SigNoz + MCP up
uv sync
uv run swarmscope provision                         # dashboards, alerts, views via MCP
uv run swarmscope demo --chaos   --agents 6 --tasks 6
uv run swarmscope demo --guarded --agents 6 --tasks 6
uv run swarmscope analyze --run-id <chaos_run_id>
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:8080 and filter any panel by the printed swarm.run_id.

Takeaway

Concurrent agents are a distributed system. Span links plus one shared resource_key attribute turn them back into one you can reason about. The SigNoz feature that carried the most weight was the MCP server's write tools: agents provision their own observability and read it back to act in the same loop. That is the shortest path I found from "agents are opaque" to "agents heal themselves".

Try it

Top comments (0)