Manage OTel Collectors at Scale with OpAMP

If you run more than a handful of OpenTelemetry Collectors, you already know the pain: a config change means SSHing into boxes, redeploying DaemonSets, or babysitting a Git pipeline per cluster, and you never quite trust that every agent is running the config you think it is. OpAMP fixes exactly that. It is a protocol that lets a central server push configuration to a fleet of Collectors, watch their health, and roll changes out in stages, without you touching each host. This post walks through how OpAMP works, the two ways a Collector can speak it, and the config you need to wire one up.

The problem OpAMP solves

A single Collector is easy. A hundred of them, spread across clusters, VMs, and edge nodes, is a fleet-management problem that has nothing to do with telemetry itself. Every observability team eventually builds some version of the same thing: a way to ship a new pipeline config, confirm it actually applied, and back it out when a processor starts dropping spans.

Without a management protocol you end up gluing that together from ConfigMaps, Ansible runs, and dashboards that only tell you an agent is alive, not what config it is actually running. Config drift creeps in. One node keeps an old sampling rate for months because its rollout quietly failed and nobody noticed.

OpAMP, the Open Agent Management Protocol, is the OpenTelemetry answer to this. Splunk donated it to the project in 2022, and it has since become the standard control channel for the Collector. It is worth pairing with a clear-eyed view of what a Collector actually is versus lighter agents; the OpenTelemetry Collector vs Grafana Alloy comparison covers that trade-off if you are still choosing a data plane.

What OpAMP actually is

OpAMP is a client/server network protocol for remote management of large fleets of data-collection agents. It is transport-flexible: agents connect to the server over either plain HTTP or a WebSocket, and the WebSocket path gives you a persistent bidirectional channel so the server can push a new config the moment you save it.

The protocol is deliberately narrow. It handles a specific set of jobs:

• Remote configuration: the server sends a config, the agent applies it and reloads.
• Health and status reporting: agents report whether they are healthy and what they are doing.
• Effective config reporting: agents send back the config they are actually running, so you can detect drift.
• Own-telemetry reporting: agents can stream their own metrics, logs, and traces about themselves.
• Package and version management: the server can discover an agent's version and, optionally, push binary updates.

Notice what is not in that list: OpAMP does not define what your telemetry pipeline looks like. It carries an opaque config blob to the agent and lets the agent decide what to do with it. For a Collector, that blob is just your normal Collector YAML. OpAMP is the envelope, not the letter.

That separation is the whole design. The OpenTelemetry management docs describe OpAMP as the recommended path for fleet management, and the protocol spec is published at opentelemetry.io/docs/specs/opamp.

Two ways a Collector speaks OpAMP

There are two distinct integration points, and mixing them up is the most common early mistake.

Approach	What it is	What it manages
opamp extension	An extension compiled into the Collector	Reports health, effective config, and identity to the server
OpAMP Supervisor	A separate process that wraps the Collector	Full lifecycle: applies remote config, restarts, and reports on the Collector's behalf

The built-in opamp extension is the lightweight option. It lets a Collector announce itself to an OpAMP server and report status, but the extension alone cannot rewrite the Collector's config and restart it, because a running process cannot swap out the config file it booted from and cleanly reload every component.

The Supervisor closes that gap. It is a small parent process that launches the Collector as a child, holds the OpAMP connection itself, and owns the Collector's lifecycle. When a new config arrives, the Supervisor writes it to disk, restarts the Collector against it, and reports the result upstream. For actual remote-configuration-driven fleet management, the Supervisor is the path you want. It lives in the opentelemetry-collector-contrib repository under cmd/opampsupervisor.

Wiring up the Supervisor

The Supervisor takes its own config file, usually supervisor.yaml, which is separate from the Collector config it manages. Here is a representative example:

server:
  endpoint: wss://opamp.example.com:4320/v1/opamp
  tls:
    insecure_skip_verify: false
capabilities:
  accepts_remote_config: true
  reports_effective_config: true
  reports_own_metrics: true
  reports_own_logs: true
  reports_health: true
  reports_remote_config: true
agent:
  executable: /usr/local/bin/otelcol-contrib
storage:
  directory: /var/lib/otelcol-supervisor

Three blocks matter here.

The server block points at your OpAMP backend. The wss:// scheme selects the WebSocket transport, and 4320 is the conventional OpAMP port used across the project's examples. Keep insecure_skip_verify at false in anything real; you are opening a control channel that can change what runs on your hosts, so certificate verification is not optional.

The capabilities block is a set of explicit opt-ins. Nothing is implied. If you want the server to be able to push config, you must set accepts_remote_config: true. If you want drift detection, reports_effective_config: true is what sends the running config back. Turning these on individually means you can start conservative (health only) and add remote config later once you trust the setup.

The agent block tells the Supervisor which binary to run and manage, and storage is where it persists the last-known remote config so a restart does not lose it.

You start the Supervisor, not the Collector, and let it own the child process:

$ otelcol-supervisor --config /etc/otelcol/supervisor.yaml

From here on you never start the Collector directly. The Supervisor connects to the server, sends an AgentDescription that identifies this instance, and waits for config. When you push a new pipeline from the server, the Supervisor lands it on disk and cycles the Collector.

What a remote config flow looks like

Once an agent is connected with accepts_remote_config enabled, the loop is simple to reason about:

1. You edit a Collector config in the server's UI or API and target a set of agents.
2. The server sends the config over the open connection.
3. The Supervisor writes it to its storage directory and restarts the Collector against it.
4. The Collector boots, and the Supervisor reports back the new effective config and health.
5. The server marks the rollout applied for that agent, or surfaces an error if the Collector rejected the config.

Step 4 is the part teams underrate. Because the agent reports its effective config, you get a closed loop: the server does not just assume the push worked, it sees the config the Collector is genuinely running. That is how you catch the node that silently kept an old sampling rate. A dashboard built on reports_effective_config shows you real drift instead of a green checkmark that means nothing.

Health, identity, and self-telemetry

The reporting capabilities are worth turning on even before you trust remote config. With reports_health: true, each agent tells the server whether it is up and functioning, which beats inferring liveness from whether metrics happen to be flowing. Health here means the Collector's own view of itself, including whether its pipelines started cleanly.

Identity comes from the AgentDescription message. Every connecting agent sends a set of attributes about itself: hostname, OS, Collector version, and any custom labels you attach. Those labels are the backbone of fleet management, because they are how you target a subset of agents. You push a config to service.namespace=payments and only those Collectors receive it. Getting your labeling scheme right early is more important than the config content itself; without good labels, every rollout is all-or-nothing.

With reports_own_metrics: true, the Collector streams its internal metrics (queue sizes, dropped spans, export failures) as part of the same channel. Feed those into your existing metrics backend. If you scrape them with Prometheus, the Prometheus documentation covers the receiver side, and Grafana's own agent tooling documented at grafana.com shows how a similar management model looks in a different distribution.

Package management, and why to be careful

OpAMP can also push binary updates. The server can discover an agent's version and, with the right capability enabled, deliver a new package so the agent upgrades or downgrades itself. On paper this is the dream: patch a Collector CVE across a thousand hosts from one console.

In practice, treat auto-update as the most dangerous capability in the protocol and turn it on last. A bad config push restarts a Collector; a bad package push replaces the binary on every targeted host at once. Stage it the way you would any other production rollout: a canary group first, watch health and effective-config reporting, then widen. The protocol gives you the mechanism, not the judgment. Keep package management off until your health and config feedback loops are boringly reliable.

Rolling out across a real fleet

Scale is where the labeling discipline pays off. A sane rollout pattern looks like this:

1. Tag everything. Attach environment, region, and service labels to every agent via its identifying attributes.
2. Canary by label. Push a config change to a small, clearly labeled canary group first.
3. Watch effective config. Confirm the canary agents report the new config as their effective config, not just that they acknowledged the push.
4. Watch health. Give it long enough to catch a pipeline that starts fine but fails under load.
5. Widen in waves. Expand to the next label group, then the rest, with the same two checks each time.

This is the same staged-rollout thinking you would apply to a Kubernetes deployment, and it composes well with cost and reliability work already in your pipeline. If you are instrumenting application workloads at the same time, the collector fleet you manage with OpAMP is what those pipelines feed into; see How to Set Up LLM Observability with OpenTelemetry and, for cluster-scale patterns, LLM Observability on Kubernetes for the data-plane side of the same system.

Picking a server

OpAMP is only half a system; you also need a server that speaks it. You have two routes. You can run a managed or open-source OpAMP backend such as BindPlane, which grew out of the same observIQ work that seeded the protocol, and get a UI, agent inventory, and config management out of the box. Or you can build against opamp-go, the reference server and client implementation, if you want the control channel wired directly into your own platform.

For most teams, starting with an existing server is the right call. The value of OpAMP is the fleet view and the safe rollout mechanics, and rebuilding those from the reference libraries is a real engineering investment. Start managed, learn the operational patterns, and only build your own server if you have a platform reason the off-the-shelf options cannot meet.

Where this leaves you

OpAMP turns a pile of independently configured Collectors into a fleet you can actually operate. The mental model is small: the Supervisor owns the Collector's lifecycle, the capabilities block is a set of explicit opt-ins, and effective-config reporting is what makes the whole thing trustworthy instead of hopeful. Start with health and effective-config reporting so you can see your fleet, add remote config once you trust the feedback loop, and leave package auto-update for last.

If you are still deciding whether the full Collector is even the right data plane for your fleet, settle that first, then bring OpAMP in to manage whatever you land on. The protocol is agnostic about the pipeline; it just makes running a lot of them survivable.