Your Solace Prometheus Exporter Deserves a Better Pipeline

Your Prometheus Setup Works. That's Exactly the Problem.

You've got Solace running. You've got the Prometheus Exporter scraping it. Grafana dashboards looking sharp. Everything's fine.

So why am I about to tell you to add another component to this stack?

Bear with me.

The Setup Nobody Questions

The standard open source observability stack with Solace looks somewhat like this:

It works on day one. It works on day 180. And then someone says "we're adding Grafana Cloud for the SRE team" or "can you also ship these metrics to Datadog?". Suddenly your clean little pipeline grows a second head. Then a third.

Before you know it, you've got three separate scrape configs all hitting the same exporter at different intervals, three slightly different label schemas because you relabeled things differently per backend, and a Monday morning where two dashboards show different values for the same metric at the same time.

That last one is fun to explain in a postmortem.

One Scrape to Rule Them All

Here's the alternative:

The OpenTelemetry Collector sits between your exporter and everything downstream. It scrapes the exporter exactly once, then fans out to as many backends as you need, simultaneously, with the same data, at the same timestamp.

No drift. No duplicate scrape load. No diverging label configs.

In this demo we run two backends: Prometheus for alerting and existing dashboards, VictoriaMetrics as a long-term storage backend with better compression and a built-in query UI. One Collector feeds both from a single scrape. The full config is in the "Try It" section at the end.

One thing that often gets overlooked in larger deployments: the Solace exporter scrape is not cheap. With hundreds of VPNs and thousands of topics, a single scrape can take several seconds. With three backends scraping independently, you're running that operation three times, potentially overlapping. The Collector runs it once, regardless of how many exporters you add downstream.

Beyond fan-out, the Collector also handles enrichment. Every metric gets stamped with a service label before it reaches any backend:

processors:
  attributes/enrich:
    actions:
      - key: service
        value: broker-1
        action: insert

Every backend, current and future, receives the same label set without any per-backend configuration. You're not maintaining this in three places; you're maintaining it in one.

Your TSDB Has a Long Memory

TSDB storage grows with cardinality. The Solace exporter exposes a comprehensive metric set by design and not everything in it is worth storing. The problem: once a high-cardinality metric is in your TSDB, removing it is painful.

The Collector's filter processor runs before storage, once, for all backends:

processors:
  filter/drop_noise:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - "solace_vpn_connections_service_mqtt"

Set it once. All current and future backends automatically get the filtered dataset. No per-backend drift.

What the Collector Can't Do

The OTel Collector is a pipeline, not a computation engine. One limit worth knowing upfront: you can't calculate new metrics from existing ones inside the Collector.

Take this example: you want solace_vpn_connections_percentage derived from solace_vpn_connections / solace_vpn_quota_connections. Reasonable thing to want. Not possible in the Collector. OTTL (the Collector's transformation language) operates on one datapoint at a time and has no access to other metrics within the same scrape.

For derived metrics, the right place is your query engine:

# Prometheus recording rule
groups:
  - name: solace_derived
    rules:
      - record: solace_vpn_connections_percentage
        expr: solace_vpn_connections / solace_vpn_quota_connections

VictoriaMetrics supports the same syntax. The trade-off: recording rules are per-backend, so a metric defined in Prometheus doesn't automatically exist in VictoriaMetrics. If you need a derived metric in all backends, you need to define it in all of them.

This is worth being deliberate about. Filtering and enrichment belong in the Collector. They apply universally and have no query-engine dependencies. Calculations and aggregations belong in the query engine. That's what it's built for. Mixing the two leads to logic scattered across configs that nobody remembers.

"But Now I Have a New Failure Point"

It's a fair concern. The Collector is an additional component in your metrics path.

The practical answer: the Collector is stateless and fast to restart. Per-exporter retry logic and configurable in-memory buffering mean transient backend outages don't immediately cause gaps. Running it as a sidecar alongside the exporter keeps the operational footprint small. It's not fragile, but it does need to be in your runbooks.

The trade-off is real. For a single backend, the direct pipeline is simpler and sufficient. For anything more complex, the Collector pays back its operational cost quickly.

The Security Angle

In larger deployments, Prometheus and your Solace brokers often live in different network zones. The OTel Collector can sit at that boundary: it handles TLS termination, injects authentication tokens for cloud backends, and means your Prometheus instance never needs a route to the broker network.

That's not a Prometheus feature. That's an architecture feature you get for free with the Collector in the middle.

When to Not Bother

Single backend. Small deployment. You're the only one looking at these metrics. Stick with direct scraping; it's simpler and you have nothing to gain.

Add the Collector when:

• You have two or more metrics backends (or expect to)
• You need consistent labels across broker clusters or regions
• You want cardinality control before storage
• You're in a network-segmented enterprise environment

Try It in 10 Minutes

All config files are available on GitHub. Spin everything up with the files below. You'll need these config files:

docker-compose.yaml

services:
  solace:
    image: solace/solace-pubsub-standard:10.26.0
    shm_size: 1g
    ulimits:
      core: -1
      nofile:
        soft: 2448
        hard: 1048576
    ports: ["8080:8080", "55554:55555"]
    environment:
      username_admin_globalaccesslevel: admin
      username_admin_password: admin

  exporter:
    image: solacecommunity/solace-prometheus-exporter:v1.20.1
    environment:
      SOLACE_SCRAPE_URI: http://solace:8080
    ports: ["9628:9628"]
    depends_on: [solace]

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.155.0
    volumes:
      - ./collector.yaml:/etc/otel/config.yaml
    command: ["--config=/etc/otel/config.yaml"]
    ports: ["8889:8889"]

  prometheus:
    image: prom/prometheus:v3.5.4
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--web.enable-remote-write-receiver"
    ports: ["9090:9090"]

  victoria-metrics:
    image: victoriametrics/victoria-metrics:v1.146.0
    command: ["-retentionPeriod=1d"]
    ports: ["8428:8428"]

collector.yaml

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: solace
          scrape_interval: 10s # for demo purpose
          static_configs:
            - targets: ["exporter:9628"]
          metrics_path: /solace-det

processors:
  filter/drop_noise:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
          - "solace_vpn_connections_service_mqtt"
  attributes/enrich:
    actions:
      - key: service
        value: broker-1
        action: insert

exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
  otlphttp/vm:
    endpoint: "http://victoria-metrics:8428/opentelemetry"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [filter/drop_noise, attributes/enrich]
      exporters: [prometheusremotewrite, otlphttp/vm]

Run docker compose up.

After startup, Prometheus is at localhost:9090 and VictoriaMetrics at localhost:8428/vmui. Query solace_up in either to confirm both backends are receiving the same data from a single scrape. Check that both backends show identical service labels. That's the enrichment working from a single config, not duplicated per backend.

VictoriaMetrics

Prometheus

Note: you don't have to use Remote Write. If you prefer ServiceMonitors, swap the prometheusremotewrite exporter for the prometheus exporter instead:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

Prometheus then scrapes otel-collector:8889/metrics like any other target, via scrape config or ServiceMonitor. No --web.enable-remote-write-receiver flag needed, no changes to your existing Prometheus setup. The rest of the pipeline stays identical.

Conclusion

Start with the Collector from day one, even if you only have one backend. The setup cost is an afternoon. The migration cost later, when your second backend lands and you're untangling three diverged scrape configs, is much higher.

The pipeline that works today is fine. The pipeline that still works when your observability stack doubles in complexity is better.