Signoz monitoring

How we monitor a multi-tenant Kubernetes SaaS across 6 regions (21B metric points/day)

Open
Full disclosure upfront: I work at SigNoz, and this is our engineering team's write-up. Posting because the architecture itself should be useful regardless of what tool you use.

Context: We run a multi-tenant SigNoz Cloud across 3 regional K8S clusters (US/EU/IN). Each tenant gets an isolated namespace with their own SigNoz instance, ClickHouse, and OTel collector. Shared infra (Nginx, OTel gateway, Redpanda) is pooled per cluster.

About 4 years ago, our internal monitoring (which watched all of this) kept crashing under its own telemetry volume. The write-up covers the rebuild:

Daemonsets (one per node) for local metric/log/trace collection, with annotation-driven per-container scraping and not pod-level. We built this ~6 months before the OTel community started considering container-level discovery.

Deployments on a dedicated node pool for synthetic probing of customer endpoints and watching the K8s API for cluster-level events (including persisting K8s events past the default ~1h retention, which has been invaluable for post-incident debugging).

Envoy → OTel Gateway → Redpanda → central SigNoz instance as the buffered pipeline. V1 tried Envoy-only load balancing and it didn't work cuz distributing an overwhelming load across more instances just gives you more overwhelmed instances.

Opt-in via pod annotations so we're not dealing with unnecessary telemetry.