Monitor a multi-tenant Kubernetes
Context: We run a multi-tenant SigNoz Cloud across 3 regional K8S clusters (US/EU/IN). Each tenant gets an isolated namespace with their own SigNoz instance, ClickHouse, and OTel collector. Shared infra (Nginx, OTel gateway, Redpanda) is pooled per cluster.
About 4 years ago, our internal monitoring (which watched all of this) kept crashing under its own telemetry volume. The write-up covers the rebuild:
Daemonsets (one per node) for local metric/log/trace collection, with annotation-driven per-container scraping and not pod-level. We built this ~6 months before the OTel community started considering container-level discovery.
Deployments on a dedicated node pool for synthetic probing of customer endpoints and watching the K8s API for cluster-level events (including persisting K8s events past the default ~1h retention, which has been invaluable for post-incident debugging).
Envoy → OTel Gateway → Redpanda → central SigNoz instance as the buffered pipeline. V1 tried Envoy-only load balancing and it didn't work cuz distributing an overwhelming load across more instances just gives you more overwhelmed instances.
Opt-in via pod annotations so we're not dealing with unnecessary telemetry.
Top comments (0)