DEV Community

Soheil Daliri
Soheil Daliri

Posted on

7 Ways to Deploy kube-prometheus-stack in Kubernetes

#kubernetes #prometheus #observability #devops

# Seven Common kube-prometheus-stack Deployment Architectures for Kubernetes

Choosing a monitoring architecture is not just a Kubernetes decision—it's a business decision. Every deployment model trades off operational simplicity, cost, scalability, isolation, and observability.

This guide explores seven common deployment architectures used in production, explains how each works, highlights the hidden trade-offs, and identifies the organizations that benefit most from each approach.

Scenario 1 — Centralized Monitoring (Shared Cluster Monitoring)

The Blueprint

A single kube-prometheus-stack is deployed in a dedicated monitoring namespace.

The stack consists of:

  • Prometheus
  • Alertmanager
  • Grafana
  • kube-state-metrics
  • node-exporter

Prometheus is granted a ClusterRole that allows it to discover monitoring targets throughout the entire Kubernetes cluster.

Application namespaces only deploy exporters and monitoring resources such as:

  • ServiceMonitor
  • PodMonitor
  • PrometheusRule

Prometheus automatically discovers and scrapes these resources across namespaces.

The Hidden Catch

Centralization simplifies operations, but every application contributes to the same Prometheus instance.

A slow or misbehaving exporter does not stop Prometheus from scraping the rest of the cluster. However, many slow exporters increase scrape duration, CPU usage, memory consumption, and overall scheduling pressure.

Another concern is scale. As the number of monitored services and active time series grows, a single Prometheus eventually becomes the bottleneck.

Best Fit For

  • Small to medium Kubernetes clusters
  • SaaS companies
  • E-commerce platforms
  • Organizations with a centralized Platform or SRE team

Not Ideal For

  • Multi-tenant environments requiring strict data isolation
  • Very large clusters producing tens of millions of active series

Scenario 2 — Namespace-Scoped Monitoring

The Blueprint

Each namespace deploys its own independent kube-prometheus-stack.

Every namespace owns its own:

  • Prometheus
  • Alertmanager
  • Grafana
  • Persistent Volume

RBAC is limited to namespace scope using Roles instead of ClusterRoles.

Teams operate independently from one another.

The Hidden Catch

Isolation comes at a significant operational cost.

For example, 30 namespaces may require:

  • 30 Prometheus instances
  • 30 Grafanas
  • 30 Alertmanagers
  • 30 Persistent Volumes
  • 30 Helm releases

Upgrading or backing up the monitoring platform becomes increasingly complex.

Best Fit For

  • Managed Kubernetes platforms
  • Air-gapped customer environments
  • Highly regulated industries
  • Organizations requiring complete tenant isolation

Not Ideal For

  • Cost-sensitive environments
  • Small engineering teams

Scenario 3 — Prometheus Agent with Central Storage (Hybrid)

The Blueprint

Instead of running a full Prometheus server everywhere, each cluster or namespace runs Prometheus Agent.

Agent mode is enabled via --enable-feature=agent on classic Prometheus, or by setting the dedicated agent mode field when running the Prometheus Operator's Prometheus CRD, depending on your Prometheus version.

Agent mode:

  • Scrapes metrics
  • Performs service discovery
  • Maintains a lightweight WAL for reliability
  • Streams metrics through remote_write

It does not expose:

  • Query API
  • Local dashboard
  • Local long-term TSDB
  • Local alert evaluation

A centralized backend such as:

  • Thanos Receive
  • VictoriaMetrics Cluster
  • Grafana Mimir

stores all metrics, executes recording rules, and serves Grafana dashboards.

The Hidden Catch

Although Agent mode reduces resource usage dramatically, it depends heavily on the availability of the remote storage.

Temporary network interruptions are handled through the local WAL, but prolonged outages eventually exhaust the local buffer and lead to dropped samples.

Since rules execute centrally, local clusters cannot evaluate alerts independently.

Best Fit For

  • Large enterprises
  • Multi-region Kubernetes deployments
  • Organizations operating thousands of services
  • Central Platform Engineering teams

Not Ideal For

  • Home labs
  • Small clusters
  • Teams without experience operating distributed observability platforms

Scenario 4 — Multi-Cluster Monitoring

The Blueprint

Each Kubernetes cluster deploys its own kube-prometheus-stack.

A global observability platform such as:

  • Thanos
  • Grafana Mimir
  • VictoriaMetrics

aggregates metrics from every cluster into a single global view:

  • Cluster A → feeds into the global query layer
  • Cluster B → feeds into the global query layer
  • Cluster C → feeds into the global query layer
  • The global query layer sits on top and gives operators a unified, cluster-wide view

Developers continue using local Prometheus, while operations teams gain cluster-wide dashboards.

The Hidden Catch

Operating multiple Prometheus servers introduces additional complexity:

  • Object storage
  • Global deduplication
  • Cross-cluster authentication
  • Long-term retention
  • Disaster recovery

These systems require dedicated operational expertise.

Best Fit For

  • Enterprises operating multiple Kubernetes clusters
  • Multi-region deployments
  • Hybrid cloud environments

Not Ideal For

  • Single-cluster deployments

Scenario 5 — Batch Jobs and Push-Based Metrics

The Blueprint

Most applications should expose metrics for Prometheus to scrape.

However, short-lived batch jobs disappear before Prometheus has a chance to collect their metrics.

For these workloads, Pushgateway provides a temporary metrics endpoint.

Typical examples include:

  • CronJobs
  • Data export jobs
  • Machine learning training
  • Database migrations

Prometheus scrapes Pushgateway instead of the completed job.

The Hidden Catch

Pushgateway is not a replacement for service discovery.

Metrics remain stored after a job finishes unless explicitly deleted.

Without cleanup, stale metrics can generate misleading dashboards and false alerts.

Pushgateway should never be used for long-running applications such as:

  • Django
  • Redis
  • PostgreSQL
  • RabbitMQ

These services should always expose pull-based metrics.

Best Fit For

  • Batch workloads
  • Scheduled jobs
  • CI/CD pipelines

Not Ideal For

  • Long-running services that can be scraped directly
  • Environments without a process to clean up stale pushed metrics

Scenario 6 — Service Mesh Monitoring

The Blueprint

When using a service mesh such as Istio, every pod contains an Envoy sidecar.

Envoy automatically exposes networking metrics including:

  • Request rate
  • Error rate
  • Latency
  • Traffic volume

A Prometheus server scrapes both:

  • Envoy metrics
  • Application metrics

using separate scrape configurations.

Application-specific metrics such as database connection pools or cache statistics continue to come from the application itself.

The Hidden Catch

Service mesh telemetry provides excellent network visibility but does not replace application instrumentation.

Running a service mesh also introduces additional CPU, memory, and operational complexity.

Best Fit For

  • Large microservice platforms
  • Platform engineering organizations
  • Teams managing complex service-to-service communication

Not Ideal For

  • Small Kubernetes clusters
  • Early-stage startups

Scenario 7 — Managed Observability Platforms

The Blueprint

Instead of managing Prometheus yourself, you deploy a lightweight vendor agent.

Examples include:

  • Datadog
  • New Relic
  • Grafana Cloud
  • Google Cloud Managed Service for Prometheus
  • Azure Monitor

Depending on the vendor, metrics may be:

  • scraped directly by the agent,
  • forwarded using remote_write,
  • or collected from an existing Prometheus deployment.

Dashboards, alerting, storage, and retention are managed by the provider.

Worth noting: some of these platforms (Grafana Cloud, Google Cloud Managed Service for Prometheus) are PromQL-compatible, so your existing dashboards, alert rules, and queries mostly carry over. Others (Datadog, New Relic) use their own proprietary query languages, which means migrating in or out later is a heavier lift. That compatibility question is often the real deciding factor for teams evaluating vendors, more so than sticker price.

The Hidden Catch

Managed platforms dramatically reduce operational effort but introduce new concerns:

  • Ingestion costs
  • Vendor lock-in
  • Internet dependency
  • Data sovereignty
  • Compliance restrictions

High-cardinality metrics can become extremely expensive.

Best Fit For

  • Startups
  • Small DevOps teams
  • Organizations without dedicated observability engineers

Not Ideal For

  • Highly regulated industries
  • Organizations prohibited from sending telemetry outside their infrastructure

Universal Challenge — Metric Cardinality

Every architecture shares one common challenge: metric cardinality.

A metric becomes a unique time series for every distinct label combination.

For example:

http_requests_total{
    endpoint="/login",
    status="200",
    namespace="prod",
    pod="api-17",
    customer_id="12345"
}
Enter fullscreen mode Exit fullscreen mode

If customer_id has 10,000 possible values, Prometheus creates 10,000 different time series for that single metric.

High-cardinality labels dramatically increase:

  • Memory usage
  • CPU utilization
  • Disk storage
  • Query latency
  • SaaS ingestion costs

Common high-cardinality labels include:

  • customer_id
  • session_id
  • uuid
  • request_id
  • trace_id

Whenever possible, remove these labels before ingestion using metric_relabel_configs.

Choosing the Right Architecture

Ask yourself the following questions.

1. How many Kubernetes clusters do you operate?

  • One cluster → Scenario 1
  • Multiple clusters → Scenario 4

2. How large is your environment?

  • Small to medium → Scenario 1
  • Large enterprise → Scenario 3

3. Do you require strict tenant isolation?

  • No → Scenario 1
  • Yes → Scenario 2

4. Are you monitoring short-lived batch jobs?

  • Yes → Add Scenario 5
  • No → Use pull-based monitoring

5. Are you running a service mesh?

  • Yes → Add Scenario 6
  • No → Standard Prometheus scraping is sufficient

6. Do you want to operate Prometheus yourself?

  • Yes → Scenarios 1–6
  • No → Scenario 7

Final Recommendation

For a home lab or a small production cluster, begin with Scenario 1.

Deploy a single kube-prometheus-stack in a dedicated monitoring namespace, retain approximately 15 days of metrics, and learn Prometheus, Alertmanager, Grafana, and ServiceMonitors thoroughly.

Only introduce architectures such as Prometheus Agent, Thanos, or multi-cluster observability when operational requirements justify the added complexity.

Like Kubernetes itself, observability should evolve with your platform. Start simple, master the fundamentals, and scale only when your business truly needs it.

Top comments (0)