7 Ways to Deploy kube-prometheus-stack in Kubernetes

Soheil Daliri — Sat, 11 Jul 2026 07:02:09 +0000

# Seven Common kube-prometheus-stack Deployment Architectures for Kubernetes

Choosing a monitoring architecture is not just a Kubernetes decision—it's a business decision. Every deployment model trades off operational simplicity, cost, scalability, isolation, and observability.

This guide explores seven common deployment architectures used in production, explains how each works, highlights the hidden trade-offs, and identifies the organizations that benefit most from each approach.

Scenario 1 — Centralized Monitoring (Shared Cluster Monitoring)

The Blueprint

A single kube-prometheus-stack is deployed in a dedicated monitoring namespace.

The stack consists of:

Prometheus
Alertmanager
Grafana
kube-state-metrics
node-exporter

Prometheus is granted a ClusterRole that allows it to discover monitoring targets throughout the entire Kubernetes cluster.

Application namespaces only deploy exporters and monitoring resources such as:

ServiceMonitor
PodMonitor
PrometheusRule

Prometheus automatically discovers and scrapes these resources across namespaces.

The Hidden Catch

Centralization simplifies operations, but every application contributes to the same Prometheus instance.

A slow or misbehaving exporter does not stop Prometheus from scraping the rest of the cluster. However, many slow exporters increase scrape duration, CPU usage, memory consumption, and overall scheduling pressure.

Another concern is scale. As the number of monitored services and active time series grows, a single Prometheus eventually becomes the bottleneck.

Best Fit For

Small to medium Kubernetes clusters
SaaS companies
E-commerce platforms
Organizations with a centralized Platform or SRE team

Not Ideal For

Multi-tenant environments requiring strict data isolation
Very large clusters producing tens of millions of active series

Scenario 2 — Namespace-Scoped Monitoring

The Blueprint

Each namespace deploys its own independent kube-prometheus-stack.

Every namespace owns its own:

Prometheus
Alertmanager
Grafana
Persistent Volume

RBAC is limited to namespace scope using Roles instead of ClusterRoles.

Teams operate independently from one another.

The Hidden Catch

Isolation comes at a significant operational cost.

For example, 30 namespaces may require:

30 Prometheus instances
30 Grafanas
30 Alertmanagers
30 Persistent Volumes
30 Helm releases

Upgrading or backing up the monitoring platform becomes increasingly complex.

Best Fit For

Managed Kubernetes platforms
Air-gapped customer environments
Highly regulated industries
Organizations requiring complete tenant isolation

Not Ideal For

Cost-sensitive environments
Small engineering teams

Scenario 3 — Prometheus Agent with Central Storage (Hybrid)

The Blueprint

Instead of running a full Prometheus server everywhere, each cluster or namespace runs Prometheus Agent.

Agent mode is enabled via --enable-feature=agent on classic Prometheus, or by setting the dedicated agent mode field when running the Prometheus Operator's Prometheus CRD, depending on your Prometheus version.

Agent mode:

Scrapes metrics
Performs service discovery
Maintains a lightweight WAL for reliability
Streams metrics through remote_write

It does not expose:

Query API
Local dashboard
Local long-term TSDB
Local alert evaluation

A centralized backend such as:

Thanos Receive
VictoriaMetrics Cluster
Grafana Mimir

stores all metrics, executes recording rules, and serves Grafana dashboards.

The Hidden Catch

Although Agent mode reduces resource usage dramatically, it depends heavily on the availability of the remote storage.

Temporary network interruptions are handled through the local WAL, but prolonged outages eventually exhaust the local buffer and lead to dropped samples.

Since rules execute centrally, local clusters cannot evaluate alerts independently.

Best Fit For

Large enterprises
Multi-region Kubernetes deployments
Organizations operating thousands of services
Central Platform Engineering teams

Not Ideal For

Home labs
Small clusters
Teams without experience operating distributed observability platforms

Scenario 4 — Multi-Cluster Monitoring

The Blueprint

Each Kubernetes cluster deploys its own kube-prometheus-stack.

A global observability platform such as:

Thanos
Grafana Mimir
VictoriaMetrics

aggregates metrics from every cluster into a single global view:

Cluster A → feeds into the global query layer
Cluster B → feeds into the global query layer
Cluster C → feeds into the global query layer
The global query layer sits on top and gives operators a unified, cluster-wide view

Developers continue using local Prometheus, while operations teams gain cluster-wide dashboards.

The Hidden Catch

Operating multiple Prometheus servers introduces additional complexity:

Object storage
Global deduplication
Cross-cluster authentication
Long-term retention
Disaster recovery

These systems require dedicated operational expertise.

Best Fit For

Enterprises operating multiple Kubernetes clusters
Multi-region deployments
Hybrid cloud environments

Not Ideal For

Single-cluster deployments

Scenario 5 — Batch Jobs and Push-Based Metrics

The Blueprint

Most applications should expose metrics for Prometheus to scrape.

However, short-lived batch jobs disappear before Prometheus has a chance to collect their metrics.

For these workloads, Pushgateway provides a temporary metrics endpoint.

Typical examples include:

CronJobs
Data export jobs
Machine learning training
Database migrations

Prometheus scrapes Pushgateway instead of the completed job.

The Hidden Catch

Pushgateway is not a replacement for service discovery.

Metrics remain stored after a job finishes unless explicitly deleted.

Without cleanup, stale metrics can generate misleading dashboards and false alerts.

Pushgateway should never be used for long-running applications such as:

Django
Redis
PostgreSQL
RabbitMQ

These services should always expose pull-based metrics.

Best Fit For

Batch workloads
Scheduled jobs
CI/CD pipelines

Not Ideal For

Long-running services that can be scraped directly
Environments without a process to clean up stale pushed metrics

Scenario 6 — Service Mesh Monitoring

The Blueprint

When using a service mesh such as Istio, every pod contains an Envoy sidecar.

Envoy automatically exposes networking metrics including:

Request rate
Error rate
Latency
Traffic volume

A Prometheus server scrapes both:

Envoy metrics
Application metrics

using separate scrape configurations.

Application-specific metrics such as database connection pools or cache statistics continue to come from the application itself.

The Hidden Catch

Service mesh telemetry provides excellent network visibility but does not replace application instrumentation.

Running a service mesh also introduces additional CPU, memory, and operational complexity.

Best Fit For

Large microservice platforms
Platform engineering organizations
Teams managing complex service-to-service communication

Not Ideal For

Small Kubernetes clusters
Early-stage startups

Scenario 7 — Managed Observability Platforms

The Blueprint

Instead of managing Prometheus yourself, you deploy a lightweight vendor agent.

Examples include:

Datadog
New Relic
Grafana Cloud
Google Cloud Managed Service for Prometheus
Azure Monitor

Depending on the vendor, metrics may be:

scraped directly by the agent,
forwarded using remote_write,
or collected from an existing Prometheus deployment.

Dashboards, alerting, storage, and retention are managed by the provider.

Worth noting: some of these platforms (Grafana Cloud, Google Cloud Managed Service for Prometheus) are PromQL-compatible, so your existing dashboards, alert rules, and queries mostly carry over. Others (Datadog, New Relic) use their own proprietary query languages, which means migrating in or out later is a heavier lift. That compatibility question is often the real deciding factor for teams evaluating vendors, more so than sticker price.

The Hidden Catch

Managed platforms dramatically reduce operational effort but introduce new concerns:

Ingestion costs
Vendor lock-in
Internet dependency
Data sovereignty
Compliance restrictions

High-cardinality metrics can become extremely expensive.

Best Fit For

Startups
Small DevOps teams
Organizations without dedicated observability engineers

Not Ideal For

Highly regulated industries
Organizations prohibited from sending telemetry outside their infrastructure

Universal Challenge — Metric Cardinality

Every architecture shares one common challenge: metric cardinality.

A metric becomes a unique time series for every distinct label combination.

For example:

http_requests_total{
    endpoint="/login",
    status="200",
    namespace="prod",
    pod="api-17",
    customer_id="12345"
}

If customer_id has 10,000 possible values, Prometheus creates 10,000 different time series for that single metric.

High-cardinality labels dramatically increase:

Memory usage
CPU utilization
Disk storage
Query latency
SaaS ingestion costs

Common high-cardinality labels include:

customer_id
session_id
uuid
request_id
trace_id

Whenever possible, remove these labels before ingestion using metric_relabel_configs.

Choosing the Right Architecture

Ask yourself the following questions.

1. How many Kubernetes clusters do you operate?

One cluster → Scenario 1
Multiple clusters → Scenario 4

2. How large is your environment?

Small to medium → Scenario 1
Large enterprise → Scenario 3

3. Do you require strict tenant isolation?

No → Scenario 1
Yes → Scenario 2

4. Are you monitoring short-lived batch jobs?

Yes → Add Scenario 5
No → Use pull-based monitoring

5. Are you running a service mesh?

Yes → Add Scenario 6
No → Standard Prometheus scraping is sufficient

6. Do you want to operate Prometheus yourself?

Yes → Scenarios 1–6
No → Scenario 7

Final Recommendation

For a home lab or a small production cluster, begin with Scenario 1.

Deploy a single kube-prometheus-stack in a dedicated monitoring namespace, retain approximately 15 days of metrics, and learn Prometheus, Alertmanager, Grafana, and ServiceMonitors thoroughly.

Only introduce architectures such as Prometheus Agent, Thanos, or multi-cluster observability when operational requirements justify the added complexity.

Like Kubernetes itself, observability should evolve with your platform. Start simple, master the fundamentals, and scale only when your business truly needs it.

DEV Community: Soheil Daliri

7 Ways to Deploy kube-prometheus-stack in Kubernetes

Scenario 1 — Centralized Monitoring (Shared Cluster Monitoring)

The Blueprint

The Hidden Catch

Best Fit For

Not Ideal For

Scenario 2 — Namespace-Scoped Monitoring

The Blueprint

The Hidden Catch

Best Fit For

Not Ideal For

Scenario 3 — Prometheus Agent with Central Storage (Hybrid)

The Blueprint

The Hidden Catch

Best Fit For

Not Ideal For

Scenario 4 — Multi-Cluster Monitoring

The Blueprint

The Hidden Catch

Best Fit For

Not Ideal For

Scenario 5 — Batch Jobs and Push-Based Metrics

The Blueprint

The Hidden Catch

Best Fit For

Not Ideal For

Scenario 6 — Service Mesh Monitoring

The Blueprint

The Hidden Catch

Best Fit For

Not Ideal For

Scenario 7 — Managed Observability Platforms

The Blueprint

The Hidden Catch

Best Fit For

Not Ideal For

Universal Challenge — Metric Cardinality

Choosing the Right Architecture

1. How many Kubernetes clusters do you operate?

2. How large is your environment?

3. Do you require strict tenant isolation?

4. Are you monitoring short-lived batch jobs?

5. Are you running a service mesh?

6. Do you want to operate Prometheus yourself?

Final Recommendation