Prometheus is one of the most widely used tools for monitoring Kubernetes environments. It collects metrics, evaluates alerting rules, and helps us understand what is happening inside our applications and infrastructure.
But there is an important question we sometimes overlook:
What happens when Prometheus itself becomes unavailable?
If Prometheus stops collecting metrics, dashboards may no longer reflect the current state of the cluster. Alerting rules may stop being evaluated, and the team might not receive the alerts it depends on to detect other incidents.
This creates a monitoring blind spot: the system responsible for helping us detect problems can become part of the problem.
In this article, we will explore what happens when Prometheus goes down, how to detect the failure, and how to make a monitoring architecture more resilient.
1. What breaks when Prometheus goes down?
The impact depends on how Prometheus is deployed and which components depend on it. However, several consequences are common.
Metrics stop being collected
Prometheus periodically scrapes metrics from configured targets, such as Kubernetes nodes, applications, and exporters.
When Prometheus becomes unavailable, scraping stops. Targets may continue running normally, but their metrics are no longer collected by that Prometheus instance.
After recovery, collection can resume, but the period of downtime may leave gaps in the time series.
Dashboards lose visibility
Grafana can still be running even when its Prometheus data source is unavailable. However, panels that depend on that data source may display errors, incomplete data, or no data.
This is an important distinction: a working dashboard does not necessarily mean the monitoring system is healthy.
Alerting rules may stop being evaluated
Prometheus evaluates alerting rules using the metrics it collects. If Prometheus is down, its rule evaluation stops.
If Alertmanager is running separately, it may continue processing alerts it has already received. However, it cannot receive new alerts from the unavailable Prometheus instance.
As a result, an incident can remain undetected precisely when monitoring is needed most.
2. Why monitoring Prometheus from Prometheus is not enough
Prometheus exposes its own metrics and health endpoints. These are useful for monitoring its internal state, including resource usage, request activity, and operational problems.
For example, Prometheus provides health endpoints such as:
-
/-/healthy— checks whether the server is healthy. -
/-/ready— checks whether the server is ready to serve traffic.
These endpoints are useful for health checks and orchestration.
However, there is a limitation: if Prometheus becomes completely unavailable, it cannot reliably report its own failure to the systems that depend on it.
Imagine the following situation:
- Prometheus stops responding.
- Its metrics are no longer collected.
- Its alerting rules are no longer evaluated.
- The team expects Prometheus to notify them about the outage.
The notification mechanism depends on the component that has failed.
This is why a resilient monitoring architecture needs an independent way to detect monitoring failures.
3. How can we detect a Prometheus outage?
The most important principle is to monitor the availability of the monitoring service from outside its own failure domain.
Option 1: Use an external health check
An external monitoring service or independent health-check system can periodically request the Prometheus health endpoint.
For example:
curl -f http://prometheus.example.com/-/healthy
A successful response indicates that the endpoint is responding as expected. A failed request can trigger an alert through a separate notification mechanism.
The critical detail is that the checker and its alert delivery path must not depend entirely on the Prometheus instance being monitored.
Option 2: Monitor the monitoring stack independently
A separate monitoring instance or an external observability service can check the availability of Prometheus, Grafana, and Alertmanager.
This approach can help detect situations such as:
- Prometheus becoming unreachable.
- Grafana failing to serve dashboards.
- Alertmanager becoming unavailable.
- A monitoring endpoint returning unexpected responses.
The independent system should also have its own reliable notification path, such as email or an incident-management integration.
Option 3: Use Kubernetes health probes appropriately
In a Kubernetes deployment, readiness and liveness probes can help Kubernetes determine whether a Prometheus container is ready or needs to be restarted.
These probes serve different purposes:
- Liveness: helps detect a process that needs restarting.
- Readiness: indicates whether the container should receive traffic.
However, Kubernetes probes are not a replacement for end-to-end monitoring. A container can be running while the overall monitoring service remains inaccessible to users.
4. How can we make Prometheus more resilient?
Detecting an outage is only part of the solution. The architecture should also limit its impact.
Run redundant monitoring instances where justified
For environments that require higher availability, multiple Prometheus instances can scrape the same targets independently.
If one instance fails, another may continue collecting metrics.
However, running multiple instances does not automatically provide a unified view of historical data or eliminate every monitoring gap. The architecture must account for alert duplication, storage, and how users access metrics.
Protect the data you need to retain
Prometheus stores time-series data in its local time-series database (TSDB). In Kubernetes, persistent storage can help preserve that data across pod replacement or restart.
Without suitable persistent storage, a replacement pod may start with an empty database.
Persistent storage does not eliminate gaps caused by downtime, and it is not a substitute for a backup and recovery strategy where one is required.
Test the recovery process
A resilient system should have a recovery procedure that is understood before an incident occurs.
Useful questions include:
- Does Prometheus recover automatically after a pod restart?
- Does it retain the expected historical data?
- Are alerting rules loaded correctly after recovery?
- Does Grafana reconnect to the data source?
- Does the team receive a notification if Prometheus becomes unavailable?
These checks turn assumptions about reliability into verifiable expectations.
5. The QA perspective: validate failure scenarios
Coming from a QA background, I find it useful to think beyond the happy path.
A monitoring stack should not only be validated when every component is healthy. Its behavior during failures matters just as much.
A practical validation checklist could include:
- Stop or make a test Prometheus instance unavailable.
- Verify that the independent health check detects the outage.
- Confirm that the notification reaches the intended destination.
- Restore Prometheus and verify that scraping resumes.
- Check whether historical metrics remain available.
- Verify that dashboards and alerting rules return to normal.
These tests should be performed in a controlled environment, rather than by disrupting a production monitoring system without an approved procedure.
The goal is to validate the entire failure-detection and recovery path, not just the Prometheus process.
Conclusion
Prometheus is an essential observability component, but it is not immune to failure.
When it becomes unavailable, metric collection and alert evaluation can stop, leaving teams with reduced visibility into their infrastructure. Relying exclusively on Prometheus to report its own outage creates a potential blind spot.
The solution is to combine independent health checks, appropriate redundancy, persistent storage where needed, and controlled recovery tests.
A monitoring system is only as useful as its ability to tell us when something is wrong — including when the monitoring system itself is the problem.
For me, this is also where QA and DevOps meet: reliability is not just about designing a system that should work. It is about validating what happens when it does not.
Top comments (2)
The distinction between a working dashboard and a healthy monitoring system is the key point here, because Grafana can look fine while Prometheus is silent. Do you treat the independent health check as part of the same on call rotation as the services it watches, or does it need its own escalation path so a monitoring outage is never queued behind app alerts?
Green Grafana panels over a dead Prometheus is the monitoring version of a smoke alarm with no battery