Cloud Bill Out of Control: Building a Reliable Budget Alert with OpenCost and Prometheus

You get the email on the 3rd of the month: last month's cloud invoice is 40% above forecast. You open the billing console, and it tells you exactly one useful thing — the account spent too much. Which team? Which workload? Which namespace started a nightly job that requests 64 GiB of RAM and never finishes? The billing console has no idea, because it has never heard of your namespaces.

This article shows how to close that gap on a Kubernetes or OpenShift cluster: deploy OpenCost, expose per-namespace cost as Prometheus metrics, write recording rules that turn raw allocation into dollars per hour, and fire PrometheusRule alerts during the month — not after the invoice lands.

Why native cloud budget alerts arrive too late

They are account-scoped, not workload-scoped

AWS Budgets, Azure Cost Management and GCP Budgets all operate on billing constructs: accounts, projects, subscriptions, tags on cloud resources. A Kubernetes cluster is, from the provider's point of view, a handful of EC2 instances, some EBS volumes and a load balancer. Every namespace on that cluster shares the same bill line. When the budget alert fires, you know the cluster is expensive. You still have to find out why.

They are evaluated on billing data, not live data

Billing pipelines are batch pipelines. Cost and Usage Reports are typically delivered with a delay of 6 to 24 hours, and some line items are only consolidated at the end of the billing period. A runaway Deployment scaled to 300 replicas on Friday evening can easily burn three days of budget before any native alert triggers.

The fix: measure allocation, price it locally, alert in seconds

OpenCost does something simpler and much faster: it watches what the cluster allocates (CPU requests, memory requests, persistent volume size, GPUs), multiplies it by a price per node-hour, and exports the result as Prometheus metrics. Resolution is your scrape interval, not your billing cycle. Accuracy is approximate — it is a model, not an invoice — but a 5-minute-old estimate that points at a namespace beats a 3-day-old number that points at an account.

What OpenCost actually exports

OpenCost is a CNCF project (the open-source core of Kubecost). Once deployed, it exposes a /metrics endpoint with a small, stable set of series. The ones that matter for budgeting:

Metric	Meaning
container_cpu_allocation	vCPU allocated to a container (max of request and usage)
container_memory_allocation_bytes	Bytes of RAM allocated to a container
pod_pvc_allocation	Bytes of PV bound to a pod through a PVC
node_cpu_hourly_cost	USD per vCPU-hour for a node
node_ram_hourly_cost	USD per GiB-hour for a node
node_total_hourly_cost	Full hourly price of a node
pv_hourly_cost	USD per GiB-hour for a persistent volume

Allocation metrics carry namespace, pod, container and node labels. Pricing metrics carry node or persistentvolume. The whole cost model is a join between the two. By default OpenCost uses public on-demand pricing for the detected cloud provider; you can plug in a real billing export later for better accuracy.

Deploying OpenCost on OpenShift

1. Enable user workload monitoring

On OpenShift, the platform Prometheus is read-only for you. You need the user workload stack so your own ServiceMonitor and PrometheusRule objects are picked up.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    alertmanager:
      enabled: true
      enableAlertmanagerConfig: true
    prometheus:
      retention: 30d

oc apply -f cluster-monitoring-config.yaml
oc apply -f user-workload-monitoring-config.yaml
oc -n openshift-user-workload-monitoring rollout status statefulset/prometheus-user-workload

The dedicated user-workload Alertmanager (second ConfigMap) lets you route your alerts without touching platform receivers.

2. Install OpenCost

oc new-project opencost

helm repo add opencost https://opencost.github.io/opencost-helm-chart
helm repo update

helm upgrade --install opencost opencost/opencost \
  --namespace opencost \
  --set opencost.prometheus.internal.enabled=false \
  --set opencost.prometheus.external.enabled=true \
  --set opencost.prometheus.external.url=https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 \
  --set opencost.exporter.defaultClusterId=prod-eu-west-1 \
  --set opencost.ui.enabled=false

Two notes. First, we disable the bundled Prometheus — the cluster already has one. Second, we disable the UI: for alerting we only need the exporter, and one less publicly reachable component is one less thing to secure.

Check that the exporter is producing prices:

oc -n opencost port-forward svc/opencost 9003:9003 &
curl -s localhost:9003/metrics | grep -E '^node_(cpu|ram)_hourly_cost' | head -5

If node_cpu_hourly_cost is 0, OpenCost could not identify your provider or instance types. Check the logs (oc -n opencost logs deploy/opencost) and, if needed, supply a custom pricing file.

3. Scrape it

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: opencost
  namespace: opencost
  labels:
    app.kubernetes.io/name: opencost
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: opencost
  endpoints:
    - port: http
      path: /metrics
      interval: 60s
      scrapeTimeout: 30s
      honorLabels: true

oc apply -f opencost-servicemonitor.yaml
oc -n opencost get servicemonitor

A minute later, query through Thanos:

TOKEN=$(oc whoami -t)
THANOS=$(oc -n openshift-monitoring get route thanos-querier -o jsonpath='{.spec.host}')
curl -sk -H "Authorization: Bearer $TOKEN" \
  "https://${THANOS}/api/v1/query" \
  --data-urlencode 'query=sum(node_total_hourly_cost)' | jq '.data.result'

That single number is your cluster's compute burn rate in USD per hour. Multiply by 730 for a monthly run rate.

Turning allocation into dollars per namespace

Raw metrics are not actionable. We create recording rules so alerts (and dashboards) read from pre-aggregated, cheap series.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: opencost-cost-model
  namespace: opencost
spec:
  groups:
    - name: opencost.cost.recording
      interval: 5m
      rules:
        - record: namespace:cost_cpu_hourly:usd
          expr: |
            sum by (namespace) (
                avg_over_time(container_cpu_allocation[10m])
              * on (node) group_left()
                avg by (node) (node_cpu_hourly_cost)
            )

        - record: namespace:cost_ram_hourly:usd
          expr: |
            sum by (namespace) (
                avg_over_time(container_memory_allocation_bytes[10m]) / 1024 / 1024 / 1024
              * on (node) group_left()
                avg by (node) (node_ram_hourly_cost)
            )

        - record: namespace:cost_pv_hourly:usd
          expr: |
            sum by (namespace) (
                avg_over_time(pod_pvc_allocation[10m]) / 1024 / 1024 / 1024
              * on (persistentvolume) group_left()
                avg by (persistentvolume) (pv_hourly_cost)
            )

        - record: namespace:cost_hourly:usd
          expr: |
            sum by (namespace) (
              {__name__=~"namespace:cost_(cpu|ram|pv)_hourly:usd"}
            )

        - record: namespace:cost_projected_monthly:usd
          expr: avg_over_time(namespace:cost_hourly:usd[6h]) * 730

The key PromQL idiom here is the many-to-one join: * on (node) group_left() avg by (node) (node_cpu_hourly_cost). The left side has thousands of series (one per container); the right side has one per node. group_left() tells Prometheus the left side is the "many" side and that extra labels should be kept. Without it, the query fails with a cardinality error.

avg_over_time(...[10m]) smooths the churn of short-lived pods so a single CronJob pod does not create a spike in the recorded value.

namespace:cost_projected_monthly:usd extrapolates the last 6 hours over 730 hours (the average month). This is the number a budget alert should compare against — it answers "if nothing changes, what will this namespace cost in 30 days?"

Attaching budgets with namespace labels

Hardcoding a threshold per namespace in the rule file does not scale. Instead, label namespaces and let the alert expression read the label.

oc label namespace team-payments cost-tier=medium
oc label namespace team-ml      cost-tier=large
oc label namespace sandbox-bob  cost-tier=small

For the labels to become a metric, you need kube-state-metrics with the namespace label allowlist enabled. The platform kube-state-metrics on OpenShift lives in openshift-monitoring and is not visible to the user-workload Prometheus, so deploy a tiny dedicated instance that only collects namespaces:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm upgrade --install ksm-namespaces prometheus-community/kube-state-metrics \
  --namespace opencost \
  --set 'collectors={namespaces}' \
  --set 'metricLabelsAllowlist={namespaces=[*]}' \
  --set prometheus.monitor.enabled=true

This produces kube_namespace_labels{namespace="team-ml", label_cost_tier="large"} 1. Note that - in a Kubernetes label key becomes _ in the metric label name.

Because the metric value is always 1, you can multiply by it to filter and enrich in one step.

Alerts that fire before the end of the month

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: opencost-budget-alerts
  namespace: opencost
spec:
  groups:
    - name: opencost.budget.alerts
      interval: 5m
      rules:
        - alert: NamespaceBudgetProjectionExceeded
          expr: |
            (
                namespace:cost_projected_monthly:usd
              * on (namespace) group_left(label_cost_tier)
                kube_namespace_labels{label_cost_tier="small"}
            ) > 200
          for: 2h
          labels:
            severity: warning
            team: finops
          annotations:
            summary: 'Namespace {{ $labels.namespace }} projected at ${{ printf "%.0f" $value }}/month (tier small, budget $200)'
            runbook_url: 'https://wiki.internal/runbooks/namespace-budget'

        - alert: NamespaceBudgetProjectionExceeded
          expr: |
            (
                namespace:cost_projected_monthly:usd
              * on (namespace) group_left(label_cost_tier)
                kube_namespace_labels{label_cost_tier="medium"}
            ) > 2000
          for: 2h
          labels:
            severity: warning
            team: finops
          annotations:
            summary: 'Namespace {{ $labels.namespace }} projected at ${{ printf "%.0f" $value }}/month (tier medium, budget $2000)'

        - alert: NamespaceCostSpike
          expr: |
            namespace:cost_hourly:usd
              > 3 * avg_over_time(namespace:cost_hourly:usd[7d] offset 1h)
            and
            namespace:cost_hourly:usd > 0.50
          for: 30m
          labels:
            severity: critical
            team: finops
          annotations:
            summary: 'Namespace {{ $labels.namespace }} burn rate tripled vs 7-day baseline (now ${{ printf "%.2f" $value }}/h)'

        - alert: ClusterMonthlyRunRateHigh
          expr: sum(node_total_hourly_cost) * 730 > 50000
          for: 1h
          labels:
            severity: warning
            team: finops
          annotations:
            summary: 'Cluster run rate is ${{ printf "%.0f" $value }}/month'

oc apply -f opencost-budget-alerts.yaml
oc -n opencost get prometheusrule

Three design choices worth copying:

• for: 2h on projection alerts. A projection built on 6 hours of data is noisy right after a deploy. Requiring the condition to hold for two hours removes almost all false positives.
• A floor on the spike alert (> 0.50 USD/h). Without it, a sandbox namespace going from $0.01/h to $0.04/h pages someone at 3 a.m.
• Spike detection separate from budget detection. A namespace can be well inside budget and still have a broken HPA. Catching the derivative is what saves you money mid-month.

Routing to the right team

oc -n opencost create secret generic finops-slack-webhook \
  --from-literal=url='https://hooks.slack.com/services/REPLACE/WITH/YOUR_WEBHOOK'

apiVersion: monitoring.coreos.com/v1beta1
kind: AlertmanagerConfig
metadata:
  name: finops-routing
  namespace: opencost
spec:
  route:
    receiver: finops-slack
    groupBy: [alertname, namespace]
    groupWait: 1m
    groupInterval: 10m
    repeatInterval: 12h
    matchers:
      - name: team
        value: finops
  receivers:
    - name: finops-slack
      slackConfigs:
        - apiURL:
            name: finops-slack-webhook
            key: url
          channel: '#finops-alerts'
          sendResolved: true
          title: '{{ .CommonAnnotations.summary }}'

A repeatInterval of 12 hours is deliberate: a budget alert is not an incident. You want a daily nudge, not a pager storm.

Security best practices

Give OpenCost read-only, cluster-scoped RBAC — nothing more. The Helm chart creates a ClusterRole limited to get/list/watch on pods, nodes, services, PVs, PVCs and deployments. Never grant cluster-admin, and audit it after upgrades:

oc get clusterrole opencost -o yaml | yq '.rules'

Run as non-root under a restricted SCC. OpenCost does not need privileges, host networking or host paths. On OpenShift it should run under restricted-v2. Verify:

oc -n opencost get pod -l app.kubernetes.io/name=opencost \
  -o jsonpath='{.items[0].metadata.annotations.openshift\.io/scc}{"\n"}'

Do not expose the OpenCost UI or API through a Route without authentication. Cost data is a map of your infrastructure: node types, workload sizes, which teams run what. Keep it ClusterIP and consume it through Prometheus and Grafana, which already sit behind OAuth on OpenShift.

Lock down egress and ingress with NetworkPolicy. OpenCost only needs to reach the Kubernetes API, Prometheus/Thanos and (optionally) the cloud pricing API.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: opencost-allow-monitoring-scrape
  namespace: opencost
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: opencost
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: openshift-user-workload-monitoring
      ports:
        - protocol: TCP
          port: 9003

Never put cloud billing credentials in Helm values or a ConfigMap. If you integrate a real billing export (AWS CUR, Azure Cost Export), use a Secret mounted as a file, and prefer workload identity (IRSA, Azure Workload Identity, GCP Workload Identity) so no long-lived key exists in the cluster at all. Grant that identity read access to the billing bucket only.

Treat namespace labels as a privileged control. A cost-tier=large label raises the alerting threshold tenfold. If developers can self-serve namespaces, enforce the label through a policy engine (Kyverno, OPA Gatekeeper) rather than trusting convention, and alert on namespaces that have no cost-tier label at all.

Keep secrets out of alert annotations. Annotations are rendered into Slack, email and webhooks. Use $labels and $value only; never inject environment-specific tokens or internal URLs that bypass authentication.

Conclusion

Native cloud budgets answer the question "did we spend too much last month?" OpenCost plus Prometheus answers "which namespace is about to make us spend too much, right now?" — and that is the only version of the question you can act on.

Key takeaways:

• OpenCost joins allocation metrics (container_cpu_allocation, container_memory_allocation_bytes, pod_pvc_allocation) with price metrics (node_cpu_hourly_cost, node_ram_hourly_cost, pv_hourly_cost). The entire cost model is a group_left() join.
• Use recording rules to collapse thousands of container series into one namespace:cost_hourly:usd series. Alerts and dashboards should never run the raw join.
• Alert on the projected monthly cost (hourly * 730), not on cumulative spend — projections warn you on day 3, cumulative totals warn you on day 28.
• Add a separate spike alert on the derivative, with an absolute floor to keep tiny namespaces quiet.
• Store budgets as namespace labels exposed by kube-state-metrics, so adding a team is an oc label command and not a pull request on your rules.
• Treat cost data as sensitive: read-only RBAC, no public Route, NetworkPolicy, workload identity for billing credentials.

Start with one tier and two alerts. Once the first NamespaceCostSpike catches a forgotten load test before it runs all weekend, the rest of the model justifies itself.