DEV Community

devtocash
devtocash

Posted on Originally published at devtocash.com

Kubernetes HPA Shows <unknown> Targets (FailedGetResourceMetric): How to Fix It

#kubernetes #devops #sre #autoscaling

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

What an <unknown> target actually means

kubectl get hpa shows cpu: <unknown>/50%, the replica count hasn't moved in days, and kubectl describe hpa is repeating a FailedGetResourceMetric warning. What happened: the HPA controller asked the metrics API for your pods' usage and got nothing it could use — so it did nothing. An HPA that can't read a metric never scales up, never scales down, and never errors loudly; it just holds whatever replica count it last reached. The only place the failure shows up is the HPA's own ScalingActive condition, and the text of that condition tells you which of four things is broken.

The HPA gets metrics through the Kubernetes aggregation layer: metrics.k8s.io for CPU and memory (served by metrics-server, which scrapes every kubelet), custom.metrics.k8s.io for per-pod app metrics (prometheus-adapter), and external.metrics.k8s.io for queue depths and the like (KEDA, or the adapter). A request for a metric fails at one of three points — the API isn't there, the API is there but can't serve, or the API answers but your pods don't qualify. Every FailedGetResourceMetric is one of those, and the fix is different for each.

This is the failure that bites during an incident: traffic spikes, the HPA you set up months ago doesn't add a single replica, and nobody notices because the alert is on latency, not on the autoscaler. It also cuts the other way — an HPA that scaled to 20 during a spike and then lost its metrics stays at 20 until someone fixes it, which is one of the quieter ways a cluster ends up paying for idle capacity.

Step 1: Read the HPA condition, not the TARGETS column 

kubectl describe hpa api -n prod | sed -n '/^Conditions/,/^Events/p'
Enter fullscreen mode Exit fullscreen mode 
Conditions:
  Type            Status  Reason                    Message
  ----            ------  ------                    -------
  AbleToScale     True    ReadyForNewScale          recommended size matches current size
  ScalingActive   False   FailedGetResourceMetric   the HPA was unable to compute the replica count:
                                                    failed to get cpu utilization: ...
Enter fullscreen mode Exit fullscreen mode

AbleToScale is almost always True — it means the controller could write to the target if it wanted to. ScalingActive False is the real signal. The rest of the message is one of these:

unable to fetch metrics from resource metrics API: the server could not find the
requested resource (get pods.metrics.k8s.io)
Enter fullscreen mode Exit fullscreen mode 
unable to fetch metrics from resource metrics API: the server is currently unable
to handle the request (get pods.metrics.k8s.io)
Enter fullscreen mode Exit fullscreen mode 
failed to get cpu utilization: missing request for cpu in container fluent-bit of Pod api-7d9f-x2kq
Enter fullscreen mode Exit fullscreen mode 
failed to get cpu utilization: did not receive metrics for any ready pods
Enter fullscreen mode Exit fullscreen mode 
unable to get metric http_requests_per_second: no metrics returned from custom metrics API
Enter fullscreen mode Exit fullscreen mode 
unable to get external metric prod/s0-prometheus/&{nil scaledobject.keda.sh/name: api}:
no metrics returned from external metrics API
Enter fullscreen mode Exit fullscreen mode

The first two are the metrics API itself — not found means nothing is registered to serve metrics.k8s.io; unable to handle the request means something is registered but the API server can't get a healthy answer from it. missing request and did not receive metrics mean the API works and the problem is your pods. The last two are the custom and external metrics pipelines, which have their own failure modes below. If the events have rotated, the condition message persists on the object: kubectl get hpa api -n prod -o jsonpath='{.status.conditions}' | jq.

Step 2: Is it the metrics API or your pods?

Ask the metrics API directly, bypassing the HPA. This is the fastest way to split the diagnosis:

kubectl get apiservice v1beta1.metrics.k8s.io -o jsonpath='{.status.conditions}' | jq
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/prod/pods" \
  | jq '.items[] | {pod: .metadata.name, cpu: [.containers[].usage.cpu]}'
Enter fullscreen mode Exit fullscreen mode

If the APIService doesn't exist, or its condition is Available: False, the problem is the metrics pipeline — go to fixes 1 and 2. If the raw call returns usage for your pods, the API is fine and the HPA is rejecting the pods for a reason of its own — fixes 3 and 4. kubectl top pod -n prod is a shortcut for the same check, since kubectl top calls exactly this API. If top works and the HPA doesn't, you've already ruled out metrics-server.

Scope matters here too. One HPA broken while others in the cluster work is a pod-side problem, always. Every HPA in the cluster <unknown> at the same time is the API.

Step 3: Fix the actual cause

1. could not find the requested resource: nothing serves the metrics API

There is no v1beta1.metrics.k8s.io APIService, which means metrics-server was never installed, or it was removed with a Helm uninstall that took the APIService with it. Nothing in Kubernetes provides resource metrics by default — kubeadm, kind, k3s (which bundles it), and self-managed EKS/AKS all differ, and EKS in particular does not ship it. Install it and confirm the registration:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl get apiservice v1beta1.metrics.k8s.io
Enter fullscreen mode Exit fullscreen mode

The HPA controller retries every sync period (15 seconds by default), so once the APIService reports Available: True the condition clears on its own. Give it about a minute — metrics-server needs one full scrape cycle before it has anything to serve.

2. currently unable to handle the request: the API is registered but unhealthy

This is the common one on self-managed clusters. The APIService condition tells you which half is broken:

kubectl get apiservice v1beta1.metrics.k8s.io -o jsonpath='{.status.conditions[0].message}'
Enter fullscreen mode Exit fullscreen mode

endpoints for service/metrics-server in "kube-system" have no addresses (reason MissingEndpoints) — the metrics-server pod isn't Ready. Its readiness probe fails until it has successfully scraped at least one kubelet, so a metrics-server that can't reach kubelets never becomes Ready and the APIService stays down. The reason is in its log:

kubectl logs -n kube-system deploy/metrics-server | grep -E "Failed to scrape|Failed probe" | tail -5
Enter fullscreen mode Exit fullscreen mode 
"Failed to scrape node" err="Get \"https://10.0.2.114:10250/metrics/resource\":
  x509: cannot validate certificate for 10.0.2.114 because it doesn't contain any IP SANs" node="ip-10-0-2-114"
"Failed probe" probe="metric-storage-ready" err="no metrics to serve"
Enter fullscreen mode Exit fullscreen mode

The x509 error is the kubelet presenting a self-signed serving certificate, which kubeadm and most bare-metal installs do by default. The correct fix is to have the kubelet request a real serving cert from the cluster CA — serverTLSBootstrap: true in the KubeletConfiguration, then approve the resulting CSRs (or run kubelet-csr-approver so you don't do it by hand after every node join). The pragmatic fix that most clusters actually run with is telling metrics-server not to verify:

kubectl patch deploy metrics-server -n kube-system --type=json -p '[
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}
]'
Enter fullscreen mode Exit fullscreen mode

That's acceptable on a private cluster network where kubelet traffic never crosses a boundary you don't control; on anything multi-tenant, do the CSR approach. The other scrape failure you'll see is dial tcp: lookup ip-10-0-2-114: no such host — metrics-server is trying the node's hostname first and in-cluster DNS can't resolve it. Reorder the address preference:

kubectl patch deploy metrics-server -n kube-system --type=json -p '[
  {"op":"add","path":"/spec/template/spec/containers/0/args/-",
   "value":"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname"}
]'
Enter fullscreen mode Exit fullscreen mode

If the metrics-server pod itself is in CrashLoopBackOff, its crash reason is your root cause — usually a --secure-port clash with hostNetwork, or an old image on a cluster whose kubelet dropped the /metrics/resource endpoint's old format.

failing or missing response from https://10.100.32.7:443/apis/metrics.k8s.io/v1beta1: ... i/o timeout (reason FailedDiscoveryCheck) — metrics-server is Ready, but the API server can't reach it. The kube-apiserver connects to aggregated APIs over the pod network, and three things get in the way:

  • A default-deny NetworkPolicy in kube-system. Cluster-wide default-deny rollouts hit this constantly; the API server's traffic has no pod label to select, so you need an ingress rule on metrics-server that allows the control-plane CIDR to port 4443. If you're deriving policies from observed flows, the apiserver-to-aggregated-API flow is one of the ones that only shows up when an HPA next evaluates, so it's easy to miss in a short observation window.
  • A control plane with no route to the pod network. On kubeadm clusters the API server runs on the host network and reaches ClusterIP services through kube-proxy on the control-plane node; if that node is tainted so nothing schedules there and kube-proxy isn't running, aggregated API calls hang. --enable-aggregator-routing=true on the API server makes it dial endpoint IPs directly instead. GKE private clusters have the equivalent problem as a firewall: the control plane can reach nodes on 443 and 10250 by default, and an aggregated API on 4443 needs its own allow rule.
  • hostNetwork as a workaround. Some managed distributions run metrics-server with hostNetwork: true precisely to sidestep this. It works, but then the secure port must not collide with anything on the node.

3. missing request for cpu in container ...: a container has no request

CPU utilization is a ratio — usage divided by request, summed across every container in the pod. If any single container has no resources.requests.cpu, the denominator is undefined and the HPA refuses to compute it for the whole pod. The message names the container, and it's very often not your app: a log-shipping sidecar, a metrics exporter, or something a mutating webhook injected without requests. Check what the running pods actually carry, not what your Deployment YAML says:

kubectl get pod api-7d9f-x2kq -n prod -o json \
  | jq '.spec.containers[] | {name, cpu_request: .resources.requests.cpu}'
Enter fullscreen mode Exit fullscreen mode

The direct fix is to give every container a request — the same requests that keep pods from being OOMKilled are the ones the autoscaler needs to reason about. The better fix, if the sidecar's usage has nothing to do with your load, is to scale on the app container alone. ContainerResource metrics are GA since Kubernetes 1.30:

metrics:
- type: ContainerResource
  containerResource:
    name: cpu
    container: app
    target:
      type: Utilization
      averageUtilization: 60
Enter fullscreen mode Exit fullscreen mode

This also fixes a subtler problem: a sidecar with a tiny request and real usage inflates the pod-level utilization, so the HPA scales your app on the sidecar's CPU. Pin the metric to the container that actually serves traffic.

4. did not receive metrics for any ready pods: the API works, your pods don't count

The metrics API is up, but it has no samples for the pods the HPA selected. Three causes, in order of likelihood:

The pods are new. metrics-server's default --metric-resolution is 15s, and the HPA additionally ignores a pod's CPU for --horizontal-pod-autoscaler-cpu-initialization-period (5 minutes by default) if the pod wasn't Ready throughout — so a fresh rollout can sit on <unknown> for a few minutes and then recover. If it recovers on its own, that's what you saw. This is also why a new HPA still enforces minReplicas immediately but won't scale beyond it until metrics arrive.

metrics-server can't scrape one node. It scrapes every kubelet independently; a node whose 10250 is firewalled, or whose kubelet is half-way to NotReady, simply produces no samples for its pods. If every pod behind the HPA happens to sit on that node, the HPA sees nothing. kubectl top pod -n prod -o wide shows which pods have no row, and the node column shows the pattern.

The pods aren't Ready. The HPA only counts Ready pods. A failing readiness probe — say, a dependency check that's flapping — takes the pod out of the autoscaler's view and out of the Service at the same time, so the surviving pods take more traffic while the HPA computes on fewer samples. Fix the probe, not the HPA.

When only some pods are missing metrics, the HPA doesn't error — it computes conservatively, assuming missing pods are at 100% of target when deciding to scale down and 0% when scaling up. That dampening is why partial metrics loss looks like "the HPA is sluggish" rather than <unknown>.

5. Custom and external metrics: adapter and KEDA failures

no metrics returned from custom metrics API means prometheus-adapter is registered and answering, but the query it built returned nothing. List what it actually exposes:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq '.resources[].name' | grep http
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/prod/pods/*/http_requests_per_second" | jq
Enter fullscreen mode Exit fullscreen mode

If the metric isn't in the list, the adapter's seriesQuery doesn't match anything in Prometheus. If it's listed but the second call returns an empty items, the series exists but lacks the namespace and pod labels the adapter uses to attach it to your pods — the usual culprit is a ServiceMonitor that relabels pod to something else. The rule has to map the labels you actually have:

rules:
- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total$"
    as: "${1}_per_second"
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
Enter fullscreen mode Exit fullscreen mode

For external metrics there's a trap with no error message that names it: only one APIService can serve v1beta1.external.metrics.k8s.io per cluster. Install KEDA next to a prometheus-adapter that has rules.external configured and whichever registered last silently owns the API — every ScaledObject or adapter-backed HPA on the losing side goes <unknown>. Check who owns it:

kubectl get apiservice v1beta1.external.metrics.k8s.io -o jsonpath='{.spec.service}'
Enter fullscreen mode Exit fullscreen mode

Pick one. The clean split is prometheus-adapter for custom.metrics.k8s.io with its external rules removed, and KEDA for everything external — or drop the adapter and use KEDA's Prometheus scaler for both.

Alert on it — a stalled autoscaler trips no error-rate graph

The HPA's condition is exported by kube-state-metrics, which you already scrape if you run the standard Prometheus stack:

# any HPA that has been unable to compute replicas for 10 minutes
kube_horizontalpodautoscaler_status_condition{condition="ScalingActive",status="false"} == 1
Enter fullscreen mode Exit fullscreen mode

Hold it for: 10m — a single evaluation failing during a rollout is noise. Pair it with the API server's own view of the aggregation layer, which fires before the HPAs do and tells you it's the pipeline rather than the pods:

aggregator_unavailable_apiservice{name=~"v1beta1.(metrics|custom.metrics|external.metrics).k8s.io"} == 1
Enter fullscreen mode Exit fullscreen mode

Also worth a panel, not an alert: kube_horizontalpodautoscaler_status_current_replicas against kube_horizontalpodautoscaler_spec_max_replicas. An HPA pinned at max with ScalingActive False is the expensive version of this failure — it scaled up, lost its metrics, and is now billing you for the peak.

A repeatable checklist

  1. kubectl describe hpa and read the ScalingActive condition message. Could not find / unable to handle = the metrics API; missing request / did not receive metrics = your pods; anything mentioning custom or external metrics = the adapter or KEDA. (Step 1)
  2. kubectl get apiservice v1beta1.metrics.k8s.io and kubectl top pod. If top works, metrics-server is not your problem. (Step 2)
  3. No APIService: install metrics-server, wait one scrape cycle. (Step 3.1)
  4. MissingEndpoints: read the metrics-server log — x509 IP SANs means --kubelet-insecure-tls or proper kubelet serving certs; no such host means --kubelet-preferred-address-types=InternalIP. (Step 3.2)
  5. FailedDiscoveryCheck: the API server can't reach metrics-server — NetworkPolicy on 4443, --enable-aggregator-routing, or a private-cluster firewall rule. (Step 3.2)
  6. missing request for cpu in container X: add a request to container X, or switch the metric to ContainerResource on your app container. (Step 3.3)
  7. did not receive metrics for any ready pods: new pods (wait 5 minutes), one unscrapable node (kubectl top pod -o wide), or a flapping readiness probe. (Step 3.4)
  8. Custom metrics: query the adapter's raw API; fix the seriesQuery or the label overrides. External metrics: only one owner for v1beta1.external.metrics.k8s.io — KEDA or the adapter. (Step 3.5)
  9. Alert on ScalingActive == false for 10 minutes and on aggregator_unavailable_apiservice, and watch for HPAs pinned at maxReplicas — that's the version of this bug that costs money.

Related Reading

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.

Top comments (0)