Pushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts

A CronJob exits 0, Pushgateway shows a heartbeat metric that hasn't updated in days, and Prometheus fires nothing. Every dashboard is green while the one thing the heartbeat was supposed to guarantee (that you'd know when the job stops working) has quietly stopped being true.

This is a gotcha about two Kubernetes features that are each fine on their own: the default ndots:5 DNS behavior and namespace-scoped NetworkPolicies. Put them in the same cluster with a wildcard DNS record and a push-based monitoring pattern, and you get a failure mode where the monitoring infrastructure itself is the victim, which means nothing tells you it happened.

What I expected

The heartbeat pattern for batch jobs is simple and well established. Prometheus can't scrape a pod that lived for forty seconds, so instead the job pushes a timestamp to Pushgateway when it finishes, Prometheus scrapes Pushgateway on its normal cycle, and an alert rule fires if the timestamp goes stale. A dead man's switch:

# Last line of the backup job's script
echo "backup_last_success_timestamp $(date +%s)" | curl -s --data-binary @- \
  "http://pushgateway.monitoring.svc.cluster.local:9091/metrics/job/nightly-backup"

- alert: BackupTooOld
  expr: time() - backup_last_success_timestamp > 86400
  for: 15m
  labels:
    severity: warning

If the backup breaks, the timestamp stops advancing, the expression crosses the threshold, and I get paged. That's the theory. The name in the curl command is even the full service FQDN, so DNS should be unambiguous. It is not.

What actually happened

Three independent failures stack here, and each one masks the next. That stacking is the whole gotcha; any single layer would have been a five-minute fix.

Layer 1: ndots:5 turns your FQDN into a wildcard lookup

A pod with the default dnsPolicy: ClusterFirst gets a resolv.conf that looks like this:

search ai-jobs.svc.cluster.local svc.cluster.local cluster.local lab.example.com
nameserver 10.96.0.10
options ndots:5

Note the last search domain. The kubelet appends the node's own search domains after the cluster ones, so if your nodes carry an internal domain like lab.example.com, every pod inherits it.

Now count the dots in pushgateway.monitoring.svc.cluster.local. Four. The ndots:5 option means any name with fewer than five dots is treated as relative, and the resolver walks the search list before trying the name as-is. So your "fully qualified" name generates these queries, in order:

1. pushgateway.monitoring.svc.cluster.local.ai-jobs.svc.cluster.local → NXDOMAIN
2. pushgateway.monitoring.svc.cluster.local.svc.cluster.local → NXDOMAIN
3. pushgateway.monitoring.svc.cluster.local.cluster.local → NXDOMAIN
4. pushgateway.monitoring.svc.cluster.local.lab.example.com → an answer

Step 4 is the trap. If your internal DNS has a wildcard record (*.lab.example.com pointing at your ingress controller's LoadBalancer VIP, a common setup for homelab TLS and per-app hostnames), that query matches the wildcard and returns the ingress VIP. The resolver got an answer, so it stops. The actual absolute name, the one that resolves to the real Pushgateway ClusterIP, never gets queried.

Your heartbeat is now aimed at Traefik on port 9091. I've written about this exact mechanism corrupting TLS validation in Wildcard DNS + ndots:5: The TLS Nightmare; the Pushgateway variant is nastier because the failure has no user-facing symptom at all.

The wildcard also changes the character of the failure. Without it, step 4 would return NXDOMAIN, the resolver would fall through to the absolute name, and everything would work (slower, with three wasted queries, but working). The wildcard converts a harmless inefficiency into a wrong answer.

Layer 2: NetworkPolicy makes the wrong answer unreachable, quietly

The monitoring namespace runs default-deny ingress, the pattern from Network Policies with Calico: Default Deny and Namespace Isolation. Pushgateway has an allow rule for Prometheus scrapes and for the namespaces that existed when the policy was written.

Two things go wrong at this layer. First, the connection isn't even headed to Pushgateway anymore; it's headed to a LoadBalancer VIP that sits outside the pod and service CIDRs. If the workload namespace has an egress policy scoped to in-cluster traffic, that packet gets dropped on the way out. Second, even after you fix DNS, a new namespace (say you just added an ai-jobs namespace for LLM batch work) isn't in Pushgateway's ingress allow-list, so its pushes get dropped on arrival instead.

Either way, the drop is silent by design. NetworkPolicy denies don't send RST packets or ICMP errors with most CNI configurations. The curl just hangs until its TCP timeout, which on a default curl with no --max-time can be over two minutes. The job runtime gets longer and nobody notices, because who watches the runtime of a job that's succeeding?

This is the maintenance-burden face of NetworkPolicy that doesn't show up in tutorials: an allow-list written in March is wrong by August, because namespaces get added and every new one silently falls outside the rules.

Layer 3: both the script and the alert fail open

Here's where it becomes invisible. The push was the last line of the script, and it looked like this in the failure case:

curl -s --data-binary @- "http://..." || true

The || true was there so a flaky Pushgateway wouldn't fail an otherwise-good backup. Reasonable instinct, terrible outcome: the curl times out, the job still exits 0, and Kubernetes reports the CronJob as succeeding.

The alert rule fails open too, and this part catches almost everyone. time() - backup_last_success_timestamp > 86400 only evaluates when the series exists. Pushgateway restarted at some point (it stores pushed metrics in memory unless you enable persistence), the series vanished, and the expression started returning an empty result. Empty isn't "greater than 86400." Empty is nothing. No data, no alert.

So: the DNS layer sends the heartbeat to the wrong place, the network layer drops it without an error, the script layer hides the failure from Kubernetes, and the alerting layer can't fire on a metric that doesn't exist. Four layers, zero signals.

The fix

Work from the inside out: make the failure loud first, then fix the path.

1. Make the push failure fail the job. Drop the || true, add -f so HTTP errors count, and bound the timeout so a network drop fails in seconds instead of minutes:

echo "backup_last_success_timestamp $(date +%s)" | \
  curl -fsS --max-time 10 --retry 2 --data-binary @- \
  "http://pushgateway.monitoring.svc.cluster.local.:9091/metrics/job/nightly-backup"

Notice the trailing dot on the hostname. That marks the name as absolute and bypasses the search list entirely, which makes this one command immune to the ndots problem regardless of pod configuration. It's the cheapest fix in this whole post.

2. Fix ndots at the pod level. The trailing dot fixes one command; dnsConfig fixes every lookup the pod makes:

apiVersion: batch/v1
kind: CronJob
spec:
  jobTemplate:
    spec:
      template:
        spec:
          dnsConfig:
            options:
              - name: ndots
                value: "2"

With ndots:2, any name with two or more dots gets tried as an absolute name first. pushgateway.monitoring.svc.cluster.local resolves on the first query, and even the short form pushgateway.monitoring still works because the search list is consulted after the absolute attempt fails. You lose nothing for cluster-internal names and you stop feeding four-dot FQDNs to your wildcard record.

3. Expand the NetworkPolicy with labels, not names. Enumerating source namespaces by name is what made the policy rot when a new namespace arrived. Select on a label instead:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-pushgateway-clients
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: pushgateway
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              pushgateway-client: "true"
      ports:
        - port: 9091
          protocol: TCP

Then onboarding a new batch namespace is one label away: kubectl label namespace ai-jobs pushgateway-client=true. The policy stops being a file you have to remember to edit and becomes a contract the namespace opts into. Pair it with a Kyverno rule that requires the label on namespaces containing CronJobs if you want it enforced rather than remembered.

4. Alert on absence, not just staleness. The dead man's switch needs to fire when the series is missing, because "missing" is exactly what the failure mode produces:

- alert: BackupHeartbeatMissing
  expr: |
    (time() - backup_last_success_timestamp > 86400)
    or
    absent(backup_last_success_timestamp)
  for: 15m
  labels:
    severity: critical

The absent() function returns a value of 1 when the series doesn't exist, and the or makes the rule cover both stale and gone. One caveat: absent() collapses labels, so if you push heartbeats for several jobs under one metric name with a job label, you need one absent() clause per job (or a recording-rule pattern). Tedious, but the alternative is the exact blind spot this post is about. The reasoning behind alerting on the negative space is the same one I laid out in Prometheus Alerting Rules That Don't Cry Wolf: an alert that can't fire when the pipeline feeding it dies isn't an alert, it's decoration.

5. Verify the path from inside the namespace. Don't trust the fix until you've watched the resolution happen where the workload runs:

kubectl run -n ai-jobs dbg --rm -it --restart=Never \
  --image=nicolaka/netshoot -- \
  getent hosts pushgateway.monitoring.svc.cluster.local

If that returns your ingress VIP instead of a ClusterIP, the wildcard is still winning. Follow with a curl -v --max-time 5 to port 9091 to confirm the NetworkPolicy actually passes traffic, because DNS being right says nothing about the packet getting through.

Why this matters

You'll hit this if your cluster combines three things that are each individually recommended: a wildcard DNS record for ingress convenience, default-deny NetworkPolicies, and push-based heartbeats for batch workloads. Plenty of production clusters and most serious homelabs check all three boxes. The layers interact in a way none of their docs mention, and the interaction specifically targets the observability path, so the usual answer ("the alert will catch it") doesn't apply. The alert is the thing that's broken.

The deeper lesson is about failure direction. Every layer in this chain failed open: the wildcard answered instead of erroring, the policy dropped instead of rejecting, the script swallowed the exit code, the alert evaluated to empty. Whenever you build a monitoring path, walk it backwards and ask what each hop does when its input disappears. Anything that goes quiet instead of loud needs an absent(), a -f, or a timeout wrapped around it. This applies well beyond Pushgateway; the same audit caught a gap in how I verify vector database backups, which I covered in Backing Up Qdrant Snapshots Correctly.

Cross-layer failures like this one, where DNS, network policy, and alerting each look correct in isolation, are the most expensive class of problem in infrastructure work, and they're a big part of the reliability engineering I consult on. The fix is rarely clever. It's knowing that the layers talk to each other, and testing the seams instead of the components.

What I'd tell you to do today, in order: add the trailing dot to your push URLs, add absent() to every dead man's switch you own, and run getent hosts for your internal service names from inside a workload namespace. The first two take five minutes. The third will either confirm you're fine or save you from finding out the hard way that your heartbeats have been landing in Traefik's 404 handler.