DEV Community

Cover image for ActiveGate pod not ready: readiness 503 and kubectl diagnosis
Span Observer
Span Observer

Posted on Originally published at span.observer

ActiveGate pod not ready: readiness 503 and kubectl diagnosis

#activegate #dynakube #dynatraceoperator #kubectl

You deploy a DynaKube and the ActiveGate pod gets stuck: kubectl get pods shows 0/1 in the READY column, sometimes for hours. The events mention « Readiness probe failed » with a 503 code. This guide shows how to troubleshoot an ActiveGate pod not ready step by step, without guessing: what the probe tests, what to read in events and logs, which causes to rule out in order, and what to attach to a support ticket.

Goal: get a ready ActiveGate (1/1) again, so routed OneAgents and Kubernetes API monitoring resume. Every minute lost on this pod lengthens your MTTR.

Scope: the behaviour described applies to Dynatrace SaaS and to a Dynatrace Managed cluster driven by Dynatrace Operator. The documentation pages read are the SaaS ones; for Managed, check compatibility between your cluster version and the Operator version (see Dynatrace Operator on Managed).

ActiveGate pod not ready: what the probe really tests

When the Operator creates the ActiveGate of a DynaKube, it deploys it as a StatefulSet with two HTTPS probes on port 9999. The values below come from the source code of the Dynatrace/dynatrace-operator repository (Apache 2.0 licence, main branch on 30 October 2026, file probe.go). They may change between versions: re-read the file for the version you run.

Go code from the dynatrace-operator repository: Readiness probe on /rest/health and Liveness probe on /rest/state, HTTPS port, 90-second initial delay

Probe Path Initial delay Period Failures tolerated Timeout
Readiness /rest/health 90 s 15 s 3 2 s
Liveness /rest/state 90 s 30 s 2 1 s

Three practical consequences. First, a pod at 0/1 during the first 90 seconds is normal: the first probe has not run yet. Second, according to the Kubernetes documentation, an HTTP code greater than or equal to 200 and below 400 is a success and any other code is a failure: a 503 on /rest/health therefore means the ActiveGate process answers but does not yet declare itself healthy. Third, the liveness probe is stricter: per the same documentation, a failed liveness check makes the kubelet kill and restart the container. A pod restarting in a loop does not have the same profile as a stable pod stuck at 0/1.

The pod name follows a convention visible in the sample output of the Operator support archive: <dynakube name>-activegate-0, with the StatefulSet <dynakube name>-activegate. The container is named activegate (constant in the source code).

Troubleshoot step by step

The Troubleshooting page of the Operator documentation gives the basic commands. Here is an order that avoids going in circles. Replace the pod name with yours.

  1. Observe the state. List the pods in the dynatrace namespace and the DynaKube. Note the restart count (RESTARTS column): zero points to an internal health problem, a rising number points to a liveness failure or an out-of-memory kill.
  2. Read the pod events. kubectl describe pod prints them at the bottom. Look for Readiness probe failed lines and their code, then Liveness probe failed, Back-off, or scheduling (FailedScheduling) and volume mount errors.
  3. Read the container logs with kubectl logs; after a restart, add --previous to get the previous instance’s logs.
  4. Run the Operator diagnostic: the troubleshoot subcommand, detailed below.
  5. Collect a support archive if nothing explains the 503 (last section).

kubectl commands from the Dynatrace Operator Troubleshooting page: get dynakubes, get pods, describe pod, pod logs and Operator logs

kubectl get dynakubes -n dynatrace
kubectl -n dynatrace get pods
kubectl describe pod <pod-name> -n dynatrace
kubectl get events -n dynatrace --sort-by=.metadata.creationTimestamp
kubectl logs <pod-name> -n dynatrace --previous
Enter fullscreen mode Exit fullscreen mode

The first three commands appear in the Operator documentation. --sort-by and --previous are standard kubectl options: confirm them with --help on your version.

Causes to rule out, from most to least likely

1. Out of memory (OOMKilled)

The resources field of the DynaKube spec.activeGate section carries the container requests and limits; the documentation says to adapt them to the monitored workload and lists no default value. An undersized ActiveGate can be killed and restarted. Since Operator 1.10.0, according to the release notes, the troubleshoot command reports recently killed ActiveGate pods (OOMKilled) in its troubleshoot.txt file.

2. Network and start-up dependencies

An ActiveGate that cannot reach the environment API does not become healthy. From an ephemeral pod, test the API URL as in the article on installing the Operator: a proxy, a firewall or an unrecognised certificate are the usual suspects. To choose between environment and cluster ActiveGates, see environment vs cluster ActiveGate.

3. Custom TLS certificate

According to the DynaKube parameters page, tlsSecretName names a secret holding the certificate, key and password; without it, a self-signed certificate is used. The metrics-ingest and dynatrace-api capabilities require a custom certificate. A malformed or missing secret is therefore a serious lead if you enabled one of them.

4. Scheduling and cluster resources

A pod in Pending is not a probe failure: it is a scheduling problem (resources, taints, selectors). Also note that, per the 1.10.0 release notes, the ActiveGate topology spread constraint now uses ScheduleAnyway, which makes updates easier on small clusters but restarts the ActiveGate on upgrade.

5. Dependencies between components

The same notes describe two fixes tied to start-up order: a mutual deadlock when KSPM is enabled and the in-cluster ActiveGate is the only one in the network zone, and, when networkZone is set, a wait for the routing ActiveGate to register before OneAgent is deployed. If you run an older version, the Operator update may be enough.

Use troubleshoot and the support archive

The following command runs the Operator checks (namespace, CRD and DynaKube present, API URL format, and so on):

kubectl exec deploy/dynatrace-operator -n dynatrace -- dynatrace-operator troubleshoot
Enter fullscreen mode Exit fullscreen mode

The ActiveGate check lists the pods carrying the DynaKube labels and raises a warning for each container whose last termination state is OOMKilled, with the time and exit code. A result without warnings only rules out the basic configuration and memory.

For a support ticket, the documentation describes two archives. The first comes from the Operator; it includes the ActiveGate logs when one is deployed (file <dynakube name>-activegate-0/activegate.log) and the StatefulSet manifest:

kubectl exec -n dynatrace deployment/dynatrace-operator -- dynatrace-operator support-archive --stdout > operator-support-archive.zip
Enter fullscreen mode Exit fullscreen mode

The documentation warns that, on a large cluster, running this command inside the Operator pod can cause an out-of-memory kill of its container; it offers a standalone-pod variant. The second archive comes from the ActiveGate itself (agctl, ActiveGate 1.333 and later for containers; before that, the deprecated create-support-archive command applies):

kubectl exec command running agctl support-archive create --stdout in an ActiveGate pod, redirected to a zip file, excerpt from the ActiveGate diagnostics page

The documentation states that PowerShell is not supported for this redirection: use cmd.exe on Windows. Review the archive before sending it: it may contain internal names.

How to check that it works

  • kubectl -n dynatrace get pods shows 1/1 for <dynakube>-activegate-0, and RESTARTS stops moving.
  • kubectl describe pod shows no new Readiness probe failed line after the time of your fix.
  • The DynaKube is up to date: kubectl get dynakubes -n dynatrace no longer shows a phase error.
  • On the Dynatrace side, agents are routed again and Kubernetes monitoring resumes. To be warned next time, create an alert on pods that are not ready, see Kubernetes alerts on nodes, pods and quotas.

Best practices

  • Set resources for the ActiveGate rather than leaving the pod unbounded: it is the first lever against restarts.
  • Keep a recent Operator version and read its release notes before each upgrade.
  • Capture the events before deleting the pod: a new pod wipes the trail.

Frequently asked questions

Why does my ActiveGate pod stay at 0/1 after start-up?

The readiness probe queries /rest/health over HTTPS on port 9999 after 90 seconds, then every 15 seconds. Until the answer is a success code, the pod is not ready. Read the events and logs to find out why the ActiveGate does not declare itself healthy.

Is a 503 a configuration error on my side?

Not necessarily. It means the server answers without being healthy. The cause is in the ActiveGate logs: memory, access to the environment, certificate, unfinished start-up. The documentation read gives no official list of causes for a 503 on this route: the leads above are a method, not a catalogue.

Should I delete the pod to fix it?

Only after capturing events and logs. Deleting the pod recreates it with the same configuration: if the cause is a memory limit or a wrong TLS secret, the new pod fails the same way.

Conclusion

An ActiveGate pod not ready is no mystery: a precise probe, events, logs and one diagnostic command are enough to choose between memory, network, certificate and scheduling.

This article comes from an independent blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Probe values come from the public Dynatrace Operator source code and may change with the version.

Originally published on span.observer. French version: https://span.observer/fr/activegate-kubernetes-not-ready/.

Top comments (0)