It is a classic paradox: the platform that detects every anomaly in your applications is sometimes the last to know that it is unwell itself. A data disk slowly filling up, a node running out of memory, a backup that has been failing silently for three weeks… and it is on a Monday morning, when dashboards show gaps, that the problem is discovered. Hence the need to watch Dynatrace Managed cluster health with the same rigour as any critical service.
With Dynatrace Managed, that responsibility is entirely yours. The cluster runs on your servers, with your disks, your network and your maintenance windows. Dynatrace provides health indicators and, if the cluster is connected to Mission Control, proactive support; but the underlying infrastructure — storage, operating systems, backup NFS — remains within your scope.
This article suggests a three-layer approach: what the Cluster Management Console shows, what you check at system level, and how to set up external monitoring that does not depend on the cluster itself.
What to monitor for Dynatrace Managed cluster health
A node’s components
Each Managed node hosts several components: the Dynatrace server (processing), Cassandra (metrics and time-series data), Elasticsearch (user data, sessions), transaction storage, as well as the web front end and related services. Each one has its own signs of strain.
| Component | Signal to watch | Consequence if ignored |
|---|---|---|
Data disk (/var/opt/dynatrace-managed) |
Fill rate, weekly growth | Writes stop, data loss |
| Cassandra | Node status, latencies, pending compactions | Incomplete charts, slow UI |
| Elasticsearch | Cluster status (green/yellow/red), disk space | Missing RUM sessions |
| Server | Memory, CPU, incoming processing load | Rejected data, analysis delays |
| Backup | Date and status of the last backup | Restore impossible |
| Certificates | Expiry date | Agents and users disconnected |
Capacity
Beyond the current state, the trend matters. Disk consumption depends on the number of hosts, trace volume, RUM sessions and retention periods. Growth of a few percent per week looks harmless, but it fills a disk within a few months. Size with a comfortable margin set according to your data volume, and keep the same partition size on every node. The CMC also raises an « Insufficient disk space » event when a mount point runs short.
Step-by-step tutorial
Step 1 — Read the CMC health dashboard
Open the CMC at https://{your-domain} and look at the cluster overview, then at each node’s details. Note each node’s status, disk usage per storage type, processing load and any reported health events. Take a reference screenshot during a normal period: it will serve as a baseline for comparison.
Step 2 — Check the system layer on each node
A few standard commands are enough for a daily check:
# Disk space of the Dynatrace directories
df -h /opt/dynatrace-managed /var/opt/dynatrace-managed
# Memory and load
free -h
uptime
# Dynatrace services and any failures
systemctl list-units --type=service | grep -i dynatrace
systemctl --failed
# Recent errors in the system journal
journalctl -p err --since "24 hours ago" | tail -50
The documentation mentions units such as dynatrace-server, dynatrace-cassandra, dynatrace-elasticsearch, dynatrace-nginx, dynatrace-nodekeeper, dynatrace-firewall and dynatrace-security-gateway: check the list on a node of your version and pin it in your scripts. For store health: /opt/dynatrace-managed/utils/cassandra-nodetool.sh status (every node « Up/Normal ») and curl -s 'http://localhost:9200/_cluster/health?pretty' for Elasticsearch.
Step 3 — Check the backup
Check that the NFS share is mounted on all nodes and that the latest backup is recent (setup is covered in backing up and restoring a Managed cluster):
mount | grep -i nfs
ls -lt {backup-mount-point} | head
find {backup-mount-point} -maxdepth 2 -mtime -2 | head
If nothing has changed for more than two days while the backup runs daily, open the backup configuration in the CMC and check its status.
Step 4 — Monitor certificates
echo | openssl s_client -connect {your-domain}:443 -servername {your-domain} 2>/dev/null \
| openssl x509 -noout -enddate
Alert at least 30 days before expiry (for renewal, see SSL certificates and cluster domain). Do the same for the listening port of your Cluster ActiveGates (9999 by default). Note that a Cluster ActiveGate exposed externally must use a domain different from the web UI domain.
Step 5 — Query the Cluster API
With a cluster token holding the Service Provider API permission (the only one this call requires), retrieve node status to feed it into your monitoring tool:
curl -s -o /tmp/cluster.json -w "%{http_code}\n" \
-H "Authorization: Api-Token {token}" \
"https://{your-domain}/api/v1.0/onpremise/cluster"
python3 -m json.tool /tmp/cluster.json
Any code other than 200, or a node whose operationState is not RUNNING, should trigger an alert. The response details are in the Cluster API explorer, and see automating administration with the Cluster API to go further.
Step 6 — Set up external monitoring
The key point: monitoring the cluster must not depend solely on the cluster. Two complementary options:
- Your existing infrastructure monitoring tool (the one already watching your servers) runs the checks from steps 2 to 5 and alerts through an independent channel.
- A second Dynatrace environment or cluster monitoring the first one’s nodes through OneAgent, in a cross-monitoring model. Managed nodes must not be monitored by the environment they host; the documentation suggests, for example, that a pre-production cluster monitors production and vice versa. These OneAgents consume licence. In addition, the built-in Local-Self-Monitoring environment (DDU licensing) exposes the cluster’s
dsfm:metrics without consuming licence.
Add a simple HTTP probe on the login page and on the API URL, from a point outside the cluster.
Step 7 — Enable Mission Control support
If your security policy allows it, connecting to Mission Control lets Dynatrace be informed of certain health problems and help you proactively. In offline mode, this layer does not exist: your internal checks need to be all the more complete.
Best practices
- Independent monitoring: never rely solely on the cluster to report its own failure.
- Early disk thresholds: warning at 70%, critical at 80% (adapt as needed).
- Monthly capacity review: disk growth, number of hosts, RUM volume, retention periods.
- Restore test at least once a year on an isolated environment.
- Homogeneous nodes: same sizing, same OS, same configuration.
- Runbook: document the expected response to each health alert.
Frequently asked questions
Why not let the cluster monitor itself?
Because a platform that is down cannot report its own failure. Checks must run through an independent tool or environment, and Managed nodes must not be monitored by the environment they host.
Which thresholds should you use for the data disk?
A reasonable starting point: warning at 70%, critical at 80%, and enough free space to absorb several weeks of growth. Adjust to your version, your data volume and the weekly growth you observe.
What does offline mode change for monitoring?
Without a connection to Mission Control, Dynatrace is not informed of health problems and cannot help you proactively. Your internal checks (system, backup, certificates, Cluster API) therefore need to be all the more complete.
Conclusion
Monitoring the health of a Managed cluster means combining three viewpoints: the CMC for the platform’s application view, the system for the reality of disks and services, and external monitoring so you are never blind. With a few standard scripts, regular capacity tracking and verified backups, the observability platform stops being a blind spot and becomes a service as well run as the ones it monitors. To link each indicator to the component behind it, keep the Dynatrace Managed architecture map at hand; a freshly installed cluster deserves these checks from day one.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Always check values (ports, versions, sizes) against the official documentation for your Managed version.
Originally published on span.observer. French version: https://span.observer/fr/monitoring-sante-cluster/.
Top comments (0)