A Dynatrace Managed cluster never stays still. The monitored estate grows, a team turns on Real User Monitoring for a high-traffic application, a server reaches the end of its warranty: sooner or later you need to add a node, replace one or remove one. These add/remove node operations look simple, but they directly affect the distributed databases that hold your data.
This is a typical Managed topic: you are responsible for the cluster’s capacity and topology. A poorly prepared addition can saturate the network while data is redistributed; a hasty removal can lose traces or, worse, weaken replication.
This article explains what happens during these operations, then gives two step-by-step procedures: adding a node and cleanly removing a node.
What happens when you add a node
When a new node joins the cluster, it installs the same stack as the others (server, front-end proxy, Cassandra, Elasticsearch, transaction storage). The distributed databases then move part of the data to this node to balance the load. During this phase:
- network traffic between nodes rises noticeably;
- disks on the existing nodes are under read load;
- the new node starts receiving agent traffic once it is declared operational.
The cluster stays available, but it is better to schedule the addition outside peak activity.
What happens when you remove a node
Removing a node does the opposite: its replicated data must be rebuilt elsewhere to maintain the replication factor. Traces held locally in its transaction storage, which is usually not replicated, are lost. You must therefore check that the remaining nodes have enough CPU, memory and above all disk to absorb the load.
| Operation | Typical duration | Main risk |
|---|---|---|
| Adding a node | Full synchronisation within a couple of hours | Network and disk load during redistribution |
| Removing a healthy node | At least 24 hours before removing the next node | Disks filling up on the remaining nodes |
| Replacing a failed node | Variable | A window during which replication is degraded |
These durations are indicative and depend heavily on stored volume and the network: check them in your own context.
Add/remove node: common prerequisites
- A recent, successful full backup (see backing up and restoring a cluster).
- All existing nodes healthy in the CMC and on the same version.
- The new server validated with the same prerequisites checklist as the others (OS, resources, disks, NTP, DNS).
- Inter-node ports open in both directions between the new server and each existing node.
- One topology operation at a time: do not chain an addition and a removal without waiting for the first to finish.
Step-by-step tutorial: add a node
Step 1 — Check cluster health
In the CMC, open the nodes view and check that all of them are healthy, with no update in progress. Check disk space on each node:
df -h /var/opt/dynatrace-managed
Step 2 — Prepare the new server
nproc; free -g
lsblk -o NAME,SIZE,ROTA,MOUNTPOINT
chronyc tracking
curl -sv --connect-timeout 5 https://{existing-node-ip}/ 2>&1 | grep -i connected
The server must have the same hardware profile as the existing nodes.
Step 3 — Get the add-node command from the CMC
In the CMC, open Deployment Status then Install cluster node. The console displays a wget command to run on the new server: it downloads the installer that matches the cluster version. Copy it as is, without modifying it or storing it in a public ticket. If you are reinstalling on a recently removed host, wait 72 hours before reusing the same IP address.
Step 4 — Run the command on the new server
Log in as root and paste the supplied command. It usually takes the form of a download followed by running dynatrace-managed-{version}.sh with the enrolment parameters. Follow the output and answer any questions about storage paths with the same choices as on the other nodes.
Step 5 — Follow the integration
systemctl list-units --type=service | grep -i dynatrace
sudo /opt/dynatrace-managed/utils/cassandra-nodetool.sh status
In the CMC, the node first shows the joining status; in Cassandra it is marked UJ until synchronisation completes. Do not add another node before it has finished. Once all additions are done, cassandra-nodetool.sh cleanup on the older nodes frees the space used by data they are no longer responsible for. Also watch network and disk load on the existing nodes during redistribution.
Step 6 — Update the surrounding ecosystem
- Add the node to the external load balancer if there is one in front of the cluster.
- Check that the certificate covers the new node if agents connect to it by name.
- OneAgents and ActiveGates discover new endpoints automatically; check this on a host with
oneagentctl --get-server. - Check that the backup share is mounted on the new node at the same path.
Step-by-step tutorial: remove a node
Step 1 — Assess remaining capacity
Add up the space used on all nodes and check that the remaining nodes can absorb it with a comfortable margin. With three nodes, do not remove one without adding another first: you would lose full replication.
du -sh /var/opt/dynatrace-managed/* 2>/dev/null | sort -h
Step 2 — Run a backup
Trigger or wait for a full backup and check its status in the CMC.
Step 3 — Remove the node from the CMC
Sign in through a node you are keeping. Under Deployment Status > Cluster nodes, open the node and, from the … menu, first choose Disable OneAgent traffic. The documentation recommends then waiting for the transaction storage retention period (up to 35 days), since that data is not redundant. Finally use Remove node: the node is stopped and uninstalled.
Step 4 — Wait for completion and check
Metric replicas are redistributed across the remaining nodes within 24 hours: do not remove another node before then. Then check the health of the remaining nodes, disk space and the absence of replication alerts.
Step 5 — Clean up the infrastructure
- Remove the node from the load balancer, DNS and firewall rules.
- Decommission the server according to your policy (secure disk wipe).
- Update the architecture documentation and the inventory.
Best practices
- One topology operation at a time, and never during a cluster update.
- To replace a server, add the new node first, then remove the old one.
- Schedule operations outside peak hours and let user teams know.
- Keep nodes identical: do not add a « small » node to a cluster of « large » nodes.
- Monitor the cluster itself (disks, network) with an independent tool or a second environment (see monitoring cluster health).
Frequently asked questions
Does the cluster stay available while a node is added?
Yes. Data redistribution increases network and disk load on the existing nodes, which is why the operation is best scheduled outside peak activity.
What data is lost when a node is removed?
Replicated data is rebuilt on the remaining nodes, but traces in local transaction storage, which is usually not replicated, are lost. Check beforehand that the remaining nodes have enough disk.
How do I replace a server that is reaching end of warranty?
Add the new node first, wait until it is operational, then remove the old one. With three nodes, removing a server before adding another would lose full replication.
Conclusion
Adding or removing a Managed node is a controlled operation as long as you follow a few rules: a healthy cluster, a recent backup, an identical server, enough capacity and patience while data is redistributed. The CMC does most of the work; your job is to prepare the ground and validate the result.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Always check values (ports, versions, sizes) in the official documentation for your version of Managed.
Originally published on span.observer. French version: https://span.observer/fr/ajouter-noeud-cluster-managed/.
Top comments (0)