A three-node Dynatrace Managed cluster in a single data centre survives the loss of one server. But what happens if the whole room goes down: a power cut, a major network incident, a disaster? Your observability tool disappears at exactly the moment you need it most, and restoring from a backup takes hours. This is precisely the scenario Premium High Availability is designed for.
For organisations whose business continuity plan requires two active sites, Managed offers the Premium High Availability option: a single cluster spread across two data centres, with data replicated between them. This feature is specific to Managed, since you operate both sites, the link between them and the traffic distribution.
This article explains the principle, the infrastructure constraints and the implementation approach, then details operational best practices.
The principle of Premium High Availability
With Premium High Availability, the cluster no longer just replicates data between nodes on one site (the mechanism described in Dynatrace Managed architecture): it keeps a copy of the data in each of the two data centres. Each site has its own nodes, able to serve the UI, receive agent data and store metrics and configuration.
If one data centre is lost, the other keeps running. For a site outage of up to 72 hours, the cluster automatically resynchronises data when the site returns; beyond that, a recovery procedure applies.
| Criterion | Single-site cluster | Premium High Availability |
|---|---|---|
| Loss of a node | Tolerated (from 3 nodes) | Tolerated |
| Loss of a data centre | Cluster unavailable, restore required | Service continues on the other site |
| Infrastructure | One site | Two sites and a high-performance link |
| Licence | Standard | Specific option |
| Hardware cost | N nodes | Roughly double, as each site must carry the full load on its own |
Infrastructure constraints
Symmetrical sites
Each data centre must be able to absorb the entire load on its own. In practice, you deploy the same number of nodes, with the same profile, on both sites. An undersized site would become the weak point during a failover.
Inter-site link
Synchronisation between data centres is asynchronous: it removes the 10 ms requirement that applies between nodes on one site, but round-trip latency must stay at 100 ms or less, with enough bandwidth for the replication stream on top of normal traffic. Connections between nodes on different sites must be encrypted, and Dynatrace does not supply the certificates required.
Network flows
All inter-node ports must be open between the nodes of both sites. Agents and ActiveGates must be able to reach the nodes in both data centres so they can fail over if their preferred site disappears.
Backups
Replication does not replace backups: a configuration mistake or an accidental deletion is replicated immediately. Keep regular backups on NFS storage, ideally reachable from each site (see backing up and restoring a cluster).
Step-by-step tutorial: extend a cluster to a second data centre
The exact procedure depends on the version and is partly done through the CMC and the Cluster API. The steps below describe the general approach; follow the documentation for your version for the exact screens and calls.
Step 1 — Validate the licence and version
Check that your licence model (Dynatrace Classic or Dynatrace Platform Subscription) supports Premium High Availability and that the cluster is connected to Mission Control: the option does not exist in offline mode. Plan at least three nodes per site (six in total, 30 at most, 15 per site), and bear in mind that the migration cannot be reversed.
Step 2 — Measure the inter-site link
ping -c 100 {site-b-node-ip} | tail -2
mtr -rwc 50 {site-b-node-ip}
Record average latency, jitter and packet loss. Have the network team validate the available bandwidth, taking peaks into account.
Step 3 — Prepare the second site’s nodes
Install servers identical to those on the first site and apply the prerequisites checklist (OS, disks, NTP, DNS). Check that ports are open to each node on site A:
for ip in {node-a1-ip} {node-a2-ip} {node-a3-ip}; do
curl -sv --connect-timeout 5 telnet://$ip:{port} 2>&1 | grep -E "Connected|timed out"
done
Step 4 — Back up the existing cluster
Run a full backup and check that it succeeded: this is your safety net if the extension fails. The procedure then requires backup and automatic updates to be turned off for the whole migration.
Step 5 — Declare the second data centre and add its nodes
Follow the official procedure to declare the new data centre, then add its nodes using the installation command provided by the CMC, as for any node addition. Initial replication of data to the new site can take a long time depending on volume: monitor it and avoid any other operation in the meantime.
Step 6 — Check cluster health
curl -k https://{site-a-node-ip}/rest/health
sudo /opt/dynatrace-managed/utils/cassandra-nodetool.sh status
The first call, run from site B, must answer RUNNING; the second command must list the nodes of both data centres in the UN state.
In the CMC, check that all nodes on both sites are healthy and that replication has finished.
Step 7 — Organise traffic
- Place ActiveGates on each site and use network zones (
--set-network-zone=) so that agents prefer their local site. - Configure DNS or the load balancer for
{your-domain}to serve users from both sites, with health checks.
Step 8 — Test a failover
During a planned window, isolate one site (a controlled network cut rather than a hard shutdown) and check that the UI stays reachable, agents reconnect and data keeps arriving. Then restore the site and follow the resynchronisation.
Best practices
- Size each site to carry 100% of the load, margin included.
- Monitor inter-site link latency continuously and alert when it degrades.
- Test failover at least once a year and document the runbook.
- Keep backups: replication propagates mistakes too.
- Carry out updates and topology changes one site at a time, following the documentation.
- Align Premium HA with your overall continuity plan: it is pointless if the monitored applications are not multi-site themselves.
Frequently asked questions
Does Premium High Availability replace backups?
No. Replication immediately propagates a configuration mistake or an accidental deletion to the other site. Regular backups remain essential.
Can two very distant data centres be used?
Yes, within limits. Synchronisation between sites is asynchronous, but round-trip latency must stay at 100 ms or less, and the link must carry the replication stream on top of normal traffic.
How many nodes does the second site need?
As many as the first, with the same profile: each site must be able to carry the full load on its own during a failover. Hardware cost therefore roughly doubles.
Conclusion
Premium High Availability turns a Managed cluster into a platform that survives the loss of an entire data centre. The price is real: twice the hardware, a demanding inter-site link and stricter operations. For organisations where observability is part of crisis management, however, it is the only way to make sure Dynatrace stays available when everything else is shaking.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Always check values (ports, versions, sizes) in the official documentation for your version of Managed.
Originally published on span.observer. French version: https://span.observer/fr/premium-high-availability-multi-datacenter/.
Top comments (0)