Helpdesk: "Overnight, during the backup window, HV02 dropped out of the Hyper-V cluster for about a minute. The System log on HV01 and HV03 has Event 1135: 'Cluster node 'HV02' was removed from the active failover cluster membership.' A few VMs restarted on other hosts. It happened last Tuesday too. A forum post says to raise SameSubnetThreshold to the maximum so it stops. Can we do that?"
Not yet. Event 1135 isn't a diagnosis. It's the cluster telling you the other nodes stopped hearing a node's heartbeats on every route for longer than the threshold allows, or that its Cluster service stopped. The cause is usually below the cluster: a NIC, a driver, a team or virtual switch, a physical or virtual switch, a firewall or IPsec policy change, a network profile flip, packet loss under load, or real latency between sites. Microsoft's event text says as much: the node "was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster." It's just as plain about the forum fix: "Increasing these values to their maximum might make the event and node removal go away, it just masks the problem. It doesn't fix anything." Work through it as a chain: which node, which nodes logged it, and when → which neighbor events came with it? → is the node up, isolated, or quarantined, and was the Cluster service running? → which cluster networks carry heartbeats, and are there two? → what are the heartbeat values on this cluster, really? → what does the cluster log say about routes at that second? → what did the NIC, driver, team, and switch see? → did a firewall, IPsec, profile, or antivirus change block it? → change one link → prove it stays quiet. Get-WinEvent, Get-ClusterNode, Get-ClusterNetwork, Get-ClusterNetworkInterface, Get-Cluster, Get-ClusterLog, Get-NetAdapter, Get-NetAdapterStatistics, Get-Counter, Get-NetConnectionProfile, Get-NetFirewallRule, and Test-Cluster are the sources for that chain. Raise the thresholds "to be safe" and the next hard failure takes longer to detect, while the dropped packets that caused this one are still dropping.
If the 1135s line up with live migrations, Hyper-V Live Migration Failed (Event 21502) covers the migration network and virtual switch side. If they line up with the backup window and checkpoints fail too, Hyper-V Checkpoint Operation Failed covers the VSS and storage side. This post covers "a Windows Server failover cluster (Hyper-V, SQL Server, file server, or any other role) that logs FailoverClustering Event 1135 (cluster node was removed from the active failover cluster membership), with or without 1177 quorum loss, 1069 resource failures, VMs restarting or going unmonitored, or a node that ends up isolated or quarantined, and you need a read-first path before you raise heartbeat thresholds, evict the node, or rebuild networking."
TL;DR
- Read 1135 on every node, not one. Microsoft: "This event is logged on all nodes in the Cluster except for the node that was removed." Which nodes logged which names tells you whether one node, one site link, or the whole network was the problem (Steps 1 and 2).
- A node is removed only when all its routes are down. Microsoft's heartbeat article: "If all routes are marked down for W2K8-R2-NODE2, it's removed from active failover cluster membership and the Event 1135 that you see in the first section is logged." One working cluster network keeps a node in; that's why you want two (Step 4).
- Heartbeats are UDP 3343. Microsoft: "IPv6 is the default protocol that Failover Clustering will use for its heartbeats. The heartbeat itself is a UDP unicast network packet that communicates over Port 3343." Anything that drops, delays, or blocks that traffic for long enough removes the node (Steps 7 and 8).
-
Read the thresholds before you change them. Defaults differ by Windows Server version and by Microsoft page, so
Get-Cluster | Format-List *subnet*is the only number that counts for your cluster (Step 5). -
The cluster log has the route history.
Get-ClusterLog -TimeSpanwith-UseLocalTimeshows which route was marked down and when; by default its timestamps are GMT (Step 6). - Raising thresholds is a trade, not a fix. Microsoft: "cranking up the thresholds to high values does not fix nor resolve the transient network issue, it simply masks the problem by making health monitoring less sensitive." (Steps 5 and 9).
Where in the chain did it break?
| Step | What should be true | Typical symptom when it breaks |
|---|---|---|
| 1. Symptom captured | Which node was removed, which nodes logged it, the exact times, the cluster's role and platform, and what was running at that moment | "HV02 keeps falling out of the cluster" with no times and no list of which node logged what |
| 2. Event read | Every 1135 on every node is collected, with 1006, 1069, 1127, 1129, 1130, 1177, 1592, 1554, and the isolation and quarantine events around it | One node's System log read; the pattern across nodes never seen |
| 3. Node state | You know whether the node is Up, Down, Paused, isolated, or quarantined, and whether its Cluster service stopped or the network went quiet | A quarantined node that "won't rejoin" treated as a new fault |
| 4. Cluster networks | At least two networks allow cluster communication, each is Up on every node, and you know which carries cluster, live migration, and client traffic | One network for everything; a network partitioned or down on one node; live migration on the heartbeat network |
| 5. Heartbeat values | The delay, threshold, and route history values on this cluster are read, not assumed | A forum's defaults quoted for the wrong Windows version; thresholds already at the maximum |
| 6. Cluster log | The route-down lines for the time of the 1135 are found in local time | System log in local time compared with a cluster log in GMT, an hour or more apart |
| 7. NIC and switch | Drivers, firmware, teaming, virtual switch, and receive buffers are known, and discarded packets are counted | Old NIC drivers; VMXNET3 buffers at defaults; discards climbing under backup load |
| 8. Firewall and policy | UDP 3343 is allowed, the network profile stays Domain, IPsec refreshes and antivirus filters are accounted for | A profile flip to Public, an IPsec policy refresh, or a new filter driver at the time of the 1135 |
| 9. Change | One link fixed, with the node drained first if you touch it | Drivers, thresholds, teaming, and firewall all changed in one maintenance window |
| 10. Proof | No new 1135 through the same load window, routes stay up in the cluster log, and validation is clean | Quiet for a week until the next backup or patch night |
Step 1: Capture the symptom (read-only)
Write these on the ticket first:
| Fact | Why it matters |
|---|---|
| Which node was removed, and which nodes logged 1135 for it | Microsoft's three scenarios are told apart only by comparing every node's events (Step 2) |
| The exact time of each 1135, to the second | You'll match it to the cluster log, the NIC driver events, and the network profile log (Steps 6 to 8) |
| The cluster's role and platform | Hyper-V hosts, guest clusters on VMware, SQL Server in Azure IaaS, and stretched clusters each have their own usual cause (Steps 5 and 7) |
| Sites and subnets | Same-subnet and cross-subnet heartbeats use different values, and a site link failure looks like Microsoft's Scenario B (Steps 2 and 5) |
| What was running at that moment | Backups, live migrations, patching, a switch or firewall change, a Group Policy refresh, an antivirus update |
| How often, and since when | Once after a switch reboot is a different case from every night at 01:00, and both differ from three times in an hour (quarantine, Step 3) |
Then branch on what you see:
| What you see | What it usually means (our reading unless noted) | Start at |
|---|---|---|
| Every other node logged 1135 for the same node | Microsoft's Scenario A: "This quite suggests that at the time of issue, either due to network congestion or otherwise the communication to the NODE A was lost." | Steps 7 and 8 on that node |
| Nodes in one site logged 1135 for the nodes in the other site, and the other way round | Microsoft's Scenario B: "it is indicative that there was a communication failure, over the link that connects these sites." | Steps 4 and 5 |
| Each node logged a different set of names | Microsoft's Scenario C: "Such events are possible when the network channels between the nodes are choked and the cluster communication messages are not reaching in a timely manner" | Steps 4, 6, and 7 |
| 1135 lines up with backups or live migrations | Load on a shared network or host; heartbeats delayed or dropped while it runs | Steps 4 and 7 |
| 1135 lines up with a Group Policy refresh or a firewall change | Microsoft names IPsec policy refreshes and profile changes among the causes | Step 8 |
| The removed node's own log shows its Cluster service stopped or crashed first | The node didn't lose the network; the service went away | Step 3 |
| 1177 on the remaining nodes as well | Quorum was lost, so the whole cluster stopped, not only one node | Steps 2 and 3 |
| The node is quarantined and won't rejoin | It left the cluster too many times in an hour; that's a result of 1135, not a new fault | Steps 3 and 9 |
| Guest cluster on VMware | Microsoft's VMware article: VMXNET3 dropping inbound packets | Step 7 |
| Cluster nodes in Azure IaaS | Microsoft's Azure thresholds article: the defaults "don't consider the possibility of induced latency caused by a multitenant environment" | Step 5 |
Step 2: Read the events on every node (read-only)
1135 is in the System log, from the FailoverClustering source. Microsoft's guidance is to collect them all: "Review the Event ID 1135 that you're seeing on the nodes and copy all the instances of this event." Read every node in one pass, with the events that usually come with it:
# Read-only: FailoverClustering events of interest on every node, last 7 days, oldest first
$ids = 1006, 1069, 1127, 1129, 1130, 1135, 1146, 1177, 1554, 1592, 1653, 1672, 1673, 1681
$since = (Get-Date).AddDays(-7)
(Get-ClusterNode).Name | ForEach-Object -Process {
$n = $_
Get-WinEvent -ComputerName $n -FilterHashtable @{
LogName = 'System'
ProviderName = 'Microsoft-Windows-FailoverClustering'
Id = $ids
StartTime = $since
} -ErrorAction SilentlyContinue |
Select-Object @{n='LoggedOn';e={$n}}, TimeCreated, Id, Message
} | Sort-Object TimeCreated | Format-Table -AutoSize -Wrap
Then build the who-logged-what grid (our form): one row per 1135, with the node that logged it and the node named in the message. If every row names the same node, it's Scenario A; if the names split by site, B; if they're scattered, C.
# Read-only: which node logged 1135 for which node, last 7 days
(Get-ClusterNode).Name | ForEach-Object -Process {
$n = $_
Get-WinEvent -ComputerName $n -FilterHashtable @{ LogName = 'System'; Id = 1135; StartTime = (Get-Date).AddDays(-7) } -ErrorAction SilentlyContinue |
Select-Object @{n='LoggedOn';e={$n}}, TimeCreated, @{n='Removed';e={ ($_.Message -split "'")[1] }}
} | Sort-Object TimeCreated
Read the result against Microsoft's text for each event:
| Event | Symbolic name | Microsoft's message (excerpt) | What it tells you (our reading) |
|---|---|---|---|
| 1135 | EVENT_NODE_DOWN | "was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster." | The node lost all routes, or its service stopped; the rest of this post |
| 1127 | TM_EVENT_CLUSTER_NETINTERFACE_FAILED | "Cluster network interface '%1' for cluster node '%2' on network '%3' failed." | One adapter on one node failed; the start of a route loss (Steps 4 and 7) |
| 1129 | TM_EVENT_CLUSTER_NETWORK_PARTITIONED | "Some attached failover cluster nodes cannot communicate with each other over the network." | One cluster network is split; others may still be carrying heartbeats (Step 4) |
| 1130 | TM_EVENT_CLUSTER_NETWORK_DOWN | "None of the available nodes can communicate using this network." | A whole cluster network is down; a switch or VLAN, not a node (Step 7) |
| 1592 | CLUSSVC_TCP_RECONNECT_CONNECTION_REESTABLISHED | "Network communication was reestablished. This could be due to communication temporarily being blocked by a firewall or connection security policy update." | A short break that recovered; look for the same cause as the 1135 (Step 8) |
| 1006 | NM_EVENT_MEMBERSHIP_HALT | "Cluster service was halted due to incomplete connectivity with other cluster nodes." | The node's own side: its service halted on connectivity |
| 1554 | SERVICE_NETWORK_CONNECTIVITY_LOST | "This cluster node has lost all network connectivity. It cannot participate in the cluster until connectivity is restored." | The node lost every network, not only the heartbeat path |
| 1177 | MM_EVENT_ARBITRATION_FAILED | "The Cluster service is shutting down because quorum was lost." | Enough nodes or the witness were lost that the cluster stopped (Step 3) |
| 1069 | RCM_RESOURCE_FAILURE | "Cluster resource '%1' in clustered role '%2' failed." | A consequence: a resource failed during or after the removal |
| 1146 | RCM_EVENT_RESMON_DIED | "The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted." | Resource health detection, not the network; read which resource |
| 1653 | SERVICE_NO_CONNECTIVITY | "failed to join the cluster because it could not communicate over the network with any other node in the cluster." | The node tried to come back and still can't reach anyone (Step 8) |
| 1673 | CLUSTER_NODE_ENTERED_ISOLATED_STATE | "Cluster node '%1' has entered the isolated state." | Windows Server 2016 and later: the node left membership but still hosts its VMs (Step 3) |
| 1681 | CLUSTER_GROUPS_UNMONITORED_ON_NODE | "Virtual machines on node '%1' have entered an unmonitored state." | VMs kept running on the isolated node, unmonitored |
| 1672 | EVENT_NODE_QUARANTINED | "Cluster node '%1' has been quarantined." | It left too often; it stays out until the quarantine expires or you clear it (Steps 3 and 9) |
Two details matter when you compare with forum posts. First, read the removed node's own System log for the same minute: Microsoft's heartbeat article says that on the removed node "the Cluster Service is terminated and then restarted so it can try to rejoin the Cluster", so service stop and start events there are expected after the 1135, not before it. A service that stopped before the others logged 1135 is a different case (Step 3). Second, Microsoft's 1135 article lists the companion events it expects: 1069 for resources, "Event ID 1177 for Quorum loss", and "Event ID 1006 for Cluster service halted". If 1177 is in the list, the whole cluster went down, and that changes the urgency.
Step 3: Is the node up, isolated, or quarantined? (read-only)
Read node state and the Cluster service before you read the network:
# Read-only: name, id, and state of each node
Get-ClusterNode
# Read-only: the Cluster service on every node
Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock { Get-Service -Name ClusSvc } |
Select-Object PSComputerName, Name, Status, StartType
# Read-only: Microsoft's check that the cluster answers (Windows Server 2012 and later)
Get-ClusterResource
# Read-only: isolation and quarantine settings on this cluster
Get-Cluster | Format-List -Property Quarantine*, Resiliency*
Microsoft's 1135 article checks the service early: "Is the cluster service continuously running and available on all the nodes?" Its firewall section gives the service name, "System service name: ClusSvc".
On Windows Server 2016 and later, a node that loses membership on a cluster with VMs may not go straight to Down. Microsoft's VM compute resiliency article describes two states:
| State | Microsoft's description | What to do (our reading) |
|---|---|---|
| Isolated | "The node is no longer in an active membership" and "The node continues to host the VM role" | Read the network now; it may rejoin on its own |
| Down after isolation | "after a certain period (default of 4 minutes), the VM is failed over to a suitable node in the cluster, and the node is now moved to a Down state." | The VMs moved; read why the node didn't come back |
| Quarantined | "The node is no longer allowed to join the cluster for a fixed time period (default: 2 hours)" | Find the cause before you clear it (Step 9) |
Microsoft's numbers for quarantine: "By default, a node is quarantined, if it ungracefully leaves the cluster, three times within an hour", and "No more than 25% of nodes can be quarantined at any given time". The Get-Cluster reference agrees: QuarantineThreshold "is set to 3 by default", and QuarantineDuration is "set to 7200 seconds or 2 hours by default". Microsoft also notes what isolation does to storage: for block storage (FC, FCoE, iSCSI, SAS) "The VM is placed in the Paused Critical state." while SMB-backed VMs keep running.
| Finding | What it means (our reading unless noted) |
|---|---|
| All nodes Up now, ClusSvc Running everywhere | It recovered; the evidence is in the logs, not the current state (Steps 6 to 8) |
| One node Down, ClusSvc Running on it | It's trying to join and can't; look for 1653 or 1572 on it (Step 8) |
| ClusSvc stopped first on the removed node, before the 1135 on the others | A service or host problem, not a network drop; read that node's System and Application logs for the minute before |
| Node Paused | Someone drained or paused it; that's not a 1135 cause |
| Node quarantined | Event 1672 tells you until when; Microsoft's text gives the way out, "Start-ClusterNode -ClearQuarantine" |
| 1177 on the survivors | Quorum was lost; read the witness and node votes before anything else (Get-ClusterQuorum) |
For the quorum branch, read the configuration once:
# Read-only: quorum type and witness for the local cluster
Get-ClusterQuorum
Microsoft's quorum guidance is short: "We recommend you always configure a witness if you're using Windows Server 2012 R2 and later." A two-node cluster with no witness that loses one node's heartbeats is the case where a single 1135 turns into an outage (our reading).
Step 4: Which cluster networks carry heartbeats? (read-only)
Microsoft's heartbeat article explains why the number of networks matters: each node tracks every route to every other node, and "If other routes are still considered to be up, W2K8-R2-NODE2 will remain as an active member." A node is removed only when the last route goes. Read the networks, their roles, and their metrics:
# Read-only: cluster networks, their state, role, and metric
Get-ClusterNetwork | Format-Table -Property Name, State, Role, Address, Metric, AutoMetric -AutoSize
# Read-only: every node's adapter on every cluster network, and its state
Get-ClusterNetworkInterface | Sort-Object Network, Node | Format-Table -AutoSize
Read the Role column against Microsoft's values for the network Role property:
| Role | Microsoft's name | Microsoft's description |
|---|---|---|
| 0 | ClusterNetworkRoleNone | "The network is not used by the cluster." |
| 1 | ClusterNetworkRoleInternalUse | "The network is used to carry internal cluster communication." |
| 3 | ClusterNetworkRoleInternalAndClient | "The network is used to connect client systems and to carry internal cluster communication." |
Then read the metrics. Microsoft's Metric property: "the network path with the lowest sum of metrics will be used." Microsoft's network prioritization article explains what that means for traffic: the network with the lowest metric carries cluster and CSV traffic, and the second lowest carries live migration traffic. It also explains why live migration on the heartbeat network hurts: "During a live migration, large chunks of memory are copied from one server to another as quickly as possible." and "For this reason it is recommended to have a dedicated network for live migration traffic so that it does not interfere with other important network traffic."
| Finding | What it means (our reading unless noted) |
|---|---|
| Only one network with Role 1 or 3 | Every heartbeat route depends on one path; Microsoft's validation warns about this single point of failure (Step 9) |
| Two or more networks with Role 1 or 3, all Up | Good; a 1135 then means every path failed at once, which points at something shared (a host, team, switch, or policy) |
| A network Partitioned, or one node's interface Failed or Unreachable | That's the route that went; read Step 7 on that node's adapter |
| A network Down on every node | A switch, VLAN, or physical link; Event 1130 (Step 7) |
| The live migration network has the lowest metric, or the same network carries cluster, live migration, and backup traffic | Load on that network can delay heartbeats; compare 1135 times with migrations and backups |
| AutoMetric False on a network | Someone set the metric by hand; read why before you change it |
| An iSCSI network with Role 1 or 3 | Microsoft's prioritization article recommends "Do not allow cluster network communication on this network" for storage traffic |
Microsoft's validation page gives an alternative to separate networks: "you can connect your cluster nodes with one network that is constructed with teamed network adapters, redundant switches, redundant routers, or similar hardware that removes single points of failure." If that's your design, the team and the switches are what you read in Step 7.
Step 5: What are the heartbeat values on this cluster? (read-only)
Two kinds of setting decide how long the cluster waits. Microsoft's definitions: Delay "defines the frequency at which cluster heartbeats are sent between nodes", and Threshold "defines the number of heartbeats, which are missed before the cluster takes recovery action." Read the values this cluster actually has:
# Read-only: Microsoft's form for the heartbeat values
Get-Cluster | Format-List *subnet*
# Read-only: the same, plus cross-site values (Windows Server 2016 and later) and route history
Get-Cluster | Format-List -Property SameSubnet*, CrossSubnet*, CrossSite*, RouteHistoryLength
Don't compare them with a number from a forum. Microsoft's own pages give different defaults depending on the Windows Server version they describe:
| Property | Microsoft's property reference (minimum, maximum, default) | Microsoft's tuning article (Windows Server 2012 R2, 2016, maximum) |
|---|---|---|
| SameSubnetDelay | 250, 2000, 1000 (milliseconds) | 1 second, 1 second, 2 seconds |
| SameSubnetThreshold | 3, 120, 5 | 5 heartbeats, 10 heartbeats, 120 heartbeats |
| CrossSubnetDelay | 250, 4000, 1000 (milliseconds) | 1 second, 1 second, 4 seconds |
| CrossSubnetThreshold | 3, 120, 5 | 5 heartbeats, 20 heartbeats, 120 heartbeats |
| CrossSiteDelay | (not on the reference pages fetched) | NA, 1 second, 4 seconds |
| CrossSiteThreshold | (not on the reference pages fetched) | NA, 20 heartbeats, 120 heartbeats |
| RouteHistoryLength | 0, 40, 10 | "By default the RouteHistoryLength setting is set 10, which is two times the number of default thresholds." |
The property reference pages list Windows Server 2012 as the minimum supported server. Microsoft's Azure IaaS article states the newer default: "By default, Windows Server sets the SameSubnetThreshold to 10 and SameSubnetDelay to 1000 ms." The tuning article adds a version trap: "If the Hyper-V role is installed on a Windows Server 2012 R2 Failover Cluster, the SameSubnetThreshold default will automatically be increased to 10 and the CrossSubnetThreshold default will automatically be increased to 20." Older Microsoft pages (the heartbeat article, whose examples use Windows Server 2008 R2 node names, and the Windows Server 2008 multi-site heartbeat page) still show thresholds of 5 with a range of 3 to 10. That's why the read above is the step that matters.
Microsoft's own descriptions of each value:
| Property | Microsoft's description |
|---|---|
| SameSubnetDelay | "Controls the delay, in milliseconds, between netft heartbeats. This property applies to routes on the same network subnet." |
| SameSubnetThreshold | "Controls how many heartbeats can be missed on the same subnet before the route is declared as unreachable." |
| CrossSubnetDelay | "Controls the time interval, in milliseconds, that the cluster network driver waits between sending Cluster Service heartbeats across subnets." |
| CrossSubnetThreshold | "Controls how many Cluster Service heartbeats can be missed across subnets before it determines that Cluster Service has stopped responding." |
| RouteHistoryLength | "The number of heartbeats to log after a cluster network failure." |
Then do the math (Microsoft's): "both the delay and threshold have a cumulative effect on the total health detection." Its example: a CrossSubnetDelay of 2 seconds with a CrossSubnetThreshold of 10 "means that the cluster can have a total network tolerance of 20 seconds before recovery action is taken." With the newer same-subnet defaults, the Azure article puts it as "if connectivity monitoring fails for 10 seconds, the failover Threshold is reached resulting in the unreachable that node being removed from cluster membership."
| Finding | What it means (our reading unless noted) |
|---|---|
| Thresholds at the defaults for your version | Normal; the network lost heartbeats for longer than the default window, so find out why (Steps 6 to 8) |
| SameSubnetThreshold 5 on a cluster that should have 10 | An older default on a Windows Server 2012 R2 cluster; Microsoft's tuning article recommends the higher values for those versions |
| Thresholds already high (for example 60 or 120) | Someone already masked it once, and it still fails; the network problem is bigger than the last fix |
| RouteHistoryLength 10 with raised thresholds | Microsoft recommends RouteHistoryLength "to be twice the value" of the threshold, so the cluster log keeps enough history |
| A stretched cluster across sites or subnets | Cross-subnet or cross-site values apply; Microsoft's only real case for raising them is "a multi-site scenario where nodes reside in different locations and network latency can't be overcome." |
| Nodes in Azure IaaS | Microsoft's Azure article recommends relaxed values there; read Step 9 |
Step 6: What does the cluster log say? (read-only)
The System log tells you a node was removed. The cluster log tells you which route went first and what the heartbeat counters looked like. Generate it for a window around the 1135, in local time:
# Read-only: from an empty working folder, a cluster log from every node for the last 120 minutes, local timestamps, copied here
Get-ClusterLog -TimeSpan 120 -UseLocalTime -Destination .
Microsoft's reference for the two switches that matter here: -TimeSpan "Specifies the time span in minutes for which to generate the cluster log." and -UseLocalTime "Specifies that the time stamp for each cluster log entry uses local time. By default, the timestamp uses Greenwich Mean Time (GMT)." If the 1135 was more than a couple of hours ago, raise -TimeSpan to cover it; without -Destination, Microsoft says the log is created "in the cluster reports folder C:\Windows\Cluster\Reports on each node of the cluster." With -Destination ., Microsoft's example "creates a log file for each node of the local cluster, and copies all logs to the local folder."
Then search the logs for the route lines Microsoft's Azure thresholds article shows in its example:
# Read-only: route-down, unreachable, route history, and quorum lines in every node's cluster log
Select-String -Path .\*.log -Pattern 'Marking Route', 'unreachable', 'Route history', 'Lost quorum' |
Select-Object Filename, LineNumber, Line
Microsoft's example log shows the sequence: "Signaled NetftRemoteUnreachable event", then "Marking Route from" one address "to" another "as down", then "All routes for route (virtual) local" and "are down", then "Route history" lines with "Heartbeats counter/threshold: 5/5", and finally, in that example, "Lost quorum". What to read (our reading):
| What you find | What it means |
|---|---|
Marking Route lines for one network only, and the node stayed in |
That network dropped heartbeats; the other one held. Read Step 7 for that network's adapters |
| Routes on every network marked down within the same second | Something shared went: a host, team, virtual switch, upstream switch, or a policy refresh that blocked all traffic (Steps 7 and 8) |
| "Route history" lines with counter/threshold at the limit, then the removal | The threshold was reached; compare with the time of the backup, migration, or policy refresh |
| The remote address in the unreachable line is the same node every time | That node's side (Scenario A) |
| "Lost quorum" | The survivors couldn't keep a majority; read Step 3's quorum branch |
| No route lines at all, but the node's service stopped | Not a heartbeat loss; read that node's System and Application logs |
| Nothing near the time you expected | The cluster log is GMT unless you used -UseLocalTime, or -TimeSpan didn't reach back far enough |
The route history depth is RouteHistoryLength from Step 5. If you plan to raise thresholds, raise that too, or the next log won't show enough of the heartbeats before the drop.
Step 7: What did the NIC, driver, team, and switch see? (read-only)
Microsoft's heartbeat article puts hardware first among the causes: "Actual network hardware failures. If the packet is lost on the wire somewhere between the nodes, then the heartbeats fail." Read the adapters on the node that was removed and on one that wasn't:
# Read-only: adapters with Microsoft's Driver view (driver provider, version, and date)
Get-NetAdapter -Name * | Format-Table -View Driver
# Read-only: all statistics for one adapter, including discards and errors
Get-NetAdapterStatistics -Name 'Cluster1' | Format-List -Property "*"
# Read-only: Microsoft's discarded-packets counter, 12 samples 5 seconds apart, during the busy window
Get-Counter -Counter '\Network Interface(*)\Packets Received Discarded' -SampleInterval 5 -MaxSamples 12
Microsoft's guidance on that counter: "if they are any value higher than zero, then the receive buffer needs to be adjusted up for the adapter." Its heartbeat article adds how to read it: "This counter is cumulative and only increases until the server is rebooted. Seeing a large number of packets dropped here could be a sign that the receive buffers on the network card are set too low or that the server is performing slowly and can't handle the inbound traffic." So take two readings across the backup or migration window and compare them, rather than reading one number once (our reading).
Then read the team and virtual switch, if there is one, since a heartbeat route on a team or a virtual switch depends on both:
# Read-only: LBFO teams and their state
Get-NetLbfoTeam
# Read-only: Hyper-V virtual switches, and Switch Embedded Teaming members if used
Get-VMSwitch
Get-VMSwitchTeam -Name 'SETswitch'
| Finding | What it means (our reading unless noted) |
|---|---|
| Old driver or firmware dates on the removed node, or different versions across nodes | Microsoft: "Old or out-of-date network card drivers and/or firmware. At times, a simple misconfiguration of the network card or switch can also cause loss of heartbeats." |
| Discarded packets climbing during the window | Receive buffers too small or the host too busy; Microsoft says to use the manufacturer's recommended values |
| The heartbeat network is a team, and a member flapped at the time of the 1135 | Microsoft: "If you have NIC teaming enabled on the heartbeat NIC, check Teaming software functionality as per teaming vendor's recommendation." |
| All cluster networks share one team or one virtual switch | One failure takes every route; that matches routes marked down in the same second (Step 6) |
| Guest cluster on VMware with VMXNET3 adapters | Microsoft's VMware article: "VMXNET3 adapters dropping inbound network packets because the inbound buffer is set too low to handle large amounts of traffic." |
| Guest cluster on Hyper-V or VMware with old integration components or tools | Microsoft lists "outdated NIC drivers\Integration Components (IC)\VmTools or faulty NIC adapters" |
| Discards zero, drivers current, but latency high between sites | Not packet loss; heartbeats arriving late. Microsoft: "The packets might not be lost between the nodes, but they might not get to the nodes fast enough before the timeout period expires." |
For the VMware guest case, Microsoft's 1135 article gives the adapter settings it changes, in the guest's Device Manager under the vmxnet3 adapter's Advanced tab: "Select Small Rx Buffers and increase the value. The default value is 512 and the maximum is 8192." and "Select Rx Ring #1 Size and increase the value. The default value is 1024 and the maximum is 4096." Read the current values first:
# Read-only: the vmxnet3 adapter's advanced properties (look for the Rx buffer and ring settings)
Get-NetAdapterAdvancedProperty -Name 'Ethernet0'
Step 8: Did a firewall, IPsec, profile, or antivirus change block it? (read-only)
These are the causes that make a 1135 look random, because nothing is broken afterwards. Read each one for the minute of the event.
Firewall and ports. Microsoft's 1135 article lists what the cluster needs:
| Application | Protocol | Port (Microsoft's table) |
|---|---|---|
| Cluster Service | UDP | 3343 |
| Cluster Service | TCP | 3343 (This port is required during a node join operation.) |
| RPC | TCP | 135 |
| Cluster Admin | UDP | 137 |
| Kerberos | UDP/TCP | 464 |
| SMB | TCP | 445 |
# Read-only: Microsoft's check for the cluster heartbeat rule
netsh advfirewall firewall show rule name="Failover Clusters (UDP-In)" verbose
# Read-only: every rule in the Failover Clusters group, and whether it's enabled
Get-NetFirewallRule -DisplayGroup 'Failover Clusters' | Format-Table -Property DisplayName, Enabled, Direction, Action -AutoSize
# Read-only: the IPv6 Neighbor Discovery rules Microsoft's heartbeat article names
Get-NetFirewallRule -DisplayName 'Core Networking - Neighbor Discovery*' | Format-Table -Property DisplayName, Enabled, Direction -AutoSize
Microsoft ties IPv6 to the heartbeat directly: "IPv6 is the default protocol that Failover Clustering will use for its heartbeats." Its list of causes includes IPv6 enabled with "Core Networking - Neighbor Discovery Advertisement" and "Core Networking - Neighbor Discovery Solicitation" disabled, and "If there are switches, firewalls, or routers not configured properly to allow this traffic through, you can experience issues like this." Event 1572 on a joining node says the same in its own words: "Also verify the Windows Firewall 'Failover Clusters' rules." Microsoft also says the Cluster Network Driver adapter is placed for you: "In Windows Server 2012 and Windows Server 2012 R2, the Cluster Network Driver (NETFT.SYS) adapter is automatically placed at the bottom in the binding order list."
Network profile. Microsoft: "The profile for your network connections could possibly be bouncing from Domain to Public and back to Domain again. During the transition of these changes, network I/O can be blocked." The log is Applications and Services Logs\Microsoft\Windows\NetworkProfile\Operational:
# Read-only: the current profile of each adapter
Get-NetConnectionProfile | Format-Table -Property InterfaceAlias, Name, NetworkCategory -AutoSize
# Read-only: find the exact name of the NetworkProfile Operational log on this node
$np = (Get-WinEvent -ListLog '*NetworkProfile*Operational').LogName
# Read-only: its events around the 1135 (set the window to your event time)
Get-WinEvent -FilterHashtable @{
LogName = $np
StartTime = (Get-Date '2026-10-13 01:00')
EndTime = (Get-Date '2026-10-13 01:15')
} -ErrorAction SilentlyContinue | Select-Object TimeCreated, Id, Message
IPsec policy refresh. Microsoft: "during an IPSec group policy update all IPsec Security Associations (SAs) are torn down by Windows Firewall with Advanced Security (WFAS). While this is happening, all network connectivity is blocked." If renegotiation waits on Active Directory, the block can outlast the threshold. A 1135 that lines up with the Group Policy refresh interval on nodes with connection security rules is this case until proven otherwise (our reading); Event 1592's text names the same thing: "communication temporarily being blocked by a firewall or connection security policy update."
Antivirus. Microsoft: "Anti-virus software could be interfering with this process also." Its 1135 article gives the exclusions it expects, and they're worth reading against your antivirus policy:
| Microsoft's exclusion | Applies to |
|---|---|
| "The path of the FileShare Witness" | Clusters with a file share witness |
| "The %Systemroot%\Cluster folder" | Every node |
| "Default virtual machine configuration directory (C:\ProgramData\Microsoft\Windows\Hyper-V)" | Hyper-V hosts |
| "Default virtual hard disk drive directory (C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks)" | Hyper-V hosts, plus any custom VM, VHD, replica, and snapshot directories |
| Vmwp.exe, as a process exclusion | Hyper-V hosts |
| "exclude the CSV path C:\Clusterstorage and all its subdirectories" | Clusters with Cluster Shared Volumes and live migration |
Microsoft's caution about testing it: "Just disabling the antivirus software is insufficient in most cases. Even if you disable the antivirus software, the filter driver is still loaded when you restart the computer."
Validation. Microsoft's 1135 article puts this early: "A validation and the network tests would be recommended as one of the initial troubleshooting steps to ensure there are no configuration issues that might be a cause for problems." Run only the network tests on a live cluster, and list the test names on your version first:
# Read-only: list the validation tests and categories on this version
Test-Cluster -List
# Read-only for networking: run the network communication and firewall tests only, with a named report
Test-Cluster -Include 'Validate Network Communication', 'Validate Windows Firewall Configuration' -ReportName C:\Temp\cluster-network-validation
Microsoft's descriptions of those two tests: "This test validates that tested servers can communicate with acceptable latency on all networks. The test also checks to see if there are redundant communication paths between all servers." and "This test validates that Windows Firewall is configured correctly for failover clustering on the tested servers." In the report, Microsoft's example latency failure reads "the ping latency is greater than the maximum allowed 2000 ms". Keep storage tests out of a run on a live cluster: Microsoft's -Disk and -Force notes describe disks taken offline for the duration of the storage tests.
Step 9: Change one link (a change)
Fix only what Steps 2 to 8 showed, then read the result before changing anything else. If the change touches a node, drain it first:
# A change: move this node's roles off gracefully before you touch its NICs, drivers, or switch ports
Suspend-ClusterNode -Name HV02 -Drain -Wait
# After the change and a read of Steps 4 and 7 on it: bring it back, with its roles moved back now
Resume-ClusterNode -Name HV02 -Failback Immediate
Microsoft's -Drain: "Specifies that all of the workloads are gracefully moved to other nodes while maintaining the highest levels of availability and using the best placement logic." -Failback takes NoFailBack, Immediate, or Policy.
Drivers, firmware, and buffers (Step 7). Update the NIC driver and firmware to the versions your server and NIC vendors recommend, on one drained node at a time, and raise receive buffers to the manufacturer's recommended values. For a VMware guest cluster, Microsoft's direction is to "gradually increase the number of buffers in the guest operating system and verify." Re-read the discard counter under the same load afterwards.
Network roles and metrics (Step 4). Give the cluster a second path for heartbeats, and keep storage traffic off cluster communication. Microsoft's Role values: 1 for internal cluster communication, 3 for cluster and client, 0 for not used by the cluster. Microsoft's restriction: "If this network is the only one used for internal cluster communication, the Role property may not be changed to ClusterNetworkRoleNone or ClusterNetworkRoleClientAccess."
# A change: let a second network carry cluster communication (Role 1); use your own network name
(Get-ClusterNetwork -Name 'Cluster Network 2').Role = 1
# A change, Microsoft's form: rank a network by metric (lower is preferred for cluster traffic)
$n = Get-ClusterNetwork 'Cluster Network 3'
$n.Metric = 1050
Microsoft's network prioritization article shows how to undo a manual metric: set AutoMetric back to $true on that network.
Firewall, IPv6, profile, IPsec (Step 8). Re-enable the Failover Clusters rules and the Neighbor Discovery rules in the policy that disabled them, not by hand on one node (our reading). Fix the reason a profile drops to Public (usually domain detection at boot or a DNS path; our reading). For IPsec, Microsoft's article points at the refresh and the authentication delay; exempt cluster traffic in the connection security rules or fix the DC path the renegotiation waits on (our reading).
Antivirus (Step 8). Add Microsoft's exclusions from the table above in the antivirus policy for every node, then reboot one drained node at a time so filter drivers load with the new policy.
Quarantine (Step 3). Once the cause is fixed, clear the quarantine with Microsoft's command instead of waiting out the two hours:
# A change: start the Cluster service on the node and clear its quarantine
Start-ClusterNode -Name HV02 -ClearQuarantine
Thresholds, last and in moderation (Step 5). Raise them only for the case Microsoft describes (stretched clusters, Azure IaaS, or older versions still on 5), and only after the network read. Microsoft: "In general, continuing to send frequent heartbeats but having greater thresholds is the preferred method." Its forms:
# A change, Microsoft's form: raise the same-subnet threshold (example value from Microsoft's article)
(Get-Cluster).SameSubnetThreshold = 20
# A change, Microsoft's form: keep twice as much route history as the threshold
(Get-Cluster).RouteHistoryLength = 40
# Re-read
Get-Cluster | Format-List -Property SameSubnet*, CrossSubnet*, RouteHistoryLength
Microsoft's tuning article: "These setting can be set while the cluster is up and running with no downtime and will take effect immediately with no need to reboot or restart the cluster." Its limit: "it is generally not recommended to exceed the TCP reconnect timeouts. Generally, this means not going above ~20 seconds." So a 1000 ms delay with a threshold of 20 is about the ceiling for most workloads, and its warning still applies: "increasing resiliency to network hiccups comes at the cost of increased downtime when a hard failure occurs." (The RouteHistoryLength value of 40 is the reference maximum; 40 is twice 20.)
Step 10: Prove it stays quiet (read-only, through the same load window)
Wait through the window that produced the 1135 (the backup, the migration wave, the patch night, the Group Policy refresh), then run the reads again:
# Read-only: no new 1135, 1127, 1129, 1130, 1177, or 1672 on any node since the change
$since = Get-Date '2026-10-20 18:00'
foreach ($n in (Get-ClusterNode).Name) {
Get-WinEvent -ComputerName $n -FilterHashtable @{ LogName = 'System'; Id = 1127, 1129, 1130, 1135, 1177, 1672; StartTime = $since } -ErrorAction SilentlyContinue |
Select-Object @{n='LoggedOn';e={$n}}, TimeCreated, Id
}
# Read-only: nodes, networks, and interfaces all Up
Get-ClusterNode
Get-ClusterNetwork | Format-Table -Property Name, State, Role, Metric -AutoSize
Get-ClusterNetworkInterface | Sort-Object Network, Node | Format-Table -AutoSize
# Read-only: the cluster log for the window, with no route marked down
Get-ClusterLog -TimeSpan 120 -UseLocalTime -Destination .
Select-String -Path .\*.log -Pattern 'Marking Route' | Select-Object Filename, LineNumber, Line
# Read-only: discards across the same window, compared with the reading before the change
Get-Counter -Counter '\Network Interface(*)\Packets Received Discarded' -SampleInterval 5 -MaxSamples 12
# Read-only: the network validation tests again
Test-Cluster -Include 'Validate Network Communication', 'Validate Windows Firewall Configuration' -ReportName C:\Temp\cluster-network-validation-after
Pass criteria (our reading): no 1135 on any node through at least one full cycle of the window that caused it (a week, for a weekly backup); no route marked down in the cluster log for that window; every node, network, and interface Up; discarded packets flat across the busy window; at least two networks allowed for cluster communication; the network validation tests pass without latency or firewall warnings; no node quarantined; and heartbeat values at the defaults for your version, or raised in moderation with RouteHistoryLength at twice the threshold and a written reason.
Common false starts
| False start | Why it doesn't help (our reading unless noted) |
|---|---|
| Raising SameSubnetThreshold and CrossSubnetThreshold to the maximum | Microsoft: "it just masks the problem. It doesn't fix anything." Hard failures then take minutes to detect |
| Raising the delay instead of the threshold | Microsoft: "continuing to send frequent heartbeats but having greater thresholds is the preferred method." |
| Reading the 1135 on one node only | Microsoft's scenarios A, B, and C are told apart only by comparing all nodes |
| Comparing the System log with a cluster log in GMT |
Get-ClusterLog timestamps are GMT unless you use -UseLocalTime, so the route lines appear hours away |
| Evicting and re-adding the node | The NIC, driver, switch, or policy that dropped the heartbeats comes back with it |
| Disabling antivirus to test | Microsoft: "Just disabling the antivirus software is insufficient in most cases." The filter driver stays loaded |
| Disabling IPv6 on the cluster networks | Microsoft says IPv6 is the default heartbeat protocol; its cause is the Neighbor Discovery rules being disabled, not IPv6 itself |
Running a full Test-Cluster on a live cluster |
Storage tests can take disks offline; run the network tests by name |
| Clearing quarantine before the cause is fixed | The node leaves again; three times in an hour quarantines it again |
| Using one team or one virtual switch for every cluster network | Every route then fails together, and a node with two networks still gets removed |
| Treating 1069 or 1146 as the cause | They're resource and health events that follow the removal; read the 1135 first |
When a change is justified
| Finding | Prefer | Avoid |
|---|---|---|
| Discards climbing on one node's adapter | Driver, firmware, and receive buffers to vendor-recommended values, one drained node at a time | Raising thresholds so the drops stop mattering |
| VMware guest cluster with VMXNET3 drops | Microsoft's Small Rx Buffers and Rx Ring #1 Size changes, increased gradually | Moving the guest cluster to a single vNIC |
| Only one network carries cluster communication | A second network with Role 1 or 3, or a team on redundant switches | Setting the iSCSI network to carry cluster traffic to get a second path |
| Live migration or backup traffic on the heartbeat network | A dedicated live migration network and metrics that keep cluster traffic on its own path | Turning off live migration or backups |
| Neighbor Discovery or Failover Clusters rules disabled by policy | Re-enable them in the policy that disabled them | A local rule on one node that the next policy refresh removes |
| Profile dropping to Public | Fix domain detection on that adapter | Disabling the firewall on cluster networks |
| IPsec refresh blocking traffic | Exempt cluster traffic in the connection security rules, or fix the DC path | Lengthening the threshold to outlast every refresh |
| Stretched cluster or Azure IaaS with real latency | Cross-subnet or cross-site thresholds raised in moderation, with RouteHistoryLength at twice the threshold | Same-subnet values raised to 120 for a site link problem |
| A quarantined node, cause fixed | Start-ClusterNode -ClearQuarantine |
Waiting out two hours, or evicting the node |
FAQ
What does Event 1135 "Cluster node was removed from the active failover cluster membership" mean?
The other nodes stopped receiving the node's heartbeats on every cluster network for longer than the threshold allows, or its Cluster service stopped. Microsoft's text: "The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster." It's logged on the nodes that remain, not on the removed one (Steps 1 and 2).
What port does the failover cluster heartbeat use?
UDP 3343. Microsoft: "The heartbeat itself is a UDP unicast network packet that communicates over Port 3343." TCP 3343 is used when a node joins, and IPv6 is the default heartbeat protocol, so the Neighbor Discovery rules matter too (Step 8).
What are the default SameSubnetDelay and SameSubnetThreshold values?
It depends on the version and the page you read. Microsoft's property reference lists SameSubnetDelay 1000 ms and SameSubnetThreshold 5; its tuning article lists 10 heartbeats for Windows Server 2016, and its Azure article says Windows Server sets SameSubnetThreshold to 10 by default. Read your own with Get-Cluster | Format-List *subnet* (Step 5).
Should I increase the cluster heartbeat thresholds to stop Event 1135?
Only after the network read, and in moderation. Microsoft says raising them "simply masks the problem by making health monitoring less sensitive", and that it's generally "not going above ~20 seconds". The real cases are stretched clusters, Azure IaaS, and older versions still on 5 (Steps 5 and 9).
Why is my cluster node quarantined, and how do I clear it?
On Windows Server 2016 and later, "a node is quarantined, if it ungracefully leaves the cluster, three times within an hour", for 7200 seconds by default. Fix the cause of the 1135s, then run Start-ClusterNode -ClearQuarantine, the command Microsoft's quarantine event names (Steps 3 and 9).
Related posts
- Hyper-V Live Migration Failed (Event 21502) (when the 1135s line up with live migrations on the same hosts and networks)
- Hyper-V Checkpoint Operation Failed (when the 1135s line up with the backup window and checkpoints fail too)
- Active Directory Replication Error 1722 (when the same nodes also fail RPC to domain controllers through the same firewall or NIC)
Free checklist
Raising heartbeat thresholds on every cluster, or evicting the node, before anyone read which route dropped and why, is a change made everywhere at once. So is a Conditional Access policy enforced before anyone read the report-only results. The free Conditional Access pre-enforcement checklist keeps break-glass accounts, pilot groups, and report-only review on one page:
If you want the rest of the Intune runway written down for the team (enrollment hygiene, compliance phasing, break/fix cards and read-only PowerShell snapshots), the Intune & M365 Admin Starter Pack pairs with this triage: https://cashflow4375.gumroad.com/l/joonf?utm_source=devto&utm_medium=article&utm_campaign=aps_first_sale&utm_content=failover-cluster-event-1135
Admin Pack Studio; not affiliated with Microsoft. Steps 1-8 and 10 are read-only checks: Get-WinEvent, Get-ClusterNode, Get-ClusterResource, Get-ClusterQuorum, Get-Cluster, Get-ClusterNetwork, Get-ClusterNetworkInterface, Get-ClusterLog, Select-String, Get-NetAdapter, Get-NetAdapterStatistics, Get-NetAdapterAdvancedProperty, Get-Counter, Get-NetLbfoTeam, Get-VMSwitch, Get-VMSwitchTeam, Get-NetFirewallRule, netsh advfirewall firewall show rule, Get-NetConnectionProfile, Get-Service through Invoke-Command, and Test-Cluster with only the network tests (it writes a report file). Get-ClusterLog writes log files to the destination folder. Draining and resuming nodes, updating drivers and firmware, changing adapter buffers, network roles, and metrics, changing firewall, IPsec, and antivirus policy, clearing quarantine, and changing heartbeat thresholds or route history are deliberate changes. Change one link at a time, prove it through the same load window, and check current Microsoft Learn for version-specific behavior.
Top comments (0)