The Windows workload client lost its TrueNAS shares one Thursday morning. Red-X on every drive mapped to that specific server. Mappings to other storage servers on the same VLAN stayed online. Test-NetConnection on port 445 returned TcpTestSucceeded : False. Ping timed out.
I cleared the client's ARP cache. The shares came back. Then Windows Update rebooted the client, and the shares went down again.
That second failure is what made me look properly, because the first failure had a satisfying workaround and the second one broke that workaround in a way that was going to keep breaking it. If I'd stopped at arp -d, I'd have carried a time bomb forward — Windows Update reboots are routine, and every reboot would have been a 50/50 bet on whether the shares came back.
This piece is the trace that turned a storage symptom into a network root cause: client to Nexus to endpoint, with each layer eliminating a correct-looking suspect, ending in a configuration error that had been in place since the fabric migration. The server I thought was failing wasn't failing. Nothing was failing. Two devices were simply configured with the same IP address, and ARP — behaving exactly as designed — had been quietly handing my client the wrong one.
// why storage was the first suspect
The affected TrueNAS server had been through genuine storage trouble recently. The MMX/TrueNAS migration piece documented five layers of failure during that same ReRACK window, including a hung SAS/CAM path on this same server that required a full JBOD power cycle to clear. When storage had just bitten me, storage was the natural place to look.
The symptoms were plausibly storage: SMB mappings timing out, Red-X on the shares, retries not helping. If this had been a repeat of the earlier SAS failure, I'd have expected pool-level trouble visible in the TrueNAS GUI, SMB service errors in the system logs, or at minimum a kernel message about a hung path. None of that was there. Pools online. SMB service healthy. Other storage servers on the same VLAN reachable from the same client, serving the same workload, with no complaints.
That mismatch — storage-looking symptom with no storage-layer evidence — was the first thing that didn't fit. Storage went to the back of the suspect list within about ten minutes. The symptom was Layer 7, but the fault was somewhere lower.
// the Windows ARP red herring
The next suspect was the client itself. SMB has a long history of edge cases on Windows that can be fixed by clearing some piece of cached state and letting the stack rebuild. The ARP cache is one of them. If the client has learned a stale MAC address for the server, nothing above Layer 2 is going to work until that mapping refreshes.
I ran:
arp -d 172.16.x.22
ping 172.16.x.22
arp -a | findstr 172.16.x.22
(Addresses are illustrative throughout this piece — I'll use the 172.16.x.NN pattern to abstract real addressing.)
The ping succeeded. The ARP cache populated with a MAC. The SMB mappings came back online. Everything felt resolved.
For a reader who's spent any time troubleshooting SMB on Windows, this is the moment that feels like victory and is often a trap. arp -d restoring service is a specific signal — it says the client was holding the wrong MAC for the IP it was trying to reach. It does not say why the client was holding the wrong MAC. There are at least three plausible reasons, and only one of them is a Windows bug.
I didn't think about which reason it was. I called the shares back up, verified the mappings, and moved on. For about two hours it felt like I'd fixed something.
Then the client took a Windows Update reboot. The ARP cache cleared as a side effect of the restart. Windows did a fresh ARP lookup on the TrueNAS address. The SMB mappings came up Red-X again.
That recurrence is what forced the real investigation. A workaround that works once can be a workaround. A workaround that works and then breaks on the next reboot is pointing at something underneath that nobody's looked at yet.
// tracing from client to the fabric
The question I needed to answer first was: when the client sends traffic to the TrueNAS address, where does the fabric think that traffic should go?
The TrueNAS server runs as a Proxmox VM, which attaches to the Cisco Nexus 3548 top-of-rack pair via a vPC-bonded port-channel on the hypervisor. The TrueNAS MAC should appear in both TORs' MAC address tables, learned on that port-channel.
I checked:
TOR-A# show mac address-table address MAC-A
+ 10 MAC-A dynamic 0 ... Po200
TOR-B# show mac address-table address MAC-A
+ 10 MAC-A dynamic 0 ... Po200
(MAC addresses abstracted as MAC-A, MAC-B, MAC-C throughout.)
Both TORs had the TrueNAS MAC on Po200, which is the LACP bundle to the correct Proxmox host. The vPC was consistent, the trunk allowed the workload VLAN, and port type was edge trunk as it should be. From the switch's perspective the TrueNAS server was exactly where it was supposed to be.
The fabric wasn't the problem. The fabric was doing the thing fabrics do: forwarding traffic to the MAC that owns the destination IP, as that MAC resolves through ARP, as the ARP table currently believes.
Which meant the real question was who owns the destination IP in the ARP table, and the answer to that was about to be the whole story.
// the ARP table discrepancy that unlocked everything
I checked the gateway ARP table on the Nexus for the TrueNAS address:
TOR-A# show ip arp vlan 10 | include 172.16.x.22
172.16.x.22 00:00:44 MAC-B Vlan10
The MAC in that line was not MAC-A. It was a MAC I didn't recognise — MAC-B. For an address I thought belonged to TrueNAS, the gateway's ARP table was returning the MAC of some other device.
This is the moment the investigation shifts. Up until this point I'd been looking for things that were broken — a storage pool, an SMB service, a Windows cache. Nothing was broken. Everything was working. What was happening was that the fabric was routing .22 traffic to a MAC that was not the server I thought owned .22, and it was doing that because that MAC had answered the gateway's most recent ARP request for the address.
I traced the unfamiliar MAC through the CAM table:
TOR-A# show mac address-table address MAC-B
* 10 MAC-B dynamic 0 ... Eth1/15
Eth1/15 on TOR-A. The port profile named it — a different workload node on the same VLAN. Not Proxmox, not TrueNAS. A separate storage/compute node attached directly to one of the TORs.
For completeness I checked the client's path too — its MAC resolved on TOR-B Eth1/21, the port profile matching the expected workload-client access port. Clean. The client was correctly learned, in the correct port, in the expected VLAN. No fabric-side problem on the client's side either.
The whole fabric was behaving as designed. What wasn't right was higher up: two different MAC addresses were able to answer ARP for the same IP, and the fabric had no way to know which one was supposed to.
// the endpoint reveal
The authoritative check for IP ownership is the host itself. The switch can tell you where a MAC lives. Only the endpoint can tell you what address it thinks it has.
I logged into TrueNAS directly and ran ip addr show:
3: enp6s19: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
link/ether MAC-A
inet 172.16.x.22/24 brd 172.16.x.255 scope global enp6s19
TrueNAS was configured as .22. The other workload node was configured as .22. Both devices had legitimately claimed the same address. Both were valid Layer-2 participants on the workload VLAN. Both would answer an ARP request for .22 with their own MAC.
The intended address for TrueNAS during the ReRACK was .40. I had the record of the planned allocation; .22 belonged to the other node and had belonged to it for months before the fabric migration. What had happened was that during the TrueNAS reconfiguration for the new fabric, the address .22 had gone onto the interface by mistake — one digit off, or a copy from a different line of a planning spreadsheet, or anything in that family of five-second configuration slips. Nobody caught it at the time, because .22 on the TrueNAS interface produced a device that answered ARP and accepted TCP connections. From any single-request check, it looked fine.
It only became a problem when both devices were on the VLAN at the same time and the client needed to decide which one owned .22. The client couldn't decide. The client just believed whatever ARP reply arrived first.
// what the duplicate IP actually did
ARP is one of those protocols where violating the design assumptions can stay quiet for a long time. Nothing I was watching logged a complaint — no pool errors, no SMB service faults, nothing in the fabric. NX-OS does have duplicate-address detection, and it never surfaced this one; with the second node mostly idle on that address, the two never ARPed close enough together to trip it. The vPC pair stayed consistent, HSRP didn't flap, MAC flapping didn't trigger because the two MACs were on different ports and neither of them was moving.
What the duplicate IP did was turn ARP resolution into a race. When any device on the VLAN sent an ARP request for .22, both TrueNAS and the other workload node answered. Whichever answer arrived first became the entry in the requester's ARP cache. The entry held until it timed out or the cache got cleared, at which point another race ran.
For the client, this looked like intermittent connectivity. Half the time the cache held TrueNAS's MAC and SMB worked. Half the time it held the other node's MAC, traffic went to a host that had no SMB listener on that address, and the shares came up Red-X. The 50/50 wasn't literal — timing depends on where the devices sit relative to the client, which replies win in which order — but the practical effect was that the client sometimes worked and sometimes didn't, with no deterministic rule for which.
arp -d restored service because it forced a new race. If the race landed on TrueNAS, the shares came back. If it landed on the other node, nothing changed. For my situation, the first few arp -d calls happened to land on TrueNAS, which is why the workaround felt like a real fix. It wasn't fixing anything. It was rerolling the dice, and I happened to roll well for a few hours.
The Windows Update reboot was what exposed this. A reboot clears the entire neighbour cache, which forces a fresh resolution. The post-reboot resolution landed on the other node this time, and the shares were Red-X again. That recurrence was the lucky break — if the reboot had landed on TrueNAS, I'd have gone another day or two before the next failure, and the pattern would have been even harder to see.
Clearing ARP wasn't a workaround for a Windows bug. Clearing ARP was the trigger I was using to reroll an unstable system state, without understanding that the state was unstable.
// the fix
With the duplicate IP identified, the fix was configuration-level and non-destructive. Both devices were legitimately part of the VLAN. The other workload node had owned .22 for months and was the historical owner. TrueNAS was the one with the mis-assigned address.
I verified that .40 was unused on the VLAN, by checking the gateway's ARP table for the address and getting no entry. Then changed the TrueNAS interface configuration in the GUI from .22/24 to .40/24, leaving everything else — the MAC, the subnet mask, the gateway, the DNS settings — unchanged.
On the client side, the persistent drive mappings had to point at the new address. The net use entries for the two affected shares came off and went back on with the new target:
net use G: /delete
net use H: /delete
net use G: \\172.16.x.40\JBOD1 /persistent:yes
net use H: \\172.16.x.40\JBOD2 /persistent:yes
Both drives came up on first attempt. SMB access was clean, reproducible, and stable across the next series of planned client reboots. The duplicate-IP condition was gone because only one device now claimed .22, and that device was the one that had always been the legitimate owner.
The whole corrective action took about ten minutes once the root cause was identified. The investigation that identified it took the better part of a morning.
// what I'd make instinctive
Four things from this.
When clearing ARP "fixes" a problem, test for duplicate IP before treating the client as buggy. arp -d is a legitimate diagnostic tool, but a successful ARP clear doesn't prove the client's ARP implementation is broken. It proves the client was holding one of several possible mappings, and the one it was holding wasn't the right one. There are three plausible reasons for that: the client had learned a stale MAC that no longer owned the IP (an actual client-side cache problem), the switch had learned stale CAM entries for the MAC (a fabric-side cache problem), or two devices were legitimately answering for the same IP (a configuration problem). The first two are rare on modern hardware with healthy aging timers. The third is more common than operators like to admit, and it's the one that arp -d can't fix because the next ARP exchange has the same race condition.
ARP tables answer IP → MAC. CAM tables answer MAC → port. Use both directions. The decisive moment in this investigation was comparing what the gateway ARP table thought owned the IP against what the CAM table said about where that MAC lived. The ARP table alone would have told me the resolution was wrong but not where the wrong device was. The CAM table alone would have told me a MAC was on a port but not that it was answering for an IP it shouldn't own. Both tables, read in the same session, pointed at the specific switch port on the specific TOR that was serving the wrong endpoint. That's not something you can see from either table in isolation.
The switch tells you where a MAC lives. Only the endpoint tells you what IP it thinks it owns. This is the kind of lesson that sounds obvious when written down and gets forgotten under time pressure. During the investigation I was tempted to stop at "wrong MAC answering for the IP, that's the problem" and go hunt down the owner of that MAC in the inventory. The step that actually resolved the question was logging into TrueNAS and running ip addr show — that's what proved the mis-assignment at the authoritative source. Any investigation that ends at the switch is incomplete; the switch is reporting observations, not declarations of ownership.
Static ARP is the wrong permanent answer to duplicate IP. It's tempting, once a duplicate IP is identified, to pin the correct mapping on the affected client with a static ARP entry. That hides the symptom for one client. It leaves the duplicate IP in place for every other device on the VLAN — the gateway, the DHCP server, the monitoring stack, any other client that happens to need the address. The correct action is to fix the IP ownership at the source, which is a configuration change on the mis-assigned device. Static ARP as a durable fix means you've accepted that your VLAN has a duplicate IP and chosen to paper over it on the one host that noticed.
None of these are theoretical. All four came out of this specific morning's investigation, and all four are things I'd apply next time I see a symptom that arp -d temporarily resolves.
// closing
The Article 10 rack migration piece landed on the observation that a plan can be right and still be incomplete — correctness and completeness are different axes. This piece is a smaller version of the same shape at the IP layer: a reconfiguration had been done, had been verified to the extent that single-point checks could verify it, and had been complete enough that TrueNAS came up on the VLAN and started answering queries. What the single-point check couldn't see was whether anyone else on the VLAN was also answering for the same address. That verification needs the full ARP table, not just a ping to confirm the host is reachable.
If you have anything on your network that "sometimes needs an ARP clear to come back," you probably don't have a Windows ARP bug. You have a duplicate IP condition that hasn't been investigated yet, and the ARP clear is masking it by rerolling the race. Investigate the next time the symptom appears, not the next time it's convenient — the next client reboot is going to reroll the dice regardless, and sooner or later it's going to land in a way that costs more than one morning.
Top comments (0)