DEV Community

Cover image for Debugging LACP Instability in a Transparent OPNsense Bridge
Andre Faria
Andre Faria

Posted on Edited on

Debugging LACP Instability in a Transparent OPNsense Bridge

#opnsense #networking #homelab

I run a transparent OPNsense bridge between a UniFi Dream Machine Pro and the rest of my LAN. It is deliberately boring at Layer 3. The UDM keeps routing, DHCP, DNS, firewall policy, WAN handling, and VLAN definitions. OPNsense sits inline as a Layer 2 bump in the wire.

The interesting part is that both sides of that bump use LACP.

I already wrote the build and configuration guide for this setup here. Building a Transparent LAGG (LACP) Bridge with OPNsense, UDM, and UniFi - A Practical Guide. That article explains how the bridge was built, how the LAGG devices were configured, and why I wanted the firewall to remain transparent.

This article is the other half of the story. What happens when that kind of setup fails in a non-obvious way. Not a clean outage. Not a single network down moment. Just enough instability to make everything feel wrong.

1. Topology and Failure Surface

The topology looked like this:

                          +----------------------+
                          | UniFi Dream Machine  |
                          | kantharos-udm-pro    |
                          +----------+-----------+
                                     |
                         LACP aggregate, 2 x 1G
                                     |
                            OPNsense lagg0
                            "ingresslagg"
                          igc1 + igc2, LACP
                                     |
                          +----------v-----------+
                          | OPNsense bridge0     |
                          | "laggbridge"         |
                          +----------+-----------+
                                     |
                            OPNsense lagg1
                            "egresslagg"
                          igc4 + igc5, LACP
                                     |
                         LACP aggregate, 2 x 1G
                                     |
                          +----------v-----------+
                          | UniFi USW-Lite-16    |
                          | downstream LAN       |
                          +----------------------+
Enter fullscreen mode Exit fullscreen mode

On OPNsense, the relevant interfaces were:

igc1 + igc2 -> lagg0 -> ingresslagg -> toward UDM
igc4 + igc5 -> lagg1 -> egresslagg  -> toward USW
lagg0 + lagg1 -> bridge0 -> laggbridge
Enter fullscreen mode Exit fullscreen mode

The bridge is a FreeBSD bridge. The aggregates are FreeBSD lagg(4) interfaces using LACP. OPNsense exposes those through its Interfaces > Devices UI.

The expected healthy OPNsense state is:

laggproto lacp
status: active
laggport: igcX flags=<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igcY flags=<ACTIVE,COLLECTING,DISTRIBUTING>
Enter fullscreen mode Exit fullscreen mode

Those three member states matter:

  • ACTIVE means the member is participating in the LACP bundle.
  • COLLECTING means the member may receive traffic.
  • DISTRIBUTING means the member may transmit traffic.

For an LACP link, carrier alone is not enough. A cable can show link, but if the member is not collecting and distributing, it is not a healthy participant in the aggregate.

In a transparent bridge, that distinction matters more than usual. OPNsense is not routing around the problem. It is forwarding Ethernet frames between two aggregated links, much like the OPNsense bridge documentation describes for Layer 2 forwarding and MAC learning. If one LACP member misbehaves, the symptoms can leak across the whole Layer 2 segment.

2. Symptoms and the LACP Trap

The failure did not present as a clean interruption. There was no single point where the whole LAN died and stayed dead. Instead, traffic slowed down, clients behaved inconsistently, management sessions became flaky, UniFi and OPNsense disagreed about state, and the bridge looked partially alive and partially broken.

With a single Ethernet cable, a physical failure is usually obvious. The link drops. The port goes down. The device disappears.

With LACP, a single member can become marginal while the logical aggregate still exists. Some traffic survives. Some traffic lands on the bad member. Some flows stall, some retry, and some keep working. The user-facing symptom becomes "the network is weird", which is among the least useful sentences in infrastructure.

The reason is hashing. LACP does not normally split one flow across all cables like a striped disk. The FreeBSD handbook notes that Ethernet frame ordering means traffic between two stations stays on the same physical link, while the transmit algorithm tries to balance flows across the aggregate. In my OPNsense setup, the LAGG hash was Layer 2:

laggproto lacp lagghash l2
Enter fullscreen mode Exit fullscreen mode

A simplified model:

flow A -> member 1 -> works
flow B -> member 2 -> stalls
flow C -> member 1 -> works
flow D -> member 2 -> retries
Enter fullscreen mode Exit fullscreen mode

That creates a failure mode which feels like congestion, DNS trouble, Wi-Fi trouble, controller weirdness, or firewall slowness. The central trap is simple. Partial LACP failure can masquerade as general network degradation.

3. OPNsense Evidence That the Bundle Was Actually Flapping

The strongest evidence came from OPNsense logs in the system log files (/var/log/system/system_20260605.log). Two windows mattered:

2026-06-05 02:26:32-02:28:01 UTC
2026-06-05 20:08:27-21:22:31 UTC
Enter fullscreen mode Exit fullscreen mode

During the earlier window, OPNsense saw:

igc1 and igc2 went down/up repeatedly
lagg0: link state changed to DOWN
lagg0: link state changed to UP
igc4/igc5: Interface stopped DISTRIBUTING, possible flapping
Enter fullscreen mode Exit fullscreen mode

During the major evening window:

20:08:27  lagg1 went DOWN
20:10:19  lagg1 came UP
20:19:12  lagg1 went DOWN again
20:24-20:41 igc4/igc5 continued bouncing
20:26:47  lagg0 dropped
20:34:36  lagg0 came back
21:05:10  lagg1 dropped again
21:05:44  lagg1 came back
21:22:28  lagg0 detached during final bypass/reset activity
21:22:31  lagg1 detached during final bypass/reset activity
Enter fullscreen mode Exit fullscreen mode

The most useful phrase was:

Interface stopped DISTRIBUTING, possible flapping
Enter fullscreen mode Exit fullscreen mode

That is not an application-layer symptom. It is not DNS. It is not an IP routing issue. It is not a firewall rule. It means the LACP member state changed at the link aggregation layer. A simplified LACP health path looks like this:

Physical carrier up
  v
LACP peer detected
  v
Correct partner/system/key information
  v
Member selected into aggregator
  v
Member allowed to collect and distribute traffic
Enter fullscreen mode Exit fullscreen mode

If a member stops distributing, the aggregate may still exist, but it is no longer healthy. The device has decided that member should not transmit traffic as a valid part of the bundle. The current healthy state after reconnecting the bridge looked like this:

lagg0:
  laggproto lacp lagghash l2
  laggport: igc1 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: igc2 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
  status: active

lagg1:
  laggproto lacp lagghash l2
  laggport: igc4 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: igc5 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
  status: active
Enter fullscreen mode Exit fullscreen mode

And the bridge itself:

bridge0:
  member: lagg1
    role root
    state forwarding

  member: lagg0
    role designated
    state forwarding
Enter fullscreen mode Exit fullscreen mode

That contrast matters. During the incident, OPNsense saw real LAGG instability. After remediation, it saw active LACP members and a forwarding bridge. This matches the healthy FreeBSD example where ifconfig lagg0 reports status: active and member ports with ACTIVE,COLLECTING,DISTRIBUTING flags in the FreeBSD link aggregation documentation.

4. UniFi Evidence and the UDM Trap

The UniFi side complicated the investigation because the UDM Pro did not expose this like a normal Linux LACP bond. UniFi's Port Aggregation FAQ says static LAG is not supported and aggregation uses LACP, while also calling out that gateway support is limited to specific models including the UDM Pro.

Over SSH, the UDM looked alarming:

eth6@switch0 UP
eth7@switch0 UP
lacp6 LOWER_UP
lacp7 LOWER_UP
lag0 DOWN / NO-CARRIER
Enter fullscreen mode Exit fullscreen mode

And /proc/net/bonding/lag0 showed this:

Ethernet Channel Bonding Driver: v3.7.1
Bonding Mode: load balancing (round-robin)
MII Status: down
Enter fullscreen mode Exit fullscreen mode

For a normal Linux bonding LACP bond, this would be terrible. I would expect IEEE 802.3ad, slave interfaces, MII up, aggregator details, and partner MAC information. That is not what the UDM showed.

The UniFi controller told a more coherent story. On the UDM, port 7 was the aggregate parent and port 8 was masked as a member. On the USW-Lite-16-PoE, ports 7 and 8 were both active LACP members, the aggregate was forwarding, and partner_system_id matched the OPNsense lagg1 MAC:

e4:3a:6e:5d:a0:00
Enter fullscreen mode Exit fullscreen mode

The UDM also had lagd creating LACP interface mappings for lacp6 and lacp7, while the interface details showed Realtek switch abstractions like eth6@switch0, eth7@switch0, and rtk_sw_netdev devices.

That was the debugging lesson. On appliance hardware, not every OS-level network interface is equally authoritative. The better sources of truth were UniFi controller aggregate state, USW lacp_state, OPNsense ACTIVE,COLLECTING,DISTRIBUTING, STP forwarding state, packet counters, and successful pings through the bridge. In this incident, the UDM lag0 DOWN output was suspicious, but not decisive.

5. Root-Cause Analysis

The most useful UniFi historical lines came from the UDM lagd logs:

lag0: eth7: carrier state is DOWN dropping received LACP PDU.
lag0: Failed to send PDU from eth6: Failed to write LACP data: Network is down (os error 100)
lag0: Failed to send PDU from eth7: Failed to write LACP data: Network is down (os error 100)
Enter fullscreen mode Exit fullscreen mode

This is where the investigation stopped being abstract. LACP depends on LACPDUs exchanged between the actor and partner. If a device cannot send LACP PDUs because the interface is down, or if it drops received LACP PDUs because carrier is down, the aggregate cannot stay stable.

That is different from the two devices disagree about configuration. It is closer to the link is physically unstable enough that LACP control traffic cannot reliably move.

The USW counters supported the same direction. The aggregate ports had the worst link-down history:

USW Port 7:
  link_down_count: 26
  tx_errors: 5
  tx_dropped: 5
  lag_member: true
  lacp_state: active

USW Port 8:
  link_down_count: 8
  lag_member: true
Enter fullscreen mode Exit fullscreen mode

Several ordinary ports had much lower link-down counts. Counters alone do not prove causality, but combined with OPNsense LACP distribution failures and UniFi carrier/PDU errors, they became strong supporting evidence.

There was also a reset and recovery window on the USW. That lined up with final OPNsense LAG detach events around 21:22:28 and 21:22:31. The distinction matters. Link events created by a deliberate reset are not the same kind of evidence as repeated LACP distribution failures before the reset.

After replacing the OPNsense-to-USW cable pair and restoring the bridge, the state became boring again:

igc1   up  1000baseT full-duplex
igc2   up  1000baseT full-duplex
igc4   up  1000baseT full-duplex
igc5   up  1000baseT full-duplex
lagg0  up
lagg1  up
bridge0 up
Enter fullscreen mode Exit fullscreen mode

The diagnosis was not absolute proof. Physical-layer incidents rarely hand you a signed confession. But the evidence lined up well. OPNsense saw stopped DISTRIBUTING, possible flapping, the USW aggregate ports had high link-down counts, UDM lagd logged carrier-down and PDU send failures, and replacing the cable pair restored stable LACP state.

My final classification was a likely marginal or bad cable pair on the OPNsense-to-USW LACP bundle. The transparent bridge made symptoms appear wider than the failed segment, LACP hashing made the failure intermittent rather than total, the UDM LAG representation added noise, and manual reset actions added extra log events.

6. Commands, Checks, and Lessons

These were the checks that mattered.

OPNsense LACP state 

ifconfig lagg0
ifconfig lagg1
Enter fullscreen mode Exit fullscreen mode

Healthy output should show status: active and member ports with ACTIVE,COLLECTING,DISTRIBUTING, matching the examples in the FreeBSD handbook.

laggproto lacp
status: active
laggport: igc1 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igc2 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
Enter fullscreen mode Exit fullscreen mode

OPNsense bridge state 

ifconfig bridge0
Enter fullscreen mode Exit fullscreen mode

Healthy output should show both LAGG members forwarding, consistent with OPNsense's bridge documentation.

member: lagg1
  state forwarding

member: lagg0
  state forwarding
Enter fullscreen mode Exit fullscreen mode

OPNsense logs and counters 

tail -f /var/log/system/latest.log
netstat -I lagg0 -w 1
netstat -I lagg1 -w 1
Enter fullscreen mode Exit fullscreen mode

Bad signs are lagg0 or lagg1 link state changes, Interface stopped DISTRIBUTING, and repeated member link-down events. Good signs are increasing packets and bytes with no new errors.

UniFi LAG surface 

ip -d link show dev eth6
ip -d link show dev eth7
ip -d link show dev lacp6
ip -d link show dev lacp7
ip -d link show dev lag0
tail -n 160 /var/log/lagd.log
Enter fullscreen mode Exit fullscreen mode

On the UDM Pro, do not panic at lag0 DOWN alone. In this case, the more useful signals were lagd carrier-down lines, failed PDU sends, and the UniFi controller view of aggregate members.

UniFi controller state

The controller view should agree with UniFi's port aggregation model. On the USW, look for active LACP members, the expected partner_system_id, and stp_state set to forwarding. On the UDM, look for the aggregate parent port and the masked member port.

After the fix

Monitor OPNsense, UniFi, and end-to-end reachability. OPNsense should show active LACP members that collect and distribute. UniFi should show active LACP state, forwarding STP state, and counters that stop increasing after the fix. End-to-end checks should confirm UDM to OPNsense reachability, OPNsense to gateway reachability, stable DHCP and DNS, and no VLAN-specific weirdness.

The important thing is not the absolute historical counter value. Historical counters may already be dirty. The important thing is whether they continue increasing after the fix.

The lessons were simple:

  • LACP instability often looks like general network weirdness.
  • Link up is not enough. LACP member state matters.
  • Appliance operating systems can hide the real dataplane behind strange abstractions.
  • Label physical topology before you need to debug it under pressure.
  • Replace suspect cables earlier than pride wants you to.

The technical explanation was deep. The fix was still copper.

Top comments (0)