DEV Community

Cover image for Debugging LACP Instability in a Transparent OPNsense Bridge
Andre Faria
Andre Faria

Posted on

Debugging LACP Instability in a Transparent OPNsense Bridge

#opnsense #networking #homelab

I run a transparent OPNsense bridge between a UniFi Dream Machine Pro and the rest of my LAN. It is deliberately boring at Layer 3: the UDM keeps routing, DHCP, DNS, firewall policy, WAN handling, and VLAN definitions. OPNsense sits inline as a Layer 2 bump in the wire.

The interesting part is that both sides of that bump use LACP.

I already wrote the build/configuration guide for this setup here: Building a Transparent LAGG (LACP) Bridge with OPNsense, UDM, and UniFi - A Practical Guide. That article explains how the bridge was built, how the LAGG devices were configured, and why I wanted the firewall to remain transparent.

This article is the other half of the story: what happens when that kind of setup fails in a non-obvious way.

Not a clean outage. Not a single "the network is down" moment. Just enough instability to make everything feel wrong.

1. Topology and Failure Surface

The topology looked like this:

                          +----------------------+
                          | UniFi Dream Machine  |
                          | kantharos-udm-pro    |
                          +----------+-----------+
                                     |
                         LACP aggregate, 2 x 1G
                                     |
                            OPNsense lagg0
                            "ingresslagg"
                          igc1 + igc2, LACP
                                     |
                          +----------v-----------+
                          | OPNsense bridge0     |
                          | "laggbridge"         |
                          +----------+-----------+
                                     |
                            OPNsense lagg1
                            "egresslagg"
                          igc4 + igc5, LACP
                                     |
                         LACP aggregate, 2 x 1G
                                     |
                          +----------v-----------+
                          | UniFi USW-Lite-16    |
                          | downstream LAN       |
                          +----------------------+
Enter fullscreen mode Exit fullscreen mode

On OPNsense, the relevant interfaces were:

igc1 + igc2 -> lagg0 -> ingresslagg -> toward UDM
igc4 + igc5 -> lagg1 -> egresslagg  -> toward USW
lagg0 + lagg1 -> bridge0 -> laggbridge
Enter fullscreen mode Exit fullscreen mode

The bridge is a FreeBSD bridge. The aggregates are FreeBSD lagg(4) interfaces using LACP. OPNsense exposes those through its Interfaces > Devices UI.

The expected healthy OPNsense state is:

laggproto lacp
status: active
laggport: igcX flags=<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igcY flags=<ACTIVE,COLLECTING,DISTRIBUTING>
Enter fullscreen mode Exit fullscreen mode

Those three member states matter:

  • ACTIVE: the member is participating in the LACP bundle.
  • COLLECTING: the member may receive traffic.
  • DISTRIBUTING: the member may transmit traffic.

For an LACP link, carrier alone is not enough. A cable can show link, but if the member is not collecting and distributing, it is not a healthy participant in the aggregate.

In a transparent bridge, that distinction matters more than usual. OPNsense is not routing around the problem. It is forwarding Ethernet frames between two aggregated links, much like the OPNsense bridge documentation describes for Layer 2 forwarding and MAC learning. If one LACP member misbehaves, the symptoms can leak across the whole Layer 2 segment.

2. Symptoms: Instability, Not Interruption

The failure did not present as a clean interruption.

There was no single point where the whole LAN died and stayed dead. Instead, the network became unstable:

  • traffic slowed down
  • clients behaved inconsistently
  • management sessions became flaky
  • UniFi and OPNsense did not always describe the same state
  • LACP state changed underneath the transparent bridge
  • the bridge looked partially alive and partially broken

This is exactly the sort of fault LACP makes annoying.

With a single Ethernet cable, a physical failure is usually obvious. The link drops. The port goes down. The device disappears.

With LACP, a single member can become marginal while the logical aggregate still exists. The point of a Link Aggregation Group is that multiple full-duplex point-to-point links are treated as one logical link, but the physical members still exist underneath. Some traffic survives. Some traffic lands on the bad member. Some flows stall, some retry, and some keep working. The user-facing symptom becomes "the network is weird", which is among the least useful sentences in infrastructure.

The reason is hashing. LACP does not normally split one flow across all cables like a striped disk. The FreeBSD handbook notes that Ethernet frame ordering means traffic between two stations stays on the same physical link, while the transmit algorithm tries to distinguish flows and balance them across the aggregate. Depending on the device and configuration, that hash may use Layer 2, Layer 3, or Layer 4 fields. In my OPNsense setup, the LAGG hash was Layer 2:

laggproto lacp lagghash l2
Enter fullscreen mode Exit fullscreen mode

A simplified model:

flow A -> member 1 -> works
flow B -> member 2 -> stalls
flow C -> member 1 -> works
flow D -> member 2 -> retries
Enter fullscreen mode Exit fullscreen mode

That creates a failure mode which feels like congestion, DNS trouble, Wi-Fi trouble, controller weirdness, or firewall slowness. It is not always obvious that the problem is a physical member inside an aggregate.

This is the central trap: partial LACP failure can masquerade as general network degradation.

3. OPNsense Evidence: The Bundle Was Actually Flapping

The strongest evidence came from OPNsense logs in the system log files (/var/log/system/system_20260605.log). Two windows mattered:

2026-06-05 02:26:32-02:28:01 UTC
2026-06-05 20:08:27-21:22:31 UTC
Enter fullscreen mode Exit fullscreen mode

During the earlier window, OPNsense saw:

igc1 and igc2 went down/up repeatedly
lagg0: link state changed to DOWN
lagg0: link state changed to UP
igc4/igc5: Interface stopped DISTRIBUTING, possible flapping
Enter fullscreen mode Exit fullscreen mode

During the major evening window:

20:08:27  lagg1 went DOWN
20:10:19  lagg1 came UP
20:19:12  lagg1 went DOWN again
20:24-20:41 igc4/igc5 continued bouncing
20:26:47  lagg0 dropped
20:34:36  lagg0 came back
21:05:10  lagg1 dropped again
21:05:44  lagg1 came back
21:22:28  lagg0 detached during final bypass/reset activity
21:22:31  lagg1 detached during final bypass/reset activity
Enter fullscreen mode Exit fullscreen mode

The most useful phrase was:

Interface stopped DISTRIBUTING, possible flapping
Enter fullscreen mode Exit fullscreen mode

That is not an application-layer symptom. It is not DNS. It is not an IP routing issue. It is not a firewall rule. It means the LACP member state changed at the link aggregation layer. A simplified LACP health path looks like this:

Physical carrier up
  v
LACP peer detected
  v
Correct partner/system/key information
  v
Member selected into aggregator
  v
Member allowed to collect and distribute traffic
Enter fullscreen mode Exit fullscreen mode

If a member stops distributing, the aggregate may still exist, but it is no longer healthy. The device has decided that member should not transmit traffic as a valid part of the bundle. The current healthy state after reconnecting the bridge looked like this:

lagg0:
  laggproto lacp lagghash l2
  laggport: igc1 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: igc2 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
  status: active

lagg1:
  laggproto lacp lagghash l2
  laggport: igc4 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: igc5 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
  status: active
Enter fullscreen mode Exit fullscreen mode

And the bridge itself:

bridge0:
  member: lagg1
    role root
    state forwarding

  member: lagg0
    role designated
    state forwarding
Enter fullscreen mode Exit fullscreen mode

That contrast matters. During the incident, OPNsense saw real LAGG instability. After remediation, it saw active LACP members and a forwarding bridge. This matches the healthy FreeBSD example where ifconfig lagg0 reports status: active and member ports with ACTIVE,COLLECTING,DISTRIBUTING flags in the FreeBSD link aggregation documentation.

4. UniFi Evidence: Correct Controller State, Weird UDM Internals

The UniFi side complicated the investigation because the UDM Pro did not expose this like a normal Linux LACP bond. UniFi's own Port Aggregation FAQ says static LAG is not supported and aggregation uses LACP, while also calling out that gateway support is limited to specific models including the UDM Pro.

Over SSH, the UDM showed:

eth6@switch0 UP
eth7@switch0 UP
lacp6 LOWER_UP
lacp7 LOWER_UP
lag0 DOWN / NO-CARRIER
Enter fullscreen mode Exit fullscreen mode

And /proc/net/bonding/lag0 showed:

Ethernet Channel Bonding Driver: v3.7.1
Bonding Mode: load balancing (round-robin)
MII Status: down
Enter fullscreen mode Exit fullscreen mode

The sysfs bonding view was suspicious too:

/sys/class/net/lag0/bonding/mode       balance-rr 0
/sys/class/net/lag0/bonding/slaves     empty
/sys/class/net/lag0/carrier            0
/sys/class/net/lag0/operstate          down
Enter fullscreen mode Exit fullscreen mode

For a normal Linux bonding LACP bond, this would be terrible. Linux bonding documentation describes enslaving interfaces through /sys/class/net/<bond>/bonding/slaves and shows /proc/net/bonding/<bond> output listing the slave interfaces and their MII status. I would expect something closer to:

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Slave Interface: ...
MII Status: up
Aggregator ID: ...
Partner Mac Address: ...
Enter fullscreen mode Exit fullscreen mode

That is not what the UDM showed, but the UniFi controller showed a more coherent story.

On the UDM:

Port 7:
  op_mode: aggregate
  aggregate_members: [7, 8]
  up: true
  speed: 1000

Port 8:
  aggregated_by: 7
  masked: true
  up: true
  speed: 1000
Enter fullscreen mode Exit fullscreen mode

On the USW-Lite-16-PoE:

Port 7:
  op_mode: aggregate
  aggregate_members: [7, 8]
  aggregate_num_ports: 2
  lacp_state:
    - member_port: 7, active: true, speed: 1000
    - member_port: 8, active: true, speed: 1000
  partner_system_id: e4:3a:6e:5d:a0:00
  stp_state: forwarding

Port 8:
  aggregated_by: 7
  lag_member: true
  stp_state: forwarding
Enter fullscreen mode Exit fullscreen mode

The partner_system_id is important. It matched the OPNsense lagg1 MAC:

e4:3a:6e:5d:a0:00
Enter fullscreen mode Exit fullscreen mode

That told me the USW was actually negotiating LACP with OPNsense. The UDM also had lagd involved:

Created LACP interface mapping: lacp6 -> eth6
LAG lag0: Interface mapping eth6 -> lacp6
Created LACP interface mapping: lacp7 -> eth7
LAG lag0: Interface mapping eth7 -> lacp7
LAG lag0: Switch port driver: RealtekTag, use_realtek_tag: false
Enter fullscreen mode Exit fullscreen mode

The SSH details for the UDM interfaces showed Realtek switch abstractions:

eth6@switch0:
  vlan protocol 802.1ad id 4088

eth7@switch0:
  vlan protocol 802.1ad id 4087

lacp6:
  rtk_sw_netdev

lacp7:
  rtk_sw_netdev
Enter fullscreen mode Exit fullscreen mode

So the UDM Pro was not behaving like a simple Linux host with two Ethernet slaves under an 802.3ad bond. It appeared to model LACP through UniFi's lagd and Realtek switch pseudo-interfaces. The Linux lag0 object looked like a control-plane artefact, not the whole dataplane truth.

That was the debugging lesson: on appliance hardware, not every OS-level network interface is equally authoritative. The better sources of truth were:

  • UniFi controller aggregate state
  • USW lacp_state
  • OPNsense ACTIVE,COLLECTING,DISTRIBUTING
  • STP forwarding state
  • packet counters moving without errors
  • successful pings through the bridge

In this incident, the UDM lag0 DOWN output was suspicious, but not decisive.

5. Root-Cause Analysis: Following the Physical Evidence

The most useful UniFi historical lines came from the UDM lagd logs:

lag0: eth7: carrier state is DOWN dropping received LACP PDU.
lag0: Failed to send PDU from eth6: Failed to write LACP data: Network is down (os error 100)
lag0: Failed to send PDU from eth7: Failed to write LACP data: Network is down (os error 100)
Enter fullscreen mode Exit fullscreen mode

This is where the investigation stopped being abstract. LACP depends on LACPDUs exchanged between the actor and partner; the Linux bonding documentation describes the LACPDU exchange used by 802.3ad mode, and the UniFi FAQ describes LACP as the protocol that helps both ends agree on aggregation settings. If a device cannot send LACP PDUs because the interface is down, or if it drops received LACP PDUs because carrier is down, the aggregate cannot stay stable.

That is different from: The two devices disagree about configuration.
It is closer to: The link is physically unstable enough that LACP control traffic cannot reliably move.

That points toward physical-layer causes such as a bad cable, a bad termination, a damaged connector, marginal port, electrical noise or PHY/link partner issue. The USW counters supported the same direction. The aggregate ports had the worst link-down history:

USW Port 7:
  link_down_count: 26
  tx_errors: 5
  tx_dropped: 5
  lag_member: true
  lacp_state: active

USW Port 8:
  link_down_count: 8
  lag_member: true
Enter fullscreen mode Exit fullscreen mode

For comparison, several ordinary ports had much lower link-down counts:

Port 1:  link_down_count 1
Port 2:  link_down_count 2
Port 3:  link_down_count 1
Port 9:  link_down_count 1
Port 11: link_down_count 1
Port 13: link_down_count 1
Port 14: link_down_count 1
Enter fullscreen mode Exit fullscreen mode

Counters alone do not prove causality. Ports can accumulate link-down counts from normal reboots, unplugging, reprovisioning, or moving devices. But combined with OPNsense LACP distribution failures and UniFi carrier/PDU errors, they become strong supporting evidence.

There was also a reset/recovery window on the USW:

2026-06-05 20:27:31 UTC
  USW-Lite-16-PoE adopted_at marker

2026-06-05 21:21:28 UTC
  switch disconnected

2026-06-05 21:22:13 UTC
  switch connected

2026-06-05 21:22:52 UTC
  switch provisioned

2026-06-05 21:23:05 UTC
  DHCPACK for USW-Lite-16-PoE, 192.168.1.21

2026-06-05 21:23:08 UTC
  controller state back ON
Enter fullscreen mode Exit fullscreen mode

That lined up with OPNsense seeing final LAG detach events around:

2026-06-05 21:22:28 UTC
2026-06-05 21:22:31 UTC
Enter fullscreen mode Exit fullscreen mode

That distinction matters. The reset caused some link events. The earlier instability was the thing being investigated. This is also why I kept the OPNsense and UniFi timelines separate: link events created by a deliberate reset are not the same kind of evidence as repeated LACP distribution failures before the reset.

After replacing the OPNsense-to-USW cable pair and restoring the bridge, the state became boring again:

igc1   up  1000baseT full-duplex
igc2   up  1000baseT full-duplex
igc4   up  1000baseT full-duplex
igc5   up  1000baseT full-duplex
lagg0  up
lagg1  up
bridge0 up
Enter fullscreen mode Exit fullscreen mode

The expected evidence for a marginal cable pair would be:

LACP member flaps on OPNsense egress LAG
USW aggregate ports show high link-down counts
UniFi logs show carrier down or LACP PDU failures
Symptoms are intermittent rather than hard-down
Replacing the cables restores stable LACP state
Enter fullscreen mode Exit fullscreen mode

The observed evidence was:

OPNsense lagg1 and igc4/igc5 flapped.
OPNsense logged "stopped DISTRIBUTING, possible flapping".
USW aggregate ports 7/8 had high link-down counts.
UDM lagd logged carrier-down and LACP PDU send failures.
After replacing the cable pair, OPNsense showed all members ACTIVE,COLLECTING,DISTRIBUTING.
USW showed both LACP members active.
Bridge forwarding and traffic counters looked normal.
Enter fullscreen mode Exit fullscreen mode

That is a good match, although other possible causes still existed (like a bad physical port on the USW, bad physical port on the OPNsense box, UniFi LAG implementation bug triggered by reset/provisioning, transient controller reprovisioning issue, electrical noise near the cable run or two separate faults overlapping).

But the cable-pair theory was the simplest explanation that fit the observed data and the successful fix.

My final classification:

Likely root cause:
  marginal/bad cable pair on the OPNsense-to-USW LACP bundle

Contributing factors:
  transparent bridge made symptoms appear wider than the failed segment
  LACP hashing made the failure intermittent rather than total
  UniFi's UDM LAG representation was misleading through /proc/net/bonding
  manual reset/bypass actions added extra log noise

Confirmed recovery condition:
  OPNsense LAG members active/collecting/distributing
  USW LACP state active
  bridge members forwarding
  packet counters moving without errors
  management reachability restored
Enter fullscreen mode Exit fullscreen mode

Not absolute proof. Physical-layer incidents rarely hand you a signed confession. But the logs, counters, and recovery behavior all pointed in the same direction.

6. Commands, Checks, and Lessons

These were the commands and checks that mattered.

OPNsense: check LACP state 

ifconfig lagg0
ifconfig lagg1
Enter fullscreen mode Exit fullscreen mode

Healthy output, matching the examples in the FreeBSD handbook:

laggproto lacp
status: active
laggport: igc1 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igc2 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
Enter fullscreen mode Exit fullscreen mode

For the USW-facing side:

laggport: igc4 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igc5 flags=<ACTIVE,COLLECTING,DISTRIBUTING>
Enter fullscreen mode Exit fullscreen mode

OPNsense: check the bridge 

ifconfig bridge0
Enter fullscreen mode Exit fullscreen mode

Healthy output. OPNsense's bridge documentation describes bridges as Layer 2 switching constructs with MAC learning, and optionally RSTP/STP to prevent loops:

member: lagg1
  state forwarding

member: lagg0
  state forwarding
Enter fullscreen mode Exit fullscreen mode

OPNsense: watch logs during reconnection 

tail -f /var/log/system/latest.log
Enter fullscreen mode Exit fullscreen mode

Bad signs:

lagg0: link state changed to DOWN
lagg1: link state changed to DOWN
Interface stopped DISTRIBUTING, possible flapping
igc4: link state changed to DOWN
igc5: link state changed to DOWN
Enter fullscreen mode Exit fullscreen mode

OPNsense: sample counters 

netstat -I lagg0 -w 1
netstat -I lagg1 -w 1
Enter fullscreen mode Exit fullscreen mode

Good signs:

packets increasing
bytes increasing
errs 0
colls 0
Enter fullscreen mode Exit fullscreen mode

UDM: inspect UniFi's LAG surface 

ip -d link show dev eth6
ip -d link show dev eth7
ip -d link show dev lacp6
ip -d link show dev lacp7
ip -d link show dev lag0
Enter fullscreen mode Exit fullscreen mode

In this case, the relevant details were:

eth6@switch0 UP
eth7@switch0 UP
lacp6 rtk_sw_netdev
lacp7 rtk_sw_netdev
lag0 bond mode balance-rr, no slaves, carrier 0
Enter fullscreen mode Exit fullscreen mode

The lesson: do not panic at lag0 DOWN alone on the UDM Pro. It may not represent the actual hardware dataplane.

UDM: inspect lagd 

tail -n 160 /var/log/lagd.log
Enter fullscreen mode Exit fullscreen mode

Bad lines:

carrier state is DOWN dropping received LACP PDU
Failed to send PDU ... Network is down
Starting deconfiguration
Removing LACP interface
Enter fullscreen mode Exit fullscreen mode

Normal lines:

Created LACP interface mapping: lacp6 -> eth6
Created LACP interface mapping: lacp7 -> eth7
Enter fullscreen mode Exit fullscreen mode

UniFi controller API: inspect device port state

The controller view should agree with UniFi's port aggregation model: sequential aggregate member ports, LACP rather than static LAG, and forwarding state on the aggregate.

For the USW:

{
  "port_idx": 7,
  "op_mode": "aggregate",
  "aggregate_members": [7, 8],
  "lacp_state": [
    { "active": true, "member_port": 7, "speed": 1000 },
    { "active": true, "member_port": 8, "speed": 1000 }
  ],
  "partner_system_id": "e4:3a:6e:5d:a0:00",
  "stp_state": "forwarding"
}
Enter fullscreen mode Exit fullscreen mode

For the UDM:

{
  "port_idx": 7,
  "op_mode": "aggregate",
  "aggregate_members": [7, 8],
  "up": true,
  "speed": 1000
}
Enter fullscreen mode Exit fullscreen mode

And port 8:

{
  "port_idx": 8,
  "aggregated_by": 7,
  "masked": true,
  "up": true,
  "speed": 1000
}
Enter fullscreen mode Exit fullscreen mode

What to monitor after the fix

On OPNsense:

status: active
all members ACTIVE,COLLECTING,DISTRIBUTING
Enter fullscreen mode Exit fullscreen mode

On UniFi:

lacp_state active on both members
stp_state forwarding
link_down_count not increasing
errors not increasing
drops not increasing
Enter fullscreen mode Exit fullscreen mode

End-to-end:

UDM can reach OPNsense management
OPNsense can reach the UDM gateway
clients keep stable DHCP/DNS
no VLAN-specific weirdness appears
Enter fullscreen mode Exit fullscreen mode

The important thing is not the absolute historical counter value. Historical counters may already be dirty. The important thing is whether they continue increasing after the fix.

The lessons were simple:

  • LACP instability often looks like general network weirdness.
  • Link up is not enough; LACP member state matters.
  • Appliance operating systems can hide the real dataplane behind strange abstractions.
  • Label physical topology before you need to debug it under pressure.
  • Replace suspect cables earlier than pride wants you to.

The technical explanation was deep. The fix was still copper.

Top comments (0)