Disclosure up front: I run IPYE, a company that leases IP space. It gets one short paragraph at the end; the rest is plain networking.
Two links between a server and a switch should give twice the capacity and survive the loss of either cable. Link aggregation delivers both, but not in the way many people expect: it does not make one connection twice as fast, and it can fail in confusing ways when the two ends disagree about the bundle. LACP (Link Aggregation Control Protocol, from IEEE 802.1AX, formerly 802.3ad) is the standard way to negotiate a bundle so that both ends agree. This article explains what LACP does, how traffic is distributed, a Linux bonding configuration, and how to verify and troubleshoot it. Examples use 192.0.2.10/26 on the bond and 2001:db8:100:10::10/64 for IPv6.
What LACP negotiates
An aggregated link (a LAG, bond, or port-channel) treats several physical ports as one logical interface. Two ways to build it:
- Static: both ends are configured to treat the ports as a bundle, with no negotiation. If cabling or configuration is wrong on one side, nothing detects it, and traffic can loop or vanish.
- LACP: the ends exchange LACPDUs and only add a port to the bundle if the partner agrees and is the same partner on all the member ports. A port with a mismatch is left out instead of corrupting the bundle.
LACP has active and passive modes. An active end sends LACPDUs; a passive end only replies. At least one side must be active. The LACPDU rate is slow (every 30 seconds) or fast (every second); fast detects failures sooner, though a link failure that the physical layer reports (loss of signal) is noticed immediately in either case. LACP's value is chiefly in catching cabling and configuration mistakes and in detecting a link that is up but not passing frames.
Hashing: how flows pick a link
Frames of one conversation should stay on one link to avoid reordering, so the sender chooses the member port by hashing fields from the packet. Choices on Linux's xmit_hash_policy:
| Policy | Fields hashed | When it fits |
|---|---|---|
| layer2 | Source and destination MAC | Many different peers on the same segment |
| layer2+3 | MAC plus IP addresses | Traffic to many different addresses |
| layer3+4 | IP addresses plus TCP/UDP ports | Many flows between few hosts (best distribution) |
| encap2+3, encap3+4 | Fields of the inner packet of tunnels | VXLAN, GRE and similar traffic |
Consequences follow directly. One TCP connection uses one link. A single iperf3 stream across a two-link bundle reaches the speed of one link, not two; run several parallel streams (iperf3 -P 4) to see the aggregate. Hashing is statistical, so with two links you might see an uneven split such as 70/30 by chance, particularly with few flows. And hashing applies only to outgoing traffic from each end: the server decides its transmit distribution, and the switch independently decides what it sends to the server. Configure both sides sensibly.
Linux bonding configuration
With iproute2:
ip link add bond0 type bond mode 802.3ad miimon 100 lacp_rate fast xmit_hash_policy layer3+4
ip link set eth1 down
ip link set eth2 down
ip link set eth1 master bond0
ip link set eth2 master bond0
ip link set bond0 up
ip addr add 192.0.2.10/26 dev bond0
ip -6 addr add 2001:db8:100:10::10/64 dev bond0
Member interfaces must be down when you enslave them. miimon 100 checks link state every 100 milliseconds. The bond takes its MAC from the first member by default, which matters if you have MAC-based filtering. To persist it, use systemd-networkd ([NetDev] Kind=bond with [Bond] Mode=802.3ad), NetworkManager, or your distribution's tools.
The switch side needs a matching LAG in LACP active or passive mode on the two ports, with the same speed and duplex, and the same VLAN configuration for the bundle as a whole. Always apply VLAN and trunk settings to the LAG interface, not to the individual members. Naming and syntax vary by vendor, so use your switch documentation.
Verifying
cat /proc/net/bonding/bond0
ip -d link show bond0
The bonding status lists each member, its link state, and, for 802.3ad, the aggregator ID and the partner's system MAC. Both members should show the same aggregator ID and the same partner MAC. Different partner MACs mean the cables lead to different switches that are not an MLAG pair, and the bundle will use only one of them. Check the switch side with its LAG status: both ports should show as bundled or up in the port-channel.
Test failure behaviour on purpose. Run ping against the host and disconnect one cable (or ip link set eth1 down), then watch loss: with LACP and a healthy bond, the outage should be brief, typically a fraction of a second to a few seconds depending on detection. Restore it and confirm the member rejoins.
Common problems
- Both sides passive. No LACPDUs are sent; no bundle forms.
- Members to different switches without MLAG. Standard LACP requires all members to terminate on the same logical partner. Multi-chassis designs (MLAG, vPC, or similar) exist for this, with vendor-specific behaviour.
- Mismatched speeds or VLANs. A member that differs from the others is excluded.
- LAG up but poor distribution. Check the hash policy against your traffic; a pair of servers exchanging one flow cannot balance.
- PXE or boot environments. Early boot code may not speak LACP. Some switches provide a fallback to a single active port; learn what yours does.
- MTU. Set the bond and members to the same MTU.
Alternatives
Linux also provides active-backup mode (one link in use, the other standby, no switch configuration), which is robust and simple when you want redundancy but not bandwidth. LACP is appropriate when you need both, and when you can configure both sides.
About IPYE
IPYE (https://my.ipye.com) leases IPv4, IPv6, and ASN space; a /48 IPv6 block is free for the first year. LOA, ROA, rDNS, and WHOIS are handled by IPYE. Locations: Los Angeles, New York, Ashburn, Salt Lake City, London, Amsterdam, and Frankfurt. Questions: ipye@turnel.com.
Disclosure: I run IPYE
Top comments (0)