DEV Community: FirstPassLab

22 Fortinet Patches, 2 Ivanti Auth Bypasses, 9 Intel UEFI Bugs — Your March 2026 Patching Cheat Sheet

FirstPassLab — Sat, 04 Apr 2026 16:56:43 +0000

Fortinet dropped 22 security patches on March 11, 2026, including a FortiOS authentication bypass (CVE-2026-22153) that lets unauthenticated attackers slip past LDAP-based VPN and FSSO policies. The same patch cycle addresses a heap buffer overflow (CVE-2025-25249) in FortiOS and FortiSwitchManager enabling remote code execution. Ivanti simultaneously patched a high-severity auth bypass in Endpoint Manager. If you manage FortiGate firewalls, Ivanti EPM, or Intel-based infrastructure, here's your prioritized action plan.

TL;DR: FortiOS 7.6.0–7.6.4 has an auth bypass that grants unauthorized network access without credentials. Patch to 7.6.5+ immediately if you use Agentless VPN or FSSO with LDAP.

The Fortinet Patch Dump: What Actually Matters

Fortinet released fixes for 22 security defects across its portfolio. Here's the breakdown of what you need to care about:

CVE	Product	CVSS	Impact	Exploited?
CVE-2026-22153	FortiOS 7.6.0–7.6.4	7.2	Auth bypass (LDAP/VPN/FSSO)	No
CVE-2025-25249	FortiOS, FortiSASE, FortiSwitchManager	7.4	RCE via heap overflow	No
CVE-2026-24018	FortiClientLinux	7.4	Local privesc to root	No
CVE-2026-30897	FortiOS API	5.9	Stack overflow / code exec	No
N/A	FortiWeb	High	Auth rate-limit bypass	No
N/A	FortiSwitchAXFixed	High	Unauthorized cmd exec	No

None are currently exploited in the wild. But Fortinet's track record says exploitation follows disclosure by days, not weeks. CVE-2026-24858 — a related FortiOS SSO auth bypass — was actively exploited in January 2026 with attackers creating rogue admin accounts before patches even shipped.

CVE-2026-22153: The Auth Bypass You Need to Fix Today

This is a CWE-288 authentication bypass in FortiOS that lets an unauthenticated attacker bypass LDAP authentication for Agentless VPN or FSSO policies. Successful exploitation grants unauthorized access to network resources without valid credentials.

The catch: it requires a specific LDAP server configuration. But Agentless VPN and FSSO are exactly the features that enterprise networks deploy at scale. If your FortiGate authenticates remote users or maps AD users to firewall policies via FSSO, you're in the blast radius.

Affected: FortiOS 7.6.0 through 7.6.4
Fix: Update to FortiOS 7.6.5+

CVE-2025-25249: Heap Overflow → Remote Code Execution

A heap-based buffer overflow (CWE-122) in the cw_acd daemon of FortiOS and FortiSwitchManager. Remote unauthenticated attackers can execute arbitrary code via crafted requests.

The version spread is brutal:

FortiOS 7.6.0–7.6.3
FortiOS 7.4.0–7.4.8
FortiOS 7.2.0–7.2.11
FortiOS 7.0.0–7.0.17
FortiOS 6.4.0–6.4.16
FortiSwitchManager 7.2.0–7.2.5
FortiSwitchManager 7.0.0–7.0.5

That covers essentially every FortiOS release train still in production.

Ivanti: Two More Auth Problems

Ivanti released patches in Endpoint Manager 2024 SU5:

CVE	CVSS	Impact
CVE-2026-1603	7.4	Auth bypass exposing credential data
CVE-2026-1602	5.3	SQL injection

CVE-2026-1603 is the bigger concern — an auth bypass that exposes credential data remotely. Given Ivanti's history (CVE-2025-22457 in Connect Secure was a zero-day RCE exploited before the patch), don't sit on this one.

Your Prioritized Patching Plan

Based on severity, exploitability, and typical network exposure:

Priority 1: FortiOS 7.6.x (CVE-2026-22153) — This Week

If you run Agentless VPN or FSSO with LDAP authentication, this is job #1.

# Check your current FortiOS version
get system status

# Verify LDAP server configuration
show user ldap

# Check if Agentless VPN or FSSO is configured
show user fsso
diagnose debug application fssod -1

Priority 2: FortiOS/FortiSwitchManager (CVE-2025-25249) — Within 2 Weeks

The heap overflow affects nearly all FortiOS versions. Attack complexity is high, but impact is full RCE. Schedule alongside your CVE-2026-22153 patching.

Priority 3: FortiClientLinux (CVE-2026-24018) — Next Maintenance Window

Local privesc to root via symlink following. If you deploy FortiClient on Linux endpoints, queue it up.

Priority 4: Ivanti EPM (CVE-2026-1603) — Within 2 Weeks

Update to EPM 2024 SU5. Auth bypass + credential exposure = potential cascade.

Priority 5: Intel UEFI Firmware — Next Quarterly Window

Intel published advisory INTEL-SA-01234 with nine UEFI vulnerabilities across 45+ processor models. Five rated high severity. Requires local access, so lower urgency — but UEFI compromises persist across OS reinstalls.

Why Does Fortinet Keep Having Auth Bypasses?

This is the pattern that should concern you: multiple authentication bypass vulnerabilities within Q1 2026 alone.

CVE-2026-24858 was actively exploited as a zero-day in January:

Attackers created unauthorized local admin accounts on FortiGate appliances
Downloaded full device configurations (including VPN credentials)
Modified firewall policies to enable persistent access

Now CVE-2026-22153 arrives — another auth bypass, same vulnerability class (CWE-288). This suggests a systemic issue in how FortiOS handles authentication flows.

If Fortinet is your primary perimeter defense:

Don't rely solely on FortiGate for auth — integrate with a dedicated IdP
Enforce MFA at every layer
Monitor for config changes via FortiAnalyzer or SIEM
Consider defense-in-depth beyond a single-vendor auth stack

Post-Patch: Check for Pre-Patch Exploitation

Even after patching, verify these weren't exploited before your update window:

For CVE-2026-22153 (LDAP bypass):

# Check for unexpected VPN sessions
diagnose vpn tunnel list
get vpn ssl monitor

# Review admin login history
diagnose sys admin list

# Check for unauthorized policy changes
execute log filter device 0
execute log filter category 1
execute log display

For CVE-2025-25249 (heap overflow):

# Monitor crashlog for cw_acd daemon
diagnose debug crashlog read

# Check for unexpected processes
fnsysctl ls -la /tmp/

For Ivanti EPM (CVE-2026-1603):

Review EPM audit logs for unusual auth events
Check for new/modified admin accounts
Monitor SQL query logs for injection patterns

The Bigger March 2026 Picture

This wasn't just Fortinet and Ivanti. March 2026 was one of the heaviest patch loads of the year:

Microsoft: 83 vulnerabilities including two publicly disclosed zero-days
Adobe: 80 vulnerabilities across eight products
SAP: Critical NetWeaver flaws
Siemens, Schneider, Moxa, Mitsubishi Electric: ICS/OT patches

If you haven't built an automated patch validation pipeline (test → staging → production), March 2026 is your wake-up call.

Originally published at FirstPassLab. More deep dives on network security engineering at firstpasslab.com.

Disclosure: This article was adapted from original research with AI assistance in editing and formatting.

SRv6 uSID Is Replacing MPLS at Scale — Here's How to Migrate with Working IOS XR Configs

FirstPassLab — Fri, 03 Apr 2026 22:56:35 +0000

If you work in service provider networking, the question is no longer whether your network will move from MPLS to SRv6 — it is when. By early 2026, over 85,000 Cisco routers are running SRv6 uSID in production across more than 60 operators worldwide. Swisscom, Rakuten Mobile, SoftBank, and Jio Platforms have either completed or are actively executing their SRv6 uSID migrations.

This is not a lab curiosity anymore. It is the dominant transport transformation in the SP space.

In this article, we'll break down what SRv6 uSID actually is, why it matters, how to configure it on Cisco IOS XR, and how to execute a lossless migration from legacy MPLS or SR-MPLS to SRv6 uSID F3216.

What Is SRv6 uSID and Why Should You Care?

SRv6 (Segment Routing over IPv6) encodes forwarding instructions directly into IPv6 addresses. Each Segment Identifier (SID) is a 128-bit IPv6 address, and a list of SIDs in the IPv6 Segment Routing Header (SRH) describes the packet's path through the network.

The problem with early SRv6 implementations was overhead. A full 128-bit SID per hop adds up fast — four waypoints meant 64 bytes of SRH, which created MTU headaches in real networks.

SRv6 micro-SIDs (uSIDs) solve this by compressing multiple segment instructions into a single 128-bit IPv6 address, called a uSID Carrier. The dominant production format is F3216:

32-bit uSID Block — identifies the SRv6 domain
16-bit uSID IDs — identifies specific nodes or functions

A single uSID carrier encodes up to 6 micro-SIDs, which means you can describe 18 source-routing waypoints in only 40 bytes of overhead. That is comparable to or better than an MPLS label stack for the same path complexity.

Pro Tip: When planning your SRv6 addressing scheme, choose your 32-bit uSID block prefix carefully. This prefix identifies your entire SRv6 domain, and changing it later requires touching every node. Treat it like your BGP AS number — pick it once, document it everywhere.

SR-MPLS vs. SRv6 uSID: Key Differences

Attribute	SR-MPLS	SRv6 uSID (F3216)
Encoding	MPLS label stack	IPv6 SRH with compressed uSIDs
Path description	Label per hop (4 bytes each)	Up to 6 waypoints per 16-byte carrier
Data plane	MPLS forwarding	Native IPv6 forwarding
Programmability	Limited to label operations	Full IPv6 extension header programmability
Network slicing	Complex, requires dedicated LSPs	Native support via FlexAlgo + uSID

The programmability advantage is significant. With SRv6, you can define custom behaviors (called SRv6 functions) at each segment endpoint — enabling service chaining, VPN overlay binding, and traffic engineering that would require multiple protocol interactions in MPLS.

Configuring SRv6 uSID on Cisco IOS XR

Let's walk through a complete SRv6 uSID deployment on a Cisco 8000 Series router running IOS XR.

Step 1: Define SRv6 Locators

The locator is the foundation of your SRv6 configuration. It defines the prefix that the router will use to generate its local SIDs:

segment-routing srv6
 encapsulation
  source-address fcbb:bb00:0001::1
 !
 locators
  locator myuSID
   micro-segment behavior unode psp-usd
   prefix fcbb:bb00:0001::/48
  !
 !
!

Key points:

The source-address is the IPv6 address used as the outer source in SRv6-encapsulated packets. It must be reachable in your IGP.
micro-segment behavior unode psp-usd enables the F3216 uSID format with Penultimate Segment Pop (PSP) and Ultimate Segment Decapsulation (USD) behaviors.
The /48 prefix defines the locator block. The first 32 bits (fcbb:bb00) are the uSID block, and bits 33-48 (0001) are this node's uSID ID.

Pro Tip: Always use psp-usd as your default uSID behavior. PSP removes the SRH at the penultimate hop, reducing processing overhead on the endpoint node. USD handles decapsulation cleanly when this node is the final destination.

Step 2: Integrate with IS-IS

IS-IS advertises SRv6 locator reachability across the network:

router isis CORE
 is-type level-2-only
 net 49.0001.0000.0000.0001.00
 address-family ipv6 unicast
  metric-style wide
  segment-routing srv6
   locator myuSID
  !
 !
 interface Loopback0
  address-family ipv6 unicast
  !
 !
 interface HundredGigE0/0/0/0
  point-to-point
  address-family ipv6 unicast
  !
 !
!

When applied, the router advertises its /48 locator prefix into IS-IS. Other routers install this as a route in their IPv6 RIB, enabling SRv6 forwarding.

Step 3: Enable BGP and EVPN Overlays

For L3VPN and EVPN services, bind the uSID locator to BGP:

router bgp 65001
 address-family vpnv4 unicast
 !
 address-family vpnv6 unicast
 !
 address-family l2vpn evpn
 !
 segment-routing srv6
  locator myuSID
 !
 neighbor 2001:db8::2
  remote-as 65001
  update-source Loopback0
  address-family vpnv4 unicast
  !
  address-family l2vpn evpn
  !
 !
!

evpn
 segment-routing srv6
  locator myuSID
 !
!

With this, BGP allocates SRv6 uSIDs for VPN prefixes and advertises them to PE peers. The receiving PE uses the uSID to forward encapsulated traffic directly over the IPv6 underlay — no LDP, no RSVP, no separate MPLS signaling stack.

Migration Strategy: Ship in the Night

The most critical question for any production network: how do we get from MPLS to SRv6 without dropping traffic?

Cisco's recommended approach is Ship in the Night — running both forwarding planes simultaneously during the transition.

State 1: Initial (Legacy MPLS or SR-MPLS)

Your network runs traditional MPLS with LDP/RSVP, or SR-MPLS with format1 SIDs. All services use MPLS transport. No SRv6 configuration exists.

State 2: In-Migration (Dual Mode)

You deploy SRv6 uSID locators alongside the existing MPLS configuration. Both forwarding planes coexist:

Existing MPLS/SR-MPLS control plane continues to signal labels and forward traffic
SRv6 uSID locators are advertised in IS-IS concurrently
BGP and EVPN are configured with uSID locators on all PE nodes
Traffic can flow over either transport depending on which SIDs the ingress PE selects

This is the critical phase. You must enable overlay F3216 locators under BGP and EVPN on all PE nodes before cutting over any services.

State 3: End State (SRv6 uSID Only)

Once all services are verified on SRv6 uSID transport, you remove legacy MPLS configuration.

Pro Tip: Use the delayed-delete command when removing format1 locators during cutover. This keeps old SIDs active in the forwarding table for a configurable period while F3216 SIDs take over. A 60-second delay is usually sufficient for BGP to reconverge.

Migration Caveats

Line card reloads may be required during hardware profile transitions on some platforms. Plan maintenance windows.
Cisco 8000 Series (K100, A100 ASICs) and 8700-MOD support dual-mode natively. Verify your hardware.
For mixed-vendor networks, check draft-ietf-spring-srv6-mpls-interworking for SRv6-MPLS gateway interworking standards.

Verification and Troubleshooting

Verify SRv6 Locator Status

RP/0/RP0/CPU0:PE1# show segment-routing srv6 locator

Name          Prefix                    Status
----          ------                    ------
myuSID        fcbb:bb00:0001::/48       Up

If status shows Down, check:

No prefix conflict with another locator on the same node
IS-IS is configured to advertise the locator
The platform supports the specified uSID behavior

Verify IS-IS SRv6 Advertisement

RP/0/RP0/CPU0:PE1# show isis segment-routing srv6 locators detail

IS-IS CORE Level-2 SRv6 Locators:

  Locator: myuSID
    Prefix: fcbb:bb00:0001::/48
    Topology: IPv6 Unicast
    Algorithm: 0
    Metric: 1
    Advertised: Yes

Verify BGP SRv6 SID Allocation

RP/0/RP0/CPU0:PE1# show bgp vpnv4 unicast vrf CUSTOMER-A 10.1.1.0/24

BGP routing table entry for 10.1.1.0/24, Route Distinguisher: 65001:100
  Local
    fcbb:bb00:0001:: (via SRv6 SID: fcbb:bb00:0001:e004::)
    Origin IGP, metric 0, localpref 100, valid, sourced, best
    SRv6 SID: fcbb:bb00:0001:e004::
      Function: End.DT4
      Locator: myuSID

The End.DT4 function indicates a decapsulation-and-lookup behavior for IPv4 VPN traffic — the SRv6 equivalent of a VPN label in MPLS.

Common Troubleshooting Scenarios

SRv6 locator is Up but BGP is not allocating SIDs:
Ensure segment-routing srv6 locator myuSID is configured under both router bgp and evpn.

Traffic is blackholing after enabling uSID locators:
Verify all transit routers have IS-IS SRv6 locator routes in their IPv6 RIB. A single router without the locator route will drop SRv6-encapsulated packets.

MTU issues after migration:
SRv6 encapsulation adds a minimum of 40 bytes (IPv6 outer header) plus SRH overhead. Core links need at least 9216-byte MTU.

Key Takeaways

SRv6 uSID is production-ready at scale. 85,000+ routers deployed globally across 60+ operators. This is not early-adopter tech.
F3216 solves the MTU problem. Six micro-SIDs per 128-bit carrier means SRv6 overhead is comparable to MPLS label stacks.
Ship in the Night enables lossless migration. Run both planes in parallel, validate per-service, cut over at your own pace.
Configuration is straightforward. Three building blocks — locators, IS-IS integration, and BGP/EVPN binding — cover the majority of deployments.
Open-source support is growing. FRRouting 10.5 now supports SRv6 uSID, extending this beyond Cisco-only shops to SONiC and other FRR-based platforms.

The MPLS-to-SRv6 migration is the defining infrastructure shift for service providers in 2026. Start with a lab, validate with Ship in the Night, and build operational confidence before touching production. The technology is ready — the question is whether you are.

Originally published at FirstPassLab. For more deep dives on networking protocols and infrastructure, visit firstpasslab.com.

AI Disclosure: This article was adapted and edited with AI assistance. All technical content is based on published vendor documentation, IETF standards, and real-world deployment data.

9 AppArmor Kernel Bugs Hidden Since 2017 — Root Escalation, Container Escape, and 12.6M Linux Systems Exposed

FirstPassLab — Fri, 03 Apr 2026 16:54:24 +0000

Nine critical vulnerabilities in Linux AppArmor — collectively dubbed CrackArmor by the Qualys Threat Research Unit — let any unprivileged local user escalate to root, escape container isolation, and crash entire systems via kernel panic. These flaws have existed in every kernel since v4.11 (April 2017). If you run infrastructure on Ubuntu, Debian, or SUSE — and if you use Kubernetes, your nodes almost certainly do — this is a patch-now situation.

Over 12.6 million enterprise Linux instances run with AppArmor enabled by default. Here's what broke, why it matters for your containers and infrastructure, and exactly how to check and fix it.

The Attack Surface: What CrackArmor Actually Does

CrackArmor exploits a confused deputy flaw in AppArmor's kernel implementation. AppArmor is the Mandatory Access Control (MAC) framework that confines processes under security profiles — it ships enabled by default on Ubuntu, Debian, and SUSE. The nine vulnerabilities let an attacker trick privileged processes into performing actions they shouldn't.

Here's the attack matrix:

Attack Vector	Mechanism	Impact
Profile manipulation	Write to `/sys/kernel/security/apparmor/.load`, `.replace`, `.remove`	Disable protections on any service
Privilege escalation	Leverage setuid binaries (sudo, postfix) to modify AppArmor profiles	Full root from unprivileged user
Container escape	Load crafted "userns" profile to bypass namespace restrictions	Break Kubernetes/container isolation
Denial of service	Trigger recursive stack exhaustion via deeply nested profiles	Kernel panic and reboot
KASLR bypass	Out-of-bounds read during profile parsing	Disclose kernel memory layout

The Qualys advisory puts it simply: "This is comparable to an intruder convincing a building manager with master keys to open restricted vaults that the intruder cannot enter alone." The attacker doesn't need special permissions — they manipulate the privileged machinery that already exists.

Why This Matters for Your Containers and Infrastructure

AppArmor isn't just an abstract kernel feature. It's the trust boundary for a massive amount of production infrastructure:

Kubernetes clusters. According to Kubernetes docs, AppArmor profiles are the recommended mechanism to "restrict a container's access to resources." CrackArmor breaks that restriction entirely. An attacker inside any container can escape to the host, then pivot to other containers — controllers, databases, monitoring.

Linux-based appliances. Many firewalls, SDN controllers, and network appliances ship Ubuntu or Debian under the hood with AppArmor enabled. A local exploit on any of these devices means full control.

NFV and edge deployments. Containerized network functions and edge computing nodes use AppArmor to isolate workloads. A container escape in these environments doesn't just compromise one function — it can expose the entire control plane.

CI/CD systems and jump boxes. If your build infrastructure or management stations run affected distros, an attacker with unprivileged access (stolen SSH creds, compromised service account) gets root.

Infrastructure Component	AppArmor Exposure	Risk Level
Kubernetes nodes (Ubuntu/Debian)	AppArmor profiles per pod	Critical — container escape
Linux-based firewalls/appliances	Often enabled by default	Critical — root = device control
NFV / edge platforms	Default MAC enforcement	High — lateral movement
CI/CD runners / jump boxes	Varies	High — pivot to production
Red Hat / CentOS / Fedora	SELinux (not AppArmor)	Not affected

How the Container Escape Works

This is the most dangerous chain for anyone running Kubernetes or Docker.

Ubuntu's user-namespace restrictions were specifically designed to prevent unprivileged users from creating fully-capable namespaces. CrackArmor bypasses this by loading a specially crafted "userns" profile for /usr/bin/time, enabling an attacker to create namespaces with full capabilities.

In a Kubernetes environment:

Attacker gains shell inside a container (compromised app, RCE in a dependency)
Exploits CrackArmor to escape to the host node
From the host, accesses all other containers on that node
Kubernetes AppArmor security boundary = nullified

The DoS path is equally nasty: deeply nested profiles trigger recursive removal routines that overflow the 16KB kernel stack, causing immediate kernel panic. An unexpected reboot of your K8s node is a production outage.

Am I Affected? Check in 10 Seconds

# Check if AppArmor is loaded
aa-status 2>/dev/null && echo "AppArmor ACTIVE - check kernel version" || echo "AppArmor not active"

# Check kernel version (v4.11+ is vulnerable if AppArmor is active)
uname -r

Run this across your infrastructure — not just servers. Check:

Kubernetes worker/control-plane nodes
CI/CD runners (GitHub Actions self-hosted, GitLab runners, Jenkins agents)
Docker hosts
Network appliances and firewalls on Linux
Jump boxes and bastion hosts

Affected: Any distro using AppArmor + kernel ≥ v4.11 without March 2026 patches

Distribution	Affected?	Patch Status
Ubuntu (all supported)	Yes — AppArmor default	`apt update && apt upgrade`
Debian (bookworm, trixie)	Yes — AppArmor default	`apt update && apt upgrade`
SUSE / openSUSE	Yes — AppArmor default	`zypper refresh && zypper update`
Red Hat / CentOS / Fedora	No — uses SELinux	Not affected
Alpine Linux	Varies	Check `aa-status`

Patch Now: Step-by-Step

Ubuntu / Debian

sudo apt update && sudo apt upgrade -y linux-image-$(uname -r)
sudo reboot

SUSE

sudo zypper refresh && sudo zypper update kernel-default
sudo reboot

Yes, reboots are required — this is a kernel-level fix.

Post-Patch: Audit Profile Integrity

# List all loaded profiles and enforcement mode
aa-status

# Check for unexpected profiles
ls /etc/apparmor.d/

# Verify no profiles were modified recently
find /etc/apparmor.d/ -mtime -7 -ls

Harden Kubernetes After Patching

# Ensure AppArmor annotations are enforced on pods
metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/my-container: runtime/default

Configure admission controllers to reject pods without AppArmor profiles — a post-patch hardening step that prevents future profile manipulation.

CVE Status: Don't Wait for Numbers

As of mid-March 2026, no CVE identifiers have been assigned. The upstream kernel team typically assigns CVEs 1-2 weeks after fixes land in stable releases. Qualys has published the full technical advisory and proof-of-concept details.

Do not wait for CVE assignment to justify patching. The exploit details are public.

Context: How CrackArmor Compares

Vulnerability	Attack Vector	Impact	Key Difference
CrackArmor (AppArmor)	Local unprivileged	Root + container escape	Requires local access first
Fortinet FortiOS CVE-2025-24472	Remote unauth	Super-admin access	No local access needed
Ivanti Connect Secure CVE-2025-22467	Authenticated remote	Remote code execution	Needs valid credentials

CrackArmor requires local access — but in environments where attackers already have a foothold (compromised container, stolen SSH key, malicious insider), it turns limited access into total control.

Three Takeaways

Defense in depth isn't optional. AppArmor was one layer. When it failed, containers, namespaces, and privilege boundaries all failed together. Layer your controls independently — don't rely on a single MAC framework.
Know your attack surface. CrackArmor needs local access first. Your SSH hardening, network access controls, and authentication policies are the real first line. If an attacker can't get local access, CrackArmor is irrelevant.
Patch management is engineering, not ops. The ability to rapidly identify, test, and deploy kernel patches across heterogeneous infrastructure is a core engineering competency — not an afterthought.

Originally published at FirstPassLab. For more deep dives on network security, cloud architecture, and infrastructure engineering, visit firstpasslab.com.

🤖 AI Disclosure: This article was adapted from the original blog post with AI assistance. The technical content, vulnerability analysis, and remediation steps are based on primary sources including the Qualys CrackArmor advisory and vendor security bulletins.

386 Global Outages in One Week: What ThousandEyes Q1 2026 Data Reveals About Modern Network Fragility

FirstPassLab — Thu, 02 Apr 2026 22:53:25 +0000

Cisco ThousandEyes tracked between 199 and 386 global network outage events per week during Q1 2026, with a 62% spike during the last week of February. The defining outage pattern of 2026 isn't broken components — it's systems interacting in ways nobody designed for.

If your monitoring stops at your network boundary, you're blind to the failures that actually hit your users.

The Numbers: Q1 2026 Outage Data

ThousandEyes monitors ISPs, cloud service providers, conferencing services, and edge networks (DNS, CDN, SECaaS). Here's the week-by-week breakdown:

Week	Global Outages	WoW Change	U.S. Outages
Dec 29 – Jan 4	199	−14%	71
Jan 5 – Jan 11	255	+28%	135
Jan 12 – Jan 18	263	+3%	149
Jan 19 – Jan 25	236	−10%	148
Jan 26 – Feb 1	314	+33%	156
Feb 2 – Feb 8	264	−16%	157
Feb 9 – Feb 15	247	−6%	136
Feb 16 – Feb 22	239	−3%	114
Feb 23 – Mar 1	386	+62%	184
Mar 2 – Mar 8	304	−21%	124
Mar 9 – Mar 15	272	−11%	155
Mar 16 – Mar 22	277	+2%	144

The January 5–11 week saw U.S. outages surge 90% (71 → 135) as operations resumed after the holiday change-freeze period. Global outages increased 178% from November to December 2025, rising from 421 to 1,170 monthly incidents.

The Biggest Outages: Who Went Down and Why

The highest-profile incidents hit Tier 1 carriers, cloud platforms, and critical infrastructure:

Date	Provider	Duration	Regions	Root Cause Pattern
Jan 6	Charter/Spectrum	1h 43m	U.S. + 9 countries	Node migration across NYC, DC, Houston
Jan 17	TATA Communications	23m	14 countries	Cascading failures Singapore → U.S. → Japan
Jan 27	Cloudflare	2h 23m	U.S. + 4 countries	Chicago → Winnipeg → Aurora expansion
Jan 27	Lumen	1h 5m	U.S. + 13 countries	Oscillating DC → Detroit → LA → DC
Feb 10	Hurricane Electric	25m	U.S. + 12 countries	Dallas → Atlanta → Charlotte → NYC
Feb 17	Cogent	1h 20m	U.S. + 4 countries	Recurring Denver node failures
Feb 20	Cloudflare BYOIP	1h 40m	Global	Automated maintenance withdrew customer IP prefixes
Feb 26	GitHub	1h	U.S. + 6 countries	Washington D.C. centered
Mar 4	PCCW	48m	14 countries	Marseille → LA → Hong Kong cascade
Mar 6	ServiceNow	1h 3m	29 countries	Austin → Seattle → Chicago node migration
Mar 20	Arelion (Telia)	1h 38m	18+ countries	Ashburn → DC → Dallas → Newark expansion

The Cloudflare BYOIP incident (Feb 20) is the most instructive: a bug in an automated maintenance task caused Cloudflare to unintentionally withdraw customer IP address advertisements from the global routing table. No human made a mistake — the automation itself created the failure.

Cogent appeared twice (Feb 17 and Mar 12), both times centered on Denver — a pattern that multi-path SD-WAN failover is specifically designed to survive.

The Cost: $14K–$23.7K Per Minute

Enterprise downtime costs between $14,000 and $23,750 per minute depending on organization size (EMA, ITIC, BigPanda 2026). Over 90% of midsize and large companies report hourly costs exceeding $300K.

Industry	Avg. Hourly Cost	Key Risk Factor
Financial Services	$1M – $9.3M	Real-time transaction processing
Healthcare	$318K – $540K	Patient safety + HIPAA fines
Retail / E-commerce	$1M – $2M (peak)	Lost sales + customer churn
Manufacturing	$260K – $500K	Supply chain disruption
Automotive	$2.3M	Assembly line stoppages
Telecommunications	$660K+	Service credits + customer churn

Global 2000 companies collectively lose $400 billion annually from unplanned downtime.

Root Causes: It's the Network, and It's Us

Network and connectivity issues are the #1 cause of IT service outages (31%), per the Uptime Institute's 2024 Data Center Resiliency Survey. Within that:

Configuration/change management failures: 45% — BGP route policies, OSPF area design, SD-WAN overlay topology. Understand blast radius before executing changes.
Third-party provider failures: 39% — Cogent, Lumen, Charter all had repeated outages. Multi-homed BGP with RPKI validation is the engineering response.
Software/system failures: 36% — 64% of these stem from config/change issues. 44% of respondents say network changes cause outages "several times a year."

Human error contributes to 66–80% of all downtime. Of those, 85% stem from staff not following procedures (47%) or flawed processes (40%). Only 3% of organizations catch all mistakes before impact.

The New Pattern: Autonomous Agent Interaction Failures

This is the section that matters most for 2026 and beyond.

ThousandEyes identifies autonomous agents — auto-scalers, AIOps platforms, remediation bots, intent-based automation — as the single biggest emerging risk. The pattern is no longer "something broke" but "systems interacting in ways nobody anticipated."

Three 2025 incidents that define this pattern:

AWS DynamoDB (Oct 2025): Two independent DNS management components operated correctly within their own logic. A delayed component applied an older DNS plan at the precise moment a cleanup operation deleted the newer plan. Neither malfunctioned — their timing interaction created the failure.

Azure Front Door (Oct 2025): A control plane created faulty metadata. Automated detection correctly blocked it. The cleanup operation triggered a latent bug in a different component. Every system did its job. The interaction produced the outage.

Cloudflare Bot Management (Nov 2025): A configuration file exceeded a hard-coded limit. The generating system operated correctly. The proxy enforcing the limit also operated correctly. The output of one system exceeded the constraints of another.

The proliferation of agents creates three specific risks:

Cascading failures: Agents make decisions in milliseconds. When one agent reacts to another's output, mistakes propagate before humans detect degradation.
Optimization conflicts: A performance agent, a cost-reduction agent, and a reliability agent may work against each other simultaneously.
Intent uncertainty: When one agent changes a route, other agents must determine whether the change was intentional. Get that wrong and agents start undoing each other's work.

5-Layer Defense Strategy

Layer 1: End-to-End Observability Beyond Your Boundary

Traditional SNMP traps capture what happens inside your infrastructure. The Q1 data shows outages cascading across Tier 1 carriers (Arelion across 18 countries), cloud platforms (ServiceNow across 29 countries), and edge networks simultaneously. You need visibility into dependencies you don't own — ThousandEyes, Catchpoint, and Kentik provide Internet-wide path analysis.

Layer 2: Multi-Homed BGP with RPKI Validation

Cogent's recurring Denver outages demonstrate why single-carrier dependency is unacceptable. Implement BGP RPKI Route Origin Validation with at least two upstream providers. Configure BGP communities and local preference to steer traffic away from degraded paths automatically. IX peering adds a third failover path.

Layer 3: Automated Change Validation

45% of outages come from config/change failures. Every network change needs pre-deployment validation. Network digital twins (Batfish, ContainerLab) simulate route policy impact before production. Pair with Terraform IaC for auditable, reversible changes.

Layer 4: Agent Coordination as a Design Concern

If your network runs auto-scalers, AIOps remediation, and intent-based policies, define interaction boundaries. Establish rate limits on automated changes. Implement circuit breakers that halt cascading automation when change velocity exceeds thresholds. This is the evolution of network automation from scripting to architecture.

Layer 5: Redundancy Matched to Financial Exposure

90% of organizations require minimum 99.99% availability — only 52.6 minutes of annual downtime. At $14K/min for midsize businesses, that's $736K of maximum tolerable loss per year. Calculate your specific exposure: Annual Revenue ÷ Total Working Hours = Hourly Revenue at risk. That number justifies geographic distribution, SD-WAN multi-path failover, and dual-DC designs.

Action Items Right Now

Map your carrier dependencies — run traceroutes from multiple vantage points and identify single-carrier paths
Implement RPKI if you haven't — route origin validation prevents the BGP hijacks and leaks that contributed to several Q1 incidents
Audit your automation guardrails — do your auto-scalers and remediation bots have rate limits and circuit breakers?
Calculate your per-minute downtime cost — make the business case for observability investment concrete
Schedule a real failover test — untested failover is no failover

The Q1 2026 data proves that we're building more capable networks that are simultaneously more fragile. We spent 20 years building redundancy. Now we need coordination.

Originally published at FirstPassLab. For more deep dives on network engineering and infrastructure resilience, check out firstpasslab.com.

This article was adapted from original research with AI assistance. The data, sources, and technical analysis have been verified against the cited references.

Why Multi-AZ Failed: Lessons from the First Kinetic Attack on a Major Cloud Region

FirstPassLab — Thu, 02 Apr 2026 16:52:57 +0000

When Iranian drones struck AWS data centers in the UAE and Bahrain on March 1, 2026, they didn't just destroy server racks — they invalidated the multi-AZ assumptions that most cloud architectures are built on. AWS responded by waiving all March charges for ME-CENTRAL-1 and ME-SOUTH-1, an unprecedented move. Here's what engineers need to understand about what failed and how to design around it.

TL;DR: Multi-AZ is not a disaster recovery plan when the threat is geopolitical. Only tested, cross-region failover with active data replication protects against the physical destruction of an entire cloud region.

What Actually Happened

Shahed 136 drones struck two AWS data center facilities, causing structural damage, power grid disruption, and water damage from fire suppression systems. The AWS Service Health Dashboard confirmed:

ME-CENTRAL-1 (UAE): 2 of 3 AZs impaired (mec1-az2, mec1-az3)
ME-SOUTH-1 (Bahrain): 1 of 3 AZs lost power entirely (mes1-az2)
84+ services offline including EC2, S3, DynamoDB, Lambda, RDS, and the Management Console

Regional customers — Careem, Alaan, Tabby, and banking services — went down immediately (CNBC, 2026).

Impact Detail	ME-CENTRAL-1 (UAE)	ME-SOUTH-1 (Bahrain)
AZs Impaired	2 of 3	1 of 3
Services Affected	84+	60+
Power Status (Late March)	Partially restored	Still restoring mes1-az2
Customer Migration	Active to unaffected regions	Active to unaffected regions

The Billing Waiver Created a Second Problem

AWS waived all March usage charges — unprecedented. But the waiver also removed Cost and Usage Report (CUR) data from billing dashboards. As Cory Quinn pointed out, for most enterprises the CUR isn't just an invoice — it's the authoritative record of what infrastructure exists. Compliance teams, auditors, and FinOps teams all build on it.

AWS later clarified the data wasn't deleted, just filtered from standard reports. But the lesson is clear: your billing data is also your infrastructure inventory. If your DR playbook doesn't account for billing data availability during a region-wide failure, you have an audit gap.

Why Multi-AZ Didn't Protect Workloads

This is the critical engineering lesson. Multi-AZ distributes across physically separate data centers within a single region, but all AZs sit within the same metro area and share the same geopolitical threat envelope.

Three assumptions that broke:

1. Multi-AZ ≠ Multi-Region

AZs within ME-CENTRAL-1 are ~50-100 km apart. A coordinated strike targeting a metropolitan area reaches multiple AZs. Engineers running workloads across all three AZs still experienced degradation because the surviving AZ couldn't absorb full regional load.

2. Control Plane Failures Cascade

Even where data plane instances survived (mec1-az1), the control plane was disrupted. Customers couldn't launch new instances, modify security groups, or execute failover automation. If your DR runbook requires API calls to the impaired region's control plane, your failover is dead on arrival.

3. Shared Dependencies Are Invisible

Services running in healthy AZs had hidden dependencies on impaired zones — internal load balancers, DNS resolution, IAM authentication endpoints. These cross-AZ dependencies aren't documented in customer-facing architecture diagrams.

Architecture Pattern	Protects Against	Does NOT Protect Against
Multi-AZ (same region)	Single AZ failure, hardware failure	Regional disaster, military strike
Multi-Region (active-passive)	Full region outage	Data lag during failover, control plane dependency
Multi-Region (active-active)	All above + zero RPO failover	Complexity, cost, global routing challenges
Multi-Cloud	Single provider failure	Doubled operational complexity

A Practical Redesign Framework

Here's how to rethink your architecture:

Tier 1: Region Risk Assessment. Before deploying to any region, evaluate the sovereign risk profile. Map regions against active conflict zones, not just latency numbers. AWS operates in the UAE, Bahrain, and is investing $5.3B in Saudi Arabia. Each region has a different threat model.

Tier 2: Cross-Region Data Replication. Implement async or sync replication to a geographically and politically distant region. S3 Cross-Region Replication, DynamoDB Global Tables, Aurora Global Database. RPO under 1 minute requires active-active with Global Accelerator routing.

Tier 3: Tested Failover. "Untested failover is no failover." Schedule quarterly game days where you actually cut traffic from one region. Organizations that never tested ME-CENTRAL-1 failover discovered missing encryption keys, expired credentials, and incomplete replication during the crisis.

Tier 4: Decouple Data Residency from Compute. If regulations require data in a specific country, architect so compute/serving can operate from a different region while maintaining data locality compliance.

Industry Impact

This was the first confirmed kinetic attack destroying a major cloud provider's infrastructure. Israel reportedly struck a Tehran data center on March 11 (Jerusalem Post), confirming both sides view digital infrastructure as strategic targets.

Counterintuitively, Amazon's stock rallied ~3% — investors betting the incident accelerates cloud spending on resilience. Oracle's Middle East regions experienced zero incidents during the same period, validating the multi-cloud argument for critical workloads.

Compared to Previous Outages

Outage	Cause	Duration	Regions	Services Down
AWS us-east-1 (Dec 2021)	Scaling bug	~10 hours	1	20+
AWS ME-CENTRAL-1 (Mar 2026)	Drone strikes	Weeks	2	84+
Azure (Jan 2023)	WAN routing misconfig	~5 hours	Multiple	15+
Google Cloud (Apr 2023)	Paris region power failure	~12 hours	1	10+

Physical infrastructure cannot be rebooted. It must be rebuilt. Organizations with infrastructure defined in Terraform or Ansible redeployed in hours. Those relying on ClickOps are still migrating a month later.

Five Things to Do Right Now

Audit region dependencies: aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId,Placement.AvailabilityZone]'
Verify cross-region replication — check actual RPO/RTO metrics, not just config
Schedule a real failover test within 30 days
Review your CUR data pipeline for gaps during crisis scenarios
Document a geopolitical risk matrix for every region where you run workloads

The cloud is not an abstraction. It's concrete, steel, and cooling systems sitting on land that exists inside a geopolitical reality. Engineers who build truly resilient multi-region architectures will define the next decade of enterprise cloud design.

Originally published at firstpasslab.com. More deep dives on cloud networking, infrastructure security, and network architecture at FirstPassLab.

AI Disclosure: This article was adapted from original research with AI assistance for editing and formatting. All technical claims are sourced and linked. The original article contains full source citations.

Stop Accepting BGP Routes on Trust Alone: Deploy RPKI ROV on IOS-XE and IOS XR Today

FirstPassLab — Wed, 01 Apr 2026 22:54:32 +0000

If you run BGP in production and you're not validating route origins with RPKI, you're accepting every prefix announcement on trust alone. That's the equivalent of letting anyone walk into your data center and plug into a switch because they said they work there.

BGP RPKI Route Origin Validation (ROV) is the mechanism that changes this. With 500K+ ROAs published globally, mature validator software, and RFC 9774 formally deprecating AS_SET, there's no technical barrier left. Here's how to deploy it on Cisco IOS-XE and IOS XR.

How RPKI ROV Actually Works

RPKI (Resource Public Key Infrastructure) cryptographically binds IP prefixes to the autonomous systems authorized to originate them. Three components make it work:

Route Origin Authorizations (ROAs) — Signed objects published by prefix holders in RPKI repositories. A ROA states: "AS 65001 is authorized to originate 192.0.2.0/24 with a maximum prefix length of /24."

RPKI Validators — Servers (Routinator, Fort, OctoRPKI) that download ROA data from the five RIR trust anchors, validate cryptographic signatures, and build a validated cache.

RPKI-to-Router Protocol (RTR) — RFC 8210 protocol that transports the validated cache to your BGP routers. Each prefix gets tagged:

Valid: Prefix and origin AS match a ROA
Invalid: A ROA exists but origin AS or prefix length doesn't match
NotFound: No ROA published for this prefix

⚠️ Critical distinction: NotFound ≠ Invalid. ~60% of the global table is still NotFound. Dropping those would black-hole most of the internet. The safe policy: drop Invalid, accept everything else, prefer Valid.

Connecting to RPKI Validators

IOS-XE Configuration

router bgp 65001
 bgp rpki server tcp 10.0.0.50 port 8282 refresh 300
 bgp rpki server tcp 10.0.0.51 port 8282 refresh 300

IOS XR Configuration

router bgp 65001
 rpki server 10.0.0.50
  transport tcp port 8282
  refresh-time 300
 !
 rpki server 10.0.0.51
  transport tcp port 8282
  refresh-time 300
 !

The refresh 300 sets a 300-second poll interval as a safety net. Incremental updates (Serial Notify) arrive between polls.

🔑 Always deploy at least two validators from different implementations (e.g., Routinator + Fort). If your only validator goes down and the RTR session expires, the router flushes its cache and silently disables ROV. You won't even know it happened.

Applying ROV Policy — Start Soft

Without a route-map acting on validation state, the router knows which routes are Invalid but does nothing about it. Here's where enforcement happens.

Phase 1: Tag and Deprioritize (Weeks 1-4)

route-map RPKI-POLICY permit 10
 match rpki invalid
 set community no-export additive
 set local-preference 50
!
route-map RPKI-POLICY permit 20
 match rpki valid
 set local-preference 200
!
route-map RPKI-POLICY permit 30
 match rpki not-found
 set local-preference 100
!
router bgp 65001
 address-family ipv4 unicast
  neighbor 203.0.113.1 route-map RPKI-POLICY in

This does three things:

Invalid routes → local-preference 50 + no-export community (least preferred, not propagated)
Valid routes → local-preference 200 (strongly preferred)
NotFound routes → local-preference 100 (functional, but not as trusted)

Why not hard-drop Invalid immediately? Some Invalid states come from stale or misconfigured ROAs by other operators. You need to observe before you enforce.

Phase 2: Hard Enforcement (After Monitoring)

route-map RPKI-POLICY deny 10
 match rpki invalid
!
route-map RPKI-POLICY permit 20
 match rpki valid
 set local-preference 200
!
route-map RPKI-POLICY permit 30
 match rpki not-found
 set local-preference 100

Now Invalid routes are silently dropped at the edge. Route hijacks don't propagate into your network.

Verification and Troubleshooting

Check Validator Connectivity

Router# show bgp rpki servers

BGP RPKI Server:
  Server Address: 10.0.0.50
  Server Port: 8282
  Server State: established
  Serial Number: 47
  Record Count: 482631

Key checks:

Server State must be established — if connecting or down, check reachability and validator health
Record Count should be 400K-500K+ — if 0, the validator isn't syncing

Check a Specific Prefix

Router# show bgp ipv4 unicast 1.0.0.0/24

BGP routing table entry for 1.0.0.0/24
  RPKI validation state: valid
  Origin AS: 13335
  ROA: 1.0.0.0/24, maxlen /24, origin-as 13335

Monitor Invalid Routes

Router# show bgp ipv4 unicast rpki invalid

This is your daily monitoring command during initial deployment. Cross-reference Invalid prefixes against the ROA database to separate genuine hijacks from ROA misconfigurations.

Common Issues

Problem	Cause	Fix
Validator session flapping	MTU issues on RTR path	Check path MTU; large cache responses trigger fragmentation
High record count but no Valid routes	Route-map not applied to neighbor	Verify `neighbor X route-map RPKI-POLICY in`
Prefix shows NotFound when you expect Valid	ROA not published or expired	Check ROA status in RPKI repositories directly
Validator shows 0 records	Not syncing with RIR trust anchors	Verify validator config syncs with all 5 RIRs

💡 Set up syslog/SNMP traps for RPKI session state changes. A validator failure should be a P1 alert, not something you discover during the next change window.

Beyond ROV: What's Coming

RPKI ROV validates the origin AS only. It can't detect route leaks where a legitimate AS improperly announces a prefix it received from a peer.

ASPA (Autonomous System Provider Authorization) is the emerging standard that encodes customer-provider AS relationships, enabling AS path validation. NIST is building test tools (the BRIO framework), and early adoption is expected in 2026-2027.

RFC 9774 deprecated AS_SET, which simplifies ROV — every route now has a single unambiguous origin AS. If you're still generating AS_SET in aggregate routes, update your aggregate-address configs now. Compliant peers will increasingly reject those routes.

Key Takeaways

RPKI ROV is production-ready — 500K+ ROAs, mature validators, no excuses
Start soft — tag Invalid with low local-pref for 2-4 weeks, then hard deny
Two validators minimum — single validator failure = silent ROV death
ROV is layer one — complement with ASPA and BGP session auth (TCP-AO) as they mature
Fix your aggregation — RFC 9774 killed AS_SET; update your configs
Alert on validator failures — treat RTR session drops as P1 incidents

BGP security is no longer optional for any network participating in the global routing system. RPKI ROV gives you a concrete, deployable mechanism to verify route origins today. Configure it, monitor it, enforce it.

Originally published at FirstPassLab. For more protocol deep dives and lab guides, visit firstpasslab.com.

AI Disclosure: This article was adapted with AI assistance. All technical content, configurations, and recommendations are based on documented Cisco IOS-XE/XR behavior, RFC specifications, and industry best practices. Human-reviewed for accuracy.

Ditch the vPC Peer-Link: EVPN ESI Multi-Homing on Nexus 9000 with NX-OS 10.6.x (Config + Verification)

FirstPassLab — Wed, 01 Apr 2026 16:56:45 +0000

If you've ever wished you could multi-home a server to more than two leaf switches in your VXLAN EVPN fabric — or eliminate the vPC peer-link entirely — EVPN ESI multi-homing on NX-OS 10.6.x is the answer. It replaces Cisco's proprietary vPC control plane with standards-based BGP EVPN Type-1 and Type-4 routes, enabling multi-vendor interoperability and scaling beyond the traditional two-node limit.

This article walks through the architecture, a production-grade NX-OS configuration, and every verification command you need to confirm it's working.

Why ESI Multi-Homing Matters

vPC has served data center engineers well for over a decade. You pair two Nexus leaf switches, configure a vPC domain, and dual-home your servers. It works — but with well-known limitations:

Two-node limit: vPC is strictly a two-switch technology. You cannot multi-home a server to three or four leaf switches.
Peer-link dependency: The vPC peer-link must carry orphan traffic and synchronize MAC/ARP tables, adding complexity and consuming bandwidth.
Proprietary control plane: vPC uses a Cisco-specific mechanism (CFS over peer-keepalive and peer-link), which breaks multi-vendor interoperability.

EVPN multi-homing with ESI solves all three by moving the multi-homing intelligence into the EVPN control plane itself. Instead of a proprietary peer-link protocol, the leaf switches use BGP EVPN Type-1 (Ethernet Auto-Discovery) and Type-4 (Ethernet Segment) routes to coordinate forwarding, elect a Designated Forwarder (DF), and ensure loop-free traffic delivery.

Note: ESI multi-homing and vPC can coexist in the same NX-OS 10.6.x fabric. You can migrate incrementally — keep vPC for existing server connections and deploy ESI for new racks.

Core Concepts

Before diving into configuration, understand four key mechanisms:

Ethernet Segment Identifier (ESI)

An ESI is a 10-byte identifier that uniquely represents a multi-homed link bundle (e.g., a LAG connecting a server to two or more leaf switches). Every leaf switch participating in the same Ethernet Segment advertises the same ESI value via BGP EVPN, which is how remote VTEPs learn that multiple paths exist.

NX-OS 10.6.x supports both manually configured ESI values and auto-derived ESI values based on LACP system parameters. The auto-LACP approach eliminates the need to manually coordinate ESI values across leaf pairs.

EVPN Route Types 1 and 4

Type-1 (Ethernet Auto-Discovery per ES): Advertised by each leaf in the Ethernet Segment. Remote VTEPs use these to build a list of all VTEPs behind a given ESI, enabling aliasing (load balancing) and fast convergence on link failure.
Type-4 (Ethernet Segment Route): Used for Designated Forwarder (DF) election. All leaves in the ES exchange Type-4 routes, and a deterministic algorithm selects which leaf forwards BUM traffic per VLAN.

Designated Forwarder Election

When a broadcast or multicast frame arrives at the fabric, only one leaf per Ethernet Segment should forward it to the locally connected host. The DF election process ensures exactly one forwarder per VLAN per ES. NX-OS 10.6.x supports preference-based DF election.

Split Horizon and Aliasing

Split horizon prevents BUM traffic received from an ES member from being forwarded back to the same ES.
Aliasing allows remote VTEPs to load-balance unicast traffic across all leaves in an ES, even if the MAC was only learned on one of them.

Configuration: ESI Multi-Homing on NX-OS 10.6.x

We assume the underlay (OSPF or eBGP), NVE interface, and BGP EVPN overlay are already in place.

Step 1: Enable EVPN ESI Multi-Homing

! Enable EVPN ESI multi-homing globally
nv overlay evpn
feature nv overlay
feature bgp
feature interface-vlan
feature vn-segment-vlan-based

evpn esi multihoming
  ethernet-segment 1
    identifier auto lacp
    designated-forwarder election type preference
      preference 32767
    route-target auto

! Associate the Ethernet Segment with a port-channel
interface port-channel10
  description ESI-to-Server-Rack1
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 100,200
  ethernet-segment 1
  no shutdown

This tells NX-OS to automatically derive the ESI from the LACP system ID and port-channel key. The preference 32767 sets this leaf as the preferred DF. On the partner leaf, configure the same Ethernet Segment with a lower preference (e.g., preference 16384).

Step 2: VXLAN Fabric Integration

Map VLANs to VNIs and configure anycast gateways:

vlan 100
  vn-segment 10100
vlan 200
  vn-segment 10200

fabric forwarding anycast-gateway-mac 0001.0001.0001

interface Vlan100
  no shutdown
  vrf member TENANT-1
  ip address 10.100.0.1/24
  fabric forwarding mode anycast-gateway

interface Vlan200
  no shutdown
  vrf member TENANT-1
  ip address 10.200.0.1/24
  fabric forwarding mode anycast-gateway

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10100
    ingress-replication protocol bgp
  member vni 10200
    ingress-replication protocol bgp
  member vni 50001 associate-vrf

Step 3: BGP EVPN Overlay for ESI Routes

The BGP EVPN session to spine route reflectors carries Type-1 and Type-4 routes. Verify that extended communities are enabled:

router bgp 65001
  router-id 10.255.0.1
  neighbor 10.255.0.100
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
      send-community both
  neighbor 10.255.0.101
    remote-as 65001
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
      send-community both

evpn
  vni 10100 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10200 l2
    rd auto
    route-target import auto
    route-target export auto

Verification and Troubleshooting

Verify Ethernet Segment Status

Leaf-1# show evpn esi
ESI: 0000.0000.0001.0001.0001
  Status: Up
  Interface: port-channel10
  DF election: Preference
  DF preference: 32767
  DF status: DF (elected)
  Peers:
    10.255.0.2 (preference 16384) - Non-DF

Confirms Leaf-1 is elected as the Designated Forwarder. The partner leaf at 10.255.0.2 shows as Non-DF.

Verify BGP EVPN Type-1 and Type-4 Routes

Leaf-1# show bgp l2vpn evpn route-type 1
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.255.0.1:3
*>l[1]:[0000.0000.0001.0001.0001]:[0]/120
      10.255.0.1           100      0 i
*>i[1]:[0000.0000.0001.0001.0001]:[0]/120
      10.255.0.2           100      0 i

Leaf-1# show bgp l2vpn evpn route-type 4
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.255.0.1:3
*>l[4]:[0000.0000.0001.0001.0001]:[10.255.0.1]/184
      10.255.0.1           100      0 i
*>i[4]:[0000.0000.0001.0001.0001]:[10.255.0.2]/184
      10.255.0.2           100      0 i

Both local (*>l) and remote (*>i) Type-1 and Type-4 routes should appear with matching ESI values.

Verify Aliasing on Remote Leaves

Remote-Leaf# show l2route evpn mac all | include 10100
Topology ID  Mac Address    Prod  Next Hop(s)
10100        aabb.cc00.0100 BGP   10.255.0.1, 10.255.0.2

Two next-hop addresses confirms aliasing is working — unicast traffic gets load-balanced across both ESI member leaves.

Common Troubleshooting Scenarios

DF election not converging:

Check that both leaves have the same ESI configured (show evpn esi)
Verify Type-4 routes are being exchanged (show bgp l2vpn evpn route-type 4)
Confirm LACP is operational on both ends (show port-channel summary)

Duplicate BUM frames on the host:

DF election has failed and both leaves are forwarding BUM traffic
Verify designated-forwarder election type preference is configured consistently
Check for Type-4 route filtering on the spine route reflectors

MAC flapping on remote leaves:

Usually caused by an ESI mismatch — one leaf has the ESI configured while the other does not
Verify with show evpn esi on both leaves and ensure ESI values are identical

ESI vs. vPC: When to Use Each

Criteria	vPC	ESI Multi-Homing
Max leaf switches per group	2	2+ (standards allow more)
Control plane	Proprietary (CFS)	BGP EVPN (RFC 7432)
Multi-vendor support	No	Yes
Peer-link required	Yes	No
Maturity on NX-OS	10+ years	NX-OS 10.6.x (new)
Incremental migration	N/A	Can coexist with vPC

For greenfield deployments on NX-OS 10.6.x, ESI multi-homing is the forward-looking choice. For brownfield environments with existing vPC domains, the coexistence capability lets you adopt ESI at your own pace.

Key Takeaways

EVPN ESI multi-homing replaces the proprietary vPC peer-link with standards-based BGP EVPN Type-1 and Type-4 routes, enabling multi-vendor interop and scaling beyond two-node pairs.
NX-OS 10.6.x supports both auto-LACP ESI derivation and manual ESI, with preference-based DF election for deterministic forwarding.
Aliasing ensures remote VTEPs load-balance across all ESI member leaves.
Coexistence with vPC makes incremental migration practical.
Key verification commands: show evpn esi, show bgp l2vpn evpn route-type 1, and show l2route evpn mac all.

Originally published at firstpasslab.com.

This article was adapted from original content with AI assistance. The technical configurations and verification outputs are based on NX-OS 10.6.x documentation and lab testing.

Volt Typhoon Weaponized SOHO Routers at Scale — Here's Your Zero-Trust Playbook for the Remote Edge

FirstPassLab — Tue, 31 Mar 2026 22:54:04 +0000

The FCC just banned all new foreign-made consumer routers from US sale (effective March 23, 2026), but here's what most coverage misses: the ban doesn't fix the actual problem. Millions of unpatched SOHO routers already deployed in your remote workers' homes are the real attack surface — and three Chinese state-sponsored campaigns (Volt Typhoon, Flax Typhoon, Salt Typhoon) have been weaponizing them for years.

This post breaks down the technical reality behind the ban, why it might actually increase the US attack surface, and — most importantly — a concrete zero-trust playbook for removing the home router from your enterprise trust chain entirely.

What the FCC Actually Banned

The FCC's Public Safety Bureau issued DA 26-278 on March 20, 2026. The order adds every consumer-grade router manufactured outside the US to the FCC's Covered List. New models can't get the FCC ID required for legal sale.

Date	Action
March 23, 2026	FCC ceases all new equipment authorizations for covered foreign-made routers
September 2026	Retailers prohibited from importing new inventory of covered devices
March 2027	Maintenance Waiver expires — security patches from covered jurisdictions require federal audit

The ban does not affect: routers already purchased, previously authorized models, or enterprise/carrier-grade equipment.

Here's the supply chain math that matters: China and Taiwan manufacture 60–75% of all consumer routers globally. The US produces ~10%. Supply disruption isn't hypothetical — it's arithmetic.

The Typhoon Campaigns: How SOHO Routers Became Attack Infrastructure

The FCC explicitly cited three Chinese state-sponsored campaigns as justification. Each exploited SOHO routers differently:

Campaign	Technique	Enterprise Impact
Volt Typhoon	Hijacked end-of-life SOHO routers as proxy infrastructure; targeted power grids, water systems	VPN tunnels from compromised home routers provided direct pivot into enterprise networks
Flax Typhoon	Built Raptor Train botnet from compromised IoT/SOHO devices	Mass credential harvesting through residential IP addresses
Salt Typhoon	Embedded in telecom networks using compromised routers as persistent footholds	Long-term access to communications infrastructure; lateral movement across operator networks
CovertNetwork-1658	Password spraying via thousands of compromised SOHO routers	Evasive attack infrastructure rotating residential IPs to bypass detection

The CISA/NSA Joint Advisory documented that US-based processor architectures were involved in over 90% of the compromises. Vendors like Cisco, Juniper, Netgear, and Fortinet were all exploited. Geographic origin was secondary to the actual attack vector: unpatched firmware, default credentials, and exposed management interfaces.

The Paradox: This Ban Might Increase Your Attack Surface

Here's the part that should concern every security engineer. Analysis from the Internet Governance Project at Georgia Tech argues that banning the newest, most secure Wi-Fi 7/8 routers from dominant manufacturers forces consumers to either pay substantially more for US-made alternatives or — more likely — keep their older, more vulnerable devices longer.

Compare the security posture across router generations:

Feature	Modern Wi-Fi 7	Wi-Fi 6	Legacy Wi-Fi 5 and older
Encryption	WPA3 mandatory	WPA3 supported	WPA2 only (KRACK-vulnerable)
Firmware Updates	Active auto-updates	Active with manual check	End-of-life — no patches
Hardware Security	Secure Boot + TPM	Firmware signing	Minimal or none
Management Exposure	Cloud-managed, no open ports	Mixed	Often exposes UPnP, Telnet, HTTP admin

The enterprise takeaway: regardless of what the FCC does about new hardware, your security posture cannot depend on the home router. Treat every remote edge as hostile.

Zero-Trust Playbook: Remove the Home Router from Your Trust Chain

1. Deploy ISE Posture Assessment for All Remote Access

Evaluate the endpoint before granting network access — not the router. Configure posture policies that check OS patch level, endpoint protection status, disk encryption, and host-based firewall state.

# Authorization Policy (simplified)
Rule: Remote_VPN_Posture
  Condition: Network Device Group == VPNs AND Posture_Status == NonCompliant
  Result: Redirect to Client Provisioning Portal (ACL: POSTURE_REDIRECT)

Rule: Remote_VPN_Compliant
  Condition: Network Device Group == VPNs AND Posture_Status == Compliant
  Result: PermitAccess (dACL: FULL_ACCESS)

Posture decisions are binary: compliant or non-compliant. Non-compliant endpoints get remediation instructions, not network access. This removes the SOHO router from the trust equation entirely.

2. Migrate from Traditional VPN to ZTNA

Traditional site-to-site and remote-access VPN architectures implicitly trust the network path, including the home router. ZTNA flips the model: authenticate the user and device per-session, directly to the application, with no reliance on the underlying network.

Architecture	Trust Model	Home Router Dependency
Traditional RA-VPN	Trusts the tunnel endpoint (includes home network path)	High — router compromise can intercept or manipulate tunnel
Split-tunnel VPN	Trusts partial path; internet traffic exits locally	Medium — local traffic is fully exposed
ZTNA	Zero trust — per-session, per-app authentication	None — connection is user-to-app, router is irrelevant

3. Enforce SWG and DNS Security on Every Endpoint

Even with ZTNA, remote endpoints still generate DNS queries and web traffic that traverse the home router. Deploy a Secure Web Gateway and DNS-layer security (like Cisco Umbrella or Cloudflare Gateway) on every managed endpoint:

DNS queries route to secure resolvers regardless of DHCP-assigned DNS from the home router
Web traffic inspection occurs at the cloud proxy, not the SOHO device
Intelligent proxy decrypts and inspects suspicious HTTPS connections

4. Segment Remote Access with Micro-Zones

Don't grant flat network access to VPN users. Use Security Group Tags (SGTs) or dynamic ACLs to segment remote workers into micro-zones based on role, device posture, and application requirements. A compromised remote endpoint should never have Layer 3 reachability to your DC management plane.

5. Monitor for Residential IP Anomalies

The CovertNetwork-1658 campaign used thousands of compromised residential IPs for password spraying. Your SOC should:

Flag authentication attempts from residential ISP ranges that don't match known employee locations
Correlate VPN login geolocation with HR employee records
Alert on unexpected residential IP blocks, especially from broadband providers in regions where you have no employees

The March 2027 Firmware Cliff

The FCC's Maintenance Waiver expires in March 2027. After that date, security patches for foreign-made legacy devices originating from covered jurisdictions may require a secondary federal audit. Millions of currently-deployed routers could effectively become permanently unpatched.

For security teams, this creates a hard deadline:

Accelerate ZTNA migration — remove the home router from the trust chain before the firmware cliff hits
Deploy managed CPE — issue corporate-managed access points or routers to critical remote workers
Enforce endpoint-only security — ensure every security function (firewall, DNS, VPN, posture) runs on the managed endpoint, not the SOHO device

Supply Chain Reality Check

Vendor	Manufacturing Base	Ban Impact
TP-Link	China (Shenzhen)	Directly affected — no new consumer model authorizations
Netgear	Contract manufacturing in China, Vietnam	Affected unless production shifts
Linksys	China, Vietnam	Affected for China-manufactured models
Starlink	Texas, USA	Exempt — manufactured domestically
Juniper/HPE	Flextronics (China, Canada, Mexico)	Partially affected; pursuing Conditional Approval

As Greyhound Research chief analyst Sanchit Vir Gogia put it: "This is about control, not just compromise. Routers sit at the network edge, but functionally they are part of the control plane of the enterprise."

TL;DR

The FCC banned all new foreign-made consumer routers (March 23, 2026)
The Typhoon campaigns weaponized SOHO routers to infiltrate US critical infrastructure at scale
The ban might actually make things worse by slowing router upgrade cycles
Your fix isn't a different router — it's removing the router from your trust chain entirely
Deploy ISE posture + ZTNA + SWG + micro-segmentation + residential IP monitoring
March 2027 firmware cliff makes this urgent

Originally published on FirstPassLab. More deep dives on network security architecture at firstpasslab.com.

AI Disclosure: This article was adapted from original research with AI assistance for formatting and style optimization. All technical content, data points, and cited sources have been verified against their original publications.

Equinix's 280-DC AI Fabric Just Changed the DCI Game — Here's the Architecture Under the Hood

FirstPassLab — Tue, 31 Mar 2026 16:55:05 +0000

Equinix launched the Distributed AI Hub on March 11, 2026 — spanning 280 data centers across 77 markets. Powered by Fabric Intelligence, it automates connectivity, routing, and security policy enforcement for distributed AI workloads. For network engineers, this is the clearest signal yet that manual DCI provisioning is being replaced by intent-based orchestration.

Let's break down the architecture.

The Three-Component Stack

Component	Function	Network Impact
AI-Ready Backbone	High-bandwidth transport fabric	400 Gbps physical ports, 100 Gbps virtual connections
Fabric Intelligence	Software-defined orchestration	Real-time telemetry, automated routing, policy enforcement
AI Solutions Lab	Architecture validation across 20 locations	Pre-deployment testing for DCI and AI topologies

For engineers working with VXLAN EVPN multi-site DCI, this is a familiar pattern scaled to an unprecedented level. Equinix is abstracting the underlay complexity into a managed service — the engineering challenge shifts from building the fabric to integrating with it.

How Fabric Intelligence Changes DCI Operations

Fabric Intelligence is a software layer on top of Equinix Fabric with real-time awareness, AI-driven automation, and policy enforcement for next-gen AI workloads. Here's what it means in practical terms:

Real-time telemetry and observability. Live telemetry feeds across the entire Fabric mesh. For engineers accustomed to polling SNMP counters or scraping streaming telemetry from individual routers, this is centralized, cross-domain observability across dozens of metro areas — the kind of visibility that traditionally required building a custom network digital twin.

Automated routing and segmentation. Rather than manually configuring BGP peering sessions or adjusting ECMP weights, Fabric Intelligence dynamically adjusts routing based on workload requirements. This is intent-based networking applied to the interconnection layer — define performance and security requirements, the platform handles path selection.

Policy enforcement at scale. Palo Alto Networks Prisma AIRS is embedded directly in the Hub. Security policies enforced at the infrastructure layer from day one. No more bolt-on security where firewall rules lag behind connectivity changes.

According to Equinix CBO Jon Lin, Q4 2025 saw over 4,500 deals closed in a single quarter, with ~60% of the largest deals driven by AI workloads. That volume can't be served by manual provisioning.

The APAC Demand Problem

Asia-Pacific added 1,557 MW of new DC capacity in 2025, bringing the total to 13,763 MW — yet vacancy rates shrank from 12.4% to 10.9% (Cushman & Wakefield, H2 2025). Record construction couldn't keep pace with AI demand.

Metric	Value
Total APAC DC capacity (2025)	13,763 MW
New capacity added	1,557 MW
Vacancy rate	10.9% (down from 12.4%)
Development pipeline	19.37 GW
Bangkok forecast growth (2026-2030)	10.3x
Jakarta forecast growth (2026-2030)	4.4x

Investment highlights: CapitaLand committed US$874M (Singapore + Osaka), Nvidia-backed Reflection AI announced $6.7B in South Korea, AirTrunk secured Japan's largest-ever DC financing at $1.2B.

For network engineers, these emerging Southeast Asian markets mean greenfield DCI designs with less legacy — but less mature peering ecosystems and higher latency challenges.

Architecture Deep Dive: Three Planes

Transport Plane: 400G-Ready Backbone

400 Gbps physical ports with QSFP-DD and OSFP transceiver support, plus 100 Gbps Fabric virtual connections. This is the WAN/metro DCI equivalent of what NVIDIA Spectrum-X does within a single AI cluster.

Control Plane: Fabric Intelligence Orchestration

The orchestration layer integrates with AI workload schedulers and cloud providers. When a Kubernetes cluster in Tokyo needs training data staged in Singapore, Fabric Intelligence handles virtual connection provisioning, QoS attachment, and route optimization — tasks that would traditionally require configuring BGP communities, adjusting DSCP markings, and verifying end-to-end latency manually.

Security Plane: Prisma AIRS at the Edge

Prisma AIRS runs locally on Equinix Network Edge — AI-powered threat detection without backhauling to a centralized security stack. For distributed AI inference where microseconds matter, inline security at the interconnection point eliminates the hairpinning latency penalty.

Skills That Matter Now

Based on the technical requirements in the Equinix architecture and APAC buildout:

400G transport and optics — QSFP-DD, OSFP, ZR/ZR+, FEC options, fiber capacity planning. The market is moving toward 800G coherent for metro DCI.

EVPN-VXLAN multi-site DCI — Even though Equinix abstracts the underlay, enterprises connecting their own fabrics still need EVPN Type-5 routes, multi-site BGW peering, and VNI-to-VRF mappings.

AI traffic engineering — AI inference workloads are bursty and latency-sensitive, often requiring RoCEv2 semantics across DCI links. DSCP marking for AI traffic classes, ECN configuration, PFC tuning.

Multi-cloud overlay architecture — The Hub connects to AWS, Azure, GCP, and dozens of smaller clouds. Designing overlays that span Transit Gateway, Azure vWAN, and GCP NCC alongside Equinix Fabric.

Intent-based networking and automation — Fabric Intelligence is IBN for the DCI layer. Engineers who can write NETCONF/RESTCONF calls for Equinix Fabric or build Terraform modules for multi-cloud overlays will have a distinct advantage.

The Career Shift

Traditional DCI Role	Emerging Distributed AI Role
Manual cross-connect provisioning	API-driven fabric orchestration
Static BGP peering configuration	Intent-based routing automation
Bolt-on firewall insertion	Embedded security policy
Per-link capacity planning	AI workload-aware traffic engineering
Single-metro DCI design	Multi-region, multi-cloud overlay architecture

Continental AG deployed NVIDIA GPU clusters and IBM storage inside Equinix to support ADAS AI workloads — achieving 14x increase in AI experiments. The engineering work was designing the interconnection topology for distributed GPU clusters with consistent latency, not rack-and-stack.

The operating model is shifting from CLI-driven config to API-driven orchestration. The foundational protocols (EVPN, BGP, VXLAN, QoS) don't change — the interface does.

TL;DR

Equinix launched the largest AI-ready DCI fabric: 280 DCs, 77 markets, Fabric Intelligence orchestration
APAC DC vacancy is shrinking despite record construction — AI demand outpaces everything
The architecture has three planes: 400G transport, intent-based control, inline Prisma AIRS security
Skills to prioritize: 400G optics, EVPN-VXLAN DCI, AI traffic engineering, multi-cloud overlays, automation
The career shift: manual provisioning → API-driven orchestration

Originally published at FirstPassLab.

Disclosure: This article was adapted from the original blog post with AI assistance. Technical content has been reviewed for accuracy.

An AI Hacker Beat 99% of Humans in 6 CTF Competitions for $12.92 Each — Here's the Defensive Playbook

FirstPassLab — Mon, 30 Mar 2026 22:55:09 +0000

Tenzai's autonomous AI hacker outperformed 99% of 125,000 human competitors across six elite capture-the-flag hacking competitions in March 2026, completing multi-step exploit chains for an average cost of $12.92 per platform. This isn't a research demo — it's a production-grade offensive AI system built by Israeli intelligence veterans with $75 million in seed funding and a $330 million valuation, and it fundamentally changes the threat model that every security engineer must defend against.

Key Takeaway: AI-driven offensive security has crossed the threshold from theoretical to operational — autonomous agents can now chain multiple exploits, bypass authentication, and escalate privileges faster and cheaper than most human penetration testers, making zero trust microsegmentation and AI-driven behavioral analytics mandatory rather than aspirational.

What Exactly Did Tenzai's AI Hacker Accomplish?

Tenzai's autonomous hacking agent competed across six major CTF platforms — websec.fr, dreamhack.io, websec.co.il, hack.arrrg.de, pwnable.tw, and Lakera's Agent Breaker — achieving top 1% rankings on every single one. According to CEO Pavel Gurvich, the agent outperformed more than 125,000 human security researchers, completing challenges that span web application hacking, binary exploitation, and AI prompt injection attacks. The total cost across all six platforms was under $5,000, with individual competition runs averaging $12.92 and completing in approximately two hours each.

What makes this different from previous AI security milestones is the complexity of exploit chaining. In one documented Dreamhack challenge (difficulty 8/10, only 17 human solvers, no public writeups), Tenzai's agent independently:

Discovered a Server-Side Request Forgery (SSRF) vulnerability
Identified a prototype pollution weakness in the class-transformer library
Escalated privileges to administrator
Chained all three attacks together to achieve Remote Code Execution against a Redis instance via CVE-2022-0543

According to Tenzai's engineering blog, the agent managed state across attack paths, tracked leads, and coordinated sub-agents for technical exploitation — behaviors that previously required experienced human pentesters.

Metric	Tenzai AI (2026)	Typical Human CTF Player
CTF ranking	Top 1% across 6 platforms	Varies widely (median ~50th percentile)
Average cost per competition	$12.92	$0 (human time not counted)
Average completion time	~2 hours	Days to weeks
Exploit chaining capability	Autonomous multi-step chains	Requires significant experience
Vulnerability classes covered	Web, binary, AI/prompt injection	Usually specialized in 1-2 areas

This follows a pattern. In 2025, XBOW became the first AI to reach #1 on HackerOne's leaderboard by finding real-world vulnerabilities. Anthropic's Claude ranked in the top 3% of a Carnegie Mellon student CTF. But Tenzai's achievement represents a step change: elite-level performance across multiple platforms simultaneously, against professional researchers rather than students.

Why Traditional Network Defenses Cannot Keep Pace

Static perimeter defenses — signature-based IPS rules, manually maintained ACLs, and periodic vulnerability scanning — operate on human timescales. According to Knostic CEO Gadi Evron, the time from vulnerability discovery to working exploit has collapsed from days or weeks to hours with AI assistance.

Traditional firewall rule sets that assume known attack patterns become fundamentally inadequate when the attacker adapts in real time. A firewall running static ACL entries or an IPS relying solely on Snort signature updates faces an adversary that can generate novel exploit chains faster than signature databases refresh.

The core problem is deterministic versus adaptive. A signature-based IPS catches known patterns — specific byte sequences, known CVE exploitation attempts, documented protocol anomalies. An AI attacker operates probabilistically, testing variations, mutating payloads, and chaining exploits that individually might pass signature inspection.

Consider the SSRF-to-RCE chain Tenzai demonstrated: each individual step — a crafted HTTP request, a prototype pollution via JSON parsing, a Redis command injection — might not trigger any single IPS signature. The attack's power lies in the combination, and that combination is now automated.

The Economics Make This Worse

The cost barrier that once limited advanced offensive capabilities to nation-states has evaporated. Tenzai's entire six-competition run cost under $5,000. Pavel Gurvich warned that this capability is "rapidly getting out of the realm of nations and military intelligence organizations and into the hands of college kids who may have very different incentives." When a sophisticated multi-step exploit chain costs $12.92 to execute, every network becomes worth probing.

What Defensive Architecture Actually Works Against Autonomous AI Attacks?

Defending against AI-driven offensive tools requires three architectural layers operating simultaneously:

Zero trust microsegmentation to limit blast radius
AI-driven behavioral analytics for real-time detection
Continuous automated red teaming to find vulnerabilities first

According to SecurityWeek's Cyber Insights 2026 report, "zero trust will be less about conceptual frameworks and more about operational architecture, especially within the LAN."

Layer 1: Zero Trust Microsegmentation

Zero trust microsegmentation assumes every network segment is compromised and enforces identity-based access at the workload level. Using SGT-based segmentation (e.g., Cisco ISE + TrustSec, or equivalent vendor solutions), you can enforce policies where a compromised web server in VLAN 100 cannot reach the database tier in VLAN 200 even if the attacker has valid Layer 3 connectivity.

The critical configuration involves Security Group Tags (SGTs) assigned dynamically via 802.1X or MAB authentication, with enforcement via SGACL on access-layer switches or inline SGT tagging on data center platforms.

In a traditional flat network, Tenzai's AI could chain SSRF into lateral movement across subnets in minutes. With SGT enforcement, each lateral movement attempt hits an identity-based policy check that the AI must independently compromise — multiplying the attack complexity exponentially.

Layer 2: AI-Driven Behavioral Analytics in the SOC

Signature-based detection fails against novel exploit chains. Behavioral analytics platforms — Cisco Secure Network Analytics, Vectra AI, Darktrace — establish baseline traffic patterns and flag statistical anomalies. According to IBM research, AI-powered security reduces Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR) by correlating events across network flows, endpoint telemetry, and identity systems simultaneously.

For network engineers, this means exporting NetFlow/IPFIX from your infrastructure to analytics platforms isn't optional anymore. When an AI agent generates anomalous DNS queries during SSRF exploitation or initiates unusual east-west traffic patterns during lateral movement, behavioral analytics catches what signatures miss.

Layer 3: Continuous Automated Red Teaming

The defensive equivalent of AI offensive tools is continuous automated penetration testing. Rather than annual pentests that produce stale results, organizations deploy AI-driven red team agents that continuously probe their own infrastructure. The industry is shifting from "scan and patch" to "agentic red teaming" — AI agents that reason about attack paths, chain vulnerabilities, and test defenses 24/7.

Practical takeaway: if you're not testing your access policies and firewall rules with automated tools at least monthly, an AI attacker will find the gaps you missed.

What the Industry Experts Are Saying

According to Gadi Evron, cofounder and CEO of AI security company Knostic, hackers have already had their "singularity moment." Evron told Forbes: "Tenzai now showing how their agents win at 99% of six CTFs shows a maturity of the capability in the market, even though the proliferation of such capabilities to pretty much everybody is already there, and growing."

The startup ecosystem confirms the trend:

Company	Funding	Focus	Relevance
Tenzai	$75M seed, $330M valuation	Autonomous offensive AI	Demonstrates AI attack capability
Native	$42M	Multi-cloud security policy	Automated defense across cloud providers
XBOW	Undisclosed	AI bug bounty hunting	#1 on HackerOne leaderboard (2025)
Knostic	Undisclosed	AI security posture	Threat intelligence and AI risk assessment

Practical Defensive Checklist

Implement these regardless of your vendor stack. Each item directly counters a capability that AI offensive tools like Tenzai have demonstrated:

Deploy microsegmentation at Layer 2/3: Configure identity-based security groups on all access-layer switches. Enforce group-based ACL policies between security zones. Verify enforcement is active.
Enable behavioral analytics: Export Flexible NetFlow from all L3 infrastructure to your analytics platform. Baseline normal east-west traffic. Alert on deviations exceeding 2 standard deviations.
Implement encrypted traffic analytics: Enable ETA (or equivalent) to detect malicious patterns within encrypted flows without decryption.
Automate red team testing: Deploy continuous penetration testing tools against your own infrastructure. Run automated scans against access policies and firewall rule sets monthly at minimum.
Reduce MTTD with AI-driven SOC tools: Integrate firewall and IPS event data with your SIEM. Configure correlation rules that detect multi-step attack chains, not individual events.
Segment management planes: Isolate network management interfaces (SSH, SNMP, RESTCONF) into dedicated VRFs with ACLs that restrict access to jump hosts only.

FAQ

Can AI really hack better than humans?

Yes, with caveats. Tenzai's AI ranked in the top 1% across six CTF platforms, outperforming 125,000 human competitors. CEO Pavel Gurvich notes "there is still a small group of exceptional hackers who outperform current AI systems." The gap is closing rapidly.

How much does it cost to run an AI hacking agent?

$12.92 per competition on average, under $5,000 total across all six platforms. This makes advanced offensive capabilities affordable far beyond nation-state actors.

What defensive strategies work against AI-powered attacks?

Three layers: zero trust microsegmentation to limit lateral movement, AI-driven behavioral analytics for real-time anomaly detection, and continuous automated red teaming to find vulnerabilities before attackers do.

How fast can AI exploit a vulnerability compared to humans?

The time from vulnerability discovery to working exploit has collapsed from days/weeks to hours with AI assistance. Tenzai's agent completed entire multi-step exploit chains in under two hours on average.

Originally published at firstpasslab.com.

AI Disclosure: This article was adapted from original research with AI assistance for editing, formatting, and Dev.to optimization. All technical analysis, data, and source citations are from the original article.

One Antenna, 16 Beams: How Metamaterial RF Lenses Are Replacing Hundreds of Wi-Fi APs in Stadiums

FirstPassLab — Mon, 30 Mar 2026 16:57:34 +0000

What if you could replace 500–1,500 access points in a stadium with 30–60 antenna units — and get better performance? That's exactly what MatSing's lens antenna technology is doing in production deployments right now.

MatSing's new MS-16.16W45 WiFi 6E lens antenna generates 16 independent beams with 4x4 MIMO from a single mount point, covering thousands of simultaneous users in the 5.125–7.125 GHz band. Unveiled at MWC Barcelona in March 2026, this technology uses metamaterial refraction — not reflection or electronic phase shifting — to fundamentally change how we approach high-density wireless design.

For any network architect dealing with stadium, arena, or large campus deployments, this represents the most significant antenna innovation in three decades.

How Does the Lens Antenna Actually Work?

MatSing's antenna operates on RF refraction — the same principle as a telescope refracting light through a convex lens. The patented metamaterial lens is engineered from composite materials with precisely tuned dielectric properties that bend radio waves at controlled angles, directing energy into distinct sectorized beams.

"Our lens antenna operates much like an eye does — receiving and sending signals from multiple directions through a single lens" — Leo Matytsine, EVP and co-founder of MatSing (RCR Wireless News)

Here's why this matters compared to existing antenna tech:

Parabolic dishes reflect signals off a curved surface → one beam per reflector
Phased arrays use electronic phase shifters → hardware self-interferes at high beam density
Lens antennas pass RF energy through the lens → dozens of independent feeds from one unit, no self-interference

The practical result: where a traditional high-density deployment needs 200–500 distributed APs, MatSing achieves equivalent or superior coverage from 2–3 centralized lens positions. Each lens handles multiple frequency bands simultaneously — Sub-6 GHz (LTE/5G), C-Band, and WiFi 6E — without separate antenna systems per band.

Metamaterial Construction and Beam Formation

The lens is constructed from layered metamaterials — engineered composites where the internal structure (not the chemical composition) determines electromagnetic behavior. MatSing's materials achieve a gradient refractive index across the lens surface, focusing RF energy entering at different angles into separate, tightly controlled beams.

Each beam maintains physical isolation from adjacent beams. In traditional deployments, co-channel interference (CCI) between closely spaced APs is the primary capacity limiter. With lens-generated beams, isolation comes from physics, not software-based mitigation.

MS-16.16W45 WiFi 6E Specifications

Feature	MS-16.16W45 Specification
Frequency Band	5.125–7.125 GHz (WiFi 6E full band)
Independent Beams	16
MIMO Configuration	4x4 per beam
Coverage Model	Centralized, single mount point
Target Environment	Stadiums, arenas, high-density venues
Multi-Band Support	Yes (Sub-6, C-Band, WiFi 6E)

16 beams × 4x4 MIMO = theoretical throughput of up to 4.8 Gbps per beam on 160 MHz channels, or 76.8 Gbps aggregate from a single antenna unit.

Compare that to a traditional setup: 16+ Cisco Catalyst 9136 APs, each with its own mounting hardware, cabling, and PoE switch port, plus hours of RF tuning to manage inter-AP interference.

Production Deployments: The Numbers

Allegiant Stadium, Las Vegas

60 multibeam lens antennas provide multi-band, multi-carrier connectivity for over 65,000 fans. All three major US carriers share the same physical antenna infrastructure as a neutral-host DAS.

"With just 16 MatSing multibeam lens antennas we were able to cover the field and stands for C-Band for the carrier" — Steve Dutto, DGP President (AFL Wireless)

Coachella Music Festival

100,000+ attendees in a single square mile — traditional cellular connectivity consistently failed under the strain of simultaneous social media uploads. MatSing provided 96 sectors from a single installation point, reaching devices up to 240 feet away.

Venue	Antennas	Capacity	Key Metric
Allegiant Stadium	60	65,000+ fans	All 3 carriers, neutral host
Coachella	Single installation	100,000+ attendees	96 sectors, 240 ft range
Multiple NFL Stadiums	Varies	12,000–100,000	Multi-carrier, multi-band

Head-to-Head: Lens Antenna vs. Traditional High-Density Wi-Fi

Attribute	Traditional Panel APs	MatSing Lens Antenna
Antennas per venue	500–1,500	30–60
Mounting locations	Hundreds	2–10 centralized
Co-channel interference	High (complex RF tuning)	Low (physically isolated beams)
Multi-carrier support	Separate systems per carrier	Neutral host, 1–5 carriers per lens
Cable runs	500+	30–60
Maintenance	Distributed troubleshooting	Centralized access
Band support	Single-band per AP model	Multi-band simultaneous

The operational savings are just as compelling as the capital ones: firmware updates and hardware replacements hit 30–60 centralized units instead of hundreds of under-seat installations. That's a 70–80% reduction in annual maintenance labor hours.

What About Cisco's Approach?

Cisco's response has been hyper-directional antennas with Catalyst 9136 and 9166 APs — "top-down" placement from overhangs and under seating decks with highly directional radiation patterns. It works, but still requires hundreds of individual APs and all the associated infrastructure. MatSing represents a fundamentally different philosophy: fewer, more capable antenna positions vs. many distributed points of presence.

When Should You Consider Lens Antenna Architecture?

Lens antennas deliver the strongest ROI in environments with three characteristics:

Ultra-high user density — 5,000+ simultaneous connections
Limited mounting infrastructure — historic venues, open-air festivals
Multi-carrier requirements — neutral-host cellular + Wi-Fi

For a typical corporate campus or office building, traditional AP deployments remain more practical and cost-effective. The crossover point appears to be around 10,000–15,000 users in a defined venue footprint.

Where Lens Antennas Fall Short

Cost per unit — premium pricing, justified only at scale
Location services — fewer triangulation points than distributed APs (reduced BLE-based RTLS accuracy)
Multi-floor offices — heavy wall attenuation favors per-floor AP placement
Best in open spaces — stadium bowls, festival grounds, convention halls with line-of-sight

Wi-Fi 7 Forward Compatibility

The lens architecture is inherently frequency-agnostic — the physics of refraction work across bands. MatSing can extend the platform to Wi-Fi 7 (802.11be) by engineering feeds for 320 MHz channels and 6 GHz upper band extensions.

For architects planning 3–5 year infrastructure investments, a single physical lens installation can be upgraded with new feed modules as standards evolve — avoiding the forklift replacement cycle that traditional AP deployments face every 4–5 years.

Key Takeaways

MatSing's metamaterial lens uses RF refraction to generate 16+ independent beams from a single unit
Allegiant Stadium replaced hundreds of traditional antennas with 60 lens units serving 65,000+ fans
Physical beam isolation eliminates the co-channel interference problem that plagues dense AP deployments
The technology is best suited for venues with 10,000+ users; traditional APs remain better for offices and campuses
Wi-Fi 7 compatibility is built into the physics — future upgrades are feed module swaps, not forklift replacements

This article was adapted from the original deep dive on FirstPassLab. For more wireless architecture and networking content, check us out there.

AI Disclosure: This article was adapted and edited with AI assistance. All technical content is based on cited sources and verified deployment data. The original research, analysis, and editorial direction are human-driven.

From BIG-IP to Kubernetes: What F5's CNCF Gold Membership Signals for Network Engineers

FirstPassLab — Sun, 29 Mar 2026 22:53:17 +0000

F5 upgraded to Gold Membership in the Cloud Native Computing Foundation (CNCF) on March 26, 2026, during KubeCon + CloudNativeCon Europe in Amsterdam. If you've spent years configuring BIG-IP virtual servers, iRules, and health monitors, this move directly affects your career trajectory — F5 is betting its future on Kubernetes-native networking, and your existing skills transfer more than you'd expect.

TL;DR: F5's CNCF Gold Membership accelerates the convergence of traditional application delivery controllers with Kubernetes-native networking. Gateway API, OpenTelemetry, and service mesh skills are now essential for anyone working in infrastructure.

Why This Matters: The Numbers

According to the CNCF 2025 Annual Cloud Native Survey:

98% of organizations have adopted cloud native technologies
82% run Kubernetes in production (up from 66% in 2023)
NGINX powers roughly 34% of all web servers globally

When the company behind your BIG-IP fleet and the world's most-deployed reverse proxy doubles down on Kubernetes, the signal is clear.

"Expanding to Gold Membership in the CNCF reflects our dedication to fostering innovation and collaboration in the cloud native ecosystem," said Kunal Anand, Chief Product Officer at F5.

The ingress-nginx Retirement Changes Everything

In November 2025, the Kubernetes project announced the retirement of the community-maintained ingress-nginx controller, citing maintenance challenges and security concerns. This creates a clear opening for F5's commercial NGINX Ingress Controller.

Here's how they compare:

Feature	Community ingress-nginx (Retired)	F5 NGINX Ingress Controller
Configuration	Annotations (`nginx.ingress.kubernetes.io/`)	CRDs (`VirtualServer`, `Policy`)
Protocol Support	HTTP/HTTPS primarily	HTTP, gRPC, TCP, UDP
WAF Integration	Limited	NGINX App Protect built-in
Gateway API	Partial support	Full Gateway API conformance
Commercial Support	Community only	F5 enterprise support

If your organization runs ingress-nginx today, migration planning is now a near-term operational requirement.

BIG-IP Next for Kubernetes: Your ADC Skills in Container Form

BIG-IP Next for Kubernetes extends F5's traditional ADC capabilities into container environments — a single control point for ingress, egress, security, and visibility. The key difference from traditional BIG-IP is the deployment context: these functions now run as Kubernetes-native workloads, managed through Kubernetes APIs rather than TMSH or the BIG-IP GUI.

Your understanding of virtual servers, pools, iRules, and health monitors translates directly. The orchestration layer changes from CLI/GUI to Kubernetes manifests and Helm charts.

Gateway API: The Networking Abstraction You'll Actually Like

The Kubernetes Gateway API replaces the legacy Ingress resource with a role-based, protocol-flexible model. F5 is a key contributor, and their Gold Membership deepens that influence.

Concept	Legacy Ingress	Gateway API
Role Separation	Single resource, single owner	GatewayClass → Gateway → Route (multi-role)
Protocol Support	HTTP/HTTPS only	HTTP, TCP, UDP, gRPC, TLS passthrough
Cross-Namespace	Not supported	Built-in `ReferenceGrant` mechanism
Extensibility	Annotations (vendor-specific)	Policy attachment model (standardized)
Status Reporting	Minimal	Rich status conditions per resource

The role separation mirrors how networking teams already work:

GatewayClass = infrastructure provider (analogous to provisioning a BIG-IP)
Gateway = cluster operator (analogous to creating a VIP)
HTTPRoute/TCPRoute/GRPCRoute = application developer (analogous to pool members and virtual servers)

If you've ever configured virtual servers, VIPs, and routing policies on traditional load balancers, Gateway API provides a familiar mental model wrapped in Kubernetes-native semantics.

AI Inference: Why This Is a Networking Problem

"Inference relies on scalable infrastructure, which is a fundamentally cloud native challenge enabled by CNCF technologies," said Jonathan Bryce, Executive Director of CNCF.

AI inference endpoints require:

Low-latency load balancing — distributing requests across GPU-backed pods with health-aware routing
Protocol flexibility — gRPC for model serving (TensorFlow Serving, Triton), HTTP/2 for API gateways
Observability — OpenTelemetry traces and metrics across the inference pipeline
Security — mTLS between services, WAF at ingress, rate limiting per client

Every item on that list maps to skills network engineers already possess. The platform changed from Cisco IOS to Kubernetes. The engineering principles didn't.

CNCF Membership Tiers: A Quick Reference

Tier	Key Benefits	Notable Members
Silver	Community access, event discounts, project participation	Startups, regional SIs
Gold	Deeper project collaboration, enhanced visibility	F5, Viettel
Platinum	Governing Board seat, voting rights, strategic reviews	Google, AWS, Microsoft, Red Hat, Cisco

Tracking which vendors hold Platinum and Gold positions reveals where the industry is investing.

What You Should Do This Week

Deploy a K8s lab with NGINX Ingress Controller — Install K3s or kind, deploy F5 NGINX IC, configure VirtualServer CRDs. This is the hands-on equivalent of configuring virtual servers on BIG-IP.
Study the Gateway API spec — Read the official docs and implement GatewayClass, Gateway, and HTTPRoute resources.
Instrument with OpenTelemetry — Deploy an OTel Collector and export traces/metrics from NGINX. This builds the observability muscle that AI inference environments demand.
Track CNCF project graduates — Monitor Sandbox → Incubating → Graduated transitions. These predict which technologies become enterprise defaults within 12-24 months.

The convergence of traditional ADC vendors with Kubernetes-native networking isn't a future trend — it's happening in production today.

FAQ

What does F5's CNCF Gold Membership mean for network engineers?
Deeper investment in Kubernetes-native tools like NGINX Ingress Controller and Gateway API. Expect tighter integration between traditional ADC capabilities and cloud native infrastructure.

Is Kubernetes knowledge required for network engineering roles?
Increasingly, yes. With 82% of production containers on Kubernetes and AI inference driving demand, understanding CNI plugins, service mesh, and Kubernetes networking is becoming essential.

What is the Kubernetes Gateway API?
The next-generation routing standard replacing legacy Ingress. Role-based configuration, cross-namespace routing, and multi-protocol flexibility. F5 is a key contributor.

Originally published at firstpasslab.com.

AI Disclosure: This article was adapted from original research with AI assistance. All technical facts, statistics, and quotes were verified against their cited sources. The adaptation for Dev.to was AI-assisted.