DEV Community

manuelerastus
manuelerastus

Posted on

Your FortiGate static route is perfect. SD-WAN is ignoring it.

#networking #fortinet #vpn #troubleshooting

I lost the better part of three days to one FortiOS behaviour while connecting two sites over IPsec. It bit me four times: three different destination subnets, then once more in the return direction. Each time the firewall policy, the phase 2 selectors and the static route all looked correct, and traffic still vanished or leaked out to the internet.

This post is the explanation I wish I'd had, and the diagnostic order that finds it in minutes. Addresses and interface names are changed.

The setup

Two sites, each behind a FortiGate, joined by route-based IPsec tunnels. Site A has two internet links bundled into SD-WAN. Someone had added an SD-WAN rule to load-balance internet traffic across both:

config system sdwan
    config service
        edit 1
            set name "wan_load_balance"
            set mode load-balance
            set dst "all"
            set src "all"
            ...
        next
    end
end
Enter fullscreen mode Exit fullscreen mode

Then I added a new remote subnet, 10.2.0.0/24, behind the tunnel to-siteB:

config router static
    edit 0
        set dst 10.2.0.0 255.255.255.0
        set device "to-siteB"
    next
end
Enter fullscreen mode Exit fullscreen mode

Firewall policy allowing LAN to the tunnel: done. Phase 2 selectors covering both subnets: done. Tunnel up: yes.

From a real host on the LAN, nothing reached Site B.

Why the static route loses

FortiOS doesn't go straight to the routing table. Roughly, the forwarding decision checks, in order:

  1. Policy routes (config router policy)
  2. SD-WAN service rules
  3. The routing table (FIB)

SD-WAN service rules are policy routes under the hood. A rule matching src all / dst all therefore claims every forwarded packet before the routing table is consulted. However specific your static route is, it never gets a vote.

The packet goes out of an SD-WAN member link, source-NATed to that link's public IP, towards a private address on the internet. It doesn't arrive anywhere, and nothing logs a "deny", because by its own rules nothing was denied.

Note that the built-in implicit SD-WAN rule follows the routing table. The trap is an explicit catch-all rule, which many deployments add for load balancing.

Why it looked fine at first

My first test was a ping from the FortiGate itself:

execute ping-options source 10.1.0.1
execute ping 10.2.0.10
Enter fullscreen mode Exit fullscreen mode

It worked. Self-originated traffic isn't forwarded through the same policy-route path as traffic arriving from a LAN host, so it takes the static route and goes into the tunnel.

Lesson one: test with real traffic from a real host. A ping from the firewall proves the tunnel works, not that forwarding works.

The fix, part one: outbound

Add a policy route ahead of the SD-WAN rules for the new destination:

config router policy
    edit 0
        set input-device "lan"
        set src "10.1.0.0/255.255.255.0"
        set dst "10.2.0.0/255.255.255.0"
        set output-device "to-siteB"
    next
end
Enter fullscreen mode Exit fullscreen mode

Traffic from the LAN now entered the tunnel. The remote side saw the packets arrive and replied.

Replies still never reached the LAN host.

The fix, part two: the return path

This is the one that cost the most time. A reply comes out of the tunnel, gets decrypted, and then needs a second routing decision to reach the directly connected LAN. That decision goes through the same order, and the catch-all SD-WAN rule claims it too.

The symptoms looked like a hardware fault:

  • Tunnel counters showed decrypted packets increasing.
  • The sniffer showed nothing leaving the LAN interface.
  • No session, no log entry.

diagnose debug flow finally showed what was happening:

Match policy routing id=...: to 10.1.0.25 via ifindex-<wan member>
find a route: gw-<isp gateway> via <wan member>
...
policy-0 is matched, act-drop
Denied by forward policy check (policy 0)
Enter fullscreen mode Exit fullscreen mode

Policy 0 is FortiOS's implicit deny. The packet only reached it because routing had already chosen the WAN link for a reply meant for the LAN, and no firewall policy allows tunnel-to-WAN.

The fix is the mirror image:

config router policy
    edit 0
        set input-device "to-siteB"
        set src "10.2.0.0/255.255.255.0"
        set dst "10.1.0.0/255.255.255.0"
        set output-device "lan"
    next
end
Enter fullscreen mode Exit fullscreen mode

Lesson two: every new tunnel destination needs a policy route in both directions. The return direction is not inherited.

The diagnostic order that finds it fast

  1. Use a real host. Generate traffic from a LAN machine, not the firewall.
  2. Sniff it. 
   diagnose sniffer packet any "host 10.2.0.10" 4 30
Enter fullscreen mode Exit fullscreen mode

If you see it leaving a WAN member with a NATed source instead of the tunnel, SD-WAN has it.

  1. Check which policy routes are being hit. 
   diagnose firewall proute list
Enter fullscreen mode Exit fullscreen mode

A hit_count of 0 on your new rule means it never matches. Compare it with a working sibling rule.

  1. Clear stale sessions before retesting. Offloaded sessions keep their old path until cleared: 
   diagnose sys session filter dst 10.2.0.10
   diagnose sys session clear
Enter fullscreen mode Exit fullscreen mode
  1. Trace the flow. Still stuck? diagnose debug flow names the exact policy route and gateway chosen. It's easiest to capture from the GUI's debug flow tool, because streaming live debug over a non-interactive SSH session is unreliable.
  2. Trust get over show. In a few edge cases show omitted fields that were set. get inside edit <id> gives the live values.

What didn't help

I tried these before finding the cause. None of them touch the routing decision, so none of them helped:

  • Disabling ASIC/NPU offload on the firewall policy
  • Disabling NPU offload on the phase 1 interface
  • Widening the firewall policy to all / all

If the sniffer shows the packet leaving the wrong interface, the problem is routing. Firewall policy changes won't fix it.

Takeaways

  • An SD-WAN rule matching dst all outranks every static route for forwarded traffic.
  • Add explicit policy routes for internal and tunnel destinations, in both directions.
  • Prove forwarding with traffic from a real host. A firewall-originated ping proves only the tunnel.
  • When a decrypted packet "disappears", trace the second routing decision before blaming the hardware.

Better still, scope the load-balancing rule to internet destinations in the first place, for example with a negated RFC 1918 destination group. Then internal traffic never matches it.

Top comments (0)