DEV Community

Tommy Brunn
Tommy Brunn

Posted on

Tailscale VPN Exit Node with Split-Tunneling using WireGuard and Namespaces

#security #privacy #networking #linux

Like all other mid-thirties nerds, I run a janky homelab out of my living room. In addition to being my daily driver for gaming and coding, my trusty desktop computer also serves as a media server and content acquisition system.

In a previous iteration, I had been using Tailscale with Mullvad as a VPN provider. This was extremely easy to set up and worked pretty great, but it came with two downsides:

  1. All traffic had to be routed through a Mullvad exit node. The internet is already a challenging place to access thanks to Cloudflare inundating anyone using Firefox and Linux with captchas, but if you use a VPN exit node it gets even worse.
  2. Part of my subscription cost goes to funding the far-right ethno-fascist party Örebropartiet, via Mullvad co-founder Daniel Berntsson.

I took this as an opportunity to improve my setup and get rid of Mullvad.

Goals

  1. Expose services I have running on my desktop in my Tailnet, with MagicDNS working.
  2. QBittorrent exits via VPN at all times.
  3. Any device in my Tailnet can exit via VPN, if I allow it. This includes the desktop that also acts as the exit node.

The overall solution that I am aiming for will look something like this:

          tailnet (MagicDNS, DERP relays)
                    ▲                              ▲
                    │ tailscale0                   │ tailscale-exit
                    │ (root ns)                    │ (ivpn ns)
┌───────────────────┴──────────────────────────────┼────────────────────┐
│   desktop            tailscaled #1               │                    │
│                                                  │ ← passes through   │
│   eth0 ── internet                               │                    │
│   veth-tsr 10.99.0.1/30 ────────────────┐        │                    │
└─────────────────────────────────────────┼────────┼────────────────────┘
                                          │        │
┌─────────────────────────────────────────┼────────┼────────────────────┐
│ ivpn namespace                          │        │                    │
│   ivpn                 tailscaled #2 ───┘        │                    │
│     (--advertise-exit-node)                      │                    │
│   qBittorrent-nox                                │                    │
│   veth-tsn 10.99.0.2/30                          │                    │
│   vpn-interface ── WireGuard ────────────────────┴──▶ IVPN           │
└───────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

This article serves in large part to remind myself how this was actually set up, in case I have to do it again sometime, but perhaps it can be useful also to others.

VPN namespace

Before involving Tailscale, where things start getting a bit complicated, I first followed Split tunneling using Wireguard and namespaces to set up just the VPN namespace and running Qbittorrent-nox inside that namespace.

The linked article walks through all the steps. Rather than using NordVPN, I chose to use IVPN. They have a similar focus on privacy as Mullvad does, and while I honestly can't say the service is as good as Mullvad's, they get a pass for not funding far-right parties in my country.

At the end of that article, what I ended up with was a Wireguard namespace called "ivpn" which was connected to an IVPN endpoint. Qbittorrent-nox, which is running as a SystemD service, is set up to use that namespace via NetworkNamespacePath=/var/run/netns/ivpn.

At this point, goal number 2 is achieved. All traffic from Qbittorrent now goes through the VPN. If I want to funnel some other program through the VPN, I can do so, like sudo ip netns exec ivpn <command>, but it's obviously not very ergonomic. Let's use Tailscale to solve this.

Tailscale in the root namespace

Setting up my desktop to be a regular Tailscale client is no problem. The tailscale package in Arch already comes with a systemd service definition, so it's just a matter of enabling it with sudo systemctl enable --now tailscaled.service.

Tailscale will automatically update /etc/resolv.conf to point to the Tailscale resolver, so MagicDNS will just work out of the box. I think I may have run tailscale up once manually first, in order to authenticate.

At this point, goal 1 is achieved. Other devices on my Tailnet can now access services I expose on my desktop. This is really nothing special - it's just a standard Tailscale setup.

Tailscale VPN exit node

Here's where things get dicey. In Tailscale, a device can advertise itself as an exit node, allowing other clients to exit through it. The normal way to do this would be to spin up a cloud VM somewhere and configure it to be a Tailscale exit node. But I already have this perfectly good namespace right here on my desktop where all the traffic gets routed through my VPN.

So the plan is to run a second instance of tailscaled inside the VPN namespace. My physical machine will end up showing up as two different devices in Tailscale - one is the root namespace where I am just a normal Tailscale client, and one is the ivpn namespace which is going to be a Tailscale exit node. In theory, I should be able to select my ivpn device as an exit node from my desktop, and route all the traffic from my desktop via the exit node and to IVPN, just the same as if it was two physically separate devices.

Let's try:

# /etc/systemd/system/tailscale-exit.service
[Unit]
Description=Tailscale exit node in the ivpn namespace
Documentation=https://tailscale.com/docs/
After=wg-netns@ivpn.service
Required=wg-netns@ivpn.service
BindsTo=wg-netns@ivpn.service
PartOf=wg-netns@ivpn.service

[Service]
Type=exec

# Tailscaled rewrites /etc/resolv.conf through the bind-mount, clobbering the
# upstream nameserver in /etc/netns/ivpn/resolv.conf, so we have to add that back
# on startup.
ExecStartPre=/usr/bin/mkdir -p /etc/netns/ivpn
ExecStartPre=/usr/bin/bash -c 'echo "nameserver <VPN_DNS_RESOLVER>" > /etc/netns/ivpn/resolv.conf'

ExecStart=/usr/bin/ip netns exec ivpn /usr/sbin/tailscaled \
        --state=/var/lib/tailscale-exit.state \
        --socket=/run/tailscale-exit.sock \
        --tun=tailscale-exit \
        --port=41641

ExecStartPost=/usr/bin/bash -c 'for i in {1..50}; do [ -S /run/tailscale-exit.sock ] && exit 0; sleep 0.2; done; exit 1'

ExecStartPost=/usr/bin/ip netns exec ivpn /usr/bin/tailscale \
        --socket=/run/tailscale-exit.sock up \
        --accept-dns=true \
        --advertise-exit-node \
        --hostname=%H-ivpn \
        --reset

ExecStopPost=/usr/bin/ip netns exec ivpn /usr/sbin/tailscaled \
        --socket=/run/tailscale-exit.sock --cleanup

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

If you wanted to have multiple exit nodes available, for example in different countries supported by your VPN provider, you should template the file above instead of hardcoding "ivpn".

This defines the service tailscale-exit.service, and as you can see we run tailscale in the ivpn network namespace instead of the root.

Starting this with sudo systemctl enable --now tailscale-exit.service makes a new device $(hostname)-ivpn show up in the Tailscale device list. Before it becomes available as an exit node, you have to allow it in the Tailscale admin panel.

Using this as an exit node from a different device, such as a phone, now works. You can test it out by selecting the exit node and then querying something like ipinfo.io and seeing that the public IP is that of the VPN provider.

At startup, tailscaled in the vpn namespace will likely complain about a few optimizations we can make. I didn't notice any difference, but following Tailscale's documentation, I added this override to set some parameters:

# /etc/systemd/system/wg-netns@ivpn.service.d/override.conf
[Service]
# https://tailscale.com/docs/features/exit-nodes?tab=linux#advertise-a-device-as-an-exit-node
ExecStartPost=/usr/bin/ip netns exec %i bash -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
ExecStartPost=/usr/bin/ip netns exec %i bash -c 'echo 1 > /proc/sys/net/ipv4/conf/all/forwarding'
ExecStartPost=/usr/bin/ip netns exec %i bash -c 'echo 1 > /proc/sys/net/ipv6/conf/all/forwarding'

# GRO offload quirk on the WG interface
# https://tailscale.com/docs/reference/best-practices/performance#linux-optimizations-for-subnet-routers-and-exit-nodes
ExecStartPost=-/usr/bin/ip netns exec %i bash -c '/usr/bin/ethtool -K <vpn-interface> rx-udp-gro-forwarding on rx-gro-list off 2>/dev/null || true'
Enter fullscreen mode Exit fullscreen mode

Same host client

Selecting my new exit node from the root tailscale client on the same machine, tailscale up --exit-node=$(hostname)-ivpn --accept-dns=true --exit-node-allow-lan-access=true, nothing works.

$ nslookup google.com
;; communications error to 100.100.100.100#53: timed out
Enter fullscreen mode Exit fullscreen mode

In fact, it manages to also kill the exit node, which goes offline shortly after selecting it. This is quite impressive if you think about it, as that's running in a different network namespace. Nothing the root daemon does to root-namespace routing should be able to kill connectivity inside the vpn namespace, unless the namespace's traffic actually transits the root routing table - which it does!

When the root daemon selects the exit node, it installs 0.0.0.0/0 dev tailscale0 in its routing table:

$ ip route show table 52
default dev tailscale0
<tailnet ips> dev tailscale0
throw 127.0.0.0/8
throw 172.17.0.0/16 
throw 172.18.0.0/16
throw 192.168.49.0/24
throw 192.168.50.0/24
throw 192.168.122.0/24
Enter fullscreen mode Exit fullscreen mode

Tailscale exempts its own sockets from that table using fwmark 0x80000, which is why a normal node can still reach DERP while using an exit node. But the kernel WireGuard module's packets don't carry that mark. They get matched by the Tailscale table's default route and shoved into the tunnel, addressed to the exit node, which lives inside the namespace whose underlay you just broke.

We can confirm this by running:

ip route get <VPN_ENDPOINT_IP>
Enter fullscreen mode Exit fullscreen mode

If this returns dev tailscale0 instead of the physical interface, that's the bug.

It's a chicken-and-egg problem: the IVPN tunnel's underlay is being routed through the tunnel that depends on the IVPN tunnel.

We can solve this the same way that Tailscale does, by marking the underlay to keep it out of Tailscale's routing table. We could piggyback on their existing routing rule by just tagging the packets from our vpn interface with the same mark:

ip netns exec ivpn wg set <vpn-interface> fwmark 0x80000
Enter fullscreen mode Exit fullscreen mode

But that's quite fragile. If they change their mark, this breaks. Instead, we can define our own routing rule and mark.

Instead of executing this manually or from some systemd unit, we can have wg-netns do it for us when it sets up the wireguard namespace, as per the original split-tunneling article. All that's needed is to add the mark to the interface in the wireguard config file:

# /etc/wireguard/ivpn.yaml
name: ivpn
base_netns: null
managed: true
dns-server: [<vpn-dns-server>]
interfaces:
- name: <vpn-interface-name>
  fwmark: 0x1000 # Adding mark
  mtu: 1420
  address:
  - <vpn-address>
  private-key: <vpn-private-key>
  peers:
  - public-key: <vpn-public-key>
    endpoint: <vpn-peer-endpoint>
    allowed-ips:
    - 0.0.0.0/0
    - ::/0
Enter fullscreen mode Exit fullscreen mode

Then we add a the routing rules in our wg-netns override (sudo systemctl edit wg-netns@ivpn.service), to make sure that those packets directly use the main routing table instead of Tailscale's:

# wg-netns@ivpn.service.d/override.conf
[Service]
# Exclude the ivpn packets (marked by 4096/0x1000 in ivpn.yaml) from Tailscale routing
ExecStartPost=/usr/bin/ip rule add fwmark 4096/4096 lookup main priority 5000
ExecStartPost=/usr/bin/ip rule add fwmark 4096/4096 lookup default priority 5001
ExecStartPost=/usr/bin/ip rule add fwmark 4096/4096 unreachable priority 5002

ExecStopPost=-/usr/bin/ip rule del priority 5000
ExecStopPost=-/usr/bin/ip rule del priority 5001
ExecStopPost=-/usr/bin/ip rule del priority 5002
Enter fullscreen mode Exit fullscreen mode

After restarting wg-netns@ivpn.service, the exit node now works. We can confirm this by making a request through the vpn namespace:

sudo ip netns exec ivpn curl -s https://ipinfo.io

# We can also see that it's being routed to the physical interface,
# though we have to mark it manually
ip route get <VPN_ENDPOINT_IP> mark 0x1000
Enter fullscreen mode Exit fullscreen mode

Real path between the two daemons

With the way things are currently configured, every byte goes ISP → Tailscale DERP → back to my machine → IVPN → out. This is extremely inefficient, considering all I want to do is shove a byte from my machine to another namespace on the same machine and then to IVPN.

To resolve this, we have to give a data path between the two namespaces, using a veth pair:

# wg-netns@ivpn.service.d/override.conf

# This file also contains the directives enabling IP forwarding,
# the GRO tweak, and the routing rules. I'm omitting for brevity.

[Service]
# Clean up direct link in case it already exists
ExecStartPre=-/usr/bin/ip link del veth-tsr

# Set up direct veth link to avoid relaying
ExecStartPost=/usr/bin/ip link add veth-tsr type veth peer name veth-tsn
ExecStartPost=/usr/bin/ip link set veth-tsn netns %i
ExecStartPost=/usr/bin/ip addr add 10.99.0.1/30 dev veth-tsr
ExecStartPost=/usr/bin/ip link set veth-tsr up
ExecStartPost=/usr/bin/ip netns exec %i ip addr add 10.99.0.2/30 dev veth-tsn
ExecStartPost=/usr/bin/ip netns exec %i ip link set veth-tsn up

# Bypass Tailscale's route table
ExecStartPost=/usr/bin/ip rule add to 10.99.0.0/30 lookup main priority 5003
# Remove rule
ExecStopPost=/usr/bin/ip rule del priority 5003

# Remove link
ExecStopPost=/usr/bin/ip link del veth-tsr
Enter fullscreen mode Exit fullscreen mode

The namespace daemon enumerates 10.99.0.2 as a local endpoint and advertises it. The root daemon's marked sockets look up main, find the connected /30, and go direct. After restarting everything, tailscale status should no longer show relay (<somewhere>) and tailscale ping $(hostname)-ivpn should take 0ms.

End result

At this point, all three goals are achieved:

  1. My desktop root namespace acts as a regular Tailscale client, which can expose any services running on it to the rest of the tailnet.
  2. Qbittorrent-nox runs in the vpn namespace, so all traffic exits via IVPN.
  3. A second tailscale daemon runs in the same vpn namespace, advertising itself as an exit node, which can be used either by other devices in my tailnet or by the tailscale daemon running in the root namespace on the same physical machine.

Unlike my previous setup, this does not depend on me to remember to enable Tailscale before starting Qbittorrent, and I can choose whether I want to exit regular traffic over the VPN or just directly via my ISP. MagicDNS just works, so I can resolve other devices on my tailnet and vice-versa. And not a single SEK goes to political parties spreading hatred in my country.

I cobbled this together, so there's probably improvements that can be made. It feels a bit fragile to be so dependent on details of how Tailscale sets up routing, and I'm not 100% sure of the ordering of my systemd units, but overall I still think this is a much better option than what I had earlier.

Because the root Tailscale daemon just runs as a regular Tailscale client, and all the complexity is in the exit-node, I can just manage my Tailscale connection using whatever Tailscale GUI I want. For me, that's this gnome extension that integrates with the quick settings menu.

Top comments (0)