Tailscale Kernel TUN in Unprivileged LXC: Direct SSH Without Userspace Networking

tailscale up --tun=userspace-networking gets you a green dot in the admin console and almost nothing else. The node appears in your tailnet, tailscale status looks healthy, and then you try to SSH into that container from your laptop and the connection hangs until TCP gives up. Two lines in the LXC config file fix it, and the container stays unprivileged.

That's the whole post, really. But those two lines only make sense once you understand why every guide pushes you toward userspace mode in the first place, and what you're giving up by staying there.

Who should care

Anyone running services in unprivileged LXC containers on Proxmox who wants those containers to be real tailnet members with their own 100.64.0.0/10 address. Not reachable through something else. Reachable directly, over WireGuard, with a kernel network interface that ip addr can see.

If you're already routing everything through a subnet router, you have a working setup and this is an optional upgrade. I covered that pattern in Tailscale Subnet Routers. Treat this as the next rung on the ladder: instead of one node advertising routes on behalf of everyone else, each container carries its own identity, its own ACL surface, and its own direct path to peers.

What userspace networking actually costs you

Every LXC-and-Tailscale guide I've read lands on the same instruction: pass --tun=userspace-networking and move on. It works because it sidesteps the problem entirely. Rather than asking the kernel for a TUN device, tailscaled runs a userspace TCP/IP stack (gVisor's netstack) inside its own process and never opens /dev/net/tun.

Those costs stay invisible until you trip over one.

Outbound traffic needs a proxy. In userspace mode, tailscaled exposes SOCKS5 and HTTP proxies on a local port. Nothing on the system routes to 100.64.0.0/10 automatically, because there is no interface and no route. Every client has to be told about the proxy:

# userspace mode: this is the only way out
export ALL_PROXY=socks5://localhost:1055/
curl http://100.64.0.20:8080/health

Miss that env var in a systemd unit, a cron job, or a nested container, and the traffic silently takes the normal default route instead. No error. It just goes somewhere else, which is the worst failure mode a network can have.

Inbound to a normal daemon doesn't happen. Your sshd binds 0.0.0.0:22 on the container's LAN interface. Packets arriving over the tailnet terminate inside tailscaled's netstack, and there's no path from netstack to a socket the kernel owns unless you build one explicitly with tailscale serve, or you switch to Tailscale SSH where tailscaled itself is the SSH server. Tailscale SSH genuinely works in userspace mode, which is why plenty of people never notice the limitation. But that's tailscaled's SSH implementation, not OpenSSH. If you care about host key pinning, authorized_keys command restrictions, Match blocks, or an sshd_config you've tuned, you now maintain two parallel SSH stories on the same box.

UDP is a mess. Netstack's UDP support has been partial for years. Concrete symptom: mosh does not work. mosh-server starts fine, prints its port and key, and the client sits there forever because the datagrams never arrive. If you SSH over flaky mobile links and lean on mosh to survive roaming, userspace mode is a dead end.

None of this is a bug. It's a documented tradeoff, and for a container that only needs to call out to a couple of HTTP endpoints, it's a perfectly reasonable one. The problem is that most guides present it as the answer rather than the fallback it actually is.

The false starts

First instinct, recommended constantly on forums: make the container privileged. Set unprivileged: 0, restore, done, TUN works. It also hands the container a root that maps to real host root, which throws away the single most valuable property of an unprivileged container. For a machine that's about to sit on a VPN and accept inbound connections from anywhere, that trade is exactly backwards.

Second instinct: --features nesting=1. That flag shows up in nearly every Proxmox thread about containers doing unusual things, and it's genuinely required for Docker-in-LXC and systemd cgroup delegation. It does nothing for TUN. Nesting controls whether the container can see and mount its own cgroup hierarchy and its own procfs/sysfs views. Device access is a completely separate mechanism. Set it, restart, observe zero change, spend twenty minutes wondering what you got wrong.

AppArmor is the third dead end. There's a pile of advice suggesting lxc.apparmor.profile: unconfined to get device access working. That's a sledgehammer for a problem AppArmor isn't causing. The default lxc-container-default-cgns profile does not block /dev/net/tun usage; it blocks a set of mount, ptrace, and /proc write operations, none of which sit in the path here. Dropping confinement to fix a device permission issue is the kind of change that looks like it worked and quietly widens the blast radius. I've seen this pattern bite people in other contexts too, which is the same theme as the runc sysctl trap: the fix that "works" is usually the one that removed a boundary you wanted.

What's actually stopping you is much narrower than any of that.

The actual fix

Three conditions have to hold: the host has the tun module loaded, the container's device cgroup allows that specific char device, and the device node is bind-mounted into the container's filesystem.

1. Load tun on the host and persist it

On the Proxmox host:

lsmod | grep -w tun || modprobe tun
ls -l /dev/net/tun
# crw-rw-rw- 1 root root 10, 200 Aug 14 09:12 /dev/net/tun

Persist it so a host reboot doesn't quietly undo everything:

echo tun > /etc/modules-load.d/tun.conf

Do this before the next step. The bind mount below has a source path on the host. If /dev/net/tun doesn't exist when the container starts, the mount fails and the container refuses to start, which is a confusing way to learn that a kernel module wasn't loaded.

Repeat this on every node in the cluster. A container that migrates to a host without tun loaded will fail to start there, and you'll be debugging a mount error at the worst possible moment.

2. Add two lines to the container config

Edit /etc/pve/lxc/<CTID>.conf on the host that currently owns the container:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

That's it. No unprivileged: 0, no AppArmor changes, no nesting. Two details matter in the second line. The target path dev/net/tun has no leading slash, because LXC resolves mount targets relative to the container rootfs. And create=file tells LXC to create the target inode if it doesn't already exist, which it won't on a fresh container.

The first line uses cgroup v2 syntax. Anything running Proxmox 8.x is on the unified hierarchy, so lxc.cgroup2.devices.allow is what you want. Older setups on cgroup v1 used lxc.cgroup.devices.allow with the same argument format; if you're copying a snippet from a 2019 forum post, check which one it uses.

3. Stop and start the container from the host

This part catches people. Rebooting from inside the container does not re-read the config file, because the config is consumed by LXC on container creation, not by the init system inside it. You need a full stop/start cycle driven from the host:

pct stop <CTID> && pct start <CTID>

Until you do that, /dev/net/tun simply won't exist inside the container, and you'll assume the config was wrong.

4. Drop the userspace flag and bring Tailscale up

If tailscaled was previously running in userspace mode, that flag lives in /etc/default/tailscaled:

# /etc/default/tailscaled - before
FLAGS="--tun=userspace-networking"

# after
FLAGS=""

Then restart and authenticate:

systemctl restart tailscaled
tailscale up --hostname=app-container-01 --accept-dns=false

I set --accept-dns=false on servers by habit. MagicDNS rewriting /etc/resolv.conf on a box that already has opinions about DNS is a fight you don't need, especially if you're running your own resolver like AdGuard Home. Turn it on deliberately if you want it.

Verifying it actually worked

Four checks, in order. Each one fails differently, which is what makes them useful.

# 1. The device exists and is world-writable
ls -l /dev/net/tun
# crw-rw-rw- 1 nobody nogroup 10, 200 Aug 14 09:12 /dev/net/tun

# 2. A real kernel interface, not a netstack fiction
ip addr show tailscale0
# 3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 ...
#     inet 100.64.0.31/32 scope global tailscale0

# 3. Routes for the tailnet are installed in the container's table
ip route | grep 100.64
# 100.64.0.0/10 dev tailscale0 scope link

# 4. Peer connectivity is direct, not relayed
tailscale ping 100.64.0.20
# pong from db-node (100.64.0.20) via 192.0.2.14:41641 in 1ms

That fourth line is the one worth reading carefully. via <ip>:<port> means a direct WireGuard tunnel between the two hosts. If it says via DERP(sfo) instead, you're relaying through Tailscale's infrastructure, which still works but adds latency and depends on someone else's servers. tailscale netcheck will tell you whether UDP is reachable and whether you're behind a NAT that's blocking direct paths.

Then test the thing you came for:

ssh user@100.64.0.31       # OpenSSH, your config, your keys
mosh user@100.64.0.31      # UDP works now

Why it works

Three separate mechanisms have to cooperate, and the reason so much bad advice exists is that people conflate them.

Device cgroup allowlist. An unprivileged LXC container starts with a deny-by-default device policy and a short allowlist covering the basics: /dev/null, /dev/zero, /dev/full, /dev/random, /dev/urandom, /dev/tty, and the pty devices. The TUN driver registers as char device major 10, minor 200. It's not on that list, so any open() on it returns EPERM regardless of file permissions. Adding c 10:200 rwm grants read, write, and mknod for exactly one device node and nothing else. That's the surgical part: you're not opening a category, you're naming a device.

The bind mount. Passing the cgroup check gives permission to use the device, but the container's /dev is a fresh tmpfs populated by LXC. Creating the node yourself with mknod inside the container fails, because unprivileged containers don't hold CAP_MKNOD in a namespace where it means anything for device creation. Bind-mounting the host's existing node sidesteps that entirely: the inode already exists, LXC just makes it visible at the right path.

Namespaced capability. Opening the device is only half the operation. Actually creating an interface requires the TUNSETIFF ioctl, which needs CAP_NET_ADMIN. Here's the part that makes the whole approach safe: CAP_NET_ADMIN is evaluated against the user namespace that owns the network namespace. The container's root has full CAP_NET_ADMIN over its own netns, and zero authority over the host's. So tailscaled can create tailscale0, set addresses, and install routes inside the container while remaining completely unable to touch host networking. No host privilege is granted at any point.

About that nobody:nogroup

Ownership showing as nobody:nogroup inside the container is expected, not a symptom. The host node is owned by uid 0. An unprivileged container maps its uid 0 to something like host uid 100000, and host uid 0 falls outside the mapped range, so the kernel reports it as the overflow uid (65534) which resolves to nobody. The mode is 0666, so every process in the container can open it anyway. People see nobody:nogroup, assume permissions are broken, and go chase a chown that will never work. Nothing to fix.

Direct paths versus the subnet router model

With a kernel interface, the container installs a route for 100.64.0.0/10 and encrypts packets itself. Traffic to a peer leaves as UDP on tailscaled's port (41641 by default) over the container's normal LAN interface, straight to the peer's endpoint. No hairpinning through a subnet router, no dependency on one container staying healthy, no single node whose reboot takes out remote access to a dozen services.

The ACL story improves too. A subnet router advertises a CIDR, so your policy can only reason about IP ranges. Per-node membership lets you tag containers and write rules against identity:

"acls": [
  {
    "action": "accept",
    "src":    ["tag:admin"],
    "dst":    ["tag:lxc-app:22", "tag:lxc-app:3000"]
  }
]

Tag the container at join time with tailscale up --advertise-tags=tag:lxc-app, using an auth key whose owner is permitted to assign that tag. Now access is a property of the machine, not a property of whatever subnet it happens to sit in. That distinction matters a lot once more than one person needs access, and it's the same principle behind two-tier service accounts for agents: scope by identity, not by network position.

Before you restart anything

Step 3 requires stopping the container, which means killing every process inside it. Worth five minutes of preparation:

• Save your terminal state. If you live in tmux inside that container, prefix + Ctrl-s with tmux-resurrect saves the session layout and running programs. Losing a session with twelve panes of half-finished debugging is a self-inflicted wound.
• Confirm a recent backup. pct listsnapshot <CTID> or a check in Proxmox Backup Server. The config change is easy to revert, but "the container won't start now" is much less stressful with a restore point.
• Know how you'll get back in. pct enter <CTID> from the host works regardless of container networking. If your only access path is SSH over the very network you're changing, fix that first.
• Record the change. Two lines in a config file are trivially forgettable six months later when a fresh container mysteriously lacks TUN. Put them in whatever provisioning script or template you use, alongside the rest of your cluster build notes.

Lessons learned

Device access is not privilege. The instinct to reach for unprivileged: 0 comes from treating container security as a binary. It isn't. The device cgroup, capability sets, mount entries, and idmaps are independent knobs, and almost every "just make it privileged" answer online is someone who found one knob and gave up on the rest. Name the exact device you need and grant that.

Flags are not interchangeable. nesting=1 fixes cgroup visibility. AppArmor profiles gate mounts and ptrace. Neither touches /dev. Reading the actual mechanism took less time than trying flags at random, and I'd skip straight to the mechanism next time.

Load the module first, always. Container start failing on a missing bind-mount source is a confusing error to debug backwards, and the /etc/modules-load.d/tun.conf file has to exist on every node the container can migrate to. Cluster-wide, not per-node.

Pick one SSH story. Running both OpenSSH and Tailscale SSH on the same node means two authorization models, two audit trails, and two places to revoke access. With a kernel TUN device you get to use plain OpenSSH over the tailnet, so I turn Tailscale SSH off on these containers rather than leaving both paths live.

The surprise was how boring the fix is. Two config lines, one module, one restart. All the complexity lives in understanding why the defaults deny it, which is generally where the interesting part of infrastructure work hides. If you're designing this kind of access model for a team rather than a homelab, that's a conversation I have often through GuatuLabs, and the boundary questions get more interesting once compliance and multiple operators enter the picture.

One caveat worth stating plainly: this gives the container a kernel TUN device, not the right to advertise subnet routes. If you also want it acting as a subnet router, you need IP forwarding enabled inside the container and --advertise-routes on the Tailscale side, which is a different set of tradeoffs. For the common case (a container that should be its own node, reachable directly, with working UDP and your real sshd), the two lines above are the entire job.