DEV Community

Guatu
Guatu

Posted on • Originally published at guatulabs.dev

Unprivileged LXC + Docker: The runc Sysctl Permission Trap

sysctl: setting key "net.ipv4.ip_local_port_range": Permission denied

I saw this error while trying to tune the network stack for a high-concurrency service running in Docker, which itself was hosted inside an unprivileged LXC container on Proxmox. The weird part? I was root inside the container.

I expected that since I had already enabled nesting=1 and keyctl=1 in the LXC configuration, Docker would have the necessary permissions to modify kernel parameters via runc. In a standard VM, this is trivial. In a privileged container, it just works. But in an unprivileged container, the user namespace mapping creates a wall that runc cannot climb.

What actually happened is a collision between systemd (v243+), runc, and the Linux kernel's security model for unprivileged user namespaces. When you run an unprivileged LXC, the root user inside the container is actually a non-privileged user on the Proxmox host (usually UID 100000).

The kernel prevents these mapped users from modifying sysctl settings because those settings are often global or namespace-specific in ways that could allow a container to crash the host or leak information. runc, the runtime Docker uses, tries to apply these settings during container creation, but the kernel returns a permission denied error. Because of how some Docker versions handle this, the error is sometimes swallowed, and your app just runs with the wrong defaults.

If you're building a production-grade homelab, you probably don't want to just switch to a privileged container. That's a security nightmare. Instead, you have to move the configuration "up" the chain.

The fix is to apply the sysctl settings at the LXC level before the container fully initializes, or directly on the host if the parameter isn't namespaced. Since we want to keep the host clean, using an LXC pre-start hook is the cleanest way to inject these settings.

On the Proxmox host, you can add a hook to the container's configuration file (usually in /etc/pve/lxc/ID.conf).

# Add this to your LXC .conf file on the Proxmox host
lxc.hook.pre-start = /usr/bin/echo "net.ipv4.ip_local_port_range = 1024 65535" >> /etc/sysctl.d/99-lxc.conf
Enter fullscreen mode Exit fullscreen mode

However, for most users, the most reliable method is to define the parameter in the host's sysctl.conf if it's a global setting, or use the lxc.sysctl directive in the config file:

# Example Proxmox LXC config snippet
arch: amd64
cores: 2
memory: 2048
net0: name=eth0,bridge=vmbr0,ip=10.0.0.x/24,gw=10.0.0.1
ostype: ubuntu
unprivileged: 1
features: nesting=1,keyctl=1
# Inject the sysctl here
lxc.sysctl.net.ipv4.ip_local_port_range = 1024 65535
Enter fullscreen mode Exit fullscreen mode

After adding this, you have to restart the container. If you just restart the Docker daemon inside the LXC, the kernel parameter won't update because the LXC boundary is where the restriction lives.

This trap is common when you're trying to optimize networking or memory management (like vm.max_map_count for Elasticsearch) inside a nested environment. If you've dealt with the headache of GPU passthrough on Proxmox, you know that the gap between "it's a container" and "it's an unprivileged container" is where most of the pain lives.

One last thing to watch out for: UID shifts. If you're mounting NFS shares into these containers to provide storage for your Docker volumes, you'll hit the UID mismatch. The container thinks it's root (UID 0), but the host sees UID 100000. I've spent hours debugging "Permission Denied" on volumes only to realize I needed to chmod 0777 the host directory or properly map the IDs in the .conf file.

If you're scaling this into a larger cluster, I highly recommend moving these workloads to bare-metal Kubernetes. I wrote about my experience with Longhorn for bare-metal storage, and while the initial setup is heavier than an LXC, you stop fighting the Proxmox container permission war and start dealing with standard K8s primitives.

Top comments (0)