DEV Community

Cover image for Home Lab Networking: DNS, VPN, VLANs, and Reverse Proxy Flow
FlatlinerAtheist | CLI-NERDS
FlatlinerAtheist | CLI-NERDS

Posted on Originally published at clinerds.com

Home Lab Networking: DNS, VPN, VLANs, and Reverse Proxy Flow

#networking #dns #selfhosting #vpn

When a self-hosted app "is down", the app is often fine and something on the way to it is not, such as a client asking the wrong resolver or a proxy that cannot reach its backend. This guide builds the path from client to container one hop at a time, covering internal DNS, VLANs, a reverse proxy with valid TLS, container exposure, and remote access. It ends with a troubleshooting routine that walks the same path.

Draw the path before you change anything

Write down every hop a request takes before you touch config. When something breaks, you test each hop in order instead of guessing, and the drawing shows which component owns each decision.

laptop 10.10.30.25 (VLAN 30, clients)
  -> DNS: resolver 10.10.20.2 says jellyfin.home.example.com is 10.10.20.5
  -> router/firewall: VLAN 30 to 10.10.20.5 tcp/443 is allowed
  -> reverse proxy: Caddy on 10.10.20.5:443 terminates TLS, picks a backend by hostname
  -> app network: Docker network "proxy"
  -> container: jellyfin:8096
Enter fullscreen mode Exit fullscreen mode

Use your own names and addresses, and keep the drawing next to your config.

Pick a domain and run split-horizon DNS

Use a subdomain of a real domain you own, such as home.example.com. Avoid .local: it is reserved for multicast DNS (RFC 6762), and clients running mDNS (macOS, many Linux desktops) may send those lookups to mDNS instead of your resolver, causing slow or failed resolution. The name home.arpa is reserved for home networks by RFC 8375 and is safe to use, but no public certificate authority will issue certificates for it. A real domain lets you get trusted certificates without running your own CA.

Split-horizon DNS means your local resolver answers jellyfin.home.example.com with the LAN address of the proxy, while public DNS has no record or points somewhere else. Pi-hole (local DNS records), AdGuard Home (DNS rewrites), and Unbound can all do this. With Unbound, add a file such as /etc/unbound/unbound.conf.d/home.conf:

server:
    interface: 10.10.20.2
    access-control: 10.10.0.0/16 allow
    local-zone: "home.example.com." transparent
    local-data: "jellyfin.home.example.com. IN A 10.10.20.5"
    local-data: "photos.home.example.com. IN A 10.10.20.5"
Enter fullscreen mode Exit fullscreen mode

By default Unbound listens only on localhost and refuses other clients, which is why interface and access-control are there. The transparent zone answers the names you list and resolves everything else normally. Check the syntax before restarting:

sudo unbound-checkconf && sudo systemctl restart unbound
Enter fullscreen mode Exit fullscreen mode

Then compare the local and public answers:

dig @10.10.20.2 jellyfin.home.example.com +short
dig @1.1.1.1 jellyfin.home.example.com +short
Enter fullscreen mode Exit fullscreen mode

The first should print 10.10.20.5. The second should print nothing or your public record. Hand out 10.10.20.2 as the DNS server in DHCP on every VLAN. A second resolver for redundancy needs the same local records, or answers will depend on which one a client asks. For the public side, the DNS lookup tool shows what resolvers outside your network return.

VLANs and default-deny rules between them

A VLAN plan gives each class of device its own subnet so the router's firewall sits between them. A /24 per VLAN with the VLAN ID in the third octet is easy to read:

VLAN Name Subnet Holds
10 Management 10.10.10.0/24 switch and hypervisor admin, router UI
20 Servers 10.10.20.0/24 resolver, reverse proxy, Docker hosts
30 Clients 10.10.30.0/24 laptops, phones
40 IoT 10.10.40.0/24 TVs, cameras, smart plugs
50 Guests 10.10.50.0/24 visitor Wi-Fi

Avoid 192.168.0.0/24 and 192.168.1.0/24 for lab networks, since hotel and friends' networks often use them and will collide when you connect over VPN. If you have to fit everything in one /24, the subnetting guide walks through a VLSM plan, and the subnet calculator checks the ranges.

On the switch, an access port carries one VLAN untagged, so the device plugged into it needs no VLAN configuration. A trunk port carries several VLANs, each frame tagged with its 802.1Q VLAN ID, and connects the switch to the router, another switch, a hypervisor, or an access point with several SSIDs. Reconfigure ports from a connection that does not pass through the port you are changing, or you can cut yourself off.

Hosts in the same VLAN talk directly at layer 2, so the firewall only controls traffic between VLANs. Start from deny and allow specific flows. On a Linux router, the forward chain in /etc/nftables.conf could look like this (NAT rules omitted, interface names are examples):

table inet filter {
    chain forward {
        type filter hook forward priority filter; policy drop;
        ct state established,related accept
        ct state invalid drop

        # management may reach everything
        iifname "vlan10" accept

        # every VLAN, guests included, may use the resolver
        ip daddr 10.10.20.2 meta l4proto { tcp, udp } th dport 53 accept

        # clients and VPN users reach the reverse proxy on HTTPS only
        iifname { "vlan30", "wg0" } ip daddr 10.10.20.5 tcp dport 443 accept

        # your own VPN peer may reach management
        iifname "wg0" ip saddr 10.10.99.2 oifname "vlan10" accept

        # Home Assistant may reach IoT devices; IoT cannot start connections back
        ip saddr 10.10.20.10 oifname "vlan40" accept

        # internal VLANs may reach the internet
        iifname { "vlan20", "vlan30", "vlan40", "vlan50" } oifname "wan0" accept

        log prefix "fwd-drop: " drop
    }
}
Enter fullscreen mode Exit fullscreen mode

IoT has no rule toward VLAN 10, so the policy drops it. Guests get the resolver and the internet and nothing else; point them at a public resolver instead if you would rather not expose internal names. The router's own admin interface is reached through the input chain, not forward, so give that chain a drop policy too and allow admin ports only from VLAN 10.

Test before applying, and keep a console session (physical or hypervisor) open in case a rule cuts off SSH:

sudo nft -c -f /etc/nftables.conf && sudo nft -f /etc/nftables.conf
Enter fullscreen mode Exit fullscreen mode

Run sudo systemctl enable nftables so the file loads at boot. Dropped packets appear in sudo journalctl -k | grep fwd-drop. Remove the log rule once things work if it gets noisy. On OPNsense or pfSense the same policy is written as rules on the interface where traffic enters, evaluated top to bottom.

One reverse proxy with DNS-01 certificates

Send every web service through one reverse proxy. TLS and access logs then live in one place, and the firewall needs a single allow rule (tcp/443 to 10.10.20.5). Caddy, Traefik, and Nginx all work; the examples use Caddy.

For certificates, use the ACME DNS-01 challenge. The certificate authority checks a TXT record at _acme-challenge.<name> in public DNS, so it never connects to your network and no inbound port is needed. DNS-01 is also the only way to get a Let's Encrypt wildcard such as *.home.example.com, and a wildcard keeps individual hostnames out of public Certificate Transparency logs.

Caddy's standard build does not include DNS provider plugins. Build an image with the one for your DNS host (Cloudflare here):

FROM caddy:2-builder AS builder
RUN xcaddy build --with github.com/caddy-dns/cloudflare

FROM caddy:2
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
Enter fullscreen mode Exit fullscreen mode

The Caddyfile:

*.home.example.com {
    tls {
        dns cloudflare {env.CF_API_TOKEN}
        resolvers 1.1.1.1
    }

    @jellyfin host jellyfin.home.example.com
    handle @jellyfin {
        reverse_proxy jellyfin:8096
    }

    handle {
        abort
    }
}
Enter fullscreen mode Exit fullscreen mode

resolvers makes Caddy check the challenge record against public DNS instead of your split-horizon resolver. Limit the API token to that one zone with the permissions the plugin's README lists, and keep it in an env file with mode 600 rather than in the Caddyfile. Traefik does the same with a dnsChallenge certificate resolver, and Nginx users can run certbot with a DNS plugin such as python3-certbot-dns-cloudflare.

Keep container ports off 0.0.0.0

A Compose port mapping like "8096:8096" listens on every address of the host. Docker manages its own iptables rules, and published traffic is forwarded to the container before ufw's host rules see it, so ufw deny 8096 does not block it. Publish only the proxy, and let apps join the proxy's network:

services:
  caddy:
    build: ./caddy
    ports:
      - "10.10.20.5:443:443"
    env_file: caddy.env
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
    networks: [proxy]

  jellyfin:
    image: jellyfin/jellyfin
    networks: [proxy]

networks:
  proxy:
    name: proxy

volumes:
  caddy_data:
Enter fullscreen mode Exit fullscreen mode

The caddy_data volume stores certificates, so they survive restarts. For a tool you only use from the host itself, bind it to loopback, for example "127.0.0.1:9000:9000". The Docker Compose stack guide covers the rest of the Compose layout.

Verify what is listening:

sudo ss -tlnp
Enter fullscreen mode Exit fullscreen mode 
LISTEN  0  4096  10.10.20.5:443   0.0.0.0:*  users:(("docker-proxy",pid=1873,fd=7))
LISTEN  0  4096   127.0.0.1:9000   0.0.0.0:*  users:(("docker-proxy",pid=1902,fd=7))
Enter fullscreen mode Exit fullscreen mode

Any 0.0.0.0 or [::] entry you did not intend is reachable from the host's own VLAN and from anything the firewall lets through. The port check tool tests your public IP from outside.

Remote access without port-forwarded admin panels

Do not forward router, hypervisor, or NAS admin interfaces to the internet. Use a VPN and reach them as if you were at home.

WireGuard needs one forwarded UDP port, conventionally 51820/udp. It does not answer packets that fail authentication, so a scan gets no response. A client config that routes only lab traffic through the tunnel and uses your resolver:

[Interface]
PrivateKey = <client-private-key>
Address = 10.10.99.2/32
DNS = 10.10.20.2

[Peer]
PublicKey = <server-public-key>
Endpoint = vpn.example.com:51820
AllowedIPs = 10.10.10.0/24, 10.10.20.0/24, 10.10.99.0/24
PersistentKeepalive = 25
Enter fullscreen mode Exit fullscreen mode

With WireGuard running on the router, the wg0 rules in the firewall example decide what each peer can reach, so treat the tunnel as one more zone.

Tailscale usually needs no inbound port at all, which helps if your ISP puts you behind CGNAT. Run a subnet router on a lab host with sudo tailscale up --advertise-routes=10.10.20.0/24, approve the route in the admin console, and enable IP forwarding on that host. By default, subnet-routed traffic reaches the LAN from the subnet router's own address, so firewall rules see it as that host. Add a split DNS entry for home.example.com pointing at 10.10.20.2 so names resolve remotely.

Troubleshooting hop by hop

Walk the drawing from the top and stop at the first hop that fails.

Name resolution comes first. From the client:

dig jellyfin.home.example.com +short
Enter fullscreen mode Exit fullscreen mode

A public IP or empty answer means the client is not using your resolver. Check resolvectl status on systemd-resolved systems, and check whether the browser uses DNS over HTTPS, which bypasses the local resolver. If you recently changed public records, cached answers may still be in play, as explained in DNS propagation.

Next, the firewall. Ping may fail because the rules only allow 443, so test the port itself:

nc -vz 10.10.20.5 443
sudo traceroute -T -p 443 10.10.20.5
Enter fullscreen mode Exit fullscreen mode

A timeout here points at a firewall rule or VLAN assignment. The TCP traceroute (install the traceroute package if missing) shows whether packets reach the gateway, 10.10.30.1, at all.

Then the proxy, with DNS taken out of the picture:

curl -v --resolve jellyfin.home.example.com:443:10.10.20.5 https://jellyfin.home.example.com/
Enter fullscreen mode Exit fullscreen mode

A certificate error points at the proxy's TLS setup. A 502 means the proxy is up but cannot reach the backend. On the proxy host, sudo ss -tlnp 'sport = :443' confirms it is listening on the right address.

Finally, the container network:

docker network inspect proxy --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{println}}{{end}}'
docker run --rm --network proxy curlimages/curl -sI http://jellyfin:8096/
Enter fullscreen mode Exit fullscreen mode

Both containers should be listed, and any HTTP status line from the second command means the path works. "Could not resolve host" means the backend is not attached to the proxy network, since Docker's internal DNS only resolves containers on a shared user-defined network.

Checklist

  • The path is drawn with real names and addresses.
  • Local and public dig answers differ as intended.
  • Every VLAN defaults to deny, with named exceptions.
  • Only the reverse proxy publishes a port, and ss -tlnp agrees.
  • Admin interfaces are reachable only from VLAN 10 or the VPN.
  • The DNS lookup and port check tools show nothing public you did not intend.

Top comments (0)