The homelab started flat. One /24, everything on it. My workstation, the NAS, the Proxmox host, and — over time — a growing list of workloads sharing the same broadcast domain because that was the path of least resistance.
For a while, that was fine. A homelab running one workload doesn't need segmentation any more than a house needs an office door.
Then I stood up an Akash provider.
An Akash provider is, in shape, a Kubernetes cluster that accepts inbound tenant workloads from the internet — real deployments, paying for compute, containers I didn't write landing in namespaces on my hardware. The provider itself is documented at github.com/jjozzietech/akash-provider-ops-public — this piece is about the network underneath it. The containerisation posture itself is fine. I trust the isolation model. But trust isn't a network design. And the network at that moment had the tenant workload cluster sitting on the same subnet as my workstation, my NAS, and my Proxmox management interface.
That was the moment I stopped thinking of the rack as a home network with extra boxes, and started thinking of it as a small data centre.
This piece is the network design that came out of that shift. I'll cover the layout, the rules that hold it together, and the Nexus and Proxmox configs that anchor it — with the specifics of my own deployment sanitised. It's not a step-by-step replication guide. It's the design pattern, with enough of the shape to be useful and enough restraint to not double as a recon document for my own rack.
// the original design
The flat layout looked like this:
home lan — 192.168.1.0/24
opnsense (perimeter)
cisco nexus (dumb L2 switching)
proxmox host
workload VMs (all on the same subnet)
What it got right: zero routing complexity, everything reachable from everywhere, fast to stand up. If you're running one project on a homelab, this is the correct design. Don't over-engineer it.
What stopped working, as soon as the second project landed on the rack, was that the design had no answer to three questions:
- How do I write different firewall policies for different workloads when they all share a subnet?
- How do I keep a compromise in workload A from reaching workload B?
- How do I troubleshoot when every packet in the rack lives in the same broadcast domain?
The last one is the operational tell. When "which workload is chatty on the wire right now?" becomes a question you can't answer from a switchport counter, the design has outgrown the problem it was designed for.
The industry term for what I was worried about is east-west traffic — traffic between workloads on the same tier, as opposed to north-south traffic between workloads and the outside world. Flat L2 doesn't give you any policy handle on east-west. You just have to trust that nothing on the wire behaves badly.
That's a defensible assumption for a house LAN. It's not a defensible assumption for infrastructure that hosts tenant workloads.
// the reframe: stop thinking residential
Commercial data centres don't put every workload in the same subnet. They isolate services into dedicated network segments, each with its own routing policy, its own firewall posture, its own blast radius. When something goes wrong, the fault is bounded. When something needs a different security posture, it gets one. When a workload needs to move to different hardware or a different site, the network context comes with it.
That's the model I took. Every project on the rack gets:
- Its own VLAN
- Its own /24
- Its own firewall policy at the routing boundary
- No default trust with any other VLAN
The hypervisor stays on management. Workloads live on their own VLANs. Ops access is explicit, not implicit.
A quick note on hardware, because it matters. The switch doing this work is a Cisco Nexus 3548P — enterprise-grade L3 switching, ACLs applied inbound on the SVI, inter-VLAN routing at line rate on the switching ASIC rather than punted to a CPU. A UniFi or Mikrotik can do most of what's below, and if you're building this pattern on consumer gear, that's a reasonable place to start. What you give up is the ability to enforce policy at wire speed with the ACL evaluated in hardware. On a homelab-sized rack it's rarely the bottleneck. On a rack that hosts real workloads with real tenants, I wanted the ceiling raised.
// the architecture
Here's the layout, in the same YAML metadata style I use across the site:
management: 192.168.1.0/24 workstation, opnsense, proxmox host
vlan N (workload-a): 10.N.N.0/24 first project
vlan M (workload-b): 10.M.M.0/24 second project
vlan …: … future
Actual VLAN IDs and subnets in my deployment differ. The pattern is what matters, and I'll get to why the pattern is the way it is in the "what I'd do differently" section.
Three design rules underpin this.
The hypervisor stays on management, VMs live on workload VLANs. The Proxmox host is not a workload. It's the platform workloads run on. Putting it on the management VLAN means the host stays reachable when a workload VLAN goes sideways — backups continue, console access works, snapshots run. If the Proxmox host is on the same VLAN as a workload and that VLAN's routing breaks, you've just lost your recovery path.
One /24 per workload, gateway on the Nexus. The L3 boundary between VLANs is where policy lives. That means the Nexus SVI, not OPNsense. OPNsense's job is different, and I'll get to it.
Workload VLANs are peers, and no VLAN trusts another by default. East-west is denied. ACLs grant the exceptions. If workload A ever needs to talk to workload B, that's a deliberate rule, not a side effect of shared broadcast domain.
// what the nexus does
Three sanitised snippets, each with a sentence on intent.
VLAN definition:
vlan 10
name workload-a
Nothing exciting. The VLAN is a container; the interesting policy lives on the SVI.
SVI with gateway and inbound ACL:
interface Vlan10
description workload-a-gw
ip address 10.10.10.1/24
ip access-group acl-workload-a in
no shutdown
The ip access-group acl-workload-a in line is the important one. Inbound on the SVI means traffic entering the Nexus from the workload VLAN is what gets filtered. Any packet leaving workload A's subnet passes through that ACL before it gets routed anywhere else. That's how east-west enforcement happens at the routing boundary.
Trunk to a Proxmox node:
interface Ethernet1/3
description pve01-trunk
switchport mode trunk
switchport trunk allowed vlan 10,20,30
spanning-tree port type edge trunk
no shutdown
The switchport trunk allowed vlan list is explicit. Only VLANs I've deliberately added to the trunk can traverse it. New workload comes online → the VLAN gets added to the allow-list. Mundane, but it means the trunk isn't a wide-open pipe by default.
I leave the native VLAN unset on workload trunks. Tagging is explicit on the Proxmox side, which I'll cover shortly.
// ACLs are the actual security mechanism
VLAN segmentation isolates broadcast domains. That's it. Two VLANs on the same switch, both trunked to the same Proxmox host, can absolutely talk to each other — as soon as one of them has a route to the other. The route is provided by the SVI. Which is why the SVI is where policy has to live.
The pattern, not the rules, for a workload VLAN's ACL looks like this:
acl-workload-a:
permit management-workstation -> workload-a (ops access in)
permit workload-a -> dns, ntp (outbound essentials)
deny workload-a -> 10.0.0.0/8 (block east-west)
deny workload-a -> 172.16.0.0/12 (block east-west)
deny workload-a -> 192.168.0.0/16 (block east-west)
permit workload-a -> any (internet last)
Walking through the logic:
Rule order matters. The three RFC1918 denies sit before the permit-any-internet. Reverse them and the denies never evaluate — the permit-any matches first and every internal network is reachable. This is the single most common way I've seen this pattern implemented incorrectly.
The management workstation gets an explicit permit at the top. Without it, I lose ops access to the workload VLAN — SSH, monitoring, any control-plane traffic. With it, only that one host can reach in. Everything else that tries to reach workload A from a private range hits the RFC1918 denies further down.
DNS and NTP get their own permit lines. Cleaner than carving exceptions into the deny block. The workload needs to resolve names and know what time it is; both are explicit.
No VLAN-to-VLAN permits exist. If workload A needs to talk to workload B in the future, that'll be a deliberate exception — added, logged, reviewed. Default deny is the actual default, not aspirational.
What's not in this article: the per-host permits, the port restrictions inside the workload VLAN, the exact management-range definition, and the north-south inbound permits handled at the OPNsense boundary. The shape is the point. The full ruleset stays on the rack.
// dividing the labour: nexus vs opnsense
There's a clean split between the two devices.
The Nexus handles east-west: inter-VLAN routing and policy. Everything that stays inside the rack.
OPNsense handles north-south: WAN, NAT, port forwards, outbound NAT, anything that crosses the perimeter to or from the internet.
Two devices, two audit surfaces, two separate rulesets. Both stay simpler as a result.
The alternative — collapsing everything into OPNsense as a router-on-a-stick — is a design I see in a lot of homelab writeups, and it works. But it has two costs. First, inter-VLAN traffic becomes CPU-bound on the firewall instead of running at line rate on the switch. Second, north-south and east-west rules end up in the same ruleset, which makes both harder to reason about over time. You end up asking "does this rule apply to WAN traffic or inter-VLAN?" every time you touch the config, and the answer depends on interface assignments buried elsewhere.
Keeping the two roles physically separate — Nexus for east-west, OPNsense for north-south — is more disciplined. And on the Nexus, inter-VLAN routing doesn't touch the firewall at all, which means an OPNsense reboot doesn't interrupt workload-to-workload traffic. (It also doesn't enable workload-to-workload traffic, because the Nexus ACL denies it by default. But the point is that the firewall isn't in the east-west path either way.)
// the proxmox side
The Proxmox network config on the host is deliberately simple. One VLAN-aware Linux bridge, one LACP bond feeding it, management IP on the untagged side.
Here's the shape, sanitised — real interface names, redacted IPs:
auto lo
iface lo inet loopback
iface ens7f0 inet manual
iface ens7f1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves ens7f0 ens7f1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
auto vmbr0
iface vmbr0 inet static
address 192.168.1.10/24
gateway 192.168.1.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
Two physical NICs (ens7f0 and ens7f1) bond together into bond0 using LACP (802.3ad) — the switch side is configured as a matching port-channel. That gives me two paths to the Nexus, active/active, with automatic failover if one link drops.
The bond feeds a single Linux bridge, vmbr0, configured as VLAN-aware. The bridge trunks every VLAN from 2 to 4094 (in practice, only the VLANs actually on the switch's allow-list traverse it). The hypervisor's management IP sits on the bridge itself on the untagged VLAN — which is the management VLAN, reachable from the workstation and the rest of the management infrastructure.
The VM side is one line of config per VM: the NIC gets a VLAN tag set in the Proxmox GUI. The VM sees a regular Ethernet interface. The bridge handles tagging on the way out to the bond, and the Nexus routes based on the tag. That's the whole story on the host.
Notably absent: a separate bridge per VLAN. That design works, but it doesn't scale — every new VLAN means a new bridge, a new set of uplinks to manage, and a growing surface of things that can drift out of sync between hosts. One VLAN-aware bridge trunked to the switch does the same job with dramatically less state to maintain.
// what I'd do differently
Two things: one process, one design. Plus the honest note on the migration itself.
Process: design the ACLs before the workload goes live, not after.
The segmentation itself I got right the first time. The ACLs I got right eventually — but I built them reactively, as workloads landed and I discovered gaps. New workload goes live → I stand up the VLAN and SVI → I permit the ops access I need to configure it → the workload comes up → I notice traffic leaking somewhere it shouldn't → I tighten the ACL. Each iteration got closer to correct. But each iteration also had a window where the ACL was more permissive than I wanted, and one of those windows was long enough to matter.
The right approach is to have a template for both directions of policy before you provision anything. The east-west deny block as a ready-to-paste chunk. The north-south permit list — which ports OPNsense will forward inbound, from where — drafted alongside the workload's own deployment plan. Apply both at the moment the VLAN is created. The workload never exists in a permissive state, even briefly. The "did I remember to tighten that rule?" question never comes up.
This is the single thing that separates the segmentation I have now from segmentation I'd have designed on day one if I could start over.
Design: I'd keep the VLAN scheme exactly as it is.
The pattern is VLAN N maps to 10.N.N.0/24. VLAN 10 lives in 10.10.10.0/24. VLAN 11 lives in 10.11.11.0/24. And so on.
It's opinionated and I'll defend it. The payoff is that troubleshooting is self-documenting. A packet capture shows a host in 10.10.10.0/24 and I know without looking that it's on VLAN 10. An ACL entry references 10.11.11.0/24 and I know without looking it's targeting VLAN 11. Every debugging session, every ACL review, every firewall rule audit is faster because the VLAN ID and the subnet are the same number.
The downside is you're locked into a /24 per VLAN, and you can't ladder easily beyond VLAN ~250 before you start running out of clean second-octet space. For a homelab, that's not a real constraint. If it ever becomes one, the failure mode is switching to a different scheme for the overflow VLANs — the existing ones don't need to change.
Migration: rolling, and it was fine — mostly.
The migration itself was rolling rather than greenfield. I had running workloads, no maintenance window, and no ability to stand up a parallel network to cut over to. So each project moved one at a time: stand up the new VLAN, ACL, and SVI on the Nexus; add the VLAN to the Proxmox trunk allow-list; retag the VM's NIC; update any OPNsense port forwards to the new IP; verify; move on to the next.
Here's the one that stung. My Storj storage node went dark for longer than I'd like to admit during the TrueNAS cutover. TrueNAS moved from the flat network to its own VLAN — new subnet, new IP. The Storj app runs as a TrueNAS CORE plugin, so it inherited the new IP automatically. What didn't inherit anything automatically was the OPNsense port-forward, which cheerfully continued sending inbound Storj traffic to TrueNAS's old address. From my side, everything looked fine — TrueNAS was up, the plugin was running, SMB shares were serving internal clients without complaint. From the Storj network's side, my node had stopped accepting inbound connections. It took a reputation dip before I noticed.
The failure mode is generic and worth naming: when a workload moves subnets, every downstream reference to its old address is now stale. Port-forwards, reverse proxies, DNS records, anything with a hardcoded IP in a config, anything scraping Prometheus on the old address. My cutover checklist was fine for the workload itself. It was not fine for everything that talked to the workload. That's the kind of gap a pre-provisioned north-south permit template — with every port and destination mapped out before the move — closes automatically. My checklist now includes an explicit "audit every reference to the old IP" step. It hasn't happened again.
// scaling: why this pays back
The marginal cost of adding workload N+1 is now:
- Pick the next VLAN ID and its matching /24.
- Define the VLAN and SVI on the Nexus, with the template ACL applied.
- Add the VLAN to the Proxmox trunk allow-list.
- Tag the VM's NIC.
- Configure north-south forwarding on OPNsense if the workload takes inbound traffic.
That's a fifteen-minute job for a workload that doesn't need custom firewall policy, and maybe an hour if it does. Not a redesign. Not a rework of anything already running.
The same logical layout transfers to colocation or a commercial data centre with the physical topology changing but nothing else. The VLAN IDs, the subnets, the ACLs, and the OPNsense rules all move as-is. That was one of the original design constraints — build a homelab network that doesn't need to be rethought when the rack does — and it's the part I'm most quietly proud of.
// closing
Most homelabs don't need any of this. If you're running a Plex box and a media NAS and maybe one game server, flat L2 is the right answer. The complexity here only starts paying back when you're running multiple workloads with materially different trust postures — something you built yourself next to something you didn't, something that accepts inbound traffic from the internet next to your workstation, something in production next to something in perpetual dev.
The moment you cross that line, segmentation stops being over-engineering and starts being basic operational hygiene.
This design pattern is the substrate the rest of the rack runs on. The next few pieces in this series build on it — OPNsense as the perimeter, TrueNAS as the storage layer, and eventually the workload-specific pieces where individual services get their own writeups. All of it assumes the network underneath looks like this.
If you're running mixed workloads on a flat network today, the question isn't whether to segment. It's how many workloads you want to add before you do.
Top comments (0)