Hi All!
New to the Fediverse from the reddit exodus, I’ve gotten into Self-hosting around January this year and have been loving learning about Networks and how they are structured and communicate and I love the projects that come out of managing a home lab.
As I’ve built up my home lab, from a single node to 3, I’ve been trying to think of how to structure my network to segment it in such a way that my homelab is on its own segmented network, whether this be VLAN or separate LAN (though I’ve heard terrible things about double NAT), and have that whole segment of traffic be pushed through a VPN tunnel. Unless that is not necessary? Part of the reason I want to make this post is so that people who are around the same point in their home lab adventure who might have similar questions can come to this thread to discuss particulars about the manner at hand. I’ll probably be structuring future questions in such a manner that allows people to discuss and nail down topics they may be struggling with wrapping their head around.
Back to my question, so let me give a better lay of the land. I am running Proxmox as my hypervisor on all these nodes, I have a generic ISP-provided router (ActionTek T3200 if interested in the model) that handles LAN routing and WiFi. Currently all my nodes are hard-lined to the router and I rely on proxmox default firewall atm, I haven’t dug into how to properly configure any of that since I wanted a separate solution, not sure the security implications of just using proxmox’s firewall so chime in if you know.
So all my nodes are hooked up to my router, but I have a Layer 2 switch I got for free (supports basic VLANs as well as some other basic features) and I want to configure my 3rd node to run OPNsense for my routing and VLAN tagging. The 3rd node will sit on the edge of the Router and the Switch, meaning Router connects to Ethernet port 1 on node 3 and Ethernet port 2 on node 3 connects to the switch and would be providing the LAN and internet access from my understanding. Node 3 will also be running a VPN tunnel to provide remote access as well as providing protection for my *Arr downloads. So the routing for my homelab should go from this:
Node 1, Node 2, Node 3 ------- ISP Router ------ Internet
to this:
Node 1, Node 2–Switch/Node3—ISP Router --Internet
Now my understanding is that structuring it in such a way means that if, for example, Node 1 which hosts my *Arr network were to pull a compromised download then its damage would only go as far as where the VLAN ends, ie would only affect my homelab network because its segmented in such a way that my devices connected to wifi would be unaffected.
I’ve just started to tinker around with configuring OPNsense when I got a sense for how to structure my network. I’m trying to virtualize it, which is a bit unconventional but not unheard of. I was able to spin up a VM running the installer and it gets through the install just fine but I am unable to reach the address provided. It’s 192.168.1.1 which is off my IP range. This might be where I need a bit of help understanding, but shouldn’t it give me an IP address that’s in my Routers IP Range? Maybe not, maybe its a sort of DMZ type thing? I’m not all too familiar so give a shout if you know something. I am thinking that the WAN and LAN ports are just getting switched during install and I need to interrupt the install and manually delegate those ports and then I’ll be able to connect? I haven’t had some solid time to dig deep on this so I figured rambling on a forum with tech savvy individuals might at the very least provide me with some insight and more understanding.
So I suppose my question is what do you think of my thought process? Am I missing anything major in my understanding? How should you think about configuring your firewall and VPN? How do you setup VLANs to allow communication where necessary between VLANs and Wifi network? Apologies that this got so long, I was trying to keep it brief but also give enough info on my environment. Let me know if there’s any questions. I’d also be interested in resources if my topics just point to needing a better understanding of networking generally. Thanks for your time
You must log in or register to comment.
This sounds like a fun project! I recently just ripped out and redid the network segmentation on my 3-node proxmox cluster too.
Originally I had everything in a /16, but that was causing some routing problems because I actually needed to static route a /24 that was in the /16 to a VM for VPN. Anyways, I’m going to try to dig through your post and give some advice. This is all just personal opinion on how I’d set stuff up after over a decade of homelabbing/home infra, so ya know, take and leave what you want.
It sounds like you want to use one of your Proxmox nodes with a VM running OPNSense as your router? I’d highly discourage this. I know you call your setup a lab, but it’s running the *arrs and probably a streaming server; and there’s nothing worse than planning a movie night and having your networking be down. Also, it’ll make it easier to recover from a power outage or hardware failure, keep your network config much simpler, and provide physical boundaries between machines increasing security.
So, I’d say unless you’re fine with the possibility of extended outages, use dedicated hardware for network. I’m partial to pfsense’s netgate, it’s a good price and a lot of bang for your buck, and it comes from an awesome open source project. I use Unifi, though I wouldn’t necessarily recommend it due to some shady stuff the company has done/said over the last few years.
OPNSense looks neat, but the only reason I see to use it over pfsense is the integrated IDS/IPS, which is just a nice gui over suricata and a proofpoints subscription. Personally I’d run suricata in a VM and mirror WAN traffic to it via pfsense. This way a VM isn’t in your critical network path, but IDS is available and easy to manage.
Don’t forget, when you separate stuff into VLANs, it forces the traffic to go up to the router, and back down to the switch. This means any inter-VLAN traffic has a 1gb limit on it. So if you ever upgrade your servers with 10gb nics, if you setup VLANs incorrectly, you won’t get that performance. Or if you just have a lot of traffic, you’ll start getting TCP slow starts and retransmissions and it may play havoc on your network. That’s why many usually just don’t bother with VLANing, it gives you network isolation, but comes at the cost of increased routing.
As for routing, all VLANs will route between eachother automatically. As obvious as it is, just think of two VLANs as two separate physical switches plugged into the same router. By default those two switches will be able to communicate with each other through the router, but they can’t directly communicate with one an other (which would have higher throughput/bandwidth).
DMZ is interesting, in my mind it basically came from a time when networks had a hard shell and a soft interior (wrt security). I don’t DMZ because I have host level firewalls and network firewalls to do LAN segmentation. But, that isn’t to say it’s a bad idea if you’re up for it. Basically a DMZ (demilitarized zone) is a VLAN where you’d put stuff like a mail server, DNS servers, and maybe an HTTP server. Stuff you’d want to expose to the internet, as well as into your local network. The idea is that if one of those servers were compromised, you wouldn’t want it to have full access to your local network. So instead you split off a DMZ network so if a host in it gets compromised, it wouldn’t provide the attackers a good base to pivot into your local network from. I don’t expose any services to the internet, except a VPN, and that VPN definitionally needs a lot of access to my local network to be useful, so I don’t partition it off into its own DMZ. I’m not a network admin, so that’s just my interpretation of it.
As for structure, this is where I ended up: 10.0.0.0/24 - LAN management - Stuff like Unifi/Pfsense admin panels 10.0.1.0/23 - LAN - Where most of my normal stuff goes 10.99.0.0/24 - OOB Administration - Things like IPMI and BMCs end up here 10.99.1.0/24 - Administration - Things like Proxmox VMs end up here 10.99.2.0/24 - Core network - Things like VPN, DNS, backups, basically important network services. 10.99.3.0/24 - Services - Things like *arr, etc. end up here. I actually run K8s via kubespray in Proxmox, so for me this is actually my metalLB service IP range. 10.99.100.0/24 - VPN IP pool - I static IP my VPN clients instead of masquerade, so they get an IP out of this pool, the VPN instance acts as a router, and I static route from my main router back to the VPN instance.
My LAN is a separate IP range that then routes to this. And I have a separate /24 for my 10gb network for Ceph.
So yeah! I don’t know if that helps at all, feel free to ask questions to clarify. If you still really want to install OPNSense in a VM I can give you some tips on that as well.
Dude this is awesome, thanks for your comment! It’s exactly the type of engagement I was hoping for, someone to help with a few key concepts. I’ll definitely be taking your advice to heart it sounds like you’ve been through the ringer with your own setup
I agree with you that running OPNsense in a VM is less than ideal but I figured as it would only affect my lab, I can deal with the occasional outage and this is more so I can learn first before going all in on a solution such as a netgate, I appreciate the suggestion there too. I’m not dead set on OPNsense I just figured a gui would be easier to navigate and it looked nice. I haven’t heard of suricata and mirroring WAN connection to pfsense, that definitely makes sense.
As for the VLANs I’m not dead set on that in particular either, the switch I have supported it and it made the most sense as to how to segment my network in that way. So you are saying you can also segment a LAN connection with just a firewall? It would make sense as you can set rules as to how a device can communicate with other devices on the network, is it rather cumbersome to set it up in that way? And on VLAN performance I would assume that would not be a problem as I am the only user who would be connecting to my media server, unless I had a bunch of services sending and receiving shitloads of data essentially?
I really appreciate all your help! You definitely sound like you’ve been doing this for a decade haha, very useful stuff. I might pop back and ask more questions later if you don’t mind
For sure, love talking about my home lab! And it’s an easy way to help get a bit of content on Beehaw, albeit a bit scatterbrained haha
First, the OSI model! Yeah, this is a very “first you must invent the universe” moment, sorry (not sorry). Basically for you the important bits are layer 2 and layer 3 (though I HIGHLY recommend knowing the OSI model forwards and backwards in its entirety).
Layer 2, the datalink layer, uses MAC addresses and is how to computers on a LAN communicate directly with one an other. Layer 3, the IP layer, uses (you guessed it) IP addresses and is what allows you to communicate between networks (it’s where we get the term internet: inter-network). Layer 2 is where switching takes place, layer 3 is where routing takes place.
So, if two computers are in a single LAN (or vLAN), they will communicate through a switch only. If two computers are not in the same LAN/vLAN, they can only have connectivity through a router. It’s really important to understand a switch only sees MAC addresses, so you can’t firewall at the switch level.
If some of this is new to you, I highly recommend reading networking 101 and 201 from this site. It seemed like a good resource.
Okay, with that in mind, next you really need to understand bridges. Any *nix box can be act as a switch and/or router. By default when you install Proxmox it creates a bridge on the machine’s default network interface. This bridge interface is effectively a switch. Every VM then “plugs” into this switch, giving it access to your network’s router, and fetches its IP via DHCP. You can use
ip linkandbrctl showto see it. Take a look at the related debian docs too.Setting up OPN in Proxmox
So with OPNSense, you have a few ways of setting it up, but you always need at least two LANs. One which is shared by OPNSense and your edge router (the ActionTek), and the other is shared by OPNSense and the VMs running in Proxmox. I’d recommend just using two vLANs as the best solution for now. One vLAN will be for your edge router, and the other will be for your OPN router.
First I’d setup your new switch so that every Proxmox host is on a trunk port. This allows tagged traffic from the host, meaning you can specify a vLAN for each VM network interface. Make sure you get this working first, here’s a decent guide, and make sure to read the proxmox network docs too. Now everything should be on the native vLAN (special vlan 1, containing untagged traffic).
Next just add a second interface to your OPNSense with a different vLAN (maybe 100), this’ll be your OPN vLAN. Now you can move VMs to your OPN network by changing their vLAN to the OPN network’s vLAN in Proxmox. Easy peasy!
Finally you’ll want to setup proper connectivity between your routers. If you were running a cooler edge router you could setup BGP, but for now static routes will suffice. Setup a static route for your OPN network in your edge router, and point it to your OPN VM’s IP address (the IP the edge router assigned it, 192.168.1.X). Finally create the opposing route in your OPN router for the edge router network. This way you don’t need a double NAT.
Okay, really sorry if none of that made sense. I really did try to keep it concise :S
I really appreciate all your input, its really helped me connect a few dots that needed connecting so I really do appreciate it my man. I’m aware of the OSI model and I was hoping my work with my homelab would help me understand better, but I’ve still got a ways to go. I really appreciate your explanations on Layer 2 and Layer 3, I understood the definitions but your explanation helps me connect the idea to real world examples. I also understood bridges somewhat but relating it to a switch makes it simple to understand and configure.
Thanks for the links as well, I will be messing around with this config tonight so I will be using your comments as a resource, so really, thank you! Can I ask for permission to PM you if I have any questions related to homelab/networking? You and I seem very similar in our scatterbrain-ness lol, so I figured it might make things easier if that makes sense. I will definitely be doing more reading and research, the first link you provided seems like an absolute blessing of a resource.
Oh on my first point I misunderstood, this will not replace my router, it will only be used by my proxmox network
I run all my lab servers/services/etc in their own /16 on my home net. Nothing is publicly routed in over my WAN IP- if I want to expose a service, it goes through Nginx Proxy Manager to my local service via a ZeroTier tunnel.
I would strongly encourage you to not expose any of the *arr services (particularly your download node) to your WAN IP. PIA’s desktop app does a pretty good job of forcing a full tunnel with a VPN kill switch, so you never have to worry about your ISP catching onto what you’re doing.
So you’re saying your services run on a separate subnet? 255.255.0.0? How would you connect from your home pc connected to your home WiFi? I assume have the vpn running on the machine on a different subnet and also have it running in front of the service, the vpn would give your home computer an IP on the /16 subnet range? Am I correct in that assumption?
I suppose I need to get OPNsense actually working and providing a different subnet in the first place before worrying about all this, I appreciate your input! I understand about exposing the WAN IP, I’m assuming VPN tunnel for those specific services would protect my WAN IP as it would just send all my traffic to the VPN provider and then out to the actual destination, again correct me if I’m wrong. I don’t think I understand how the actual routing would work, how to hook the services into nginx proxy manager and how to know which ports to close and what not, but I suppose I’m not at that step quite yet
Yes, I’ve got separate subnets & vlans for a few things. My PCs/phone/tablets/etc, homelab, IoT devices (i.e. loads of Govee bulbs/ropes, gaming consoles, oven, etc), Guest (all isolated from everything else internal) and one for my roommate. I’m on a Unifi Dream Machine Pro so setting up traffic rules to allow certain traffic from PC vlan to homelab (and the other way) was pretty straightforward.
As for the VPN, yes a full tunnel would force all traffic over the VPN, but for all but my *arr stuff that’s overkill. I just join all my VMs to Zerotier and force traffic from the public LB in via their VPN IP, but the VMs can still pull yum updates and anything else they want over my WAN link.
Oh boy, new post character limit is working alright😄
“It’s 192.168.1.1 which is off my IP range. This might be where I need a bit of help understanding, but shouldn’t it give me an IP address that’s in my Routers IP Range?”
To solve this important question: You can change the OPNsense interface IPs either in the command line of OPNsense (login root/opnsense), or you can chane the IP of your management machine temporally to access this IP range and thus the OPNsense GUI.
Backup the OPNsense VM in Proxmox first, then you’re free to play/mess up however you want ;-)
