Zerotier bypasses proxmox firewall

[vanburen.brian]

Hello, this is more of a proxmox question than a Zerotier one, but there does seem to be a lot of people using both from my reading of this forum, so I figured I’d post this here in case anyone has any insights.

For starters, this is my first time trying to use proxmox. I have a web development background, but extremely limited networking / IT knowledge. So this is all very new to me; I’m just saying that in case it helps formulate responses.

In my Proxmox setup, I have two VMs. One is a ubuntu server 22.04.3 for the sole purpose of self-hosting a zerotier controller (using ztnet which has an excellent UI for self-hosted network controllers). The other is another ubuntu server instance hosting web-based services. My hopes are to create a zerotier private network to allow members to access these services. It’s volunteer work for a local spiritual / religious organization, so security isn’t of utmost importance, but I want to use this as a good learning experience.

I followed instructions I could find to setting up the firewall in proxmox and applying it to each of the VMs. I am trying to make sure that the only access can be through either the local network, or the zerotier network, and that the only ports that can be accessed are the ones for the services on that VM. It took a bit of reading, but I was able to get everything to work with one gaping exception - accessing the devices using their IP address on the zerotier network completely bypasses the firewall. Once I turned the firewall on (before adding rules), it properly blocked me from using SSH or accessing the services on their local IPs (for example, 192.168.1.44:3777) but I could still SSH and access them if I used the zerotier IP (10.12.35.123:3777).

After doing quite a bit of reading, I found that in proxmox firewall needs to be enabled on the Network Device for the VM in order for it to work properly. Zerotier creates its own network device, inside the VM. My best guess here is that I somehow need to make sure that the firewall is being applied to this network device as well.

Looking in the proxmox webgui, if I go to the VM > Hardware, I can see ‘Network Device’ there with the one it created, where I have enabled the firewall. When SSH’d into the VM, I can use ifconfig to see the network device that I see in the web gui, with the same MAC address. Also, using ifconfig, I can see the network device created by Zerotier, with its MAC address. I am wondering if I can somehow make the firewall apply to that network device. I see that I can add more network devices, but my gut tells me I can’t just add a new one, using the MAC address that I see for the zerotier device in the VM, and have it work like I want. I haven’t even tried that yet because I am concerned about creating some type of conflict.

So, I am wondering how I can make sure that the firewall rules apply to the members of the zerotier network accessing the VM. That being said, I am a total neophyte to this and I realize that what I am trying to do might not be the best way to try to accomplish my goals. I’ve read stuff about bridging but honestly this is all still quite a bit over my head. I know that zerotier also has some abilities that are similar to what a firewall does, but I am not familiar with those and would need some guidance there as well if that’s a better solution. If I’m not following best practices here, or my approach is just completely wrong, I’d be happy to be pointed in the right direction.

[l0crian]

Whenever you create a VPN between 2 devices, you have 2 parts of the traffic; the underlay and the overlay. The underlay is the path that is used to create the VPN (like the internet). The overlay is the actual VPNs, and the data that is sent tunneled across those VPNs. When you use a network level firewall like you’re doing with Proxmox, it will only impact the ability to filter on the underlay, since it’ll have no idea what is in the encrypted payload. So you can filter traffic to prevent ZeroTier from coming up, but you won’t be able to prevent that SSH traffic over ZeroTier. You’d need to filter with a host firewall on Ubuntu (or use ZeroTier’s great Flow Rules Engine) if you wanted policy enforcement on that traffic.

I’m not very familiar with Proxmox, but one option that may be available is to have ZeroTier installed directly on Proxmox instead of within the VMs. With this, Proxmox would be aware of both the underlay and overlay interfaces, allowing for filtering for both parts of the traffic.

[vanburen.brian]

Thanks for your insights. I have tried the flow rules, but am not getting the desired results.

The network consists of two server endpoints the clients would be interested in: a NAS, and the one VM with web-based services. The NAS is secured by its own firewall. So the only other thing I really need to do is make sure the clients don’t talk to each other over Zerotier and then to make sure the only allowed traffic to the VM with the services are for SSH and the ports with web-based services.

I found the FAQ that shows how to block communication between two clients here ( Rules FAQ | ZeroTier Documentation ). That works. My rule to limit traffic to the VM, however, does not work. Here’s what I have:

# This is a default rule set that allows IPv4 and IPv6 traffic but otherwise behaves like a standard Ethernet switch.
drop
  not ethertype ipv4
  and not ethertype arp
  and not ethertype ipv6
;
# Block communitcation between clients https://zerotier.atlassian.net/wiki/spaces/SD/pages/222330881/Client+Isolation
tag server
  id 2
  enum 0 No
  enum 1 Yes
  default No;

break not tor server 1;
# Reject anything to services VM host other than ports used for its services
break
  ztdest xxxxxxxxxx
  and not dport 9443 # portainer
  and not dport 81 # nginx proxy manager
  and not dport 5556 # service A
  and not dport 22 # ssh
;
# Accept anything else. This is required since default is 'drop'.
accept;

When I save this, I cannot communicate with the services VM at all. I can communicate with the NAS, so I know it’s something specifically wrong about that rule, but I can’t figure it out.

[l0crian]

If you want to limit ZeroTier traffic to just client to server, it’ll likely be better to change the default behavior to drop instead of accept. You don’t really need tags because you don’t need that scalability. I’ve been liking using Macros in the rules to make things a little cleaner. You can do something like this:

# This is a default rule set that allows IPv4 and IPv6 traffic but otherwise behaves like a standard Ethernet switch.
drop
  not ethertype ipv4
  and not ethertype arp
  and not ethertype ipv6
;

# Allows traffic to server on a defined port; allows bidirectional traffic
macro serveraccess($port)
  accept
      ztdest xxxxxxxxxx and dport $port or
	  ztsrc xxxxxxxxxx and sport $port
  ;
;

include serveraccess(22)
include serveraccess(81)
include serveraccess(5556)
include serveraccess(9443)

drop;

Adding allowed ports to the server is as easy as just adding more “include serveraccess()” lines.

[vanburen.brian]

Thanks for your suggestions here. I actually prefer to have the default be drop instead of accept for security purposes, so I like that. The only challenge here is that I have the NAS on the network also that I’ll have to add some rules for. Those include a few ports for its web-based services but also for accessing the drives themselves for the file sharing (SMB). Looks like it might be pretty straight forward though based on this ( networking - What ports do I have to open for SMB sharing? - Super User ).

Are you aware of any mechanism to inspect the ZT traffic and troubleshoot potential issues? My main concern here is something doesn’t work, and I have no way to figure out why. With a firewall I can inspect logs to see why access may be blocked, but with ZT I’m not aware of anything like that and am concerned I’d have a hard time troubleshooting.

[l0crian]

You can modify the macro and pass through a node address as an additional variable. This will provide a little more utility, so that it can be used to filter access to both your NAS and that services VM.

# Allows traffic to server on a defined port; allows bidirectional traffic
macro serveraccess($ztaddress,$port)
  accept
      ztdest $ztaddress and dport $port or
	  ztsrc $ztaddress and sport $port
  ;
;
# Allow ARP messages; required when default drop
accept ethertype arp;

# NAS Access
include serveraccess(0123456789,445)   # smb

# Services VM Access
include serveraccess(fedcba9876,22)    # ssh
include serveraccess(fedcba9876,81)    # nginx proxy manager
include serveraccess(fedcba9876,5556)  # service A
include serveraccess(fedcba9876,9443)  # portainer

For SMB, I would only allow TCP/445. Ports 137-139 are for NetBIOS for network discovery, but it’s better to use internal DNS if you want to be able to access the share via a hostname. This does make it so you need to know an IP or hostname to access the share, it won’t pop up if someone’s host is looking for shares on the network, but is generally considered more secure. If you have a generally ubiquitous environment like Windows, then you can map the share to a users domain account via group policy.

And yeah, unfortunately a log to see which rule is being hit isn’t present to my knowledge, but hopefully that is something they’ll add later since it’ll be pretty useful. There is an API for network metrics that you can read about here: https://github.com/zerotier/ZeroTierOne/blob/dev/README.md#prometheus-metrics. While this doesn’t show which rule is being hit, it does have a hit count of packets between specific peers (look for ‘zt_peer_packets’). You can at least look at that to glean which direction the issue is in.

Beyond that, you kind of need to approach it like debugging code. Comment out sections, or add temporary rules to focus in on where in your flow rules the issue resides. Doing that along with tcpdump in the ZeroTier interface can be useful in seeing where traffic is/isn’t getting to.