I have a setup I am testing and I cannot get my throughput past 65Mbps…
The setup is the following:
debian host → opnsense w/ zt → internet → MT RB5009 w/ zt → Palo Alto VM → windows host
The debian/opnsense side is running on a proxmox server connected to a 300mbps symmetric connection. Standard downloads and uploads from this side easily hit 275Mbps. The windows side is a 1.3Gbps/42Mbps Xfinity connection and I can easily hit those speeds on the Windows host to/from the internet.
Opnsense is running the zerotier client on one side and the RB5009 is running the Mikrotik zerotier container on the other side. Peer connections between the two are direct with no relay involved. Connectivity is pure layer 3 routing between the debian and windows hosts with no NAT involved.
The debian host is running openspeedtest server in a docker container on port 3000. Testing across the zerotier network I am getting 65Mbps down and about 40Mbps up. I created a port forward in the OpnSense firewall and testing across the internet to the OpnSense public IP on port 3000, I am easily getting 225Mbps down and 42Mbps up… So it is not an issue with the speed test server setup.
When running across the ZT network, CPU usage is low on the everything on the debian host side, and the RB5009 occasionally sees peak single cpu core usage no more than 60%. So I do not believe it is a resource issue.
If i had to venture a guess, the Palo Alto VM is preventing a direct connection. Palo Alto and ZeroTier generally don’t play well together unless you give Palo Alto lots of money to enable a sane NAT configuration.
Maybe I wasn’t clear, but the Palo is pure Layer 3 routing with no NAT involved. If you look again at my diagram, it is the Mikrotik RB5009 which has the Xfinity internet connection where the NAT is taking place… If you also read through my post again, you will see that I said the ZT were direct connection paths and that when I take ZeroTier out of the equation, which still uses the Palo Alto and the Mikrotik, I get 275Mbps from the debian server.
Doing a speed test from the Windows box behind the Palo and Mikrotik out to Ookla SpeedTest, I get full 1.3Gb/42Mbps through the Palo.
Looking at the RB5009’s processor, while it is an ARM64 it does seem like the clock speed can be kind of low (down to 300MHz). I’m not sure how its auto scaling works but it is a Marvell and they aren’t particularly performant in my experience. I’d check your clock speed at various times to see what’s reported.
The current version of ZeroTier is unfortunately bound to a single core but we’re working to change that in an upcoming version which should help in situations like this.
I was just able to complete an additional test from another Windows machine with the zerotier client on it. Download speed was 60Mbps and this was with the Mikrotik ZT interface disabled.
So the setup for this test was:
Windows host → Fortigate firewall → Mikrotik RB5009 → internet → OpnSense w/ zt → Debian host
This bypasses the ZT client on the Mikrotik (also made sure it was disabled) and makes a direct connection from the Windows host to the ZT client on OpnSense. Result was 60Mbps download and 42Mbps upload. So it lookslike ZT is being limited to about 60Mbps here…
And to show capability of the Windows host on internet speeds tests, test to the OOkla site via a web browser gives me 850Mbps down and 42Mbps up. The download limitation is the 1Gb connection to the Fortigate firewall.