Hi. My zt network has two geographically separated OpenWrt routers. The LANs are 192.168.1.0 (…196.1) and 192.168.2.0 (…196.111). Both have bridging enabled on the controller:
The 192.168.196.150 address is a cloud instance that does route override.
Is this the correct routing setup? Are the routes actually necessary, considering NAT already done by openwrt when udb port 9993 is correctly forwarded?
The docs say “additional setup needed on device” - presumably OpenWrt doesnt need any additional setup beyond the firewall zones/device settings, as it already has ip forwarding enabled…
Without knowing the specifics of your network, that looks fine.
It doesn’t look like you’re doing any bridging (stretching Layer2 between sites), you appear to only be doing routing (route between subnets at different site).
The routes will be necessary to talk between sites. UDP/9993 is part of the underlay. It is what is used to bring up the ZeroTier network, so it’ll be separate from all of your overlay routes (192.168.1.0/24, 192.168.2.0/24, 192.168.196.0/24).
The main additional setup on OpenWRT will be the firewall stuff you mentioned. The only additional thing you may need to do is allow the use of the default if/when needed. You would just do that when SSH’ed into one of your routers: Here’s the command (insert your networkID):
Based on your setup, you can remove bridging on the routers (as well in ZT Central).
The 2 sites will talk like you would talk to something on the internet (evaluating the Layer3 destination IP hop by hop). Here’s a quick packet walk:
Let’s say you have a host (Host1) at site 1 with an IP of 192.168.0.100 that wants to talk to 192.168.2.100 (Host2) at site 2.
Host1 generates a packet with Source:192.168.0.100 / Destination:192.168.2.100
It looks how to get to 192.168.2.100; noticing it doesn’t have a direct route, so it follows its default gateway to OpenWRT1.
OpenWRT looks at the packet and looks at its routing table and sees it has a direct route to 192.168.2.0/23 (which would include 192.168.2.100), via 192.168.196.111, so it forwards that packet to OpenWRT2 that owns the 192.168.196.111.
OpenWRT2 looks at its routing table and sees that 192.168.2.0/23 is directly attached, so it sends an ARP message to its local LAN to see what MAC address owns 192.168.2.100, and once received will send the final packet down to 192.168.2.100.
The return from 192.168.2.100 to 192.168.0.100 will simply reverse the source/destination and send back over the reverse path.
192.168.1.0/23 isn’t a valid subnet, since it falls in the middle of 192.168.0.0/23. It’s just correcting the route.
192.168.0.0/23 covers 192.168.0.0 to 192.168.1.255
If you wanted to cover 192.168.1.0-192.168.1.255, that would be 192.168.1.0/24
Another thing to add, if you SSH into your OpenWRT routers, you can run either of these commands to see the routing tables on those boxes (you can also see them in Luci, but I prefer to see them from Linux):
Yes, you’ve basically created a site-to-site VPN. If you wanted to have your own root servers, you’ll likely want to place them on the underlay, so how you do your site-to-site network won’t necessarily impact that.
Your routing tables should generally look like this (plus your other routes):
If one of the remote routes is missing from one of the site routers, that usually means (from what I’ve seen), that ZeroTier sees something local that would be overlapping at the site (or the gateway under the ‘via’ section is incorrect).
It could also be that the node doesn’t currently allow managed routes. We can do something similar to what we did for the default:
sudo zerotier-cli set <networkId> allowManaged=1
You can see your current ‘set’ settings using this cli command:
That seems to have fixed it! Routes are appearing on both sides. Also zerotier-cli peers shows that all connections are direct whereas before there were two or three relay leafs.
I just had to reinstall this local router… Dont know what happened, possibly i made a mistake in building zt from source. The way it was set up before i always saw all direct connections.
Thank you very much for your guidance, ive learned some from this.
