I have the following setup:
Node1 → router1 (wan ip, 100.1.1.1, assigned ip from the building network) → public ip (building fiber internet some valid ip here a.b.c.d)
Node2 → router2 (wan ip 100.1.1.2, assigned from the building network) → public ip (same as above a.b.c.d)
Now when I connect between node1 and node2, it will be on relay mode and it is slow. But I can access to either node1 or node2 with direct connection from any node outside a.b.c.d.
I think it is theoretically possible node1 and node2 can be connected in direct mode? Any hints on how I could do it?
Yeah, that’s a bit odd. Have you tried pinging each other using the wan address inside the building network (ie 100.1.1.x)? I mean, if that works then ZT should be able to establish a direct link between them.
That is quite common that customers are not allowed to communicate directly between each other.
Use of your own server in the close proximity to a.b.c.d may help, see Private Root Servers | ZeroTier Documentation
Possibly at L2 level to prevent DHCP spoofing etc but with public L3 issues I’d say that specific subnet is misconfigured. Also, CGNAT should normally allow inter L3 communication and only in that case will Private root servers help. But maybe that “Building network” doesn’t follow best practice…
@Honglet, what does the local “Building” subnet look like?
I have limited knowledge to that. It used to be the case that 100.1.1.1 and 100.1.1.2 can’t ping each other. Then I called the tech support and asked them to put these two addresses under the same VLAN. Now they can ping each other, but zt couldn’t establish a direct connection.
I installed zerotier on one of the routers, which is node 8e1b*. You can see that I can have a direction connection. However, node 7dbc*, which is within this router, is in relay mode.
You might think I’m asking a lot of stupid questions but it’s somewhat hard to follow because I’m missing a network topology that explains the different parts of your setup, but anyhow here we go:
8e1b is your main router (btw, what model is it?)
192.168 is your local network?
What WAN (internet) ip address du you get on you router?
7dbc is a node on you local network?
I’m just guessing here but it looks like you ran into a double-nat problem since the router is able to get a direct link. Just curious, why do you need ZT on your local devices (like the 7dbc) if the router can manage all the ZT network traffic for you?
Because my current router is slow, when it handles all zt traffic, it is quite slow. Until I get a dedicated routing machine, I’d like to be able establish direct connection.
Ok, then there is only one piece missing and that is the building network. If the whole building is sharing 100.1.1.1 as the public ip there must be an intermediate network between you and that address.
Picture this:
A (LAN ) <=> B (Building network/router w NAT) <=> C (Internet)
Where:
A: is your LAN (192.168)
B: is the building network that connects you router A to internet C via a dedicated building router B using NAT (or possible CG-NAT)
C: is the public IP 100.1.1.1 shared between all users in the building.
What does these commands say using the OpenWRT ssh login:
root@OpenWrt:~# ip address
root@OpenWrt:~# ip route
root@OpenWrt:~# ifconfig
root@OpenWrt:~# ifstatus wan
Maybe I didn’t make it clear. 100.1.1.1 is the static ip that is assigned to my router by building’s network. The building has another public ip which is something a.b.c.d that is different than 100.1.1.1.
Let me do this
N1 (some device) <=> A1 (router, with openwrt installed and runs zerotier) <=> B (building router) <=> N0 (internet)
N2 (another device) <=> A2 (router) <=> B (building router)
Now N0 can have direct connection using zt with either N1 and N2 without any problem.
N2 can occasionally connect directly to A1 using zt.
N1 and N2 as always in relay mode.
What I am trying to fix is to have N1 and N2 be in direct connection mode.
Ok, I get it. However, then you actually get your own public IP (100.1.1.1 from Verizon) that can’t be shared by anyone else. What IP do you get from whatismyipaddress.com?
I think this is the last confusion. 100.1.1.1 is a static in the sense that it is a internal ip from the building network, but it is not a public ip. In whatismyipaddress.com, all devices will see a.b.c.d which is the shared ip by the entire building network.
Then you probably have two different issues that need to be addressed.
First I have to ask if you are just using the IP 100.1.1.1 as a hypothetical example? The reason I’m asking is that the IP range actually belongs to a public network and using it as an internal intermediate private network (ie as in the building network) might cause serious side effects.
And secondly, because your router WAN address 100.1.1.1 differs from the public IP we are back to the double-NAT problem, ie A (LAN IP 192.168 - NAT1 ) => B (Building subnet 100.1.1.x - NAT2) <=> C (Internet IP a.b.c.d).
IP hypothetical flow:
N1 ZT Node1 = 192.168.1.10
A1 router: gw addr 192.168.1.254 NAT-1 using WAN address 100.1.1.1 to B router at 100.1.1.254
Building router 100.1.1.254 NAT-2 to internet using WAN address a.b.c.d
The other way around to N2 (NAT3/NAT4)
If the path through N1 NAT1/NAT2 and on to N2 NAT3/NAT4 cannot be opened by, for example, hole punching, ZT won’t be able to establish a direct link. If the building’s internal network allows direct communication at IP level within the same subnet (as it ought to), you should be able to communicate directly between 100.1.1.1 and 100.1.1.2 but since this doesn’t work, something is missing. Sometimes it might help to open “helper ports” in the router and point those to the zt node using dst-nat.
However, it will be difficult to continue this hypothetical discussion since you are not willing to share more of concrete details.
Now I really understand your dilemma and given what you explained here there really shouldn’t be any problem with a direct link between N1 and N2. Really strange indeed!
My guess is that there is a configuration issue in the router(s) and/or possibly in the nodes as well. What os are you using to run the nodes btw?
Just to rule out any possible odd firewall rules or possble double nat if you happen to use masquerade/src-nat in also the n1/n2 nodes you might temporary connect (route) your 1 and 100 networks together by bypassing the firewall and add routes like “route add 192.168.100/24 dest 10.65.1.142” on A and “route add 192.168.1/24 dest 10.65.1.147” on router B.
If you’re able to reach all nodes on both networks, you can test again and see if ZT can possibly establish a direct link this time.
