Zerotier routes not added for the correct interface

I’ve recently upgraded ZT to 1.8.4, and am noticing a problem with the way ZeroTier adds static routes on linux (CentOS 7). The routes are added upon startup of the zerotier-one daemon as expected, but they appear to be getting added before the ZT interface (zt44xi7gkv) is up in some cases. In my network I have three static routes corresponding to the ranges in rfc1918. Sometimes the routes get added via the zt44xi7gkv interface in which case traffic flows as expected. Other times they get added using my VM’s interface, venet0, in which case traffic to those ranges is not reachable. In these cases the zt44xi7gkv interface is still up, but I suspect it came up after the routes were already added. If I restart the zerotier-one daemon, some, or even all of them might get added correctly, or not. Is this a known issue?

Non-working case:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 venet0
10.0.0.0        100.64.10.47    255.0.0.0       UG        0 0          0 **venet0**
100.64.10.0     0.0.0.0         255.255.255.0   U         0 0          0 zt44xi7gkv
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 venet0
172.16.0.0      100.64.10.47    255.240.0.0     UG        0 0          0 **venet0**
192.168.0.0     100.64.10.47    255.255.0.0     UG        0 0          0 **venet0**

Semi-working case: (172.16.0.0, 192.168.0.0 correct, 10.0.0.0 not correct)

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 venet0
10.0.0.0        100.64.10.47    255.0.0.0       UG        0 0          0 **venet0**
100.64.10.0     0.0.0.0         255.255.255.0   U         0 0          0 zt44xi7gkv
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 venet0
172.16.0.0      100.64.10.47    255.240.0.0     UG        0 0          0 zt44xi7gkv
192.168.0.0     100.64.10.47    255.255.0.0     UG        0 0          0 zt44xi7gkv

Expected result:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 venet0
10.0.0.0        100.64.10.47    255.0.0.0       UG        0 0          0 zt44xi7gkv
100.64.10.0     0.0.0.0         255.255.255.0   U         0 0          0 zt44xi7gkv
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 venet0
172.16.0.0      100.64.10.47    255.240.0.0     UG        0 0          0 zt44xi7gkv
192.168.0.0     100.64.10.47    255.255.0.0     UG        0 0          0 zt44xi7gkv

Hello,
thanks for writing. There’s a similar report here https://github.com/zerotier/ZeroTierOne/issues/1498
(More the second half of the comments)

This will be easier to reproduce.

What happens if you leave and rejoin the network?

Leaving and then rejoining the network results in the routes being added correctly. After doing this, I then stopped the daemon and started it back up again, but the routes were no longer correct, pointing to the venet0 interface instead of the zt44xi7gkv interface. Leaving and rejoining is a fine workaround for now. Hopefully this can be resolved in a future update?

Definitely. Thanks for the info.

Hello again.
I can’t reproduce on centos 7 vm. Any ideas? Any network manager scripts?

I only have the usual ifcfg- network config scripts in /etc/sysconfig/network-scripts/. There doesn’t seem to be anything special about them. I found three entries in /etc/NetworkManager/dispatch.d/, but they aren’t custom. They look to be stock scripts for mail, time, etc.

@zt-travis Is there anything else I can do to help demonstrate this problem since you’re having trouble reproducing it on your lab vm?

hey thanks for checking. We have a fix, just need to get it into a release soon. See the github issue for a little more info if you like.