I recently upgraded a CentOS 7 vps running openvz to zerotier 1.6.4. After the upgrade zerotier would no longer start, complaining about not being able to open the tun device:
$ /usr/sbin/zerotier-one
ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory
The permissions on the tun device appear correct and the tun module still appears to be enabled in the kernel.
$ ls -al /dev/net/tun
crw-rw-rw- 1 root root 10, 200 Feb 23 06:44 /dev/net/tun
$ cat /dev/net/tun
cat: /dev/net/tun: File descriptor in bad state
I noticed that if I prevent zerotier from dropping privileges (-U) it will start, but it will remain in TUNNELED mode. I ruled out any iptables issues by disabling the firewall. Everything is functional in tunneled mode, just slow.
Are these known issues? Are there any suggestions for how to further troubleshoot?
Yes! SELinux was set to targeted (and always has been), however disabling it as a test has now let this new verseion of zerotier start without having to disable the permissions drop. I’d have thought the appropriate SELinux permissions would be set as part of the [rpm] installation.
Unfortunately disabling SELinux did not resolve the main issue - i’m still only able to run in tunneled mode.
Tunneled mode is a tough one, usually routing related, double NAT for sure causes issues. if you traceroute is your 2nd hop another local ip/different local subnet.
The vps has a public ip address, so there’s no rfc1918 and therefore no nat. The next hop gateway is the first address on the same subnet as the vps’ address.
Ensure iptables/firwalld/any other iptables config wrappers aren’t blocking ZeroTier from incoming & outgoing UDP packets. At least port 9993 must be able to send & receive UDP.
If I ping this hosts zerotier address while running tcpdump on it’s external interface, I can see udp/9993 coming inbound into the box but I never see udp/9993 sent. Not a single packet.
My symptoms appear to be similar to those described by a few users in an old issue reported on github
What can I do to further troubleshoot why zerotier isn’t responding to the incoming udp/9993 packets?
Are you running Centos inside an OpenVZ container/vps? It’s likely an OpenVZ configuration issue. OpenVZ VPS’s are more like containers than full VMs, so settings on the host also apply to the containers running on it. I’d start looking there. Beyond that I don’t have any other suggestions.
Yes, this CentOS 7 system is running inside on OpenVZ 7. Since it’s a vps I don’t have access to the host to view it’s settings, but the tun interface does seem to be working as expected, otherwise zerotier wouldn’t function at all. Could the provider somehow have enabled the tun interface but only for tcp and not udp? Seems unlikely?
I agree, it’s possible there could be a host configuration issue, but I know this problem started when I upgraded zerotier from 1.4.6 to 1.6.4. If there are no other ways to troubleshoot this I guess i’ll have to roll back to 1.4.6 and see what happens.
Just a quick update on my progress. I rolled back to 1.4.6 and continue to experience the problem with being stuck in tunneled mode. I set allowTcpFallbackRelay to false in my local.conf and then started capturing traffic on the external interface with tcpdump, expecting to find traffic (udp/9993) going to one of the four ipv4 addresses associated with root.zerotier.com. I didn’t see any traffic to those addresses so I ran an strace against the pid of the zerotier-one process and found the system calls where it’s trying to send packets to those servers. The sendto() system call seems to be returning ENETUNREACH when trying to talk to the roots. There are many occurrences of this error.
FD #11 looks to be the udp/9993 socket and there are similar failed system calls for another udp port - 61004 which is also associated with the zerotier-one process. There is a default route in the routing table and there are no other interfaces besides eth0 and the interface zerotier creates. All four root servers respond to ping so there isn’t a routing issue; something else must be causing the ENETUNREACH. I’m a bit beyond my level of expertise but really need to solve this problem. Suggestions welcome.
Same issue also appearing in 1.6.4 on an OS X client. Issue does not appear with other clients with a different version on the same network and port forwarding has no impact.
@monte you can try stopping zerotier and deleting your peers.d directory in /var/lib/zerotier-one and starting zerotier, if 1.4.6 is still stuck after downgrading