Ubuntu 22.04 No ping / No connection / Not routing / Destination Host Unreachable

My first post… I’m a bit nervous.

For some reason, the “feature-rich” Ubuntu 22.04 network environment got me stumped - there must be a setting many people cannot find to correct this issue as the tickets get closed or abandoned.
My Windows 10 PCs are Zerotier networking perfectly, but Ubuntu 22.04 won’t. I have compared the dump files, the network and settings, the peers between Windows and Ubuntu and cannot find any obvious differences. I have also loaded Zerotier on a Raspberry PI2, Raspbian GNU/Linux 11 (bullseye) armv7l and it works fine.

As for Ubuntu remedies, I have tried everything Google can provide, from firewall, reinstall, remove configs, remove peers.

Any assistance appreciated. RX Packets = 0 - what would be the potential cause of that?
Maybe a software bug?

Ubuntu 22.04

$ ufw status
Status: inactive

Interface:
zteb4mbztl: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2800
inet 192.168.192.186 netmask 255.255.255.0 broadcast 192.168.192.255
inet6 fc9f:d233:d454:342f:47a9::1 prefixlen 40 scopeid 0x0
inet6 fe80::4423:b0ff:fe01:d6ed prefixlen 64 scopeid 0x20
inet6 fdb1:5644:912e:8477:4599:9354:342f:47a9 prefixlen 88 scopeid 0x0
ether 46:23:b0:01:d6:ed txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6417 bytes 1208221 (1.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 100 0 0 enp0s25
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 enp0s25
192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s25
192.168.192.0 0.0.0.0 255.255.255.0 U 0 0 0 zteb4mbztl

I am sure this will not matter
$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

$ traceroute 192.168.192.132
traceroute to 192.168.192.132 (192.168.192.132), 64 hops max
1 192.168.192.186 3048.801ms !H 3071.946ms !H

$ ping 192.168.192.132
PING 192.168.192.132 (192.168.192.132) 56(84) bytes of data.
From 192.168.192.186 icmp_seq=1 Destination Host Unreachable

$ zerotier-cli status
200 info 54342f47a9 1.10.1 ONLINE

$ zerotier-cli listnetworks
200 listnetworks b25644912e847746 MUISNET 46:23:b0:01:d6:ed OK PRIVATE zteb4mbztl fdb1:5644:912e:8477:4599:9354:342f:47a9/88,fc9f:d233:d454:342f:47a9::1/40,192.168.192.186/24

$ zerotier-cli listpeers
200 listpeers 10dad0cb51 - -1 - LEAF
200 listpeers 62f865ae71 50.7.252.138/9993;7293;2069 220 - PLANET
200 listpeers 778cde7190 103.195.103.66/9993;7293;8212 276 - PLANET
200 listpeers b15644912e 35.206.79.5/42863;2289;2034 257 1.10.1 LEAF
200 listpeers c5b41e5297 - -1 - LEAF
200 listpeers cafe04eba9 84.17.53.155/9993;3350;7124 169 - PLANET
200 listpeers cafe9efeb9 104.194.8.134/9993;7293;6989 304 - PLANET
200 listpeers da41607c8e - -1 - LEAF
200 listpeers dc90349aaa - -1 - LEAF
200 listpeers e8c1c916f1 - -1 - LEAF
200 listpeers f463655262 - -1 1.10.1 LEAF

Raspberry PI

zteb4mbztl: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2800
inet 192.168.192.100 netmask 255.255.255.0 broadcast 192.168.192.255
inet6 fdb1:5644:912e:8477:4599:9310:dad0:cb51 prefixlen 88 scopeid 0x0
inet6 fc9f:d233:d410:dad0:cb51::1 prefixlen 40 scopeid 0x0
inet6 fe80::4467:5eff:fefe:5a15 prefixlen 64 scopeid 0x20
ether 46:67:5e:fe:5a:15 txqueuelen 1000 (Ethernet)
RX packets 9115 bytes 2046099 (1.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 91 bytes 11242 (10.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

#ping 192.168.192.132
PING 192.168.192.132 (192.168.192.132) 56(84) bytes of data.
64 bytes from 192.168.192.132: icmp_seq=1 ttl=128 time=488 ms
64 bytes from 192.168.192.132: icmp_seq=2 ttl=128 time=483 ms

zerotier-cli status
200 info 10dad0cb51 1.10.1 ONLINE

Etc…

Installed Ubuntu Desktop 22.04 on another machine, also the same result as above, ZeroTier on that version of Ubuntu is broken, also tried the snap package install instead of the curl script method. Only option I have not tried is a ZeroTier Docker image on Ubuntu 22.04.

Added another Linux based Raspberry Pi (4) to the network an it is working fine.

Hello,
thanks for writing!

from the output of listpeers, it’s not making direct connections to many of the nodes.
I see ufw is off, but can we see what is in iptables? iptables-save or iptables -S

Can it ping itself at 192.168.192.186?

Sure! Anything to to help. As requested. But remember I uninstalled and installed the snap package. There are changes.

OS: Ubuntu 22.04.1 LTS x86_64

'# zerotier status
200 info c73cc018eb 1.10.0 ONLINE

zteb4mbztl: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.192.114 netmask 255.255.255.0 broadcast 192.168.192.255
inet6 fe80::44b0:b8ff:feee:89af prefixlen 64 scopeid 0x20
ether 46:b0:b8:ee:89:af txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 6 overruns 0 frame 0
TX packets 41539 bytes 4463982 (4.4 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

'# zerotier-cli listpeers
200 listpeers
200 listpeers 62f865ae71 50.7.252.138/9993;12185;1949 225 - PLANET
200 listpeers 778cde7190 103.195.103.66/9993;12185;11908 329 - PLANET
200 listpeers b15644912e 35.206.79.5/42863;10638;10638 309 1.10.1 LEAF
200 listpeers cafe04eba9 84.17.53.155/9993;5739;5514 174 - PLANET
200 listpeers cafe9efeb9 104.194.8.134/9993;12185;11879 305 - PLANET
200 listpeers da41607c8e - -1 - LEAF
200 listpeers dc90349aaa - -1 - LEAF
200 listpeers f463655262 - -1 1.10.1 LEAF

'# Generated by iptables-save v1.8.7 on Thu Sep 8 18:11:20 2022
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p udp -m multiport --dports 5353 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 4000 -j ACCEPT
COMMIT
'# Completed on Thu Sep 8 18:11:20 2022

'# ufw status
Status: inactive

'# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 100 0 0 enp0s25
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 enp0s25
192.168.1.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s25
192.168.192.0 0.0.0.0 255.255.255.0 U 0 0 0 zteb4mbztl

Can it ping itself at 192.168.192.186? Yes, new IP

'# ping 192.168.192.114
PING 192.168.192.114 (192.168.192.114) 56(84) bytes of data.
64 bytes from 192.168.192.114: icmp_seq=1 ttl=64 time=0.021 ms
64 bytes from 192.168.192.114: icmp_seq=2 ttl=64 time=0.058 ms
^C
— 192.168.192.114 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 0.021/0.039/0.058/0.018 ms

Let me know if you need anything else.

thanks. looks like iptables is wide open.

is this machine behind the same router that the working machine is behind?
I’d assume so, but… it’s not working and the others are.

Hmm, the MTU on the zt interface should be 2800. That’s the default for zerotier networks anyways.

Is some kind of ubuntu network manager thing messing with the zerotier interfaces?

is this machine behind the same router that the working machine is behind?

Yes - it is.

I tried the Docker container, same result. The machines are connected to a switch (no network management configured/hub mode), then NAT router → internet → NAT Router → WiFi → Win 10 (I do remote desktop support for my 90 year old mom!). The working Win 10 PC is connected to the next port on the switch (i have swapped the cables around as well). I must be running this beautiful ZT network and Nomachine setup flawlessly for 2 years, just never tried ZeroTier on a Linux machine.

MTU on the zt interface should be 2800.

That was my doing. I tested the 1500 MTU limitation, had a Cisco config issue once. Will change it back. Yes, Ubuntu 22.04 has Network Manager in place of Systemd-networkd , and I must admit it is still dark arts for me.

I used the command ifconfig zteb4mbztl mtu 1500 up will change it back, forgot about it.

I downloaded the ZeroTier source code from GitHub and built it. Same result as the package installs. I wish I had the skills to read the code.

Something strange is going on!

Is there a VPN running on the ubuntu?

Lot of people including ZeroTier devs are using ZeroTier on Ubuntu 22.04.

I’d say look into the router, but you have multiple other devices behind it working…
What happens if you power cycle it? Or click the “clear NAT state” button if it happens to have one.
If that’s not too disruptive.

1 Like

Found it! It was a relentless effort. I watched a youtube video where the protocol is explained.

Now I have:

zteb4mbztl: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2800
inet 192.168.192.93 netmask 255.255.255.0 broadcast 192.168.192.255
ether 46:71:c0:c6:84:dd txqueuelen 1000 (Ethernet)
RX packets 868 bytes 289006 (289.0 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1212 bytes 258935 (258.9 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

And a ping! And a Connection! And routing.

'# ping 192.168.192.132
PING 192.168.192.132 (192.168.192.132) 56(84) bytes of data.
64 bytes from 192.168.192.132: icmp_seq=1 ttl=128 time=379 ms
64 bytes from 192.168.192.132: icmp_seq=2 ttl=128 time=376 ms

I can’t explain why Win 10 and the Raspbian OS has less of an issue with it.

That fixed it.

Thank you so much for having a look at the issue.