Zerotier on RPi keeps dropping packages

Hello everyone,
I am trying to setup Zerotier on my Raspberry Pi 4B.
It works flawlessly on my Android and Windows devices but not yet on my Linux RPi.

I wrote myself a small script to try and recreate my installation process and help find the problem. No luck so far.

I did not uninstall anything! So it is a basic Raspberry Pi OS which is up to date (fresh install).

Does anybody see what the problem is? Could the bridge be the problem? Or does zerotier on RPi require something else to work? Or something else you might see?

I want to access the server running zerotier as well as all devices behind it. This works from all other clients, so only the RPi is not stable. I am able to ping and connect to devices, but e.g. SFTP keeps disconnecting after a few KB and pings drop a lot of packages.
I am trying to also bridge the zerotier network to other devices using an AP, so don’t be surprised about my bridging attempt).

ping 192.168.0.101
PING 192.168.0.101 (192.168.0.101) 56(84) bytes of data.
64 bytes from 192.168.0.101: icmp_seq=1 ttl=64 time=200 ms
64 bytes from 192.168.0.101: icmp_seq=2 ttl=64 time=88.1 ms
64 bytes from 192.168.0.101: icmp_seq=3 ttl=64 time=127 ms
64 bytes from 192.168.0.101: icmp_seq=4 ttl=64 time=105 ms
64 bytes from 192.168.0.101: icmp_seq=5 ttl=64 time=88.0 ms
64 bytes from 192.168.0.101: icmp_seq=6 ttl=64 time=400 ms
64 bytes from 192.168.0.101: icmp_seq=20 ttl=64 time=469 ms
64 bytes from 192.168.0.101: icmp_seq=21 ttl=64 time=90.0 ms
64 bytes from 192.168.0.101: icmp_seq=22 ttl=64 time=97.6 ms
64 bytes from 192.168.0.101: icmp_seq=23 ttl=64 time=98.7 ms
64 bytes from 192.168.0.101: icmp_seq=24 ttl=64 time=88.0 ms
64 bytes from 192.168.0.101: icmp_seq=43 ttl=64 time=182 ms
64 bytes from 192.168.0.101: icmp_seq=44 ttl=64 time=83.6 ms
64 bytes from 192.168.0.101: icmp_seq=45 ttl=64 time=89.5 ms
64 bytes from 192.168.0.101: icmp_seq=46 ttl=64 time=79.8 ms
64 bytes from 192.168.0.101: icmp_seq=47 ttl=64 time=80.5 ms
64 bytes from 192.168.0.101: icmp_seq=48 ttl=64 time=86.1 ms
^C
--- 192.168.0.101 ping statistics ---
48 packets transmitted, 17 received, 64.5833% packet loss, time 48284ms
rtt min/avg/max/mdev = 79.836/144.246/468.522/111.682 ms


My current script:
#!/bin/bash

sudo apt-get install iptables-persistent
curl -s https://install.zerotier.com | sudo bash
sudo zerotier-cli join MYNETWORK

sudo zerotier-cli listnetworks
sudo zerotier-cli set MYNETWORK allowManaged=0
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf;
sudo systemctl enable systemd-networkd;
sudo systemctl enable systemd-resolved;
sudo systemctl enable systemd-timesyncd;
sudo zerotier-cli set MYNETWORK allowManaged=0
cat << EOF | sudo tee /etc/systemd/network/25-bridge-br0.network
[Match]
Name=br0

[Network]
Address=192.168.0.3/24
Gateway=192.168.0.1
DNS=192.168.0.1
EOF

cat << EOF | sudo tee /etc/systemd/network/br0.netdev
[NetDev]
Name=br0
Kind=bridge
EOF

cat << EOF | sudo tee /etc/systemd/network/25-bridge-br0-zt.network
[Match]
Name=ztwdjlnlsc

[Network]
Bridge=br0
EOF

cat << EOF | sudo tee /etc/systemd/network/25-bridge-br0-wlan1.network   
[Match]
Name=wlan1

[Network]
Bridge=br0
EOF
sudo zerotier-cli set MYNETWORK allowManaged=0
sudo tail -n+0 /etc/systemd/network/*
sudo echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables
sudo iptables -A FORWARD -p all -i br0 -j ACCEPT
sudo iptables-save
sudo iptables-save > /etc/iptables/rules.v4
sudo ip6tables-save > /etc/iptables/rules.v6
sudo systemctl enable netfilter-persistent

Check the output of zerotier-cli peers. If there are lots of entries with RELAY, you likely have a firewall blocking traffic, or a router preventing direct peer to peer communication.

Thank you @zt-grant , I will try that during my next iteration.

I found an AP tool that would make AP mode much more convenient but cannot yet get it to work with zerotier. So I need to solve that before returing to fixing this :frowning:

Opened an issue here: RaspAP with zerotier

Seems maybe zerotier and RaspAP don’t like each other out of the box.

So, I checked the peers and only one is a RELAY. The only RELAY is my Windows 10 PC, which was not running during testing.

Maybe I have my basics wrong though.

Let’s call the RPi I am tryint ot get to work “client” and my other one at home “server”.

If I want to access my zerotier network at home, do I need a bridge on my client? Or should it work by simply installing and joining?
My server is setup using this tutorial Bridge local network

I want to be able to access the server and the LAN behind it from my client. And of course I still need to be able to access the internet.

I have now made a clean install and only run these commands (I removed some info from name of course):

sudo apt-get install iptables-persistent
curl -s https://install.zerotier.com | sudo bash
sudo zerotier-cli join d5e5fb6537
sleep 2
sudo zerotier-cli listnetworks
sleep 2
sudo zerotier-cli set d5e5fb6537 allowManaged=1
sudo zerotier-cli set d5e5fb6537 allowGlobal=1
sudo zerotier-cli set d5e5fb6537 allowDefault=1
sleep 2
sudo sysctl -p
sleep 2

I did not add any routing or similar. It is otherwise a fresh system.

Here is my route info. I noticed that inet and broadcast are not the same IP range. Is this normal?
Happy to supply any other info :slight_smile:

pi@raspberrypi:~ $ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     128.0.0.0       UG    5000   0        0 zt12345678
default         www.huaweimobil 0.0.0.0         UG    303    0        0 wlan0
128.0.0.0       192.168.0.1     128.0.0.0       UG    5000   0        0 zt12345678
192.168.0.0     0.0.0.0         255.255.255.0   U     303    0        0 wlan0
192.168.0.0     0.0.0.0         255.255.254.0   U     0      0        0 zt12345678

pi@raspberrypi:~ $ ifconfig
eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether dc:c6:32:c6:1a:89  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 275  bytes 27283 (26.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 275  bytes 27283 (26.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.39  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::52f9:c0c3:c247:c519  prefixlen 64  scopeid 0x20<link>
        ether dc:c6:32:c6:1a:89  txqueuelen 1000  (Ethernet)
        RX packets 5150  bytes 3002029 (2.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3970  bytes 793720 (775.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

zt12345678: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2800
        inet 192.168.0.3  netmask 255.255.254.0  broadcast 192.168.1.255
        inet6 fe80::74ce:c1ff:fe13:3da9  prefixlen 64  scopeid 0x20<link>
        inet6 2a02:810d:c200:688:74ce:c1ff:fe13:3da9  prefixlen 64  scopeid 0x0<global>
        ether 76:ce:c1:13:3d:c9  txqueuelen 1000  (Ethernet)
        RX packets 3180  bytes 2538977 (2.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2131  bytes 318466 (311.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

pi@raspberrypi:~ $ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
pi@raspberrypi:~ $ 

From my understanding, in therory, without adding any routing, iptables rules etc., I should be able to ping my server right out of the box, should I not?

Will continue the discussion here on reddit so we do not have double threads.

Right now it seems that I cannot get zerotier up and running.
Error connecting to the ZeroTier service:Please check that the service is running and that TCP port 9993 can be contacted via 127.0.0.1.

Additional info on how RaspAP works were given:

  1. RaspAP uses minimal iptables rules to configure NAT only. It does not block any ports or manipulate traffic beyond basic FORWARD policy chains for a routed AP and, optionally, WireGuard and OpenVPN. Check the rules on your system with sudo iptables -L -v -n.
  2. dhcpcd is used to configure TCP/IP across all of the available network interfaces. dnsmasq is used for DHCP and DNS. Bridging is done between the AP interface (wlan0 is typical) and eth0 only. No other interfaces are used. systemd-networkd is used to create and populate the bridge. The exact method is functionally identical to the official Raspberry Pi bridged AP tutorial, so suggest you look there for details.

RaspAP’s default settings are completely exposed and may be modified to suit your needs Default settings - RaspAP Documentation. These settings are contained within the /config folder

Unfortunately I am still struggling to understand the different network approaches of different operating systems and apps. So I cannot say if one of the above is causing the issue.

Does zerotier support debugging? Could not find anything.
I would like to find out what is preventing the service from starting.

Service seems to be running

service --status-all
[+] zerotier-one

So i just saw this in your script above. d5e5fb6537 is not a network ID. That’s the Node ID of the network controller. The network ID will have 16 digits, starting with the 10 digits of the network controller ID.

Additionally, it looks like you’re attempting to do a hardware bridge to a Wireless network. I don’t think that’s allowed per the 802.11 specifications.

The network ID was shortened for privacy sake (stated above). I am of course using the correct one I always use :slight_smile:

What do you mean with hardware bridge?
Yes, I have two wifi interfaces (one built in and one usb). One of those is a hotspot and the other a client. This should not be an issue (and explicitely supported by RaspAP). Why/how would this prevent zerotier from running?

P.S.: In addition to checking the service I also ran telnet 192.168.0.39 9993.
So I checked the port using the IP of my wlan0. And it connected. So the port should be open, if I am not mistaken. Or can/must I perform the test differently?

You’re creating a bridge device (br0) with a ZT interface and a Wireless interface. This isn’t always supported. I don’t recall if it’s a driver support thing for the wireless card, or an 802.11 support thing.

1 Like

Ah, okay. So this might only with with eth0?
But this is then only on linux? Because on Win and Android zerotier works perfectly fine with wifi.

But I have now created a fresh setup without the bridge. So right now only zerotier was installed and RaspAP was installed. Zerotier is not part of the bridge yet as it will not even start as long as RaspAP is running (full description in reddit and also main information above… check service and port…).

Correct, your bridge may only be able to be done with an ethernet port, not with a wireless port.

I’m not folloiwng you here. If you have a linux device joined to a wireless network, of course ZeroTier will work. You just may not be able to bridge the zerotier device with a wireless device.

Maybe I should open a fresh thread.

I am trying a new approach. For now, there is no bridge involved with zerotier.

I am running RaspAP with wlan0 as client and wlan1 as AP.
I then installed zerotier. But I cannot even join zerotier because I am getting the service/pott error I posted earlier. At the stage, no bridge is involved. Zerotier won’t even run.

As noted by Automoderator on Reddit, we ZT devs pay much closer attention over here, than on reddit

Also, your own message above says ZeroTier appears to be running. How are you determining that it’s not running?

What specific error messages are you seeing?

Because this is what I get when I try to join my network or list networks or do anything else with zerotier-cli.

Plus, there is no zt interface being created (none listed with ifconfig).

Then you likely have something blocking 127.0.0.1:9993. I’ve never used this “RaspAP” package you’re speaking of but i’m betting it messes with IPTable rules quite extensively and is likely doing something to block access.

Try running without RaspAP. If ZeroTier works before RaspAP is installed, and doesn’t after, it’s RaspAP causing your issues.

According to RaspAP support, they do not.

They do say they’re setting up NAT & Forward rules in iptables. I’m betting that’s where your issue is. Try running ZeroTier without RaspAP installed. If ZeroTier works before RaspAP is installed, and doesn’t after, it’s RaspAP causing your issues.