I have a ZeroTier client, running in a Docker container, setup as a gateway connected to multiple ZeroTier networks. When the container first starts and for a period of time afterwards, everything works correctly. The container and the clients on my local network can connect to a ZeroTier client running on a remote network. However, after a period of time, the connection dies and the container and clients can no longer connect to the remote ZeroTier client. It appears to be the same issue mentioned by bartmichu here: Connection problem · Issue #1349 · zerotier/ZeroTierOne · GitHub. What’s interesting is that it’s always the same network that stops working. I’ve also looked at the Coma Bug issue here: Coma Bug Update, but I’m not seeing the same symptoms as described. My Planet/Root servers never show RELAYED and everything in ZeroTier Central shows Online. I’ve even setup my own Moon servers to see if that helped, but the issue persists.
When the client is in this failed state, I am able to connect to other remote ZeroTier clients on different ZeroTier networks that the client is joined to just fine. It’s only the one particular ZeroTier network that is the issue. I complied the ZeroTier client with ZT_DEBUG=1 and when I look at the logs, I see the following:
learned new path 103.195.103.66/9993 to 778cde7190 (packet 60d0e984fa99d8da local socket 140003660416624 network 0000000000000000)
MAC failed for packet 2d419386b39b98e8 from af75b2954a(103.195.103.66/9993)
MAC failed for packet 686a550643ad9606 from af75b2954a(103.195.103.66/9993)
learned new path 103.195.103.66/9993 to 778cde7190 (packet 60d0e984faa1e5e4 local socket 140003660415040 network 0000000000000000)
learned new path 50.7.73.34/9993 to 61d294b9cb (packet 60d0ec31f047f132 local socket 140003660415040 network 0000000000000000)
MAC failed for packet 2ad312b6f78af479 from af75b2954a(103.195.103.66/9993)
MAC failed for packet 51d15a226c0c8adc from af75b2954a(103.195.103.66/9993)
MAC failed for packet 901e45ad2a971d22 from af75b2954a(103.195.103.66/9993)
MAC failed for packet 852cfb14a1169c41 from af75b2954a(103.195.103.66/9993)
MAC failed for packet b47361d99ee079e6 from af75b2954a(103.195.103.66/9993)
MAC failed for packet e217d9139824ab50 from af75b2954a(103.195.103.66/9993)
MAC failed for packet 121c956ed1d37f3c from af75b2954a(103.195.103.66/9993)
MAC failed for packet 5e25f0850a64418a from af75b2954a(103.195.103.66/9993)
MAC failed for packet a0dadf5ca54f6fac from af75b2954a(103.195.103.66/9993)
MAC failed for packet 8760510b76b364ea from af75b2954a(103.195.103.66/9993)
MAC failed for packet 97a62d16b7eec546 from af75b2954a(103.195.103.66/9993)
MAC failed for packet b677d31c07ed74d6 from af75b2954a(103.195.103.66/9993)
MAC failed for packet 3beaddd5b894717b from af75b2954a(103.195.103.66/9993)
learned new path 50.7.73.34/9993 to 61d294b9cb (packet 60d0ec31f05873d1 local socket 140003660416624 network 0000000000000000)
MAC failed for packet 617bfe9bea791259 from af75b2954a(103.195.103.66/9993)
MAC failed for packet ae5828f685ed2e0b from af75b2954a(103.195.103.66/9993)
MAC failed for packet cca1d61872b7ffd1 from af75b2954a(103.195.103.66/9993)
MAC failed for packet 8c695d33820e8919 from af75b2954a(103.195.103.66/9993)
MAC failed for packet 5e05d7a2b14d6749 from af75b2954a(103.195.103.66/9993)
Also while the client is in this state, using another ZeroTier client connected to the same local network and same ZeroTier network, I can connect to the remote ZeroTier client just fine. If I restart/recreate the container things start working again for a period of time, but will always eventually fail. I’ve also taken packet captures from the ZeroTier client when in this failed state and only see ARP requests, but no responses for the remote ZeroTier client.
I’m stumped and could use some help on where to look next. Thanks!