Can see bidirectional traffic between nodes on physical network but not inside zt network

[yellowlm]

Hello guys,

I have a few Linux (Rocky 8) nodes connected through zerotier. All of them have public IPs (v4+v6) and there is no NAT between them. (Except on the machine where zerotier is installed. These are routers themselves.) However, one of them is weird recently. On the physical network (multiple pppoe links), I can see traffic (v4+v6) coming and going with other nodes using tcpdump. But I cannot ping anyone inside the zerotier network (or do anything else with other nodes). Can someone suggest some directions I can look at? Any suggestions would be appreciated.

Here are what I already have:

1. iptables and ip6tables do allow incoming traffic to zt’s primary and secondary ports. Outgoing traffic is not restricted. ICMP and ICMPv6 are allowed from any to any on all interface.
2. zerotier central shows the IPv6 address of the problematic nodes and it reports last seen less than a minute.
3. zerotier-cli peers shows all peers are directly connected except for one planet server.
4. zerotier-cli bond list (I’m using multipath, balance-aware) shows nodes are connected through multiple links and if I zerotier-cli bond show anyone of them. I could see a list of links with latency, eligibility, etc. (some links are not eligible but many are.)
5. tcpdump reports only the local host’s outgoing traffic on the zt interface. No broadcast or anything from any nodes.

PS: I have some weird routing policies (using iptables -j MARK and ip rules) on some of the nodes but they seem not to affect zt’s connectivity on other nodes.