1.6.0 clients will not communicate off lan segment

Community Support
1

After upgrading to 1.6.0 I’m noticing odd networking behavior.

I have a zt network joined to my guest network (192.168.69.0/24) via a raspberry pi acting as a layer 2 bridge. The zt network occupies the upper quarter of the /24’s range (192 - 254). It’s configured as a /26 so that the zt connected resources will not be able to interact with other devices on the guest network but will be able to reach each other as well as selected resources on another network (192.168.28.0/23) via a firewall (192.168.69.193).

The ZT network is configured with two managed routes:

192.168.28.0/23 via 192.168.69.193
192.168.69.192/26 (LAN)

This config worked great for well over a year until I upgraded to 1.6.0 yesterday. Now the zt clients (3 macs and two ios devices) will only respond to traffic originating from IPs in the /26 but do not respond to anything in the /23. I have verified the bridge still works correctly by testing with pings from the firewall; the issue appears to be with the clients themselves.

Here’s a tcpdump from a client being pinged from the firewall:

IP 192.168.69.193 > 192.168.69.210: ICMP echo request, id 28542, seq 0, length 64
IP 192.168.69.210 > 192.168.69.193: ICMP echo reply, id 28542, seq 0, length 64
IP 192.168.69.193 > 192.168.69.210: ICMP echo request, id 28542, seq 1, length 64
IP 192.168.69.210 > 192.168.69.193: ICMP echo reply, id 28542, seq 1, length 64
IP 192.168.69.193 > 192.168.69.210: ICMP echo request, id 28542, seq 2, length 64
IP 192.168.69.210 > 192.168.69.193: ICMP echo reply, id 28542, seq 2, length 64

…and here’s a tcpdump from the same client being pinged from a different interface on the same firewall:

IP 192.168.29.198 > 192.168.69.210: ICMP echo request, id 11972, seq 0, length 64
IP 192.168.29.198 > 192.168.69.210: ICMP echo request, id 11972, seq 1, length 64
IP 192.168.29.198 > 192.168.69.210: ICMP echo request, id 11972, seq 2, length 64

Notice there’s no reply from the client at all.

Thoughts?

2

hello!
did the bridge interface get messed up somehow when zerotier restarted?

are the routes getting applied on your macs? the phones are on 1.6 too?

3

Ha! I thought it was the bridge interface at first and took it down and redid it. It was only after the problem remained that I began to look elsewhere.

The ios devices are running the latest version from the Apple Store. They show up in my.zerotier as 1.5.1 rather than 1.6.0 though.

It’s worth noting that all the traffic to the firewall traverses the bridge - I would think if it was the problem, both tcpdumps would look the same. Also the second dump shows the client isn’t even sending a reply for the bridge to drop.

4

Oh yes, I forgot to mention the routes show up in the client:

Destination Gateway Flags Netif Expire
192.168.28/23 192.168.69.193 UGSc feth826
192.168.69.192/26 link#19 UC feth826 !
192.168.69.193 link#19 UHLWIi feth826 !

5

The one thing jumping out at me is this route:

192.168.28.0/23 via 192.168.69.193

There’s no member on the network with the address 192.168.69.193 assigned, so to the clients running ZeroTier there’s no route back.

Your Bridge’s IP per ZeroTier’s configuration is 192.168.69.150. That’s outside the range of your /26 managed route. So the computers on ZeroTier have no route to actually get there.

6

It’s a layer 2 bridge, there’s no IP or route required for the bridge to work.

193 is the firewall on the other side of the bridge. The clients clearly can communicate with it (as shown in the tcpdump).

7

Looks like the pings are never making it to the clients. You can tcpdump on the ZeroTier adapter on the client itself to verify.

To me it points to a misconfiguration of your bridge/router/firewall.

8

The tcpdump are from the zt client. The pings are clearly making it there; it is clearly not replying to anything not in the lan segment (/26). Can you please read through my initial post before responding again?

9

I’ve got some new info.

I connected a mac running Catalina (macOS 10.15) and zt 1.6.0 client and it has full connectivity. The macs that will not communicate off of the /26 are running Big Sur (macOS 11.0.1) and zt 1.6.0.

I know Apple changed the networking stack between the two OSes so… that’s a thing.

BTW, the iOS device which exhibits the same, non-working behavior is running iOS 14.2.1. Unfortunately, I don’t have an older iOS device to test with.

10

Same behavior under 1.6.1

11

Maybe I’m seeing this as well:

13

New update. With all the Big Sur Macs updated to 1.6.1, I tried again and same result (no pinging off the /26). I deleted and recreated the managed route on my.zerotier (because, why not?):

192.168.28.0/23 via 192.168.69.193

And every mac works as expected; they’re able to reach everything in the /26 and the /23.

The iOS devices still do not work (even after updating to 1.6.0).

14

The issue returned this evening on one of the macs (but not all!)

Focusing on the problem zt client I checked the routes first and zt routes show up correctly:

	Destination		Gateway			Flags	Netif	Expire
	192.168.28/23		192.168.69.193		UGSc	feth826       
	192.168.69.192/26	link#8			UC	feth826      !
	192.168.69.193		link#8			UHLWIi	feth826      !

Satisfied I saw the correct routes I then pinged the gateway on the other side of the bridge to make sure it’s reachable:

	PING 192.168.69.193 (192.168.69.193) from 192.168.69.210: 56 data bytes
	64 bytes from 192.168.69.193: icmp_seq=0 ttl=64 time=1.987 ms
	64 bytes from 192.168.69.193: icmp_seq=1 ttl=64 time=2.309 ms
	64 bytes from 192.168.69.193: icmp_seq=2 ttl=64 time=18.675 ms
	--- 192.168.69.193 ping statistics ---
	3 packets transmitted, 3 packets received, 0.0% packet loss
	round-trip min/avg/max/stddev = 1.987/7.657/18.675/7.792 ms

Everything still looks good, so I ping devices on the 192.168.28/23 subnet and…

	PING 192.168.28.28 (192.168.28.28) from 192.168.69.210: 56 data bytes
	64 bytes from 192.168.28.28: icmp_seq=0 ttl=64 time=12.334 ms
	64 bytes from 192.168.28.28: icmp_seq=1 ttl=64 time=3.702 ms
	64 bytes from 192.168.28.28: icmp_seq=2 ttl=64 time=2.088 ms
	--- 192.168.28.28 ping statistics ---
	3 packets transmitted, 3 packets received, 0.0% packet loss
	round-trip min/avg/max/stddev = 2.088/6.041/12.334/4.498 ms

which is great then I ping a 192.168.29.x address (also in 192.168.28/23)

	PING 192.168.29.3 (192.168.29.3) from 192.168.69.210: 56 data bytes
	ping: sendto: No route to host
	ping: sendto: No route to host
	ping: sendto: No route to host
	--- 192.168.29.3 ping statistics ---
	3 packets transmitted, 0 packets received, 100.0% packet loss

I tried deleting and recreating the route in Zerotier Central (as I did before) and the results are the same.

15

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.