I have a relatively simple existing configuration that has been rock solid and worked perfectly fine for months. In summary, two networks (A & B) with two different private address spaces, a few hosts on each. A Pi4 on each network with the zerotier client configured, the network config on the Pis set up to route traffic for the counterpart network via zerotier.
Network A has a router (UDMP) configured with a route to network B via the Pi on its local network.
Network B has a router (ASUS AC3100) configured with a route to network A via the Pi on its local network, the Pi is directly connected to the switch in the router. In addition, network B has a second ASUS router meshed with the first (master) over WiFi, there is no ethernet backhaul. Clients associated with the APs built either of the network B routers have full visibility of the remote network via the Pi gateway, everything works as advertised.
Yesterday, I relocated the Pi4 running the zerotier client from the master AC3100 router to the other one in the mesh - as in connected to the switch built into the meshed router. There is where things become strange. At this point, from a laptop connected via WiFi on network B, I can ping systems on network A, I can ssh into systems on network A, but I cannot access a web server running on network A, and a VOIP phone cannot access the phone switch on network A. If I relocated the zerotier Gateway Pi back to the master router, HTML and SIP begin to work again. I could imagine the configuration working or not work, but not partially working, which appears to make no sense.
It is unclear to me at this point if there is some odd behavior going on in the ASUS mesh implementation, and/or coupling in some way with how zerotier is implemented. One further item, I know the mesh is working since there is an IP camera connected to the mesh router that is accessible from both sites quite happily.
I have resolved the situation for now by leaving the gateway in its original location, but since I am beginning demolition of that part of the building tomorrow, I really need to move it, and would also really like to fill the gap in my knowledge.