LAN to LAN using ZeroTier

[robert.perry]

We have tried to use ZeroTier in several different use cases. The RAS type VPN access seems like a great solution, and works with almost perfect access. However, the situation we face problems with is the site to site configuration. We have tried with Windows hosts as well as OPNSense routers, and seem to get similar results. I am hoping someone can shed some light on what we need to do to get the same reliability as we have with RAS type VPN access.

Fresh build of two nodes of OPNSense routers for example, with one being the head end at a site that has the 10.10.0.0/16 network, and the other that has the 10.11.0.0/16 network. We have also tried smaller subnets/networks to confirm our findings. Our plans were to use Z/T to connect these sites together and utilize the routing between them. Basically a site to site router over the Z/T network.

Basic build of OPNSense with firewall disabled, no IPV6, and the ZT plugin installed. Fully updated as well.

All traffic between the ZT interfaces works fine and no errors or issues. That is until we add the /16 or other subnets as routes within the ZT portal. At that time, we start to drop packets and in most cases have an unusable ZT connection. We know ZT is stable in other cases as we have used it as a RAS VPN into several different locations for support.

After further testing, we have found that adding specific subnets that do NOT include the interface address of the ZT host appear to work fine. You can also add specific hosts to the routes and it works fine. We just wanted to specify the network for the entire location/network, and not be limited.

Is there something we are missing? Maybe we just don’t understand the route options.

This is the same for any host we have tried to do this with. Windows, Linux, VMs based on OPNSense, etc.

Any advise or help is greatly appreciated.

[zt-grant]

Sounds like you’re another victim hitting the “Software Laser” mode, covered in this GitHub Issue. BSD, the underlying OS to OPNSense, lacks a system call available on other systems that allow us to check & prevent sending ZeroTier packets over ZeroTier and in these situations, especially with bridging involved, ZeroTier will end up in a feedback loop.

To prevent this, you need to dig into the local.conf file. On the OPNSense router on Site A, we put the LAN address of Site B like so:

{
  "phsycial": {
    "10.11.0.0/16": {
      "blacklist": true,
    }
  }
}

On Site B, we use the address block of Site A:

{
  "phsycial": {
    "10.10.0.0/16": {
      "blacklist": true,
    }
  }
}

These configs should fix the feedback loop. On normal BSD systems, the local.conf file lives in /var/db/zerotier-one (long with other stateful information like the node’s Identity). If the file doesn’t exist, you can create it. You may have to restart zerotier for changes to the file to take effect.