Hi, regarding the post Dynamic Multipoint VPN with ZeroTier and VyOS.
Great articles though I’ve probably missed some important details as I simply don’t understand the purpose of the implementation.
What is the basic idea behind adding an internal gateway or boundary protocol on top of ZeroTier, which already has built-in capabilities for scaling, fast routing, and direct communication between nodes?
Hey @Boilerplate4U,
Thanks for checking out the articles.
The design is meant for someone that has a large multi-site deployment where site-to-site connectivity is desired over Hub and Spoke.
ZeroTier does a lot of things very well and is uniquely suited for this solution due to it creating a virtual LAN instead of a collection of point-to-points like other overlay technologies (Tailscale, NetBird, etc…). There are however a few limitations when using only ZeroTier for an SD-WAN Solution.
Each ZeroTier network is limited to 128 managed routes, to include any LAN routes. In order to scale out a design past 127 sites (assuming only one summary per site + LAN subnet), you would need to create another ZeroTier network and stitch them together. This would be analogous to regions in a traditional Multipoint design. Regions can help scale the network, but it will cause routing to be suboptimal between spokes in different regions. Using BGP in the design can easily scale to Millions of routes even with cheap hardware. And because you can simply use Next-Hop-Unchanged behavior with BGP to point to the remote node, you can always create on-demand site-to-site tunnels even if you have 1000’s of sites.
ZeroTier doesn’t support ECMP for managed routes. If you configure multiple identical routes, only one managed route will be be present in a device at a time. This prevents optimal scaling of networks. Imagine that you have a 5Gbps circuit at a site, and can only do 1.5Gbps of ZeroTier traffic on your node. You could scale out horizontally with 4 routers to achieve an aggregate of 6Gbps of throughput via ECMP. You’re unable to do that with Managed Routes, but can easily do that with BGP.
Since there is no inherit health check with ZeroTier Managed routes, it becomes fairly easy to blackhole traffic. This is because the route is always present at remote sites, even if the node it’s pointed to is down. With BGP, health checks and rerouting of traffic is built at the heart of BGP. If a node goes down, and you have redundancy built into the overlay of your network, BGP will reconverge without any admin intervention.
ZeroTier managed routes don’t have a unique preference attached to them. This means if you were able to have multiple routes, they would at best be ECMP routes (like mentioned earlier, only one is installed anyways). This means if you want traffic to prefer a high speed 5Gbps circuit over a 5G Cellular backup, you can’t select it short of doing longest match routing (which eats into that 128 route limit). With BGP, there are countless mechanisms to make a route more preferred to others, and vastly more scalable as you can attach BGP communities to match traffic without needing to explicitly define a route.
ZeroTier can do what it can do, and no more. When using BGP, you can run services like MPLS for multitenancy and microsegmentation. Within a single ZeroTier network, you can have 100 customers and prevent them from talking without even touching the flow rules. This really opens up ZeroTier as a bolt-on SD-WAN solution for multiple Network OSes.
Ultimately networks are built to requirements, and some customer’s requirements will drive them to a solution like this, and some will be more than happy with letting ZeroTier handle everything for smaller deployments. Hopefully that helped answer some of your questions, let me know if you have any more or need more detail.
This topic was automatically closed after 30 days. New replies are no longer allowed.