ZeroTier and two ISPs

Community Support
1

Hi,

Imagine I have a box with two physical interfaces - each assuring that my exit traffic is going via different ISP.

I would like to run two zerotier overlay networks each taking different ISP (technically different exit interface).

How can I set it such that I force locally overlay A to take say eno1 and overlay B to take eno2 ? From box OS both have default routes with different metrics so from plane rib lookup this is not going to help.

Alternative to such setup where each ISP is hanging off different interface there can be scenario where there is single interface and two ISPs are just different getaways on the attached subnet … so support in zero tire for such model would be also be pretty cool to have.

Any help or pointer to already posted answer on this very welcome !

Many thx,
Robert Raszuk

2

I’m not sure I understand 100% what you want, but it sounds like you could just join the box to 2 ZeroTier networks and then push a default route on each network for each ISP.

3

Hi,

Today tunnel is created and routed over host OS RIB.

I need a knob to force the ZT to take specific exit interface or use specific next hop on the single interface.

If I build two (or N) ZT networks all will take same default route from host OS RIB to connect to peers.

Many thx,
R.

4

There are no knobs to turn that can set different networks to use different physical network cards. It may be possible to do with multiple instances of ZeroTier running by using the blacklist feature local.conf.

5

Can you say a little more about your use case, @robert1 ? I don’t think I’ve heard this request before that I recall.

6

@zt-travis at all,

Sure will be happy to describe it. As you know any good SDWAN has ability to dynamically measure path performance between any site and choose alternative underlay path to reach it.

Of course it only applies if your site/home/office has more then one ISP upstream which is more and more very common.

So assume site_1 has two ISPs and there is bunch of other sites. Site_2 would run some form of performance evaluation (continues or periodic, active or passive or both etc …) and measure quality to reach other sites over both ISPs. Once detected that to reach site_2 ISP_A is better or to reach site_3 ISP_B is better it can influence the exit underlay path to reach remote sites.

Of course I am not asking here for the full feature … I have been doing such measurements and path steering - only thing which is missing is ability to tell locally to ZT to use either interface_X to get to sites_Z or to tell that in case host is connected to only one subnet set that to get to sites_Y you need to use GW_Y.

It would be super useful to get such ability.

Poor mens solution would be to actively monitor all underlay destinations ZT is using, map it to overlay ZT network addresses then influence underlay routing according to the measured performance.

In this case ZT would not be aware at all about anything and the only thing which would be very helpful would be to get some easy way to locally get full such underlay to overlay table mapping from ZT. Maybe such mapping can be already exported in some way ?

Many thx you folks are doing super great product !

Cheers,
R.

7

Thanks!

We do have advanced features for using multiple ISPs

This would automatically steer traffic over the best link per peer.
From the explanation, I’m not sure you’d need knobs to make it per VL2 network.

8

Hi,

Link aggregation (or link bonding) is a completely different feature. LAGs combine multiple physical links (usually p2p) to a single L2 adjacency.

I am talking about different L3 peering exits. Link aggregation at best could work with multiple parallel links to same ISP not across two different ISPs.

Many thx, R.

9

Hello @robert1,

It is difficult to determine exactly what you mean. However, with multipath you can define a bond to a specific peer and in that bond you can tell ZT to only use a certain set of interfaces. Would this be a solution to your problem? It’s peer-specific, not network-specific.

Additionally, multipath is designed to work across multiple ISPs, that’s kind of the point.

10

Let me try one more time … a bit differently :slight_smile:

I have build zero tier network across hosts residing on 11 sites. My office site has two ISPs. To get to 5 sites ISP1 is better and to get to the other 5 sites ISP2 is better.

How do I tell ZT to go via ISP1 as primary ISP to sites 1-5 and to go via ISP2 as primary ISP to sites 6-10 ? When primary ISP fails the backup ISP is to be used.

Multipath when used blindly (ie. per flow) is not the best idea in networking. Commonly it is being used for spreading the load not for quality enhancement of the network.

Much better is to use multipath wisely based on real time measurements like unidirectional delays, jitter, packet drops, app TCP retransmissions etc …

Thx, R.

11

Sure. You can try something like the following in your local.conf:

{
	"settings": {
		"policies": {
			"bond-that-prefers-ISP1": {
				"links": {
					"en1": {
						"failoverTo": "en2",
						"mode": "primary"
					},
					"en2": {
						"mode": "spare"
					}
				},
				"failoverInterval": 10000,
				"basePolicy": "active-backup"
			},
			"bond-that-prefers-ISP2": {
				"links": {
					"en2": {
						"failoverTo": "en1",
						"mode": "primary"
					},
					"en1": {
						"mode": "spare"
					}
				},
				"failoverInterval": 10000,
				"basePolicy": "active-backup"
			},
			"peerSpecificBonds": {
				"aaaaaaaaaa": "bond-that-prefers-ISP1",
				"bbbbbbbbbb": "bond-that-prefers-ISP1",
				"cccccccccc": "bond-that-prefers-ISP2",
				"dddddddddd": "bond-that-prefers-ISP2",
				"eeeeeeeeee": "bond-that-prefers-ISP2"
			}
		}
	}
}

This uses the active-backup mode of multipath so only one link is used at a time but it will failover from ISP1’s interface to ISP2’s interface (or the reverse) if the link quality drops too low. And again this would only work for specific peers and cannot be configured on a per-network basis.

I hope this helps, if you decide to try this keep in mind that multipath is currently beta and development is happening on the dev-multipath branch.

12

I haven’t had the pleasure of having multiple internet connections recently. @zt-joseph what would be some good options for doing this automagically? Balance Aware?

Link quality#

ZeroTier measures various properties of a link (such as latency, throughput, jitter, packet loss ratio, etc) in order to arrive at a quality estimate…

13

Hi @zt-joseph

Ok this is exactly what I was asking for for deployment scenario A (multiple interfaces).

Would we have the very same ability but instead of “links” (en1, en2 etc …) be able to specify the gateway addresses gw1, gw2 etc … sitting all on the same subnet ?

Simply I have compute nodes with single interface but on that subnet I have multiple routers each attached with different gw_X address and each serving different ISP. Having such ability would be 100% covering the deployments.

Now to both scenarios - what are exactly the failover criteria ?

Can I configure it ? Can the switchover be exposed via API externally (for example I would like to switch when my external controller decides so).

Also do you plan to provide ability of policy routing based on the src/dst/port of the packets to the preferred bond with fallback to backup ?

Many many thx,
Robert

14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.