MultiPath performance Tuning

Community Support Linux
1

Cross post from Reddit. Hopefully someone can help here. Original Reddit post: https://www.reddit.com/r/zerotier/comments/wuqumy/zerotier_multipath_not_using_full_bandwidth/

Morning all.

I have a linux vm setup with 2 internet links (1Gb/50Mb Cable modem and 2Gb/200Mb FTTH). I have Zerotier installed on it and box in a datacenter (2Gb link on it) and have set Zerotier to use balance-aware (also tried balance-rr and balance-xor) and, although it does seem to be using both links for balance-aware and balance-rr, it does not seem to be able to use the full bandwith of the links… it is topping out at around 300mb/s down and 150mb/s up. the box in the house has 8 cores (Xeon Scalable V1) and 16Gb RAM. the box upstream has 4 cores (Xeon E3 V2) and 8GB RAM.

What am i missing? Any performance tuning stuff i can do to make things faster?

[EDIT] 2 things to clarify: 1: box in house is running Ubuntu 22.04. Box in the cloud is running Debian (latest). also, neither box is behind a firewall.

[EDIT2] Bit more digging. Seems that when i am downloading at a high speed (managed to get around 550Mb/s just now) the upstream server (a slightly different one, a Skylake 2 cores @ 2Gz) is running 70-90% CPU usage (one core is hitting 100%, the other around 70-80%). The home machine has 8 cores assigned, so it seems to be doing a bit better…

Is there anything at a config level i can tweak to make this work faster?

2

Hello again Tiernan.

I’d suggest trying balance-xor if the protocol you’re sending over ZT is sensitive to re-ordering (TCP for instance). If you’re only sending something like UDP then balance-rr is suitable. I would suggest not using balance-aware as that did not mature fully yet and would probably not be much better than ordinary balance-xor.

Specifically what numbers are you getting when you use balance-xor?

Thanks.

3

just checked and updated all local.config files to use balance-xor, and getting around 160mb/s down… weirdly, getting more upload (210mb). Looking at iftop for the individual links, i can see both links are around equal on traffic in, but wan 1 is pushing 50mb/s up and wan 2 is pusing a little over 160… wan 1 tops out at 50, wan 2 is 200mb…

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.