Questions about bandwidth and ipv6

sefi · March 6, 2022, 7:00pm

i have been using zerotier for a while and have always had bandwidth issues even though zerotier-cli peers shows all the connections as direct, which is supported by iftop as well.

for example, an iperf3 on my LAN gives me these speeds:

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec                  receiver

the same speedtest through zerotier (again these two devices are on the same LAN) gives me about a quarter the bandwidth:

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   314 MBytes   263 Mbits/sec                  sender
[  4]   0.00-10.00  sec   313 MBytes   263 Mbits/sec                  receiver

I also recently had the idea to enable ipv6 and try a speedtest:

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   734 MBytes   615 Mbits/sec                  sender
[  4]   0.00-10.00  sec   734 MBytes   615 Mbits/sec                  receiver

the result appears to be ~2x better than over ipv4.

so my questions are:

why is zerotier causing such a performance penalty, especially on ipv4 (but also on ipv6)?
why am i getting better speeds on ipv6 than ipv4?
is there a way to get the ipv4/ipv6 speeds closer to my raw LAN?
if not, is there a way to “prefer” ipv6 over ipv4 with zeronsd e.g. when i try to access a webpage through zerotier?

crest_zerotier · March 7, 2022, 3:12pm

Have you checked with tcpdump (or Wireshark) if there is a difference in fragementation and/or discovered PMTU?

sefi · March 7, 2022, 6:11pm

Thanks for the suggestion! I found two filters for tcpdump that I ran on both the ethernet interface and on the zerotier interface the iperf3 server:

tcpdump -s0 -p -ni eth0 'icmp and icmp[0] == 3 and icmp[1] == 4'
tcpdump -i eth0 '((ip[6:2] > 0) and (not ip[6] = 64))'

neither of these had any output while i ran iperf3 -c to the server. Are there other things i should be looking out for in tcpdump? I’m not very familiar with it.

update: i ran sudo tcpdump -s 0 -i ztnfapkhyv -w ztnhost4.pcap host <ip> on the receiving server, opened it in wireshark and saw these errors a couple times:

No.	Time	Source	Destination	Protocol	Length	Info
1777	0.165847	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=1421974 Ack=1 Win=131712 Len=1448 TSval=2578994892 TSecr=1914704243
1778	0.165883	10.144.0.196	10.144.0.10	TCP	1514	[TCP Previous segment not captured] 51288 → 5201 [ACK] Seq=1556638 Ack=1 Win=131712 Len=1448 TSval=2578994894 TSecr=1914704247
1779	0.165912	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=1423422 Win=1638656 Len=0 TSval=1914704297 TSecr=2578994892 SLE=1556638 SRE=1558086
1780	0.165934	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=1558086 Ack=1 Win=131712 Len=1448 TSval=2578994894 TSecr=1914704247
1781	0.165959	10.144.0.10	10.144.0.196	TCP	78	[TCP Dup ACK 1779#1] 5201 → 51288 [ACK] Seq=1 Ack=1423422 Win=1638656 Len=0 TSval=1914704297 TSecr=2578994892 SLE=1556638 SRE=1559534
1782	0.165991	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=1559534 Ack=1 Win=131712 Len=1448 TSval=2578994894 TSecr=1914704247
1783	0.166011	10.144.0.10	10.144.0.196	TCP	78	[TCP Dup ACK 1779#2] 5201 → 51288 [ACK] Seq=1 Ack=1423422 Win=1638656 Len=0 TSval=1914704297 TSecr=2578994892 SLE=1556638 SRE=1560982
1784	0.166031	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=1560982 Ack=1 Win=131712 Len=1448 TSval=2578994894 TSecr=1914704248
1785	0.166054	10.144.0.10	10.144.0.196	TCP	78	[TCP Dup ACK 1779#3] 5201 → 51288 [ACK] Seq=1 Ack=1423422 Win=1638656 Len=0 TSval=1914704298 TSecr=2578994892 SLE=1556638 SRE=1562430
1786	0.166077	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=1562430 Ack=1 Win=131712 Len=1448 TSval=2578994894 TSecr=1914704248

No.	Time	Source	Destination	Protocol	Length	Info
37504	3.310510	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=36385390 Ack=1 Win=131712 Len=1448 TSval=2578997625 TSecr=1914707430
37505	3.310540	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=36386838 Ack=1 Win=131712 Len=1448 TSval=2578997626 TSecr=1914707430
37506	3.310570	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=36388286 Ack=1 Win=131712 Len=1448 TSval=2578997626 TSecr=1914707430
37507	3.310618	10.144.0.10	10.144.0.196	TCP	78	[TCP Dup ACK 37346#13] 5201 → 51288 [ACK] Seq=1 Ack=36156606 Win=3145728 Len=0 TSval=1914707442 TSecr=2578997610 SLE=36173982 SRE=36389734
37508	3.310624	10.144.0.196	10.144.0.10	TCP	1514	[TCP Fast Retransmission] 51288 → 5201 [ACK] Seq=36156606 Ack=1 Win=131712 Len=1448 TSval=2578997626 TSecr=1914707431
37509	3.310652	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=36158054 Win=3144320 Len=0 TSval=1914707442 TSecr=2578997626 SLE=36173982 SRE=36389734
37510	3.313429	10.144.0.196	10.144.0.10	TCP	1514	[TCP Out-Of-Order] 51288 → 5201 [ACK] Seq=36158054 Ack=1 Win=131712 Len=1448 TSval=2578997630 TSecr=1914707431
37511	3.313472	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=36159502 Win=3142912 Len=0 TSval=1914707445 TSecr=2578997630 SLE=36173982 SRE=36389734
37512	3.313515	10.144.0.196	10.144.0.10	TCP	1514	[TCP Out-Of-Order] 51288 → 5201 [ACK] Seq=36159502 Ack=1 Win=131712 Len=1448 TSval=2578997630 TSecr=1914707432
37513	3.313565	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=36160950 Win=3141504 Len=0 TSval=1914707445 TSecr=2578997630 SLE=36173982 SRE=36389734
37514	3.313640	10.144.0.196	10.144.0.10	TCP	1514	[TCP Out-Of-Order] 51288 → 5201 [ACK] Seq=36160950 Ack=1 Win=131712 Len=1448 TSval=2578997630 TSecr=1914707433
37515	3.313692	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=36162398 Win=3140096 Len=0 TSval=1914707445 TSecr=2578997630 SLE=36173982 SRE=36389734
37516	3.313735	10.144.0.196	10.144.0.10	TCP	1514	[TCP Out-Of-Order] 51288 → 5201 [ACK] Seq=36162398 Ack=1 Win=131712 Len=1448 TSval=2578997630 TSecr=1914707438
37517	3.313776	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=36163846 Win=3138688 Len=0 TSval=1914707445 TSecr=2578997630 SLE=36173982 SRE=36389734
37518	3.313820	10.144.0.196	10.144.0.10	TCP	1514	[TCP Out-Of-Order] 51288 → 5201 [ACK] Seq=36163846 Ack=1 Win=131712 Len=1448 TSval=2578997630 TSecr=1914707439
37519	3.313866	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=36165294 Win=3137280 Len=0 TSval=1914707445 TSecr=2578997630 SLE=36173982 SRE=36389734
37520	3.313911	10.144.0.196	10.144.0.10	TCP	1514	[TCP Out-Of-Order] 51288 → 5201 [ACK] Seq=36165294 Ack=1 Win=131712 Len=1448 TSval=2578997630 TSecr=1914707439
37521	3.313952	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=36166742 Win=3135872 Len=0 TSval=1914707445 TSecr=2578997630 SLE=36173982 SRE=36389734
37522	3.313992	10.144.0.196	10.144.0.10	TCP	1514	[TCP Out-Of-Order] 51288 → 5201 [ACK] Seq=36166742 Ack=1 Win=131712 Len=1448 TSval=2578997630 TSecr=1914707439
37523	3.314034	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=36168190 Win=3134464 Len=0 TSval=1914707445 TSecr=2578997630 SLE=36173982 SRE=36389734

No.	Time	Source	Destination	Protocol	Length	Info
74572	6.372875	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=72752078 Ack=1 Win=131712 Len=1448 TSval=2579000279 TSecr=1914710493
74573	6.372905	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=72753526 Ack=1 Win=131712 Len=1448 TSval=2579000279 TSecr=1914710495
74574	6.372935	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=72754974 Ack=1 Win=131712 Len=1448 TSval=2579000279 TSecr=1914710495
74575	6.373174	10.144.0.10	10.144.0.196	TCP	78	[TCP Dup ACK 74439#11] 5201 → 51288 [ACK] Seq=1 Ack=72523294 Win=3145728 Len=0 TSval=1914710504 TSecr=2579000260 SLE=72575422 SRE=72756422
74576	6.384348	10.144.0.196	10.144.0.10	TCP	1514	[TCP Fast Retransmission] 51288 → 5201 [ACK] Seq=72523294 Ack=1 Win=131712 Len=1448 TSval=2579000289 TSecr=1914710504
74577	6.384450	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72524742 Win=3144320 Len=0 TSval=1914710516 TSecr=2579000289 SLE=72575422 SRE=72756422
74578	6.388318	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72524742 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74579	6.388418	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72526190 Win=3142912 Len=0 TSval=1914710520 TSecr=2579000293 SLE=72575422 SRE=72756422
74580	6.388491	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72526190 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74581	6.388570	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72527638 Win=3141504 Len=0 TSval=1914710520 TSecr=2579000293 SLE=72575422 SRE=72756422
74582	6.388632	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72527638 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74583	6.388702	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72529086 Win=3140096 Len=0 TSval=1914710520 TSecr=2579000293 SLE=72575422 SRE=72756422
74584	6.388795	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72529086 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74585	6.388875	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72530534 Win=3138688 Len=0 TSval=1914710520 TSecr=2579000293 SLE=72575422 SRE=72756422
74586	6.388967	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72530534 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74587	6.389045	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72531982 Win=3137280 Len=0 TSval=1914710520 TSecr=2579000293 SLE=72575422 SRE=72756422
74588	6.389122	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72531982 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74589	6.389193	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72533430 Win=3135872 Len=0 TSval=1914710521 TSecr=2579000293 SLE=72575422 SRE=72756422
74590	6.390093	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72533430 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74591	6.390158	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72534878 Win=3134464 Len=0 TSval=1914710522 TSecr=2579000293 SLE=72575422 SRE=72756422
74592	6.390222	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72534878 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74593	6.390289	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72536326 Win=3133056 Len=0 TSval=1914710522 TSecr=2579000293 SLE=72575422 SRE=72756422
74594	6.390360	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72536326 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74595	6.390420	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72537774 Win=3131648 Len=0 TSval=1914710522 TSecr=2579000293 SLE=72575422 SRE=72756422
74596	6.390481	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72537774 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74597	6.390526	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72539222 Win=3130240 Len=0 TSval=1914710522 TSecr=2579000293 SLE=72575422 SRE=72756422
74598	6.390589	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72539222 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74599	6.390636	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72540670 Win=3128832 Len=0 TSval=1914710522 TSecr=2579000293 SLE=72575422 SRE=72756422
74600	6.390687	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72540670 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74601	6.390738	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72542118 Win=3127424 Len=0 TSval=1914710522 TSecr=2579000293 SLE=72575422 SRE=72756422
74602	6.390786	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72542118 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74603	6.390854	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72543566 Win=3126016 Len=0 TSval=1914710522 TSecr=2579000293 SLE=72575422 SRE=72756422
74604	6.390916	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72543566 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74605	6.390981	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72545014 Win=3124608 Len=0 TSval=1914710522 TSecr=2579000293 SLE=72575422 SRE=72756422
74606	6.391047	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72545014 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74607	6.391106	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72546462 Win=3123200 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74608	6.391167	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72546462 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74609	6.391211	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72547910 Win=3121792 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74610	6.391261	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72547910 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74611	6.391310	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72549358 Win=3120384 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74612	6.391358	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72549358 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74613	6.391405	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72550806 Win=3118976 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74614	6.391460	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72550806 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74615	6.391512	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72552254 Win=3117568 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74616	6.391564	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72552254 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74617	6.391613	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72553702 Win=3116160 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74618	6.391670	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72553702 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74619	6.391733	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72555150 Win=3114752 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74620	6.391811	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72555150 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74621	6.391877	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72556598 Win=3113344 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74622	6.391947	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72556598 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74623	6.392014	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72558046 Win=3111936 Len=0 TSval=1914710523 TSecr=2579000293 SLE=72575422 SRE=72756422
74624	6.392091	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72558046 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74625	6.392151	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72559494 Win=3110528 Len=0 TSval=1914710524 TSecr=2579000293 SLE=72575422 SRE=72756422
74626	6.392221	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72559494 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74627	6.392284	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72560942 Win=3109120 Len=0 TSval=1914710524 TSecr=2579000293 SLE=72575422 SRE=72756422
74628	6.392376	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72560942 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74629	6.392438	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72562390 Win=3107712 Len=0 TSval=1914710524 TSecr=2579000293 SLE=72575422 SRE=72756422
74630	6.392529	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72562390 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74631	6.392591	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72563838 Win=3106304 Len=0 TSval=1914710524 TSecr=2579000293 SLE=72575422 SRE=72756422
74632	6.392659	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72563838 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74633	6.392735	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72565286 Win=3104896 Len=0 TSval=1914710524 TSecr=2579000293 SLE=72575422 SRE=72756422
74634	6.392823	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72565286 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74635	6.392915	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72566734 Win=3103488 Len=0 TSval=1914710524 TSecr=2579000293 SLE=72575422 SRE=72756422
74636	6.392979	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72566734 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74637	6.393041	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72568182 Win=3102080 Len=0 TSval=1914710524 TSecr=2579000293 SLE=72575422 SRE=72756422
74638	6.393095	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72568182 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74639	6.393161	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72569630 Win=3100672 Len=0 TSval=1914710525 TSecr=2579000293 SLE=72575422 SRE=72756422
74640	6.393230	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72569630 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74641	6.393290	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72571078 Win=3099264 Len=0 TSval=1914710525 TSecr=2579000293 SLE=72575422 SRE=72756422
74642	6.393354	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72571078 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74643	6.393413	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72572526 Win=3097856 Len=0 TSval=1914710525 TSecr=2579000293 SLE=72575422 SRE=72756422
74644	6.393465	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72572526 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74645	6.393527	10.144.0.10	10.144.0.196	TCP	78	5201 → 51288 [ACK] Seq=1 Ack=72573974 Win=3096448 Len=0 TSval=1914710525 TSecr=2579000293 SLE=72575422 SRE=72756422
74646	6.393590	10.144.0.196	10.144.0.10	TCP	1514	[TCP Retransmission] 51288 → 5201 [ACK] Seq=72573974 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516
74647	6.393654	10.144.0.10	10.144.0.196	TCP	66	5201 → 51288 [ACK] Seq=1 Ack=72756422 Win=3023744 Len=0 TSval=1914710525 TSecr=2579000293
74648	6.393722	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=72756422 Ack=1 Win=131712 Len=1448 TSval=2579000293 TSecr=1914710516

No.	Time	Source	Destination	Protocol	Length	Info
123799	10.050951	10.144.0.196	10.144.0.10	TCP	67	51287 → 5201 [PSH, ACK] Seq=145 Ack=5 Win=131712 Len=1 TSval=592742424 TSecr=1914704170
123800	10.050990	10.144.0.10	10.144.0.196	TCP	66	5201 → 51287 [ACK] Seq=5 Ack=146 Win=65152 Len=0 TSval=1914714182 TSecr=592742424
123801	10.051023	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=121200102 Ack=1 Win=131712 Len=1448 TSval=2579003464 TSecr=1914714164
123802	10.051037	10.144.0.10	10.144.0.196	TCP	66	5201 → 51288 [FIN, ACK] Seq=1 Ack=121200102 Win=3145728 Len=0 TSval=1914714182 TSecr=2579003459
123803	10.051073	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=121201550 Ack=1 Win=131712 Len=1448 TSval=2579003464 TSecr=1914714164
123804	10.051084	10.144.0.10	10.144.0.196	TCP	54	5201 → 51288 [RST] Seq=1 Win=0 Len=0
123805	10.051107	10.144.0.10	10.144.0.196	TCP	54	5201 → 51288 [RST] Seq=1 Win=0 Len=0
123806	10.051140	10.144.0.196	10.144.0.10	TCP	1514	51288 → 5201 [ACK] Seq=121202998 Ack=1 Win=131712 Len=1448 TSval=2579003464 TSecr=1914714164
123807	10.051171	10.144.0.10	10.144.0.196	TCP	67	5201 → 51287 [PSH, ACK] Seq=5 Ack=146 Win=65152 Len=1 TSval=1914714183 TSecr=592742424
123808	10.051175	10.144.0.10	10.144.0.196	TCP	54	5201 → 51288 [RST] Seq=1 Win=0 Len=0

I also saw these errors to a lesser extent with zerotier over ipv6. Over the raw lan i only saw the [RST] errors at the end.

zt-travis · March 8, 2022, 10:24pm

Hey thanks for writing. To clarify, you enabled ipv6 on your physical network or your zerotier network?

sefi · March 9, 2022, 8:21pm

ipv6 is enabled on both my LAN and on zerotier. I don’t believe my internet uplink is ipv6-capable though.

after some more research and testing I think this might be a linux-specific issue. I ran across this stackexchange question that also said hamachi has a similar issue on linux, which gave me the idea to test zerotier performance between mac<->windows.

surely enough, my iperf3 tests on LAN vs zerotier were pretty much identical, in fact for some reason zerotier seems to even outperform my LAN which is a bit funny.

I will continue to investigate but progress will likely slow to a crawl the lower level I try to go–if anyone with expertise has any insight I would greatly appreciate it!

system · April 8, 2022, 8:22pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.