Inconsistent behaviour between docker and windows/android clients

[PovilasID]

Hey,

Context: I am using zerotier to centralize my device monitoring. So on devices I have a Prometheus exporter running and monitoring server calls out to it read data 60-80kb.

Docker container times out or takes over a minute to get a response for http requests on non 80 port.
Any time I try to get http request on port say 9100 it times out while on 80 it works just fine. When I test it on Windows machine on an android phone the pages are loaded in under a second.

However, if I run a docker container and try to read the data it just times out after a couple of minutes. I thought maybe it’s related to ports, so spun up docker playground and the same problem occurred but running iperf3 worked fine:

Here is the iperf3 between server to rule out general conectivity:

iperf3 -c 192.168.2.1
Connecting to host 192.168.2.1, port 5201
[  5] local 100.104.98.116 port 35282 connected to 192.168.2.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.80 MBytes  23.5 Mbits/sec    0    923 KBytes       
[  5]   1.00-2.00   sec  2.20 MBytes  18.5 Mbits/sec    2    827 KBytes       
[  5]   2.00-3.00   sec  1.25 MBytes  10.5 Mbits/sec    3    652 KBytes       
[  5]   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec    2    459 KBytes       
[  5]   4.00-5.00   sec  1.25 MBytes  10.5 Mbits/sec    3    306 KBytes       
[  5]   5.00-6.00   sec  1.22 MBytes  10.2 Mbits/sec    0    309 KBytes       
[  5]   6.00-7.00   sec  2.40 MBytes  20.2 Mbits/sec    1    242 KBytes       
[  5]   7.00-8.00   sec  1.21 MBytes  10.2 Mbits/sec    0    260 KBytes       
[  5]   8.00-9.00   sec  1.16 MBytes  9.75 Mbits/sec    0    263 KBytes       
[  5]   9.00-10.00  sec  2.38 MBytes  19.9 Mbits/sec    0    263 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  17.1 MBytes  14.4 Mbits/sec   11             sender
[  5]   0.00-10.14  sec  14.2 MBytes  11.7 Mbits/sec                  receiver

I am just confused.
Here is my docker compose zerotier service dfinition:

  ZT:
    image: "zerotier/zerotier:latest"
    container_name: ZT
    hostname: ZT
    volumes:
      - ./zerotier_config:/var/lib/zerotier-one
    cap_add:                                    # Required for zerotier to work
      - net_admin
      - sys_admin
    devices:
      - "/dev/net/tun"

I am testing from inside the container