Diagnosing an OFFLINE machine

Community Support Linux
1

I have an Ubuntu 18.04 box. It has zerotier setup but it is showing as offline

Any recommendations to diagnose? I suspect the machines outgoing traffic is being blocked somehow, I see inbound ports to open, are there known outbound paths I can ask to be opened up that may help ??


$ /d/h/dev> sudo zerotier-cli info -j
{
 "address": "185eabf82d",
 "clock": 1655313210441,
 "config": {
  "physical": null,
  "settings": {
   "allowTcpFallbackRelay": true,
   "portMappingEnabled": true,
   "primaryPort": 9993,
   "softwareUpdate": "disable",
   "softwareUpdateChannel": "release"
  }
 },
 "online": false,
 "planetWorldId": 149604618,
 "planetWorldTimestamp": 1644592324813,
 "publicIdentity": "185eabf82d:0:f0e28ab91dfe3b22e3336f1b0fcee2b543cf2772bf6027eadfbced658e2701644de6306de4534dec9e48b1b555532e4757e96c7eaa71dd59d2a36f9fdc47c060",
 "tcpFallbackActive": false,
 "version": "1.6.5",
 "versionBuild": 0,
 "versionMajor": 1,
 "versionMinor": 6,
 "versionRev": 5
}

$ /d/h/dev> sudo zerotier-cli info
200 info 185eabf82d 1.6.5 OFFLINE

$ /d/h/dev> sudo zerotier-cli listpeers
200 listpeers <ztaddr> <path> <latency> <version> <role>
200 listpeers 62f865ae71 - -1 - PLANET
200 listpeers 778cde7190 - -1 - PLANET
200 listpeers 8056c2e21c - -1 1.8.9 LEAF
200 listpeers cafe04eba9 - -1 - PLANET
200 listpeers cafe9efeb9 - -1 - PLANET

$ /d/h/dev> sudo zerotier-cli peers
200 peers
<ztaddr>   <ver>  <role> <lat> <link> <lastTX> <lastRX> <path>
62f865ae71 -      PLANET    -1 RELAY
778cde7190 -      PLANET    -1 RELAY
8056c2e21c 1.8.9  LEAF      -1 RELAY
cafe04eba9 -      PLANET    -1 RELAY
cafe9efeb9 -      PLANET    -1 RELAY
2

You likely have firewall rules blocking ZeroTier from communicating. Check ufw (i believe the default on Ubuntu) or iptables. At the very least, it needs to be able to send & receive packets on UDP port 9993. Beyond that, it’s possible your router may be blocking traffic. How to fix that is dependent on your router.

Also, you’re on version 1.6.5. 1.10.0 was released last week. You may want to upgrade

3

Thanks for the quick response. Yeah I suspect outgoing from 9993 is blocked. I have put a request into IT to open that. Digging into this in parallel

ufw is inactive but looking at

xxx@yyyyyyyy /d/h/dev [3]> sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
f2b-sshd   tcp  --  anywhere             anywhere             multiport dports ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain f2b-sshd (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

sort of says it all

4

Yeah iptables looks like it’s on accept across the board there, so it’s not the firewall on your machine. Hopefully your IT department can help you out. Ideally you need incoming & outgoing UDP for things to work properly.

5

IT Finally opened up the ports but we still show status OFFLINE. Are there any clever ideas on how to test that outgoing or incoming UDP ports are really open, I genuinely do not trust IT. My normal trick of using TCP won’t work since it is UDP

dev@connectria /d/h/dev> sudo systemctl restart zerotier-one.service
dev@connectria /d/h/dev> sudo zerotier-cli status
200 info 185eabf82d 1.6.5 OFFLINE
6

nevermind the system suddenly came online and started working about 30 minutes after I posted this.

7

I am also having occasional disconnects with the system telling me it is offline. It seems this started with the update to version 1.10. I will see if downgrading will help.

8

Update version 1.8.9 also has this problem. Is anyone also having these problems on Ubuntu server 22.04 LTS?

9

Same problem here with some servers. Example bellow is a Debian 11. The same machine have a OpenVPN that not stop working for months. Firewall is completely off. Tcpdump reports thats packets are out an in in 9993 and other ports:

09:04:33.062445 eno1  Out IP 192.168.1.10.38062 > 50.7.252.138.9993: UDP, length 137
09:04:33.062452 eno1  Out IP 192.168.1.10.30202 > 50.7.252.138.9993: UDP, length 137
09:04:33.062465 eno1  Out IP 192.168.1.10.9993 > 104.194.8.134.9993: UDP, length 137
09:04:33.062476 eno1  Out IP 192.168.1.10.38062 > 104.194.8.134.9993: UDP, length 137
09:04:33.062483 eno1  Out IP 192.168.1.10.30202 > 104.194.8.134.9993: UDP, length 137
09:04:33.175989 eno2  In  IP 103.195.103.66.9993 > 192.168.1.10.9993: UDP, length 629
09:04:33.255201 eno2  In  IP 104.194.8.134.9993 > 192.168.1.10.38062: UDP, length 629
09:04:33.285850 eno2  In  IP 84.17.53.155.9993 > 192.168.1.10.9993: UDP, length 629
09:04:33.420511 eno2  In  IP 50.7.252.138.9993 > 192.168.1.10.38062: UDP, length 629

But status stay offline and peers shows all RELAYED. Suddenly, they become ONLINE for a while, sometimes RELAYED and some direct. Other machine (an old Debian 7) in the same network stay online not relayed all the time, but I never can ping them.
OpenVPN have a log thats helps, but I can’t find any in ZT.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.