ZeroTier not working on router running FreshTomato firmware

Community Support Linux
, ,
1

I can successfully connect to my ZeroTier with the router and join my ZT network with an automatic IP assigned my ZT. However, I can’t ping the router from from any device on my ZT network nor have the router ping any device my my ZT network. However any device that’s connected to the router, be it Wi-Fi or wired, could ping all the devices on my ZT network.

My local network IP addresses are 192.168.1.* range and the ZeroTier network is 192.168.192.*. My guess its the Iptables rules or route that have to be added or some other configuration I’m missing. Any feedback would be much appreciated.

1 Like
2

I have almost the same problem as yours (only differing is my IP ranges), with as little success as you.
Router appears online in the members list in my.zerotier.com, and also in the router’s terminal executing zerotier-cli info.
From the LAN I can ping the router’s ZT IP from devices zerotiered or not but can’t access its web interface at the same IP (I can on the LAN IP).
From zerotiered devices elsewhere I can’t ping or access the interface.

Today I staqrted logging the router’s iptables and saw a lot of DROPs. almost all related to its ZT virtual interface, but I’m yet to analize them.
You don’t mention what router you have (I have a Netgear R7000), but maybe you can enable and review your router’s firewall logs, and maybe give insight to each other.

Have a nice weekend!

3

Yea we have the same router Netgear R7000, but that doesn’t matter its the firmware that the ZT is running on which is Fresh Tomato. I’m pretty sure that it has something to do with Iptables and the way the traffic is routed on ZT interface. The primary suspect here is NAT because ZT is on a different ip range. I think the solution has something to do with PREROUTING and POSTROUTING. What I’m going to do is since I have ZT running on Ubuntu as well, I’m going to see how the iptables route traffic on Ubuntu and compare them to tomato iptables thru SSH. I will keep you posted.

1 Like
4

Good that we have the same hardware and OS. I mentioned it only to justify my not knowing how you should check your router’s firewall log. In case you or some other reader are as noob as I, just enable it in the web interface in Administration > Logging > Syslog > Log internally, take notice or set the log file path and select to log connections “If blocked by the firewall”.
Then ssh into the router and do tail -F /path/for/your/logfile for a live view or nano /path/for/your/logfile for a static snapshot.

At once I logged a lot of DROPped traffic from my ZT interface and though “gotcha!”.
I executed iptables -S and found nothing specific to my ZT interface, but noticed that the default INPUT and FORWARD policies were set to DROP.
Not wanting to set defaults to ACCEPT and leave it wide open (like this related video), I ran iptables -A INPUT -i zt+ -j ACCEPT and iptables -A FORWARD -i zt+ -j ACCEPT. Still no luck, more googling. “Discovered” that iptables -A appends the rule at the end after the traffic has been dropped before, so I deleted those rules and reinserted at the top with iptables -I INPUT 1 -i zt+ -j ACCEPT and iptables -I FORWARD 1 -i zt+ -j ACCEPT. Still no luck (but almost no DROPs in the log, so that’s something?).

And that’s where I am now. Was going to try to temporarily set the default policies to ACCEPT, but I’m going to investigate the PREROUTING and POSTROUTING you mentioned.
I have ZT running just fine on a RaspberryPi with Debian, so I’ll check those rules.

Thanks for your feedback!

5

OK good new IT WORKS !!! I have you thank you for giving me the idea about checking the logs, however I did it a bit differently. I OPKG tcpdump -i ZT interface, and what I saw shocked me, as i began pinging from ZT node to the router I could see the router replying to my pings, however what it was replying was the shocking part, the router was broadcasting its own ZT ip address and asking the node where that address was on the network !!! and that’s when it hit me, the issue isn’t iptables, well its that too but not yet, its ARP. Basically the ZT assigns it an IP address but there isn’t any ARP happening on the router for it to know whos who on the ZT network, nor does it even know itself. BOOM !!! The whole time I was thinking it was iptables it was really just ARP !!! So I first started by adding itself on the ARP table [ arp -s (ip address of the ZT router) (MAC address ZT Router) -i ZT interface ] than I had to add its MAC address on one of the ZT node on the network in its ARP table. And boom, I finally had a ping going on both devices over the ZT network. I also should mention that I doing this on a router that is in wireless bridge mode, so its not WAN facing, its the second router behind the main router which isn’t FT or even capable of running ZT. I’m pretty sure that the whole ARP thing is a FT issue and not a ZT one. I got the ping working to and from the ZT to other nodes both local and remote and SSH as well both ways. Only thing that so far hasn’t worked as I was hoping for is SSH -D (port) user@ip/hostname, for a proxy connection so that I could reach the router web admin page from any ZT node on the network, but its probably just iptables again. If anybody could figure out why the ARP Table doesn’t automatically update itself once the ZT interface is up and running on the router would be much appreciated. Thanks !!

1 Like
6

EXCELLENT, I’m glad you succeeded! Good thinking looking into tcpdump instead of firewall logs.

I did as you did, pinged the router from another ZT host while tcpdumping at the same node in another terminal, and I got the same as you did, a lot of 08:12:51.817853 ARP, Request who-has {Router's ZT IP} tell {TestHost's ZT IP}, length 28, i.e., “somebody tell me where is this IP that I can’t find?”.
Then I did arp in the ZT test host and there it was: {Router's ZT IP} (incomplete) {ZTinterface}, i.e., “I’ve met this but don’t know where it is”.
So I added the arp record both in the ZT router arp -s {Router's ZT IP} {Router's ZT MAC} -i {ZTinterface} and on the test node arp -s {Router's ZT IP} {Router's ZT MAC} and pinged again. No luck: I still can’t ping in either direction between the ZT router and the ZT test host, despite tcpdum now getting 08:57:44.495053 IP {TestHost's ZT IP} > {Router's ZT IP}: ICMP echo request, id 7, seq 1, length 64. I thought “now it’s the firewall that’s blocking the pings”, but inserting iptables -I INPUT 1 -p icmp -j ACCEPT on the router did not help. They have always pinged just fine via their LAN IPs.
So, in my case it’s something else besides the arp issue.

As I understand, you had to add to the arp tables of both your FT+ZT router and your ZT test host to be able to ping? If that’s the case, that wouldn’t help my end goals, which are to be able to access non-ZT capable LAN devices from ZT devices elsewhere.
Interestingly, I can ping the router’s ZT IP from a non-zerotiered host in the same LAN as the router (and the router’s LAN IP, of course) without jumping any arp hoops.

I must mention that my R700 IS WAN facing, not in wifi-bridge mode like yours.

Next: I will set-up a fresh Debian VM,configure it as a ZT router with these instructions, and if that works I will compare iptables and arp tables with those on the R7000. I might as well end up settling for that, a dedicated machine as a bridge, but I find more elegant to use the router itself (and it says so in those instructions).
I’m also considering using Tailscale, but I have read everywhere that it’s more difficult than ZT.

7

Also make sure to add the right -i interface on the test ZT node of the local ZT interface not the remote one. Also run nano /proc/sys/net/ipv4/ip_forward on the router and see that its says 1 and not 0. If its 0 than run this command echo 1 > /proc/sys/net/ipv4/ip_forward. Yes that’s correct I had to add both ARP tables on the router ZT and all other ZT node on the network. Also try this commands iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT and iptables -A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT than Iptables-save to commit and don’t reboot router at any point. As far as your end goal that i believe is very much possible with Managed Routes on your ZT Admin Dashboard, that’s actually on my to do list but later on once i master the art of the ARP. As far as tailscale goes, it doesn’t work at all on FT, there are 2 packages of it on OPKG but are both buggy and don’t even connect to the tailscale server. Also check that your ZT is in TAP and in TUN, command is iptuntap, TAP is s virtual interface operating on a OSI model level 2 a TUN is a VPN operating on OSI model level 3. So ZT is not a VPN in the traditional sense of the word. I will be destroying my whole setup shortly and setting this whole thing up again from scratch and posting the precise steps taken to get it all working. I will also be setting this up both on a WAN sided Router and than on wireless bridge but this time on a separate subnet so that the router has to have its DHCP server running unlike what I have now which is just a dummy AP. The goal is to have ZT up and running in all scenarios. Keep me posted on your progress !!

1 Like
8

Hi again!
(little) Progress report as requested.

Regarding arp:
I checked if I specified the right interface when adding to arp tables, and found out that 1: the zt interfaces have the same ztklh2unun name on the router and both Linux hosts I have on ZT, and 2: I forgot to specify the ZT interfaces when adding to arp tables, and it does not seem to matter, as the ZT interface gets included in the arp record (I guess the OS infers those from the IP?).
Pinging from a Linux ZT node only works with MACs in each other’s arp tables.
Pinging from a Windows ZT node works regardless of MACs being present or not in each other arp tables.
I rebooted the router to check arp entries persistence and I had to add them again, but LAN arp entries remain (or are rebuilt at boot, maybe some OSI layer thing). To be researched later.

Regarding your iptables suggestions:
I just added ACCEPTs anything from -i zt+ at the top of the INPUT and FORWARD chains (OUTPUT default policy is already ACCEPT) and they get re-added via startup script. No worries there.

Regarding Managed Routes:
I already have {ZT network} via LAN and {LAN network} via {router ZT IP}. These settings mirrors those of the ZT network at work which uses a OPNsense router and where I can access non zerotiered LAN devices without trouble from everywhere (set up following the video I linked in my first comment days ago).

Tailscale: Discarded as per your advice.

Regarding TAP and TUN:
TUN is listed in lsmod, but ip tuntap returns just ztklh2unun: tap.
modprobe tap returns module tap not found in modules.dep
I don’t know what to make of these.

Yesterday I set up a Debian VM to test soon as a ZT router, see what it does in a full linux environment and deduce what Freshtomato lacks.
And lastly, good look with your setting up everything from scratch, I’m looking forward for your progress and maybe spot which step I missed!

9

OK HUGE Update !!! Im pretty certain of myself that I perfected the Fresh Tomato Zerotier process. Starting off from ground zero, ZT GUI Admin Config Restore Default Config Erase All Data in NVRAM memory (thorough), Next Install Entware this time I did it in JFFS so no USB needed, Administration > JFFS Enable, Execute When Mounted – mount -o bind /jffs /opt
than click Format/Erase, reboot. Next Start TUN and ZT on reboot,
#Administration > Scripts > Init >
lsmod | grep -q “tun” || modprobe tun
zerotier-one -d

#Make sure TCP Forwarding is set to 1 not 0
echo 1 > /proc/sys/net/ipv4/ip_forward

#Next to get pings working, SSH and type ip a if you see your ZT interface like this zt6hfejwkf:<BROADCAST,MULTICAST,NOARP,ALLMULTI,NOTRAILERS,UP,LOWER_UP>
We need to change it to this
zt6hfejwkf:<BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP>

#Add to Administration > Scripts > Firewall
ifconfig zt6hfejwkf arp
ifconfig zt6hfejwkf trailers

Reboot

Fully TESTED on FT Router ARMv7 FT Firmware 2024.1 with ZT 1.10.4
Works in Wireless Client Mode which means second 192.168.2.1 router behind the main router 192.168.1.1, So its double NATd
Works in Wireless Ethernet Bridge Mode which means second router behind the main router on the same subnet, So single NAT
Tested and will work on Entware ZT on a USB (external) or JFFS (internal) storage.
Also all done WITHOUT any changes or modification to IPTABLES what so ever. My method is persistent and will survive reboot. ip tuntap should return zt6hfejwkf : tap. The only thing that it lacks is that the ZT Interface has ARP disabled on it and ifconfig easily fixes that problem. When attempting your first ping with other ZT nodes give it some time, like 5 mins to get all the ARP cache on the ZT server properly set up for traffic flow. Also you should see in your FT GUI Status Device List the ZT interface pop up with the IPs of the other nodes, feel free to add it to your DHCP Reservation and customize hostname and static ARP even IP Traffic. And that’s it you should now have yourself a fully working ZT on an FT Router !! Congrats !!

1 Like
10

Hi, those are very good news!

Starting off from ground zero …

I’ve been toying with that, taking a safety-net backup, restore defaults but not restore the backup but redo it manually from notes. I’ve been putting it off because I would be left without internet for the duration (and I’m presently doing the tests remotely from work, via a zerotiered Windows PC at home that would get disconnected if I restore the router defaults)

I did it in JFFS so no USB needed…

I have forgotten that I could install Entware and ZT on internal JFFS, but I remember reading about flash storage getting worn-out with repeated writes, and I must have taken the removable USB route based on that (perhaps you might want to consider switching to a replaceable USB drive now that you have the wrinkles ironed-out). Unless forced, I think I’ll stick with the USB drive since it is already configured, mounted on /opt, with EW and ZT operational, and showing online on zerotier’s web admin page.

#Administration > Scripts > Init >
lsmod | grep -q “tun” || modprobe tun
zerotier-one -d

Was already done, but with more convoluted commands from one of the procedures I read.
I have [ $(lsmod | grep “tun” | wc -l) -eq 0 ] && modprobe tun that gets the same result, but I like your syntax more and might replace mine with yours (but omit the zerotier-one -d command because that gets me a second ZT process).

#Make sure TCP Forwarding is set to 1 not 0

My /proc/sys/net/ipv4/ip_forward alerady have a “1” after reboot, I guess some configuration from the GUI does that.

#Next to get pings working, SSH and type ip a if you see your ZT interface like this
zt6hfejwkf: <BROADCAST,MULTICAST,NOARP,ALLMULTI,NOTRAILERS,UP,LOWER_UP>
We need to change it to this
zt6hfejwkf: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP>
#Add to Administration > Scripts > Firewall
ifconfig zt6hfejwkf arp
ifconfig zt6hfejwkf trailers

I had
ztklh2unun: <BROADCAST,MULTICAST,NOARP,ALLMULTI,NOTRAILERS,UP,LOWER_UP>
followed your ifconfig arp and ifconfig trailers commands and now I have
ztklh2unun: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP>
Then I commented my two iptables commands, added your ifconfig with my ZT interface in my firewall script, rebooted… and lost access between any two LAN hosts, by name or by IP, including to the router web interface. Strangely enough, I stil can connect from work to my zerotiered devices on my LAN (except the router itself as before) and still had LAN access to the router but only via ssh (why ssh and not http?). I could do nvram unset script_fire, nvram commit, rebooted and recovered normal access.
Then I tried with only ifconfig ztklh2unun arp, again lost accesses and again had to nvram unset script_fire.
Then I tried with only ifconfig ztklh2unun trailers, and did not lose access, but also did not answer pings from the ZT IP.

The rest of my “symptoms” remain unchanged: My router still shows online in zerotier’s web admin page and it and non-zerotiered devices behind it are still not accessible. ip tuntap has always returned ztklh2unun: tap.
I will proceed setting up the Debian VM as a router, unless I (or you) notice something missing from my setup.

11

I would flush out all iptables rules just to make sure that the arp isn’t affected by them and leave them default. Other than that I’m guessing it as to be an issue between layer 2 and layer 3 of the OSI model. However since your ZT router is WAN facing, I would just go the OpenVPN route and have that as a server running on your router rather than ZT. Also you can have ZT traffic tunneled thru the ZT network or have ZT traffic tunneled thru OpenVPN, which is something I might to try to do next. That opens up the possibly to do at point to point tunneling thru the ZT network reaching other nodes on the network who cant run ZT on their system but who can run OpenVPN. Which means that ZT could work alongside wireguard and tinc and even nebula.

12

Hello there!
I do flush iptables between experiments (in fact I don’t save them, and they get flushed at the router scheduled reboot every day at 3AM). But I think I have some mess between layer 2 and layer 3 (I haven’t done much networking since the '00s)
Anyway, this weekend I took note of my basic settings, restored the router to defaults, reconfigured the basics manually, and began again from scratch based on ZeroTier’s official steps.
I tried the bridged approach, which required to have ZT and LAN on the same segment (which I don’t like) and of course made a mess with conflicts with the other two ZTs I’m connected (my mom’s and my office’s). ! spent half of sunday getting weird IPs, only to eventually find that I was getting those from the router at work!.
Ended up disconnecting from ZT at home to have internet again and I’m now ordering IPs, but plan to leave the bridged approach for routing, which is what’s working on OPNsense at work.

I can’t use OpenVPN or Wireguard (don’t know about Nebula) since those need a public IP, which I don’t have since I changed my old DSL for fiber, which uses GNAT.

13

Also forgot to mention to enable IPv6 and UPnP and NAT-PMP on the FT Router Router Config Tips | ZeroTier Documentation, without that even with ARP fully functioning it wont fully work. Also found another way of doing the same thing of a mesh overlay network but this time I did it with openvpn instead. Openvpn VPC cloud free tier that acts as a relay point called OpenVPN Cloud Connexa, and all that is needed is a openvpn client on the FT router side and instead of a TAP you will have yourself a TUN point to point with no need for ARP, and unlike ZT that has a relay server running, this will be fully customizable by the end user. I tried it myself and it works, was able to go thru the openvpn and in to the ZT network. Try it out if you want, I will do a full walkthrough at a later point.

14

Hi again!
I have not made much progress with my issue beyond trying to learn iptables and routing in depth, so didn’t update. Did manage to turn the DROPs into ACCEPTs from zerotier in the router logs, but still the router doesn’t answer pings at its ZT IP, and of course does not route. I guess other ZT devices in my LAN set up their own tunnels.
I have indeed IPv6 and UPnP disabled in my router (not sure if NAT-PMP), will research that when I get home.

15

IPv6 and UPnP and NAT-PMP has to be enabled according to that link I left.DHCPv6 with Prefix Delegation and the other in the Port Forwarding section.

16

Brief update:
Enabling IPv6 (any service type), UPnP and NAT-PMP did nothing, either all three or individually. IPv6 gets DUID, address and DNS from my ISP without issue. With UPnP and NAT-PMP enabled the table “Forwarded Ports” started listing mappings, but none from the router (either LAN or ZT side).

Regarding OpenVPN, I believe that needs a public IP exclusive to me, and my ISP gives me a private IP behind their CGNAT fiber ONU (the public IP is shared with half the town). I think there are ways around that, but that would be exchanging a complication for another.

I will check later the other tips in the link you shared.

17

The IPv6 should be DHCPv6 with Predix Delegation on the FT Router

No the OpenVPN doesn’t need a public ip. I have it set up on the FT router in repeater mode so its not WAN facing and still works just the way I have my ZT setup. So it should work even behind CGNAT since my FT router is double NATd. Remember its OpenVPN Client not Server on the FT Router so it reaches out to a relay point. So it basically the same as how ZT network works to create a WAN Mesh network. Basically its a VPC like AWS Cloud. Also just got the ZT setup on my TV with HDMI amazon FireTV stick, so any TV could be part of my ZT Mesh if it has an HDMI port. LOL

18

The IPv6 should be DHCPv6 with Predix Delegation on the FT Router

Tried all DHCPv6 service types and none made a difference, with or without UPnP and NAT-PMP.
Weird thing is that while zerotier-cli info returns online, zerotier-cli peers lists other nodes, ip a shows the zt interface with IPs (both v4 and v6), and my.zerotier.com shows the FT router as online and with IPs… I still don’t get a response when pinging the router’s ZT IP from another ZT node or viceversa.
Weirder is that tailing the log shows all ZT incoming and outgoing packets as ACCEPTed.

OpenVPN: I would have sworn that it needed an exclusive public IP, but that makes sense given that you are using it as a client and not a server. Does OpenVPN offer a public relay point like ZT does? If so, it is an option, but I would like to find out why ZT works with all my devices but the router.

19

If that’s the case than your probably dealing with an extremely hostile network. Best bet is to give openvpn a shot. Public IP is only needed to the server and not the client side. Yes it does operate as a relay point that you have full control over unlike ZT that handles the relay server side for you, but the setup is way easy than ZT, no ARP OSI 2 or 3 layers config needed, and just 2 IP Tables rules for pings and SSH to the router from any other openvpn client. I tested it on a FT MIPS netgear router and could say with certainty it work perfectly.

20

But if it’s my ISP that is hostile, why it is only the router that is not accesible while other nodes in my LAN (two PCs, one RapberryPI) are reachable from ZT hosts elsewhere (and via that same router at that)?
That seems to indicate something lacking in my FT router and not with my LAN or ISP.

The stubborn in me still wants to find out what is getting in the way of ZT… But I will give OpenVPN a shot indeed. I have a couple doubts with OpenVPN, though:
1- if I connect from my work PC to my home LAN via OpenVPN, does my work PC start to use my LAN as a default route, cutting me off from the office servers? Yes I’m rusty.
2- I have three ZT networks: one with total access for me and only selected LAN hosts for family members. Am I right in thinking that OpenVPN offers that kind of granularity also?

Thanks for all your suggestions!

p.s
Last night I tried adding a firewall rule to always accept incoming and outgoing ICMP traffic
iptables -A INPUT -j ACCEPT -p icmp --icmp-type 8
iptables -A OUTPUT -j ACCEPT -p icmp --icmp-type 8
Before that I could only ping the router’s LAN IP, after that I could also ping the router’s ZT IP, but only from the LAN and not from a ZT node elsewhere. I still can’t ping any ZT node from the router.