Just want to share my experience to the community about this great tools.
Zerotier is great tools and super powerful, great help for people WFH nowaday.
PLEASE also note that end point protection is super important when using ZT, you don’t want your devices get compromise or get ransomware because of any one of users.
- OPNsense install ZT and OSPF is straight forward. skipped.
- PFsense install ZT will need shell access. it still easy.
Check the ZT version and find the package you want to install at here: https://pkg.opnsense.org/
then in shell run this command:
pkg add https://pkg.opnsense.org/xxxxxxxxxxxxxx/latest/All/zerotier-xxxxx.txz
next install the web interface, find it in github: pfSense-pkg-zerotier
upload to pfsense, you can use upload file function in GUI under Diag>command.
then run “pkg install /path/pfSense-pkg-zerotier.pkg”
- reference in chinese: removed, new user 2 links in a post.
then join you network (ZT bridge selected) , add a allow all firewall rule for the interface, then begin your test. after test add rules suit you.
you may need to aware that Ospf has its own protocol, not TCP nor UDP.
Setup OSPF, its easy and straight forward. Skipped.
so basically, installation and setup is pretty easy. But the real world is challenging.
After everything is setup ( I setup 3 nodes test with OPSF), I did a ping test, first 2 min is good, but after 2 mins, packet loss start showing up, then a few mins, its back to normal.
i kept monitor OSPF and found OPSF flap every 2-3 mins. at first I thought it was some OSPF settings, but I was wrong, wasted many hours here, so don’t want you guys repeat my steps. I noticed the flood in ZT interface, so I use packet capture to see what is those flood. And then found some internal subnet IP traffic flooding. After google it, some call it software laser, and the solution is add a blacklist in this file:
/var/db/zerotier-one/local.conf
{
“physical”: {
“192.168.0.0/16”: { “blacklist”: true }
}
}
Reference: links removed.
According to some source this should be fixed in 2021 but I still having this problem.
Change this file in OPNsense and PFsense is different, it caused me hour to found out.
you can create your file in PFsense, default did not have this file.
OPNsense has this file, but you cannot edit it directly in shell, it works but not after reboot. found out you need to use GUI to add this in OPNsense, because when you reboot it reload what is saved in GUI database.
After that, I think is all good, so reboot all PFsense and OPNsense.
then PFsense not working…
the ZT was started at boot
the OSPF saw the ZT interface for less than a min, then its down and removed from OSPF. because the ZT is not yet connected. OPN sense does not have this issue because you can lock the ZT interface, I guess.
Even the ZT UP and connected the OSPF did not automatically enable the interface, need manual restart FRR service.
Tried many methods delay startup of FRR service, cmdshell, etc. and wasted some hours.
found out what I need to do is, use PFsense FRR Global setting RAW Config, you need to use RAW config for this, because GUI no option for this.
so first copy your running config to saved config, then add “no shutdown” under ZT interface.
my config look like this (only the ZT interface part is shown here):
interface zt3tkicmx12345678
ip ospf network point-to-multipoint
ip ospf area 0.0.0.0
no shutdown
with this, pfsense can be all good after reboot.
now pfsense and opnsense all good after reboot.
OSPF up > ZT UP > routes propagated > ping test between network > OSPF route not flap anymore, no flood, no packet loss, all good.
Cento7 also tested, straight forward, no such problems. but I thought GUI firewall is easier for someone need to maintain it, then PFsense or OPNsense may be more easier for them.
Hopes this could save you some time.
Cheers and Stay safe.
Edit : as a new user only allow 2 links in a post, so some ref links are removed.