More Synology Questions

Sorry in advance for the length.

I’ve read the various posts here regarding Synology. Mostly they seem kind of vague. Perhaps because of this, my questions aren’t yet detailed, technical ones. What I’m after is some clarity before I decide how to proceed. Let me explain.

My company, a locally owned daily newspaper, while fairly small, nonetheless spans 3 offices in 3 cities. Thanks to Covid, we now support a fully mobile workforce. When I think back to what our daily operations looked like just a few years ago, I find it all unrecognizable compared to today. Nobody here wants to go back there.

The key to all of this has been ZeroTier. As we’ve grown our virtual LAN, I’ve become a rabid evangelist for the platform. I tell friends and colleagues that ZT has become as fundamental to our operation as wire. This is not an exaggeration. There simply is no better solution for us. All of you at ZT should be very proud of yourselves. You are rock stars.

Replacing our previous strategy of monolithic file servers, rooted as it was in the 90’s, with Synology units has been no less impactful. Among the many benefits has been the ability to give our photographers a proper archiving solution for images. PhotoStation in DSM6 really is world-class. It gives us the ability to have literally decades worth of published photos - hundreds of thousands of them - at our fingertips, all with full IPTC metadata on display. This is only one of the benefits we’ve seen from using Synology. There are many more. This has become a strategic decision that we won’t be able to walk back for some time, if we ever do.

Here’s where things begin to get murky. I get that Synology threw a curve-ball at ZT with the privilege issues in DSM7. They also threw us a curve ball by killing PhotoStation. I’m no happier with some of their recent decisions than you are. Until I can chart us a course away from PhotoStation, we’re stuck on DSM 6.2.4.x. Fortunately, DSM6 will not be EOL until some time in 2023, so I have a little room to maneuver.

My issue is that supporting all of this will present more and more administrative and technical challenges going forward, not to mention trying to explain it all to management.

So, my questions are these;

Is ZT 1.4.0 still OK to use? Do I have to worry about the protocol changing, orphaning a version this old? Will I eventually have to freeze versions on all my clients to keep this running?

Since DSM6 is not EOL until 2023, is there a chance in hell of getting newer, native builds of ZT for DSM6? Screw the GUI. A purely CLI version as a native SPK not requiring Docker would be plenty good enough. I doubt using SSH to join a network would be a serious hurdle for most of ZT’s target audience. Given the spotty nature of Docker support across Synology’s platform, and the number of users I’ve seen on Syno forums also digging their heels in on DSM6, the idea doesn’t seem unreasonable until DSM6 is officially EOL. I doubt I’m the only one who wants this.

If ZT is unwilling to build these newer versions, is a community build using spksrc feasible/acceptable? I thought about trying it, but It’s not something I want to devote resources to if it’s a dead end.

In one of the posts I read here, someone claimed that Synology does allow root access for packages if they’re signed by Synology. Is there anything to this? Do they have a vetting process for developers (like Apple, for example) that would provide the needed code signature, opening the door to native packages for DSM7?

I get that I’m glossing over the use of Docker here. I’ve tried the official, documented procedure on two different machines, multiple times. I’ve also tried several of the recipes floating around on the net. Nothing has worked yet. I’m sure I can get it to work, but before I beat my head against Docker any further, I thought it would be best to make sure it’s going to be my only choice.

1 Like

Hello dbronson.

Thank you for the kind words! You’re right that others are dealing with the same issue. I’ve whipped up a batch of fresh 1.8.6 packages for DSM 6.2.4 but since I no longer have a Synology unit running DSM 6 I haven’t tested them. If you could try these out and let me know which models worked or didn’t work that would be very helpful.

Is ZT 1.4.0 still OK to use? Do I have to worry about the protocol changing, orphaning a version this old? Will I eventually have to freeze versions on all my clients to keep this running?

Our plans are to keep all future 1.X series versions backwards compatible. (For example I intentionally keep my father on a 1.1.4 client to watch out for breakages.)

In one of the posts I read here, someone claimed that Synology does allow root access for packages if they’re signed by Synology. Is there anything to this? Do they have a vetting process for developers (like Apple, for example) that would provide the needed code signature, opening the door to native packages for DSM7?

Correct, we’re beginning talks with Synology to explore this.

I’ve tried the official, documented procedure on two different machines, multiple times. I’ve also tried several of the recipes floating around on the net. Nothing has worked yet.

If you ever do re-attempt this just let me know where you get hung up and I can assist. I use the docker image exclusively.

Good luck and I look forward to hearing if these packages work for you!

Sorry for the delay. I was sick for awhile, then had to catch up, etc…

Anyway, I tried to keep the install GUI only, since I thought this would be easiest for most people. First, I tried simply manually installing. This resulted in the Syno package manager stuck with a perpetual “Loading…” spinner. Checking ZT-Central, the machine was still offline. I refreshed the DSM page, then did an uninstall, and installed via GUI again.

This time, it seemed to work. Central shows the machine online, and the version is now at 1.8.6. Via ssh, synopkg shows the package as “started.”

Sadly, nothing else works. I deliberately turned off the Syno firewall to eliminate variables, but I can’t even ping the ZT address of the box. Very strange.

Incidentally, this is the Denverton build on a DS1819+ with DSM 6.2.4-25556 Update 5.

Thoughts?

Addendum;

ps shows: /usr/local/zerotier/bin/zerotier-one -d

CLI reports ONLINE.

Again, the Syno firewall is off.

Next, I uninstalled/reinstalled 1.4 via the GUI. Weirdly, the same symptoms are now happening. I’m going to reboot the box…

Yikes. Things got even weirder. Central now shows online, and version 1.4. Just to be thorough, synopkg and ps look normal. I even verified the presence of /dev/net/tun. Still no traffic, whereas it all worked before I began this. Now I’m dead in the water.

Interestingly, if I ping the ZT address in question from a ZT node outside my LAN, nothing. If I ping it from another node on my LAN, nothing. However, if I ping it from the Syno CLI, the pings come back.

Also, just on a lark, I port-scanned the physical LAN IP. Port 9993 is open. There are also a couple high ports open that my gut tells me look like older versions of ZT - 34373 & 4.

Fortunately this is not a production machine - it’s my personal one at home, so I can leave it like this for a few days if you need more info.

Oh, and FYI - my attempts with Docker produced very similar symptoms. Everything seemed to go by the book. Central showed the device as online, and the correct version. However, no traffic would flow. I would stop/delete the container, reinstall 1.4 SPK, and all would return to normal. The difference this time is that it has not returned to normal.

This made me suspect something was up with my machine, but the same thing happened on 3 different ones, so I can’t seem to narrow it down. It’s very possible I’m missing something obvious.

Took a little longer than I had hoped, but I discovered that there was both a regression in our package and a bug in Synology’s synonetd which results in broken packages. I’ve fixed the regression in the package and also included a workaround for the bug in synonetd. The new 1.8.7 packages are here.

The workaround will keep a small watchdog daemon running to check for when synonetd erroneously deletes routes needed by ZT and will re-add them.

When starting the package you may notice a little sputtering connectivity but this is normal and should stabilize 15s after starting the ZT package.

Please note that this package stores identities in a different location on the FS and will be completely deleted when the package is uninstalled so make sure to back up your identities. (/var/lib/zerotier-one)

I hope this helps!

OK - before I begin, will this new version pick up the existing identity automatically, or do I need to relocate it?

UPDATE: Since it’s my personal machine, it’s easy for me to use a new identity, so I backed up the identity anyway, then charged ahead, assuming it would generate a new one.

zerotier-cli info now gives → 401 info {}

Telling it to join produces no result (just empty brackets again.) Central doesn’t see it. I even rebooted just in case. Same deal.

I apologize for wasting your time on that one. I also just saw the same issue and it’s because I moved the ID storage location to the recommended spot on the FS but failed to provide a symlink in my latest package revision. Fixing now.

While you wait you can use zerotier-cli -D/var/packages/zerotier/var info

Sorry. Took the evening off.

That worked perfectly. My box is online and traffic is flowing.

I’ll hold off on deploying this at work until you’re happy with it, but otherwise, I’m very impressed.

Thank you VERY much.

P.S. (If you like, I’ll mark your last post above as the accepted solution, or I can wait till you’re satisfied with a final build. Your call.)

Your testing has been very helpful. Thanks.

If you re-download from this link the packages have been updated (sorry about the same rev number, it’s still 1.8.7-1): Index of /RELEASES/1.8.7/dist/synology/

This update will intentionally destroy existing identities so make sure you back them up if needed, removing identities seems like the cleanest and most correct way to do things even if inconvenient. It also fixes a bug where the route watchdog daemon ate a little too much CPU.

I’d roll this out slowly and if you find bugs feel free to report back and I can keep iterating on the package for you.

For reference, here’s the new script that keeps checking the routes and re-adding them: ZeroTierOne/start-stop-status.sh at dev · zerotier/ZeroTierOne · GitHub

Wow, this is awesome. I’ll begin with it tonight/tomorrow at home, then I’ll cautiously update the Syno at work over the weekend so the users don’t see any glitches.

I also manage some Syno boxes for friends, so with any luck I’ll be able to test something other than a Denverton build for you in the next week or so. I’ll report anything interesting here.

One final question while I’m thinking about it. What I think I’m seeing is older ZT installs opening random ephemeral ports, but with newer builds, they seem to count up from 29994. Do I have that right? If so, that predictability will allow me to tighten up my firewall a tiny bit, and every little bit helps.

Awesome, and that is orrect, in older versions the ephemeral ports were randomly generated for NAT purposes but the seed was the node’s identity which caused issues with some routers that have poor NAT implementation. The newer versions of ZT try to be a little more intelligent about port randomization and this should result in slightly more predictable ports. If that isn’t enough you can manually set ports via local.conf.

1 Like

I successfully applied and tested the latest Denverton build. For my own notes, and anyone who comes after, here’s a step-by-step;

1 - Log in to DiskStation Manager 6, but not via ZeroTier. I did it remotely using a machine on my home LAN to reach DSM, but I did not use a direct ZT connection for obvious reasons. If your Syno is local, you have no problems.

2 - SSH into your Syno, ideally from the same machine as step 1. This presumes you’ve set it up for SSH access, and have basic familiarity with things like sudo. If not, Google is your friend. You’ll want to set up ZT from the command line.

3 - Log in to ZeroTier Central. Updating ZT will generate a new identity and you need to have ZTC up to manage the new node.

4 - Using the DSM Package Center, click on “Installed” in the left sidebar, and uninstall any previous version of ZT.

5 - Still in Package Center, click on “Manual Install.” Locate the spk file you downloaded from Joseph’s post above, and install it, clicking through any warnings about security. Tell it to run after installation.

6 - In the Syno CLI, you can now type;

sudo zerotier-cli info

…and you should see;

200 info ‹node ID› 1.8.7 ONLINE

7 - Great. Everything is working. Get your network ID from ZTC, and;

sudo zerotier-cli join ‹network ID›

This should result in;

200 join OK

8 - Congratulations. You’re connected. Now, in ZTC, add a name and description for the new node. In my case, my Syno uses an address I statically assign, so before I authorized the new node, I deleted the address from the old ID, deauthorized it, and deleted it. I then gave the address to the new node, and finally authorized it. Done. For extra credit, you can try to ping the new node to make sure traffic is flowing.

This is definitely a few more steps than the typical, dead-simple update/install we’ve come to expect from ZT, but IMO it’s simpler, and more approachable for many users than using Docker. It should also cover machines that can’t run Docker at all.

Mad respect for Joseph for all his help. He didn’t have to take the time to do this, yet he did. Over the years I’ve been involved with many contractors, and open-source projects where dismissiveness, and arrogance were a barrier to getting anything done. This thread is the complete opposite of that. At every turn, I’ve found myself genuinely pleased that we chose ZeroTier. Thanks again, guys.

1 Like

For the record, @BB3CPO reported success with geminilake in another thread. Thanks!

Thanks for reporting back and thanks for the help testing!

Hmm. I arrived at work this AM to find my Syno at home unreachable. Central showed it online, but no pings. I got in via ssh by hopping through my laptop, and zerotier-cli showed a status of “OFFLINE.” I used synopkg to stop/start the service, and all is well.

I obviously can’t reasonably claim this has anything to do with this build, but this has never happened before, ever. I’ll keep an eye on it, and try to get more info if it should happen again.

Hmm, that is odd. Anything you find we can try to interoperate into a revision. My device has been working but I’ll keep an eye out. Thanks.