I’m very impressed by the good design and ease of use of Zerotier.
I’m however a bit concerned about the security of ZT network controllers.
The documentation clearly states that network controller identity (i.e. private key, correct me if I’m wrong) must not be compromised. A compromission would allow the attacker to make any kind of modifications to the networks managed by this controller (accept new hosts, modify rules, etc …).
However, I don’t see anywhere in the documentation a reference to a proper way to secure this identity.
For instance, it seems this identity cannot be stored in a HSM (hardware security module). This would reduce the risk in the sense that the attacker will never (or very hardly) be able to copy the identity. Hence once the compromission is detected and remedied, the attacker has no more access to the identity so no more ability to interfer with the ZT network.
This is kind of a concern for me, especially for ZT hosted network controllers. I have good confidence that ZT pals did their best to protect them, but as usual, no one is perfect. And the fact that ZT pals have such great powers means that they suddenly become a very interesting target of attack from my point of view.
Is there anyone out here who could develop an argument around this (e.g. explain me that I’m plain wrong or second my thoughts) ?
Given the nature of your question about security of our hosted network identity keys, I’m not going to go into a lot of detail of our internal security practices in my answer here. Suffice to say, the only people in the organization that have access to the keys are those that need access to the keys to do their job. Access is audited as well. Same goes for access to the machines our controllers are hosted on.
You’re correct that currently the identity cannot be stored in an HSM. Requiring an HSM for our hosted controllers would actually hinder us more than help. For one, we’d need 30 (and growing) HSMs for our controllers. It would also tie controllers to individual machines, and right now they float around between machines as needed. Additionally, back on May 4 GCP in us-west2 had a 4 hour outage. We were down for just over an hour of that because we were able to move everything to a different region before GCP figured out & fixed their problem. Shipping 30+ HSMs half way across the country would take have taken considerably longer
You’re also correct we don’t specify how to securely store your own identities for your own nodes or self hosted controllers. The “Proper” way often depends on the use case & risk tolerance. All that being said, the default state of a ZeroTier install only allows those with administrator access to the machine to access the private identity. If a user so desires, said identity could be a link to /dev/shm (on Linux) so that the identity is never written to disk. It’s up to the end user to ensure the data is present prior to starting ZeroTier in this configuration.
All that being said. Yes it’s important to secure your identity key. How you do that depends on you, your use case, and your risk tolerance. Having the private identity key alone is not enough to change network configurations, though. There’s nothing in the ZT protocol itself that allows the reading or changing of network configuration remotely that having the private key would enable. At most, it enables a denial of service attack. While that may still be problematic, it’s not retrieving & changing network configuration details.
Thank you very much for your very detailed explanation. Very appreciated to have a quick answer with such a quality of information.
A very important point for me is your last point about the fact that compromission of the private key would not allow much modifications of the networks except a denial of service (which is still a concernbut much lower in a risk evaluation from my point of view).
However, the reasons behind this point are not very clear for me. And the documentation is not stating the same (cf section “Controller Security Considerations” in paragraph 2.2.1 of the manual). Could you please develop a bit your reasoning ?
I’m pleased and aligned with all other points, once again, very appreciated to find such sound arguments / point of views. Maybe those elements should find a place somewhere in the manual or I’m a FAQ. I might have missed them, but they are of prime importance to me (and probably to many other corporate teams).
For the scenario in the manual to be effective, one would also have to get all of the network IDs & configurations. In a self hosted controller situation, this is all stored on disk in the ZeroTier working directory. For our hosted controllers, this is not the case.
