Cisco NX-OS for remote management

cyrus · March 2, 2024, 3:54am

Hello!, has anybody explored installing Zerotier client in the management plane of Cisco NX-OS based switches?.

They seem to support installation of third party software through RPM files:

CiscoDevNet/NX-SDK/blob/master/readmes/NXSDK_in_NXOS.md

# Deploying NX-SDK Application in NX-OS

# Table of Contents

  - [Description](#description)
  - [NX-OS Build Environment](#nx-os-build-environment)
  - [Supported Languages](#supported-languages)
  - [Custom Application Development Flow using NX-SDK](#custom-application-development-flow-using-nx-sdk)
    * [1. Install NX-SDK](#1-install-nx-sdk)
      + [a) NX-OS Build Toolchain](#a-nx-os-build-toolchain)
      + [b) Get NX-SDK toolkit](#b-get-nx-sdk-toolkit)
    * [2. Building Custom Applications](#2-building-custom-applications)
    * [3. Unit Testing Custom Application](#3-unit-testing-custom-application)
    * [4. Packaging Custom Application](#4-packaging-custom-application)
      + [a) Auto-generate RPM package using Script](#a-auto-generate-rpm-package-using-script)
      + [b) Manually-generate RPM Package](#b-manually-generate-rpm-package)
    * [5. Installing Custom Application in NX-OS Switch](#5-installing-custom-application-in-nx-os-switch)
    * [6. Running Custom Application in Switch](#6-running-custom-application-in-nx-os-switch)
    * [7. Verify if Custom Application is running](#7-verify-if-custom-application-is-running)
    * [8. Stop Custom Application in NX-OS Switch](#8-stop-custom-application-in-nx-os-switch)

This file has been truncated. show original

l0crian · March 2, 2024, 4:54am

N9K has support for Dockers with host networking. That’d ultimately be a cleaner way to install it. Should work just fine for MGMT functions. Only issue might be /dev/tun.

Dockers are also supported on a number of IOS-XE devices like models in the Cat9K family. It might be possible to install it on those as well, but I’m not sure if you’ll be able to use it for MGMT.

cyrus · March 2, 2024, 10:06pm

I expect to receive a N5K-C5548UP for testing, any chance there?, where should I start?

l0crian · March 3, 2024, 2:04am

Docker support only appears to be in N3K/N9K from 9.2.1. Here’s documentation for N9K:

I wouldn’t expect it to be supported on N5K/N7K.

I did do some testing earlier with an N9000v running 10.2.1, and I was able to install ZeroTier in a container, and even bring up the control-plane, but as I mentioned, I wasn’t able to mount the ‘/dev/net/tun’ module. It appears to never have been loaded into the Kernel based on modprobe.

I did further testing with WireGuard, and was able to build WireGuard between the Nexus and another system. The dockers on Nexus get thrown into the management namespace by default, so I was even able to ping the mgmt0 interface IP through the WireGuard tunnel. I wasn’t able to SSH into it however, even though I can obviously SSH into the mgmt IP when not through that tunnel. Probably some intentional namespace wonkiness Cisco did to protect the Bash Shell.

My expectation is that the Linux sub-system is going to be very limiting by design, so I’m not sure how useful it will be ultimately to try to create an on-system OOB solution. You may be able to create something outside of ZeroTier, though it would require really digging into the namespaces in use on it. ZeroTier will have issues with the TAP/TUN module not being built into the Kernel, so getting it working would be an uphill battle.

Really the best way to manage Nexus using ZeroTier would be to build a dedicated Out of Band network and connect the management VRF into it. OOB networks I think is one of the strongest ZeroTier use-cases for enterprise customers. With that said, I’m not discouraging you from testing further. At the very minimum, it’d be a fun project to mess with. If you’re able to make any progress with it, I think it would have a lot of utility for a number of users.