Direct Kubernetes API access?

[davosian]

Is it possible to use ZT to connect to a kubernetes cluster through kubectl from a machine across the internet? Currently the cluster has been assigned a 192.168.x.x local IP and is therefore only reachable in its home network. Would I need some kind of jump host with ZeroTier running or is a more direct connection possible, e.g. through a pod or some kind of service running inside the cluster?

[zt-grant]

For kubectl access, you’ll need a router or bridge running zerotier to get you into the network the cluster IP is in. This may be possible to do via a pod if it’s assigned an address in the same network as the cluster IP.

Outside of kubectl, it’s also easily possible to run ZeroTier as a sidecar container in a pod and access the services in the pod via ZeroTier with something like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kube-hello-world
  name: kube-hello-world
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kube-hello-world
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kube-hello-world
    spec:
      containers:
      - image: my/nginx-server:0.0.1
        imagePullPolicy: IfNotPresent
        name: kube-hello-world
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - image: zerotier/zerotier:1.6.5
        args: [ "8056c2e21c000001"]
        imagePullPolicy: IfNotPresent
        name: kube-zt
        resources: {}
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - SYS_ADMIN
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /dev/net
          name: dev-net
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /dev/net
          type: ""
        name: dev-net

Note: that this is a very naive implementation. The identity is regenerated on every start of the container.

ZeroTier can also be run as a DaemonSet for access to the individual nodes in a cluster as well, if you so desire.

[davosian]

Wow, definitively more than one option. The sidecar approach might actually be a good fit for my use case. Will give it a shot Thanks @zt-grant

[cadayton]

@davosian Let me know how this works out for you. I’m interesting in doing the same but hoping to find enough people to contribute device to the cluster.

[davosian]

Well, some initial testing was successful: I can attach the sidecar to a sample podinfo container, join the ZT network and access the pod at its http port 9898 just fine:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: dev
spec:
  minReadySeconds: 3
  revisionHistoryLimit: 5
  progressDeadlineSeconds: 60
  strategy:
    rollingUpdate:
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9797"
      labels:
        app: podinfo
    spec:
      containers:
        - name: podinfod
          image: ghcr.io/stefanprodan/podinfo:6.0.3
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 9898
              protocol: TCP
            - name: http-metrics
              containerPort: 9797
              protocol: TCP
            - name: grpc
              containerPort: 9999
              protocol: TCP
          command:
            - ./podinfo
            - --port=9898
            - --port-metrics=9797
            - --grpc-port=9999
            - --grpc-service-name=podinfo
            - --level=info
            - --random-delay=false
            - --random-error=false
          env:
            - name: PODINFO_UI_COLOR
              value: "#34577c"
          livenessProbe:
            exec:
              command:
                - podcli
                - check
                - http
                - localhost:9898/healthz
            initialDelaySeconds: 5
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command:
                - podcli
                - check
                - http
                - localhost:9898/readyz
            initialDelaySeconds: 5
            timeoutSeconds: 5
          resources:
            limits:
              cpu: 2000m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 64Mi
        - image: zerotier/zerotier:1.6.5
          args: ["mynetworkid"]
          imagePullPolicy: IfNotPresent
          name: kube-zt
          resources: {}
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
                - SYS_ADMIN
            privileged: true

Now I am trying to add the sidecar to a helm chart which does not provide for sidecar containers through its values file. I am trying to modify the deployment with kustomize patching, but I have not been successful so far (which is more related to my kustomize knowledge than anything else).

Then I would have to address the issues mentioned by @zt-grant. For one, I do want to reuse the identity created and do not want it to rejoin the network each time a new pod comes up.

[cadayton]

Thanks for the info.