Hello, I have a ZeroTier router pod running on my Kubernetes cluster and I want to provide access to certain pods using IP addresses that overlap with the ZeroTier IP block.
The Pod’s zt interface is has joined a ZeroTier network and has an IP address of 172.25.26.21 which was given to it by the ZeroTier controller.
The pods that I want to allow access to on the ZeroTier network are outside of the range of IP addresses ZeroTier will assign to clients, but is still within the netblock.
I can ping the ZeroTier pod (172.25.26.21) from a ZeroTier client and it works, but cannot access the pods on the same network. Here’s a diagram of the network:
Here’s an example of a service I want to give access to over the ZeroTier network running on the cluster:
apiVersion: v1
kind: Service
metadata:
labels:
app: mosquitto
name: mosquitto-zt
namespace: mynamespace
spec:
externalTrafficPolicy: Local
ports:
- name: mosquitto-mqtt
protocol: TCP
port: 1883
targetPort: mosquitto-mqtt
selector:
app: mosquitto
loadBalancerIP: 172.25.26.101
type: LoadBalancer
I have setup NATing both directions on the ZeroTier router pod:
# zerotier-cli listnetworks
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks XXXXXXXXX K8s Pods. No Gateway. XX:XX:XX:XX:XX:XX OK PRIVATE ztkiqtnfto 172.25.26.21/25
/ # ip route
default via 10.244.0.1 dev eth0
10.244.0.0/24 dev eth0 scope link src 10.244.0.153
10.244.0.0/16 via 10.244.0.1 dev eth0
172.25.26.0/25 dev ztkiqtnfto scope link src 172.25.26.21
# iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- any zt+ anywhere anywhere
0 0 MASQUERADE all -- any eno1 anywhere anywhere
141 19672 MASQUERADE all -- any eth0 anywhere anywhere
One notable thing is that for some reason, there was an FQDN request in ARP:
# tcpdump -i ztkiqtnfto -A
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ztkiqtnfto, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:10:57.561060 ARP, Request who-has mosquitto-zt.mynamespace.svc.cluster.local tell 172.25.26.86, length 28
.........;{......V.........e
09:10:58.515072 ARP, Request who-has mosquitto-zt.mynamespace.svc.cluster.local tell 172.25.26.86, length 28
I’m not sure how that happened, in the MQTT client, I put in the Pod’s IP address 172.25.26.101.
I also tried connecting to some mock webservices I setup as pods, but didn’t see a lookup or any traffic in the tcpdump (They are listening on 0.0.0.0, so should accept from any interface/ip).
I’m not sure how to go about having the Pod forward traffic to and from the network. For reference I am using zyclonite/zerotier:router-1.12.2 for the router with ZEROTIER_ONE_GATEWAY_MODE set to both, and sinamics/ztnet:0.5.10 for the Web-UI