Using ipv6 Only to Connect to Zerotier

I have a ipv6 only server and am using a tunnel to grant access to ipv4 network on the server. However, all connections (web server, vnc, etc.) except for ping and ssh results in a timeout when I have ipv4 enabled for the server. Is there a way to use ipv6 only to connect to zerotier on this server, as it functions just fine without ipv4 connection?