Simple Use Case Scenarios?

[Pushshift]

First, I am really impressed with this product. My use case is that I have T-mobile 5G home internet and want to link a remote machine (Linode) to take advantage of a static IP address. Ideally I would just want to route any traffic that hits the public IP on the Linode box and then it will forward over the Zerotier tunnel to my on premise network and traffic would be routed accordingly. I’m not sure what the networking name is for “forward everything from one IP to another IP and have responses go back through the original IP” but let me try asking a more simple question.

I want to forward all port 80 / 443 web requests to the Public IP (linode box) to my web server on premise (and have responses go back out through the linode box. I have installed Zerotier services on both machines and I can confirm that both machines can ping one another over the IPs at each end of the tunnel.

Let’s say the public IP is 1.2.3.4 and the tunnel IPs are 192.168.194.1 (Linode public side) and 192.168.194.2 (On premise web server). What I would like is all requests to 1.2.3.4:80 and 1.2.3.4:443 get routed through the tunnel to the web server and responses from the web server get routed back through the tunnel to go back to the end user through the linode public IP. From my (poor) memory of a networking class over a year ago, I know I need to enable “net.ipv4.ip_forward=1” on both machines in /etc/sysctl.conf. This part has been completed successfully. Another part that I (barely) remember is that there are probably post-routing IP table rules that need to be created and probably masquerading needs to be enabled so responses are sent back correctly.

Can someone point me to something in the knowledge base that goes step by step how to forward traffic for a port over the tunnel (or show it here for extra points).

My last question is – can I have all traffic transparently routed to a machine on-premise (port 22 would go to port 22 on-premise, etc.) – is that much different than the previous scenario?

Thanks again for all your help!

[Pushshift]

I am a big fan of updating and replying to my own questions when I find the answers just in case someone in the future has a similar question and can derive value from my own.

I have now figured out how to route web requests appropriately, but I’m still trying to figure out how to do it without using MASQUERADE (which replaces the source address). The reason this is problematic is that the HTML logs will show the originating IP as the VPN public IP and not the true client IP. Below is how I got it working.

Using the same IPs as my original question:

Public VPS IP: 1.2.3.4
Public On-premise IP (The Tmobile 5G Home Internet Side): Dynamic / changes

VPS Tunnel IP: 192.168.191.1
On Premise Web Server Tunnel Side: 192.168.191.2

The goal is to first get port 80 requests made to the VPS public IP to get to the webserver successfully.

Here are the iptable commands:

iptables -t nat -I PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.191.2:80
iptables -t filter -A FORWARD -d 192.168.191.2/32 -p tcp --dport 80 -j ACCEPT

The above two iptable rules (entered on the VPS machine) will reroute requests coming in on the public IP destined for port 80 and send those requests to the on-premise web server. After using tcpdump, I was able to verify that the packets were getting to the machine, but these two rules alone still didn’t route the web server responses back through the VPS server. Basically with these two iptable rules, the request is forwarded to the web server but then the webserver sees the original source IP and tries to send the reply directly from the web server (which won’t work).

Adding the following rule (to the VPS machine) fixes that issue but then adds a new problem:

iptables -t nat -A POSTROUTING -j MASQUERADE

This rule essentially replaces the source IP information in the packets with the IP of the VPS host. The web server will now send the replies back to the VPS machine as we would want but the problem is the source IP which would usually have the IP of the client (end user with a browser making a request to the web server) now has the IP of the VPS machine. So we’re able to contact the web server via the VPS public IP and run a web server with a static IP address but the web server logs will only have the VPS ip as the only client making requests to it.

Unfortunately iptables only works at layers 3/4 (from memory) so we need to find another solution. I was able to find that solution but then it breaks internet requests on the web server. If we remove the MASQUERADE iptables rule and then add the following routes (on the web server machine on-premise), everything works and the true client IP shows up in the logs but other things start to break:

Remove MASQUERADE and route all traffic through VPN and add the following to web server

sudo ip route del default
sudo ip route add default via 192.168.191.1 dev ztc3q5l333

These two routes force the web server to send traffic over the tunnel back to the VPS machine, but it breaks other things like IPv4 internet routing on the web server.

I’m making a lot of progress but I’m stuck here at the moment. What I need to figure out is a way to selectively send only packets forwarded from the VPS machine and mark those as ones that need to take the tunnel route. I think the solution is some combination of iptable / route rules – but I’m not sure what at the moment.

I’ll keep trying and report back if I find a solution that works.