Changing the secondary port fixed it, thanks!
The computers are connected with different ISPs, and both routers are provided by the ISP itself so I have little control over them. I suppose either the router goes crazy or the ISP decides to block the ports.