Over the last year I have seen this issue on 3 sites (separate networks). It seems to happen on linux nodes, both VMs with Ubuntu 16 and 20, as well as R-Pi. And when only two nodes are on a network, as well as more (10 or so).
This happens almost like clockwork. I have a non-critical setup now to test on. I can set up all sorts of long term tests if someone is interested in engaging and getting to the root cause.
Great to hear from you. I look forward to helping get this resolved.
I have a master node (MN), running on a linux AWS (East) instance. So their (virtual) routers, firewalls.
I have one slave node in California (CA), behind a small business router. I suspect I can find out more
I have one slave node (KT) in the Canary Islands. I will have to research routers.
I have one slave node (CAVM) running on a VMWare VM in California. Same small business router.
The CA and KT nodes are the ones producing these regular disconnects. There is most likely double NATting at KT, but unlikely at CA. I can run any montoring tools you recommend, including tcpdump at any node. I have tried looking at a tcpdump, but did not see anything. I wonder if some other app is stepping on our IP ports but I dont know how to check for that.
I would like to take one pass at checking/monitoring before calling up my remote team and customers to start researching router hardware. Any recommendations? If not, I will start the research.