ip rules to route packets back out are missing

[cedar stone]

You seem to support the capability of having a "multi-homed" setup, whereby you have multiple network interfaces connected to the HAOS system. However, it seems that you've missed a crucial step in setting up the Linux routing underneath, if I'm reading this correctly (difficult, because I'm not sure which container I'm actually in), but it explains my problem very well.

When setting up multi-homed systems, it's vital that you use connection marking and policy based routing to route reply packets back out on the network interface they were received from.

Network Manager has some notion of policy routing, and I believe that's what you're using deep under the covers.

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-policy-based-routing-to-define-alternative-routes_configuring-and-managing-networking

Unfortunately, I don't seem to be able to get to the root OS layer, and verify my guesses, and browsing the supervisor source code it's difficult to see what you're doing, but keyword searches don't seem to turn up anything related to policy based routing.

FYI this is what I had for each of my VNICs on my prior "core" based system.

iface eth1 inet static
    address 10.0.0.220
    netmask 255.255.255.0
        up iptables -t mangle -A OUTPUT -m conntrack --ctorigdst 10.0.0.220 -j CONNMARK --set-mark 2
        up ip rule add fwmark 2 table 2
    up ip rule add from 10.0.0.220 table 2
        up ip route add 10.0.0.0/24 dev eth1 table 2
        up ip route add default via 10.0.0.1 dev eth1 table 2

auto eth2
iface eth2 inet static
        address 10.0.100.2
        netmask 255.255.255.0
        up iptables -t mangle -A OUTPUT -m conntrack --ctorigdst 10.0.100.2 -j CONNMARK --set-mark 3
        up ip rule add fwmark 3 table 3
        up ip rule add from 10.0.100.2 table 3
        up ip route add 10.0.100.0/24 dev eth2 table 3
        up ip route add default via 10.0.100.1 dev eth2 table 3

As you can see, I use connection mangling to tag each packet on an interface with an "fwmark". Then I use the policy routing to direct the marked packets back out of their origin interface. A similar trick should work for HAOS and would dramatically improve the multihomed experience.

For others who might end up here trying to figure out a solution: I have temporarily enabled "sloppy" connection tracking in my OPNsense firewall. This is very suboptimal because it has all sorts of security and other implications, but it means things for the most part work. (Some well configured devices will still not work, because they reject the packets "going the wrong way")

As a final note: everything I say is speculation. I can't find a way to escape the docker container to inspect the underlying host configuration so it's quite plausible I'm not seeing the full state. A documented escape mechanism to the host for diagnosing these kinds of issue would be greatly appreciated.

[frozen wind]

This might be useful to you:

• https://gist.github.com/Impact123/e9a4a07b184eb393d2ff762e3b1b0a05#run-commands-in-a-containeraddon
• https://gist.github.com/Impact123/e9a4a07b184eb393d2ff762e3b1b0a05#get-a-os-level-shell
  Read this before: https://gist.github.com/Impact123/fb086b391f7d14cb3515144fcbe4785e#background

[cedar stone]

Following the recommendations above, my hypothesis seems confirmed - you're not configuring policy routing at all, and so traffic is just going out of whichever interface "wins"

Which is absolutely less than ideal, sadly.