[TriLUG] Multiple Public Interfaces == routing confusion

[Brian via TriLUG]

Hi Gang,

Recently, I had Lumos install fiber at my house.  Now my router (a 
Debian 11.3 system) has two public interfaces: one for Lumos, and one 
for the Spectrum service I still have.

I am using iptables for firewalling, as I have yet to buckle down and 
learn about firewalld.

The router has a public IP address for each public interface.  From the 
router, I can ping and traceroute out through either interface successfully.

I have the IPv4 routing table set up simply with the Spectrum interface 
as a higher-metric default gateway.  I can ping and traceroute through 
either interface without issue.

The trouble, as the astute will have already surmised, is that any 
packets coming in on the Spectrum interface get answered by way of the 
Lumos interface since it has a higher priority in the routing table. 
I've verified this by using logging rules in for ICMP packets in 
iptables; I can clearly see an echo-request come in on the Spectrum 
interface and the echo-reply go out the Lumos interface.  Of course that 
response packet gets dropped by the first stateful firewall it 
encounters on its way back.

What's the secret sauce to ensure services answer packets through the 
correct interface?  The whole reason I've not canned Spectrum 
immediately is that the Lumos connection hasn't proven to be very 
stable, and I want to be able to get in from outside through either 
interface if one happens to be down.  Do I just need to rig up a proper 
failover setup?  How difficult is that to do?

Thanks,
-Brian