[TriLUG] BSD/Linux firewall with multiple ISP and failover?

[Jon Carnes]

On Sat, 2006-01-28 at 17:04, Aaron S. Joyner wrote:

> You want something that allows you to have multiple paths to the 
> internet, and should one of those paths die, you want to switch to using 
> the alternate path.  This is actually a very easy thing to do, and only 
> requires a second ethernet interface in the firewall in question (note 
> the word interface, not network card, as technically this could be done 
> with a managed switch, vlans, and some craziness if you want to keep 
> your existing hardware platform).  In short bullet points, assuming you 
> want to use Linux, it'll go something like this:
> 
> 1 - Get one ISP working, dhcp, whatever is required.  Shutdown that 
> interface.
> 2 - Setup the second interface, get that ISP's connection working, shut 
> that interface down.
> 3 - Pick a few reliable hosts on the internet, I'd choose 6, to use as a 
> measure of connectivity.
> 4 - Configure DHCP on the backup internet connection not to write the 
> default gateway or resolv.conf.  It helps if this connection has a 
> static ip / default gateway.
> 5 - Bring up both interfaces, and things should work as expected.  Note, 
> you won't properly respond to traffic on the secondary interface, but 
> having that interface turned up shouldn't interfere with the primary 
> interface.
> 6 - Setup iproute policy routing such that traffic leaving your 
> secondary interface has the gateway set for the secondary default 
> gateway, and if your primary is also static you may be able to do the 
> same for the primary ISP, or at worst you can leave it in the default 
> table.  This is a common technique for multi-homed servers, see here: 
> http://www.linuxjournal.com/article/7291 for more information.  This was 
> the first google result for the query ["ip rule" multihomed], fell free 
> to look for other sources of how to setup multihomed servers to get a 
> better feel for using the 'ip rule' and 'ip table' set of commands.  
> Reading to have a thorough understanding of these topics is required for 
> you to complete steps 7 and 8.
> 7 - Setup custom "ip rule"s to each of your test hosts, to ensure that 
> traffic to that test host goes over the correct interface.
> 8 - Write a short script which attempts to connect to each of your 
> primary isp's test hosts to validate that connection is valid.  If those 
> tests fail, try the secondary isp's test hosts, if those succeed, change 
> the default 'ip rule' to point t othe other table (see docs referenced 
> in step 6 for more detail).
> 
> Come back and post again if you can't get it working correctly.  :)
> 
> Good luck Greg,
> Aaron S. Joyner

Hmmm, interesting but a bit complex. I prefer to simply have the
secondary take over the IP address of the primary - when the primary
goes down.

If the internal primary interface has address 192.168.1.1, then the
fail-over firewall runs this line:
  ifconfig eth0:0 inet 192.168.1.1 netmask 255.255.255.0

===
You could initiate the fail-over with a script that uses a simple ping
to see if the Primary server is up...

#! /bin/bash
#
# Server_Check: Run a minute by minute check of the
#   Master server with internal address
#   of 192.168.1.1 (and secondary internal address
#   of 192.168.10.1), trigger Failover if
#   Master goes off-line, trigger Backdown if
#   Master goes back on-line. 
#   Run via cron - every minute
#     * * * * *   /usr/local/sbin/Server_check
######
#
# Check for existance of Trigger file
#   0 = Normal (Master is fine)
#   1 = Failover (This server has taken over)
#
if [ ! -f /root/config/trigger ]; then
    echo 0 > /root/config/trigger
fi

TRIGGER=`cat /root/config/trigger`

# Do three pings in a minute.  If all three pings fail
#  j="xxx" and we fail the server over (if it's not
#  already in that state).
#  If one of the pings works then we assume that the
#  Master is up and we return to normal (if we are not
#  already in that state).
#
j=""; 
for i in 1 2 3 ;
   do 
   ping -qc1 192.168.10.1 >/dev/null || j="x"$j; 
   sleep 10; 
done; 

if [ "xxx" = "$j" ]; then 
  if [ ! "1" = "$TRIGGER" ]; then
    /sbin/ifconfig eth0:0 inet 192.168.1.1 netmask 255.255.255.0
    echo 1 > /root/config/trigger
    echo "Primary Firewall has failed - Secondary taking over" |mail -s "ALERT: Primary Firewall is down" root
  fi
else
  if [ "1" = "$TRIGGER" ]; then
   /sbin/ifconfig eth0:0 inet 192.168.1.11 netmask 255.255.255.0
   echo 0 > /root/config/trigger
  fi
fi

===
I like this because the Fail-over server does all the checking. It uses a secondary network (192.168.10.0) that is shared with the Primary Firewall. All testing is done across the secondary network. This lets you manipulate the primary network (192.168.1.0) and move the gateway for that network anytime you want, while still letting you test to see if the Primary Firewall comes back up.

It's elegant and it works great.
Good Luck - Jon Carnes