[TriLUG] Routing...once again.

[Aaron S. Joyner]

Greg Brown wrote:

> Brian:
>
> You should have a default gateway for each nic, not just one for the 
> entire
> machine.  I assume there is a dual port fireall with 1.1 and 10.1 and a
> single Internet connection?

<rest of conversation snipped, so as not to cause further confusion>

Let me start by saying that Greg is a well meaning guy, who's generally 
on the ball.  With that peasantry out of the way, you're way off base on 
this thread Greg.  :)

To clarify, layer 3 network routing, ie. the kind done by the routing 
table on a Linux box, is not interface specific.  It's part of the 
TCP/IP stack, which chooses which lower level interface is the 
appropriate interface to send a packet out.  Consider a simple 
situation, similar to what Brian originally described.  You have two 
networks, both which are paths to the single larger network (think of it 
as the Internet if you like).  The diagram looks like this:
(fixed width fonts are good for you)

=== Intarweb ===
 ^            ^
 |            |
NetA        NetB
 ^            ^
 |            |
 \--- Bob ----/

On the left side (eth0), you have an IP address of 1.1.1.1/24, and on 
the right side (eth1), 2.2.2.2/24.  You have a default gateway pointing 
to 1.1.1.254 out the left interface.  Your routing table would look like 
this:

asjoyner at bob:~$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt 
Iface
1.1.1.0         0.0.0.0         255.255.255.0   U         0 0          0 
eth0
2.2.2.0         0.0.0.0         255.255.255.0   U         0 0          0 
eth1
0.0.0.0         1.1.1.254       0.0.0.0         UG        0 0          0 
eth0

When you address a packet to 2.2.2.10, that Linux box will look at your 
routing table, and choose the network with the best match for that 
packet, which is 1.1.1.0 with a mask of 255.255.255.0.  It will then 
route it out that interface.  That will result in an arp lookup, which 
will then allow the packet to be delivered out eth1.  If you address a 
packet to 1.1.1.10, the same occurs, eventually out eth0.  If you 
address a packet to 3.3.3.10, the routing table is consulted, and the 
best match is the network 0.0.0.0 with a mask of 0.0.0.0.  The packet is 
then addressed to 3.3.3.10 via the mac address of 1.1.1.254, which will 
(hopefully) deliver it on to it's destination.

To clarify, all TCP/IP routing is done via the route table, and there is 
only one route table per machine*.  There are not separate routing 
tables, or separate gateways, per interface**.

So why does Greg believe there are multiple routes per interface?  Well, 
the syntax of Debian's /etc/network/interfaces file is definitely 
misleading.  To boot, it probably does work in his environment, it just 
results in packets not quite flowing the way one would expect.  A 
hypothetical scenario:

In the diagram above, imagine a host (Frank) on the Intarweb (3.3.3.10) 
addresses a packet to Bob at 2.2.2.2.  The packet comes across NetB to 
Bob.  Bob gets the packet, and crafts a response.  Responses have no 
state, only a source and dest address.  The source address is 2.2.2.2, 
as Bob will respond from the same source the session was addressed to.  
Bob goes through the above routing procedure, matches the default 
gateway, and routes the packet through NetA.  Assuming NetA is a private 
network and there is a simple router involved, it will pass the packet 
along to the appropriate network and Frank will eventually receive it.  
The packet didn't travel the most intuitive path, but it did get there 
and life was happy.  This was the way TCP/IP was designed to work back 
in the day, and often does work in large networks.

The problems creep in when you start using things like residential-class 
networks and you throw in a dose of Internet paranoia.  For good 
reasons, any residential ISP isn't going to let you send traffic from 
their network, with an address they didn't give you.  So if you imagine 
1.1.1.1 and 2.2.2.2 are real Internet addresses from real ISPs, you can 
probably imagine why Time Warner won't let you send packets on their 
network claiming a source address of a Bellsouth DSL customer.  Thus, 
your packet is sent by Bob just fine, but dropped by your upstream ISP 
as a security precaution.

I have gone through the whole "how do I setup source-based policy 
routing under Linux" spiel on the list before.  It is possible to have 
multiple default gateways, for different types of traffic, as Joe Mack 
mentioned later in this thread.  The 3-second answer is "ip rules an ip 
route tables via the iproute2 package".  I'll save my and everyone 
else's time and just point to the message in the archives:
http://www.trilug.org/pipermail/trilug/Week-of-Mon-20040329/025177.html
http://www.trilug.org/pipermail/trilug/Week-of-Mon-20060619/042827.html

and a pretty good Debian-specific tutorial on the same idea:
http://www.debian-administration.org/articles/377

Aaron S. Joyner

* -- we'll expose this over-simplification before the end of the 
message, but play along for now if you know better.
** -- ip route tables aren't per interface, they're per 
type-of-traffic.  You still have to match on ip-based characteristics 
with the rules, they can't be interface-specific.  Keep reading to the 
end if you are confused, and read the links for more details.