[TriLUG] why is it slow?

[Aaron S. Joyner]

Rick DeNatale wrote:

> On 9/13/06, Aaron S. Joyner <aaron at joyner.ws> wrote:
>
>> So this post isn't entirely useless - my gut instinct is that the
>> problem is related to the "u-turn" problem as described, but I'm at a
>> loss to explain precisely the internals of why.  Assuming the NAT
>> implementation is anything close to *sane* on the embedded router, this
>> really shouldn't be a problem.  Then again, don't trust the Chinese or
>> Korean guy who wrote the firmware to have done a sensible job on his
>> first programming project.
>
>
> I've had appliance routers (SMS I think) which refused to recognize
> "U-turn" addressing.
>
> Unlike Brian, there are routers that do this, the Netgear I'm using
> now has no problem with it.
>
> Just out of curiosity how would you set up NAT routing in Linux to do
> this, with port forwarding?  For example say:
>
> wan-interface gets it's ip address from isp via dhcp
> lan-interface 192.168.1.1
> lan devices use 192.168.1.1 as gateway.
>
> for requests coming from either outside or inside:
>     http connections to wan ip address get forwarded to 192.168.1.2
>     wan ssh connections to wan ip address get forwarded to 192.168.1.3
>
Okay, so I started writing this like 30 mins ago, and discovered along 
the way that this doesn't work for this case easily at all.  In fact, I 
tried to cook up a way with just iptables, and failed miserably.  Allow 
me to summarize what I was originally thinking, and then explain why 
this case is messy, and provide a completely wacky solution or two.

In my mind, when you have a simple home network, and a Linux box, you 
run the services on the gateway.  I don't typically have additional 
servers behind my gateway which I push ports through to, via some 
mechanism ala DNAT.  If you do run services on the inside, and DNAT into 
them, you don't loop through the router for numerous reasons, including 
a) inefficient use of network links, b) weird problems like we're about 
to discuss, c) it just feels wrong.  :)  The typical way to handle this 
problem is at the name space level.  If you're addressing service Foo as 
foo.joyner.ws, then the external DNS view will return the external IP 
address, and the internal DNS view will return the internal IP address.  
In the days before views, you either ran two DNS servers with different 
zone files, or you used two different domain names (foo.int.joyner.ws, 
for example - where int is short for internal).  This way, you use the 
names, life is happy, and traffic flows logically.  If you need to use 
IPs to address the service, you better understand what IP to use from 
where.  :)

Anyway, on with why this is messy in Linux.  If you originate traffic 
internally from 192.168.1.5, and address your traffic to the external IP 
address, and have the appropriate DNAT rule, your packet gets passed 
back inside to 192.168.1.2 just fine.  The problem is that you can only 
change the destination address, or the source address.  You can do 
either one with the DNAT or SNAT targets in iptables, respectively.  
Unfortunately, both of these targets terminate rule processing and 
immediately deliver your packet on it's merry way, out the interface.  
Choosing the DNAT rule is the logical choice, as that will at least get 
your packet to where it needs to go.  The problem is, the source address 
is still 192.168.1.5, so when 192.168.1.2 writes the response packet, it 
delivers it directly to 192.168.1.5.  It does not send it back through 
the gateway (as you would sort of hope it would).  Consequentially, when 
that packet arrives, 192.168.1.5 doesn't recognize it as part of the 
stream it originated to the external IP address (naturally), so it drops 
it on the floor, and you get no where.  I googled, I looked through 
Usenet archives, I searched for quite some time for a way to change both 
the source and destination address of a packet with iptables.  Some 
people seem to be suggesting that you can just use an additional SNAT to 
fix the problem (and believe me, it seemed logical before reading the 
iptables man page, and I did try - oh did I try), but my testing proves 
out that this simply does not work.  Once the packet matches the DNAT 
rule, you get no more opportunity to match any appropriate SNAT rules, 
and vice versa.  If someone else out there knows of a way to make this 
work via iptables alone, I'd be mighty interested in it, because I can't 
come up with a way, and it sure feels like there should be one.

On to the truly wacky ways to actually make this work.  So, a few things 
came to mind immediately, all of which are horribly ugly, and if you 
implement them... well... I'm not to blame.  First off, make sure your 
DNAT rule is sufficiently specific that it doesn't capture traffic from 
the internal network (a simple -s ! 192.168.1.0/24 will suffice).  Then, 
start tacking on rules like -j REDIRECT -port 8080, to capture that port 
80 traffic and redirect it to a local port 8080.  Then, run a 
port-redirector on port 8080 which will capture the traffic, and 
originate new traffic on to the destination box.  This will get the 
right source IP address into the packet, and cause 192.168.1.2 to 
respond to the gateway, which can then pass the packets back through the 
port redirector, back to the REDIRECT session / rule, and then 
eventually back to 192.168.1.5.  Don't blame me, I said it was ugly.  I 
also promised to provide more than one ugly solution.

The other way to skin this cat, is to do weird things on the backend 
servers.  You could do something akin to port based policy routing, such 
that you add an iproute2 rule to match traffic originating from port 80 
on the web server, traveling to any host on the local network, and shunt 
it to a different route table, which only has an explicit nexthop set 
for all hosts on the network to the default gateway.  The amusing thing 
here is that you can't match packets with just iproute2 based on port, 
you only get src and dst ip address.  You can match on fwmark, from 
iptables, though.  So you mark the packets with iptables via the -j MARK 
target, then apply the route to them with the `ip rule` command.  This 
solution may or may not actually work, depending on how the gateway 
handles that incoming traffic destined for another host on the same 
network.  It's late, and I don't want to setup the above absurd solution 
to see it if will work or not.  :)  If anyone's feeling bored, let me 
know -- I think it'll work, but I'm not quite sure how you'd have to 
specify the route(s) in the alt routing table, ie. can you do a simple 
/24 with the nexthop set to the gateway, or do you need individual /32s 
for everything but the machine itself and the gateway, or perhaps some 
odd mix to make it easier to specify, with the same effect?

Let it be said that I'm a died in the wool Linux fan, and I don't think 
many people on the list will question that.  But, I was thinking while 
writing that last paragraph, that this is dramatically easier to do on 
an honest-to-god router.  Not just part of it, but all of it.  Granted, 
you have to think like a network guy, and the terminology is almost all 
different, conceptually half of it is different, and the configuration 
of things is 100% different, but once you beat the learning curve, man 
is it easier.  On the flip side, all the things that seem easy and you 
take for granted in the Linux world, become their own challenge in that 
world.  *chuckle*  I'll stick with the things I know inside and out, and 
jump through hurdles like above as required every once in a while.  In 
this world, at least I know how to overcome the bizarre edge cases 
better.  :)  Yes Ryan, this paragraph was for you.  I hope you made it 
this far in.

It's late, I'm tired, enough typing.
Aaron S. Joyner