[TriLUG] Is this block diagram fairly accurate?

[Aaron Joyner]

Please forgive my latency, I knew replying to this was going to consume
more than a bit of time, so put it off until this evening.


On Fri, Apr 5, 2013 at 10:57 AM, Steve Litt <slitt at troubleshooters.com>wrote:

> On Fri, 5 Apr 2013 10:09:40 -0400
> Aaron Joyner <aaron at joyner.ws> wrote:
>
> > It might seem helpful to show that the ARP cache is functionally very
> > similar to the routing table, just with different column data
> > (ip->mac, as opposed to ip/cidr->ip).
>
> Nice characterization! I hadn't noticed that until now.
>
> > If you can find the space to
> > illustrate a couple entries for each host,
>
> :-)  So much info, so little space. I already have so much stuff
> crammed into that diagram that it's risking font illegibility for those
> with bad vision or in presentation situations with a weak projector and
> lots of ambient light.
>
>
> > in particular that
> > computer 1&2 both have the gateway, and the gateway has both, but
> > that 1&2 don't have each other,
>
> Am I correct in assuming that the preceding sentence fragment is about
> routing -- that .1 and .2 route to .15, and .15 routes to each of them,
> but .1 and .2 don't route to each other because they are on the same
> subnet?
>

Nope, I'm specifically still talking about ARP caches.  Host1's MAC will
(generally) only appear in the router's ARP cache, not in Host2's ARP
cache.  Imagine the following hypothetical scenario:
1) You reboot everything (hosts, switches, router, etc), nothing has spoken
on the network yet.
2) Host1 attempts to address a packet to 68.68.0.1.  It consults it's
routing table, resolves the next hop as 192.168.100.15 and forwards the
packet to that address.  To do so, it consults the ARP table, and does not
find an entry.
3) Host1 sends an ARP Broadcast (destination ff:ff:ff:ff:ff:ff), saying
essentially "WHO HAS 192.168.100.15, tell 192.168.100.1".
4) The switch, seeing the ARP broadcast destination address of ff...
forwards the packet to all ports.  It adds an entry to it's forwarding
table (CAM table) for the source MAC of Host1 on the associated physical
port.
5) Host2 receives the packet, it does not respond, but depending on the OS
it *may* add an entry to its ARP cache (see
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt,
specifically arp_accept, it's typically off in Linux -- for more fun
reading, search for Gratuitous ARP).
6) Router/Gateway receives the packet, creates an entry in its own ARP
cache for Host1 based on the data in the incoming packet, then it crafts a
*unicast* reply, telling Host1 at 192.168.100.1 that it owns the IP address
192.168.100.15.
7) The switch receives the packet from Router/Gateway, consults it's
forwarding table, and delivers the unicast response only to the port of
Host1.  It also adds an entry in it's forwarding table (CAM table) for the
source MAC of Router/Gateway's interface.
8) Host1 receives the ARP reply from Router/Gateway, adds an entry to it's
ARP cache, and forwards the original packet that started this whole dance
to 192.168.100.15.

At this point, Host1 has an entry in it's ARP table for Router/Gateway, and
vice versa.  Host2 saw the ARP request packet from Host1, but probably
ignored it.  It never saw a packet from Router/Gateway.  The switch has
forwarding-table entries for both Host1 on Port 1, and Router/Gateway on
Port 3.  If you imagine Host2 also originating a packet to 68.68.0.1, it
will go through the same dance, and end up with the same
partially-populated ARP table, containing an entry for Router/Gateway, but
no entry for Host1.  This is the normal steady-state of a network where
Host1 and Host2 are talking to the outside world, but not each other (what
I'd call the "typical case" for most networks).



> > that would likely be helpful when
> > explaining the flow of arp discovery to someone.  Typing that also
> > makes me realize you didn't include an arp cache for the
> > Router/Gateway (on either of it's interfaces).  I'd recommend doing
> > that.
>
> Let me see if I can fit it in...
>
> >
> > I think you can achieve those changes by making each box a touch
> > bigger, and shifting the IP&MAC of each host to be displayed under
> > its name.  This would help prevent confusion with it being seen as
> > too-closely associated with the ARP cache, and give you room to fill
> > in example entries of the ARP cache.
> >
> > Labeling the arrows "MAC address" between the host and the Ethernet
> > indeed captures the relevant packet header for that layer of
> > transmission... but it's a bit at odds with the subnet notation on
> > the right ("192.168.100.0/24subnet").  In a perfect world, I'd think
> > of those arrows as a set of nested
> > boxes, something like [mac[ip[data]]], but I don't have any good
> > suggestions for representing that concisely around the arrows.
>
> Perhaps I could make the Ethernet lines blue on the outside (with
> subnet address) and red on the inside (labeled MAC address). Do I have
> this correct that packets between computers on the same subnet still
> have the source and destination IP address in the packet?
>

Yes, IP packets will have the source and destination IP in each packet
(gee, that sentence seems awfully redundant when I type it out :) ).

Something like the coloring you suggest might work, but I'd put the subnet
on the inside, and the MAC on the outside, personally.  It better
represents the way the encapsulation works.  Imagine that the application
originating the packet stuffs some data into it, the OS first wraps that
data in an IP frame (which adds IP/subnet info, etc), and then wraps that
IP frame in an Ethernet frame (which adds MAC src/dst).



>
> > Also,
> > I don't know that duplicating what ever is there (above and below the
> > switch) is strictly necessary?  If you have the space it probably
> > doesn't hurt. Still, it might be more clear to point out there that
> > all the Ethernet layer cares about is a single broadcast domain,
> > there's nothing technically requiring that to only be one IP subnet.
>
> Aaron, I don't understand anything in the preceding paragraph. I think
> the preceding paragraph is beyond my networking knowledge, but perhaps
> you're just using different words than the ones I'm used to.
>

Not all packets are IP packets, many other protocols could be flowing
across that Ethernet network that wouldn't necessarily be associated with
the IP subnet 192.168.100.0/24.  The simplest to visualize is probably a
parallel IP subnet.  Imagine if you added a second IP on a virtual
interface on both Host1 and Host2, eg. 10.0.0.1 and 10.0.0.2.  Or imagine
that they talked Appletalk, IPX/SPX, or some other random to each other.
 In general, the switch doesn't care about the Layer 3 protocol going
across the Ethernet network.

As was summarized elsewhere, I think it will behoove you to read briefly
about the OSI model.  Ethernet in my previous ramblings is strictly a
Layer1 and Layer2 construction.  IP addresses, subnets, etc are a Layer 3
construction.  If it's not clear after doing some background reading, let
me know and I can try to find a good primer to point you at, or summarize
at a bit more length here.


> As far as duplicating above and below the switch, what I was trying to
> do is show that 100.1, 100.2, and 100.15 are on the same subnet
> connected to the same switch. Is there a better way to do that?
>

Yeah, I see what you're going for, and it does achieve that, but I'd
suggest that there's a deeper understanding that can be portrayed in your
graph.


> I think this paragraph comes down to this question, "Will you have
> > exposition to go with this graph, or do you expect it to stand
> > alone?"
>
> After this and other evaluations, it's looking more and more like it
> will need further exposition.
>
> > It'll work fine as a thing to point at, or with some text
> > elaborating on the data flows, but if it's going to stand alone it
> > might benefit with a bit more thinking.
>
> I'd love to figure out a way to improve the diagram to the point where
> parallel explanation is unnecessary.
>
> Thanks very, very much for all this great advice.
>

I'm happy to help.  In particular, I think this is something that anyone
reading this is likely to benefit from thinking through, so it's a great
topic to ask about on the list.  I was also greatly heartened to see that
others chimed in (both on-list and off).  I'm certainly not a final
authority on all things networking, and hopefully some of the other
networking geeks lurking will chime in with explanatory flair or at least
their favorite resources to point you at.

Aaron S. Joyner