[TriLUG] Help routing between different devices

[Ed Blackman via TriLUG]

I need some help setting up routing between different network devices, 
but it's been too long since I've worked on this stuff, and google 
searches don't turn up helpful resources.

I'm trying to create a network simulated in VMs that are isolated from 
each other (see Background below for details), but am failing to get the 
first VM to be able to access the internet.

In the uplink VM, I can ping 10.0.0.2 (itself), 10.0.0.1 (brup0 on the 
host), 192.168.2.214 (br0 on the host), and 192.168.11.1 (brrt0 on the 
host), but can't ping or otherwise connect to 192.168.2.1 (the real 
router for the lan) or any other LAN or internet host.

In the host I *can* ping the VM (10.0.0.2)!

Any help or pointers would be appreciated, and I'm happy to answer any 
questions about the existing configuration.

ip addr and ip route on KVM host:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 18:03:73:30:f3:6e brd ff:ff:ff:ff:ff:ff
6: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether fe:54:00:9e:18:af brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe9e:18af/64 scope link 
       valid_lft forever preferred_lft forever
7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 18:03:73:30:f3:6e brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.214/24 brd 192.168.2.255 scope global dynamic br0
       valid_lft 43173sec preferred_lft 43173sec
    inet6 fd52:bd32:dccb:0:1a03:73ff:fe30:f36e/64 scope global dynamic mngtmpaddr 
       valid_lft 7173sec preferred_lft 1773sec
    inet6 fe80::1a03:73ff:fe30:f36e/64 scope link 
       valid_lft forever preferred_lft forever
8: brrt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 0a:ff:fe:7a:b8:5e brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.1/24 brd 192.168.11.255 scope global brrt0
       valid_lft forever preferred_lft forever
    inet6 fe80::8ff:feff:fe7a:b85e/64 scope link 
       valid_lft forever preferred_lft forever
9: brup0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 86:ef:eb:cd:d5:56 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/8 brd 10.0.0.255 scope global brup0
       valid_lft forever preferred_lft forever
    inet6 fe80::84ef:ebff:fecd:d556/64 scope link 
       valid_lft forever preferred_lft forever

default via 192.168.2.1 dev br0
10.0.0.0/8 dev brup0 proto kernel scope link src 10.0.0.1
192.168.2.0/24 dev br0 proto kernel scope link src 192.168.2.214
192.168.11.0/24 dev brrt0 proto kernel scope link src 192.168.11.1

enp0s25 is the physical NIC in the host.

br0 is the bridge for sharing the NIC for general VMs that I want to 
have unrestricted access to the internet.  These work fine.

brup0 is the bridge for the "uplink" VM.
vnet0 is the network set up by KVM for the uplink VM.
brrt0 is the (currently unused) bridge for the router VM.

ip addr and ip route in the uplink VM:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:9e:18:af brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/8 brd 10.0.0.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe9e:18af/64 scope link 
       valid_lft forever preferred_lft forever

default via 10.0.0.1 dev ens3 onlink
10.0.0.0/24 dev ens3 proto kernel scope link src 10.0.0.2

/proc/sys/net/ipv4/ip_forward is 1 on the host and in the VM.

Background:
I've been wanting to replace my OpenWRT router with one of the multi-NIC 
mini PCs that you can get these days.  I have the system, but the 
networking configuration is hairy, and I could only work on it when I 
could take the wireless down at my house, which these days is "never".   
So I thought I'd create a bunch of VMs and simulate the configuration so 
I can work on it without disturbing anyone until it's ready to be copied 
in to physical hardware.  This VM is the simulated "uplink", that the 
simulated router will get an IP from.  Eventually I will add the 
simulated router, a simulated "guest system", and a simulated "lan 
system", then work on DNS, firewall rules, etc.

-- 
Ed Blackman