18:39:19 <raub> #startmeeting
18:39:19 <Tribot> Meeting started Mon Mar  8 18:39:19 2021 UTC.  The chair is raub. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:39:19 <Tribot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:39:19 <noway2> I have "pilot2" up and running Ubuntu 20.04 LTS server edition and it is accessible via IPv6 at least.
18:39:35 <raub> #chair bdmc noway2
18:39:35 <Tribot> Current chairs: bdmc noway2 raub
18:39:37 <bdmc> Great.  DNS?
18:40:12 <raub> part of pilot upgrade?
18:40:16 <noway2> Not yet.... that I believe is handled by Pilot... where I would need to update the zone file and try it.
18:40:29 <raub> #topic 2. Current Topics
18:40:36 <noway2> Yeah, we are out of sync since meeting started ... lets get back to this later.
18:40:43 <bdmc> Exactly.   Shouldn't be difficult.
18:40:57 <raub> noway2: not anymore
18:41:03 <raub> Let's just go to pilot
18:41:19 <bdmc> noway2: Oh, don't you want to start at the end and work backwards?
18:41:31 <raub> I was looking at the 2011 pilot upgrade plan and it lists named (https://steering.trilug.org/wiki/index.php/PilotUpgradePlan#Configuration_and_Services)
18:41:43 <noway2> why not, I like eating dessert first :P
18:42:07 <raub> noway2: one of the benefits of becoming an adult
18:42:25 <bdmc> raub: As far as I remember, that is installed in Pilot, working.
18:44:04 <raub> Can we make https://steering.trilug.org/wiki/index.php/PilotUpgradePlan current?
18:44:12 <bdmc> Without actually breaking down and doing any research, here is what I remember running on Pilot:  ( here for corrections and additions )
18:45:01 <bdmc> HTTP(S), SSH, LDAP, NAMED, ( NTP? ), SMTP, IMAP(S), Home Dirs,
18:45:53 <noway2> Saturday you sent the message suggesting we consider breaking the /home (for users) off on a different machine.
18:45:56 <bdmc> raub: That sounds like a plan.  Any thoughts on bringing in any of our other documentation?
18:46:15 <bdmc> noway2: That has always been the plan.
18:46:51 <raub> noway2: IMHO we either move the services out or the home accounts
18:46:59 <raub> (first, that is)
18:47:19 <bdmc> I see perhaps three ( or four? ) machines.  One to do critical services in general, one for /home, one for mail, one for web.
18:49:18 <bdmc> Perhaps combinations.
18:49:57 <noway2> Mail and Web pretty much (have to) go together. The Let's encrypt makes use of some of the https pieces parts as you need to shut apache down to update the cert and then both the mail and web use the same cert files.
18:50:55 <raub> You can create on one box and them move cert to another
18:50:56 <bdmc> Good point.
18:51:19 <bdmc> noway2: yours, I meant.
18:51:38 <raub> I never make good points ;)
18:51:44 <bdmc> I think that they could be combined.  That's why I said three machines.
18:52:10 <bdmc> raub: Now, now.  You sometimes do.
18:52:17 <noway2> I imagine the LUG web traffic is pretty light wieght. It's mostly static pages. The mail server is the heavy lifter.
18:52:22 <raub> Don;t scare me like that
18:52:32 <raub> noway2: agreed
18:52:43 <bdmc> noway2:  Exactly.  Especially since the change to whatever its name is.
18:53:05 <bdmc> The page compiler.
18:54:39 <bdmc> One thought for Pilot2 --- well, a couple of thoughts.   Start out by building the Web machine.  Then add Mail, then move on to the next machine.
18:55:04 <bdmc> Unless you would like to start with critical services first. ( LDAP, NAMED, etc. )
18:55:10 <raub> Sounds like a plan. Web will be the quickest
18:55:44 <raub> Since it seems ldap is only used inside moya/pilot, I think that can wait
18:55:57 <noway2> Yes, priority is getting it running web first. If we can get it set with a static IP and then visible via DNS we're halfway home. Apache is pretty easy to setup and the virtual hosts can be largely copied over.
18:56:12 <bdmc> It is used to manage Home directories, but that is later, too.
18:56:32 <bdmc> noway2: Exactly.
18:56:41 <raub> We can let users know it will be down for a while
18:57:11 <bdmc> Shouldn't be long, once the new one has been built.  Just a case of switching DNS at that point.
18:57:47 <raub> bdmc: I mean user stuff in homedirs
18:58:11 <bdmc> raub: Oh, yes, but that is some time in the future.
18:58:24 <noway2> No, it shouldn't once I get it on a static IP and updated DNS.   The mail will take a little bit of doing but fortunately most of it is in main.cf which hasn't really changed.
18:58:26 <bdmc> I thought that we were talking about the web machine.
18:58:35 <raub> Hence letting people know
18:59:11 <raub> noway2: my suggestion for mail is to move the mails to mail server
18:59:31 <noway2> It almost sounds like we could use a front machine that directs to a backend machine based upon incoming port - service.
18:59:34 <bdmc> raub: Have you done your measurements, yet?
19:00:02 <raub> Yep. I can post them to the pilot upgrade plan page
19:00:16 <bdmc> noway2: Almost Moya, but if each service has its own IP address, ....
19:00:50 <bdmc> Incidentally, has any thought been given to the new names?
19:01:05 <noway2> That way, everyone can still point to "trilug.org" and behind the scene the port-address forwarding sends it to pilot, pilot2, etc or whatever the name is and it's transparent to the user.
19:01:17 <raub> Not really. I have no issue keeping the Farscape theme
19:01:42 <bdmc> raub: How much space does "all mail" require, today?
19:02:04 <raub> Let me run the check again
19:02:37 <bdmc> noway2: As far as I know ---- hmmm.  I guess that people who are using Home directories are addressing "pilog.trilug.org," today.
19:03:30 <bdmc> Of course, to me Pilot is more appropriate for Critical Services.
19:04:32 <bdmc> Web is at trilug.org, mail goes to mail.trilug.org.
19:05:15 <raub> External vs internal names
19:05:24 <raub> And, people are used to "pilot"
19:05:33 <raub> I would not change that
19:05:51 <bdmc> www.trilug.org goes to pilot.trilug.org
19:06:18 <raub> The www thingie I am not worried about. The ssh part, yes
19:06:18 <bdmc> Need a bit of re-organization of DNS, but we knew that already.
19:06:41 <bdmc> Yes, generic SSH goes to the "Home
19:06:54 <raub> Yes. Also, are we assuming we will get more than one external IP? I thought we just had one and then port forward the rest
19:06:59 <bdmc> " machine.  Each of the machines needs to respond to SSH, too.
19:07:41 <raub> bdmc: not necessarily. We could have one we can login and then access the others inside the local net
19:07:45 <bdmc> raub: You didn't read my message from Mark, did you?   We have several IPv4 addresses ( 5? ), and a whole block of IPv6.
19:08:19 <noway2> That brings up another question.... it looks like Michael was correct in that we've got the HE tunnel block in use... but nothing is responding to it.
19:08:36 <raub> Nope, I did not see that reply. I still do not know what is the point of getting one IP just to run one single service on it
19:08:52 <bdmc> Maybe so, but why not use what the data center is already providing.
19:09:05 <noway2> When I got the new machine up today with networking, DHCP ipv6, I was able to SSH into via ipv6. It was the He block, not the one the bdmc mentioned.
19:09:43 <bdmc> Because that is what the DNS is pointing to.
19:09:56 <noway2> web traffic is handled by Moya and funneled via the haproxy program, so it can be redirected easily.
19:10:13 <noway2> @bdmc do we need to do an experiment and try pointing to the other block?
19:10:56 <bdmc> I would think so.  It should be ready to go with statically-assigned addresses, but that is what DNS needs to advertise.
19:11:39 <noway2> question, where is the DHCP server... doesn't look like DHCP3 is runing on Moya.
19:11:55 <bdmc> NAMED is on Pilot.
19:12:11 <bdmc> More properly, BIND.
19:12:47 <bdmc> I notice that the DNS refers to "pilotvm.trilug.org."  Interesting.
19:13:12 <bdmc> But doesn't respond to that name.
19:14:26 <raub> noway2: kvm has its own dhcp
19:14:53 <bdmc> /etc/named.conf, if I remember correct, should show you where the tables are, probably /var/lib/named.
19:15:08 <bdmc> ( correct = correctly )
19:15:20 <raub> Yep
19:16:59 <noway2> @raub, speaking of which one of the missing pieces of information on getting the network up, was that I needed a NIC interface in the VM configuration. I thought I copied the XML from Pilot and it looked like it, but something didn't go quite right. Once I instantiated it in the vm-manager gui AND re-installed ubuntu it connected. Apparently trying to add it after the install causes Ubu to leave key pieces out.
19:17:35 <bdmc> noway2: sounds normal for Ubuntu.   B-)
19:19:14 <raub> noway2: I usually create VMs from command line, and specify the network it is to use
19:20:32 <noway2> I was running into that there was no interface, even though it declared the bridge to br-guest..... Something was off.. hidden characters or something.
19:20:50 <raub> Possibly. And, moya also has old OS
19:21:18 <bdmc> REALLY????   B-)
19:22:00 <raub> As bdmc can attest, moya has its on boinkness but it is less than pilot's right now
19:22:04 <bdmc> According to the configuration on Pilot, both IPv4 and IPv6 are expected to be configured via DHCP.
19:22:39 <bdmc> Static Addresses ( for either ) are dead easy.
19:23:18 <noway2> Static allows you to map the services like the haproxy (web).... otherwise it could change.
19:24:49 <raub> Mail is taking about 23MB (24889708 bytes) of disk space
19:25:24 <raub> unless my math is off and I have 24889708 KB ;)
19:26:01 <bdmc> Details!   B-)
19:26:29 <bdmc> Are you actually counting bytes and not something like Ks?
19:26:40 <bdmc> Or even Megs?
19:26:54 <raub> bytes because it would be easier math for script
19:27:06 <bdmc> OK.  So almost nothing.
19:27:17 <raub> The default du output is bytes, right?
19:27:47 <bdmc> I use "du -sh" which shows the units.   Let me look.
19:28:03 <raub> The h is too helpful here.
19:28:47 <bdmc> I just did "du *" and "du -sh *" and I got "2020" for the first and "2.0M" for the second.
19:29:32 <bdmc> So, to me, that would say that the default unit is KB.
19:29:47 <raub> Well then
19:30:01 <raub> It is still not horrible
19:30:05 <bdmc> That's a LEEETLE bigger.
19:30:18 <raub> As you said, details. Trifle details
19:30:24 <bdmc> Only 23 GB!
19:31:08 <noway2> Thats a big twinky.
19:31:40 <raub> I also have enough data to find what is the max individual maildir size
19:33:37 <bdmc> All of the BIND data is in /etc/bind, nothing in /var.
19:33:50 <bdmc> /etc/bind/trilug.org is very interesting.
19:34:32 <raub> Worth putting on the wiki or too sensitive?
19:35:05 <noway2> Aside from the google verify part, most of it looks like it's old partition scheme that doesn't exist
19:35:20 <raub> lovely
19:35:49 <noway2> It refers to various servers like zhaan, dargo, etc which I see pieces parts of and there are then others such as marconi and talon.
19:37:00 <noway2> The spf record points to pilotvm.trilug.org which is incorrect - part of the seen as spam problem?
19:37:22 <raub> Probably
19:39:52 <bdmc> Yes, that's what I saw, too.
19:42:01 <bdmc> It might be worth backing up that zone file and correcting a couple of things right now.  I tried a couple of outbound IPv6 connections, and then traceroute6's, and nothing.
19:42:59 <raub> Agreed
19:44:09 <bdmc> noway2: You wanted to assign an IPv6 address to Pilot2, for instance.  Perhaps assign an IPv6 address to Pilot, as well.
19:44:16 <noway2> "I tried a couple of outbound IPv6 connections, and then traceroute6's, and nothing." elaborate please. Are you saying outbound isn't working on pilot
19:44:42 <bdmc> Not using the HE Tunnel.  It seems to get a ways along, but then stalls.
19:44:45 <noway2> I do have Ipv6 (inbound) working on Pilot 2.  It got it via DHCP... now one thing that was interesting the prefix lenght I got was 128 not 64 like the others are configured.
19:45:11 <noway2> It is using the HE addressing though
19:46:24 <bdmc> What address ( of the "local" block ) do you want to assign to Pilot and Pilot2?  You could, just to confuse things, assign two different addresses to each of them, with two different interfaces.
19:48:29 <noway2> Right now dhcp gave 192.168.122.226 ... which I don't think goes anywhere... except maybe out the ipv4 gateway.  and the v6 uses fe80::5054:ff:fe85:a9a4
19:49:12 <noway2> I could asign an alias ethernet (ens7:0) and give it one of the other IPv6 addresses ... need to be careful to not assing multiple gateways though.
19:49:14 <bdmc> Yes, that is a "local" IPv4 address which can not go anywhere without help.
19:49:47 <bdmc> That is also a local IPv6 address, not the global that you need for external access.
19:50:08 <bdmc> You can add a global IPv6 address to the same interface.
19:50:09 <noway2> I thought you were asking about the local addresses..... ??? .... Yes, this does get confusing, but I think I would need / want to assing one statically and then  Moya would have to NAT it... this is done as a combo of IPtables pre-post roting and haproxy.
19:50:57 <bdmc> Right.  The IPv6 global addresses would have nothing to do with Moya, and certainly nothing to do with NAT.
19:52:06 <bdmc> Just at the moment, I'm not particularly interested in IPv4, for exactly those reasons.
19:52:54 <noway2> Agreed. I think Im starting to get a handle on the moving pieces. The local IPv4 addresses get NAT through MOYA and the bridge interface.  It looks liek the HE electric zone does too - because 2001:470:8:11ec::1 is in the br--guest bridge.
19:53:05 <bdmc> Reading back a bit, how are you getting an external address on Pilot2?
19:53:56 <noway2> DHCP - ipv6.  It gave me one of the 201:470:8 block... that is part of br-guest.
19:53:58 <bdmc> OK, maybe that last statement means something.  There has to be some connection between the fe80 address and that 2001:: address.
19:54:35 <bdmc> OK.  So you do have a second IPv6 address on that Pilot2 interface?
19:55:22 <noway2> my understanding is that fe80 is like 127.0.0.1 and 2001: is public.  No second ipv6 yet.  I was thinking of trying to instantiate one (without a gateway) and try the block you mentioned in the email to see if it can then ping respond.
19:56:23 <bdmc> Basically.  No, you have two addresses on the interface.  One, most like the MAC address, is the fe80, and the "real" address is the 2001:: one.
19:56:52 <bdmc> If you do "ip a" on that machine, you will see two IPv6 addresses on that interface.
19:57:56 <bdmc> Going way back, so you are saying that if you do "ssh -6 2001:::whatever" you get connected to Pilot2, correct?
19:58:41 <noway2> Yes on the SSH.  Yes I get two addresses, one global scope, one link scope.
19:59:58 <bdmc> OK, that was what I was asking, sort of, most recently.  We COULD add an interface to that machine and assign a static IPv6 address from our "host" block.  Shall we?
20:01:04 <bdmc> We could also do the same on Pilot.  ALTERNATIVELY, we could just change both machines from "dhcp" to "static," and assign new addresses.
20:01:18 <bdmc> ( to the existing interface )
20:01:40 <noway2> The new machine can be played with and not disrupt anything.  I can add an alias interface (another IP add) to the interface and we can see if it responds.
20:02:00 <bdmc> Sounds like a plan.
20:02:39 <noway2> The usable range from your email is 2607:fc50:4000:87::2 and up. ::3 should work.
20:03:19 <noway2> ok :1 (gateway) responds to ping6.  :2 and :3 do not.
20:05:08 <bdmc> OK.  Let's assign 3 to Pilot2.
20:06:08 <noway2> Updating the ethernet config now.
20:08:15 <bdmc> "rygel.trilug.org" will direct you to the IPv6 address.  The IPv4 for it goes who knows where.  I'll fix that.
20:12:26 <noway2> Ok... having some trouble getting the yaml file to accept the syntax ... working on it.
20:12:45 <bdmc> Perhaps square brackets around the address?
20:13:30 <bdmc> I have the DNS fixed.  I think that the odd IPv4 address was left over from the before time.
20:21:22 <noway2> Ok, let's see what happens.  I got the netplan to accept the plan. I did a reboot and we will see if the alias interface comes up.
20:29:08 <noway2> No link..... Trying some more fiddling...
20:32:38 <bdmc> raub: Have you gone back to work?
20:33:41 <bdmc> Guess so.
20:34:26 <noway2> I am here...I think I have it addressed, but it won't respond on ping.
20:34:51 <noway2> doing a reboot to possibly clean things up.... try again in a minute
20:35:07 <raub> bdmc: I had to deal with some stuff
20:39:21 <bdmc> raub: No problem.  We are just playing.
20:39:46 <raub> Shall we close the meeting or do you want to keep it going for recordkeeping purpose?
20:39:57 <noway2> For some odd reason, it's listening on both ipv6 interfaces and I don't know how it's getting the HE one. It's not in the config
20:45:07 <noway2> OK, it will respond to ping6 for it's own address, but it can't hit the gateway.  This makes it sound like the addr range of 2607:fc50:4000:87::2 is NOT accessible via our switch configuration.
20:46:19 <bdmc> Are we ready for me to contact Tech Support, and ask questions?
20:46:52 <bdmc> Incidentally, is that interface still marked "dhcp?"
20:47:09 <bdmc> That would explain where the address is coming from.
20:47:55 <bdmc> Also, do you have the gateway address configured for that interface, the new one?
20:49:22 <noway2> Who owns Moya and what (ethernet) switch is it connected to.. Is this range 'tagged' in the switch port to be in this vlan? I know I could get to it from the 2001 address but not the 2607. traceroute (inward) dies at  2607:fc50:4000:59::2 which DOES not come up on a traceroute to the gateway. the last hop, which this traceroute goes to thats common is 4000:1::4....
20:50:25 <bdmc> OK.  Let me ask.  In the mean time, I suggest that we are done for this meeting.
20:50:52 <noway2> I agree. 2.5 hours....
20:51:06 <bdmc> Doing an inbound traceroute, my last address is: 2607:fc50:4000:59::2
20:52:40 <bdmc> Also incidentally, I tried "ssh  bdmc@192.168.122.226"  from Pilot and got "no route to host."
20:54:13 <bdmc> noway2: What does "ip neigh" show from Pilot2?
20:54:37 <raub> Shall I close the meeting?
20:54:45 <noway2> ip neigh?
20:54:59 <bdmc> Hold a moment until Matt gets these answers.
20:55:00 <noway2> 192.168.122.1 dev ens7 lladdr fe:54:00:2c:52:24 STALE
20:55:01 <noway2> fe80::280a:3bff:feff:6ef2 dev ens7 lladdr fe:54:00:2c:52:24 router REACHABLE
20:55:06 <raub> K
20:55:25 <bdmc> Interesting, just those two?
20:55:29 <bdmc> OK, "ip a"
20:55:59 <noway2> Just verified I CAN ssh in via noway2@2001:470:8:11ec::1:8892
20:56:46 <noway2> ip a
20:56:47 <noway2> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
20:56:49 <noway2> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
20:56:50 <noway2> inet 127.0.0.1/8 scope host lo
20:56:51 <noway2> valid_lft forever preferred_lft forever
20:56:53 <noway2> inet6 ::1/128 scope host
20:56:54 <noway2> valid_lft forever preferred_lft forever
20:56:56 <noway2> 2: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
20:56:57 <noway2> link/ether 52:54:00:85:a9:a4 brd ff:ff:ff:ff:ff:ff
20:56:59 <noway2> inet 192.168.122.226/24 brd 192.168.122.255 scope global dynamic ens7
20:57:00 <noway2> valid_lft 3356sec preferred_lft 3356sec
20:57:02 <noway2> inet6 2001:470:8:11ec::1:8892/128 scope global dynamic noprefixroute
20:57:03 <noway2> valid_lft 3359sec preferred_lft 3359sec
20:57:05 <noway2> inet6 fe80::5054:ff:fe85:a9a4/64 scope link
20:57:06 <noway2> valid_lft forever preferred_lft forever
20:57:08 <noway2> I turned off the other address cause I thought we were done for now.
20:58:36 <bdmc> Oh.  It might be a bit hard to diagnose things with it shut off.
20:59:50 <bdmc> noway2: I presume that you are the only user account on that machine?
21:03:11 <noway2> yes, at the moment.  I can easilly add users.
21:03:45 <bdmc> I would appreciate your adding me, in case NetActuate needs me to look at anything.
21:04:00 <noway2> Ill play with the ipv6.. One thing that was weird is that it was listening on BOTH v6 ranges, though only one was configured. I am not sure where it was getting it.
21:04:46 <noway2> Sure I will add you.. Can you get in via the KVM virt-manager?
21:05:08 <bdmc> Would you also try "ip -6 r"?
21:06:01 <noway2> ::1 dev lo proto kernel metric 256 pref medium
21:06:03 <noway2> 2001:470:8:11ec::/64 dev ens7 proto ra metric 100 expires 3352sec pref medium
21:06:04 <noway2> fe80::/64 dev ens7 proto kernel metric 256 pref medium
21:06:05 <bdmc> Hmmm.  Perhaps.  Don't know.  Haven't tried it, just SSH.
21:06:06 <noway2> default via fe80::280a:3bff:feff:6ef2 dev ens7 proto ra metric 100 expires 1552sec mtu 1500 pref medium
21:06:19 <noway2> ok. hold on ...
21:06:42 <noway2> it will be passwd for right now, so I'll create it and then go ahead and change it.  THe likelihood of getting cracked is very low, but still
21:07:44 <bdmc> From Pilot: root@pilot:/etc/bind# ip -6 r
21:07:44 <bdmc> 2001:470:8:11ec::2 dev eth0  proto kernel  metric 256
21:07:44 <bdmc> 2001:470:8:11ec::/64 dev eth0  proto kernel  metric 256  expires 3548sec
21:07:47 <bdmc> fe80::/64 dev eth0  proto kernel  metric 256
21:07:50 <bdmc> default via fe80::280a:3bff:feff:6ef2 dev eth0  proto ra  metric 1024  expires 1748sec hoplimit 64
21:08:43 <noway2> ok, user bdmc.... shall I post the password or email it to you?
21:08:47 <bdmc> Looks like I jumped the gun.
21:08:57 <noway2> jumped the gun?
21:09:11 <bdmc> Oh, I thought that you told me the password, a couple of minutes ago.
21:09:43 <bdmc> "it will be passwd for right now,"
21:10:27 <noway2> passwd it is
21:10:43 <noway2> user bdmc
21:11:07 <bdmc> Could not chdir to home directory /home/bdmc: No such file or directory
21:11:49 <noway2> ok... I thought it added it...
21:13:04 <noway2> Ok. directory in ... adding you to the group and I'll add to the sudo'ers file too. May take a minute
21:13:31 <bdmc> Success.  Thank you.
21:14:33 <bdmc> OK.  I'll quit bothering you.  Talk to you both later in the week.
21:15:25 <noway2> You're good.  Been kind of slow here.  You should have sudo access.
21:16:06 <bdmc> I do, thank you.  I notice that /etc/network/interfaces is empty.
21:16:25 <bdmc> Where is it getting the configuration from?
21:16:32 <noway2> Yeah, the "new" way is to use netplan.  I don't think I like it... Might have to see if we can revert to the old way.
21:16:48 <noway2> netplan creates a yaml file, but I think it lacks some of the fine tuning that the interfaces file allows.
21:17:10 <bdmc> Oh, joy?
21:17:37 <noway2> Yeah, I think that's where  / why it was still keeping both Ipv6 addresses when only one was configured. It compiles the yaml into ???
21:17:53 <bdmc> Just like the "new" LetsEncrypt only uses "snap."
21:18:13 <noway2> great ....
21:19:21 <bdmc> Well, as I said half an hour ago, let's go our wepret says.
21:19:52 <noway2> K... catch you later.
21:20:16 <bdmc> #endmeeting