[TriLUG] The sad state of sysadmin in the age of containers

[Igor Partola via TriLUG]

I agree with some of your points, disagree with others. First, containers,
cheap VM's, cheap hosting in general made self-hosting applications much
more available to the masses. Back in the day if you wanted to host an
application, you had to find a large server, and a sysadmin willing to
carve out a chunk of that server for you to use. You had to play nice
because servers were multi-purpose. They ran databases, web servers,
directory services, and many other applications. It took a person to be in
charge of it and know everything that's going on to keep it running.

Naturally, the downside of this approach is that a single misbehaving
application, or a single bad update can crash dozens, or hundreds of
applications. It also meant that you had to a hire a separate person, a
sysadmin, to do all this work.

Nowadays, a VM or a container costs almost nothing (as in no overhead), so
you can build single-purpose servers. Want a database server, a web server,
a queue server, a task server, and a odd utilities server? No problem. It
costs the same as one big server to run all these tasks on at once. As a
sysadmin, you no longer have to worry if Joe the developer has an
application that's misbehaving. You give Joe his own VM and it's up to him
to keep the VM running. This was the first great advance.

Next, you have configuration management. Puppet/Ansible/Chef/etc. have huge
benefits of a sysadmin performing steps manually. If you are a sysadmin,
and your boss asks you to configure 1000 servers identically, except things
like networking, hostnames, etc. would you do it manually? Hopefully not.
Even if you did, then your boss asks you to add more configuration to each:
a new security policy, a dozen new libraries and utilities, etc. Would you
still do that by hand? Of course not. You can write a small shell script
that you run over ssh. But why not share that with your fellow sysadmins?
Puppet/Ansible/Chef/etc. give you the tools to do this. This is the second
great advancement. Your job as a sysadmin is infinitely easier with these
than ssh-ing to the box and fiddling with it by hand, with no record of
what you did and no repeatable process.

However, you touch on an important point: security and sane built
processes. As the two great advances above allowed any average Joe to
become a sysadmin after only a few hours of reading tutorials, it also
brought in a whole lot of people who don't know better. It's not that all
sysadmins got worse, it's that there are more of them, meaning you see a
lot more that are under-qualified. I can certainly understand the appeal:
as a developer, I don't care about the system. I want it to just work and
focus on creating the product. I will use every tool, script, and platform
to make that happen.

Where you run into problems is with people who are under-qualified as
sysadmins trying to create systems tools. This is when you see `curl -s
http://example.com/foo.sh | sudo sh`. That is bad, and is considered very
bad by lots of people. Yet some software authors don't know any better, and
the script actually does something useful so it ends up becoming a popular
method. Build layers and layers of this stuff and you get deployment
processes like you describe: untarring a .deb because the developer
couldn't figure out how to install it properly.

But is it as bleak as it seems? I don't think so. With many terrible
sysadmins (now known as devops), there also have appeared a huge number of
brilliant ones. Good sysadmins are more productive than ever, able to
manage thousands of servers at once. We have tools to verify that servers
are configured identically, that upgrades are happening properly and
securely. All distributions are still signing their packages, as they used
to, and reproducible builds are more important than ever. There is, in fact
an initiative to produce builds that are binary identical, so you can
verify that the distro-built package is exactly the same as what you can
compile yourself. People are very much paying attention to these issues.
There is even movement of the non-distro package repositories (such as
PyPI) to provide signatures to the packages they publish (they already
secure their transport, but not the source; this is a complex problem and
there are very good reasons for this, but a discussion for a different
time).

Basically, I think we are in the golden age: sysadmins are now more
productive than ever and can focus on much larger issues than before.

Igor