[TriLUG] Server Maintenance

[Jon Carnes]

On Fri, 2003-08-01 at 08:17, Hite, Danny wrote: 
> Good news. I have finally got Linux (2 Redhat 7.3 servers) implemented in my
> work environment. *dancing* w00t!!
> 
> Question: Could you guys and gals give me some recommendations on
> maintenance? Here are some of the things I was thinking about.
> 
> Disaster recovery options, Backups, Security Updating, Daily log checking,
> other tips, rescue disks, etc...

For backups, make sure you grabbing snapshots of /etc (all files and
subdirectories), also if you have used nothing but stock rpms for your
installs, you should keep a list of installed rpms (rpm -qa).

For a snapshot of /etc I would run something like:
  DAY=`date "+%b%d"`
  rpm -qa > /etc/installed_rpms
  tar -czf /home/backup/etc_$DAY.tar.gz /etc

That will enable you to regenerate a complete system from scratch
(assuming that all your configuration files are in /etc and you don't
allow any personal data on the server).  You already have a backup tool,
so it sounds like you are in good shape.  

As for Security updates, you can automate those by buying a Red Hat
Network license for your server.  The cost is cheap and you can set it
to automatically apply any security updates.  As an aside, it will also
store a lot of your servers configuration - specifically the rpms
installed on the server.

You should be running a firewall on the box, and blocking all
in/outbound ports that are not currently used by the server.  Also, if
you are extra paranoid, you should look at running Tripwire (an
intrusion detection tool).

One of my favorite things to do is store backup copies of command files
like: ps, ls, netstat, etc... all executables that would be replaced by
a root kit. Then I periodically run a diff that checks to see if the
actual file has changed from the backup.

I also run a periodic scan of network connections (using the backed up
version of netstat), and then send the diffs for that:
  /home/backup/netstat -na |grep tcp |grep -v ":80"
(In your case, you would want to grep off the connections to port 80)

There are some dandy tools for doing log checking.  I diff my logs on
remote servers (getting rid of the standard messaging cruft) and then
just send the diffs as an email).  If I get information in these mails
that I can ignore, then I go back to the server and adjust my diff
filters. The filters are basically just a series of greps:
    | grep -v 'phrase to ignore' /

Whatever is left over (if anything) is then mailed to me.

I also have certain trigger words that I scan the logs for every 10
minutes to an hour. If one of the trigger phrases comes up, then I get
an immediate notification.  An example would be someone logging into the
server from an external ip address, or something as simple as:
  grep error /var/log/messages

> These are things I of course do on the M$ side, but what are some ways to
> approach this in Linux?
> 
> Server 1 is used for a www.phpbb.com installation and apache of course. It
> also uses sendmail to delivery email messages RE: the phpbb forum. This
> server is on the DMZ/SSN of my firewall with only port 80 forwarded from
> external to DMZ/SSN. SMTP is enabled outbound ONLY for sendmail.
> 
> Server 2 is on my corp intranet running as a Jabber server. There are no
> external connections, besides web surfing, on this server.
> 
> The backup software I use is Veritas' NetBackup. I do know that there is a
> client for *nix so I will probably look into that.
> 
> Sorry if I am asking WAY too many question in this email, but in the end I
> guess that I really am asking is 'What is the best way to keep a Linux
> server "purring like a kitten"?'
> 
> _____________________
> Danny Hite
> MCP
> Network Administrator
> Per-Se Technologies
> (919)564-2100
> _____________________