If you can't tell from my recent series of blog entries, I have spent a lot of time lately maintaining the CodeSherpas Server Farm and provisioning a few new machines for clients. I have several more entries pending, but they all rely on higher-order sysadmin techniques to keep track of all the 'noise' that a server can generate; so I thought I'd pull those out into a separate entry.
A running server generates a lot of system notifications. Tools like LogWatch, Monit, DenyHosts, LSM, etc. all send out emails when they find something. I have seen machines that send emails to the root user, to an email address set up specifically to receive them, and to the person who installed a specific tool - all at the same time. Without a unified view into the happenings of the system, an opportunity is lost and a hacker can slip through.
Technique #1 - Unify and Forward
The first part of this technique is obvious - unify all those email addresses into one destination - I typically prefer the root user on the machine itself, although I have also set up a user account specifically for this purpose. When done consistently, this is easy to maintain; The confusion over multiple email addresses only arises when the people installing tools think they need an original answer to this question. But having all those great system notifications don't do much good if they never leave the machine... do they?
The second part of this technique involves a few levels of indirection (and don't all good solutions, really?), and it uses a little-known trick of the linux-sysadmin gurus - for ".forward" file.
In the root users home directory, I create a file named ".forward". In this text file, I put an email address - your own email address would work - and now anything that gets sent to the root user simply gets forwarded to that email address.
At CodeSherpas, we use one more layer of indirection on top of that... the .forward file on our servers forwards to a special email address like "[email protected]" (and that isn't the real address, so don't bother spamming it). That address is simply an alias that gets rotated between the CodeSherpas - whoever is on syste duty is responsible for getting those messages.
Of course, the ultra-paranoid will tell you that an intruder can delete the .forward file, and I'll stop getting notices. Thats true - but if an intruder has gotten far enough to delete files in the root's home directory, then I can't trust any output from the machine. My trip wires should have gone off long before they have gotten that far.
Technique #2 - If it does't email, make it!
While many tools send emails as part of their normal operation, there are many that don't. thats ok - we can make them!
Take lynis, for instance. Lynis is an easy to install system checker that runs on just about every flavor of *nix out there. It generates a great report that makes system hardening recommendations, checks to make sure keys haven't expired, verifies firewall rules are in effect, and all kinds of other things (worth a blog entry or more by itself). It is a command line tool that is easy to run, but I don't want to have to remember to run it... We should never send a human to do a computers job.
I want to create a shell script that takes the output from lynix and mails it to root@localhost, and I want to set it to run once a week. This is trivial:
#!/bin/sh ( /root/lynis/lynis -c -Q -q ) | /bin/mail -s "Lynis Weekly Run for $HOSTNAME" root@localhost
There are two magic lines here:
/root/lynis/lynis -c -Q -q
This tells linus to do a complete system check, do it without stopping for human intervention, and only report problems.
| /bin/mail -s "Lynis Weekly Run for $HOSTNAME" root@localhost
And this line takes the output, creates a subject that includes the host name of the machine (useful when you are getting several of these a week), and email it to root@localhost (which has the .forward file, as described above).
I put that shell script in /etc/cron.weekly/lynis.sh, and voila! Every week the system checks itself and emails the current CodeSherpa sysadmin watchdog any findings.
That shell script above can easily be modified; just keep in mind that the command you put in between the parentheses should not require any human interaction, and ideally should report only problems (humans learn to ignore emails they get every day that say "all is well").