On the Path

Securing SSH

In one of my last posts, I mentioned I was going to write a post on securing SSH. As it turns out, I really like most of this advice, so rather than recreate it, I'm just going to comment and expand on it. Go read that article, then the rest of this will make sense.

Non-standard ports?

Item #5 in that list, about running on non-standard ports, I'm not that fond of. As we saw in an earlier blog post, it is trivial to port scan and find out where stuff is running, so changing the port doesn't do much to thwart an attack from someone analyzing your servers. It *would* thwart an attack from a script kiddie, but we are going to be secure enough to block those kinds of attacks. On the other hand, changing the port means we have a small configuration nuisance to have to remember whenever we are dealing with our system, configuring other services, etc... so I see exchanging minimal-if-any gain for the equivalent of a rock in my shoe a losing proposition. I see this more as security-through-obscurity than part of defense-in-depth, do I don't do it.

Filter SSH at the firewall?

Depending on your environment that might be good advice, but I don't do it. Have you ever been on the road and had to fix something on one of your servers? Can you imagine stopping at a Starbucks, hopping on their wifi, only to realize you can't ssh into your box? I don't have to imagine it; I have been there. As mobile as I am, I just cannot anticipate the IP addresses I'll need to connect from. But thats ok, because there is a better way.

Pull Up the Drawbridge

What if your server were smart enough to detect when it is under attack and do something defensive to stop or slow it? This is a technique I'll talk more about in future articles, but I'm going to introduce now with a tool called 'denyhosts'.

With denyhosts, you can easily set up your server to:

1. notice when a particular ip address has failed several login attempts
2. reconfigure your server to block traffic from that IP address
3. email you to let you know something is happening

And here is a real, live email I got from one of my servers this morning (I changed my machine name, but the attacker IP address and name is real):

Date: Fri, 13 Aug 2010 11:55:01 GMT
From: denyhosts 
To: root@machine.example.com
Subject: DenyHosts Report from machine.example.com
Added the following hosts to /etc/hosts.deny:
217.173.26.248 (dsl-26-248.n-chelny.ru)

Thank you, denyhosts! One more intruder stopped before I even knew there was a problem. This is not a rarity either - Of 9 servers that I currently administer for my company and other clients, attacks are blocked multiple times a day, every day.

On my typical CentOS configuration, I can easily install denyhosts with:

sudo yum install denyhosts

and start it with:

sudo /sbin/service denyhosts start

Use chkconfig to make sure its also set to restart on a reboot. On your linux flavor you'll have to adjust that to your package manager of choice.

The config file (located at /etc/denyhosts.conf or /etc/denyhosts/denyhosts.conf, depending on your packaging) is well documented and self-explanatory... take a look and set it up to email you when it takes any defensive action.

Nmap and Netstat: Analyzing Your Server's Public Exposure

My last two blog entries have talked about ways of locking down services to specific ports and IPs. In this entry, I want to show you two tools for inspecting your running server so you can see the result of your hard work, and identify other things that might need to be locked down.

Knowing What Your Server is Running

The first tool we are going to look at is netstat. Log into your linux server, and type this:

sudo netstat -lnpt

you'll see some output that looks like this (server names and IP address in this blog post have been changed to protect my clients):

[dbock@example.com ~]# sudo netstat -lnpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address   Foreign Address PID/Program name   
tcp        0      0 127.0.0.1:3306  0.0.0.0:*       28190/mysqld        
tcp        0      0 127.0.0.1:11211 0.0.0.0:*       1600/memcached      
tcp        0      0 127.0.0.1:631   0.0.0.0:*       2281/cupsd          
tcp        0      0 127.0.0.1:25    0.0.0.0:*       2425/sendmail 
tcp        0      0 :::80           :::*            3112/httpd          
tcp        0      0 :::22           :::*            2251/sshd           
tcp        0      0 :::443          :::*            3112/httpd

I'll leave it up to you to look at the man page and see what other parameters you can pass to netstat, but this is the view that matters most right now. This is a list of every service that is listening on an ethernet port, the name and process ID of that service, and the port it is listening on. This is pure gold from a security standpoint... many times I have done this to find an ftp service running that no one knew about, or some other tool like webmin.

Armed with this knowledge and the approach taken in the previous blog entries, you can now lock and and/or stop services you don't need. For instance, I really don't think that cups (the common unix printing system) is necessary on this server - I'll go ahead and stop that.

From that list above, we can see that mysql and memcache are locked down to localhost(127.0.0.1) as we planned... but how do we really *know* thats the case? What does the 'outside world' see when they look?

From the Outside Looking In

The next tool we are going to use is nmap. Nmap is an awesome tool that does a zillion different things for mapping out a network... in this example, we are going to use just one of its features - showing us the open ports running on a server.

You are going to run netstat from a machine other than your server, most likely your development box. We are going to do a port scan of your server, but before we do, a few words of caution - a port scan can be seen as hostile activity if you do it to someone else's server - at the very least it is rude. It is also possible that port scanning someone's server will put your IP address on that server's deny list, at least for a little while (and we'll see how to do that in a few blog posts).

With all that disclaimer out of the way, type this from your favorite shell:

nmap -A www.your-favorite-server.com

You'll get back something that looks like this:

Starting Nmap 5.00 ( http://nmap.org ) at 2010-08-11 02:28 UTC
Interesting ports on xxx.xxx.xxx.xxx:
Not shown: 997 closed ports
PORT    STATE SERVICE  VERSION
22/tcp  open  ssh      OpenSSH 4.3 (protocol 2.0)
|  ssh-hostkey: 1024 23:66:ab:b3:4f:45:70:77:79:65:4e:8b:37:61:71:88 (DSA)
|_ 2048 a2:e5:af:bd:33:70:77:79:65:ce:a8:d4:82:4f:5c:45 (RSA)
80/tcp  open  http     Apache httpd 2.2.3 ((CentOS))
|_ html-title: Apache HTTP Server Test Page powered by CentOS
443/tcp open  ssl/http Apache httpd 2.2.3 ((CentOS))
|_ html-title: Apache HTTP Server Test Page powered by CentOS

Wow... look at all that information! First, lets pat ourselves on the back - from the outside looking in, no one can tell we are running mysql or memcache. The only thing they can see is http(on port 80), https(on port 443), and ssh(on port 22). It is a little disconcerting that they can see specific tools and version numbers, as that advertises potential vulnerabilities we might be subject to. Not as shocking is that we can see keys for ssh - if you know how ssh works, public keys like this have to be, well, public.

We have effectively hidden a lot of information from a would-be attacker doing a port scan... there are a few things they have learned though that could be useful... the most important of which is "if I want to attack the content on this web server, I know which instance of ssh I should start poking at".

In my next several blog posts, we'll learn about some minimal security configuration and practices for SSH, as well as a technique called "Service Address Isolation" so we can separate ssh from the web services. And in a few posts hen I talk about Firewalling with APF, we might even talk about portknocking.

A Couple nmap Tricks

In closing, I'll leave you with a few things about nmap to pique your curiosity. Try running the same kind of scan with -v (for verbose) or -O (for Operating System fingerprinting)... you'll see a lot of information about your server you'd be surprised is so easy to determine.

Securing MySQL in 10 minutes

Much like my last entry on memcache, I'm not trying to write the ultimate security guide for securing a mysql server... I'm just outlining the kinds of steps you should take to have a level of security on par with the locks on your car... Enough to encourage the average plunderer to move along to another target.

Listen Only Where You Want to Hear

Just like memcache, the default mysql install is listening patiently on all your server's ethernet devices for someone to connect. So, just like memcache, we need to tell mysql to only listen on the ethernet device we care about. For your average single-server web host, you only need to listen to your local host.

Find your my.conf file. On servers I manage, this is either at

/etc/my.conf

or

/etc/mysql/my.conf

. This configuration file will contain all kinds of goodies about performance tuning your mysql server, connection details for your mysql client, etc. you want to find the section with the heading

[mysqld]

and add this line:

bind-address = 127.0.0.1

If you are in some kind of multi-machine environment, you'll want to set that to whatever your private IP address is - under no circumstances should this be an ip address the general public can reach.

Secure Your Root Password

When you first start mysql on a new install, it prints out a message that says something like "Don't forget to set a root password!", and gives you a couple of command-line examples. But guess what? A lot of people never get around to setting these (and I have been as guilty of that as anyone). Here are those commands again:

mysqladmin -u root password “newpassword”
mysqladmin -u root -h localhost password “newpassword”

Set yourself up a real root password. Don't forget to write it on a sticky note and put it on your monitor. Feel free to email it to me so I can keep a copy safe and sound for you (just kidding).

Use an Application-Specific User

If you're reading this blog, you probably have a Java, PHP, or Ruby application that is connecting to your mysql server and accessing data in a database. When you set everything up the first time you were happy just to get it working, and you probably used that root user we just locked down. We want to create a user specifically for our application, and give them just the abilities they need for our database.

Lets assume the name of that database is 'schpoo', and you want to create a user named 'my_app_user'. We want to give that user access to the database. There are several ways to do this in mysql; here is how I do it. Start by logging onto the mysql command-line tool:

[dbock@my.example ~]$mysql -u root -p

you'll be prompted to enter your shiny new root password. After you connect, we'll type:

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> GRANT ALL ON schpoo.* to 'my_app_user'@'localhost' IDENTIFIED BY '';
Query OK, 0 rows affected (0.11 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.06 sec)

Note how we specify 'localhost' above? That is going to allow this user only from the local machine. If you want to connect from another machine on a private network, put the IP address in there instead. You can even have the same user have different passwords from different IP addresses.

Grant ALL? Seriously?

Well, I'm speaking with my Ruby on Rails hat on - and I'm generally going to use that user to read and write to the database as well as run migrations, so 'ALL' makes sense. If you want an extra dose of paranoia, you can grant different sets of privileges to different users - but doing that properly would become advice more tuned to configuring, say Hibernate, than mysql.

A Note About Passwords

We all know its bad to reuse passwords... so how do manage passwords for a dozen clients across a dozen applications, and not reuse passwords?

There is a great little command-line linux tool used for generating universally unique identification strings, called uuidgen. Here's some sample output:

[dbock@my.example ~]$ uuidgen
b70b8753-bbc0-4881-9b81-5e00baacd39a
[dbock@my.example ~]$ uuidgen
8f79aadd-29a0-41b7-bf66-6ad23ea49f04
[dbock@my.example ~]$ uuidgen
ea56ef7f-556e-44d5-a808-8baebc74c3d1

Those make great passwords. Since I typically store my application credentials in a secure location on the server and copy them into place when I deploy my client application, I don't have to remember those passwords. I typically don't have to worry about typing them either. That is a useful technique that is really easy when you're using a deploy tool like capistrano, but that is a different blog entry.

In Closing

Setting appropriate passwords, limiting privileges, and only listening on ports you expect traffic from is hardly top-secret 'advice from the mysql experts', but it is advice most mysql installs would benefit from. Give yourself 10 minutes and audit a server you are responsible for today.

Securing Memcache in 2 Minutes

A couple of weeks ago I was setting up a few servers for a new client. Part of that setup was an instance of memcache, and I took a few minutes to show a colleague the 'how and why' of a secure memcache install. She said "You should blog that". It made it to my 'to do' list, but I hadn't gotten around to it until now.

This slashdot story was an inspiration to me. I can't believe the high-profile websites that had security problems with their memcache instances; clearly this blog entry is a needed public service.

Understanding the Problem

Before we can secure memcache appropriately, lets understand the problem, and also look at how we can test our secure installation. For the sake of this tutorial I'm going to talk about memcache running on linux systems. My examples will be against a CentOS 5.3 install, but this should apply to just about any flavor of linux. Your mileage on Windows may vary. I'm also not going to go into detail on what memcached is and why you'd use it as part of a web aplication architecture - suffice to say that for high performance web apps, it can be an important part of your application's performance.

The problem is actually pretty simple - memcached is built for speed, not for security, and it does nothing to secure itself. As far as memcache's authors are concerned, this isn't a problem... it just means you have to secure it yourself. The problem is the people don't do that.

Since its not secure, this means that random people on the internet can discover your memcache, read things from it, and even write into it. The kinds of security problems that could create are limited only by your imagination and the kinds of data you put into your cache.

Lets assume I have an insecure memcache instance running at my.example.com. I can log into that server, and connect to my memcache instance with telnet:

[dbock@my.example ~]$ telnet localhost 11211
Trying 127.0.0.1...
Connected to my.example.com (127.0.0.1).
Escape character is '^]'.

Notice I'm connecting to port 11211 - this is the standard memcache port. In this example, I successfully connected, so lets store a value in the cache:

add catchphrase 0 0 29
intelligence for sale or rent

and memcache returns

STORED

so far, so good. We just stuck a piece of data into our memcache.

but guess what? ANYONE, ANYWHERE on the internet can now get that piece of data. Log out of your machine and try this:

[dbock@my.desktopmachine] ~]$ telnet my.example.com 11211
Trying 192.0.32.10...
Connected to my.example.com (192.0.32.10).
Escape character is '^]'.

Wow... we connected. That seems risky... Now type:

get catchphrase

and memcache responds

VALUE catchphrase 0 29
intelligence for sale or rent
END

Ouch. We just connected to our server and pulled out our super-secret catch phrase. We don't want or need memcache to be listening to the outside world.

Securing the Server

If you are dealing with a multi-server environment, the solution will be more complicated than this - you'll want a firewall, a private network, and other complexity to separate the 'outside world' from your 'inside world' (contact me for advice if thats the case). If you are dealing with one server though, the solution is simple - just don't listen to your ethernet connection.

Mysql has an option to specify what address it listens to... so rather than listen to everything, we ony want to listen to the localhost - 127.0.0.1

On my CentOS box, I can do this by simply editing the

/etc/sysconfig/memcached

file, changing the line

OPTIONS=""

to

OPTIONS="-l 127.0.0.1"

and restart memcache with

sudo /sbin/service memcached restart

retry the steps above and you'll find that outside the server, telnet can't connect anymore.

The memcached settings file might be someplace else on other linuxes, depending on their file conventions... The memcacheinstall for your platform should give you a clue where this is, as this file is also needed to specify the amount of memory in the cache, etc.

I hope this prevents an attack against your server, and good luck!