Securing Memcache in 2 Minutes

A couple of weeks ago I was setting up a few servers for a new client. Part of that setup was an instance of memcache, and I took a few minutes to show a colleague the 'how and why' of a secure memcache install. She said "You should blog that". It made it to my 'to do' list, but I hadn't gotten around to it until now.

This slashdot story was an inspiration to me. I can't believe the high-profile websites that had security problems with their memcache instances; clearly this blog entry is a needed public service.

Understanding the Problem

Before we can secure memcache appropriately, lets understand the problem, and also look at how we can test our secure installation. For the sake of this tutorial I'm going to talk about memcache running on linux systems. My examples will be against a CentOS 5.3 install, but this should apply to just about any flavor of linux. Your mileage on Windows may vary. I'm also not going to go into detail on what memcached is and why you'd use it as part of a web aplication architecture - suffice to say that for high performance web apps, it can be an important part of your application's performance.

The problem is actually pretty simple - memcached is built for speed, not for security, and it does nothing to secure itself. As far as memcache's authors are concerned, this isn't a problem... it just means you have to secure it yourself. The problem is the people don't do that.

Since its not secure, this means that random people on the internet can discover your memcache, read things from it, and even write into it. The kinds of security problems that could create are limited only by your imagination and the kinds of data you put into your cache.

Lets assume I have an insecure memcache instance running at my.example.com. I can log into that server, and connect to my memcache instance with telnet:

[dbock@my.example ~]$ telnet localhost 11211
Trying 127.0.0.1...
Connected to my.example.com (127.0.0.1).
Escape character is '^]'.

Notice I'm connecting to port 11211 - this is the standard memcache port. In this example, I successfully connected, so lets store a value in the cache:

add catchphrase 0 0 29
intelligence for sale or rent

and memcache returns

STORED

so far, so good. We just stuck a piece of data into our memcache.

but guess what? ANYONE, ANYWHERE on the internet can now get that piece of data. Log out of your machine and try this:

[dbock@my.desktopmachine] ~]$ telnet my.example.com 11211
Trying 192.0.32.10...
Connected to my.example.com (192.0.32.10).
Escape character is '^]'.

Wow... we connected. That seems risky... Now type:

get catchphrase

and memcache responds

VALUE catchphrase 0 29
intelligence for sale or rent
END

Ouch. We just connected to our server and pulled out our super-secret catch phrase. We don't want or need memcache to be listening to the outside world.

Securing the Server

If you are dealing with a multi-server environment, the solution will be more complicated than this - you'll want a firewall, a private network, and other complexity to separate the 'outside world' from your 'inside world' (contact me for advice if thats the case). If you are dealing with one server though, the solution is simple - just don't listen to your ethernet connection.

Mysql has an option to specify what address it listens to... so rather than listen to everything, we ony want to listen to the localhost - 127.0.0.1

On my CentOS box, I can do this by simply editing the

/etc/sysconfig/memcached

file, changing the line

OPTIONS=""

to

OPTIONS="-l 127.0.0.1"

and restart memcache with

sudo /sbin/service memcached restart

retry the steps above and you'll find that outside the server, telnet can't connect anymore.

The memcached settings file might be someplace else on other linuxes, depending on their file conventions... The memcacheinstall for your platform should give you a clue where this is, as this file is also needed to specify the amount of memory in the cache, etc.

I hope this prevents an attack against your server, and good luck!

Comments

[Mbuckbee]

I think this is excellent advice, but I'd suggest that it is secondary security to firewalling off ANY ports on your server that should not be accessed from outside.

[619Cloud]

Good article, but generally servers have iptables in place, and only open ports necessary like 80, 443, etc. Memcache should almost always be accessible ONLY to the local network.

[David Bock]

I agree with both of you, but only as part of a 'defense in depth' approach. Firewalling an open port can solve the problem, but never opening it to listen in the first place avoids having a problem to solve.

It is important to secure memcache itself first, then have a firewall secure the whole machine. I use an iptables-based firewall called APF. That would be worth another entry, or perhaps two- one on APF, and one on 'defense in depth'.

By the way, mysql has the same 'flaw' as memcache... It listens on port 3306, and should be configured to just bind to a specific Ethernet device (127.0.0.1 if used as in the example above).

[Sajal Basu]

3 options to protect your valuable memcash

1. Use non standard ports

2. Use IP addresses to listen (on amazon this works wonderfully)

3. Run Memcached as non standard user.