Patch for memcached on public network

If you want to know what memcached is all about, check out my recent post about memcached on The Rackspace Cloud Blog.

In order to use memcached in the cloud, you may need to run it on a public network. This introduces a rash of security concerns. Originally memcached was only intended for use on private networks that were not available to the public, so there was no attempt made to provide access controls in the memcached server. There are no concepts of users, passwords, or any access control at all.

If you do run your memcached on a public interface you could use iptables or other host-based firewall rules to limit what IP addresses can access your memcached. However, if you are using a platform hosting service that other subscribers share with you, then others may be able to make connection from the same IP address(es) as you. This means that even if you did limit access to your memcached by IP address it’s possible that some other subscriber of the same hosting service could access your memcached, and cause you all sorts of security problems.

Here is a custom patched memcached 1.4.0 x86_64 RPM I wrote that adds a command line option ‘S’ to disable ‘flush_all’ and ‘stats detail on’ . The original 1.4 source, a SPEC file for RHEL5 and CentOS5 and the patch are both included in the SRPM. By disabling these commands with the -S option in /etc/sysconfig/memcached (OPTIONS=”-S”) you can prevent would-be hackers from dropping all your cached items, or finding out what the names are of the keys you are using. The memcached maintainers want to do this a different way, so this patch won’t be included in the base memecahced source tree.

The right long-term solution is to build multi-tenant features directly into memcached. I’m aware that Dustin Sallings at NorthScale has started some work of this sort, and has a working proof of concept. It’s not yet mature, and is generally incompatible with the current release of memcached, so it’s not yet suitable for production use. The main idea is that a TCP/IP connection to memcached could be authenticated with SASL, and limited to it’s own view of what’s inside memcahced.

My patch does not change how memcahced works, except for what it does when you enter the commands that I’m disabling. It will be just as stable as memcached 1.4.0 without the patch. The only difference is that you won’t have the ‘flush_all’ command, and you won’t have access to detailed stats either.

If you want to flush your entire cache, simply reconfigure your application to begin using a new “secret” key prefix, and you’ll have the functional equivalent of a flush_all because none of the prior cached data will be accessed by your application any more. The old data will simply expire or LRU out of the cache and be replaced by new data naturally.

By using a simple “secret” text prefix to all your keys, you will ensure that hackers won’t know how to access your data in the cache.  Consider prepending a reasonably long test string to the beginning of every key you store and access. Don’t make it too long, or that will multiply the number of packets required to get the data in and out of the cache, but something long enough that it won’t be easily guessed.

This patch does not make memcached bulletproof. An attacker can still do a bunch of SET commands to fill your cache with junk, and force your hot content out. They can still irritate it with a bunch of ‘stats sizes’ commands in a loop, or try to guess your secret prefix by randomly generating keys as a brute force attack until they find your content. For these reasons, you should only use this for storing data that’s not mission critical. There’s lots of data in this category that could really speed up your system under high load if you stored it in memcahed, but is not particularly sensitive to tampering.

Some have argued that this sort of a patch offers a false sense of security. I completely agree. Only use this if you know that your memcached installation will still not be secure, and that the security weakness could be exploited to ultimately hack your application. It will just be a little bit less insecure than it is without the patch.

I have seen memcached used in situations where only statistics are stored and accessed in memcached (instead of generating log files, statistical counters are stored in the cache). The application can do strict checking of the data it gets back from the cache, and not use it in any way that could lead to a security compromise. For example, make sure that all values returned are only numeric, and within acceptable value boundaries. An application of this sort would be appropriate with this patch.

I was thinking of making a better version of this patch that would allow you to specify an IP address (potentially 127.0.0.1 for example) that would have access to all commands that you define in a restricted access class. This way you could configure what IP address(es) could access what commands. Implementing this will require slowing memcached down a bit for all commands. I’ll plan to join forces with the others who are also interested in memcached multi-tenant features and produce a suitable solution that allows for secure deployments in insecure networks.