Charter

• A database optimized for Cloud infrastructure and Web applications
• Design for massive concurrency on modern multi-cpu architecture
• Optimize memory for increased performance and parallelism
• Open source, open community, open design

Scope

• Re-designed modular architecture providing plugins with defined APIs
• Simple design for ease of use and administration
• Reliable, ACID transactional

There are many exciting changes, such as optimizing everything for 64-bit CPU’s and Multi-Core. You can’t hardly even buy 32-bit and Single Core servers nowadays if you want them. It makes no sense to have software that’s optimized for these antiquated hardware designs. No effort is spent optimizing software to work with rotational hard drives because SSD drives are the way of the future. All the language collations have simply been replaced with UTF-8 only, because the web uses UTF-8. Plus, this is tested with 41 different language translations. Drizzle has a new scheduler. The legacy MySQL scheduler was designed to work for a thread-per-session setup. In Drizzle, sessions are handled independently from the threads. The new scheduler allows this to work.

Drizzle uses InnoDB as its default storage engine, which is great for OLTP. It also supports the PBXT storage engine. There are available plugins for the InnoDB Embedded Engine and HailDB which will soon be the new default. DDL Operations (like ALTER TABLE) can actually roll back in the event that something goes wrong in the process, rather than leaving you with incomplete or corrupt data.

The code base in Drizzle has been fully modernized, and brought up to today’s standards of C++ with extensive use of the C++ STL to replace MySQL’s usage of obscure custom data type implementations that offered no real benefit compared to what the STL has today. Another example of improvements in this area is the replacement of the legacy REGEX implementation with a more standard library. All of these changes reduce the amount of Drizzle source code dramatically compared to MySQL. Less code and simpler code means less bugs, plain and simple. Drizzle is well on its way to being an ideal fit for web applications that need a reliable, and high performance transactional database.

Features in Drizzle7 Beta

• New micro kernel
• Migration Tool
• Instance Catalog Support
• Universal Replication
• User query analysis
• Mutli-core Support

What “Beta” means

• Your data is safe. Transactional engine by default and stable for over 2 years.
• Upgrade the system in-place without exporting/importing data.
• Replication is still being tested.

In Microsoft terms, it means that this project would have launched about a year ago. In Google terms, it probably would have launched six months ago. Simply put, if you trust your data to a MySQL system today running InnoDB, you should feel comfortable trying Drizzle. There have been some changes to the InnoDB setup, such as the elimination of the FRM files from disk which eliminate possible inconsistency between the state on disk and the state in InnoDB. I am in the process of moving a few of my produciton applications to use the Drizzle Beta. If you’re an accomplished system administrator and DBA, you should seriously consider putting at least one of your production applications on Drizzle now, and see how it works for you.

What’s Next?

• Beta announced today 2010-09-29.
• GA February 2011
• GA May 2011 for Multi-Tenancy features that allow an arbitrary number of logical databases (Schemas, Tables, etc.) to exist concurrently with full data isolation between them. This allows for individual security and resource controls (Threads, Memory, IO), and individual database backups, rather than system level backups. This feature will be called “Catalogs”.

Download Drizzle

Time to get started with the beta. Download the beta today!

• 100% non-blocking
• Unlimited horizontal scalability
• Unlimited vertical scalability
• Supports Sharding
• Supports Clustering
• Exceeds write performance of all other databases
• Unparalleled concurrency support
• Write-and-forget

Here is a chart that illustrates write latency and throughput with various different thread concurrency:

As you can see, as the number of concurrent writers increases, throughput increases proportionally. No matter how many threads run concurrently, latency remains at zero.

Support in MySQL

You may be thrilled to know that this data storage system is fully supported in MySQL using the Blackhole Storage Engine written by Brian Aker. Anyone considering MongoDB should give this alternative some consideration, as it exhibits the same level of data loss for new data pending writes before a node failure. Plus, MySQL has been around for a long time, and this storage engine is the single most reliable storage engine that MySQL ever produced.

Recently you may have noticed my writing about Drizzle, but that’s not the only database system I love. I’m also a fan of Cassandra, and I’m proud to work with the same company sponsoring both projects.

Drizzle is the way to go if you want an SQL system, and Cassandra is the way to go if you have a huge data set or if you have a data insert/update rate that’s too high for and RDBMS to keep up with.

In order to use memcached in the cloud, you may need to run it on a public network. This introduces a rash of security concerns. Originally memcached was only intended for use on private networks that were not available to the public, so there was no attempt made to provide access controls in the memcached server. There are no concepts of users, passwords, or any access control at all.

If you do run your memcached on a public interface you could use iptables or other host-based firewall rules to limit what IP addresses can access your memcached. However, if you are using a platform hosting service that other subscribers share with you, then others may be able to make connection from the same IP address(es) as you. This means that even if you did limit access to your memcached by IP address it’s possible that some other subscriber of the same hosting service could access your memcached, and cause you all sorts of security problems.

Here is a custom patched memcached 1.4.0 x86_64 RPM I wrote that adds a command line option ‘S’ to disable ‘flush_all’ and ‘stats detail on’ . The original 1.4 source, a SPEC file for RHEL5 and CentOS5 and the patch are both included in the SRPM. By disabling these commands with the -S option in /etc/sysconfig/memcached (OPTIONS=”-S”) you can prevent would-be hackers from dropping all your cached items, or finding out what the names are of the keys you are using. The memcached maintainers want to do this a different way, so this patch won’t be included in the base memecahced source tree.

The right long-term solution is to build multi-tenant features directly into memcached. I’m aware that Dustin Sallings at NorthScale has started some work of this sort, and has a working proof of concept. It’s not yet mature, and is generally incompatible with the current release of memcached, so it’s not yet suitable for production use. The main idea is that a TCP/IP connection to memcached could be authenticated with SASL, and limited to it’s own view of what’s inside memcahced.

My patch does not change how memcahced works, except for what it does when you enter the commands that I’m disabling. It will be just as stable as memcached 1.4.0 without the patch. The only difference is that you won’t have the ‘flush_all’ command, and you won’t have access to detailed stats either.

If you want to flush your entire cache, simply reconfigure your application to begin using a new “secret” key prefix, and you’ll have the functional equivalent of a flush_all because none of the prior cached data will be accessed by your application any more. The old data will simply expire or LRU out of the cache and be replaced by new data naturally.

By using a simple “secret” text prefix to all your keys, you will ensure that hackers won’t know how to access your data in the cache.  Consider prepending a reasonably long test string to the beginning of every key you store and access. Don’t make it too long, or that will multiply the number of packets required to get the data in and out of the cache, but something long enough that it won’t be easily guessed.

This patch does not make memcached bulletproof. An attacker can still do a bunch of SET commands to fill your cache with junk, and force your hot content out. They can still irritate it with a bunch of ‘stats sizes’ commands in a loop, or try to guess your secret prefix by randomly generating keys as a brute force attack until they find your content. For these reasons, you should only use this for storing data that’s not mission critical. There’s lots of data in this category that could really speed up your system under high load if you stored it in memcahed, but is not particularly sensitive to tampering.

Some have argued that this sort of a patch offers a false sense of security. I completely agree. Only use this if you know that your memcached installation will still not be secure, and that the security weakness could be exploited to ultimately hack your application. It will just be a little bit less insecure than it is without the patch.

I have seen memcached used in situations where only statistics are stored and accessed in memcached (instead of generating log files, statistical counters are stored in the cache). The application can do strict checking of the data it gets back from the cache, and not use it in any way that could lead to a security compromise. For example, make sure that all values returned are only numeric, and within acceptable value boundaries. An application of this sort would be appropriate with this patch.

I was thinking of making a better version of this patch that would allow you to specify an IP address (potentially 127.0.0.1 for example) that would have access to all commands that you define in a restricted access class. This way you could configure what IP address(es) could access what commands. Implementing this will require slowing memcached down a bit for all commands. I’ll plan to join forces with the others who are also interested in memcached multi-tenant features and produce a suitable solution that allows for secure deployments in insecure networks.