Most simple web clusters rely on a single database sever that all the application servers connect to. This way, all of the application servers have concurrent access to the same data. This can be problematic in the elastic use case when workloads increase, and more servers are added to the cluster. If the work is bottle-necked on storing or accessing data in the database server, adding additional application servers will not help. It will actually make the problem worse.

I spoke on a panel at Zendcon yesterday, which was covered in an Infoworld article where my remarks were published. The article says:

Panelists also debated use of SQL and database connectivity in clouds. SQL as a design pattern for storage “is not ideal for cloud applications,” said Adrian Otto, senior technical strategist for Rackspace Cloud. Afterward, he described SQL issues as “typically the No. 1 bottleneck” to elasticity in the cloud. With elasticity, applications use more or fewer application servers based on demand. Otto recommended that developers who want elasticity should have a decentralized data model that scales horizontally. “SQL itself isn’t the problem. The problem is row-oriented data in an application,” which causes performance bottlenecks, said Otto.

The author Paul Krill did a good job here of accurately reporting my position on this subject. Data stored in databases are arranged in tables of rows and columns. A new piece of data adds a new row. Each row has multiple columns that separate fields of a single record of data in the table. The truth is that most web applications work very well with this data design pattern. Those should continue to use SQL databases with row oriented data. However, there are some applications where data may be arranged differently to make reading the data more efficient.

If you have a big table of data, and you want to pull out just a little bit of it using a query, the database server must determine the location of that data in the table by consulting the table’s index, and return the desired portion that matches the constraints given in the query. This makes the reading of data relatively expensive from a computational perspective. If data were instead arranged in lots of columns instead, it could be retrieved more efficiently, and the data could be more easily distributed over a larger number of servers yielding the horizontal scalability that cloud applications want. This works very well in cases where the number of reads are very high, but the data is not updated very frequently in proportion to the reads.

Let’s use a blog application as an example. Blog posts are written once, and maybe updated a few times, possibly once each time a comment is submitted. However, on a busy web site, a blog post may be read millions of times. If the posts were stored in a column oriented storage system like Cassandra, they could be quickly and easily retrieved using the id number of the blog post. The listing of recent blog posts can also be arranged in a column so that the front page of the blog site with the listing of the articles can be generated. Using this approach requires that the data be properly arranged as it’s stored, putting the computational burden on the (infrequent) write rather than on the (frequent) read.

Using a distributed system to store data in columns allows the data to be evenly distributed over an arbitrary number of servers, eliminating the central data bottleneck. Adding more servers in the correct proportion of application servers and storage servers can result in true horizontal scalability, meaning that the capacity increases as a direct proportion of how many servers are in the cluster.

Why doesn’t everyone do this already? For some good reasons:

1. The concept of running applications in clouds is still relatively new. The related technology is still maturing.
2. Existing software tends to use SQL already. If you want to use an existing CMS platform, chances are it will require a central SQL database.
3. Most heavy-read workloads can be scaled well using data caching techniques. If applications don’t write data very often, it may not be necessary to scale beyond a single database server.
4. You must anticipate exactly how the application will use the data, and arrange it just right.
5. It may be harder to analyze the data. Once your data is arranged in a column store, you may not be able to query it in arbitrary ways. You may only be able to pull it out using it’s id numbers, or by systematically scanning all of it to find the parts you want.
6. Distributed data storage (aka: NoSQL) systems like Cassandra, Hbase, Redis, etc. are complicated, and there is a considerable learning curve associated with setting them up and maintaining them. In some cases these systems are not as good in terms of data durability or data consistency as the prevailing SQL database systems. These tradeoffs can be difficult to navigate.
7. Today’s software developers are very familiar with SQL as a data storage and access paradigm. They can very quickly develop software that relies on the ACID qualities of a SQL database.

If you have an application that you want to deploy into a cloud, and you want it to be very elastic, you should think about the subject of how you arrange your data. If you use a centralized data design, you will probably have scalability bottlenecks when you add lots of servers. You should aim to decentralize the data in a way that you can easily add more servers to horizontally scale the environment, and not stumble on the limits of the database server. This is particularly important in situations where you need the application to write a lot of data, and a cache is not a suitable solution for you.

Over time, the reasons why not to use column oriented data will begin to shrink, and better tools and services will make it easier to do. Until then, I suggest that you carefully consider if you need maximum elasticity. If not, then it’s perfectly appropriate to keep using the same centralized row-oriented data paradigm. Use a cache like memcached in cases where you have heavy reads, and when it’s acceptable to show slightly outdated information to readers. The truth is that traditional solutions work really well for most web applications. However, if you have one of the more unique situations where you need true horizontal scalability, take a good look at a different arrangement for your data, and the systems and tools to make that possible for you in the cloud.

Charter

• A database optimized for Cloud infrastructure and Web applications
• Design for massive concurrency on modern multi-cpu architecture
• Optimize memory for increased performance and parallelism
• Open source, open community, open design

Scope

• Re-designed modular architecture providing plugins with defined APIs
• Simple design for ease of use and administration
• Reliable, ACID transactional

There are many exciting changes, such as optimizing everything for 64-bit CPU’s and Multi-Core. You can’t hardly even buy 32-bit and Single Core servers nowadays if you want them. It makes no sense to have software that’s optimized for these antiquated hardware designs. No effort is spent optimizing software to work with rotational hard drives because SSD drives are the way of the future. All the language collations have simply been replaced with UTF-8 only, because the web uses UTF-8. Plus, this is tested with 41 different language translations. Drizzle has a new scheduler. The legacy MySQL scheduler was designed to work for a thread-per-session setup. In Drizzle, sessions are handled independently from the threads. The new scheduler allows this to work.

Drizzle uses InnoDB as its default storage engine, which is great for OLTP. It also supports the PBXT storage engine. There are available plugins for the InnoDB Embedded Engine and HailDB which will soon be the new default. DDL Operations (like ALTER TABLE) can actually roll back in the event that something goes wrong in the process, rather than leaving you with incomplete or corrupt data.

The code base in Drizzle has been fully modernized, and brought up to today’s standards of C++ with extensive use of the C++ STL to replace MySQL’s usage of obscure custom data type implementations that offered no real benefit compared to what the STL has today. Another example of improvements in this area is the replacement of the legacy REGEX implementation with a more standard library. All of these changes reduce the amount of Drizzle source code dramatically compared to MySQL. Less code and simpler code means less bugs, plain and simple. Drizzle is well on its way to being an ideal fit for web applications that need a reliable, and high performance transactional database.

Features in Drizzle7 Beta

• New micro kernel
• Migration Tool
• Instance Catalog Support
• Universal Replication
• User query analysis
• Mutli-core Support

What “Beta” means

• Your data is safe. Transactional engine by default and stable for over 2 years.
• Upgrade the system in-place without exporting/importing data.
• Replication is still being tested.

In Microsoft terms, it means that this project would have launched about a year ago. In Google terms, it probably would have launched six months ago. Simply put, if you trust your data to a MySQL system today running InnoDB, you should feel comfortable trying Drizzle. There have been some changes to the InnoDB setup, such as the elimination of the FRM files from disk which eliminate possible inconsistency between the state on disk and the state in InnoDB. I am in the process of moving a few of my produciton applications to use the Drizzle Beta. If you’re an accomplished system administrator and DBA, you should seriously consider putting at least one of your production applications on Drizzle now, and see how it works for you.

What’s Next?

• Beta announced today 2010-09-29.
• GA February 2011
• GA May 2011 for Multi-Tenancy features that allow an arbitrary number of logical databases (Schemas, Tables, etc.) to exist concurrently with full data isolation between them. This allows for individual security and resource controls (Threads, Memory, IO), and individual database backups, rather than system level backups. This feature will be called “Catalogs”.

Download Drizzle

Time to get started with the beta. Download the beta today!

Recently you may have noticed my writing about Drizzle, but that’s not the only database system I love. I’m also a fan of Cassandra, and I’m proud to work with the same company sponsoring both projects.

Drizzle is the way to go if you want an SQL system, and Cassandra is the way to go if you have a huge data set or if you have a data insert/update rate that’s too high for and RDBMS to keep up with.

It’s the time the hypervisor scheduled something else to run instead of something within your VM. This might be time for another VM, or for the Hypervisor host itself. If no time were stolen, this time would be used to run your CPU workload or your idle thread.

There is some disagreement circulating about whether the Hypervisor will steal idle time, or only preempted time. In other words, it has been suggested that stolen time is where your local kernel scheduler within the VM wanted to run something but the Hypervisor made that impossible. I have found that stolen time does in fact count borrowed idle time, where the local scheduler actually had nothing to run. For example, here are some vmstat values from a VM that’s got a very low cpu workload on it:

vmstat -S M 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0    121     42     53    460    0    0     0     1    0    1  0  0 89  0 10
 0  0    121     42     53    460    0    0     0    28 1014   39  0  0 90  0 10
 0  0    121     42     53    460    0    0     0     0 1016   36  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1024   32  0  0 93  0  7
 0  0    121     42     53    460    0    0     0     0 1019   40  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1015   32  0  0 90  0 10
 0  0    121     42     53    460    0    0     0     0 1022   34  0  0 92  0  8
 0  0    121     42     53    460    0    0     0     0 1016   36  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1013   34  0  0 92  0  8
 0  0    121     42     53    460    0    0     0     0 1028   43  0  0 93  0  7

As you can see, user time (us), system time (sy), and iowait time (wa) are zero, but idle time is not 100%. This normally indicates that your system is doing something, but in this case idle time is actually the sum of the id and st columns.

In this example, I really don’t care that I have a nonzero st column because my workload is basically idle all the time anyway.

If you are on a cloud host where you purchase a small sliver of a server, you should expect to see nonzero values in this column when you run vmstat. If you have a heavy CPU load and need more processing power, you can solve this problem by upgrading to a larger VM server size so that you command a larger portion of the physical host.

Okay, here’s something that fixes that problem: the Remus Project. The approach is brilliant. On regular intervals it ships the changed memory registers from one host to the other. Memory reading does not need to be replicated, only writes, and writes to the same location don’t all need to be replicated, only the most recent write. The primary node simply delays its response to TCP/IP packets (output buffering) until after it has confirmed that the standby node has received the replicated memory data. Very very clever.

Here are the key features listed on the Remus web site:

• The backup VM is an exact copy of the primary VM. When failure happens, it continues running on the backup host as if failure had never occurred.
• The backup is completely up-to-date. Even active TCP sessions are maintained without interruption.
• Protection is transparent. Existing guests can be protected without modifying them in any way.

Okay, I’ve been running HA systems in multiple geographies now for about a decade. I’ve experimented with lots and lots of clustering and replication technology. Most of the time when I hear about something new, I cringe and wonder if it’s just another thing that’s using the same old tricks I’ve been using for years, or if its something truly innovative and truly open source. Before you go making comments that VMWare has this feature or that feature, relax. This post is not about VMWare. It’s about open source Xen.

Now, you might already be wondering if this would work if you separated the two nodes to run in separate locations. The short answer is maybe. You would still need a very clever network configuration to re-route your traffic dynamically to the new location. For those of us that do operate our own Autonomous Systems, that may seem possible with a BGP route update. But here’s the bummer… The additional latency it would introduce would bring your performance to a screeching halt. You could probably afford to have about 25ms of average latency between two locations and get away with it. The cut-over would still be better than nothing, but you’d better have a rock solid network in there, and you’d better be ready to pump lots of bandwidth over it. Plan for 100Mb/sec if you checkpoint every 100ms.

This would be great for a high read application like a web cache, or some memcached applications. People ask on the memcached mailing list all the time how they can set up replication and HA. The answer is always “it’s a cache… not a database.”. Well, for those of you that want to do HA for a memcached system, give Remus a try.

Let’s not stop there. Imagine you have a SIP call control platform or Trixbox system, and you don’t want to lose all your active calls in the event of a system crash? Pretty much any mission critical application that supports long running connections over TCP/IP

Remus has been around for some time, so why am I so excited now? It’s now part of Xen! You don’t need to do anything special on the master or slave node to use it! Whoot! Now I’m impressed. Anyone out there have experience running it? I’d love to hear your thoughts.

I use the TimeMachine software built into Leopard (and newer) OSX. I use a locally connected USB2. A Firewire drive would also be good. Here is a drive that I like because it has lots of capacity, reasonably affordable, compact, and runs quietly.

Fantom GreenDrive Pro 2TB eSATA and USB 2.0 7200RPM 32MB External Hard Drive

Another that’s half the capacity, but cheaper:

Fantom GreenDrive 1TB USB 2.0 and eSATA External Hard Drive

Now, for home use a 2TB drive is probably enough for all your computers. At first I networked them all together to use just one drive on one of my computers shared to all the others so that all the backups were on the one big drive. I later decided that every computer should have it’s own drive for backups. Why? A few reasons:

1. To conserve electricity. When you are using the computer is when the backup snapshots should be taken and archived. When the computer is asleep, may not respond over the network depending on how it’s set up, meaning you need to keep that host machine powered up all the time wasting electricity.
2. Each computer does its backups when they get used, and in the idle time before they fall asleep again. It works much better for me this way.
3. Immediate restores. Having a local drive on each computer makes restoration super fast. It’s not like a network or tape backup where you need to wait for your data to transfer back on to your hard drive to begin using it.

It’s easy to set up Time Machine. Connect the drive, open “Time Machine Preferences” and select the drive.

I re-initialized mine using the disk utility first so that it had a journaled MacOS filesystem on it instead of the default FAT partitioning that comes from the factory.

One really nice thing about Time Machine is that you can easily revert to a prior point in time in the event you accidentally mess something up, get a virus, or whatever. It’s about the easiest tool I’ve ever used. it automatically rotates backups hourly, daily, weekly, etc and deletes old backups automatically to make room for new ones. It’s totally automatic whereas with other tools you need to set that all up yourself.

This sort of local backup does not help if your house or office gets burglarized or burns down because you lose both the primary and backup copy of the data.

Another option is to use JungleDisk to back your data up to the cloud. That has the advantage of only paying for the storage you actually use, the backups are off site, so if you have theft or fire, you can still restore, potentially somewhere else. A disadvantage is that it requires adequate internet connectivity. Your upload speed needs to be fast enough to accommodate all of the data you produce within each backup interval. If your network is already constrained on available bandwidth, running backups over it could potentially aggravate matters. In short, if you have a big fat internet connection, then use JungleDisk.

Rule 1 – Cache is Your Friend

Rule 2 – Don’t write to the database in real time

Rule 3 – Use a “Stateless” design whenever possible

Rule 4 – Avoid Unnecessary External Dependencies

Rule 5 – CMS Plugins

Rule 6 – HTTP Includes

Rule 7 – Coming Soon

Rule 8 – Coming Later

Rule 9 – Coming Later

Rule 10 – Coming Later

Yep, if you follow all 10 of the rules, you’ll probably have a really good cloud based web app.

In order to use memcached in the cloud, you may need to run it on a public network. This introduces a rash of security concerns. Originally memcached was only intended for use on private networks that were not available to the public, so there was no attempt made to provide access controls in the memcached server. There are no concepts of users, passwords, or any access control at all.

If you do run your memcached on a public interface you could use iptables or other host-based firewall rules to limit what IP addresses can access your memcached. However, if you are using a platform hosting service that other subscribers share with you, then others may be able to make connection from the same IP address(es) as you. This means that even if you did limit access to your memcached by IP address it’s possible that some other subscriber of the same hosting service could access your memcached, and cause you all sorts of security problems.

Here is a custom patched memcached 1.4.0 x86_64 RPM I wrote that adds a command line option ‘S’ to disable ‘flush_all’ and ‘stats detail on’ . The original 1.4 source, a SPEC file for RHEL5 and CentOS5 and the patch are both included in the SRPM. By disabling these commands with the -S option in /etc/sysconfig/memcached (OPTIONS=”-S”) you can prevent would-be hackers from dropping all your cached items, or finding out what the names are of the keys you are using. The memcached maintainers want to do this a different way, so this patch won’t be included in the base memecahced source tree.

The right long-term solution is to build multi-tenant features directly into memcached. I’m aware that Dustin Sallings at NorthScale has started some work of this sort, and has a working proof of concept. It’s not yet mature, and is generally incompatible with the current release of memcached, so it’s not yet suitable for production use. The main idea is that a TCP/IP connection to memcached could be authenticated with SASL, and limited to it’s own view of what’s inside memcahced.

My patch does not change how memcahced works, except for what it does when you enter the commands that I’m disabling. It will be just as stable as memcached 1.4.0 without the patch. The only difference is that you won’t have the ‘flush_all’ command, and you won’t have access to detailed stats either.

If you want to flush your entire cache, simply reconfigure your application to begin using a new “secret” key prefix, and you’ll have the functional equivalent of a flush_all because none of the prior cached data will be accessed by your application any more. The old data will simply expire or LRU out of the cache and be replaced by new data naturally.

By using a simple “secret” text prefix to all your keys, you will ensure that hackers won’t know how to access your data in the cache.  Consider prepending a reasonably long test string to the beginning of every key you store and access. Don’t make it too long, or that will multiply the number of packets required to get the data in and out of the cache, but something long enough that it won’t be easily guessed.

This patch does not make memcached bulletproof. An attacker can still do a bunch of SET commands to fill your cache with junk, and force your hot content out. They can still irritate it with a bunch of ‘stats sizes’ commands in a loop, or try to guess your secret prefix by randomly generating keys as a brute force attack until they find your content. For these reasons, you should only use this for storing data that’s not mission critical. There’s lots of data in this category that could really speed up your system under high load if you stored it in memcahed, but is not particularly sensitive to tampering.

Some have argued that this sort of a patch offers a false sense of security. I completely agree. Only use this if you know that your memcached installation will still not be secure, and that the security weakness could be exploited to ultimately hack your application. It will just be a little bit less insecure than it is without the patch.

I have seen memcached used in situations where only statistics are stored and accessed in memcached (instead of generating log files, statistical counters are stored in the cache). The application can do strict checking of the data it gets back from the cache, and not use it in any way that could lead to a security compromise. For example, make sure that all values returned are only numeric, and within acceptable value boundaries. An application of this sort would be appropriate with this patch.

I was thinking of making a better version of this patch that would allow you to specify an IP address (potentially 127.0.0.1 for example) that would have access to all commands that you define in a restricted access class. This way you could configure what IP address(es) could access what commands. Implementing this will require slowing memcached down a bit for all commands. I’ll plan to join forces with the others who are also interested in memcached multi-tenant features and produce a suitable solution that allows for secure deployments in insecure networks.

Cloud Mobile lets you manage your files stored on Cloud Files from the palm of your hand using your iPhone. It’s exciting to see software companies like Proactive Apps, LLC using Rackspace’s public REST API’s to build useful tools like this one. Founder of Proactive Apps, Marc Jones, quotes:

“We made the decision to support The Rackspace Cloud, and specifically Cloud Files, in the first version of Cloud Mobile based on their open, easy-to-use API and their support of open cloud standards. The combination of Cloud Files strong feature set and the open API allowed us to create an iPhone app that leveraged both platforms.”

Check out the Features:

Features

• Supports multiple Rackspace Cloud accounts
• Container count and disk space for account
• List containers
• Icons designate container status: empty, not empty, CDN status
• Create new containers
• Delete containers
• View container details: object count, disk space, and CDN info
• CDN enable or disable containers; set TTLs; view CDN URL
• Upload photos and movies (direct capture / library) from your iPhone to a container
• List objects in a container
• Delete objects
• View object details: size, etag, content type, last modified
• View / add / edit / delete object metadata
• View or play objects supported on the iPhone: images, audio, video
• Built-in links to Cloud Files and Rackspace Cloud status pages

Screenshots

“What’s so cool?” – The coolest things you can do with this app are:

1. Take a photo or video on your iPhone, upload it to Cloud Files, enable the CDN feature, and share the link.

2. Once you have photos, audio, and video on Cloud Files, you can stream them directly to your iPhone and view them in the native player at the click of a button.

3. This is the easiest possible way to change the TTL of a CDN enabled item we’ve seen so far.

Marc Jones:

“…Snap a pic or grab some video, upload it to Cloud Files, CDN enable your container with one touch, and share the link – it’s that easy.”

A future revision of this app will include support for management of other cloud services from The Rackspace Cloud including Cloud Servers. This app might just be the perfect companion for your iPhone.