Most simple web clusters rely on a single database sever that all the application servers connect to. This way, all of the application servers have concurrent access to the same data. This can be problematic in the elastic use case when workloads increase, and more servers are added to the cluster. If the work is bottle-necked on storing or accessing data in the database server, adding additional application servers will not help. It will actually make the problem worse.

I spoke on a panel at Zendcon yesterday, which was covered in an Infoworld article where my remarks were published. The article says:

Panelists also debated use of SQL and database connectivity in clouds. SQL as a design pattern for storage “is not ideal for cloud applications,” said Adrian Otto, senior technical strategist for Rackspace Cloud. Afterward, he described SQL issues as “typically the No. 1 bottleneck” to elasticity in the cloud. With elasticity, applications use more or fewer application servers based on demand. Otto recommended that developers who want elasticity should have a decentralized data model that scales horizontally. “SQL itself isn’t the problem. The problem is row-oriented data in an application,” which causes performance bottlenecks, said Otto.

The author Paul Krill did a good job here of accurately reporting my position on this subject. Data stored in databases are arranged in tables of rows and columns. A new piece of data adds a new row. Each row has multiple columns that separate fields of a single record of data in the table. The truth is that most web applications work very well with this data design pattern. Those should continue to use SQL databases with row oriented data. However, there are some applications where data may be arranged differently to make reading the data more efficient.

If you have a big table of data, and you want to pull out just a little bit of it using a query, the database server must determine the location of that data in the table by consulting the table’s index, and return the desired portion that matches the constraints given in the query. This makes the reading of data relatively expensive from a computational perspective. If data were instead arranged in lots of columns instead, it could be retrieved more efficiently, and the data could be more easily distributed over a larger number of servers yielding the horizontal scalability that cloud applications want. This works very well in cases where the number of reads are very high, but the data is not updated very frequently in proportion to the reads.

Let’s use a blog application as an example. Blog posts are written once, and maybe updated a few times, possibly once each time a comment is submitted. However, on a busy web site, a blog post may be read millions of times. If the posts were stored in a column oriented storage system like Cassandra, they could be quickly and easily retrieved using the id number of the blog post. The listing of recent blog posts can also be arranged in a column so that the front page of the blog site with the listing of the articles can be generated. Using this approach requires that the data be properly arranged as it’s stored, putting the computational burden on the (infrequent) write rather than on the (frequent) read.

Using a distributed system to store data in columns allows the data to be evenly distributed over an arbitrary number of servers, eliminating the central data bottleneck. Adding more servers in the correct proportion of application servers and storage servers can result in true horizontal scalability, meaning that the capacity increases as a direct proportion of how many servers are in the cluster.

Why doesn’t everyone do this already? For some good reasons:

1. The concept of running applications in clouds is still relatively new. The related technology is still maturing.
2. Existing software tends to use SQL already. If you want to use an existing CMS platform, chances are it will require a central SQL database.
3. Most heavy-read workloads can be scaled well using data caching techniques. If applications don’t write data very often, it may not be necessary to scale beyond a single database server.
4. You must anticipate exactly how the application will use the data, and arrange it just right.
5. It may be harder to analyze the data. Once your data is arranged in a column store, you may not be able to query it in arbitrary ways. You may only be able to pull it out using it’s id numbers, or by systematically scanning all of it to find the parts you want.
6. Distributed data storage (aka: NoSQL) systems like Cassandra, Hbase, Redis, etc. are complicated, and there is a considerable learning curve associated with setting them up and maintaining them. In some cases these systems are not as good in terms of data durability or data consistency as the prevailing SQL database systems. These tradeoffs can be difficult to navigate.
7. Today’s software developers are very familiar with SQL as a data storage and access paradigm. They can very quickly develop software that relies on the ACID qualities of a SQL database.

If you have an application that you want to deploy into a cloud, and you want it to be very elastic, you should think about the subject of how you arrange your data. If you use a centralized data design, you will probably have scalability bottlenecks when you add lots of servers. You should aim to decentralize the data in a way that you can easily add more servers to horizontally scale the environment, and not stumble on the limits of the database server. This is particularly important in situations where you need the application to write a lot of data, and a cache is not a suitable solution for you.

Over time, the reasons why not to use column oriented data will begin to shrink, and better tools and services will make it easier to do. Until then, I suggest that you carefully consider if you need maximum elasticity. If not, then it’s perfectly appropriate to keep using the same centralized row-oriented data paradigm. Use a cache like memcached in cases where you have heavy reads, and when it’s acceptable to show slightly outdated information to readers. The truth is that traditional solutions work really well for most web applications. However, if you have one of the more unique situations where you need true horizontal scalability, take a good look at a different arrangement for your data, and the systems and tools to make that possible for you in the cloud.

The more I found the less I liked any of them. Some of them actually had bugs or typos and did not work at all, and most of them would incorrectly validate a credit card number that was all zeros.

I wrote my own that I’m pretty happy with. It’s a good deal more efficient that most that I found. It does not repeat the same math on the same figures like some of them out there do.

The function contains 7 lines of code. Can you make this function better without making it harder to read and understand? Please let me know.

Over the years I’ve administered email systems that provided service to thousands of end user’s mailboxes. In the early years in the 1990’s most woes of a mail system administrator were about how to instrument the setting up of email accounts and related client settings, and changing passwords when they were forgotten by end users.

As the internet became more and more commercialized, spam exploded in our face. Everyone hates spam. Mail administrators hate it with a passion. They are doing everything they can to try and fight it… they filter, they black-hole, they tattle to abuse@whatever.com about it. Sometimes their own users send spam, and they get black-holed and need to jump through hoops to undo the damage.

At the time I reached my breaking point I managed email for about a dozen domain names, probably about two hundred mailboxes in total. I hated it. I hated every waking moment of it. The RBL’s that worked one day did not work the next. I’m convinced that e-mail system administration is the nastiest dirtiest job there is for a sysadmin.

People kept suggesting to me that I outsource email, which I shrugged off. I had problems with outsourcing:

1) I’m Paranoid about Uptime.

It’s hard for me to trust other people, let alone trust a company. And trusting a company with something as important as my email??? No way. I’m a control freak, and I was going to keep control at all costs. Yes, I hated email system administration. I wasn’t even a sysadmin any more, but I still did it just so that I could control it. It needed to be highly available. I simply could not trust anyone to do it better than me.

2) I’m Paranoid about Security.

Although email is inherently an insecure communication mechanism, all sorts of highly sensitive information is in there anyway. What would happen if a competitor would somehow get control of our email and read it. They could learn all of our secrets. No way, I’m keeping control of the security so that I know it’s locked down as much as humanly possible.

3) I’m Paranoid about Reliability and Control.

If something goes wrong, I want to be able to fix it quick. If I host it, I have full control of everything in the system. I can find what’s wrong and fix it fast. I’m really good at that.

I became a source code contributor for an open source email filtering system called bogofilter that uses Bayes filters to learn what’s spam and what’s not and filter based on that. I thought my spam filtering setup was the bomb! It worked great!

I got busier and busier with my work. I administered my email systems less and less. The better they worked, the less I would work on them because I had other fish to fry. The spammers got smarter and smarter, and soon enough my super cool spam filtering setup was becoming less and less effective.

So in 2006 something happened. I got super frustrated with spam administration. I was tired of having to keep finding or inventing better mouse traps to trap that nasty spam. So I thought to myself… There is an unlimited desire to send spam. Why? Because it works. If it did not work, the spammers would not be so determined to keep doing it. They are doing everything they can to outsmart you to get mail in your inbox. They keep getting smarter and smarter.

I thought some more… It’s like viruses. The hackers keep making better viruses, and the virus scanner software companies keep making their virus scanners better to clean them up and block them out. I needed something like virus scan, but for my email. I thought about all the technical ways to do it. I started hunting the web to find answers. I just wanted SOMEONE… anyone to handle this spam nonsense for me.

In the process, I stumbled across a company called “Webmail.us” (Later acquired by Rackpace and now called “Rackspace Email”). They had a great web site, said (at the time) they had 700,000 mailboxes in service. They had a complete spam filtering solution built in. The mailbox hosting was cheap. So cheap I could not ignore it. They were charging less for complete hosting of mailboxes than I was willing to pay for outsourced spam filtering.

In 2006 I did an experiment. I put my own domain name where I get my home email on webmail.us to see how it worked. I told myself that if it worked really well that I might switch all my email over to it, and wash my hands of email sysadmin work and all the spam nonsense that goes along with it. I did it for a month. It worked great. It was fast, it never went down. I got no spam. I was thrilled!

I did the unthinkable. I outsourced my email!

One by one I migrated all of my domains, and all my mail users over to the hosted system. I have never looked back. The system has been rock solid. The few problems I’ve seen over the past three years have been really minor, and solved more quickly than I would have been able to solve using my own systems. I had been converted.

I was so happy to finally be free of all the nuisance of administering email and spam filtering systems. It was great. Years later I ended up working with Rackspace, and told them the story of how I used and loved the email platform. I later met the people behind the system, and it was no wonder that it works as well as it does.

If you are still administering your own email… especially if you are running an Exchange system in your own office building. You need to take a serious look in the mirror and ask yourself why you are not outsourcing it to Rackspace Email. The truth is:

1) It’s more expensive to host it internally. Run the numbers.
2) Your uptime it a lot worse. Measure it.
3) Your security is no stronger. Audit it.
4) You are paranoid, just like me. Yes, you are.

You trust your bank with your money. You trust your phone company not to spy on all your phone calls. You do this stuff without worrying about it. These things are much bigger leaps of trust than outsourcing your email.

From me to you… do yourself a favor. Run the same experiment I did. You’ll be delighted. I work for Rackspace now, so my view is corrupt, right? Don’t take my word for it, because you’re paranoid. Just try it and see.

A system where software applications may be run in an environment composed of utility cloud services in a logically abstract environment.

Definition of PaaS:

Platform as a Service. A Cloud Platform offered by a service provider as a hosted service which facilitates the deployment of software applications without the cost and complexity of acquiring and managing the underlying hardware and software layers.

Examples of PaaS:

1. Google AppEngine
2. Force.com

The key drawback of most PaaS services that exist today is vendor lock-in. Implementing an application using a proprietary platform means that it becomes difficult or impossible to move a deployed application from one service provider to another, unless they have compatible Cloud Platforms

I plan to address this problem by embracing open source solutions that allow numerous service providers to host applications in a compatible PaaS model where applications can be easily moved with little or no modification required.

Keep an eye on the OpenStack project in the upcoming weeks and months for blueprints for a PaaS system for the whole world to use. I invite you to participate in the design and future implementation of the cloud platform that will end vendor lock-in concerns, and simplify application hosting for software developers.

This announcement is on the heels of another recent announcement that the public web delivery of files from the Cloud Files service now uses the world’s leading content distribution network from Akamai. This means that if you are looking for somewhere to host an extensive media or video archive, Cloud Files is definitely worth your consideration.

In economics we learn about the concept of Supply and Demand. Simply put, as something becomes more scarce, the price will adjust upward until an equilibrium is met between the supply and the demand. This holds true for situations where the pricing can be adjusted.

So, what happens when the pricing is fixed? You need some strategy to deal with the scarcity. A discipline known as queueing theory can describe this. Let’s think about a common example of a system where the demand (work) exceeds the supply (capacity). I will explain these in terms of system engineering, as they would pertain to a software system.

In the grocery store there are a finite number of checkout clerks. We will call the lines of waiting shoppers queues. If queue length increases, the system is experiencing congestion. When this happens the store manager calls in more workers (store employees) to help with checkouts. This capacity expansion process may continue until all checkstands are open, or until there are no more available workers to add. This condition is the maximum capacity of the system.  If we are at maximum capacity, we hope that the congestion is eliminated. If expanding the capacity did not alleviate the congestion, then the shoppers must wait. New shoppers entering the store may see the long queues, and decide to leave. Some people standing in line may abandon the line, and leave the store without completing their intended purchase.

Having multiple shorter queues gives shoppers the illusion that the congestion is lower than it actually is. Some checkout queues move more quickly than others, so having one queue per checkstand is not a fair system. A shopper may wait longer than they deserve because of what queue they select. Sometimes shoppers will jump from one queue to another if they feel they won’t have to wait as long.

Lesson One: Use a combined queue instead of multiple separate queues

When all shoppers have equal priority, it’s better to use a common queue that’s serviced by multiple workers. This way every shopper in the queue waits a fair share of the congestion backlog. Nobody is forced to wait longer because of queue selection, since there is only one. This also eliminates the inefficiency of changing between queues when one is slow.

Lesson Two: When using a priority queue, limit the number of priority waiters

This leads to the temptation to have multiple classifications of shared queues. If you want to have a sense of a premium service for select priority shoppers, you can create two queues. The priority queue is for your select priority shoppers, and the main queue is for everyone else. This works when the population of priority shoppers is smaller than the general population and they arrive in the line infrequently. In this situation, the next available worker can select a shopper to service from the priority queue before they service someone from the main queue. If there are too many priority shoppers, the main queue does not get serviced, which leads to further congestion, and eventually the waiters will abandon the main queue.

Lesson Three: Use Admission Control

Now, you are familiar with the example of using the store, and simple queues, and the concept of consolidating multiple queues into a single queue, or the combination of a single queue and a priority queue. These are generally easy concepts to implement in software, and for most cases where congestion is rare, they work fine. However there are some conditions where they don’t work at all. Remember that in software systems, queued jobs may or may not be able to abandon the queue. Here are examples of when a simple queue with no admission control will fail:

1. When you have the ability to create an unlimited number of workers, but they all rely on some shared pool or pools of limited resources. This is called a concurrency bottleneck. For example, when you have a finite amount of CPU resources, but the ability to create an effectively unlimited number of threads. There is an upper limit to the number of additional throughput gained by adding more threads when the CPU is congested.
2. When there is an unlimited number of jobs (waiters) to service, that arrive suddenly in bursts or steadily at a rate that’s faster than the combined capacity of all your workers. This is called a work overflow.

The concept of admission control allows you to implement a policy controlling how much work you will accept into your queue(s) and at what rate. One simple policy is a maximum queue length. This length (limit) can be chosen by considering your desired maximum wait time (max) and your average service time (t) using the formula:

limit = ( max / t )

When limit jobs are queued, you must not queue new work, but instead refuse the work. Some protocols have a response code that can be used for this. For example, in HTTP applications, you can return a 502 Server Busy response with an optional Retry-After header that indicates to the client that they may retry the request at the specified later time when the congestion may be gone.

A more advanced policy may use a strategy where the source of work and the type of request, and apply a different policy. For example, you may limit the rate at which you accept requests from a given IP Address or geography. You may prioritize write requests over read requests. If an individual resource is congested, you may reject requests for that resource, but accept other requests.

There are a number of advantages to using an admission control policy:

1. You don’t end up overloading your system to the point where service quality degrades for everyone due to extreme congestion conditions.
2. You can temporarily smooth out your work pattern so that the work can be completed at some later time when a temporary surge in demand has subsided.
3. It is very easy to monitor for congestion. A simple external check of your service will return an error during congestion conditions.
4. If the system has elastic capability to add more worker resources, this can be triggered when queue lengths are consistently non-zero over a span of time, and reduced when queue lengths are consistently zero over a span of time.

Lesson Four: Beware of the Thundering Herd and the Convoy Effect

If you have a workload that experiences sudden bursts where new work is rapidly queued, and you have not properly limited the number of workers to a level of concurrency that you can manage with your available system resources, then you may experience a condition known as resource starvation. This can lead to a complete system failure, typically in a cascading series of related failures. When your system is in a state of resource starvation, it is too busy handling multitudes of concurrent work requests that it’s unable to make meaningful progress servicing the work. Nothing completes in a reasonable time because you are too busy switching between all the requests making tiny increments of progress on each.

A Convoy Effect is where you have numerous threads or processes that are blocked (suspended making no progress) on some resource (like an I/O operation, or a lock) and then cause additional congestion when the blocking condition clears. At this point all the runnable threads are synchronized together in a convoy, requesting other scarce resources (like Disk I/O bandwidth to write a lock file) and that the existence of the convoy is causing the delays to be further delayed than if there were simply a long queue of work. For example:

1. A long queue of requests is present.
2. All workers dequeue the requests rapidly, but all of them need to read and write to the same file.
3. Concurrent reads are allowed with a shared read lock, and concurrent writes are protected by an exclusive write lock.
4. The underlying lock implementation uses a spinlock.
5. Multitudes of concurrent readers cause write access to the file to be substantially slowed down because the disk drive is chattering doing all the reads.
6. All workers become synchronized on the exclusive write lock, slowing down the rate of progress, and essentially causing the system to be in a state of live lock.

A Thundering Herd is a situation where a multitude of serialized processes are blocked waiting on an event. When the event happens, all processes become runnable, but only one of them can be serviced at a time, so all the others must become blocked again waiting on a new event. This condition causes throughput of work to be suboptimal because of the wasted effort of the blocked processed getting woken up and going back to sleep. This is generally solved by only waking one of the blocked processes at a time.

So, a Thundering Herd will typically lead to a Convoy Effect, leaving your system in a critical state of congestion. Solve this by using sensible admission control, and more sophisticated queuing strategies.

Lesson Five: Multiplex de-queue of identical requests

If you have a queue of requests that are likely to have multiple identical queued entries, you can optimize your service of that queue by using a Multiplexing technique. Instead of each worker simply taking one request off the queue and servicing it, consider this approach instead:

1. Read the first request on the queue.
2. Scan remaining entries in the queue for other identical requests, de-queuing them together as a batch. You might de-queue up to some maximum number per batch, or de-queue all matches in a single batch. This depends on how large your responses will be, and how many items you have in queue.
3. Process the request, sending the response to all clients in the batch at once.

Note: this is only appropriate in use cases where out-of-order responses are gracefully handled by the client.

You will need to be careful that your queue implementation will allow you to safely de-queue any entry in the queue without leading to a race condition where multiple workers are trying to de-queue the same entries simultaneously. You may decide that batching requests when they are queued up is more efficient than searching for matching requests when it’s time to de-queue. It would work in a similar way:

1. Treat all queued entries as batches, with at least one request in each, up to some maximum.
2. Hash the request as you receive it, and check to see if you have an existing entry or entries in the queue for the given request.
3. If have a matching entry in the queue, then temporarily lock that entry, and add the new request’s client details (or connection handle) to the existing entry.
4. If the batch is full, iterate to the next, creating a new batch when you reach the end of the list.
5. Workers simply read the next batch from the queue, and send the response to all the associated clients.

In general, you want your workers de-queueing work faster than you can queue it up. If you do your batching upon receipt of the request, you avoid the risk that the efficiency cost of the batching does not lead to a Convoy Effect.

Feedback welcome.

FDIC: Failed Bank List

The FDIC is often appointed as receiver for failed banks. This page contains useful information for the customers and vendors of these banks. This includes information on the acquiring bank (if applicable), how your accounts and loans are affected, and how vendors can file claims against the receivership. This list includes banks which have failed since October 1, 2000.

The data had a link to download the data in CSV format, so I decided to quickly plot it and take a look.

Bank Failures betwween 2000 and 2011

The interesting part of the data happens between 2006 and 2011. Because we are only part way through January 2011 at the time of this post, I excluded January’s data from the chart. Already two banks have failed in 2011.

Bank Failures between 2006 and 2011

As you can clearly see, we had a lot of volatility between Mid 2009 and today. The trend is clearly visible back through 2008 as well. Almost all of these bank closures resulted in an acquisition of the accounts by another bank. Of the 351 failures shown above, only 24 resulted in “No Acquirer”. Those were:

+--------------------------------+------------------+-------+------+
| bank_name                      | city             | state | year |
+--------------------------------+------------------+-------+------+
| NextBank, NA                   | Phoenix          | AZ    | 2002 |
| New Century Bank               | Shelby Township  | MI    | 2002 |
| AmTrade International Bank     | Atlanta          | GA    | 2002 |
| Bank of Alamo                  | Alamo            | TN    | 2002 |
| Dollar Savings Bank            | Newark           | NJ    | 2004 |
| MagnetBank                     | Salt Lake City   | UT    | 2009 |
| Omni National Bank             | Atlanta          | GA    | 2009 |
| FirstCity Bank                 | Stockbridge      | GA    | 2009 |
| First Bank of Beverly Hills    | Calabasas        | CA    | 2009 |
| New Frontier Bank              | Greeley          | CO    | 2009 |
| Silverton Bank, NA             | Atlanta          | GA    | 2009 |
| Community Bank of West Georgia | Villa Rica       | GA    | 2009 |
| Community Bank of Nevada       | Las Vegas        | NV    | 2009 |
| Platinum Community Bank        | Rolling Meadows  | IL    | 2009 |
| Citizens State Bank            | New Baltimore    | MI    | 2009 |
| RockBridge Commercial Bank     | Atlanta          | GA    | 2009 |
| Barnes Banking Company         | Kaysville        | UT    | 2010 |
| Advanta Bank Corp.             | Draper           | UT    | 2010 |
| Centennial Bank                | Ogden            | UT    | 2010 |
| Waterfield Bank                | Germantown       | MD    | 2010 |
| Lakeside Community Bank        | Sterling Heights | MI    | 2010 |
| Arcola Homestead Savings Bank  | Arcola           | IL    | 2010 |
| Ideal Federal Savings Bank     | Baltimore        | MD    | 2010 |
| First Arizona Savings, A FSB   | Scottsdale       | AZ    | 2010 |
+--------------------------------+------------------+-------+------+

According to a Wikipedia page about US Banking, this is from a total population of 8430 FDIC insured banks. The total failure rate is 4.13%, and the total that had ho acquiring institution was 0.3%. So, it’s not quite as shocking as it seems at first glance.

Now, I wonder to myself in today’s economic climate who’s banking with the likes of First Arizona Savings in Scottsdale, AZ. I suppose it does not matter too much I suppose, since your deposits are insured by the FDIC up to $250,000 and the FDIC will repo the bank if it’s at serious risk. So if you happen to have a huge bank balance (>$250K), consider finding some sensible investment strategy for that money. The risk of leaving it in a little community bank is pretty frightening.

Part of the reason that the code count is smaller is because the non-relevant parts have simply been removed, but there was also a lot of effort put into modernizing the code base, and using C++. Using Boost and other foundational software projects as building blocks, the need for internal implementations for basic things is greatly reduced. For example, the MySQL code base has a REGEX implementation in it. This was removed in Drizzle, and replaced by PCRE, which is expected to change again soon to use Boost. By using libraries that are leveraged by numerous projects, the probability of a lower bug count is an additional benefit. Simply put, with more smart people looking at and using the software, and improving its weaknesses, the more likely it is to be high quality.

Charter

• A database optimized for Cloud infrastructure and Web applications
• Design for massive concurrency on modern multi-cpu architecture
• Optimize memory for increased performance and parallelism
• Open source, open community, open design

Scope

• Re-designed modular architecture providing plugins with defined APIs
• Simple design for ease of use and administration
• Reliable, ACID transactional

There are many exciting changes, such as optimizing everything for 64-bit CPU’s and Multi-Core. You can’t hardly even buy 32-bit and Single Core servers nowadays if you want them. It makes no sense to have software that’s optimized for these antiquated hardware designs. No effort is spent optimizing software to work with rotational hard drives because SSD drives are the way of the future. All the language collations have simply been replaced with UTF-8 only, because the web uses UTF-8. Plus, this is tested with 41 different language translations. Drizzle has a new scheduler. The legacy MySQL scheduler was designed to work for a thread-per-session setup. In Drizzle, sessions are handled independently from the threads. The new scheduler allows this to work.

Drizzle uses InnoDB as its default storage engine, which is great for OLTP. It also supports the PBXT storage engine. There are available plugins for the InnoDB Embedded Engine and HailDB which will soon be the new default. DDL Operations (like ALTER TABLE) can actually roll back in the event that something goes wrong in the process, rather than leaving you with incomplete or corrupt data.

The code base in Drizzle has been fully modernized, and brought up to today’s standards of C++ with extensive use of the C++ STL to replace MySQL’s usage of obscure custom data type implementations that offered no real benefit compared to what the STL has today. Another example of improvements in this area is the replacement of the legacy REGEX implementation with a more standard library. All of these changes reduce the amount of Drizzle source code dramatically compared to MySQL. Less code and simpler code means less bugs, plain and simple. Drizzle is well on its way to being an ideal fit for web applications that need a reliable, and high performance transactional database.

Features in Drizzle7 Beta

• New micro kernel
• Migration Tool
• Instance Catalog Support
• Universal Replication
• User query analysis
• Mutli-core Support

What “Beta” means

• Your data is safe. Transactional engine by default and stable for over 2 years.
• Upgrade the system in-place without exporting/importing data.
• Replication is still being tested.

In Microsoft terms, it means that this project would have launched about a year ago. In Google terms, it probably would have launched six months ago. Simply put, if you trust your data to a MySQL system today running InnoDB, you should feel comfortable trying Drizzle. There have been some changes to the InnoDB setup, such as the elimination of the FRM files from disk which eliminate possible inconsistency between the state on disk and the state in InnoDB. I am in the process of moving a few of my produciton applications to use the Drizzle Beta. If you’re an accomplished system administrator and DBA, you should seriously consider putting at least one of your production applications on Drizzle now, and see how it works for you.

What’s Next?

• Beta announced today 2010-09-29.
• GA February 2011
• GA May 2011 for Multi-Tenancy features that allow an arbitrary number of logical databases (Schemas, Tables, etc.) to exist concurrently with full data isolation between them. This allows for individual security and resource controls (Threads, Memory, IO), and individual database backups, rather than system level backups. This feature will be called “Catalogs”.

Download Drizzle

Time to get started with the beta. Download the beta today!

About OpenStack Object Storage

OpenStack Object Storage was originally developed by Rackspace, and was released as Open Source Software earlier this year as part of the OpenStack Project. It was written for hosting the Rackspace Cloud Files service. It’s original project code name was swift, so you may see references to that in various documentation.

OpenStack Object Storage aggregates commodity servers to work together in clusters for reliable, redundant, and large-scale storage of static objects. Objects are written to multiple hardware devices in the datacenter, with the OpenStack software responsible for ensuring data replication and integrity across the cluster. Storage clusters can scale horizontally by adding new nodes, which are automatically configured. Should a node fail, OpenStack works to replicate its content from other active nodes. Because OpenStack uses software logic to ensure data replication and distribution across different devices, inexpensive commodity hard drives and servers can be used in lieu of more expensive equipment. [1]

The system uses a flat namespace, and has a concept an account (how you access the system),  a container (like a directory) and an object (like a file). You can have an arbitrary number accounts each with an arbitrary number of containers. Each container can hold an arbitrary number of objects.

OpenStack Object Storage is very good for is storing unstructured data using an object name as a lookup key (like a filename). You access your data from a web client using the web service REST API, not like a filesystem. Download an object (like a file) using an HTTP GET request, fetch object metadata with an HTTP HEAD request, delete an object with an HTTP DELETE request, etc. There are multiple language bindings so you can access your files in OpenStack Object Storage from your favorite language natively (Java, Python, Perl, PHP, .NET, etc.).

The system has no central point of failure, so it’s extremely fault tolerant, and the data and related metadata are distributed throughout the system, so there are no central scalability constraints. You can store arbitrary amounts of data in the system in both large and small sizes. It performs very well, even under very high levels of concurrency. It keeps multiple replicas of each object, so it’s reliable, and the storage is very durable, without any expensive hardware. You don’t need any RAID on any of the servers unless you want it for additional performance.

Use OpenStack Object Storage For…

Here are some good use cases for OpenStack Object Storage:

• Storing media libraries (photos, music, videos, etc.)
• Archiving video surveillance files
• Archiving phone call audio recordings
• Archiving compressed log files
• Archiving backups (<5GB each object)
• Storing and loading of OS Images, etc.
• Storing file populations that grow continuously on a practically infinite basis.
• Storing small files (<50 KB). OpenStack Object Storage is great at this.
• Storing billions of files.
• Storing Petabytes (millions of Gigabytes) of data.

Recognize the Limitations

Objects must be <5GB

This is an arbitrary size limit, but it can not be set to an unlimited value because of the system design. If you want to store a backup something larger than 5GB, you’ll need to have a way of breaking it up into chunks, and storing some manifest of the parts so you can later join them back together again when you want to download the data and use it again.

Not a Filesystem

Uses a REST API, or a language binding that consumes the REST API. It does not use the typical POSIX filesystem semantics like open(), read(), write(), seek(), and close().

No User Quotas

There are no maximums that can be configured on a per-user basis to limit how much storage is used.

No Directory Hierarchies

You can create an arbitrary number of containers, but there is no nested container capability. You can simulate a directory structure using creative object names, but this is limited to a maximum string length. If you only need a shallow hierarchy, or don’t have long directory names, this might be fine. Just remember that I warned you this is generally a bad idea.

No writing to a byte offset in a file

The only way to update a file is to essentially overwrite it. The system creates a new version of an object each time you upload one with the same name.

No ACL’s

Per-Container ACL’s will probably be added in a later release. Per-Object ACL’s will probably not be supported, but maybe.

No Append Support

It’s possible that this may be added at a later time using a versioning trick.

No File Locking

Most filesystems integrate with the kernel to offer advisory locking. This is not possible with OpenStack Object Storage.

Eventual Consistency

Don’t expect version consistency between multiple nodes when data is being updated.

If you upload a new version of an object, and immediately GET that object from another client, you may get a previous version of the file. There is no way to know which version of a given object the system is responding with, unless you set version metadata on each object yourself. If there is any problem with the network, you may get outdated versions of objects, or be able to see objects that were deleted, but the local node may not yet know are deleted.

No Support for Data Encryption

You must encrypt the data yourself. The current version does not have SSL support either. Use an SSL proxy to work around this by terminating the SSL sessions on the same network where the OpenStack Object Storage system runs.

Not Compatible With Web Browsers

You must supply a storage token header to authorize each request. Regular web browsers can’t do this. This can be solved using a proxy between the client and the system to handle token authentication. This is not a problem is you are using one of the language bindings. They will take care of this when you integrate your web app with the system.

Not a Database

It supports no querying or processing of data on the servers. All you can do is list the objects within a given container. There is no way to search based on object metadata. You need to keep your own external search indexes.

Don’t try to frequently update large objects.

All updates produce a new version of an object, because objects are immutable.

Don’t store unlimited objects per container

You can store as many objects in a container as you wish. However, your per-object upload latency will increase considerably one you reach a certain point. I found the optimal number of objects per container to be just under one million. This number will vary depending on your equipment, and how heavy of a workload it’s subjected to.

Changing Swift Into a Filesystem

You might think of using FUSE to access objects and containers in OpenStack Object Storage as files and directories with a filesystem interface, but you’ll quickly discover that this is only really good for very simple use cases. Most of the things you need to implement what we think of as a filesystem are missing.

If you are a developer, and you are thinking of building a filesystem on top of OpenStack Object Storage using objects as blocks, that could possibly work, but would probably not perform very well compared to existing alternatives that are actually designed for distributed block storage. The blocks would need to be pretty large to keep the network/protocol overhead down. Frequent writing is not likely to work well. Most users of filesystems are not expecting eventual consistency behavior. They want strong data consistency. You would also want some strategy to handle read/write concurrency with some locking capability. Plus, you would need to have a way to keep track of the blocks like a filesystem does in some data structure or database. Frankly speaking, OpenStack Object Storage is probably not the right tool for the job.

Conclusion

You should probably only use OpenStack Object Storage for use cases it’s intended for. If what you really want is a clustered filesystem, you’re probably better off looking at other solutions like Lustre, GlusterFS, GFS, OCFS, etc. Keep in mind that each of these have their own strengths and weaknesses. Pay particular attention to what they are designed for, and use them accordingly. If you want to use OpenStack Object Storage for something that it was designed for, then you will probably be very happy with it. Keep in mind that it’s a blob storage system. It’s not a filesystem, not a file server, not a database, etc. To learn more about OpenStack Object Storage, please check out the Developer Documentation.