CAMP is an open standards effort in OASIS that is expected to enter public review in the fall of 2013. I serve as one of the members of the editing team for the CAMP specification, and have presented it at a few different events. The thing that excites me about this standard is that it offers a practical path to portability of cloud applications between public and private clouds, and even between different public clouds. See the presentation for details about what this standard is, and what it’s not.

The first stage of software maturity is Development. This is the point where the software works for the first time, and has unknown quality. Not much testing has happened yet, and you should not be surprised to find bugs, or even see the program fail completely.

The next stage is Experimental, or Proof-of-Concept. This is the point where the software works, and all of the unit tests pass. The quality of the software depends on the quality and code coverage of the unit tests. No documentation exists yet.

The next stage is Alpha. This means that the software is feature complete and ready for people to try using it to verify that it does what it’s supposed to. It is ready for a limited “Preview Release” of the software to a list of users who will try it and produce feedback. At this stage documentation is limited, and is usually the version written by the software developers themselves.

Next comes Beta. This means that the software developers feel that the software is done, and that there are no more significant changes to make. They identify a version called a “Release Candidate” It’s ready for a large population of public users to run the software on a wide variety of situations and setups. Some documentation may be provided, but it usually still a work-in-progress by this point. This is really the last chance to catch any serious bugs before the software is labeled “Stable”. Beta quality software is usually provided for a limited time free of charge.

Finally comes Stable. This is when a version is considered a “Release” or GA. This means that the software has performed well with no major issues for a considerable user population over a reasonable length of time. There is complete documentation for the software, and for the support teams responsible for it. Every product development group has different standards for what Stable means, and what quality a Release needs to be. Some are better than others. People are expected to pay for Stable software and services.

Now, here is something that a lot of companies do. When they have a solution that is Experimental or Alpha quality, and they don’t have the resources to test it completely themselves, they do a form of a release that is inappropriately named a “Private Beta”. This is a misleading oxymoron of a name that really means “the software has not been completely tested yet, but it passes unit tests.” The correct term for the release of something at this quality level is Preview Release which indicates that the software may not yet have achieved Stable quality.

If you are a software development manager, or en entrepreneur who produces software, please remember this advice: There is no such thing as a Private Beta. If it needs to be Private or Limited, then it’s not Beta quality yet. If it’s Beta, then it’s ready for unlimited use by the public. If your software is still Experimental or Alpha quality at the time you want to release it, then call it a “Preview Release” to be truthful about its quality.

You might have heard news recently about the open source Linux operating system being used to control state of the art spacecraft and satellites. A new Linux driven satellite is being sent up soon that will have an Android phone on it. There is certainly an appeal to the open source nature of Android and its apps. Do you think the phone will hold up in the harsh environment of space?

I don’t like HTTP Basic Auth for a number of different reasons, and have avoided it for many years. When asked by a colleague today why I don’t like it, I began thinking of all the reasons why. Here is the full list:

1) HTTP Basic is less secure than a token based system because the username and password credentials are carried by every single request between the client and the server. Username and password credentials do not typically expire, and if they do, it’s not very often. Using a token allows the token to be regularly expired (typically in a number of minutes), and the actual account credentials are only ever transmitted between the client and the authentication service to acquire the valid token to start with, and upon expiration.

2) HTTP Basic is typically protected by routing all content over SSL/TLS protocols. Although SSL/TLS is assumed to be secure, it is possible to defeat their security, and it’s better not to depend on it for every request.

3) The effective attack surface for a system that supports HTTP Basic is much bigger than one that uses tokens issued from a central auth service.

• With HTTP Basic Auth, each web server needs to be able to validate usernames and passwords. This means that it must have a copy of the password hashes, or that it must have access to a remote system that can validate them. An attacker need only compromise any one of the service endpoints in order to intercept the credentials and potentially re-exploit a given account even after a significant period of time has elapsed.
• The amount of server-side source code that could contain bugs that are vulnerable to security exploits will continue to grow over time as the system grows and new features are added, whereas if the auth service is standardized and centralized, it can be more easily protected, is less subject to change, and does not need to be scaled out to the same extent.

4) For performance reasons, security credentials may need to be cached on the servers that handle high volumes of requests. This raises another set of concerns about keeping the cached data secure, and adds complexity to operational tasks such as the revocation of access to an account. If account information is widely cached, it may be complicated to remove it everywhere.

5) Systems that use token based authentication could offer a service over HTTP that does not use SSL/TLS for high performance use cases where the overhead of encryption is not acceptable. Special encrypted read-only tokens could be issued for this purpose. Such tokens could be carried in web browsers using cookies. With HTTP Basic, special read-only accounts would be needed to produce the equivalent. Automated password expiration in that case would be awkward, and usually impractical.

6) For web use cases, there is no good way to allow a web user to “log out” of an HTTP Basic Auth session. Although there are some convoluted and stupid workarounds, the only reliable method is to ask users to quit their web browser in order to log out. Yuck!

If there is any doubt, please don’t use HTTP Basic Auth. I’m asking you nicely.

The workloads shown are:

• Idle: Power consumption after 10 minutes of system idle
• Read: The system streaming an HD movie file
• Database: While running a database benchmark (mixed read/write)
• Max Write: Maximum Write Throughput of the drive

I already knew that SSD drives used less power than rotational drives when idle/reading, but what surprised me is how much power savings they offer when under mixed and heavy write workloads. I think that a typical server would have about 50% idle time, and about 50% mixed use in a typical production workload. If that assumption is true, then the power consumption comparison would look something like this:

This tells us that SSD drives are 5.4 times more energy efficient than ordinary SATA drives. There is a 5.24 Watt difference in power consumption between them. The more idle your servers are, the more dramatic this difference will be. Most enterprise workloads are surprisingly idle, so this should actually be a conservative picture of the actual difference. This convinced me that today’s SSD drives do in fact use a lot less power than SATA drives. The SSD drives also cost a lot more. There have been plenty of studies on the relative performance of SSD versus SATA for various workloads, and there are clear performance advantages to using them for a variety of reasons. Rather than look at that some more, I decided to pursue the answer to a question that your CFO would like to know:

Can the additional cost of SSD drives be offset by energy savings alone?

To judge the cost savings, imagine you have a room full of servers. Let’s say you have 500 servers running in a single data center, and you want to remove all SATA drives and replace them with SSD drives to conserve power, and allow you to maybe pack in some more servers before you build a new data center. Let’s see how this would work. If you assume that each sever consumes 250 Watts of power, and that each one has 4 hard drives installed in it on average, then perhaps you could save 5.24 x 4 = 20.96 Watts for each server or 20,960 Watts for your data center.

Electricity costs vary depending on where you buy it, and how much you buy. In Los Angeles, prices are about $0.20 per Kilowatt Hour. If you have a data center that pays a lot less, then the savings will be less dramatic for you. Your servers run 24 x 7 x 365, so the total annual electricity to buy for our example data center is:

250 Watts x 24 x 365 x 500 Servers = 1,095,000,000 / 1000 = 1,095,000 Kilowatt Hours

1,095,000 Kilowatt Hours @ $0.20 each = $219,000 annually

It my actually cost more than this to both power and cool the equipment, but the savings ratios are the same regardless. The costs are directly proportional to the power consumption.

Your SSD hard drives in this scenario would save you about:

20.96 Watts x 24 x 365 x 500 Servers / 1000 = 91,805 Kilowatt Hours

91,805 Kilowatt Hours @ $0.20 = $18,361 annually

If your hard drives last you five years, then that’s about $127,195 in electricity cost savings over the life of the equipment. If you picked SSD hard drives that cost about $250 each, then you would have spent:

500 Servers x 4 SSD Drives x $250 = $500,000

Guess what… the savings over 5 years by using SSD drives exclusively instead of SATA does not justify the cost of the drives to begin with. Perhaps if you could source SSD drives at $62 each, then you might have a compelling reason to switch to SSD technology for power savings. When you buy power in large volumes, it can cost less, so that can widen the price gap further. I suppose this is why we don’s see SATA drives piling up in dumpsters across the country. You can perform your own analysis for your particular situation.

Data centers are expensive to build. The most expensive parts of them are the power and cooling infrastructure. If your data center is “full” because your power/cooling capacity is maxed out, you can swap out your SATA hard drives for SSD drives. You may reclaim about 8% of your power capacity allowing you to add more servers. Less power consumption means less heat produced, so it’s like getting a bigger data center. In my example above, there would be enough power capacity freed up to allow 40 more servers to be installed at the original power consumption level. The additional capacity afforded to you by using SSD drives may be worth a small fortune to you.

There are at least three good reasons for using SSD drives in servers. I suggest giving it some serious consideration.

Reasons why NOT to switch to SSD drives

1. If you need a lot of storage. The cost per GB of the SSD storage is considerably higher than the cost per GB of SATA storage, even considering the performance and power savings.
2. If you are constantly writing to the hard drives over and over. SSD drives to have a limited duty cycle, and in general may be less durable than regular hard drives. Eventually they do wear out, just for different reasons than drives with moving parts. However, most of the drives that are on the market today are rated for MTBF durability that’s comparable to what traditional hard drives offer.
3. You run your data center on solar power (yeah, sure you do). Seriously, if your cost for power is dirt cheap, and you need a lot of storage, then regular hard drives may be a better value for you.

The talk appears to be getting good ratings, so I may give similar presentations to other groups, or speak more about related topics with UUASC. If you attended this talk and want to make suggestions for topics to consider for the next one, please comment below.

1. Use XenCenter to add a new disk to the existing VM, making it the size you want to srink the current server to.
• Select the VM from the list on the right, click Storage.
• Click “Add…” and create the new volume the size you want it.
2. Log into the VM, (you can use the Console tab in XenCenter) and start ‘Disk Management’ and Initialize the new disk.
• You can find it in Start->Administrative Tools->Computer Management.
• Click on the new disk. Mine showed up as “Disk 1, Unknown… Not Initialized”.
• If you click on the words “Not Initialized” it will be selected.
• Next select Action->All Tasks->Initialize Disk. Select MBR and OK.
3. Format a new partition on your new volume (you might be able to skip this step, did not try).
• Right click on the black “Unallocated” partition.
• Select “New Simple Volume”.
• Click Next twice, the size will be defaulted to the full disk, and specify a new drive letter.
• In my example, I use E:\.
• Format as NTFS using Quick Format, and click Finish.
• Once you see formatting finish, and the new partition turns blue and is marked “Healthy” (Primary Partition). Proceed to the next step.
4. In the VM, download and run XenConvert.
• I used version 2.4.1.
• It requires Microsoft .NET v4.0, so you may need to download that from Microsoft and install it before running XenConvert.
5. Start XenConvert and select From: Volume and To: Volume.
• Set the Source Volume to your boot drive (C:).
• Set the Destination Volume to your new drive (E:).
• Say “Yes” to the warning about losing free space.
• Click Convert, and accept the warning about erasing data on your Destination volume (E:)
• Go have a coffee, or something, and come back later.
6. When XenConvert is finished, use “Disk Management” again in the VM to activate the new partition.
• Right Click the new partition and select “Mark Partition As Active”.
7. Shut down the VM.
• I did this from inside the VM using Start->Shut down.
8. Now re-order the drives on the VM so the new drive is in position 0.
• In XenCenter, select the VM, and select the Storage tab.
• Detach the original drive by selecting it and clicking the “Detach” button.
• Select the new drive, and click Properties, and set the Position to 0.
9. Now you can boot the VM and voila! It’s now srunken to the size you wanted.
• If it all works the way you want it you can go delete the original drive in XenCenter to reclaim the space.

Most simple web clusters rely on a single database sever that all the application servers connect to. This way, all of the application servers have concurrent access to the same data. This can be problematic in the elastic use case when workloads increase, and more servers are added to the cluster. If the work is bottle-necked on storing or accessing data in the database server, adding additional application servers will not help. It will actually make the problem worse.

I spoke on a panel at Zendcon yesterday, which was covered in an Infoworld article where my remarks were published. The article says:

Panelists also debated use of SQL and database connectivity in clouds. SQL as a design pattern for storage “is not ideal for cloud applications,” said Adrian Otto, senior technical strategist for Rackspace Cloud. Afterward, he described SQL issues as “typically the No. 1 bottleneck” to elasticity in the cloud. With elasticity, applications use more or fewer application servers based on demand. Otto recommended that developers who want elasticity should have a decentralized data model that scales horizontally. “SQL itself isn’t the problem. The problem is row-oriented data in an application,” which causes performance bottlenecks, said Otto.

The author Paul Krill did a good job here of accurately reporting my position on this subject. Data stored in databases are arranged in tables of rows and columns. A new piece of data adds a new row. Each row has multiple columns that separate fields of a single record of data in the table. The truth is that most web applications work very well with this data design pattern. Those should continue to use SQL databases with row oriented data. However, there are some applications where data may be arranged differently to make reading the data more efficient.

If you have a big table of data, and you want to pull out just a little bit of it using a query, the database server must determine the location of that data in the table by consulting the table’s index, and return the desired portion that matches the constraints given in the query. This makes the reading of data relatively expensive from a computational perspective. If data were instead arranged in lots of columns instead, it could be retrieved more efficiently, and the data could be more easily distributed over a larger number of servers yielding the horizontal scalability that cloud applications want. This works very well in cases where the number of reads are very high, but the data is not updated very frequently in proportion to the reads.

Let’s use a blog application as an example. Blog posts are written once, and maybe updated a few times, possibly once each time a comment is submitted. However, on a busy web site, a blog post may be read millions of times. If the posts were stored in a column oriented storage system like Cassandra, they could be quickly and easily retrieved using the id number of the blog post. The listing of recent blog posts can also be arranged in a column so that the front page of the blog site with the listing of the articles can be generated. Using this approach requires that the data be properly arranged as it’s stored, putting the computational burden on the (infrequent) write rather than on the (frequent) read.

Using a distributed system to store data in columns allows the data to be evenly distributed over an arbitrary number of servers, eliminating the central data bottleneck. Adding more servers in the correct proportion of application servers and storage servers can result in true horizontal scalability, meaning that the capacity increases as a direct proportion of how many servers are in the cluster.

Why doesn’t everyone do this already? For some good reasons:

1. The concept of running applications in clouds is still relatively new. The related technology is still maturing.
2. Existing software tends to use SQL already. If you want to use an existing CMS platform, chances are it will require a central SQL database.
3. Most heavy-read workloads can be scaled well using data caching techniques. If applications don’t write data very often, it may not be necessary to scale beyond a single database server.
4. You must anticipate exactly how the application will use the data, and arrange it just right.
5. It may be harder to analyze the data. Once your data is arranged in a column store, you may not be able to query it in arbitrary ways. You may only be able to pull it out using it’s id numbers, or by systematically scanning all of it to find the parts you want.
6. Distributed data storage (aka: NoSQL) systems like Cassandra, Hbase, Redis, etc. are complicated, and there is a considerable learning curve associated with setting them up and maintaining them. In some cases these systems are not as good in terms of data durability or data consistency as the prevailing SQL database systems. These tradeoffs can be difficult to navigate.
7. Today’s software developers are very familiar with SQL as a data storage and access paradigm. They can very quickly develop software that relies on the ACID qualities of a SQL database.

If you have an application that you want to deploy into a cloud, and you want it to be very elastic, you should think about the subject of how you arrange your data. If you use a centralized data design, you will probably have scalability bottlenecks when you add lots of servers. You should aim to decentralize the data in a way that you can easily add more servers to horizontally scale the environment, and not stumble on the limits of the database server. This is particularly important in situations where you need the application to write a lot of data, and a cache is not a suitable solution for you.

Over time, the reasons why not to use column oriented data will begin to shrink, and better tools and services will make it easier to do. Until then, I suggest that you carefully consider if you need maximum elasticity. If not, then it’s perfectly appropriate to keep using the same centralized row-oriented data paradigm. Use a cache like memcached in cases where you have heavy reads, and when it’s acceptable to show slightly outdated information to readers. The truth is that traditional solutions work really well for most web applications. However, if you have one of the more unique situations where you need true horizontal scalability, take a good look at a different arrangement for your data, and the systems and tools to make that possible for you in the cloud.

The more I found the less I liked any of them. Some of them actually had bugs or typos and did not work at all, and most of them would incorrectly validate a credit card number that was all zeros.

I wrote my own that I’m pretty happy with. It’s a good deal more efficient that most that I found. It does not repeat the same math on the same figures like some of them out there do.

The function contains 7 lines of code. Can you make this function better without making it harder to read and understand? Please let me know.

Over the years I’ve administered email systems that provided service to thousands of end user’s mailboxes. In the early years in the 1990’s most woes of a mail system administrator were about how to instrument the setting up of email accounts and related client settings, and changing passwords when they were forgotten by end users.

As the internet became more and more commercialized, spam exploded in our face. Everyone hates spam. Mail administrators hate it with a passion. They are doing everything they can to try and fight it… they filter, they black-hole, they tattle to abuse@whatever.com about it. Sometimes their own users send spam, and they get black-holed and need to jump through hoops to undo the damage.

At the time I reached my breaking point I managed email for about a dozen domain names, probably about two hundred mailboxes in total. I hated it. I hated every waking moment of it. The RBL’s that worked one day did not work the next. I’m convinced that e-mail system administration is the nastiest dirtiest job there is for a sysadmin.

People kept suggesting to me that I outsource email, which I shrugged off. I had problems with outsourcing:

1) I’m Paranoid about Uptime.

It’s hard for me to trust other people, let alone trust a company. And trusting a company with something as important as my email??? No way. I’m a control freak, and I was going to keep control at all costs. Yes, I hated email system administration. I wasn’t even a sysadmin any more, but I still did it just so that I could control it. It needed to be highly available. I simply could not trust anyone to do it better than me.

2) I’m Paranoid about Security.

Although email is inherently an insecure communication mechanism, all sorts of highly sensitive information is in there anyway. What would happen if a competitor would somehow get control of our email and read it. They could learn all of our secrets. No way, I’m keeping control of the security so that I know it’s locked down as much as humanly possible.

3) I’m Paranoid about Reliability and Control.

If something goes wrong, I want to be able to fix it quick. If I host it, I have full control of everything in the system. I can find what’s wrong and fix it fast. I’m really good at that.

I became a source code contributor for an open source email filtering system called bogofilter that uses Bayes filters to learn what’s spam and what’s not and filter based on that. I thought my spam filtering setup was the bomb! It worked great!

I got busier and busier with my work. I administered my email systems less and less. The better they worked, the less I would work on them because I had other fish to fry. The spammers got smarter and smarter, and soon enough my super cool spam filtering setup was becoming less and less effective.

So in 2006 something happened. I got super frustrated with spam administration. I was tired of having to keep finding or inventing better mouse traps to trap that nasty spam. So I thought to myself… There is an unlimited desire to send spam. Why? Because it works. If it did not work, the spammers would not be so determined to keep doing it. They are doing everything they can to outsmart you to get mail in your inbox. They keep getting smarter and smarter.

I thought some more… It’s like viruses. The hackers keep making better viruses, and the virus scanner software companies keep making their virus scanners better to clean them up and block them out. I needed something like virus scan, but for my email. I thought about all the technical ways to do it. I started hunting the web to find answers. I just wanted SOMEONE… anyone to handle this spam nonsense for me.

In the process, I stumbled across a company called “Webmail.us” (Later acquired by Rackpace and now called “Rackspace Email”). They had a great web site, said (at the time) they had 700,000 mailboxes in service. They had a complete spam filtering solution built in. The mailbox hosting was cheap. So cheap I could not ignore it. They were charging less for complete hosting of mailboxes than I was willing to pay for outsourced spam filtering.

In 2006 I did an experiment. I put my own domain name where I get my home email on webmail.us to see how it worked. I told myself that if it worked really well that I might switch all my email over to it, and wash my hands of email sysadmin work and all the spam nonsense that goes along with it. I did it for a month. It worked great. It was fast, it never went down. I got no spam. I was thrilled!

I did the unthinkable. I outsourced my email!

One by one I migrated all of my domains, and all my mail users over to the hosted system. I have never looked back. The system has been rock solid. The few problems I’ve seen over the past three years have been really minor, and solved more quickly than I would have been able to solve using my own systems. I had been converted.

I was so happy to finally be free of all the nuisance of administering email and spam filtering systems. It was great. Years later I ended up working with Rackspace, and told them the story of how I used and loved the email platform. I later met the people behind the system, and it was no wonder that it works as well as it does.

If you are still administering your own email… especially if you are running an Exchange system in your own office building. You need to take a serious look in the mirror and ask yourself why you are not outsourcing it to Rackspace Email. The truth is:

1) It’s more expensive to host it internally. Run the numbers.
2) Your uptime it a lot worse. Measure it.
3) Your security is no stronger. Audit it.
4) You are paranoid, just like me. Yes, you are.

You trust your bank with your money. You trust your phone company not to spy on all your phone calls. You do this stuff without worrying about it. These things are much bigger leaps of trust than outsourcing your email.

From me to you… do yourself a favor. Run the same experiment I did. You’ll be delighted. I work for Rackspace now, so my view is corrupt, right? Don’t take my word for it, because you’re paranoid. Just try it and see.