1. Use XenCenter to add a new disk to the existing VM, making it the size you want to srink the current server to.
• Select the VM from the list on the right, click Storage.
• Click “Add…” and create the new volume the size you want it.
2. Log into the VM, (you can use the Console tab in XenCenter) and start ‘Disk Management’ and Initialize the new disk.
• You can find it in Start->Administrative Tools->Computer Management.
• Click on the new disk. Mine showed up as “Disk 1, Unknown… Not Initialized”.
• If you click on the words “Not Initialized” it will be selected.
• Next select Action->All Tasks->Initialize Disk. Select MBR and OK.
3. Format a new partition on your new volume (you might be able to skip this step, did not try).
• Right click on the black “Unallocated” partition.
• Select “New Simple Volume”.
• Click Next twice, the size will be defaulted to the full disk, and specify a new drive letter.
• In my example, I use E:\.
• Format as NTFS using Quick Format, and click Finish.
• Once you see formatting finish, and the new partition turns blue and is marked “Healthy” (Primary Partition). Proceed to the next step.
4. In the VM, download and run XenConvert.
• I used version 2.4.1.
• It requires Microsoft .NET v4.0, so you may need to download that from Microsoft and install it before running XenConvert.
5. Start XenConvert and select From: Volume and To: Volume.
• Set the Source Volume to your boot drive (C:).
• Set the Destination Volume to your new drive (E:).
• Say “Yes” to the warning about losing free space.
• Click Convert, and accept the warning about erasing data on your Destination volume (E:)
• Go have a coffee, or something, and come back later.
6. When XenConvert is finished, use “Disk Management” again in the VM to activate the new partition.
• Right Click the new partition and select “Mark Partition As Active”.
7. Shut down the VM.
• I did this from inside the VM using Start->Shut down.
8. Now re-order the drives on the VM so the new drive is in position 0.
• In XenCenter, select the VM, and select the Storage tab.
• Detach the original drive by selecting it and clicking the “Detach” button.
• Select the new drive, and click Properties, and set the Position to 0.
9. Now you can boot the VM and voila! It’s now srunken to the size you wanted.
• If it all works the way you want it you can go delete the original drive in XenCenter to reclaim the space.

In economics we learn about the concept of Supply and Demand. Simply put, as something becomes more scarce, the price will adjust upward until an equilibrium is met between the supply and the demand. This holds true for situations where the pricing can be adjusted.

So, what happens when the pricing is fixed? You need some strategy to deal with the scarcity. A discipline known as queueing theory can describe this. Let’s think about a common example of a system where the demand (work) exceeds the supply (capacity). I will explain these in terms of system engineering, as they would pertain to a software system.

In the grocery store there are a finite number of checkout clerks. We will call the lines of waiting shoppers queues. If queue length increases, the system is experiencing congestion. When this happens the store manager calls in more workers (store employees) to help with checkouts. This capacity expansion process may continue until all checkstands are open, or until there are no more available workers to add. This condition is the maximum capacity of the system.  If we are at maximum capacity, we hope that the congestion is eliminated. If expanding the capacity did not alleviate the congestion, then the shoppers must wait. New shoppers entering the store may see the long queues, and decide to leave. Some people standing in line may abandon the line, and leave the store without completing their intended purchase.

Having multiple shorter queues gives shoppers the illusion that the congestion is lower than it actually is. Some checkout queues move more quickly than others, so having one queue per checkstand is not a fair system. A shopper may wait longer than they deserve because of what queue they select. Sometimes shoppers will jump from one queue to another if they feel they won’t have to wait as long.

Lesson One: Use a combined queue instead of multiple separate queues

When all shoppers have equal priority, it’s better to use a common queue that’s serviced by multiple workers. This way every shopper in the queue waits a fair share of the congestion backlog. Nobody is forced to wait longer because of queue selection, since there is only one. This also eliminates the inefficiency of changing between queues when one is slow.

Lesson Two: When using a priority queue, limit the number of priority waiters

This leads to the temptation to have multiple classifications of shared queues. If you want to have a sense of a premium service for select priority shoppers, you can create two queues. The priority queue is for your select priority shoppers, and the main queue is for everyone else. This works when the population of priority shoppers is smaller than the general population and they arrive in the line infrequently. In this situation, the next available worker can select a shopper to service from the priority queue before they service someone from the main queue. If there are too many priority shoppers, the main queue does not get serviced, which leads to further congestion, and eventually the waiters will abandon the main queue.

Lesson Three: Use Admission Control

Now, you are familiar with the example of using the store, and simple queues, and the concept of consolidating multiple queues into a single queue, or the combination of a single queue and a priority queue. These are generally easy concepts to implement in software, and for most cases where congestion is rare, they work fine. However there are some conditions where they don’t work at all. Remember that in software systems, queued jobs may or may not be able to abandon the queue. Here are examples of when a simple queue with no admission control will fail:

1. When you have the ability to create an unlimited number of workers, but they all rely on some shared pool or pools of limited resources. This is called a concurrency bottleneck. For example, when you have a finite amount of CPU resources, but the ability to create an effectively unlimited number of threads. There is an upper limit to the number of additional throughput gained by adding more threads when the CPU is congested.
2. When there is an unlimited number of jobs (waiters) to service, that arrive suddenly in bursts or steadily at a rate that’s faster than the combined capacity of all your workers. This is called a work overflow.

The concept of admission control allows you to implement a policy controlling how much work you will accept into your queue(s) and at what rate. One simple policy is a maximum queue length. This length (limit) can be chosen by considering your desired maximum wait time (max) and your average service time (t) using the formula:

limit = ( max / t )

When limit jobs are queued, you must not queue new work, but instead refuse the work. Some protocols have a response code that can be used for this. For example, in HTTP applications, you can return a 502 Server Busy response with an optional Retry-After header that indicates to the client that they may retry the request at the specified later time when the congestion may be gone.

A more advanced policy may use a strategy where the source of work and the type of request, and apply a different policy. For example, you may limit the rate at which you accept requests from a given IP Address or geography. You may prioritize write requests over read requests. If an individual resource is congested, you may reject requests for that resource, but accept other requests.

There are a number of advantages to using an admission control policy:

1. You don’t end up overloading your system to the point where service quality degrades for everyone due to extreme congestion conditions.
2. You can temporarily smooth out your work pattern so that the work can be completed at some later time when a temporary surge in demand has subsided.
3. It is very easy to monitor for congestion. A simple external check of your service will return an error during congestion conditions.
4. If the system has elastic capability to add more worker resources, this can be triggered when queue lengths are consistently non-zero over a span of time, and reduced when queue lengths are consistently zero over a span of time.

Lesson Four: Beware of the Thundering Herd and the Convoy Effect

If you have a workload that experiences sudden bursts where new work is rapidly queued, and you have not properly limited the number of workers to a level of concurrency that you can manage with your available system resources, then you may experience a condition known as resource starvation. This can lead to a complete system failure, typically in a cascading series of related failures. When your system is in a state of resource starvation, it is too busy handling multitudes of concurrent work requests that it’s unable to make meaningful progress servicing the work. Nothing completes in a reasonable time because you are too busy switching between all the requests making tiny increments of progress on each.

A Convoy Effect is where you have numerous threads or processes that are blocked (suspended making no progress) on some resource (like an I/O operation, or a lock) and then cause additional congestion when the blocking condition clears. At this point all the runnable threads are synchronized together in a convoy, requesting other scarce resources (like Disk I/O bandwidth to write a lock file) and that the existence of the convoy is causing the delays to be further delayed than if there were simply a long queue of work. For example:

1. A long queue of requests is present.
2. All workers dequeue the requests rapidly, but all of them need to read and write to the same file.
3. Concurrent reads are allowed with a shared read lock, and concurrent writes are protected by an exclusive write lock.
4. The underlying lock implementation uses a spinlock.
5. Multitudes of concurrent readers cause write access to the file to be substantially slowed down because the disk drive is chattering doing all the reads.
6. All workers become synchronized on the exclusive write lock, slowing down the rate of progress, and essentially causing the system to be in a state of live lock.

A Thundering Herd is a situation where a multitude of serialized processes are blocked waiting on an event. When the event happens, all processes become runnable, but only one of them can be serviced at a time, so all the others must become blocked again waiting on a new event. This condition causes throughput of work to be suboptimal because of the wasted effort of the blocked processed getting woken up and going back to sleep. This is generally solved by only waking one of the blocked processes at a time.

So, a Thundering Herd will typically lead to a Convoy Effect, leaving your system in a critical state of congestion. Solve this by using sensible admission control, and more sophisticated queuing strategies.

Lesson Five: Multiplex de-queue of identical requests

If you have a queue of requests that are likely to have multiple identical queued entries, you can optimize your service of that queue by using a Multiplexing technique. Instead of each worker simply taking one request off the queue and servicing it, consider this approach instead:

1. Read the first request on the queue.
2. Scan remaining entries in the queue for other identical requests, de-queuing them together as a batch. You might de-queue up to some maximum number per batch, or de-queue all matches in a single batch. This depends on how large your responses will be, and how many items you have in queue.
3. Process the request, sending the response to all clients in the batch at once.

Note: this is only appropriate in use cases where out-of-order responses are gracefully handled by the client.

You will need to be careful that your queue implementation will allow you to safely de-queue any entry in the queue without leading to a race condition where multiple workers are trying to de-queue the same entries simultaneously. You may decide that batching requests when they are queued up is more efficient than searching for matching requests when it’s time to de-queue. It would work in a similar way:

1. Treat all queued entries as batches, with at least one request in each, up to some maximum.
2. Hash the request as you receive it, and check to see if you have an existing entry or entries in the queue for the given request.
3. If have a matching entry in the queue, then temporarily lock that entry, and add the new request’s client details (or connection handle) to the existing entry.
4. If the batch is full, iterate to the next, creating a new batch when you reach the end of the list.
5. Workers simply read the next batch from the queue, and send the response to all the associated clients.

In general, you want your workers de-queueing work faster than you can queue it up. If you do your batching upon receipt of the request, you avoid the risk that the efficiency cost of the batching does not lead to a Convoy Effect.

Feedback welcome.

Part of the reason that the code count is smaller is because the non-relevant parts have simply been removed, but there was also a lot of effort put into modernizing the code base, and using C++. Using Boost and other foundational software projects as building blocks, the need for internal implementations for basic things is greatly reduced. For example, the MySQL code base has a REGEX implementation in it. This was removed in Drizzle, and replaced by PCRE, which is expected to change again soon to use Boost. By using libraries that are leveraged by numerous projects, the probability of a lower bug count is an additional benefit. Simply put, with more smart people looking at and using the software, and improving its weaknesses, the more likely it is to be high quality.

Charter

• A database optimized for Cloud infrastructure and Web applications
• Design for massive concurrency on modern multi-cpu architecture
• Optimize memory for increased performance and parallelism
• Open source, open community, open design

Scope

• Re-designed modular architecture providing plugins with defined APIs
• Simple design for ease of use and administration
• Reliable, ACID transactional

There are many exciting changes, such as optimizing everything for 64-bit CPU’s and Multi-Core. You can’t hardly even buy 32-bit and Single Core servers nowadays if you want them. It makes no sense to have software that’s optimized for these antiquated hardware designs. No effort is spent optimizing software to work with rotational hard drives because SSD drives are the way of the future. All the language collations have simply been replaced with UTF-8 only, because the web uses UTF-8. Plus, this is tested with 41 different language translations. Drizzle has a new scheduler. The legacy MySQL scheduler was designed to work for a thread-per-session setup. In Drizzle, sessions are handled independently from the threads. The new scheduler allows this to work.

Drizzle uses InnoDB as its default storage engine, which is great for OLTP. It also supports the PBXT storage engine. There are available plugins for the InnoDB Embedded Engine and HailDB which will soon be the new default. DDL Operations (like ALTER TABLE) can actually roll back in the event that something goes wrong in the process, rather than leaving you with incomplete or corrupt data.

The code base in Drizzle has been fully modernized, and brought up to today’s standards of C++ with extensive use of the C++ STL to replace MySQL’s usage of obscure custom data type implementations that offered no real benefit compared to what the STL has today. Another example of improvements in this area is the replacement of the legacy REGEX implementation with a more standard library. All of these changes reduce the amount of Drizzle source code dramatically compared to MySQL. Less code and simpler code means less bugs, plain and simple. Drizzle is well on its way to being an ideal fit for web applications that need a reliable, and high performance transactional database.

Features in Drizzle7 Beta

• New micro kernel
• Migration Tool
• Instance Catalog Support
• Universal Replication
• User query analysis
• Mutli-core Support

What “Beta” means

• Your data is safe. Transactional engine by default and stable for over 2 years.
• Upgrade the system in-place without exporting/importing data.
• Replication is still being tested.

In Microsoft terms, it means that this project would have launched about a year ago. In Google terms, it probably would have launched six months ago. Simply put, if you trust your data to a MySQL system today running InnoDB, you should feel comfortable trying Drizzle. There have been some changes to the InnoDB setup, such as the elimination of the FRM files from disk which eliminate possible inconsistency between the state on disk and the state in InnoDB. I am in the process of moving a few of my produciton applications to use the Drizzle Beta. If you’re an accomplished system administrator and DBA, you should seriously consider putting at least one of your production applications on Drizzle now, and see how it works for you.

What’s Next?

• Beta announced today 2010-09-29.
• GA February 2011
• GA May 2011 for Multi-Tenancy features that allow an arbitrary number of logical databases (Schemas, Tables, etc.) to exist concurrently with full data isolation between them. This allows for individual security and resource controls (Threads, Memory, IO), and individual database backups, rather than system level backups. This feature will be called “Catalogs”.

Download Drizzle

Time to get started with the beta. Download the beta today!

This is very exciting for consumers of Cloud Computing because:

1. It allows individual companies to run their own clouds inside their own data centers and on their own equipment using the same scalable technology that powers some of the largest cloud infrastructures in the world.
2. An individual company can develop applications on their own cloud, and know with confidence that they can run their application on a number of different public clouds without any special adapter software. They need only find a cloud computing or cloud storage provider that uses OpenStack software to ensure compatibility.
3. If an application is hosted on one OpenStack public cloud, it can be easily moved to another without changes to the application source code, and without using any cloud middleware. This can completely eliminate all fears relating to single-vendor lock-in.
4. Applications can be run simultaneously in multiple clouds using the exact same software that needs to only implement a single API for universal access to computing and object storage resources.

First of all, let’s review some definitions:

• Bandwidth: The amount of data that can be passed along a communications channel in a given period of time.
• Latency: The time it takes for a packet to cross a network connection, from sender to receiver.
• Speed: Fast and rapid moving, going, traveling, proceeding, or performing; swiftness.
• Throughput: The quantity data transmitted by a computer network over a given period of time.

Now, all of these terms are related, and I want to highlight some of the minutia here:

Bandwidth

The higher the bandwidth is on a network connection, the more data it’s capable of transmitting in a given period of time. Higher bandwidth is better.

Latency

This is very very important, because latency effectively limits the amount of bandwidth you can consume if you are using a synchronous data transmission, like a TCP/IP download. Lower latency is better, and will yield faster speed.

Throughput

Throughput is another way of expressing speed. The higher the throughput, the faster your network communications will be. Note that your maximum possible throughput is your bandwidth. Actual throughput is equal to or less than your bandwidth.

Speed

If your network is high speed, you should observe high bandwidth, low latency, and high throughput.

Latency and Bandwidth are Inversely Proportional

For TCP/IP transmissions, the higher your latency is, the lower your throughput will be. Let’s explore why. The most common use of TCP/IP is for the web, which uses the HTTP protocol. HTTP works by making a TCP/IP connection to a remote server, issuing a request for a document, and then receiving the response. The protocol is text based. A simple HTTP transmission is illustrated below.

Client Request:

GET / HTTP/1.1
User-Agent: Wget
Host: www.example.com

Server Response:

HTTP/1.1 200 OK
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Tue, 15 Nov 2005 13:24:10 GMT
ETag: "b300b4-1b6-4059a80bfd280"
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Connection: Keep-Alive
Date: Wed, 18 Nov 2009 22:36:34 GMT
Age: 1010
Content-Length: 438

  Example Web Page

You have reached this web page by typing "example.com",
"example.net",
  or "example.org" into your web browser.

These domain names are reserved for use in documentation and are not available
  for registration. See &lta href="http://www.rfc-editor.org/rfc/rfc2606.txt">RFC
  2606</a>, Section 3.

Here is a trace of the TCP/IP packets that make up that request:

14:57:47.146665 IP 192.168.144.2.39556 > 192.0.32.10.80: S 3717672264:3717672264(0) win 5840
14:57:47.220092 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 1 win 183
14:57:47.220309 IP 192.168.144.2.39556 > 192.0.32.10.80: P 1:123(122) ack 1 win 183  (GET Request)
14:57:47.300962 IP 192.0.32.10.80 > 192.168.144.2.39556: P 1:728(727) ack 123 win 4502  (200 OK Response)
14:57:47.300993 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 728 win 228
14:57:47.302035 IP 192.168.144.2.39556 > 192.0.32.10.80: F 123:123(0) ack 728 win 228
14:57:47.375475 IP 192.0.32.10.80 > 192.168.144.2.39556: . ack 124 win 4502
14:57:47.375499 IP 192.0.32.10.80 > 192.168.144.2.39556: F 728:728(0) ack 124 win 4502
14:57:47.375510 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 729 win 228

Notice that there are 10 packets in the above trace. It’s a three way handshake to set up the TCP session, then a round trip to send the data, then two more round trips to close down the connection. Each time the server receives a packet from the client, the connection may wait in the server’s connection queue to be processed, which can further increase the interactive protocol latency. Consider the impact of high latency on a connection like this. Suppose that it takes 0.2 seconds for each round trip. That connection would have a total throughput of 727 bytes downloaded in 0.8 seconds. That’s a rate of 909 Bytes/sec. Maybe your internet connection is 15 Mb/sec. bandwidth did not matter. Latency caused the throughput to be low.

Now, you might be wondering why we can’t just improve networking technology to make latency lower. We can, but that’s not going to help much, because we are still bounded by the speed of light, among other factors. The speed of light is slow when you consider the distance it has to travel to cross continents on the earth. Let’s look at some match to explain that:

• The speed of light in vacuum is 299,792,458 m/s.
• The speed of light in fiber optic cable is ~200,000,000 m/s.
• The distance from Anaheim, CA to New York is 4,494,898 meters
• The one-way latency to New York is 4,494,898 / 200,000,000 = 22.47ms
• The round-trip time between Anaheim, CA and New York is 44.95ms
• The current ping time from Anaheim, CA to New York is 72 ms

Tracing the route to sl-gw33-nyc.sprintlink.net (144.228.243.82)
  1 sl-crs1-ana-0-14-2-0.sprintlink.net (144.232.11.9) 0 msec
    sl-crs2-ana-0-14-2-0.sprintlink.net (144.232.11.11) 0 msec
    sl-crs1-ana-0-14-2-0.sprintlink.net (144.232.11.9) 4 msec
  2 sl-crs2-fw-0-13-3-0.sprintlink.net (144.232.19.197) 28 msec
    sl-crs2-fw-0-9-5-0.sprintlink.net (144.232.20.130) 28 msec
    sl-crs1-fw-0-3-3-0.sprintlink.net (144.232.9.65) 28 msec
  3 sl-crs2-kc-0-0-0-2.sprintlink.net (144.232.19.141) 40 msec
    144.232.20.57 40 msec
    sl-crs1-kc-0-5-5-0.sprintlink.net (144.232.24.9) 40 msec
  4 sl-crs2-chi-0-13-5-0.sprintlink.net (144.232.20.109) 52 msec
    sl-crs1-chi-0-1-0-3.sprintlink.net (144.232.18.214) 56 msec
    sl-crs2-chi-0-15-2-0.sprintlink.net (144.232.24.206) 52 msec
  5 sl-crs1-nyc-0-8-0-3.sprintlink.net (144.232.18.123) 72 msec
    sl-crs2-nyc-0-8-0-1.sprintlink.net (144.232.20.119) 72 msec
    sl-crs1-chi-0-10-3-0.sprintlink.net (144.232.9.148) 72 msec
  6 sl-gw33-nyc-14-0-0.sprintlink.net (144.232.6.56) 72 msec *
    sl-gw33-nyc-15-0-0.sprintlink.net (144.232.6.58) 72 msec

This round trip time includes all of the switching and routing to get the packet through its full round trip. That means that even if all switching and routing were instantaneous, and we had a perfectly straight fiber path between all points on the earth, that we could only reduce latency by about 40%. We can not accelerate the speed of light, so without a significant advance in data transmission technology (perhaps a quantum physics approach) we must accept the speed of light as a performance boundary.

Making Web Sites Faster

If you’re a web content publisher, you can set up your systems to work around these natural limitations. One way to make interactive web performance faster is to place copies of your data in various geographic locations that are physically closer to your end users. Using a CDN for your media content is one way to do this. You can also make your web server as fast as possible so that your dynamically generated content can be processed as quickly as possible. Using memcached to speed up your web application can help. Also, take a look at some best practices for web developers for good performance.

It’s the time the hypervisor scheduled something else to run instead of something within your VM. This might be time for another VM, or for the Hypervisor host itself. If no time were stolen, this time would be used to run your CPU workload or your idle thread.

There is some disagreement circulating about whether the Hypervisor will steal idle time, or only preempted time. In other words, it has been suggested that stolen time is where your local kernel scheduler within the VM wanted to run something but the Hypervisor made that impossible. I have found that stolen time does in fact count borrowed idle time, where the local scheduler actually had nothing to run. For example, here are some vmstat values from a VM that’s got a very low cpu workload on it:

vmstat -S M 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0    121     42     53    460    0    0     0     1    0    1  0  0 89  0 10
 0  0    121     42     53    460    0    0     0    28 1014   39  0  0 90  0 10
 0  0    121     42     53    460    0    0     0     0 1016   36  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1024   32  0  0 93  0  7
 0  0    121     42     53    460    0    0     0     0 1019   40  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1015   32  0  0 90  0 10
 0  0    121     42     53    460    0    0     0     0 1022   34  0  0 92  0  8
 0  0    121     42     53    460    0    0     0     0 1016   36  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1013   34  0  0 92  0  8
 0  0    121     42     53    460    0    0     0     0 1028   43  0  0 93  0  7

As you can see, user time (us), system time (sy), and iowait time (wa) are zero, but idle time is not 100%. This normally indicates that your system is doing something, but in this case idle time is actually the sum of the id and st columns.

In this example, I really don’t care that I have a nonzero st column because my workload is basically idle all the time anyway.

If you are on a cloud host where you purchase a small sliver of a server, you should expect to see nonzero values in this column when you run vmstat. If you have a heavy CPU load and need more processing power, you can solve this problem by upgrading to a larger VM server size so that you command a larger portion of the physical host.

One piece of advice that I continually dispense is to try to reduce dependencies on remote web sites when coding your own. The problem strikes most dramatically when you run a very busy site, and you have some feed or resource that you download from a remote site. That remote site crashes, and oops, so does yours. It also happens when your busy site gets more traffic than the corresponding requests to the remote site can handle.

I ran into this again today. One site that I host was consuming a remote feed from a site that has a much smaller capacity than my customer does. The site on my end gets over 10 million page views a day (peak ~2000 page views per second). The capacity mismatch became very apparent when something went wrong on the remote end.

The code logic was:

1. If you have a cached version of the feed, and its fresh, then use it.
2. If the cached entry is expired, then fetch a new one, and replace the one in cache.

This logic is fundamentally flawed for busy sites. It seems sensible, but think about what happens when the cached entry expires, and the remote site is responding very slowly. All of a sudden a stampede of requests start stacking up, all trying to get the feed in parallel. It crashes the remote site even worse. The remote site tries to reboot, and you quickly crash it again. The sequence repeats indefinitely.

Why? Because the window of time during which the cache is invalid gets wider and wider as the remote site gets slower and slower. The longer that window is open, the more traffic the remote site will get from cache misses.

A clean solution is to update the cache asynchronously using a scheduled batch job that keeps a local cache of the data. Only attempt to update the cache when it has actually changed. The logic in the web appication changes to:

1. Always use the data in the cached file.

The feed site is consulted on regular intervals using a scheduled batch job (cron), and the cached data is updated if it’s able to get a response. If the remote site is down or too slow, then the application simply continues to use the version it had before. Problem solved!

Why is this not a best practice for all web developers? Because most web sites don’t get enough traffic for it to matter much. But, if you’ve got a busy site, and you don’t want it to crash when your remote feeds do, then you might want to consider getting that data asynchronously, or at least use a cache update procedure that’s serialized.

Here is an example of a non-blocking serialization approach that works for PHP applications.

So all you web developers out there who like to consume RSS feeds on the server-side of your web application… don’t say I didn’t warn you. Go look at all your code and make sure you don’t have an dependency on a remote site. If you do, you now know at least two ways to solve that problem.

I use the TimeMachine software built into Leopard (and newer) OSX. I use a locally connected USB2. A Firewire drive would also be good. Here is a drive that I like because it has lots of capacity, reasonably affordable, compact, and runs quietly.

Fantom GreenDrive Pro 2TB eSATA and USB 2.0 7200RPM 32MB External Hard Drive

Another that’s half the capacity, but cheaper:

Fantom GreenDrive 1TB USB 2.0 and eSATA External Hard Drive

Now, for home use a 2TB drive is probably enough for all your computers. At first I networked them all together to use just one drive on one of my computers shared to all the others so that all the backups were on the one big drive. I later decided that every computer should have it’s own drive for backups. Why? A few reasons:

1. To conserve electricity. When you are using the computer is when the backup snapshots should be taken and archived. When the computer is asleep, may not respond over the network depending on how it’s set up, meaning you need to keep that host machine powered up all the time wasting electricity.
2. Each computer does its backups when they get used, and in the idle time before they fall asleep again. It works much better for me this way.
3. Immediate restores. Having a local drive on each computer makes restoration super fast. It’s not like a network or tape backup where you need to wait for your data to transfer back on to your hard drive to begin using it.

It’s easy to set up Time Machine. Connect the drive, open “Time Machine Preferences” and select the drive.

I re-initialized mine using the disk utility first so that it had a journaled MacOS filesystem on it instead of the default FAT partitioning that comes from the factory.

One really nice thing about Time Machine is that you can easily revert to a prior point in time in the event you accidentally mess something up, get a virus, or whatever. It’s about the easiest tool I’ve ever used. it automatically rotates backups hourly, daily, weekly, etc and deletes old backups automatically to make room for new ones. It’s totally automatic whereas with other tools you need to set that all up yourself.

This sort of local backup does not help if your house or office gets burglarized or burns down because you lose both the primary and backup copy of the data.

Another option is to use JungleDisk to back your data up to the cloud. That has the advantage of only paying for the storage you actually use, the backups are off site, so if you have theft or fire, you can still restore, potentially somewhere else. A disadvantage is that it requires adequate internet connectivity. Your upload speed needs to be fast enough to accommodate all of the data you produce within each backup interval. If your network is already constrained on available bandwidth, running backups over it could potentially aggravate matters. In short, if you have a big fat internet connection, then use JungleDisk.

Rule 1 – Cache is Your Friend

Rule 2 – Don’t write to the database in real time

Rule 3 – Use a “Stateless” design whenever possible

Rule 4 – Avoid Unnecessary External Dependencies

Rule 5 – CMS Plugins

Rule 6 – HTTP Includes

Rule 7 – Coming Soon

Rule 8 – Coming Later

Rule 9 – Coming Later

Rule 10 – Coming Later

Yep, if you follow all 10 of the rules, you’ll probably have a really good cloud based web app.