First of all, let’s review some definitions:

• Bandwidth: The amount of data that can be passed along a communications channel in a given period of time.
• Latency: The time it takes for a packet to cross a network connection, from sender to receiver.
• Speed: Fast and rapid moving, going, traveling, proceeding, or performing; swiftness.
• Throughput: The quantity data transmitted by a computer network over a given period of time.

Now, all of these terms are related, and I want to highlight some of the minutia here:

Bandwidth

The higher the bandwidth is on a network connection, the more data it’s capable of transmitting in a given period of time. Higher bandwidth is better.

Latency

This is very very important, because latency effectively limits the amount of bandwidth you can consume if you are using a synchronous data transmission, like a TCP/IP download. Lower latency is better, and will yield faster speed.

Throughput

Throughput is another way of expressing speed. The higher the throughput, the faster your network communications will be. Note that your maximum possible throughput is your bandwidth. Actual throughput is equal to or less than your bandwidth.

Speed

If your network is high speed, you should observe high bandwidth, low latency, and high throughput.

Latency and Bandwidth are Inversely Proportional

For TCP/IP transmissions, the higher your latency is, the lower your throughput will be. Let’s explore why. The most common use of TCP/IP is for the web, which uses the HTTP protocol. HTTP works by making a TCP/IP connection to a remote server, issuing a request for a document, and then receiving the response. The protocol is text based. A simple HTTP transmission is illustrated below.

Client Request:

GET / HTTP/1.1
User-Agent: Wget
Host: www.example.com

Server Response:

HTTP/1.1 200 OK
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Tue, 15 Nov 2005 13:24:10 GMT
ETag: "b300b4-1b6-4059a80bfd280"
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Connection: Keep-Alive
Date: Wed, 18 Nov 2009 22:36:34 GMT
Age: 1010
Content-Length: 438

  Example Web Page

You have reached this web page by typing "example.com",
"example.net",
  or "example.org" into your web browser.

These domain names are reserved for use in documentation and are not available
  for registration. See &lta href="http://www.rfc-editor.org/rfc/rfc2606.txt">RFC
  2606</a>, Section 3.

Here is a trace of the TCP/IP packets that make up that request:

14:57:47.146665 IP 192.168.144.2.39556 > 192.0.32.10.80: S 3717672264:3717672264(0) win 5840
14:57:47.220092 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 1 win 183
14:57:47.220309 IP 192.168.144.2.39556 > 192.0.32.10.80: P 1:123(122) ack 1 win 183  (GET Request)
14:57:47.300962 IP 192.0.32.10.80 > 192.168.144.2.39556: P 1:728(727) ack 123 win 4502  (200 OK Response)
14:57:47.300993 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 728 win 228
14:57:47.302035 IP 192.168.144.2.39556 > 192.0.32.10.80: F 123:123(0) ack 728 win 228
14:57:47.375475 IP 192.0.32.10.80 > 192.168.144.2.39556: . ack 124 win 4502
14:57:47.375499 IP 192.0.32.10.80 > 192.168.144.2.39556: F 728:728(0) ack 124 win 4502
14:57:47.375510 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 729 win 228

Notice that there are 10 packets in the above trace. It’s a three way handshake to set up the TCP session, then a round trip to send the data, then two more round trips to close down the connection. Each time the server receives a packet from the client, the connection may wait in the server’s connection queue to be processed, which can further increase the interactive protocol latency. Consider the impact of high latency on a connection like this. Suppose that it takes 0.2 seconds for each round trip. That connection would have a total throughput of 727 bytes downloaded in 0.8 seconds. That’s a rate of 909 Bytes/sec. Maybe your internet connection is 15 Mb/sec. bandwidth did not matter. Latency caused the throughput to be low.

Now, you might be wondering why we can’t just improve networking technology to make latency lower. We can, but that’s not going to help much, because we are still bounded by the speed of light, among other factors. The speed of light is slow when you consider the distance it has to travel to cross continents on the earth. Let’s look at some match to explain that:

• The speed of light in vacuum is 299,792,458 m/s.
• The speed of light in fiber optic cable is ~200,000,000 m/s.
• The distance from Anaheim, CA to New York is 4,494,898 meters
• The one-way latency to New York is 4,494,898 / 200,000,000 = 22.47ms
• The round-trip time between Anaheim, CA and New York is 44.95ms
• The current ping time from Anaheim, CA to New York is 72 ms

Tracing the route to sl-gw33-nyc.sprintlink.net (144.228.243.82)
  1 sl-crs1-ana-0-14-2-0.sprintlink.net (144.232.11.9) 0 msec
    sl-crs2-ana-0-14-2-0.sprintlink.net (144.232.11.11) 0 msec
    sl-crs1-ana-0-14-2-0.sprintlink.net (144.232.11.9) 4 msec
  2 sl-crs2-fw-0-13-3-0.sprintlink.net (144.232.19.197) 28 msec
    sl-crs2-fw-0-9-5-0.sprintlink.net (144.232.20.130) 28 msec
    sl-crs1-fw-0-3-3-0.sprintlink.net (144.232.9.65) 28 msec
  3 sl-crs2-kc-0-0-0-2.sprintlink.net (144.232.19.141) 40 msec
    144.232.20.57 40 msec
    sl-crs1-kc-0-5-5-0.sprintlink.net (144.232.24.9) 40 msec
  4 sl-crs2-chi-0-13-5-0.sprintlink.net (144.232.20.109) 52 msec
    sl-crs1-chi-0-1-0-3.sprintlink.net (144.232.18.214) 56 msec
    sl-crs2-chi-0-15-2-0.sprintlink.net (144.232.24.206) 52 msec
  5 sl-crs1-nyc-0-8-0-3.sprintlink.net (144.232.18.123) 72 msec
    sl-crs2-nyc-0-8-0-1.sprintlink.net (144.232.20.119) 72 msec
    sl-crs1-chi-0-10-3-0.sprintlink.net (144.232.9.148) 72 msec
  6 sl-gw33-nyc-14-0-0.sprintlink.net (144.232.6.56) 72 msec *
    sl-gw33-nyc-15-0-0.sprintlink.net (144.232.6.58) 72 msec

This round trip time includes all of the switching and routing to get the packet through its full round trip. That means that even if all switching and routing were instantaneous, and we had a perfectly straight fiber path between all points on the earth, that we could only reduce latency by about 40%. We can not accelerate the speed of light, so without a significant advance in data transmission technology (perhaps a quantum physics approach) we must accept the speed of light as a performance boundary.

Making Web Sites Faster

If you’re a web content publisher, you can set up your systems to work around these natural limitations. One way to make interactive web performance faster is to place copies of your data in various geographic locations that are physically closer to your end users. Using a CDN for your media content is one way to do this. You can also make your web server as fast as possible so that your dynamically generated content can be processed as quickly as possible. Using memcached to speed up your web application can help. Also, take a look at some best practices for web developers for good performance.

Today Sagem Orga made a press release that raised my eyebrows. They have a new SIM card (the identification chip in your GSM cell phone) that has WiFi capability right on the chip. This is exciting, because it would enable otherwise ordinary cell phones to be used as WiFi internet gateways, running both WiFi and 3G data connections at the same time.

This is something that most phones simply can not do. The ones that can do it require that a software program must be running on the phone to make it into a router that can relay WiFi signals over the web through a 3G data connection over the cell phone network. Getting this on a Blackberry, for example is a huge nuisance, if your service provider supports it at all.

Well, that nuisance may be a thing of the past once the new “SIMFi” technology hits the market. Imagine just plugging in the snazzy new card into your phone, joining its WiFi network from your laptop, and accessing the internet from practically anywhere. How cool is that!?!

There has been a discussion on Slashdot about this. One of the interesting commentary was about the need for a 2.4 GHz antenna, which can actually fit fine on the SIM card itself, as long as it’s bent around a bit. An obvious question with any WiFi product is “what’s the implication on battery life?”. It will definitely be shorter. Hopefully this device will have some sort of a tunable transmit power adjustment for the WiFi signal so power consumption can be kept to a minimum. After all, your laptop and your cell phone will only be an arm’s length apart when you are using this setup anyway, so range is not a major concern.

Yes, I do love technical gadgets. The thought of where this could go is very exciting. I’ll be the first on the waiting list for this!

It’s the time the hypervisor scheduled something else to run instead of something within your VM. This might be time for another VM, or for the Hypervisor host itself. If no time were stolen, this time would be used to run your CPU workload or your idle thread.

There is some disagreement circulating about whether the Hypervisor will steal idle time, or only preempted time. In other words, it has been suggested that stolen time is where your local kernel scheduler within the VM wanted to run something but the Hypervisor made that impossible. I have found that stolen time does in fact count borrowed idle time, where the local scheduler actually had nothing to run. For example, here are some vmstat values from a VM that’s got a very low cpu workload on it:

vmstat -S M 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0    121     42     53    460    0    0     0     1    0    1  0  0 89  0 10
 0  0    121     42     53    460    0    0     0    28 1014   39  0  0 90  0 10
 0  0    121     42     53    460    0    0     0     0 1016   36  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1024   32  0  0 93  0  7
 0  0    121     42     53    460    0    0     0     0 1019   40  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1015   32  0  0 90  0 10
 0  0    121     42     53    460    0    0     0     0 1022   34  0  0 92  0  8
 0  0    121     42     53    460    0    0     0     0 1016   36  0  0 91  0  9
 0  0    121     42     53    460    0    0     0     0 1013   34  0  0 92  0  8
 0  0    121     42     53    460    0    0     0     0 1028   43  0  0 93  0  7

As you can see, user time (us), system time (sy), and iowait time (wa) are zero, but idle time is not 100%. This normally indicates that your system is doing something, but in this case idle time is actually the sum of the id and st columns.

In this example, I really don’t care that I have a nonzero st column because my workload is basically idle all the time anyway.

If you are on a cloud host where you purchase a small sliver of a server, you should expect to see nonzero values in this column when you run vmstat. If you have a heavy CPU load and need more processing power, you can solve this problem by upgrading to a larger VM server size so that you command a larger portion of the physical host.

One piece of advice that I continually dispense is to try to reduce dependencies on remote web sites when coding your own. The problem strikes most dramatically when you run a very busy site, and you have some feed or resource that you download from a remote site. That remote site crashes, and oops, so does yours. It also happens when your busy site gets more traffic than the corresponding requests to the remote site can handle.

I ran into this again today. One site that I host was consuming a remote feed from a site that has a much smaller capacity than my customer does. The site on my end gets over 10 million page views a day (peak ~2000 page views per second). The capacity mismatch became very apparent when something went wrong on the remote end.

The code logic was:

1. If you have a cached version of the feed, and its fresh, then use it.
2. If the cached entry is expired, then fetch a new one, and replace the one in cache.

This logic is fundamentally flawed for busy sites. It seems sensible, but think about what happens when the cached entry expires, and the remote site is responding very slowly. All of a sudden a stampede of requests start stacking up, all trying to get the feed in parallel. It crashes the remote site even worse. The remote site tries to reboot, and you quickly crash it again. The sequence repeats indefinitely.

Why? Because the window of time during which the cache is invalid gets wider and wider as the remote site gets slower and slower. The longer that window is open, the more traffic the remote site will get from cache misses.

A clean solution is to update the cache asynchronously using a scheduled batch job that keeps a local cache of the data. Only attempt to update the cache when it has actually changed. The logic in the web appication changes to:

1. Always use the data in the cached file.

The feed site is consulted on regular intervals using a scheduled batch job (cron), and the cached data is updated if it’s able to get a response. If the remote site is down or too slow, then the application simply continues to use the version it had before. Problem solved!

Why is this not a best practice for all web developers? Because most web sites don’t get enough traffic for it to matter much. But, if you’ve got a busy site, and you don’t want it to crash when your remote feeds do, then you might want to consider getting that data asynchronously, or at least use a cache update procedure that’s serialized.

Here is an example of a non-blocking serialization approach that works for PHP applications.

So all you web developers out there who like to consume RSS feeds on the server-side of your web application… don’t say I didn’t warn you. Go look at all your code and make sure you don’t have an dependency on a remote site. If you do, you now know at least two ways to solve that problem.

Yes, if you have similar entropy pools it is easier to break encryption dependent on it. It’s reasonably easy to work around this and make sure your entropy pool is uniquely initialized. You can consult the random manual for the Linux Kernel for information about how to seed your entropy pool with a particular set of data. If you are running an application in the cloud that utilizes encryption, and you are concerned about the initial state of your entropy pool, you can solve that. Use this procedure:

1) Seed your own pool from a long running system that has sufficient entropy in it, rather than relying on what you read from the kernel at startup.

2) Produce a network service that you use to seed your initial entropy pools. This service could be as simple as an entropy file that you create on pseudo-random time intervals, and just discard them as you serve them to cloud server instances (as they boot up) so you never serve the same one twice. At boot time from your VM, simply connect to wherever you run this service and download an input file to seed your entropy pool with. Restrict access to this so that it’s only available to your own server instances.

3) Make sure that your custom entropy pool initialization takes place prior to starting your encryption software.

4) If you are creating an AMI, or other server image that you plan to clone, be sure that it does not have a host key generated yet. Delete it and allow your initialization scripts to create it when the server is created (after step rather than making copies of the same one.

If you don’t trust what /dev/random or /dev/urandom emit, you can optionally use OpenSSL with prngd or egd as alternate entropy sources, and potentially feed in your own sensory input data. If you want to go hardcore, you could add environmental noise such as resistor noise on the microphone input of a sound card, or some other sensory data. There is existing software for doing just that. There’s all sorts of possibilities. Among them are a number of hardware solutions for RNG, most of which are pretty expensive and are not options for a cloud environment. There are sources of random numbers provided as a service from various sources.

There are things that we can do as Cloud Computing service providers to pre-initialize your entropy pools for you when the given server instance is created so the procedure above would be redundant. This still leaves the question as to the quality of the RNG available to you on a cloud server.

There are two standard randomness sources that you should know about:

/dev/random = produces actual entropy, if you have some, and blocks otherwise.
/dev/urandom = produces available entropy regardless of quality, but does not block.

The Linux kernel has a paravirtual entropy driver which provides kernel-side support for the virtual RNG hardware. The kernel compile option CONFIG_HW_RANDOM_VIRTIO enables it, and it can be built as a kernel module. There are drivers that run within the hypervisor host kernel that connect this with the RNG hardware available on the server (if any).

drivers/char/hw_random/amd-rng.ko = H/W RNG driver for AMD chipsets
drivers/char/hw_random/intel-rng.ko = H/W RNG driver for Intel chipsets
drivers/char/hw_random/virtio-rng.ko = VirtIO Random Number Generator support

How it works is the hypervisor host (dom0) runs rngd to read data from /dev/hwrandom (using the Intel or AMD modules mentoined above) and feeds it into /dev/random, then the guest VM (domU) does the same thing. The rngd can mixes data from both /dev/random and /dev/urandom so you get as much random data as you need in a non-blocking fashion. You can consult the kernel source code to learn more. Then you run rngd in the guest VM to feed that into the kernel.

What happens if multiple guest VM’s are reading this data at the same time using this arrangement? I’m not sure if it’s possible to deplete the entropy pool of the hypervisor host and produce PRNG patterns that are therefore less random. So if one guest VM emptied the entropy pool by aggressively reading from the /dev/hwrandom device, you might cause someone else’s guest VM to get less data. This could be solved if there were a simply a rate limit enforced on the consumption of RNG data allowed per guest VM. There is further discussion of that as well.

The truth is that for most needs you can have reasonably secure encryption by simply having an ordinary PRNG source like /dev/urandom that’s properly initialized with random data. I suggest that you use that approach in your cloud deployments.

He explained that the dryer has an overheat safety sensor. When the exhaust temperature gets close to the point where it might catch lint on fire, it shuts down the gas, so the dryer cools off. With no vent, the dryer cooks the air in the exhaust path. My clothes were getting hot at the start of a cycle, and just tumbling after that. No wonder they would not dry! Happily I paid him for telling me that there was nothing to fix!

I’ve had this washer and dryer for almost a decade, and have never had a problem. Now that I had my first problem, and found that it was not even the appliance that had the issue, but simply that I had never cleaned out the exhaust path. I have since learned that this should be done regularly.

After a few minutes with my shop vac sucking lint out of the exhaust hose, inside the dryer exhaust plenum, and up the external vent pipe, everything was working perfectly again. First of all, I’m happy that the dryer had this safety feature so my house did not catch fire. According to the US Consumer Product Safety Commission people die from this all the time. Secondly, I’m thrilled that there was nothing mechanically wrong with my dryer. Thirdly, I’m looking forward to saving some costs on my Gas bill which will certainly decrease now.

I am such a happy Maytag customer! I used to laugh at the Maytag Repair Man ads. Not anymore. They have earned my complete respect for their safe and reliable appliances. When you buy a washer/dryer, you want reliability. Maytag is the real McCoy.

Okay, here’s something that fixes that problem: the Remus Project. The approach is brilliant. On regular intervals it ships the changed memory registers from one host to the other. Memory reading does not need to be replicated, only writes, and writes to the same location don’t all need to be replicated, only the most recent write. The primary node simply delays its response to TCP/IP packets (output buffering) until after it has confirmed that the standby node has received the replicated memory data. Very very clever.

Here are the key features listed on the Remus web site:

• The backup VM is an exact copy of the primary VM. When failure happens, it continues running on the backup host as if failure had never occurred.
• The backup is completely up-to-date. Even active TCP sessions are maintained without interruption.
• Protection is transparent. Existing guests can be protected without modifying them in any way.

Okay, I’ve been running HA systems in multiple geographies now for about a decade. I’ve experimented with lots and lots of clustering and replication technology. Most of the time when I hear about something new, I cringe and wonder if it’s just another thing that’s using the same old tricks I’ve been using for years, or if its something truly innovative and truly open source. Before you go making comments that VMWare has this feature or that feature, relax. This post is not about VMWare. It’s about open source Xen.

Now, you might already be wondering if this would work if you separated the two nodes to run in separate locations. The short answer is maybe. You would still need a very clever network configuration to re-route your traffic dynamically to the new location. For those of us that do operate our own Autonomous Systems, that may seem possible with a BGP route update. But here’s the bummer… The additional latency it would introduce would bring your performance to a screeching halt. You could probably afford to have about 25ms of average latency between two locations and get away with it. The cut-over would still be better than nothing, but you’d better have a rock solid network in there, and you’d better be ready to pump lots of bandwidth over it. Plan for 100Mb/sec if you checkpoint every 100ms.

This would be great for a high read application like a web cache, or some memcached applications. People ask on the memcached mailing list all the time how they can set up replication and HA. The answer is always “it’s a cache… not a database.”. Well, for those of you that want to do HA for a memcached system, give Remus a try.

Let’s not stop there. Imagine you have a SIP call control platform or Trixbox system, and you don’t want to lose all your active calls in the event of a system crash? Pretty much any mission critical application that supports long running connections over TCP/IP

Remus has been around for some time, so why am I so excited now? It’s now part of Xen! You don’t need to do anything special on the master or slave node to use it! Whoot! Now I’m impressed. Anyone out there have experience running it? I’d love to hear your thoughts.

I use the TimeMachine software built into Leopard (and newer) OSX. I use a locally connected USB2. A Firewire drive would also be good. Here is a drive that I like because it has lots of capacity, reasonably affordable, compact, and runs quietly.

Fantom GreenDrive Pro 2TB eSATA and USB 2.0 7200RPM 32MB External Hard Drive

Another that’s half the capacity, but cheaper:

Fantom GreenDrive 1TB USB 2.0 and eSATA External Hard Drive

Now, for home use a 2TB drive is probably enough for all your computers. At first I networked them all together to use just one drive on one of my computers shared to all the others so that all the backups were on the one big drive. I later decided that every computer should have it’s own drive for backups. Why? A few reasons:

1. To conserve electricity. When you are using the computer is when the backup snapshots should be taken and archived. When the computer is asleep, may not respond over the network depending on how it’s set up, meaning you need to keep that host machine powered up all the time wasting electricity.
2. Each computer does its backups when they get used, and in the idle time before they fall asleep again. It works much better for me this way.
3. Immediate restores. Having a local drive on each computer makes restoration super fast. It’s not like a network or tape backup where you need to wait for your data to transfer back on to your hard drive to begin using it.

It’s easy to set up Time Machine. Connect the drive, open “Time Machine Preferences” and select the drive.

I re-initialized mine using the disk utility first so that it had a journaled MacOS filesystem on it instead of the default FAT partitioning that comes from the factory.

One really nice thing about Time Machine is that you can easily revert to a prior point in time in the event you accidentally mess something up, get a virus, or whatever. It’s about the easiest tool I’ve ever used. it automatically rotates backups hourly, daily, weekly, etc and deletes old backups automatically to make room for new ones. It’s totally automatic whereas with other tools you need to set that all up yourself.

This sort of local backup does not help if your house or office gets burglarized or burns down because you lose both the primary and backup copy of the data.

Another option is to use JungleDisk to back your data up to the cloud. That has the advantage of only paying for the storage you actually use, the backups are off site, so if you have theft or fire, you can still restore, potentially somewhere else. A disadvantage is that it requires adequate internet connectivity. Your upload speed needs to be fast enough to accommodate all of the data you produce within each backup interval. If your network is already constrained on available bandwidth, running backups over it could potentially aggravate matters. In short, if you have a big fat internet connection, then use JungleDisk.

First of all, running a “web scale” application means you have millions of end users. Running a system at that scale commands a certain level of complexity. A “cloud computing” system used to address “web scale” requirements drives complexity. The more complex a system is, the higher the risk that it will fail as a result of its own complexity. Therefore, web scale systems are more difficult to provide on a reliable basis than more simple systems.

The simple truth of the matter is that all systems fail at one time or another. No matter how well designed it is, and how well you test it, eventually something will happen that you were not prepared for, and an outage will occur. System designers must be disciplined to plan for potential problems so they can be predicted and mitigated before they occur in production. However, it’s only a matter of time until an outage does occur. Anyone who tells you that you can have a perfect reliability record forever is a blathering idiot. Don’t be tempted to align your expectations based on what idiots say.

Can you design a system to be highly reliable? Of course. Can a complex system exhibit a reliability record that’s higher than a simple one? If course. However, if the system is driven by software, and that software is complex, then it will contain human errors in a ratio proportional to its complexity. Simply put, the more code there is, the more chance it will contain bugs, or design defects. Yes, these can be mitigated, but I maintain that this problem can not be solved 100%, and that unsolved defects eventually lead to service outages.

Not convinced? In 1986 the Space Shuttle Challenger exploded. Why? Because the decision making procedures were flawed. Human error ultimately resulted in the death of seven astronauts. Blame the problem on a mechanical failure of an o-ring? No. Flawed o-ring design and a bad decision making process lead to death. The same thing happens in computer networks. Even when the software or configurations are not flawed, human error can still lead to system outages. It happens all the time.

Ever heard of a service provider offering a 100% uptime guarantee? You think that means they are going to be up 100% of the time. No, it does not. It means that you will get a discount on your next bill if the system is not up 100% of the time. In severe cases it may give you the option to terminate your service contract. That’s it, plain and simple. If you look long and hard at these guarantees, you will see that the penalties never compensate you for the actual damage of the service being unavailable. It’s a marketing tactic.

As an end user of web scale systems, set some realistic expectations for yourself. The system will break sometimes. I’m sure that your service providers will do everything they reasonable can to avoid outages. In his article, Brockmeier makes a good point that for free services there’s no simple way to extend you a discount. That does not mean that they care any less about uptime. They care. The bottom line is that ALL large scale systems have an imperfect reliability record. Compare Gmail’s reliability record with your own internal corporate email systems. Your reliability is higher? You lie! Measure it, and be honest.

So now that we are being honest, and expect that sometimes systems will fail, I’d like to make my main point. When systems do fail, keeping customers satisfied is about how you respond to the problem, and how you commit to fixing it so that it won’t keep happening. To do this well, here are some guidelines:

1) No Excuses. Customers don’t want to hear about how this problem is not your fault, or how you never expected this. Simply accept responsibility. Be sincere and humble, and commit to taking care of the problem.

2) Communicate. Focusing all your energy on the solution and ignoring the suffering subscriber base during an outage is a mistake. Take enough time to get your facts together, verify them, and use them to keep your subscribers well informed during an outage. If you notice a significant outage before your customers do, find a way to tell them before they notice. They will appreciate your proactive notification.

3) Analyze and Correct. Once service is restored, scrutinize the problem’s root cause, and find a way to prevent a recurrence of the problem.

I could keep listing more and more things here, but these three are the most important to remember.

In conclusion, I agree 100% with Brockmeier’s article, but there is more to the story. Reliability does matter. But in addition, realistic expectations matter just as much.