Latest Publications

Maytag Repair Man Reports No Problem

maytag_repair_manRecently my clothes dryer had not been working well. Clothes inside were damp and warm rather than hot and dry at the end of a cycle. We called in the Maytag repair man. Actually I booked him on-line, and got an appointment just over 24 hours later. In all fairness, he was not a Maytag repairman, but someone from their authorized service network. He listened to my description of the problem and proceeded to disconnect the exhaust vent hose attachment at the back of the dryer. OMG it was totally packed with lint!

Maytag DryerHe explained that the dryer has an overheat safety sensor. When the exhaust temperature gets close to the point where it might catch lint on fire, it shuts down the gas, so the dryer cools off. With no vent, the dryer cooks the air in the exhaust path. My clothes were getting hot at the start of a cycle, and just tumbling after that. No wonder they would not dry! Happily I paid him for telling me that there was nothing to fix!

I’ve had this washer and dryer for almost a decade, and have never had a problem. Now that I had my first problem, and found that it was not even the appliance that had the issue, but simply that I had never cleaned out the exhaust path. I have since learned that this should be done regularly.

After a few minutes with my shop vac sucking lint out of the exhaust hose, inside the dryer exhaust plenum, and up the external vent pipe, everything was working perfectly again. First of all, I’m happy that the dryer had this safety feature so my house did not catch fire. According to the US Consumer Product Safety Commission people die from this all the time. dollarSecondly, I’m thrilled that there was nothing mechanically wrong with my dryer. Thirdly, I’m looking forward to saving some costs on my Gas bill which will certainly decrease now.

Maytag LogoI am such a happy Maytag customer! I used to laugh at the Maytag Repair Man ads. Not anymore. They have earned my complete respect for their safe and reliable appliances. When you buy a washer/dryer, you want reliability. Maytag is the real McCoy.

Remus Project: Full Memory Mirroring!

Mirrored ServersImagine that you have a cluster with two machines side by side in an active/standby configuration. Let’s say you have your data replicated, and the systems are basically identical except for the IP address and hostname. You can use heartbeat to share an IP address such that if the primary fails, the secondary takes over. You can also perform the equivalent using “live migration” features in a Xen or VMWare hypervisor. The problem with these sorts of fail-overs is that any active TCP/IP sessions end up getting broken, and new connections must be established between clients and the application.

Okay, here’s something that fixes that problem: the Remus Project. The approach is brilliant. On regular intervals it ships the changed memory registers from one host to the other. Memory reading does not need to be replicated, only writes, and writes to the same location don’t all need to be replicated, only the most recent write. The primary node simply delays its response to TCP/IP packets (output buffering) until after it has confirmed that the standby node has received the replicated memory data. Very very clever.

Here are the key features listed on the Remus web site:

  • The backup VM is an exact copy of the primary VM. When failure happens, it continues running on the backup host as if failure had never occurred.
  • The backup is completely up-to-date. Even active TCP sessions are maintained without interruption.
  • Protection is transparent. Existing guests can be protected without modifying them in any way.

Xen LogoOkay, I’ve been running HA systems in multiple geographies now for about a decade. I’ve experimented with lots and lots of clustering and replication technology. Most of the time when I hear about something new, I cringe and wonder if it’s just another thing that’s using the same old tricks I’ve been using for years, or if its something truly innovative and truly open source. Before you go making comments that VMWare has this feature or that feature, relax. This post is not about VMWare. It’s about open source Xen.

Now, you might already be wondering if this would work if you separated the two nodes to run in separate locations. The short answer is maybe. You would still need a very clever network configuration to re-route your traffic dynamically to the new location. For those of us that do operate our own Autonomous Systems, that may seem possible with a BGP route update. But here’s the bummer… The additional latency it would introduce would bring your performance to a screeching halt. You could probably afford to have about 25ms of average latency between two locations and get away with it. The cut-over would still be better than nothing, but you’d better have a rock solid network in there, and you’d better be ready to pump lots of bandwidth over it. Plan for 100Mb/sec if you checkpoint every 100ms.

memcache_logoThis would be great for a high read application like a web cache, or some memcached applications. People ask on the memcached mailing list all the time how they can set up replication and HA. The answer is always “it’s a cache… not a database.”. Well, for those of you that want to do HA for a memcached system, give Remus a try.

trixbox logoLet’s not stop there. Imagine you have a SIP call control platform or Trixbox system, and you don’t want to lose all your active calls in the event of a system crash? Pretty much any mission critical application that supports long running connections over TCP/IP

Remus has been around for some time, so why am I so excited now? It’s now part of Xen! You don’t need to do anything special on the master or slave node to use it! Whoot! Now I’m impressed. Anyone out there have experience running it? I’d love to hear your thoughts.

Advice for backing up your Macs

My wife asked me today if I could give a colleague some advice for how to backup a bunch of Macs. I’ll share my advice for you here. Over the past two decades I’ve used so many different backup systems and software and hardware combinations, I can’t even count them all. So this begs the question, what do I do at home?

I use the TimeMachine software built into Leopard (and newer) OSX. I use a locally connected USB2. A Firewire drive would also be good. Here is a drive that I like because it has lots of capacity, reasonably affordable, compact, and runs quietly.

Fantom DriveFantom GreenDrive Pro 2TB eSATA and USB 2.0 7200RPM 32MB External Hard Drive

Another that’s half the capacity, but cheaper:

Fantom GreenDrive 1TB USB 2.0 and eSATA External Hard Drive

Now, for home use a 2TB drive is probably enough for all your computers. At first I networked them all together to use just one drive on one of my computers shared to all the others so that all the backups were on the one big drive. I later decided that every computer should have it’s own drive for backups. Why? A few reasons:

  1. To conserve electricity. When you are using the computer is when the backup snapshots should be taken and archived. When the computer is asleep, may not respond over the network depending on how it’s set up, meaning you need to keep that host machine powered up all the time wasting electricity.
  2. Each computer does its backups when they get used, and in the idle time before they fall asleep again. It works much better for me this way.
  3. Immediate restores. Having a local drive on each computer makes restoration super fast. It’s not like a network or tape backup where you need to wait for your data to transfer back on to your hard drive to begin using it.

It’s easy to set up Time Machine. Connect the drive, open “Time Machine Preferences” and select the drive.

I re-initialized mine using the disk utility first so that it had a journaled MacOS filesystem on it instead of the default FAT partitioning that comes from the factory.

One really nice thing about Time Machine is that you can easily revert to a prior point in time in the event you accidentally mess something up, get a virus, or whatever. It’s about the easiest tool I’ve ever used. it automatically rotates backups hourly, daily, weekly, etc and deletes old backups automatically to make room for new ones. It’s totally automatic whereas with other tools you need to set that all up yourself.

This sort of local backup does not help if your house or office gets burglarized or burns down because you lose both the primary and backup copy of the data.

Jungle DiskAnother option is to use JungleDisk to back your data up to the cloud. That has the advantage of only paying for the storage you actually use, the backups are off site, so if you have theft or fire, you can still restore, potentially somewhere else. A disadvantage is that it requires adequate internet connectivity. Your upload speed needs to be fast enough to accommodate all of the data you produce within each backup interval. If your network is already constrained on available bandwidth, running backups over it could potentially aggravate matters. In short, if you have a big fat internet connection, then use JungleDisk.