First of all, let’s review some definitions:

• Bandwidth: The amount of data that can be passed along a communications channel in a given period of time.
• Latency: The time it takes for a packet to cross a network connection, from sender to receiver.
• Speed: Fast and rapid moving, going, traveling, proceeding, or performing; swiftness.
• Throughput: The quantity data transmitted by a computer network over a given period of time.

Now, all of these terms are related, and I want to highlight some of the minutia here:

Bandwidth

The higher the bandwidth is on a network connection, the more data it’s capable of transmitting in a given period of time. Higher bandwidth is better.

Latency

This is very very important, because latency effectively limits the amount of bandwidth you can consume if you are using a synchronous data transmission, like a TCP/IP download. Lower latency is better, and will yield faster speed.

Throughput

Throughput is another way of expressing speed. The higher the throughput, the faster your network communications will be. Note that your maximum possible throughput is your bandwidth. Actual throughput is equal to or less than your bandwidth.

Speed

If your network is high speed, you should observe high bandwidth, low latency, and high throughput.

Latency and Bandwidth are Inversely Proportional

For TCP/IP transmissions, the higher your latency is, the lower your throughput will be. Let’s explore why. The most common use of TCP/IP is for the web, which uses the HTTP protocol. HTTP works by making a TCP/IP connection to a remote server, issuing a request for a document, and then receiving the response. The protocol is text based. A simple HTTP transmission is illustrated below.

Client Request:

GET / HTTP/1.1
User-Agent: Wget
Host: www.example.com

Server Response:

HTTP/1.1 200 OK
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Tue, 15 Nov 2005 13:24:10 GMT
ETag: "b300b4-1b6-4059a80bfd280"
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Connection: Keep-Alive
Date: Wed, 18 Nov 2009 22:36:34 GMT
Age: 1010
Content-Length: 438

  Example Web Page

You have reached this web page by typing "example.com",
"example.net",
  or "example.org" into your web browser.

These domain names are reserved for use in documentation and are not available
  for registration. See &lta href="http://www.rfc-editor.org/rfc/rfc2606.txt">RFC
  2606</a>, Section 3.

Here is a trace of the TCP/IP packets that make up that request:

14:57:47.146665 IP 192.168.144.2.39556 > 192.0.32.10.80: S 3717672264:3717672264(0) win 5840
14:57:47.220092 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 1 win 183
14:57:47.220309 IP 192.168.144.2.39556 > 192.0.32.10.80: P 1:123(122) ack 1 win 183  (GET Request)
14:57:47.300962 IP 192.0.32.10.80 > 192.168.144.2.39556: P 1:728(727) ack 123 win 4502  (200 OK Response)
14:57:47.300993 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 728 win 228
14:57:47.302035 IP 192.168.144.2.39556 > 192.0.32.10.80: F 123:123(0) ack 728 win 228
14:57:47.375475 IP 192.0.32.10.80 > 192.168.144.2.39556: . ack 124 win 4502
14:57:47.375499 IP 192.0.32.10.80 > 192.168.144.2.39556: F 728:728(0) ack 124 win 4502
14:57:47.375510 IP 192.168.144.2.39556 > 192.0.32.10.80: . ack 729 win 228

Notice that there are 10 packets in the above trace. It’s a three way handshake to set up the TCP session, then a round trip to send the data, then two more round trips to close down the connection. Each time the server receives a packet from the client, the connection may wait in the server’s connection queue to be processed, which can further increase the interactive protocol latency. Consider the impact of high latency on a connection like this. Suppose that it takes 0.2 seconds for each round trip. That connection would have a total throughput of 727 bytes downloaded in 0.8 seconds. That’s a rate of 909 Bytes/sec. Maybe your internet connection is 15 Mb/sec. bandwidth did not matter. Latency caused the throughput to be low.

Now, you might be wondering why we can’t just improve networking technology to make latency lower. We can, but that’s not going to help much, because we are still bounded by the speed of light, among other factors. The speed of light is slow when you consider the distance it has to travel to cross continents on the earth. Let’s look at some match to explain that:

• The speed of light in vacuum is 299,792,458 m/s.
• The speed of light in fiber optic cable is ~200,000,000 m/s.
• The distance from Anaheim, CA to New York is 4,494,898 meters
• The one-way latency to New York is 4,494,898 / 200,000,000 = 22.47ms
• The round-trip time between Anaheim, CA and New York is 44.95ms
• The current ping time from Anaheim, CA to New York is 72 ms

Tracing the route to sl-gw33-nyc.sprintlink.net (144.228.243.82)
  1 sl-crs1-ana-0-14-2-0.sprintlink.net (144.232.11.9) 0 msec
    sl-crs2-ana-0-14-2-0.sprintlink.net (144.232.11.11) 0 msec
    sl-crs1-ana-0-14-2-0.sprintlink.net (144.232.11.9) 4 msec
  2 sl-crs2-fw-0-13-3-0.sprintlink.net (144.232.19.197) 28 msec
    sl-crs2-fw-0-9-5-0.sprintlink.net (144.232.20.130) 28 msec
    sl-crs1-fw-0-3-3-0.sprintlink.net (144.232.9.65) 28 msec
  3 sl-crs2-kc-0-0-0-2.sprintlink.net (144.232.19.141) 40 msec
    144.232.20.57 40 msec
    sl-crs1-kc-0-5-5-0.sprintlink.net (144.232.24.9) 40 msec
  4 sl-crs2-chi-0-13-5-0.sprintlink.net (144.232.20.109) 52 msec
    sl-crs1-chi-0-1-0-3.sprintlink.net (144.232.18.214) 56 msec
    sl-crs2-chi-0-15-2-0.sprintlink.net (144.232.24.206) 52 msec
  5 sl-crs1-nyc-0-8-0-3.sprintlink.net (144.232.18.123) 72 msec
    sl-crs2-nyc-0-8-0-1.sprintlink.net (144.232.20.119) 72 msec
    sl-crs1-chi-0-10-3-0.sprintlink.net (144.232.9.148) 72 msec
  6 sl-gw33-nyc-14-0-0.sprintlink.net (144.232.6.56) 72 msec *
    sl-gw33-nyc-15-0-0.sprintlink.net (144.232.6.58) 72 msec

This round trip time includes all of the switching and routing to get the packet through its full round trip. That means that even if all switching and routing were instantaneous, and we had a perfectly straight fiber path between all points on the earth, that we could only reduce latency by about 40%. We can not accelerate the speed of light, so without a significant advance in data transmission technology (perhaps a quantum physics approach) we must accept the speed of light as a performance boundary.

Making Web Sites Faster

If you’re a web content publisher, you can set up your systems to work around these natural limitations. One way to make interactive web performance faster is to place copies of your data in various geographic locations that are physically closer to your end users. Using a CDN for your media content is one way to do this. You can also make your web server as fast as possible so that your dynamically generated content can be processed as quickly as possible. Using memcached to speed up your web application can help. Also, take a look at some best practices for web developers for good performance.

Okay, here’s something that fixes that problem: the Remus Project. The approach is brilliant. On regular intervals it ships the changed memory registers from one host to the other. Memory reading does not need to be replicated, only writes, and writes to the same location don’t all need to be replicated, only the most recent write. The primary node simply delays its response to TCP/IP packets (output buffering) until after it has confirmed that the standby node has received the replicated memory data. Very very clever.

Here are the key features listed on the Remus web site:

• The backup VM is an exact copy of the primary VM. When failure happens, it continues running on the backup host as if failure had never occurred.
• The backup is completely up-to-date. Even active TCP sessions are maintained without interruption.
• Protection is transparent. Existing guests can be protected without modifying them in any way.

Okay, I’ve been running HA systems in multiple geographies now for about a decade. I’ve experimented with lots and lots of clustering and replication technology. Most of the time when I hear about something new, I cringe and wonder if it’s just another thing that’s using the same old tricks I’ve been using for years, or if its something truly innovative and truly open source. Before you go making comments that VMWare has this feature or that feature, relax. This post is not about VMWare. It’s about open source Xen.

Now, you might already be wondering if this would work if you separated the two nodes to run in separate locations. The short answer is maybe. You would still need a very clever network configuration to re-route your traffic dynamically to the new location. For those of us that do operate our own Autonomous Systems, that may seem possible with a BGP route update. But here’s the bummer… The additional latency it would introduce would bring your performance to a screeching halt. You could probably afford to have about 25ms of average latency between two locations and get away with it. The cut-over would still be better than nothing, but you’d better have a rock solid network in there, and you’d better be ready to pump lots of bandwidth over it. Plan for 100Mb/sec if you checkpoint every 100ms.

This would be great for a high read application like a web cache, or some memcached applications. People ask on the memcached mailing list all the time how they can set up replication and HA. The answer is always “it’s a cache… not a database.”. Well, for those of you that want to do HA for a memcached system, give Remus a try.

Let’s not stop there. Imagine you have a SIP call control platform or Trixbox system, and you don’t want to lose all your active calls in the event of a system crash? Pretty much any mission critical application that supports long running connections over TCP/IP

Remus has been around for some time, so why am I so excited now? It’s now part of Xen! You don’t need to do anything special on the master or slave node to use it! Whoot! Now I’m impressed. Anyone out there have experience running it? I’d love to hear your thoughts.

In order to use memcached in the cloud, you may need to run it on a public network. This introduces a rash of security concerns. Originally memcached was only intended for use on private networks that were not available to the public, so there was no attempt made to provide access controls in the memcached server. There are no concepts of users, passwords, or any access control at all.

If you do run your memcached on a public interface you could use iptables or other host-based firewall rules to limit what IP addresses can access your memcached. However, if you are using a platform hosting service that other subscribers share with you, then others may be able to make connection from the same IP address(es) as you. This means that even if you did limit access to your memcached by IP address it’s possible that some other subscriber of the same hosting service could access your memcached, and cause you all sorts of security problems.

Here is a custom patched memcached 1.4.0 x86_64 RPM I wrote that adds a command line option ‘S’ to disable ‘flush_all’ and ‘stats detail on’ . The original 1.4 source, a SPEC file for RHEL5 and CentOS5 and the patch are both included in the SRPM. By disabling these commands with the -S option in /etc/sysconfig/memcached (OPTIONS=”-S”) you can prevent would-be hackers from dropping all your cached items, or finding out what the names are of the keys you are using. The memcached maintainers want to do this a different way, so this patch won’t be included in the base memecahced source tree.

The right long-term solution is to build multi-tenant features directly into memcached. I’m aware that Dustin Sallings at NorthScale has started some work of this sort, and has a working proof of concept. It’s not yet mature, and is generally incompatible with the current release of memcached, so it’s not yet suitable for production use. The main idea is that a TCP/IP connection to memcached could be authenticated with SASL, and limited to it’s own view of what’s inside memcahced.

My patch does not change how memcahced works, except for what it does when you enter the commands that I’m disabling. It will be just as stable as memcached 1.4.0 without the patch. The only difference is that you won’t have the ‘flush_all’ command, and you won’t have access to detailed stats either.

If you want to flush your entire cache, simply reconfigure your application to begin using a new “secret” key prefix, and you’ll have the functional equivalent of a flush_all because none of the prior cached data will be accessed by your application any more. The old data will simply expire or LRU out of the cache and be replaced by new data naturally.

By using a simple “secret” text prefix to all your keys, you will ensure that hackers won’t know how to access your data in the cache.  Consider prepending a reasonably long test string to the beginning of every key you store and access. Don’t make it too long, or that will multiply the number of packets required to get the data in and out of the cache, but something long enough that it won’t be easily guessed.

This patch does not make memcached bulletproof. An attacker can still do a bunch of SET commands to fill your cache with junk, and force your hot content out. They can still irritate it with a bunch of ‘stats sizes’ commands in a loop, or try to guess your secret prefix by randomly generating keys as a brute force attack until they find your content. For these reasons, you should only use this for storing data that’s not mission critical. There’s lots of data in this category that could really speed up your system under high load if you stored it in memcahed, but is not particularly sensitive to tampering.

Some have argued that this sort of a patch offers a false sense of security. I completely agree. Only use this if you know that your memcached installation will still not be secure, and that the security weakness could be exploited to ultimately hack your application. It will just be a little bit less insecure than it is without the patch.

I have seen memcached used in situations where only statistics are stored and accessed in memcached (instead of generating log files, statistical counters are stored in the cache). The application can do strict checking of the data it gets back from the cache, and not use it in any way that could lead to a security compromise. For example, make sure that all values returned are only numeric, and within acceptable value boundaries. An application of this sort would be appropriate with this patch.

I was thinking of making a better version of this patch that would allow you to specify an IP address (potentially 127.0.0.1 for example) that would have access to all commands that you define in a restricted access class. This way you could configure what IP address(es) could access what commands. Implementing this will require slowing memcached down a bit for all commands. I’ll plan to join forces with the others who are also interested in memcached multi-tenant features and produce a suitable solution that allows for secure deployments in insecure networks.

Must Reads:

memcached: More Cache = Less Cash!

Setting up memcached on Cloud Servers

Setting up memcached on Cloud Sites