Apache benchmark (ab) is not exact

I wrote an experimental web server today that keeps some internal statistics. It’s based on libev for the purpose of comparing performance to an equivalent libevent server implementation. During my benchmarking, I sent 10,000 test requests to the server using the ‘ab’ utility from the Apache httpd software distribution using various concurrency levels. What I found was that I would get extra hits to the server logged. This confused me at first, because I thought my server must somehow be corrupting its internal statistics, and showing me extra results.

I assumed I must have a bug in my server, and reviewed my code over and over, until I decided to try http_load and compare it to the results I got with ‘ab’. To my delight, the http_load client actually sent exactly the right number of requests, and my internal hit counter figure matched. I ran several comparisons to confirm it. The ‘ab’ tool does in fact measure the completion of its requests properly, but it may actually send more requests than you ask it to. That’s because it counts replies not requests.

So, if you use a concurrency setting of 1, then requests will equal responses. If you use a concurrency setting of 100, you might end up with 30 or 40 more requests that arrive at the server that the ‘ab’ client does not count in its results. Mystery solved.

I will publish results from my performance study. Initial results are showing that my libev server can produce roughly 10,000 responses per second on a single CPU. This is in the same ballpark as other high performance web servers. Stay tuned.