Apache benchmark (ab) is not exact
I wrote an experimental web server today that keeps some internal statistics. It’s based on libev for the purpose of comparing performance to an equivalent libevent server implementation. During my benchmarking, I sent 10,000 test requests to the server using the ‘ab’ utility from the Apache httpd software distribution using various concurrency levels. What I found was that I would get extra hits to the server logged. This confused me at first, because I thought my server must somehow be corrupting its internal statistics, and showing me extra results.
I assumed I must have a bug in my server, and reviewed my code over and over, until I decided to try http_load and compare it to the results I got with ‘ab’. To my delight, the http_load client actually sent exactly the right number of requests, and my internal hit counter figure matched. I ran several comparisons to confirm it. The ‘ab’ tool does in fact measure the completion of its requests properly, but it may actually send more requests than you ask it to. That’s because it counts replies not requests.
So, if you use a concurrency setting of 1, then requests will equal responses. If you use a concurrency setting of 100, you might end up with 30 or 40 more requests that arrive at the server that the ‘ab’ client does not count in its results. Mystery solved.
I will publish results from my performance study. Initial results are showing that my libev server can produce roughly 10,000 responses per second on a single CPU. This is in the same ballpark as other high performance web servers. Stay tuned.
You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.
I’m glad you posted this. I’ve been investigating the same issue today. I always get 1 or 2 more on 300 POSTs. Did you find a way around this? Did you find http_load client to be better suited?
I do tend to prefer http_load for cases where I care about controlling the total amount of work sent to the server. This matters when you are trying to make a test case reproducible The thing I still like about the ‘ab’ utility by comparison is that it can create gnuplot output format, which makes it really easy to make a chart of the results and visualize the results. The ab utility has more features overall, like the ability to handle HTTPS, control keepalive behavior, set cookies, control the content-type header, send HEAD requests, and send POST data. The http_load does not do any of that. The best solution is to patch ab so that it stops sending requests after the requested number, and add a runtime configuration flag to make that an optional mode. If I ever get around to writing that patch, I’ll post it here. If anyone else makes one, I’d love to see it.
I met the same problem that ab sends more packets than reported. I wonder if you use -k option for persistent connections and the ab version you use. The official site states that a bug realtes to this and already been fixed(https://issues.apache.org/bugzilla/show_bug.cgi?id=48497), but it still exists in later versions in my test case