Quite a few people have written about the performance failings of Net::HTTP, but until recently, to be honest, I never really cared a lot. Most of my http request needs have been fairly meagre, often not much more than hitting a url and checking the result code.
I’ve been playing with couchdb recently, and so my app does a fair amount of http requests. I’ve been using RelaxDB which uses net/http, so Net::HTTP’s performance has started to matter.
Leaning on a mature library such as libcurl gives taf2-curb a huge advantage. While eventmachine was on par speed wise, neither of the 2 http clients it includes are a complete implementation of the HTTP protocol. For example HttpClient will tell the remote server that it speaks HTTP/1.1, yet it does not support chunked encoding (mandatory part of the spec). HttpClient2 does understand chunked encoding, but doesn’t let you set headers or a body to the request. Fine for just pinging a url, but not up to the task of working with couchdb. Something to do with couchdb’s chunk encoded also seemed to confuse rfuzz.
taf2-curb does the job very nicely. On my dumb benchmark, 1000 requests for a static html page hosted on the same machine (ie we’re pretty much only testing overhead) the numbers are:
1 2 3 4 5 6 7 8 9 10 11 12 13
user system total real net/http 0.560000 0.270000 0.830000 ( 1.065960) curb 0.310000 0.170000 0.480000 ( 0.696188)
On the other extreme, these numbers corresponds to ~1 meg of data pulled from couchdb (benchmark code the same apart from the urls, and I did 100 iterations rather than 1000).
user system total real net/http 17.400000 8.900000 2.630000 ( 32.067821) curb 0.700000 1.300000 2.000000 ( 29.586022)
curb comes up squarely on top. Another thing of note during this test is cpu usage (as you might expect from the difference in user time). With Net::HTTP the ruby process running this was taking up 60-70% (on a 2.4GHz core duo), with curb it used around 5% of cpu.
The commit to switch RelaxDB from net/http to taf2-curb is here for those interested – really very straightforward stuff. There may well be more to be had by fiddling with libcurl options, I haven’t tried yet.