It’s not uncommon to want to make requests to AWS from your servers. For example you might be using S3 to store user uploads, pushing custom metrics into CloudWatch or accessing one of the datastores such as DynamoDB or SimpleDB. All of these require authentication, and getting your access key / secret key onto your instances has historically been a little messy.

New in elasticsearch 0.19 is the ability to batch searches together via the multi search api. As far as I know this doesn’t result in elasticsearch doing any less work (unlike sphinx’s multi query stuff which is able to work out when your batch contains common sub expressions and stuff like that) but it does cut out a lot of latency.

I was motivated to poke at this by some work I was doing at dressipi: a particular page required nearly 20 (similar but distinct) search queries to be run. The individual queries were quick, around 10-15ms each but repeat 20 times and this was adding up to around 250ms (and this was on my dev machine so very low latency between my app on elasticsearch). Once I reimplemented it to use multisearch it was down to about 30-40ms for the same batch of 20 queries.

I spoke on elasticsearch at lrug on June 11. You can download the slides I used if you want. Always slightly nerve wracking to stand up and speak in front of a crowded room but I usually enjoy it once I get going and yesterday was no exception. Nice to see a good sized crowd despite the competing attentions of the football and wwdc!

There are already plugins out there to allow you to write your elasticsearch custom scripts in python, javascript and groovy (in addition to the default mvel), so why not ruby?

When you’re indexing data, the world is rarely as simple as each document existing in isolation. Sometimes, you’re better off denormalizing all data into the child documents. For example if you were modelling books, adding an author field to books could be a sensible choice (even if in the database that is your authoritative datasource the data is split into separate authors and books table). It’s simple and you can easily construct queries on both attributes of the book and the author’s name.