A colleague and I recently updated an app to rails 2.3.4 and found ourselves with a slightly sticky performance problem - slow app response times, with time disappearing seemingly nowhere. New Relic also showed that requests were queueing inside mongrels which wasn’t shouldn’t have been a problem since there were plenty of mongrels for the traffic being received at the time

Our friendly Engineyard support engineer pointed us to a ticket on the Rails lighthouse describing a rather similar situation encountered during the RC phase of rails 2.3: Active record based sessions causing trouble because they weren’t returning their connection to the connection pool [1]. Connection pooling isn’t new in 2.3 but the move to rack changed the order in which things like sessions, query caching, connection pool are put up.

This ticket was mark as resolved, however looking at the fix it became clear why we were still experiencing the problem:

1
2
3
4
5
6
7

if ActionController::Base.session_store == ActiveRecord::SessionStore
  configuration.middleware.insert_before :"ActiveRecord::SessionStore", ActiveRecord::ConnectionAdapters::ConnectionManagement
   configuration.middleware.insert_before :"ActiveRecord::SessionStore", ActiveRecord::QueryCache
else
  configuration.middleware.use ActiveRecord::ConnectionAdapters::ConnectionManagement
  configuration.middleware.use ActiveRecord::QueryCache
end

(Source here)

Since our session store isn’t ActiveRecord::SessionStore, the code to put the database related middleware in the right place in the chain isn’t executed. I’m not entirely sure what to do about this - the rails initializer cannot be expected to know the details of what the session store is doing (unless part of the session store interface was a uses_active_record? flag) but as it is this very sneaky and would have taken a while to find if we hadn’t been pointed in the right direction.

[1] As of Rails 2.2, Active Record has a concept of a connection pool: a given rails instance won’t use more than a certain number of connections (5 by default) even if that rails instance was a jruby based instance with hundreds or threads. If you need a connection from the pool and they are all in use then you have to wait until one becomes available. Normally you are blissfully unaware of this - rails marks your connection as no longer used when your action has finished processing.