Space Vatican - Home

Fun with class variables

2008-08-19T22:52:00Z

Class variables are a slightly fiddly bit of ruby. They don't always quite behave the way you expect. ActiveSupport bundles up a large number of helpers to deal with the various cases. They're used extensively throughout the framework where the ability to control how things set at the framework level ripple down to subclasses (ie your models and controllers) is important to get right, but they can be pretty handy in your own apps too.

The Basics

If you're at all familiar with ruby you'll have heard @@. @@ is a bit odd because it creates variables that are visible both from the instance and the class. The value is also shared with subclasses. ActiveSupport adds cattr_accessor which creates the obvious accessor methods and initializes the value to nil. The accessors are created on both the class and instances of it (the instance writer is optional).

class Foo
  cattr_accessor :bar
end

Foo.bar #=> nil
Foo.bar= '123'
Foo.new.bar #=> '123'
Foo.new.bar = 'abc'
Foo.bar #=> 'abc'

If we create a subclass, the value is shared:

class Derived < Foo
end
Foo.bar= '123'
Derived.bar #=> '123'
Derived.bar = 'abc'
Foo.bar #=> 'abc'

Other slightly odd things can happen:

class Base
  def self.bar
    @@bar
  end

  def self.bar=value
    @@bar
  end
end

class Derived < Base
  cattr_accessor :bar
end
class Derived2 < Base
  cattr_accessor :bar
end

Derived.bar = '123'
Derived2.bar = 'abc'
Derived.bar #=> '123'

So now the subclasses have independent class variables named @@bar. However if you set Base.bar before the Derived and Derived2 classes are created [1] then everyone will be share the base class' value as before. To summarize it's rather fiddly and often unintuitive. It also does not allow the fairly common pattern of having the base class setting a default value that subclasses can override. If you don't know about this then it can lead to odd situations. You'll have code that works fine in dev mode, but not in production (since in development classes are trashed and recreated between requests which obviously interacts with all this) or tests that pass when run individually but fail when you run several of them at the same time.

Classes are objects

Classes are no exception in ruby, like most things (everything?) they are objects, and like objects they can have instance variables. These instance variables are just normal instance variables: they don't have the odd scoping rules that @@ variables have. Base classes and derived classes have completely independent values.

class Base
  @bar = '123'
  class << self
    def bar
      @bar
    end

    def bar= value
      @bar = value
    end
  end
end

class Derived < Base; end
Base.bar #=> '123
Derived.bar #=> nil
Derived.bar = 'abc'
Base.bar #=> "123"

Less prone to unwanted surprises, but not without shortfalls as you cannot make a default from a base class propagate down (without resorting to tricks with self.inherited).

Inheritable Tricks

ActiveSupport adds class_inheritable_accessor. This provides something closer in behaviour to what you might expect: base classes can provide defaults and subclasses inherit those defaults and can overwrite them without affecting other subclasses or the base class.

class Base
  class_inheritable_accessor :value
  self.value = '123'
end

class Derived < Base; end
class Derived2 < Base; end

Derived.value #=> '123'
Derived.value = 'abc'
Base.value #=> '123'
Derived2.value #=> '123

So far so good. We override the value on derive and didn't perturb anyone else. Unfortunately there are some drawbacks:

  Base.value = 'xyz'
  Derived2.value #=> '123

Oh dear. Even though Derived2 never overrode any values the change we made to Base didn't propagate. In other words the default values are baked into the subclass at the time that it is created. This is down to the way that class_inheritable_accessor is implemented: The classes have an instance variable @inheritable_attributes (like we saw above) that is a hash with all the attributes. The accessor methods just pull the values in and out of the hash. When a class is subclassed the subclass gets a copy of @inheritable_attributes. Once this has happened, nothing links the base class' attributes with the subclass.

Bags of tricks

ActiveSupport also provides class_inheritable_array and class_inheritable_hash. They both use class_inheritable_accessor as their underlying mechanism. When you set a class_inheritable_array or a class_inheritable_hash you are actually concatenating (or merging) with the value inherited from the super class.

class Base
  class_inheritable_hash :attrs
  self.attrs = {:name => 'Fred'}
end

class Derived < Base
  self.attrs = {:export => 'Pain'}
end
Derived.attrs #=> {:name => 'Fred', :export => 'Pain'}

These aren't particularly magic but are a handy shortcut.

Delegation for the nation

ActiveSupport's final trick is superclass_delegating_accessor, added in rails 2.0.1. At first it appears very similar to class_inheritable_accessor:

class Base
  superclass_delegating_accessor :properties
  self.properties = []
end

class Derived < Base; end

This time however we can do this:

  Base.properties = [:useless]
  Derived.properties #=> [:useless]

superclass_delegating_accessor creates regular instance variables in the class. The interesting bit is the reader method: it looks at the current class and checks if the appropriate instance variable is defined. If so, it returns it, if not it calls super (ie gets the superclass' instance variable) and so it. It stops when it reaches the class that created the superclass_delegating_accessor.

Unfortunately it doesn't always behave as you would expect: in place modifications will propagate upwards.

  Derived.properties << [:derived]
  Base.properties #=> [:useless, :derived]

Which is just horrible. The example is slightly less contrived when dealing with objects which you tend to modify in place. ActiveResource ran into this with its site property (an instance of URI) and rolls its own thing specially for this property that freezes things so that subclasses don't mess with their parents.

Unfortunately it's a decidedly quirky corner of ruby. Several different options with subtly different semantics that when they go wrong, go wrong in subtle ways that are easy to overlook and with no clear winner. Be careful!

[1] The important bit is actually when Derived and Derived2 create their @@bar variable. cattr_accessor does this for you so in this case the variable is created at the same time the class is.

Counting on both hands

2008-08-17T04:44:00Z

This is more of an SQL hint than anything else, but it's something I've found useful a number of times. If SQL scares you, now's the time to turn back (or crack open a beer - dutch courage).

More often than not if I'm not displaying or editing something then I'm counting. How many unpaid invoices are there, how many customers to I have who were active in the last 6 months etc...

I'm sure I'm not teaching anyone anything if I say that Rails has some helpers for this, for example

Invoice.count :all, :conditions => {:status => 'unpaid'}
Customer.count :all, :conditions => ["updated_at > ?", 6.months.ago]
Invoice.sum :total, :conditions => {:status => 'unpaid'}

From top to bottom this counts the number of unpaid invoices, the number of customers updated in the last 6 months and the total value of all unpaid invoices. Pretty much any option you can stick in a find you can also use with the calculation helpers, for example:

  Customer.sum :weight, :joins => {:orders => :products}
  Customer.sum :weight, :joins => {:orders => :products}, :group => 'customers.id'

This sums the weight of all the items ordered by our customers (ignoring for now the quantity of a product in a given order). The second example groups this by customer.

But what if you want to sum (or count) more than one thing? For example I might want to know the number and the total value of the unpaid invoices. You could of course just do one and then the other but that feels a little wasteful: we're asking the database to scan over the corresponding invoices once and then we turn around and ask it do it again. Luckily with some raw sql it's not hard to do this in one go:

  connection.select_all "SELECT count(*), SUM(total) from invoices where status='unpaid'"

But what if you wanted to count more than one thing? For example I might want a count of all outstanding invoices and a count of those with non trivial value (say more than $10). Again easy enough to write as two queries, but it would be nice to get it all back in one go.

The key to this is understanding the link between summation and counting. To use a technical term, we're interested in indicator functions. If we have:

a set X (which in our terms corresponds to all the rows in a table (or to be more correct, all the rows your query would return if it had no WHERE clause)
a subset A (the rows our WHERE clause would select).
a function I such that I(x) is 1 if x is in A and 0 if not

Then the cardinality (the number of things in it) of A is the sum of I(x) for x ranging over X.

That sounds complicated, but if you think about it, all it is saying is that to count the rows in a table you could use

SELECT SUM(1) from foos

Here our indicator function is dead simple: it always returns 1.

These three queries return the same thing

  SELECT COUNT(*) from invoices where status='pending'
  SELECT SUM(1) from invoices where status='pending'
  SELECT SUM( IF(status='pending', 1, 0) from invoices

The third form is the one we're interested in. Again it shouldn't be hard to spot why it works: for each row that matches we count 1, for all others we count 0. When we add these all up we're just going to get the number of times we counted 1, i.e. the number of matching rows. We wouldn't want to use that on its own (it won't use an index and COUNT(*) probably takes some shortcuts) but in the context of our problem it's just what we need:

  SELECT COUNT(*) as count, SUM(IF(total<10, 1, 0)) as small_invoices where status = 'pending'

will return the number of unpaid invoices and the number of invoices whose total is less than 10 dollars. Here our IF functions are our indicator functions: they return 1 if the condition evaluates to true and 0 if not.

Is it actually useful?

It's going to depend on your data and queries. The ones shown here are probably too simple for it to be of any use since they also had to be easy to explain. When you push some of your conditions from the WHERE clause into the IF statements you are effectively stopping the database from using any indexes to solve those conditions. This can hurt you, but obviously if you weren't using (or don't have) indexes that the database could have used for those conditions then you haven't lost anything.

The other basic premise is that if the database going over some set of data it might as well be counting more than one thing. So if in order to find rows satisfying condition A the database needs to scan some subset X and in order to satisfy condition B the database also needs to scan that same subset X then you're onto a winner. If on the other hand the two are completely distinct (or if X is hopelessly big) then you won't be saving much.

Typically I've used this the most in reporting style applications where having applied a common set of conditions I want to count the number of occurrences of a large number of features. Not something to be using willy-nilly, but a neat trick to have in your toolbox. Use it wisely. And as with all performance things, don't do things blindly just because you read somewhere that it's faster. Profile, measure and so on before and after and come to your own conclusions.

When ducks go mutant

2008-08-09T18:30:00Z

Ruby is a permissive language. In general we don't care what objects are, as long as they respond in certain ways to certain methods: "If it walks like a duck and quacks like a duck, I would call it a duck".

As you may recall, in Rails 2.1 there was a rewrite of the :include mechanism, however the old mechanism persists for compatiblity reasons. When using the new mechanism, the :included associations are not joined, and so if any part of your query looks references the tables that would formerly have been joined it won't work.

To work around this Rails looks at the conditions, order, select statements and so on to see if any of them mention tables other than the main table. If they do then the fallback to the old code is triggered and everything works fine. The code that detects tables in the order clause looks something like

   return [] unless order && order.is_a?(String)
   order.scan(/([\.\w]+).?\./).flatten

The code is quite simple: if we ever have "something." then that means we're using the something table.

The code that eventually adds the order to the statement looks something like

  sql << " ORDER BY #{order}"

The code for the other options is similar.

So what's the problem here? If as the api docs indicate, you pass a string containing a fragment of sql then nothing at all is wrong. However some people (I assume that the :conditions option is the origin of this habit) have taken to doing things like

  Foo.find :all, :order => ['bars']

Before 2.1, this happens to work, because the default to_s on an array just joins the strings together. However the table scanning code won't scan an array (My guess is that the explicit check for string was because people quite legitimately write :order => :name and things like that). So if you've got a select or order clause specified as an array that depends on an included able it will break when you move to rails 2.1.

It's a complete accident that this ever worked (and it breaks if you were to try anything like :order =>['name desc', 'age desc']), but that isn't a huge amount of comfort when code that has been working suddenly stops working. You could probably waste a lot of time before working out that it was specifying the order option as an array, which obviously is not a good thing (and makes people scared of upgrading). On the other hand it's hard to anticipate how people will use things and explicitly checking types and so on isn't a very rubyish thing to do and could get in the way of legitimate uses.

I'm not sure how a framework provider should handle this in the general case. It's a delicate balance between not stifling some of flexibility ruby offers and helping programmers not rely on things that only work by accident.

Nested includes and joins

2008-08-08T18:36:00Z

Nested eager loads are a not entirely obvious bit of rails syntax. It's not hard once you get it though. In a nutshell you have to tell ActiveRecord how it should walk through the associations (i.e. just listing them isn't enough). Before we get going it's worth pointing out that although I've written :include everywhere, everything I've said applies equally to :include and :joins (see my previous post on the difference between the two).

When you are nesting includes, you're building up a data structure that is inherently recursive. There are 3 rules for nested :includes:

Always use the association name. Not the table name, not the class name but the association name (whatever it is that you typed just after belongs_to, has_many etc...). A correlation is that if you don't have all your associations set up, you're dead in the water. If you've got one side of a relationship Rails won't infer the other for you.
If you want to load multiple associations from a model, use an array.
If you want to load an association and some of its child associations then use a hash. The key should be the parent association name, the value should be a description of the child associations.

Now just combine those 3 rules and apply recursively. As an aid here's a quick snippet that takes an include option (such as [:comments, {:posts => :authors}]) and describes which associations are loaded. The structure is the same as the code in activerecord that handles the :include option, so it should give some insight into how things work.

def describe associations, from = 'base'
  case associations
  when Symbol then puts "load #{associations} from #{from}"
  when Array then associations.each {|a| describe a, from}
  when Hash then
    associations.each do |parent, child|
      raise "Invalid hash - key must be an association name" unless parent.is_a?(Symbol)
      describe parent, from
      describe child, parent
    end
  end
end

This code isn't too hard to understand. It's all about the class of the associations parameter

The easy case is if it's a string or a symbol: what we've got is just the name of an association, so just go ahead and load it.
If what we've got is an array, then just call ourselves recursively on the contents of that array.
if what we've got is a hash, then for each key value pair:
- Load the association specified by the key (the parent association)
- Load the associations specified by the value (from the from the parent association).

It produces (ugly) output like this:

  describe [{:comments => :user}, :category]
load comments from base
load user from comments
load category from base

which is exactly what activerecord would do. If you see something like "load user from comments" but instances of Comment don't have an association named user then you've screwed up.

Examples

Still confused ? Here are some examples, from simple to complicated (these are purely examples of the :include syntax - don't see this as a recommendation to actually load 10 layers of nested associations). The models are from a hypothetical book selling application and are:

Book
Author
Comment
User

Books belong to authors, and users of the site can leave a comment on any book. The obvious associations are defined. In addition user has a favourite_books association and through the friends association they can list other users who taste in books they generally share.

Book.find :all, :include => :author

I hope I don't have to explain that one to anyone :-)

Book.find :all, :include => [:author, :comments]

We want to include both the author and comments, so we place the two names in an array

  Book.find :all, :include => [:author, {:comments => :user}]
  Book.find :all, :include => [{:comments => :user}]
  Book.find :all, :include => {:comments => :user}
  Book.find :all, :include => {:comments => [:user]}

In the first example we still want to include author and comments, but now we want to include an association from comments. From the 3rd rule we need a hash containing the key :comments and with corresponding value a description of the associations from the Comment model that we want to load (ie just :user in this case).

In the next 3 examples we just want to include the comments and the user from each comment. These three forms are entirely equivalent (which should be fairly obvious).

  Book.find :all, :include => {:comments => {:user => :favourite_books}}
  Book.find :all, :include => {:comments => {:user => {:favourite_books => :author}}}

Here we are loading the books' comments, the user for each comment and the favourite books for each of those users. In the second example we're also loading the author for each of those favourite books. You can keep on nesting these as far as you want.

  Book.find :all, :include => {:comments => {:user => {:favourite_books => [:author, :comments]}}}

Now we've come full circle - on each favourite book we've loaded the comments

  Book.find :all, :include => {:comments => 
    {:user => [
       :friends, 
       {:favourite_books => [:author, :comments]}
]}}
  Book.find :all, :include => {:comments => 
    {:user => [
      {:friends => :favourite_books}, 
      {:favourite_books => [:author, :comments]}
]}}
  Book.find :all, :include => {:comments => 
    {:user => 
      {:friends => :favourite_books,
       :favourite_books => [:author, :comments]
}}}

Our final examples. In addition to the favourite_books association, we're loading a user's friends, and in the second case the favourite books of those friends. The last two examples are identical: we can either have an array with two 1 item hashes, or just one hash with 2 items. We can't do that in the first example: because we're not loading any associations from friends we can't make it into a hash (what would the corresponding value be?)

Parametrised to the max

2008-07-18T22:02:00Z

If you’ve played with Rails for more that about 3 minutes you’ll know that the params hash can itself contain hashes. If you use form_for/fields_for then you’re used to things like params[:user] containing those fields pertaining to the user and things like that. But it can also contain arrays, arrays of hashes and so on.

Fundamentally html forms don’t know about any sort of structured data. All they know about is name-value pairs. Rails tacks some conventions onto parameter names which it uses to express some structure.

A lot of the time you don’t really care about how this happens. You use the helpers, rails generates its magic form element names and more magic on the other side stuffs that into your params hash. Every now and again you need to head off the beaten track, and there it’s useful to understand how things all fit together (eg if you are generating data to submit via javascript).

Boring cases & parlour tricks

First off we’ll need a way of trying out things. You could of course just mock up a form with the appropriate input elements but this would massively slow down the process of trying out stuff which is never a good thing. Instead we can tap into what Rails uses. Just open up a script/console prompt:

  ActionController::AbstractRequest.parse_query_parameters "name=fred&phone=0123456789"
    => {"name"=>"fred", "phone"=>"0123456789"}

Just stick name=value pairs (joined with &) into a string and pass it to the above function. What you get back is what would be in the params hash in your controller. From now on, for the sake of brevity I’ll just write parse instead of ActionController::AbstractRequest.parse_query_parameters.

If you’ve ever looked at the html your app generates you will have seen form inputs with names like user[name] (as generated by the text_field or form_for helpers). This notation indicates that the user parameter should be a hash, and that here we’re talking about the name key in that hash:

  parse "user[name]=fred&user[phone]=0123456789"
    => {"user"=>{"name"=>"fred", "phone"=>"0123456789"}}

The other basic structure is an array. By default, repeated parameters with the same name are ignored and only the first appearance counts.

  parse "name=fred&name=bob&name=henry"
    => {"name"=>"fred"}

If the name ends with [] then instead of discarding extra occurrences they are accumulated in an array:

  parse "aliases[]=fred&aliases[]=bob&aliases[]=dark+prince&things[]=foo"
    => {"aliases"=>["fred", "bob", "dark prince"], "things"=>["foo"]}

One use case might be a set of checkboxes which indicate which folders a user has to. Just create all your checkboxes with the name accessible_folder_ids[] and with the ids of the folders as values, for example:

  <% @folders.each do |f| %>
    <%= check_box_tag 'user[accessible_folder_ids][]', f.id %> <%= h f.name %> <br/>
  <% end %>

Stick this in a form and when you submit it, params[:user][:accessible_folder_ids] will be an array containing the ids of the folders the user can play with.

Nested parameters for dummies

We can nest hashes if we want. For example perhaps a user has an associated model such as an address:

  parse "user[name]=fred&user[address][town]=cambridge&
         user[address][line1]=4+Station+road"
    => {"user"=>{"name"=>"fred", "address"=>{"line1"=>"4 Station road", "town"=>"cambridge"}}}

A member of a hash can of course be an array. To do this you just append [] to the parameter name, as with a top level array parameter:

  parse "user[aliases][]=fred&user[aliases][]=bob&user[aliases][]=dark+prince"
    => {"user"=>{"aliases"=>["fred", "bob", "dark prince"]}}

The array can be several levels down: we can improve a previous example of a user with an address by saying that the address should have a lines array containing the lines from the address. Start with user[address][lines] and then just append [] to indicate the parameter should be an array:

  parse "user[name]=fred&user[address][town]=cambridge&user[address][lines][]=Random+house&
         user[address][lines][]=4+Station+road&user[address][lines][]=By+the+station"
    => {"user"=> {
      "name"=>"fred",
      "address"=>{ 
        "lines"=>["Random house", "4 Station road", "By the station"],
        "town"=>"cambridge"
    }}}

There’s another case in which nesting like this can be useful. Support for example that we have a list of users and we want the user to be able to change as many of them as they want with one edit operation. If we use the id of the records as the first key then this is easy:

  parse "users[1][name]=fred&users[1][email]=fred@example.com&
         users[2][name]=bob&users[2][email]=bob@example.com"
    =>{"users"=>{
      "1"=>{"name"=>"fred", "email"=>"fred@example.com"}, 
      "2"=>{"name"=>"bob", "email"=>"bob@example.com"
}}}

The corresponding controller code is similarly simple

  params[:users].each do |id, new_attributes|
    User.find(id).update_attributes new_attributes
  end

(Handling errors and invalid entries when editing multiple elements is an interesting problem in itself and is left as an exercise to the reader). If you are using the Rails form helpers, the :index option does precisely this:

  helper.text_field "person", "name"
    => '<input id="person_name" name="person[name]" size="30" type="text" />'
  helper.text_field "person", "name", "index" => 1
    => '<input id="person_1_name" name="person[1][name]" size="30" type="text" />'

Editing multiple objects

An interesting case is a form allowing us to create several models, for example to add several users to a mailing list. To do this we use parameters of the form users[][name]. This says that users is an array and we’re pushing an hash with key name onto it. So for example

  parse "users[][name]=fred&users[][email]=fred@example.com"
    => {"users"=>[{"name"=>"fred", "email"=>"fred@example.com"}]}

So how does rails know when one record is finished and the next start? Simple: if we’ve already had a users[][name] parameter and we see a new one then we can usually assume that the next one must belong to a new user

  parse "users[][name]=fred&users[][email]=fred@example.com&
         users[][name]=bob&users[][email]=bob@example.com"
    => {"users"=>[{"name"=>"fred", "email"=>"fred@example.com"}, 
                  {"name"=>"bob", "email"=>"bob@example.com"}]}

To continue with a previous example, a user might have several addresses:

  parse "user[name]=fred&user[addresses][][line]=24+bob+street&user[addresses][][town]=cambridge&
         user[addresses][][line]=1+market+square&user[addresses][][town]=bedford"
    => {"user"=>{"name"=>"fred", "addresses"=>[
        {"line"=>"24 bob street", "town"=>"cambridge"}, 
        {"line"=>"1 market square", "town"=>"bedford"}]
}}

Turtles all the way down

Hashes can be nested as much as you want:

  parse "users[a][b][c][d]=fred"
    => {"users"=>{"a"=>{"b"=>{"c"=>{"d"=>"fred"}}}}}

Arrays can’t be nested: you can can’t for example have a parameter which is an array of arrays. You can have a hash with an array parameter or an array of hashes but in general there can only be one level of ‘arrayness’. It’s easy enough to see why this is: arrays are built by repeating the same parameter name multiple times, however if you are inside an array then parameter name repetition is precisely what rails uses to determine whether it should move on to the next array element[1].

To an extent this can be sidestepped, for example instead of an array of users you can have a hash of users keyed by id. If your data structure is genuinely just an array you can always make it into a hash that is keyed by array index. For example if we had a form displaying a list of users and each user has a name and a list of aliases then the following query string parses into what we want:

  parse "users[1][name]=fred&users[1][aliases][]=joker&users[1][aliases][]=the+bat&
         users[2][name]=bob&users[2][aliases][]=bobbo&users[2][aliases][]=bobster"
  =>{"users"=>{
    "1"=>{"name"=>"fred", "aliases"=>["joker", "the bat"]},
    "2"=>{"name"=>"bob", "aliases"=>["bobbo", "bobster"]}
}}

Where it can go wrong

Consider the following query string:

  user[aliases][]=fred&user[aliases][name]=fred

This can’t work: user[aliases][] indicates that user has an aliases attribute that’s an array, but user[aliases][name] indicates that user has an aliases attribute that’s a hash. There are other ways in which this can happen, but the result is the same:

  TypeError: Conflicting types for parameter containers

[1] This is also the reason for the problem Xavier mentions in his comment. Checkboxes submit no value if they are not checked, however it’s rather convenient to create the illusion that they send (for example) 1 if the box is checked, 0 if not.

The rails check_box helper does this by adding a hidden field with the same parameter name. If the checkbox submits nothing then the hidden field “wins” and submits 0, if the checkbox is checked then it wins because it’s the first parameter. However since rails uses parameter name repetition to distinguish between elements of an array the hidden parameter causes rails to start a new array element. I don’t know of a good workaround other than using a hash instead of an array or using check_box_tag instead of check_box.

The difference between :include and :joins

2008-06-22T09:12:00Z

I think some people occasionally mix up :include and :joins (or possibly don’t know of the existance of :joins).

:include is for loading associations. Before Rails 2.1 it will always left outer join the appropriate tables, starting with rails 2.1 it will either load them with a separate query or it will join the appropriate tables.

:joins is for joining (duh). It either takes an sql fragment (eg “INNER JOIN foos on foos.id = foo_id”) or association names in a variety of ways and joins the relevant tables. For example:

:joins => :products
:joins => [:products, :customers]
:joins => [:products, {:customers => [:friends, :foes]}]
:joins => [:products, {:customers => {:friends => :parents}}]
:joins => {:order => {:customer => {:address => :some_other_association}}}

and so on. You can nest these as deeply as you want (although bear in mind that if you end up requiring a 23 way join you may want to rethink your strategy). The same notation for nested associations is used by :include. The one difference is that :joins creates inner joins for you (if you desperately need those outer joins, you can always use the string form of :joins, but you will have to write the sql fragment explicitly).

If what you want is to use attributes from the joined tables for sorting or in your conditions then both :include and :joins will work since they both cause the relevant tables to be joined. But :include then does a lot more work massaging the results that come back from the database, instantiating lots of activerecord objects, gluing together all the relationships in the appropriate manner. If all you wanted was to order or filter results based on some of the joined attributes then this work is wasted.

Just to show that I’m not making things up, I performed the following unscientific test:

Benchmark.bm(7) do |x|
  x.report("include:") do
    Customer.find :all, :include => :customer_detail, 
                 :conditions => "customer_details.date_of_birth >= '1980-01-1'"
  end  
  x.report("joins:") do 
    Customer.find :all, :joins => :customer_detail,
                 :conditions => "customer_details.date_of_birth >= '1980-01-1'"
  end
end

Some customers have provided extra details which we store in a separate table, and we want to get all customers born after 1980.

The results:

	user	system	total	real
include	1.310000	0.040000	1.350000	( 1.532910)
joins	0.350000	0.040000	0.390000	( 0.454001)

In short, don’t use :include unless you will actually be accessing those associations and want to avoid the hit caused by loading them from the database one by one.

Berlin, here we come!

2008-06-10T00:31:00Z

Hot off the presses: the Railsconf Europe proposal submitted by my colleague Paul Butcher and myself has been accepted! Join us in Berlin to here about the nifty stuff we've been up to. I can't wait!

Dealing with concurrency

2008-06-08T17:00:00Z

Sometimes you don’t want a user doing two things at once (or two users doing something to the same third party at once). Dealing with this sort of issue is quite fiddly as its easy to overlook them and hard to track them down when they happen as they tend to be heavily timing dependant[4]. There’s a few tricks in the rails toolbox for dealing with them.

There’s a related issue that this post is not about: suppose two users bring up the edit form for some object. They each make unrelated changes to the form and save, one after the other. The second user will squash the changes made by the first user. Oops. I’m not concerned about that, only with what happens if the two requests are actually concurrent, whereas here the two save requests could be minutes apart.

One way of solving this problem is for editable objects to have an edited_by association and disallowing edits if someone else is in the process of editing. Another way is to get clever about how you apply changes to attributes. While not directly relevant, the things outlined here may still be useful, for example if you go down the route of only allowing the user named in the edited_by association to edit the record then you need to be able to reliably set that field even if two users click the edit button at the same time.

And now, our feature presentation

When I was at school the teacher used to dish out ‘bon points’ when you did something good. You could later trade those in for stickers and stuff. In a world very far removed from French primary school classrooms of 1990, the teacher might have written a webapp allowing you to do this (probably complete with an obnoxious facebook app where you could show everyone how many magic stars you have).

Optimism

The people table has a magic_stars column, and we want people to be able to spend their magic stars on various perks. if someone has been a good boy, you might want to use the credit method:

def credit
  self.magic_stars += 1
end

You need to be a little bit careful, as there’s a race condition here: suppose two people try to give someone a magic star at the same time

Person 1	Person 2
loads child (3 stars)	loads child (3 stars)
adds 1	adds 1
saves (4 stars)
	saves (4 stars)

The 2 requests execute in separate mongrels (or mod_rails listeners etc…) and are completely oblivious to each other and will happily overwrite the change made by the other one. Rails’ optimistic locking will help us here: Just add an integer lock_version column to the model (don’t forget to make it default to 0) and the second ‘bad’ save will raise ActiveRecord::StaleObjectError. We can rewrite our credit method like this:

def credit
  self.magic_stars += 1
rescue ActiveRecord::StaleObjectError
  reload
  retry
end

So how does optimistic locking work? The key (unsurprisingly) is in the lock_version column. Assume that we loaded a person object with a lock_version of 4 and we now want to save it. We increment our lock version to 5. If no one else has touched this object then in the database it will still have a lock_version of 4.

Rails appends “WHERE lock_version = 4” to the update query, and then examines the number of rows that were updated (the database driver will return this). If we get 1 then we know everything went ok. If 0 rows were updated then we know that it must have been because someone touched that row and so we raise StaleObjectError.

Optimistic looking is nice because we’re not sitting on a database lock at any point, so we won’t hold up anyone else.

Optimism only gets you so far

Lets move on to another function: allowing children to swap their magic stars for some sort of perk. We need to check that they’ve got enough stars and if they do debit the appropriate amount and give them their perk. This is one of those classic times where you want a transaction: you don’t want a classroom full of 6 year olds screaming because you took their stars but didn’t give them their reward. You might write

def buy treat
  transaction do  
    if magic_stars >= treat.cost
      self.magic_stars -= treat.cost
      self.treats << treat
      save!
    end
  end
end

[1]If something bad happens halfway through then the transaction will roll back all the changes together.

There’s a big problem with this code. Suppose the child tries to buy two things in very quick succession. They’ve got 10 stars to begin with and each item costs 5. If the timing is right, then things will look like this:

Connection 1	Connection 2
start transaction
Check number of stars	start transaction
set number of stars to 5	check number of stars
add the treat	set number of stars to 5
save	add the treat
	save
commit the transaction	commit the transaction

The 2 connections walk all over each other. In this case the children will be happy: both connections will set the remaining number of stars to 5, but both items will have been bought!

So why didn’t our optimistic locking help? Because of transaction isolation: in most databases by default your transaction won’t see changes made by other, uncommitted transactions. In fact you won’t see any changes made after your transaction was started. So when rails tries to save, as far as it can tell the row hasn’t changed and the save raises no errors.

Pessimism

The name optimistic locking suggests that there is an alternative, and there is: ask the database to get an actual lock for you. You can do this in two ways in rails: you can pass :lock => true to a finder (instead of passing true you can pass an sql fragment if you want to change the type of lock acquired) or you can call the lock! method (which is effectively a reload with :lock => true). It’s important to note that a lock is held as long as the current transaction. In particular if you aren’t inside a transaction then nothing happens, or in other words

def do_something_with_lock
lock!
do_something
end

doesn’t accomplish anything at all. Inside a transaction however, you’ll get an exclusive lock and no other transaction will be able to get a lock for that row. We can rewrite our buy method to look like

def buy treat
 transaction do
   lock!
   ...
  end
end

The lock stops another connection from fiddling with that person at the same time. It’s just a row lock, so it won’t stop you editing a different person at the same time[2]. If someone called our credit method from above at the same time that would be ok, as updating the customer row to change the amount of stars they have also requires an exclusive lock, so we’re in no danger here.

We can use these locks in more general ways, even if we don’t want to actually lock the customer row. For example if you have a constraint like a person may only borrow a certain number of books then you pretty much have to do it in ruby which renders you vulnerable to the various race conditions mentioned before.

You’re not modifying any rows (just adding one to a join table), so optimistic locking is no good. You can however lock the person row[3]. You’re not actually changing it at all, just using the database as a synchronisation method. We can repackage this up as something we can use all over our app:

def exclusive
  transaction do
    lock!
    yield
  end
end

Then in your app you can do things like

  some_customer.exclusive do
    #some task 
  end

It should be an easy exercise for the reader to put this in a form where any ActiveRecord model has this exclusive method.

[1]People sometimes ask about the difference between Person.transaction, the instance method transaction and ActiveRecord::Base.transaction. There is no difference between the first two – the instance method just callsself.class.transaction. The transaction method on Person, ActiveRecord::Base or some other model class only differ if the models use a different database connection. Most of the time, the three can be used interchangeably.

[2]If you use a database without row locks (ie mysql with myisam tables) you’ll probably get something horrible like a table lock.

[3]Do be careful about deadlocks though.

[4]The sleep function can be rather helpful when you’re trying to force particular bits of code to overlap with each other in particular ways.

Squeeze your pipes

2008-05-29T18:26:00Z

It’s very easy to merrily write a web application without realising that all those images and ajax calls you’re using make it rather sluggish when you’re not just connecting to localhost. Even when you’ve uploaded it to your production or test servers chances are you’ll have pretty good bandwidth and latency between you and those servers so it can be hard to see just what it will be like for an enduser who is less well endowed in the broadband department (or even still on dialup).

It’s not just about size

Very often it’s not just about how many bytes a second you could be transfering, the amount of latency is also a critical component of what the app feels like to the end user. Luckily we can simulate both.

The piece of kit we need is a traffic shaper, a piece of software that sits in between you and the end server and modulates the flow of packets. If you’re using linux or Mac OS X you’ve probably already got everything you need. I’m a mac nerd so I’ll concentrate on that.

Luckily it’s pretty damn easy on the mac as ipfw has all the necessary bits (from 10.4 upwards). The first thing we need to setup is a pipe. To quote the man page, “A pipe emulates a link with given bandwidth, propagation delay, queue size and packet loss rate”.


  ipfw pipe 1 config bw 300Kbit/s
  ipfw pipe 2 config bw 500Kbit/s delay 100

This creates 2 pipes: one with a bandwidth of 300Kbit/s, and the other with a bandwidth of 500Kbit/s and a propagation delay of 100ms (ie when a packet goes into that pipe it won’t come out the other end for 100ms)

Having setup the pipe, you then need to add firewall rules that send traffic through those routes. For example if you’ve got your mongrel running on port 3000 you can run


  sudo ipfw add pipe 2 tcp from any to any src-port 3000
  sudo ipfw add pipe 2 tcp from any to any dst-port 3000

to send traffic to/from port 3000 through pipe number 2. When you’re done with your testing or if you mess up, run


  sudo ipfw show

to show all the firewall rules you’ve created. The output will look a little like


00300    72    18982 pipe 2 tcp from any 4500 to any
00400    24     1896 pipe 2 tcp from any to any dst-port 4500
65535 58931 55276029 allow ip from any to any

The first column is the rule number (so 300 and 400 in my case). To remove the rules just run sudo ipfw delete rule-number. This isn’t the sort of thing I’d even begin to worry about at the beginning of the project, but worth keeping at the back of your mind.

Blogging is good for the soul

2008-05-28T19:15:00Z

This might not hold true for your vitriolic rants about your ex-girlfriend and similar, but blogging technical content is great. It’s good for the community in general when there are good howtos, explanations etc… and I’m sure a lot of people like the feel of being read by many people, but it’s not just that.

It’s often said that a true test of whether you understand something is whether you can explain it to someone else. Even if you were to erase a blog post the second you finished it, chances are you will have learnt something. I always do, even if I’m writing about something I already know well. Putting something in the form of a coherent piece of writing pushes me to explore all the edge cases and ramifications which, even I was aware of, hadn’t fully thought through before.

So go and write something interesting and do yourself and the community some good. Pick something interesting you worked on recently, a common question or misunderstanding, anything will do!