R – What’s the best way to A/B test Email Subject Lines in Rails


I'm sending out a lot of emails in my latest rails app. I want to A/B test subject lines in the emails that go out. ideally i can capture two things.

1- open rate
2- whether the call to action in the email is clicked

any ideas on how to do this? i don't think (the great) 7 minute abs (http://github.com/paulmars/seven_minute_abs/tree/master) will do this because @subject is set in the model, while the ab test param for the querystring is assigned in the view, and the test versions will likely be different.


Best Solution

The Vanity gem also includes A/B testing support for emails, see the documentation. It has built in support for splitting content and measuring results via tracking pixels (images).

For example, creating the experiment:

ab_test "Invite subject" do
  description "Optimize invite subject line"
  alternatives "Join now!", "You're invited to an exclusive event."
  metrics :open

Splitting users in the email:

class UserMailer < ActionMailer::Base 
  def invite_email(user)
    use_vanity_mailer user
    mail :to => user.email, :subject =>ab_test(:invite_subject)

And measuring the result via a tracking pixel:

      <h1>Hey Joseph</h1>
        <%= vanity_tracking_image(Vanity.context.vanity_identity, :open, :host => "") %>

[Disclaimer: I do help maintain the Vanity gem.]