How To Use A/Bingo

This guide explains common usage patterns for A/Bingo. It assumes you have already successfully completed installation.

Starting a New A/B Test

A/B tests are always defined in code, and require no setup or configuration. The first time code for a particular test name executes, it does all the setup work for the test and logs the first participant. All subsequent times the test alternatives are read straight from cache, and the only DB access is to score the participant (a fast, optimized UPDATE). A single test name should never be used to refer to a second test.

A/B tests can defined in controllers or views, depending on whether you are testing program logic or merely display. The syntax is similar. Note that you can always pass a call to ab_test a block with a single variable -- this binds the test result to that variable. This is a programmer convenience for pretty, idiomatic Ruby code. 

#A controller example:
def registration
  #snipped
  #We're interested in whether this user goes on to buy new
  #credits or not, depending on how many freebies we give him.
  @user.credits = ab_test("starting_free_credits", [0,100,200], :conversion => "buy_new_credits")
  #snipped
end

#A view example without a block passed to ab_test.
#This lets you use the return value in a comparison, like "if" or "unless".
#Note an A/B test without alternatives defaults to
#flipping between true or false.
<% if (ab_test("show_mailing_list_signup")) do %>
  <p>Sign up for our email list!  <b>We don't spam!</b></p>
  <%= check_box_tag("wants_email", user.wants_email?) %>
<% end %>

#A view example with a block passed to ab_test:
<% ab_test("call_to_action", %w{button1.jpg button2.jpg}) do |button| %>
  <%= image_tag(button, :alt => "Call to action!" %>
<% end %>

#Note Rails 3 **requires** you to use the <%= block syntax, as below.  Rails 2 would raise an error if you tried this.
<%= ab_test("call_to_action", %w{button1.jpg button2.jpg}) do |button| %>
  <%= image_tag(button, :alt => "Call to action!" %>
<% end %>

You can also use A/Bingo in models: Abingo.test(test_name, alternatives)

All the test methods take an optional options parameter. Currently, the only supported option is :allow_multiple_participation. A/Bingo defaults to tracking an individual participant of a test only once. For example, if Bob refreshes your home page 100 times, he is counted as one participant in the login button test. If you turn multiple participation on, Bob would be counted 100 times, instead.

Back to the Table of Contents


Divider

Specifying Alternatives

A/B tests use two or more alternatives. (Currently, the built-in statistics module only provides significance testing for tests with exactly two alternatives.) You may specify alternatives via any of the following:

  • (nothing): A/Bingo sets you up with true and false as alternatives
  • an integer: N alternatives, from 1 to N
  • a range: every element in the range is an alternative
  • an array: every element in the array is an alternative
  • a hash of keys to values: The keys are the alternatives, the values are integral weights. For example, {:a => 3, :b => 2} will show :a 60% of the time and :b 40% of the time.

I discourage using weights because the implementation is slower than it should be. If you want to use them before I optimize it, keep the weights small ({:a => 3, :b => 2} is the same as {:a => 60, :b => 40} in terms of behavior, but it is much more efficient.)

Back to the Table of Contents


Divider

Testing A/Bingo

Obviously, with two alternatives of your display or functionality, you'll want to make sure both are functioning accurately. This can be sort of annoying to do. A/Bingo lets you cheat: if params contains a variable named the same as your test name, A/Bingo will override any automatically determined alternative with the contents of that variable. For example, appending &test1=foo to the URL in your browser will have A/Bingo return the alternative "foo" rather than doing its normal choice based on your identity. Note this is purely for testing: if you shortcircuit A/Bingo in this manner it won't collect any data for you.
This feature is disabled by default in production, because it could conceivably cause security issues. If you want to enable it, set Abingo.options[:enable_specification] = true
This feature was inspired by Assaf Arkin's Vanity testing framework, itself inspired by A/Bingo. Yay for open source!

Back to the Table of Contents


Divider

Multivariate Testing

This is an advanced topic. Feel free to skip it. Multivariate testing is a more advanced version of A/B testing, in which there are more than two alternatives. A/Bingo supports multivariate testing, but does not currently do statistical significance tests for it, so you'll have to run your own chi-squared tests.

Set up your test as seen above. There are a few ways to turn a simple array of alternatives into a powerful testing tool:

A/Bingo can serialize anything which is serializable by both ActiveRecord and the Rails cache. This means an alternative can be an array of hashes, for example. You can use this to link two (or more) experimental variables and see which combination is most effective. 

<% ab_test("multivariate_test1", [
    {:color => 'red', :text => 'Try now!},
    {:color => 'blue', :text => 'Try now!'},
    {:color => 'red', :text => 'Try later!},
    {:color => 'blue', :text => 'Try later!'}]) do |multi| %>

<span style="color: <%= multi[:color] -%>">
  <%= multi[:text] -%>
</span>

You can use this same technique to test different elements on different pages. Remember, for a given user and test name, the alternative chosen will always be the same. Thus, if you retrieve the alternative [:a, :c] on the first page in your funnel, you can be confident you'll see [:a, :c] again on the seventh page.

Smart use of this can help you "remember" e.g. promises made to the user so that they can be mentioned again later, or help test whether the order a user receives two bits of information matters, etc. (Is it better to tell them the price on page one and the money back guarantee on page two, or the other way around?)

Back to the Table of Contents


Divider

Scoring Conversions

Tracking conversions should probably be done in a controller, but may also be done in a view. Tests are only presently allowed to have one count of conversions associated with them. For example, a single change in copy on your home page can be tracked to see if it increases purchases or signups, but not both at statistics independently.

New feature: By default, A/Bingo assumes that a test is listening for conversions with the same name as its test name. You can re-use conversions across multiple tests by specifying them at test creation with the :conversion parameter. 

def purchase
  if (...) #if user actually paid us money
    bingo!("purchase")
    #Presumably, the user gets something, too.
  end
end

Back to the Table of Contents


Divider

Seeing Results

Most people will want to build their own dashboard for stakeholders to see, but A/Bingo ships with a default one to get you started. The dashboard also includes the ability to turn off tests through your browser. 

#Create a new controller.  The name is up to you -- this example
#uses abingo_dashboard_controller.rb
class AbingoDashboardController < ApplicationController
  #Declare any before_filters or similar which you need to use your authentication
  #for this controller.

  include Abingo::Controller::Dashboard
end

#You need to add the following to routes.rb:
  map.abingoTest "/abingo/:action/:id", :controller=> :abingo_dashboard

#Now, simply navigate to http://www.example.com/abingo and you'll see your dashboard.
#The templates can be customized to your liking in plugins/abingo/views.

Back to the Table of Contents


Divider

Ending Tests

There are three ways to end tests:

  1. Just delete the code and redeploy. No muss, no fuss.
  2. Use the dashboard to forcibly end a test. This will cause all users to see the alternative you specify on the dashboard. (This is often a prelude to later ripping the code out, but doesn't require a server redeploy.)
  3. Call Experiment#end_experiment!(alternative_contents) on the experiment from your own code.

Back to the Table of Contents


Divider

Statistical Significance

One key benefit of using A/B testing is that it provides hard numbers. However, numbers can be deceptive: even if A is currently showing as better than B, this could be simply due to random chance. Significance testing tells you when A is highly likely to be better than B, rather than merely better due to luck.

A/Bingo comes with one statistical test built in: the one-tailed Z-score test. If that sounds like Greek to you, you can read up on the subject. After you understand the rationale, feel free to poke around the implemenetation in Abingo::Statistics so that you can incorporate it into your custom reports and dashboards. There is a method that produces quick-and-dirty human-readable summaries to show you how it is done. 

#Showing this in the console for simplicity.
#You can also see it in the built-in dashboard.
experiment = Abingo::Experiment.find_by_test_name("login_page_redesign")
experiment.describe_results_in_words

If you wish to see the results of this method, look on the sidebar for the Get the Code link. There is a bit of an easter egg hidden behind it.

You can also easily export your data to any statistical package or custom code you prefer.

Back to the Table of Contents


Divider

Managing Identity

A/Bingo requires that you assign each user of your website an identity, which is just a persistent, unique string. This can be done very simply, but it has significant implications for user experience and tracking results.

The simplest solution boils down to "pick a random number, stuff it in a cookie, done". However, this means that User Bob will have a different identity at work and at home, and accordingly will potentially see different versions of your website from the two locations. That may be undesirable, for example if you are A/B testing prices or features in Bob's account. 

#This is the simplest code that works.
#Goes in application.rb

before_filter :set_abingo_identity

def set_abingo_identity
  if session[:abingo_identity]
    Abingo.identity = session[:abingo_identity]
  else
    session[:abingo_identity] = Abingo.identity = rand(10 ** 10).to_i.to_s
  end
end

A better solution is to integrate Bob's A/Bingo identity (which the user need never see, incidentally) with your identity management you use for login and the like 

#Assumes @user has already been set somewhere
#Also goes in application.rb
before_filter :set_abingo_identity

def set_abingo_identity
  Abingo.identity = @user.id
end

However, this breaks down for one critical use case: tracking conversions from random website visitor to registered member of the site. They go from having no identity (or, if you were clever, a random identity) to having an identity, meaning that any conversion funnels lasting over the signup step suddenly break. To get around this you can be tricky, for example by saving their randomly assigned anonymous identity in their account at signup, and then referring to it later. 

#Assumes @user has already been set somewhere
#else for logged in users.
#Also goes in application.rb
before_filter :set_abingo_identity

def set_abingo_identity
  if @user
    Abingo.identity = @user.abingo_identity
  else
    session[:abingo_identity] ||= rand(10 ** 10).to_i.to_s
    Abingo.identity = session[:abingo_identity]
  end
end

#tie in anonymous identity with registration
#Actual registration logic not shown
def register_new_user
  @user = User.new(...)
  if session[:abingo_identity]
    @user.abingo_identity = session[:abingo_identity]
  else
    @user.abingo_identity = rand(10 ** 10).to_i.to_s
  end
  @user.save
end

Back to the Table of Contents


Divider