Frequently Asked Questions

Q: I get TypeError: can't modify frozen array when using A/Bingo in production or testing modes. This happens as soon as the user sees their second distinct A/B test. What gives?

A: If you don't set Abingo.cache it will use the Rails cache. By default, Rails uses a MemoryStore. Due to a bug in Rails prior to version 2.3, this cache store freezes all objects stored inside of it, so that after being taken from the cache they cannot be modified. Either upgrade your Rails to 2.3 or later, or use a MemcachedStore in production/testing. I strongly recommend that anyway.

Q: I am used to A/B testing with Javascript. Rails is nice and all, but I crave reliability issues, page load impacts, and cross-browser compatibility problems. Can I track events from Javascript?

A: Yes! We make things easy, but you can make things hard again! Simply make an AJAX call to a controller method, and either load your alternative or track your conversion there.
#Call this action via AJAX to fetch
#whether a new player starts as a wizard
#or a warrior.  Do something with that in JS.
def default_starting_class_for_new_player
  newbie = ab_test("default_starting_class", [:wizard, :warrior])
  render :text => newbie, :layout => false
end

#Call this action via AJAX to track
#opens of JS shopping cart.
def opened_shopping_card
  bingo!("shopping_cart_test")
end

Q: I need to test conversions which happen off my site. Can you do tracking pixels?

A: Yes! Party like it is 1996! Just embed a unique ID in their image tag, and then process the request for that image in Rails.
#routes.rb
map.trackingPixel '/tp/:id.gif',
  :controller => 'conversion', :action => 'tracking_pixel'

#conversion_controller.rb
def tracking_pixel
  Abingo.identity = params[:id]
  bingo!("your-test-name-here")
  #There are better ways to do this next line.
  #But it shows the general concept.
  send_file "public/images/tracking_pixel.gif"
end

Q: Your conversion tracking doesn't match with my analytics/logs/etc. What gives?

A: A/B testing is not about accurately measuring conversions. A/Bingo might look like it gives you exact numbers for participants and conversions, but these are really best guesses, which could get screwed up by your cache resetting, bugs in your code, weird user behavior, intermediate caches, clearing cookies (if you store user identity in a cookie), etc etc.

This is also true of every stats package you use. However, accuracy doesn't matter for A/B testing, because as long as weird errors do not correlate with your experiment alternatives they come out in the wash as statistical noise. In practice, this is a safe assumption.

Q: Can I run two tests at once? Does that invalidate my stats?

A: Your college stats professor wishes to remind you that running more than one A/B test at once without guarantees that they are independent of each other runs the risk of interaction between experiments, potentially throwing off the results. Thus, there is no validity to using a one-tailed Z-score test and the numbers might as well be plucked from thin air.

OK, practical advice time? Your college stats professor is right. He also is giving impractical advice. Realistically speaking, unless you set out to test strongly correlating features, you are unlikely to have any problems. Don't take my word for it, some college stats professors agree with me. See section 5.1.3 of the PDF.

Q: Do you do pretty funnel visualizations?

A: A/Bingo plays along very well the Rails Mixpanel API that I wrote. Mixpanel generates fantastic funnel visualizations. Accordingly, I'm not going to reinvent the wheel.

All you need to do is change the code in that article a bit to synchronize your Mixpanel and A/Bingo unique identifiers, then pass the A/B tests you are running as permanent properties (Mixpanel calls them Super Properties) to Mixpanel.

Then, log into Mixpanel, open your favorite funnel, and segment on the A/B test. Voila! Sample code below.
#Shows the registration dialog
def registration
  #Business logic omitted for brevity.

  #Assign the user to be shown or not shown the guest login link.
  @show_guest_signin = ab_test("show_guest_signin")
  log_funnel("Trial Signup", 1, "Registration Form", {},
    {:show_guest_signin => @show_guest_signin})
  render :template => 'user/registration', :layout => false
end

#And in the view
<% if (@show_guest_signin) %>
  <%= link_to("Sign in as guest", login_as_guest_url) %>
<% end %>

#Conversion tracking
def process_registration
  #40 lines of business logic snipped for brevity
  if (@user.save)
    #Automatically associated with split test alternative
    #since we made it a persistent property.  Booyah.
    log_event("registration")

    #Ditto for funnels.
    log_funnel("Trial Signup", 2, "Registration Complete")

    #Track the conversion on my side of the fence, too.
    #Mixpanel doesn't do statistical significance tests.
    #Yet, anyway.
    bingo!("show_guest_signin")

    redirect_to :action => 'welcome'
  else
    ...
  end
end

Q: How do I block robots from participation in A/B tests?

A: By default, A/Bingo treats robots (such as Googlebot) as first-class citizens, meaning they get consistent participation in A/B tests, and each of Google's millions of crawling instances gets unique results. This will tend to have participants well in excess of the number of humans who come to your site (if your A/B tests are publicly visible and not behind a login), and since bots typically do not convert, that could conceivably make tests look less decisive than they are.

There is a quick fix for this: detect bots and then coerce their Abingo.identity to a constant value. This means that all bots will share the same test slot, minimizing their effects on your tests, and saving you from having to maintain state for tens or hundreds of thousands of phantom visitors.
#This goes wherever you are setting your Abingo.identity for incoming requests.
if (request.user_agent =~ /\b(Baidu|Gigabot|Googlebot|libwww-perl|lwp-trivial|msnbot|SiteUptime|Slurp|WordPress|ZIBB|ZyBorg)\b/i)
  #This prevents robots from occupying more than 1 participant slot in A/B tests.
  Abingo.identity = "robot"
else
  Abingo.identity = whatever_your_previous_logic_was
end
However, this poses a problem: not all bots identify themselves as such. Thus, if one is doing tests on publicly accessible pages, it is handy to have the user "prove" they are not a bot. One way is by forcing them to execute Javascript. Currently, only state of the art bots (like Googlebot) do full execution of Javascript, because it is pretty freaking hard at scale.

A/Bingo can defer counting of participation and conversions until after a user has proven they are human. You can implement your own heuristic for this, or use the provided one: execute the abingo_mark_human method, which proves that the user was capable of taking two numbers and adding them via Javascript. It won't stop a determined bot, but it will cut out 99.9% of your bot problems. (Another option is calling Abingo.human! any time you're sure the current Abingo.identity is a human -- for example, after they log in. You only have to do it once, but doing it multiple times won't hurt anything.)

This feature will cause one extra HTTP request, which shouldn't block browsers, but does cost the user and the server resources. Hence, it is off by default. Turn it on only if you are concerned about the impact of bots on tests you do on publicly accessible pages. To turn it on, set Abingo.options[:count_humans_only] = true in your environment files.
#This goes in your application layout, right below the end of the body.
<%= include_humanizing_javascript('/some_url', :prototype) > #The parameter to call and whether you use :prototype or :jquery.

#An alternative, if you know all requests are dynamic.
<%= include_humanizing_javascript('/some_url', :prototype) unless Abingo.is_human? >

#This goes in your routes.rb file
map.connect '/some_url', :controller => 'stat', :action => 'mark_human'  #Controller and action can be whatever you want.

#This goes in whichever controller and action you picked.
#I assume a before_filter has already set Abingo.identity properly.  If not, do so before abingo_mark_human.
def mark_human
  abingo_mark_human #Slurped into your controller automatically for you.
end

#You almost certainly want this if CSRF protection is enabled in your application.  Otherwise,
#you'll lose session contents after the Javascript calls mark_human with a POST action.
skip_before_filter :verify_authenticity_token, :only => :mark_human