Frequently Asked Questions

Q: I get TypeError: can't modify frozen array when using A/Bingo in production or testing modes. This happens as soon as the user sees their second distinct A/B test. What gives?

A: If you don't set Abingo.cache it will use the Rails cache. By default, Rails uses a MemoryStore. Due to a bug in Rails prior to version 2.3, this cache store freezes all objects stored inside of it, so that after being taken from the cache they cannot be modified. Either upgrade your Rails to 2.3 or later, or use a MemcachedStore in production/testing. I strongly recommend that anyway.

Q: I am used to A/B testing with Javascript. Rails is nice and all, but I crave reliability issues, page load impacts, and cross-browser compatibility problems. Can I track events from Javascript?

A: Yes! We make things easy, but you can make things hard again! Simply make an AJAX call to a controller method, and either load your alternative or track your conversion there.

#Call this action via AJAX to fetch
#whether a new player starts as a wizard
#or a warrior.  Do something with that in JS.
def default_starting_class_for_new_player
  newbie = ab_test("default_starting_class", [:wizard, :warrior])
  render :text => newbie, :layout => false
end

#Call this action via AJAX to track
#opens of JS shopping cart.
def opened_shopping_card
  bingo!("shopping_cart_test")
end

Q: I need to test conversions which happen off my site. Can you do tracking pixels?

A: Yes! Party like it is 1996! Just embed a unique ID in their image tag, and then process the request for that image in Rails.

#routes.rb
map.trackingPixel '/tp/:id.gif',
  :controller => 'conversion', :action => 'tracking_pixel'

#conversion_controller.rb
def tracking_pixel
  Abingo.identity = params[:id]
  bingo!("your-test-name-here")
  #There are better ways to do this next line.
  #But it shows the general concept.
  send_file "public/images/tracking_pixel.gif"
end

Q: Your conversion tracking doesn't match with my analytics/logs/etc. What gives?

A: A/B testing is not about accurately measuring conversions. A/Bingo might look like it gives you exact numbers for participants and conversions, but these are really best guesses, which could get screwed up by your cache resetting, bugs in your code, weird user behavior, intermediate caches, clearing cookies (if you store user identity in a cookie), etc etc.

This is also true of every stats package you use. However, accuracy doesn't matter for A/B testing, because as long as weird errors do not correlate with your experiment alternatives they come out in the wash as statistical noise. In practice, this is a safe assumption.

Q: Can I run two tests at once? Does that invalidate my stats?

A: Your college stats professor wishes to remind you that running more than one A/B test at once without guarantees that they are independent of each other runs the risk of interaction between experiments, potentially throwing off the results. Thus, there is no validity to using a one-tailed Z-score test and the numbers might as well be plucked from thin air.

OK, practical advice time? Your college stats professor is right. He also is giving impractical advice. Realistically speaking, unless you set out to test strongly correlating features, you are unlikely to have any problems. Don't take my word for it, some college stats professors agree with me. See section 5.1.3 of the PDF.

Q: Do you do pretty funnel visualizations?

A: A/Bingo plays along very well the Rails Mixpanel API that I wrote. Mixpanel generates fantastic funnel visualizations. Accordingly, I'm not going to reinvent the wheel.

All you need to do is change the code in that article a bit to synchronize your Mixpanel and A/Bingo unique identifiers, then pass the A/B tests you are running as permanent properties (Mixpanel calls them Super Properties) to Mixpanel.

Then, log into Mixpanel, open your favorite funnel, and segment on the A/B test. Voila! Sample code below.

#Shows the registration dialog
def registration
  #Business logic omitted for brevity.

  #Assign the user to be shown or not shown the guest login link.
  @show_guest_signin = ab_test("show_guest_signin")
  log_funnel("Trial Signup", 1, "Registration Form", {},
    {:show_guest_signin => @show_guest_signin})
  render :template => 'user/registration', :layout => false
end

#And in the view
<% if (@show_guest_signin) %>
  <%= link_to("Sign in as guest", login_as_guest_url) %>
<% end %>

#Conversion tracking
def process_registration
  #40 lines of business logic snipped for brevity
  if (@user.save)
    #Automatically associated with split test alternative
    #since we made it a persistent property.  Booyah.
    log_event("registration")

    #Ditto for funnels.
    log_funnel("Trial Signup", 2, "Registration Complete")

    #Track the conversion on my side of the fence, too.
    #Mixpanel doesn't do statistical significance tests.
    #Yet, anyway.
    bingo!("show_guest_signin")

    redirect_to :action => 'welcome'
  else
    ...
  end
end