A/Bingo vs. Alternatives

You're probably thinking "Why should I use A/Bingo when I could use something else?" For most Ruby on Rails developers, if you're thinking of something else, you're either thinking of Google Website Optimizer or, if you Googled [Rails ab testing] already, you may have heard of Seven Minute Abs.


A/Bingo vs. Google Website Optimizer

Google Website Optimizer (GWO) is a Javascript package which you can install to perform limited A/B testing. First, you define the parameters to your test in Google's web interface. Then, you copy paste code into your pages in three or four places. You read results in Google's interface.

GWO is superior to A/Bingo in one respect: it produces visualizations that non-programmers in your company will appreciate. That is the only good thing I feel compelled to mention about GWO, so I want to get it out of the way, first.

A/Bingo integrates faster and more cleanly. Making a new test in A/Bingo requires adding one line of Ruby code to your controller or view, as appropriate. Tracking the conversions requires one additional line. There is no four-page web-based workflow to go through to start a test, no tedious cut/pasting, and no stripping out code when a test needs to be disabled.

A/Bingo does not pollute your URL space. By default GWO requires you to put each alternative at a different URL. It then Javascript redirects some participants from the Control alternative URL to some of the experimental alternatives. This causes them to have load a page, execute Javascript, discard that page, load a new page, and then (finally) start interacting with your website. This skews test results for alternatives and is a terrible user experience. Also, you need to maintain that alternative URL for forever, in case someone has linked or bookmarked it. Google obsesses about factors like this on their own site, which is why Google does not use GWO internally on any site they actually care about. In comparison, Bingo Card Creator (my business) eats its own dogfood.

A/Bingo is not limited to tracking pages/page views. GWO conceptualizes alternatives as "different pages" and conversions as "the user has seen a certain page". A/Bingo lets you define what alternatives and conversions are. Do you run an online MMORPG? Want to make the alternatives starting player cash? Sure, one line of code. Want to make the conversions whether a player reaches level 5 or not? Sure, one line of code.

A/Bingo is not run by your most crucial source of traffic. Google is the most powerful business on the Internet. Google can use GWO data to whatever purpose they want. If Google removes your website from its indexes, you will see your traffic likely decline by more than half, instantly. Ponder that for a second. A/Bingo, on the other hand, is code by one guy out of an apartment in central Japan, and the data never leaves your server. I can't ruin your life if I don't like the data you send me.

A/Bingo is open source. You can audit that my code does what I say it does. GWO is a black box. If I get hit by a bus tomorrow, A/Bingo will keep working and you can maintain or patch it as needed. If Google does an A/B test and figures out that GWO is costing them more money than they want to lose, they can yank it from you instantly and there will be nothing you can do about it.


A/Bingo vs. Seven Minute Abs

Seven Minute Abs is a Rails A/B testing framework developed in March 2009 by Paul Mars. I am unaware whether it is considered under active development or not. As it is currently annointed by Google the best Rails A/B testing framework, I thought I would quickly address how A/Bingo is different:

  • A/Bingo and 7 Minute Abs have fundamentally different understandings about the nature of A/B testing.
  • A/Bingo does not require you to annotate URLs for conversion tracking.
  • A/Bingo makes heavy use of the cache for performance.
  • A/Bingo does statistical significance testing.
  • A/Bingo supports more types of alternatives, with more syntactical sugar.

Additionally, A/Bingo's syntax is designed for your productivity. Compare this example from the 7 Minute Abs documentation:

<% ab_test("linkcolor", 2) do |test,version|%>
  <% if version == 0 %>
    <%= link_to "Blue version", url_for(:ab => test.stub), {:style => "color:blue"} %>
  <% elsif version == 1 %>
    <%= link_to "Orange version", url_for(:ab => test.stub), {:style => "color:orange"} %>
  <% end %>
<% end %>

with the A/Bingo implementation.

<% ab_test("linkcolor", ["blue", "orange"]) do |color|%>
  <%= link_to "#{color.titleize} version", one_single_url, {:style => "color:#{color}"} %>
<% end %>

#Conversion code in your later action.

Once you realize how much money A/B testing makes you, you will want to run hundreds of A/B tests. A/Bingo takes the drudgery out of that.


I hold no malice in my heart towards Google Web Optimizer or Seven Minute Abs. Both of them were invaluable to me personally -- GWO for letting me cut my teeth on A/B testing, and Seven Minute Abs for showing me how to structure a Rails plugin. However, I'm a businessman and I won't mince words: A/Bingo is superior to them.