Split Testing Results for Bingo Card Creator

This page displays automatically recorded A/B testing results for Bingo Card Creator. Some of them I thought were interesting enough to do a little mini-writeup for. The other ones are minor tweaks or failed experiments. I mostly keep them here a) to have a convenient interface to check on my tests-in-progress and b) to remind folks that most A/B tests fail. That's the entire point of A/B testing: failing your way forward.

Stats geeks: All stats are delayed by up to 24 hours. If you have a burning urge to see the numbers, mouse over any percentage on this page and it will show you how it was calculated.


Divider

Sign Up Page Banner: On Theme or Not?

I have a page which hawks a Halloween bingo activity that has a graphical design distinct from that of this site. I tested the control signup page, which uses this site's branding, against an alternative which uses elements of the mini-site branding to reassure users that they are in the same place.

Desired conversion: A free trial sign-up
Results:
Halloween header: 34.44% winner
Standard header: 32.49%
Significance: Not significant


Divider

A/B Test Results (Live)

Test Name Alternatives Significance?

A/B Test Results (Completed)

Test Name Alternatives Significance?
Default Cards Printed To Maximum Fixed No(82.25%) winner
Yes(79.18%)
95%
Default Cards Printed To Maximum No(0.00%) winner
Yes(0.00%)
Not significant
User Card Limit May 2012 8(1.89%) winner
15(1.04%)
99%
Index Page Try Button Microcopy May 2012 Me Focused(13.49%) winner
Customer Focused(10.46%)
99.9%
Landing Page Aida Test Try Online(28.40%) winner
Aida Landing Page(27.00%)
Not significant
Big Bang Redesign Default(0.19%) winner
Redesign(0.19%)
Not significant
Strongly Worded Warning On Download No(95.34%) winner
Yes(93.98%)
Not significant
Registration Page Headline Get Your Free Bingo Cards(35.59%) winner
Sign Up For Bingo Card Creator(34.28%)
95%
Offer Credit Cards As Sole Checkout Option No(11.66%) winner
Yes(10.34%)
90%
Purchasing Page To Send People To Classic(1.85%) winner
Cc Checkout(1.38%)
90%
Offer Credit Cards As Checkout Option No(8.80%) winner
Yes(7.12%)
Not significant
Help Users Achieve Happiness 2 No(95.52%) winner
Yes(89.77%)
99%
Help Users Achieve Happiness No(94.24%) winner
Yes(90.00%)
90%
Hacker School Ab Example Create Your Own Bingo Cards Now!(1.41%) winner
Make Your Own Bingo Cards!(1.01%)
90%
Smashing Magazine Says 16 Pt Body Font Smashing(0.11%) winner
Default(0.11%)
Not significant
Checking For Single Card Print Jobs 2 False(83.55%) winner
True(78.25%)
99%
Checking For Single Card Print Jobs False(0.00%) winner
True(0.00%)
Not significant
Conversion Buttons Css Styling 2 Outsourced Images(12.01%) winner
Css(11.15%)
Not significant
Conversion Buttons Css Styling Outsourced Images(9.44%) winner
Css(7.74%)
90%
10 Second Delay In Printing 2 Yes(2.13%) winner
No(2.13%)
Not significant
10 Second Delay In Printing No(0.00%) winner
Yes(0.00%)
Not significant
Collect Emails On Landing Page No(45.12%) winner
Yes(42.35%)
99.9%
Ask For Sale On Download Page Yes(1.88%) winner
No(1.55%)
90%
Offer Halloween Discount Yes(2.43%) winner
No(2.41%)
Not significant
H2 On Index Page Enjoy Learning(0.00%) winner
Less Work(0.00%)
Not significant
Front Page H2 Test Less Work(-nan%) winner
Better Students(-nan%)
(Too early)
Front Page Header Create Your Own Bingo Cards!(0.00%) winner
Make Your Own Bingo Cards!(0.00%)
Not significant
Reassurity Active Yes(8.39%) winner
No(7.87%)
Not significant
Landing Page Hide Header And Footer No(28.65%) winner
Yes(27.32%)
Not significant
Identify Free Trial As Such No(12.28%) winner
Yes(11.93%)
Not significant
Purchasing Options Table Border false(10.09%) winner
true(8.80%)
Not significant
Google Checkout Offered Inline(13.16%) winner
None(7.87%)
95%
Purchasing Page Sidebar Button Off(8.96%) winner
On(8.90%)
Not significant
Purchasing Page Safety Logo Sidebar On(9.21%) winner
Off(8.46%)
Not significant
Landing Page Menu Off(28.52%) winner
On(27.44%)
95%
Trial Button Design Try Now Online(15.30%) winner
Try Online Free(14.50%)
90%
Purchasing Page Safety Logo Off(7.08%) winner
On(6.34%)
Not significant
Signup Form Button Creative (Ie Fixed) Start Making Bingo Cards(36.05%) winner
Bt Txt(35.65%)
Not significant
Landing Page Button Creative (Ie Fixed) Start Making Bingo Cards(26.72%) winner
Bt Txt(25.38%)
90%
Signup Form Button Creative Bt Txt(36.82%) winner
Start Making Bingo Cards(33.66%)
90%
Landing Page Button Creative Bt Txt(27.74%) winner
Start Making Bingo Cards(25.80%)
Not significant
Landing Page User Count Off(28.32%) winner
On(28.20%)
Not significant
Landing Page Sidebar Wording 3 Featured(27.40%) winner
Included(25.47%)
90%
Front Page Screenshot 2 Redone(0.81%) winner
Traditional(0.49%)
90%
Site Logo No Bots 2 Classic(11.05%) winner
99designs(9.92%)
99%
Offer Downloadable Software 2 On(0.26%) winner
Off(0.24%)
Not significant
Offer Downloadable Software false(0.00%) winner
true(0.00%)
Not significant
Front Page Screenshot Redone(0.00%) winner
Traditional(0.00%)
Not significant
Site Logo No Bots 99designs(0.00%) winner
Classic(0.00%)
Not significant
Location Of Javascript Head(0.01%) winner
End Of Body(0.01%)
Not significant
Site Logo 99designs(1.03%) winner
Classic(0.97%)
Not significant
Allow Incentivized Invites Off(4.43%) winner
On(4.35%)
Not significant
Trial Card Limit 15(1.52%) winner
12(0.65%)
95%
Landing Page Sidebar Wording 2 Included(0.00%) winner
Featured(0.00%)
Not significant
Landing Page Testimonial 2 Glenda(25.12%) winner
Dave(23.38%)
90%
Horizontal Conversion Button Styling Traditional(0.31%) winner
Redone(0.28%)
Not significant
Landing Page Direct Ask Next Page(23.39%) winner
On Page(23.22%)
Not significant
Landing Page Button Test 3 Traditional(15.34%) winner
Redone(12.60%)
90%
Landing Page Sidebar Wording Included(17.16%) winner
Featured(16.27%)
Not significant
Landing Page Testimonial Glenda(17.27%) winner
Dave(16.17%)
Not significant
Color Conversion Button Landing Page Blue(18.64%) winner
Red(16.59%)
Purple(14.80%)
Orange(11.07%)
Green(10.00%)
(Too early)
Conversion Button Styling Landing Page Traditional(17.01%) winner
Redone(14.14%)
95%
Registration Use Lightbox Landing Page (16.28%) winner
Ibox&Width=528(15.02%)
Not significant
Registration Use Lightbox Ibox&Width=528(0.35%) winner
(0.34%)
Not significant
Landing Page Sidebar Links On(18.48%) winner
Off(17.80%)
Not significant
Landing Page Full Spiel Off(18.65%) winner
On(17.46%)
Not significant
Conversion Button Styling Traditional(0.79%) winner
Redone(0.72%)
90%
Shopping Cart Paypal Verbiage Off(3.51%) winner
On(2.06%)
90%
Instant Access In Shopping Cart Off(4.13%) winner
On(1.32%)
99%
Suggestions On Dashboard No(63.00%) winner
Yes(56.93%)
90%
Redo Guest Test Guests Off(0.36%) winner
Guests On(0.31%)
Not significant
Signup Madlibs Versus Standard Standard(27.03%) winner
Madlibs(21.58%)
95%
Order Of First Operation Suggestions Fixed Create From Scratch First(54.34%) winner
Use Premade Card First(53.82%)
Not significant
Main Flow Button Styles Image(59.25%) winner
Html(56.18%)
90%
Offer Reward For Survey Yes(2.24%) winner
No(0.94%)
99%
Offer Survey Yes(1.07%) winner
No(0.93%)
Not significant
Order Of First Operation Suggestions Use Premade Card First(55.06%) winner
Create From Scratch First(53.83%)
Not significant
Express Path For New Trialers Dashboard First(58.55%) winner
Skip Dashboard(53.75%)
99.9%
Main Flow Submit Buttons Image(59.81%) winner
Html(58.67%)
Not significant
Lp Featured Or Baby Shower Featured(19.55%) winner
Shower(18.93%)
90%
Get The Code Link italics(3.53%) winner
bold(3.09%)
Not significant
Ask For Sale On Print Page false(0.42%) winner
true(0.26%)
Not significant
Let Users Choose Fonts false(84.41%) winner
true(81.33%)
95%
Let Users Pick Colors false(83.56%) winner
true(82.19%)
Not significant
Lightbox On Landing Page false(0.00%) winner
true(0.00%)
Not significant