Split Testing Results for Bingo Card Creator
This page displays automatically recorded A/B testing results for Bingo Card
Creator. Some of them I thought were interesting enough to do a little mini-writeup
for. The other ones are minor tweaks or failed experiments. I mostly keep them
here a) to have a convenient interface to check on my tests-in-progress and b)
to remind folks that most A/B tests fail. That's the entire point of
A/B testing: failing your way forward.
Stats geeks: All stats are delayed by up to 24 hours. If you have a burning
urge to see the numbers, mouse over any percentage on this page and it will show
you how it was calculated.
Sign Up Page Banner: On Theme or Not?
I have a page which hawks a Halloween bingo activity that has a graphical design distinct from that of this site. I tested the control signup page, which uses this site's branding, against an alternative which uses elements of the mini-site branding to reassure users that they are in the same place.
Desired conversion: A free trial sign-up
Results:
Halloween header: 34.44% winner
Standard header: 32.49%
Significance: Not significant
A/B Test Results (Live)
| Test Name | Alternatives | Significance? |
|---|---|---|
| Quick Test For Jeff | Two(0.34%) winner One(0.31%) |
Not significant |
A/B Test Results (Completed)
| Test Name | Alternatives | Significance? |
|---|---|---|
| Default Cards Printed To Maximum Fixed | No(82.25%) winner Yes(79.18%) |
95% |
| Default Cards Printed To Maximum | No(0.00%) winner Yes(0.00%) |
Not significant |
| User Card Limit May 2012 | 8(1.89%) winner 15(1.04%) |
99% |
| Index Page Try Button Microcopy May 2012 | Me Focused(13.49%) winner Customer Focused(10.46%) |
99.9% |
| Landing Page Aida Test | Try Online(28.40%) winner Aida Landing Page(27.00%) |
Not significant |
| Big Bang Redesign | Default(0.19%) winner Redesign(0.19%) |
Not significant |
| Strongly Worded Warning On Download | No(95.34%) winner Yes(93.98%) |
Not significant |
| Registration Page Headline | Get Your Free Bingo Cards(35.59%) winner Sign Up For Bingo Card Creator(34.28%) |
95% |
| Offer Credit Cards As Sole Checkout Option | No(11.66%) winner Yes(10.34%) |
90% |
| Purchasing Page To Send People To | Classic(1.85%) winner Cc Checkout(1.38%) |
90% |
| Offer Credit Cards As Checkout Option | No(8.80%) winner Yes(7.12%) |
Not significant |
| Help Users Achieve Happiness 2 | No(95.52%) winner Yes(89.77%) |
99% |
| Help Users Achieve Happiness | No(94.24%) winner Yes(90.00%) |
90% |
| Hacker School Ab Example | Create Your Own Bingo Cards Now!(1.41%) winner Make Your Own Bingo Cards!(1.01%) |
90% |
| Smashing Magazine Says 16 Pt Body Font | Smashing(0.11%) winner Default(0.11%) |
Not significant |
| Checking For Single Card Print Jobs 2 | False(83.55%) winner True(78.25%) |
99% |
| Checking For Single Card Print Jobs | False(0.00%) winner True(0.00%) |
Not significant |
| Conversion Buttons Css Styling 2 | Outsourced Images(12.01%) winner Css(11.15%) |
Not significant |
| Conversion Buttons Css Styling | Outsourced Images(9.44%) winner Css(7.74%) |
90% |
| 10 Second Delay In Printing 2 | Yes(2.13%) winner No(2.13%) |
Not significant |
| 10 Second Delay In Printing | No(0.00%) winner Yes(0.00%) |
Not significant |
| Collect Emails On Landing Page | No(45.12%) winner Yes(42.35%) |
99.9% |
| Ask For Sale On Download Page | Yes(1.88%) winner No(1.55%) |
90% |
| Offer Halloween Discount | Yes(2.43%) winner No(2.41%) |
Not significant |
| H2 On Index Page | Enjoy Learning(0.00%) winner Less Work(0.00%) |
Not significant |
| Front Page H2 Test | Less Work(-nan%) winner Better Students(-nan%) |
(Too early) |
| Front Page Header | Create Your Own Bingo Cards!(0.00%) winner Make Your Own Bingo Cards!(0.00%) |
Not significant |
| Reassurity Active | Yes(8.39%) winner No(7.87%) |
Not significant |
| Landing Page Hide Header And Footer | No(28.65%) winner Yes(27.32%) |
Not significant |
| Identify Free Trial As Such | No(12.28%) winner Yes(11.93%) |
Not significant |
| Purchasing Options Table Border | false(10.09%) winner true(8.80%) |
Not significant |
| Google Checkout Offered | Inline(13.16%) winner None(7.87%) |
95% |
| Purchasing Page Sidebar Button | Off(8.96%) winner On(8.90%) |
Not significant |
| Purchasing Page Safety Logo Sidebar | On(9.21%) winner Off(8.46%) |
Not significant |
| Landing Page Menu | Off(28.52%) winner On(27.44%) |
95% |
| Trial Button Design | Try Now Online(15.30%) winner Try Online Free(14.50%) |
90% |
| Purchasing Page Safety Logo | Off(7.08%) winner On(6.34%) |
Not significant |
| Signup Form Button Creative (Ie Fixed) | Start Making Bingo Cards(36.05%) winner Bt Txt(35.65%) |
Not significant |
| Landing Page Button Creative (Ie Fixed) | Start Making Bingo Cards(26.72%) winner Bt Txt(25.38%) |
90% |
| Signup Form Button Creative | Bt Txt(36.82%) winner Start Making Bingo Cards(33.66%) |
90% |
| Landing Page Button Creative | Bt Txt(27.74%) winner Start Making Bingo Cards(25.80%) |
Not significant |
| Landing Page User Count | Off(28.32%) winner On(28.20%) |
Not significant |
| Landing Page Sidebar Wording 3 | Featured(27.40%) winner Included(25.47%) |
90% |
| Front Page Screenshot 2 | Redone(0.81%) winner Traditional(0.49%) |
90% |
| Site Logo No Bots 2 | Classic(11.05%) winner 99designs(9.92%) |
99% |
| Offer Downloadable Software 2 | On(0.26%) winner Off(0.24%) |
Not significant |
| Offer Downloadable Software | false(0.00%) winner true(0.00%) |
Not significant |
| Front Page Screenshot | Redone(0.00%) winner Traditional(0.00%) |
Not significant |
| Site Logo No Bots | 99designs(0.00%) winner Classic(0.00%) |
Not significant |
| Location Of Javascript | Head(0.01%) winner End Of Body(0.01%) |
Not significant |
| Site Logo | 99designs(1.03%) winner Classic(0.97%) |
Not significant |
| Allow Incentivized Invites | Off(4.43%) winner On(4.35%) |
Not significant |
| Trial Card Limit | 15(1.52%) winner 12(0.65%) |
95% |
| Landing Page Sidebar Wording 2 | Included(0.00%) winner Featured(0.00%) |
Not significant |
| Landing Page Testimonial 2 | Glenda(25.12%) winner Dave(23.38%) |
90% |
| Horizontal Conversion Button Styling | Traditional(0.31%) winner Redone(0.28%) |
Not significant |
| Landing Page Direct Ask | Next Page(23.39%) winner On Page(23.22%) |
Not significant |
| Landing Page Button Test 3 | Traditional(15.34%) winner Redone(12.60%) |
90% |
| Landing Page Sidebar Wording | Included(17.16%) winner Featured(16.27%) |
Not significant |
| Landing Page Testimonial | Glenda(17.27%) winner Dave(16.17%) |
Not significant |
| Color Conversion Button Landing Page | Blue(18.64%) winner Red(16.59%) Purple(14.80%) Orange(11.07%) Green(10.00%) |
(Too early) |
| Conversion Button Styling Landing Page | Traditional(17.01%) winner Redone(14.14%) |
95% |
| Registration Use Lightbox Landing Page | (16.28%) winner Ibox&Width=528(15.02%) |
Not significant |
| Registration Use Lightbox | Ibox&Width=528(0.35%) winner (0.34%) |
Not significant |
| Landing Page Sidebar Links | On(18.48%) winner Off(17.80%) |
Not significant |
| Landing Page Full Spiel | Off(18.65%) winner On(17.46%) |
Not significant |
| Conversion Button Styling | Traditional(0.79%) winner Redone(0.72%) |
90% |
| Shopping Cart Paypal Verbiage | Off(3.51%) winner On(2.06%) |
90% |
| Instant Access In Shopping Cart | Off(4.13%) winner On(1.32%) |
99% |
| Suggestions On Dashboard | No(63.00%) winner Yes(56.93%) |
90% |
| Redo Guest Test | Guests Off(0.36%) winner Guests On(0.31%) |
Not significant |
| Signup Madlibs Versus Standard | Standard(27.03%) winner Madlibs(21.58%) |
95% |
| Order Of First Operation Suggestions Fixed | Create From Scratch First(54.34%) winner Use Premade Card First(53.82%) |
Not significant |
| Main Flow Button Styles | Image(59.25%) winner Html(56.18%) |
90% |
| Offer Reward For Survey | Yes(2.24%) winner No(0.94%) |
99% |
| Offer Survey | Yes(1.07%) winner No(0.93%) |
Not significant |
| Order Of First Operation Suggestions | Use Premade Card First(55.06%) winner Create From Scratch First(53.83%) |
Not significant |
| Express Path For New Trialers | Dashboard First(58.55%) winner Skip Dashboard(53.75%) |
99.9% |
| Main Flow Submit Buttons | Image(59.81%) winner Html(58.67%) |
Not significant |
| Lp Featured Or Baby Shower | Featured(19.55%) winner Shower(18.93%) |
90% |
| Get The Code Link | italics(3.53%) winner bold(3.09%) |
Not significant |
| Ask For Sale On Print Page | false(0.42%) winner true(0.26%) |
Not significant |
| Let Users Choose Fonts | false(84.41%) winner true(81.33%) |
95% |
| Let Users Pick Colors | false(83.56%) winner true(82.19%) |
Not significant |
| Lightbox On Landing Page | false(0.00%) winner true(0.00%) |
Not significant |
