Case Study: Growing Mobile eBook Revenue +25% with A/B Testing

By testing different pricing options, MyBook was able to increase subscriptions to their paid monthly access to their digital library by 15% and overall cash flow by 25%!

About MyBook Digital Library

MyBook is a subscription-based digital library. Launched in 2012, the platform features over 50,000+ Russian language books and targets a Russian speaking audience. Continue reading

mobile content delivery

A/B Testing Web Services for Profit in a Mobile App

We were recently challenged to explore how A/B testing different web services that were sending data such as images, content and more to and from a native mobile application could work. The purpose was to devise a rigorous way to control and test the use of services so as to identify which were driving higher levels of commercial performance.

The mobile device is a highly distracting machine. Optimizing web services can reduce frictions and pave a smoother path for our customers to purchase from us before they encounter their next distraction.

-David Shreni, Director of Global Mobile Strategy at @WalmartLabs

Continue reading

Marks & Spencer logo

Case Study: Marks & Spencer Boosts Usage +20%

M&S Digital Labs – the digital innovation group of multinational retailer Marks & Spencer – has as it’s mission the giant task of evolving the status quo of retail. 

To turn that vision into reality, the team at M&S Labs has committed itself to a lean approach to everything they do – experimenting with new digital products, experiences and business models to make shopping easier and better for their customers – and looking to user feedback and data to make decisions about what works, and what doesn’t. 

Among the lean methodologies the team has adopted is mobile A/B testing.

Continue reading

A/B Testing

A/B Testing Best Practices

A/B testing refers to a specific type of randomized experiment in which a group of users (or participants) are presented with two variations of some thing (product, email, advertisement, landing page, whatever): a Variation A, and a Variation B. The group of users exposed to Variation A is often referred to as the control group because it’s performance is held as the baseline against which any improvement in performance observed from presenting Variation B is measured. Variation A itself is sometimes the original version of the thing being tested, and existed before the test. The group of users exposed to Variation B is called the treatment. When A/B testing is used to optimize a conversion rate, the performance of the treatment is measured against that of the control using the following calculations: Continue reading

A/B testing statistical significance

How much traffic is needed for an A/B test to achieve statistically significant results?

The answer to this question varies considerably based on what you’re trying to test, and how you’re trying to test it. A few important questions to ask yourself are:

What page or part of the site are you trying to optimize?
Most of your sites’ pages will receive only a fraction of site-wide visits, and in order not to to inflate sample sizes you’ll want to perform lazy assignment (read more on that – Lazy Assignment: Get More Done by Getting Lazy)You can dig into your site’s analytics data to get a better understanding of what you’re working with traffic-wise. Continue reading

Mobile App Analytics & A/B Testing

6 Metrics Mobile Apps Should Be Optimizing

Making a living from mobile apps is tough, and getting tougher. Last year, Gartner Research forecasted that 94.5% of all mobile apps will be free to download by 2017.


But the advent of mobile apps that are connected to the Internet has made for new opportunities to tap into data to build apps that are more delightful for users and more profitable for businesses. Thus far, the majority of marketers’ energies and budgets have been focused on boosting downloads. But in a world where free apps increasingly reign supreme, a redirection of resources will need to take effect towards post-download analytics.

Continue reading

Screen Shot 2014-08-05 at 5.12.41 PM

Taste Test! What Mobile Food Apps Should Be A/B Testing

Takeout. It’s been an integral part of American life since the invention of the Automat and its decidedly turn-of-century-ish slogan: Less work for Mother. Fast forward about 100 years, and takeout has been made as easy as a thumb tap on a mobile phone. It’s a brave new world, indeed! As we become busier (or maybe just lazier), the whole food ordering process has taken a mobile turn. It makes sense: Eating is often an on-demand thing. To snatch up the quickly emerging mobile food opportunity, all of your favorite food tech companies have been moving quickly to deliver seamless (wink wink) and delightful experiences no matter where you are. Continue reading


Empathic A/B Testing

In a recent blog post, GrooveHQ founder Alex Turnbull lamented the fact that many of his team’s A/B tests generate inconclusive results. Their experience is a painful one that is increasingly being felt among businesses of all shapes and sizes as A/B testing solidifies it’s place in the web marketing tool suite. Continue reading