M&S Digital Labs – the digital innovation group of multinational retailer Marks & Spencer – has as it’s mission the giant task of evolving the status quo of retail.
To turn that vision into reality, the team at M&S Labs has committed itself to a lean approach to everything they do – experimenting with new digital products, experiences and business models to make shopping easier and better for their customers – and looking to user feedback and data to make decisions about what works, and what doesn’t.
Among the lean methodologies the team has adopted is mobile A/B testing.
“Data is important to us at M&S Digital Labs because you can’t really know or reason about anything without data; you can only have a hunch. Testing is a key part of the lean methodologies we use to build our products, and we believe that extensive testing leads to better products.
As a recent example we studied the analytics data from the Cook With M&S iOS app whilst we were debating which features to include in the Android version of the app. This allowed us to omit some features that users didn’t use, which ultimately meant we could get our MVP built and released quicker.”
-Neil Kimmett, Marks & Spencer Labs
Cook With M&S
Cook With M&S is a beautiful little app for iPhone and iPad with recipe ideas and supporting features to help at-home gourmands do some experimenting of their own!
Some innovative features like a shopping list which automatically adjusts to the number of people you’re cooking for, a full-screen cooking mode, in-app timers and push notifications of new seasonal recipes make Cook With M&S a standout favorite among users in the UK, Europe, US, Australia and UAE.
Test Goal: Increasing Usage of ‘Cooking Mode’
One particularly useful feature of Cook With M&S is the ‘Cooking Mode’, which allows users to view their recipe step-by-step, in full screen and distraction free while cooking. The recipe ingredients and cooking instructions are displayed as a nice white text on a charcoal background, making it easy on the eyes even during those teary onion-chopping moments!
In order to improve usability and ensure that the app’s users were discovering the convenience and delight of all it’s features, the team decided to take an experimental approach towards increasing the usage rate of cooking mode.
Test Goal: To increase the usage of cooking mode
Key Test Metric: Button Tap-Thru Rate (= Unique Taps / Users)
Test Hypothesis: The effects of call to action
Hypothesis: A CTA reading ‘VIEW FULL SCREEN’ will increase the button tap-thru rate.
Null Hypothesis: A CTA reading ‘VIEW FULL SCREEN’ will not increase the button tap-thru rate.
In marketing, a call to action (CTA) is an instruction intended to trigger an immediate action on behalf of the audience. CTAs often draw on the power of an imperative phrase, such as ‘Learn More’ or ‘Click Here’.
To access Cooking mode, the team at M&S Labs originally decided to place a white button directly under each recipe description in M&S signature teal, a cooking pot icon and a CTA reading ‘ENTER COOKING MODE’.
The team at M&S Labs believed, however, that their goal of increasing the usage rate of cooking mode could be achieved by altering the CTA on this button. Cooking mode is a term that is internal to the app – and so the team hypothesized that a CTA which referenced the more ubiquitous term ‘VIEW FULL SCREEN’ would bring about an increase in their key test metric.
Experimental Design: Traditional A/B Test
Using Splitforce, the team at M&S Labs was able to quickly setup a traditional A/B test in which the app’s users were each randomly assigned to one of two groups.
Group A: The control group, which received the ‘ENTER COOKING MODE’ variation.
Group B: The experimental group, which received the ‘VIEW FULL SCREEN’ variation.
This means that, for each user, there was a 50% probability of being included in either of the groups, as indicated by the ‘Split’ being set to ’50%’ for each variation.
In order to discover the best CTA for their entire user base taken as a whole, M&S Labs decided to set the sampling rate of this test to 100% and did not include any conditions defining a specific group of end users to take place in the experiment.
Test Results: +20% Bump in Usage
The team ran the test for just over four weeks, allowing them to gather a large enough sample size to generate statistically significant results and mitigate the risk of anomalous time-specific user activity.
The test covered a sample of over 19,780 users and indicated that the experimental group which received the ‘VIEW FULL SCREEN’ call to action showed a 19.56% higher button tap-thru rate than the experimental group. The nearly 20% difference observed over such a total sample size allowed us to reject the null hypothesis with virtually 100% confidence.
Result: The new experimental CTA reading ‘VIEW FULL SCREEN’ over-performed the control by 18%.
The M&S Labs team’s original instinct was to keep with the branding guidelines of the Cook With M&S product and have the CTA copy read ‘ENTER COOKING MODE’. While referring to this feature as cooking mode does more to build a distinct branded experience than the same ole’ full screen – this comes at the cost of usability, and ultimately adds friction to the process of discovering the cooking mode feature.
Full screen is technical term which has worked it’s way into the layman’s vernacular. The digitally literate are all familiar with setting slide presentations to full screen on their desktop computers, or videos to full screen on their laptops and tablets. There is therefore less cognitive effort required to understand what ‘VIEW FULL SCREEN’ means in the context of the app.
In contrast, ‘ENTER COOKING MODE’ is a term unique to M&S which is not as intuitive for users to understand – even despite being introduced to cooking mode in the app’s onboarding.
We concluded that the CTA copy has a significant effect on user experience. A/B testing different CTAs allowed M&S Labs to isolate the effect that changing this copy that would have on their goal of improving usage rates and discoverability of a key feature.
Without a rigorous approach to testing these CTAs, M&S Labs would have not gained these insights into the difficult balance between branding and usability. Moreover, had the team implemented the ‘VIEW FULL SCREEN’ variation without a proper A/B test – they would have experienced a boost in button tap-thru without being able to point to the cause with the confidence and backing of empirical evidence.
“Going forward we hope to use A/B testing to gain a deeper appreciation of the way people use our app, and investigate different ways of improving the user experience.
There are also other areas of the app we think could be improved; user testing has shown that not many people notice the “Filter” button, perhaps it could do with some new copy.”
-Neil Kimmett, Marks & Spencer Labs
We would suggest continuing to experiment with variations of the CTA copy that strike an even better balance between branding and usability, for example:
‘GET TO COOKING!’
What copy do you think will work best? Let us know and we’ll publish an update in due course.