mobile content delivery

A/B Testing Web Services for Profit in a Mobile App

We were recently challenged to explore how A/B testing different web services that were sending data such as images, content and more to and from a native mobile application could work. The purpose was to devise a rigorous way to control and test the use of services so as to identify which were driving higher levels of commercial performance.

The mobile device is a highly distracting machine. Optimizing web services can reduce frictions and pave a smoother path for our customers to purchase from us before they encounter their next distraction.

-David Shreni, Director of Global Mobile Strategy at @WalmartLabs

The Assumption

The technical performance of popular web services such as CDNs or APIs varies affects the time that it takes for a mobile app communication with such services to make use of the data being transmitted. These loading times in turn affect user behavior, which has a commercial cost in the form of shorter session lengths, lower conversion and retention rates, and higher rates of shopping cart abandonment.

The Question 

How can we apply the principles of randomized experimentation to control, measure and optimize for commercial gain the use of web services?

Technical Scenarios

We identified four different solutions to answer our question, each of which takes into account a distinct network architecture.

Scenario 1: The selection of web service(s) is hard coded into the app binary, and made at build-time.

In this scenario, we can represent the selection of a web service as an A/B test variation which calls a remote server when the app launches. One variation exists per web service – or combination of web services – and we can use event listeners to collect user behavioral data for each variation.

Scenario 2: The choice of web service(s) is made by the client at run-time with information from the server.

In this scenario, we can assume control of the selection logic from the current server and then this case becomes equivalent to Scenario 1.

Scenario 3: The choice of web service(s) is made by a server at run-time, transparently but detectably (i.e. IP address returned by the server could be used to determine which web service is being called), and a single choice is made for at least one full app session.

In this scenario, we can define segments of users dynamically at run-time based on information specific to a web service (IP address) and separate goal data by segment.

This is achieved by determining the web service being used and passing in a value which identifies that particular web service as a segmentation parameter.

Scenario 4: The choice of web service(s) is made by the server at run-time, transparently but detectably (i.e. IP address returned by the server could be used to determine which web service is being called), and a single choice is made per network request.

In this scenario, we would need to consider how combinations of services called within a single session are impacting goals. This can be achieved through a multivariate test design. To handle this scenario, we can tag goals with contextual information such as:

  • Fastest service used
  • Slowest service used
  • Fastest service speed
  • Slowest service speed
  • Load time for all assets from fastest service
  • Load time for all assets from slowest service
  • Total combined load time

The tagged goal data can then be visualized to show how overall load times affect the goal set, and aggregate the slowest load time contributions across web services.