The answer to this question varies considerably based on what you’re trying to test, and how you’re trying to test it. A few important questions to ask yourself are:
What page or part of the site are you trying to optimize?
Most of your sites’ pages will receive only a fraction of site-wide visits, and in order not to to inflate sample sizes you’ll want to perform lazy assignment (read more on that – You can dig into your site’s analytics data to get a better understanding of what you’re working with traffic-wise.
What is the minimum detectable effect you wish to observe?
It requires more data to obtain statistical significance in observing a small change than it does in a large one. For example, you’ll need more traffic to be confident in a 1% increase in conversions than a 10% increase (ceteris paribus). Setting a minimum detectable effect – or minimum effect size which you wish to observe – can help you get a higher Return On Data by allowing you to end a test with a smaller sample so long as it has been shown not to have an impact beyond your minimum requirements.
What level of significance are you willing to accept?
You’re going to need to decide what level of statistical significance is enough for you to be comfortable with declaring a winner. A 5% significance level is typically used for A/B tests – but for big changes that may potentially have a dramatic effect on your bottom line, you may want to achieve more confidence in the results before calling it. The lower the significance level, the more traffic is required to complete a test.
Check out this sample size calculation tool we built at Splitforce:. Once you get a better idea of your testing goals and relevant traffic, it can help you to gauge the potential costs and benefits of the tests you’re considering to run.