As you might have read previously on this blog, the Popsa team loves a growth experiment. We try and do as many as possible, even when we're doing a big project such as building our Android app.

I want to share the results of one of our recent experiments with you because it really does show that growth doesn't always come from the areas you spend the most time working on.

A bit of background

For many months we've had a 20% discount for new customers to help with their first order.

We talk about it on the website, in the App Store description, we visually show it in the App Store screenshots, there's another message after you install the app and we send you an email once you've signed in. All of that to make your first purchase easier. It helps us and it helps the customer. Everyone wins.

Or so that was the idea. Many people weren't actually winning. They got to the checkout having spent time picking their best photos only to then abandon.

There are many reasons why this could be, but it certainly wasn't a bad experience until that point. The vast majority of people who installed hadn't abandoned.

The easy way to fix it might have been to just lower our (already very competitive) prices; take a hit and make it harder to fund the exciting new features we're launching over the next few months. But it turned out - with a little experimentation (and a bit of psychology) - that there was a better way.

The Experiment

We ran a simple experiment to remind people of that 20% discount one last time. The difference here though, was that we showed its application, rather than just talk about it.

Have a look at this:

Before coming to the results, there are two important things to note:

1. I have normalised the data to remove the effects of the rapid growth in the number of installs we've seen this Summer. This means that each week in the chart below has been adjusted so that we only see the change this experiment has affected rather than that generated by our advertising improvements.
2. I've also abstracted away the absolute number of orders for business confidentiality reasons, but I can say that thousands of orders make up this graph.

I've used 100 as the pre-experiment baseline. Anything below this is contraction and anything above it is expansion (all by a proportionate amount).

As you can see, we've achieved a fairly seismic shift in the number of first-time customers as a result of the experiment, so much so that it makes our previous efforts look pathetic.

The post-experiment average (the green line on the right) is more than 13 standard deviations (or sigma for short) from the pre-experiment mean. Any change more than 4-sigma means there is a near-100% chance that we have a clear signal i.e. that it is not just background noise.

When we also take into account that the experiment covers many thousands of orders we can have complete confidence that the change is statistically significant.

All this means that the results are a runaway success and we have therefore decided to keep this as a permanent feature of the product as a result.

The incentive to buy was always there, we just made it even more frictionless.

Conclusions

It goes to show just how important regular experiments are not just for startups for long-established businesses too. At scale this will mean the difference of many millions in revenue and possibly countless more too...

Why? Because this change has not only generated more customers, but those people who might not have otherwise purchased now have a Popsa product in their hands. This has already encouraged at least half of them to order again and some of those will have been sent as gifts thus introducing us to more people for the first time.

So it's not only about the most advanced technology; sometimes it's the smallest details that matter most.