Apptimize: From volume to value

Think back to 2015. Most mobile platforms had little to no ability to run A/B experiments. Even when they did, experience in experimentation was nominal. Experimental tools for the web had only started appearing in 2010, so you can imagine how new this was for native mobile applications.

This case study focuses on a key product experience that changed the way we measured success and how customers interacted with our platform: the experiment setup flow.

Tiles of different components

Misguided Assumptions

Originally, users would create an experiment by naming it, adding variants, targeting an audience, and launching. After launching, they could select their metric for success in the reporting page.

Representation of the userflow

But that’s not what actually happened. Our first flawed assumption was expecting customers to know how to plan an experiment. Instead of successful experiments, we received feedback ranging from “Why isn't there any change in engagement?” to “What do you mean I should set a goal?”

One customer even thought our application would automagically create new engagement for them!

Introducing Purposeful Friction

To address this, we purposefully created a point of friction in the setup flow. Now, setting a goal is step two in the journey. This step is also required, and cannot be skipped.

Would this create confusion or drop-offs? Possibly. And yes, it did, but users didn’t log off forever. They called us, asking for help, and that was exactly the point.

Representation of the new userflow

Fewer Experiments, Greater Success

The conversations shifted from complaints to questions like “How do I know what I should put as a goal?” and "How do I create an experiment that is successful?"

Our customer success team guided them, helped them learn the steps and research needed before running an experiment, and in some cases assisted in setting up their first real successful experiment, planning with them a hypothesis and all.

This training was brought to the sales team as well, so whenever the product was introduced, anyone in the company was now able to educate the customers on best practices and work with them through their goals and ambitions.

Snippet of results page that shows different measurements with description of what it measures and how

The result? Fewer experiments, but far more perceived value added to the customers. Complaints dropped, conversations around best practices increased, and customers learned to run experiments independently. By shifting our focus from volume to capability, we created a system where both our customers and our team were set up for long-term success.