Run A Cohort Analysis, Not A Split Test

November 27, 2017 Comments Off on Run A Cohort Analysis, Not A Split Test

There are plenty of ways to make people jump through a hoop, but that doesn’t make a difference if they’ve stopped dancing a few minutes later.

This is the problem with using split tests within a community.

You can increase your conversion rates by amplifying the web copy or offering bonuses to people that register or participate.

But it’s only what happens over the long-term that really matters.

Consider this graph from a client below:

Cohort 1 (in blue), for example, begins with almost double the conversion rate of cohort 3 (in yellow).

But by week 18 this has fallen to a THIRD of cohort 3.

The big win here isn’t stopping the big drop-off at the beginning. You can do that with an array of one-off novelty ideas. Big drop-offs happen in every online community.

The big win is stopping the drop-off after around week 16 (from the beginning of the 12-week cohort). You can’t make a single tweak. Instead, you need to look holistically at the experience and make sure it’s a very fun or very relevant place to visit.

  • When people visit, are there always featured discussions taking place?
  • Were members @mentioned and included in discussions in a positive way?
  • Were there questions which relative newcomers felt informed enough to answer?
  • Were relatively newcomers encouraged and felt safe asking their own questions within the community?

Looking at the first registration or first participation metrics might seem like a smart move, but I’d focus on the post-participation experience. Turning a 0.7% to a 2.3% here (like we see above) has a HUGE payoff over the long-term.

Comment on this post at FeverBee Experts

©2019 FeverBee Limited, 1314 New Providence Wharf, London, United Kingdom E14 9PJ FEVERBEE

We’d like you to join us.

We’re a group of community professionals who push each other to the cutting edge of our work. Every day we share our experiences, highlight new expertise, and push one another to become the best community builders we can be.

The day you join, you’ll get access to 1000+ practical tips, dozens of resources, and unlimited access to the top experts in our space. You’ll also get to see how we go about building a community amongst our own audience.

It takes just 30 seconds and doesn’t cost a thing.


Level Price  
ALL COURSES The price for membership is $750.00 per month or $7,200.00 per year
Community Development Program The price for membership is $1,100.00 per month or $10,560.00 per year
Director of Community The price for membership is $8,770.00 per month