Community Strategy Insights

The latest insights on community strategy, technology, and value by FeverBee’s founder, Richard Millington

Run A Cohort Analysis, Not A Split Test

Richard Millington
Richard Millington

Founder of FeverBee

There are plenty of ways to make people jump through a hoop, but that doesn’t make a difference if they’ve stopped dancing a few minutes later.

This is the problem with using split tests within a community.

You can increase your conversion rates by amplifying the web copy or offering bonuses to people that register or participate.

But it’s only what happens over the long-term that really matters.

Consider this graph from a client below:

Cohort 1 (in blue), for example, begins with almost double the conversion rate of cohort 3 (in yellow).

But by week 18 this has fallen to a THIRD of cohort 3.

The big win here isn’t stopping the big drop-off at the beginning. You can do that with an array of one-off novelty ideas. Big drop-offs happen in every online community.

The big win is stopping the drop-off after around week 16 (from the beginning of the 12-week cohort). You can’t make a single tweak. Instead, you need to look holistically at the experience and make sure it’s a very fun or very relevant place to visit.

  • When people visit, are there always featured discussions taking place?
  • Were members @mentioned and included in discussions in a positive way?
  • Were there questions which relative newcomers felt informed enough to answer?
  • Were relatively newcomers encouraged and felt safe asking their own questions within the community?

Looking at the first registration or first participation metrics might seem like a smart move, but I’d focus on the post-participation experience. Turning a 0.7% to a 2.3% here (like we see above) has a HUGE payoff over the long-term.

Leave a comment

Your email address will not be published. Required fields are marked *

Subscribe for regular insights

Subscribe for regular insights