Run A Cohort Analysis, Not A Split Test

November 27, 2017 Comments Off on Run A Cohort Analysis, Not A Split Test

There are plenty of ways to make people jump through a hoop, but that doesn’t make a difference if they’ve stopped dancing a few minutes later.

This is the problem with using split tests within a community.

You can increase your conversion rates by amplifying the web copy or offering bonuses to people that register or participate.

But it’s only what happens over the long-term that really matters.

Consider this graph from a client below:

Cohort 1 (in blue), for example, begins with almost double the conversion rate of cohort 3 (in yellow).

But by week 18 this has fallen to a THIRD of cohort 3.

The big win here isn’t stopping the big drop-off at the beginning. You can do that with an array of one-off novelty ideas. Big drop-offs happen in every online community.

The big win is stopping the drop-off after around week 16 (from the beginning of the 12-week cohort). You can’t make a single tweak. Instead, you need to look holistically at the experience and make sure it’s a very fun or very relevant place to visit.

  • When people visit, are there always featured discussions taking place?
  • Were members @mentioned and included in discussions in a positive way?
  • Were there questions which relative newcomers felt informed enough to answer?
  • Were relatively newcomers encouraged and felt safe asking their own questions within the community?

Looking at the first registration or first participation metrics might seem like a smart move, but I’d focus on the post-participation experience. Turning a 0.7% to a 2.3% here (like we see above) has a HUGE payoff over the long-term.

Comment on this post at FeverBee Experts

©2023 FeverBee Limited, Apartment 2410, 251 Southwark Bridge Road, London SE1 6FN England FEVERBEE

We’d like you to join us.

We’re a group of community professionals who push each other to the cutting edge of our work. Every day we share our experiences, highlight new expertise, and push one another to become the best community builders we can be.

The day you join, you’ll get access to 1000+ practical tips, dozens of resources, and unlimited access to the top experts in our space. You’ll also get to see how we go about building a community amongst our own audience.

It takes just 30 seconds and doesn’t cost a thing.


Level Price  
ALL COURSES $750.00 per Month.
Community Development Program $1,100.00 per Month.
Director of Community $8,770.00 per Month.
Free Trial - Accelerator Program Free.
Community Accelerator (founding members) [annually] $250.00 per Year.
Community Accelerator and FeverBee Intelligence [monthly] $45.00 per Month.
Community Accelerator and FeverBee Intelligence [annually] $450.00 per Year.
Full Community Upgrade (Large Enterprise Only) [monthly] $750.00 per Month.
Full Community Upgrade (Large Enterprise Only) [annually] $7,600.00 per Year.