If you want me to ruin your day, ask me if you’re measuring the right things.
I’ll probably ask you what you will do differently if any of your metrics go up or down.
Then I’ll probably try to say something profound like: “If you don’t know what you will do differently because of the data, it really doesn’t matter what data you collect”
Data analysis should be the most powerful tool in your community professional toolkit.
It should tell you exactly what to stop doing, keep doing, and how to fix problems in your community.
But almost no-one is doing this today.
We spend a lot of time during our Strategic Community Management course changing how participants think about measurement.
Four Levels of Using Data
In our benchmarks, we rate data skills on a range of 1 to 4.
You can see a simple breakdown of these below:
Almost everyone begins at a level 0 (not even measuring the right things). By the end of the course, we aim to get most people to level 3 or 4.
Here’s a breakdown of what each level looks like:
Level 0 – 1 – Measuring what happened
If you’re dropping monthly metrics into a big spreadsheet to send to your boss, you’re probably at level 0.
You’ve probably selected the metrics which are easiest to get instead of those which best reflect your work.
You’re also probably not presenting the data well or making any inferences about what the data is telling you.
Level 1 is usually the level of being able to pull data (automatically) from multiple sources into a single spreadsheet with simple to read dashboards. Here’s an example we put together for a client below:
This spreadsheet automatically pulls data from Google Analytics and runs SQL queries on the platform (along with some simple calculations) to complete a useful data set.
From this you can put together some simple dashboards to show what is and isn’t working like those below:
p.s. Just from these 4 (of 10) graphs, can you spot a few major problems already?
However, none of this data means anything if you have no idea what to do with it.
This means we need to understand why a metric we care about went up or down.
LEVEL 1 – 2 – Analyzing why it happened
Amazingly only a tiny fraction of people ever bothers to do this.
If growth is rising/falling or if activity is going up or down you need to know why.
Most people prefer to guess rather than analyze the problem in detail.
To analyze why a metric went up or down, you need a decision tree and additional data.
For example, imagine the number of posts in your community drops. There can be three possible reasons for this.
- Fewer members are posting.
- Members are making fewer posts.
- Both of the above.
So you need to check the average number of posts per active member and number of members who made a post in the past month.
Let’s imagine members are posting less. This is still interesting, but it’s not useful.
To make it useful you need to know which type of members are posting less.
Hence (as you see in the table above) we like to know the average number of posts from the top 1%, top 10%, top 50% and bottom 50% of active members etc…
You can equally segment this by the average number of posts by newcomers, irregulars, veterans etc…
Is there a drop across all categories (which suggests a technology/level of interest problem) or is it focused on a single group (i.e. has the mean number of monthly posts from the top 1% dropped over time?)
Now you can look specifically at top members.
Was there a sudden drop in their level of participation or has there been a steady decline (usually speeding up until it becomes noticeable)?
This tells you if the problem is an external event (technology or product change) or if it’s a steadily declining loss of interest.
If it’s an external event you can pinpoint the date and see what else changed around that time to fix it.
If it’s a steadily declining loss of interest you need to figure out if they are losing interest in the topic altogether or just with your community?
If they participate in other communities, you can ask what they like better about that community and either incorporate features or develop new ones to bring them back.
By the end of this stage, you should be able to make really specific statements like:
The level of activity in our community has dropped in recent months because our top members are participating less. This is due to them preferring to answer questions on Quora, StackOverflow, and other online communities which are easier to use and gives them more visibility than they get from our community.
Notice how specific and concrete that is? It also rules out a lot of possible solutions which won’t work.
Most people if they see engagement dropping panic and try a bunch of silly tricks to get people to participate more (AMAs, photos, live chats, gamification etc…!).
These are wild guesses which have no chance of fixing the problem.
Now you know exactly what’s broken, but you still don’t know how to fix it.
LEVEL 2 – 3 – How To Improve It
This is the critical step.
It doesn’t matter how great you are at analyzing the problem if you don’t know how to fix it.
This means we need to put together a few ideas of how to fix the problem and test them until we get an answer.
From the above, we would probably test ways to improve the ease of community and the visibility members get from writing blog posts, asking questions, and answering questions.
First, we would look at who has solved this specific problem really well.
Asking peers helps (p.s. please stop asking for help until you’ve properly diagnosed the problem) so does your own research into existing brand communities which have managed to keep highly engaged experts.
Because every situation is different, you need to draw up a list of possible solutions to test. For example, this might include:
- Let top members speak at our annual conference.
- Help get positive PR coverage for best member contributions in the trade press.
- Feature member contributions on Twitter, in our newsletter, and link to their solutions within product documentations.
- Let top members co-create on user guides which get sent to all members who use the product.
- Develop a plugin/tool to enable members to share their external contributions easily with the community.
(aside, as a rule, don’t aim for equivalency with a competitor. Always do something a competitor can’t match).
If you’re pushed for time, you can try all of these at once. But you’re unlikely to have the resources to do it (and even if you did you wouldn’t know which was working). So try to test each individually.
Over time, you can rule out options until you find the ones that work.
Remember that any change might take 3 to 6 months to show up on the dashboard.
LEVEL 3 – 4 – Designing An Ongoing System For Improvement
But what if there isn’t an immediate problem?
This process doesn’t just explain a way to fix problems, it also shows a way to constantly improve your community.
You can inverse this process to identify what’s working well, analyze why it’s working, and then doubling down on the areas which work best.
At the very top level, you want to be dedicating more and more resources to the areas which produce the best results.
This first means identifying whether a tactic achieved its impact and deciding whether there is more potential in that impact (i.e. more people who can be reached, a bigger change in behavior etc..) if you invest more resources in the tactic.
This means you need to design a process where each month (or every 3 months) you measure which tactics are working well/not working well and then decide whether to repeat, kill, tweak, or invest more in the tactic.
The purpose of this system is to gradually allocate more and more of your resources to the areas where they will have the biggest impact.
It doesn’t matter if you have 1 or 10 hours a day, you can still allocate your limited resources (and all resources are limited) to use them as effectively as possible.
Now you have a system in place which constantly improves your community as you collect more and more data.
Amateurs use intuition, professionals use data
I’ve seen organizations waste millions of dollars on migrating platforms and hiring new staff to fix problems which they haven’t properly analyzed.
I’ve seen community professionals waste months, even years, struggling to fix problems in their communities because they haven’t properly identified what’s going wrong or feasible solutions to fix this.
I’ve also seen far too many community professionals waste far too much time and energy on tactics which aren’t working.
These are all amateur level mistakes and we’re community professionals.
Amateurs use intuition and gut instinct, professionals use data and reason.
This is why data professionals are far more effective than amateurs.
This is how we get great results from our consultancy engagements, we turn this free asset into a powerful tool to diagnose and improve the community.
Beginning on January 28, we’re going to teach you how to do this.
I hope some of you will join us.