A north star metric is a single metric that represents the success of your community.
Ideally, it’s a single metric all stakeholders can track to see if the community is delivering more value than in the past. If it goes up, you’re doing well. If it goes down, things aren’t going so well.
If you find the right metric, it’s a lot easier to get more support and build more awareness of the community. However, finding the right metric isn’t easy. Every metric has limitations and it’s not easy to find a metric which reflects your organisation’s unique goals and constraints.
In this post, I want to share some of our thinking about finding your north star and finding the perfect metric for your efforts.
Why Finding The North Star Is Painfully Difficult
Before we begin, it’s important to be mindful of why this process is painfully difficult. Most previous efforts to find a north star metric suffer from four problems.
1) They don’t reflect value. Many metrics show increases or decreases in activity, but not whether the activity is important or not. This leads to the ‘engagement trap’ problem.
2) Gaming the metrics. Many metrics are easy to game. This undermines the entire value of the metric. Good metrics always need to be accompanied by good judgment (and ethical behaviour).
3) They are driven by external factors. Many metrics change not as a result of community, but as a result of external factors outside of the community team’s control. You shouldn’t be held accountable for metrics you can’t directly influence.
4) They aren’t comparable to other metrics. It’s hard to know if the metric matters if it can’t be compared to any existing metrics. A community-focused metric is great, but how do you compare that to a return on marketing spend?
Be mindful that the challenge isn’t to find a flawless metric. There isn’t one. The challenge is to find out which flaws are acceptable and communicate these well to others.
We will go deeper into each of these flaws and make suggestions about which kinds of metrics might make sense.
The Journey Is As Important As The Destination
The process of finding the right north star metric is as important as the metric itself.
You shouldn’t come up with a metric by yourself. That’s a surefire way to set yourself up for endless years of arguing why the metric matters. Instead, you should treat this as a journey of discovery and close collaboration where you listen to the thoughts of others and bring people along on the journey.
Everyone should feel they were involved in finding the final metric. If you don’t do this, you might face a concerted effort of people to unpick the value later.
You can see a typical process to find the north star metric below:
At every stage of the process, you need to build greater understanding and keep people informed. This might begin with knowing if a precise dollar value is needed. But that’s far from the end of the journey.
Be mindful that this chart isn’t prescriptive – it’s not designed to cover every possible type of community (non-profits, partners, developers etc…), but it might be a good place to begin investigating the opportunities.
PART ONE – FINDING THE DOLLAR VALUE
Is A Precise Dollar Value Required?
The first challenge is to ascertain expectations. Perhaps the most important determination is whether you’re looking for a dollar value return or not.
The method to find a precise dollar value is completely different than if one that has a different goal.
If a dollar value is required, you need to develop and explain a methodology that leads to a precise calculation. The end result is a dollar value return which will likely be compared to other departments to determine if the community merits more investment.
Also, if a dollar value is required, you will need data not just from the community but from whatever systems store the spending and behaviour of customers. You need to combine community data with your CRM data.
If a dollar value is not required, you have more flexibility to design an approach that reflects the unique value of your community. If a dollar value isn’t required, it’s possible to determine value just with community data.
In this approach, you can find a metric that perfectly encapsulates the impact of the community – but might not assign a precise dollar value to the community. This is like the difference between measuring the value of a branding effort vs. a dollar ad spend. The former requires metrics that either show revenue driven by ads or a change in feelings and perceptions (usually acquired by questionnaires).
You begin the process then by answering some critical questions.
These questions include:
- Who will be looking at the metric?
- What do they want to know about your community?
- How will they be using the metric?
- What will good or bad look like?
Once you’re armed with answers to these questions, you can begin to figure out what kind of metric might make the most sense. Crucially, are people looking for a precise dollar value or a broader sense of impact?
For now, let’s assume a dollar value is required.
Understand The Purpose
The next step is to determine the purpose(s) of the community. We’re not talking about specific goals here, but a broad understanding of why the community exists. This can be singular or there might be multiple reasons. Often the answer is suggested by whichever department the community reports to (which is why shifting departments is such a big deal, the purpose of your community will also change).
Some simple questions to ask here might include:
- Which department does the community report to (and why?)
- Who was behind the community’s launch and what was expected?
- What are the expectations stakeholders have of the community?
If no one seems to know (or seems reluctant to own the decision), it’s often a good idea to host a stakeholder workshop to gain some sense of alignment. This is where you guide members through the process of prioritising what is the best match for a community.
You can see a typical outcome of the process below.
You don’t need specific metrics at this stage, but you should have a fairly clear purpose (or set of purposes) to define how the community helps. Is it support? Attract new customers? Retention? Or some other benefit?
The Four Methods Of Finding A Dollar Value
Once we know the purpose, we can start finding the right methodology to calculate community value to a precise dollar value.
You can see the four options below:
Each method has some obvious pros and cons. You have to decide between precision or simplicity, correlation or causation.
The former is a question of capabilities, the latter is a question of preferences.
Let’s go a little deeper into each method.
The gold standard is to run an experiment.
However, this rarely happens because experiments usually require:
- An analyst with experience in running experiments.
- 3 to 6 months to plan and execute.
- Removal of access (or not providing access) to a significant percentage of your audience.
If you don’t have the capabilities, resources, and support for this, it won’t be possible to run an experiment. This means you can only determine correlation, not causation. Also, be mindful that an experiment gives you data from a single point in time. If things change, this data might quickly become outdated.
Controlled experiments let you make statements like:
“The community causes members to spend/do [x] more than they otherwise would”
The second approach is to compare two groups (or more) by a specific condition.
The most common example is to compare spending or retention of members vs. non-members. Statistically, this isn’t too difficult to do. The catch is this provides you with correlation, not causation. This simply indicates there is a relationship between membership and the desired outcome, it doesn’t show which way the relationship goes (i.e. does retention drive engagement or engagement drive retention?).
Nor does it reveal whether there is a confounding variable that influences both variables as you can see below.
This doesn’t make the data redundant (and there are more complex calculations you can do to try and ascertain the unique impact of the community), but overall it’s a useful, interesting, signal.
This lets you make statements like:
“Community members spend/do [x] more than non-members (or comparative group)”
Note: You can (and should) be precise about how you’re comparing groups. For example, you can find groups of people who closely resemble one another before they join a community and then track their spending a year later etc…
Surveys and Polls
Surveys are a handy tool which can be used to determine the value of a community. For example, you can ask members about their spending before and after they join the community.
The upsides of using surveys and polls are obvious; they’re easy to set up and analyse the results.
However, surveys suffer from three common problems.
1) Getting enough responses. You usually need around 300 survey responses to have meaningful results. If we assume a 5% survey response rate, you must contact at least 6000 people.
2) Sampling bias. Surveys suffer from a sampling bias. The most active members are also those most likely to respond to survey requests. This means surveys tend to skew towards the happiest and highest-spending members. You need to use a quota system by activity level to gain an accurate picture.
3) Responses are often inaccurate. You can ask members questions like how much they purchase before and after the community (or even track purchases over time), but memories are often fuzzy and the answers might not be accurate (people might also give you the answers they think you want to hear).
This doesn’t mean surveys aren’t useful. For tracking trends over time, they are especially useful. They can also give you a precise value you can easily justify. For example, you can say something like:
“Members tell us they’re spending $343 per year because of the community”
But if you’re going to use them to calculate value just be mindful of the downsides as much as the upsides.
Assigning Values To Behaviors
The final approach is to assign values to member behaviours. There are two common ways to achieve this, regression analysis and alternative costs.
Regression Analysis To Assign Values To Behaviors
Let’s tackle regression (or multiple regression) analyses. This is where you look at how a change in one variable relates to another. So you might look at how changes in independent variables like asking questions, reading articles, and receiving answers affect a dependent variable such as additional sales or revenue.
This study will give you coefficients which let you assign a value to this behaviour. This helps you make statements like “We know members who ask a question spend $4 more”.
However, be mindful of the phrasing above. We’re not claiming asking a question causes the increase in spending. We’re simply asserting members who do this one thing also do this other thing. This is correlation, not causation.
This methodology offers a precise value that helps you estimate the potential value of the community and the key behaviours you might want to encourage more of.
You can see an example of this in our work with The Pragmatic Institute.
Perhaps a more familiar approach is to look at the comparative cost (i.e. what would it cost to achieve the same result if the community didn’t exist?).
The most common form of this is call deflection. This is how much it would cost to support the same number of customers if the community didn’t exist.
In the standard approach, you calculate the value of a view of an answered question and multiply it by the number of answered questions. This doesn’t mean these cost savings were realised (not many people actually reduce the size of their support team). But it gives a theoretical value to a community.
Other approaches include calculating the cost of acquiring customers, and traffic, or creating the same amount of content if the community didn’t exist. First, you find the baseline cost of the work today. Second, you determine the cost of a single incident of the behaviour happening in the community. Then you multiply this figure by the number of behaviours in the community.
This will you let assert an answer like ‘the community is potentially saving $7.04m per year in deflected tickets’
Which Approach Should You Use?
If each approach has significant advantages and disadvantages, how do you determine which is the best approach for you?
This depends upon your organisation. You shouldn’t develop the methodology alone because it might later be picked apart due to inherent flaws. If we know every approach has flaws, it’s good to determine which flaws are already acceptable to your organisation (or department) and select that approach.
This means it’s useful to investigate which methods are used to calculate the value of other activities within that function.
This might include questions such as
- How are dollar returns calculated for activities within that department today?
- Do they want to adopt a similar method for calculating community activity?
- What kind of formula or methodology are they expecting?
- How will the results be used in the decision-making process?
Find out which methodologies are already in use, and you can adopt a similar methodology to the community.
Once you know the methodology, you can zero in on the specific value metric.
Zeroing In On The Metric
It’s not possible to cover every possible metric, but it might help to cover some of the most common approaches below based on the methodology.
|Customer Support||Value Assignment||Call Deflection Method
(No. visitors to accepted solutions * % who state they received the answer * % who would otherwise have contacted support (* % in warranty) * avg. cost per ticket.
Monthly Contact Rate
No. members * (avg. contact rate of non-members - avg. contact rate of members) * avg. cost per contact.
|Controlled Experiment||Expose or withhold community from a group
Remove community access from one or more product groups and measure the increase or decrease on customer support tickets.
|Customer Retention||Paired Groups||Member vs. non-member retention rate
Avg. annual retention rate of members - avg. the annual retention rate of non-members * no. members * avg. value of a customer.
Trial completion rate
% trial completion of new community members - % trial completion of non-members * avg. value of completing trial (CLV) * no. new community members.
|Controlled Experiment||Invite one group, block another
Invite one group of prospects to join the community and hide the community from another. Compare the results after a fixed time frame.
|Customer Acquisition||Value Assignment||Track signups from the community
Calculating the avg. lifetime value of members who joined the community prior to becoming a customer.
Track conversions from the corporate site
Conversion rate of visitors to corporate site * visitors from the community.
Track direct sales
Direct sales of products and upgrades promoted to the community.
Track value of leads
Number of leads sourced from community sources * avg. value of a lead.
The right metric for your community might not be above. The key, however, is to engage with others to calculate the right metric for your organisation.
If you need help here, drop us a line.
PART TWO – IMPACT
If the value refers to a dollar return, impact refers to everything which isn’t a dollar return.
Much of the work of the community sits several levels beneath calculable value.
For example, if a salesperson uses the community to gather testimonials, how would you quantify the value of that? You can’t attribute the sale to the community, yet few organisations want to do a test comparing presentations with testimonials as opposed to those without. Yet, you know it has an impact.
If you’re not under pressure to show a dollar value return, you have a lot more flexibility to develop a method that truly reflects the impact of your community.
But be mindful of a potential problem here. The further you drift from a dollar value, the more likely it is people will question your metric. Everyone understands dollars, fewer people understand impact.
The challenge is to strike the balance between metrics that perfectly encapsulate the value of the community and those which will be accepted internally.
As a general rule, the more support there is for the community internally, the more you can design community-specific metrics.
This also serves as a good rule of thumb. The less internal support exists for the community, the more you need to skew the metrics in favour of value over impact.
Using Existing Benchmarks
The first step is to find out if there are already metrics which can be used to benchmark the community against.
These might not be dollar-value metrics. Instead, they might refer to things like member sentiment (or change in sentiment). In this case, you might compare a community by the quality of answers provided by members or the quantity of content created.
There isn’t any shortage of potential impact metrics you can use here. So begin by asking a few key questions for whatever department or purpose(s) your community serves. These questions might include:
- How are results measured?
- What metrics are important? And why?
- How are those metrics calculated?
- Who gets the metrics and what do they do with them?
- Is the community likely to be compared against them?
Based on the department your community serves, there are some common metrics you might need to use here. These include
- Net Promoter Score
- Customer Satisfaction Score
- Customer Contact Rate
- Task Completion Rate
- Product Adoption Rate
- No. Bugs Resolved.
- No Features Implemented
If you adopt one of these metrics for your community, be mindful of the context.
There are many areas where a community will naturally have advantages or disadvantages against other programs.
One client, for example, had terrible NPS scores compared with other customer support channels. The problem wasn’t the community experience. The problem was the community had been positioned as the final destination for people who hadn’t been able to solve their problems through any other channel. These are the same people who were already frustrated before they arrived in the community. Often they simply had problems which couldn’t be solved and had nowhere else to express their frustration.
Comparing the NPS scores of someone in a community vs. someone who gets personalised help on a support call doesn’t make sense unless cost is considered in the conclusion. This is why you might want to compare NPS scores/cost of each contact as a metric rather than just NPS scores.
Never take a single benchmark from another program and apply it to the community. Always consider the unique context and ensure this is reflected in your recommendations.
Is Engagement The Goal?
If you don’t have existing benchmarks to compare against, you have the option to design a unique metric for the community. The most common example is to use some measure of engagement.
We’ve talked endlessly about the challenge of using engagement metrics to serve as the north star (I wrote an entire book about it).
Put simply, the problem with using engagement is many people simply don’t value engagement. Even those who just want to see a lot of engagement in a community recognise not all engagement is good (if they disagree, try removing the spam filter).
We’ve worked with dozens of clients over the year who just want to see a highly engaged community. They might not know precisely what success looks like, but they know having a huge number of people highly engaged in an ecosystem they control is critically important.
This is where you want to probe a little deeper to find out exactly what they want to see.
- What is your definition of community?
- What kind of metrics do you want to see?
- How will you know if the community is achieving its goals?
- What does success or failure look like?
You might not need precise metrics here, but you should be able to get to a business question which you can then translate into more precise metrics.
Setting Engagement Metrics
If engagement is the goal, the next step is to translate the broad goals you’ve been given in discussions into more specific metrics.
This is where you need to ‘operationalize’ a goal. This can sound far more complicated than it really is. Essentially, you look for metrics which reflect what stakeholders have said they’re trying to achieve.
For example “I want to see a thriving community with thousands of members reflecting our diverse audience and lots of healthy discussions”
In this example, you can see the number of members, diversity of the audience, and quality of discussion matters. So you might begin testing metrics such as
- No. active members (1+ posts in the past 30 days).
- % of members from each core audience group.
- No. responses to questions.
There aren’t any rigid rules about this process so go with what works best for you. Notice how important it is to pay close attention to language. If the person had referred to thousands of ‘happy members’, we would have also incorporated satisfaction score in there too.
Develop the potential metrics and check internally if they accurately reflect the business question.
Resource: CMX Indispensable Community Talk.
Establishing Community-Driven Impact Score
If engagement isn’t a goal (but you know there aren’t existing benchmarks to use), you have the freedom to come up with a better method. This is where we recommend the Community-Driven Impact Score.
This is a simple process where you ask members what the impact is and multiply this by the no. members represented.
- On a scale of 0 to 10, to what extent did the community influence your likelihood to renew your subscription?
If you have an average score of 8.8 and 10,000 visitors, that’s a score of 88k. Your job is to make that score go up by increasing the number of visitors or improving the impact of community upon members.
In our minds, this is the purest score which reflects the unique impact of community. However, it requires deep internal support and understanding to be fully utilised.
Begin the process of finding a north star metric with the full process in mind. Once you know the process, you can start guiding others through it and find a metric which works for everyone. This includes
1) Outline the process for your colleagues (and bring everyone along on the journey). There isn’t much point in developing a north-star metric if no one else supports it. Instead, if you bring people with you on the journey of discovery you can ideally land upon a metric which works for everyone. Make sure everyone understands the process of finding your north star.
2) Make sure people understand the limitations of metrics. If you need to calculate the dollar value return, make sure everyone understands the limitations of each approach and try to find how other departments have accepted the trade-off to find the right metric.
3) Use existing methods and benchmarks where possible. If you don’t need to show a dollar value, find out if there are existing benchmarks to compare against and if not you can develop community-specific metrics for your organisation. This might mean tracking some measure of engagement or utilising the Community-Driven Impact score.
4) Repeat this process this when things change. This isn’t a one-time process. As stakeholders and corporate strategies change you will likely need to revive and update the process over time. Renew the effort as often as needed and try to expand the value of the community over time.
If you feel you need expert help to develop this process and gain internal support, contact FeverBee.
Article: The Tyranny of Metrics
Article: The Engagement Trap
Article: Don’t Use Bogus Metrics
Template: The Community ROI Flow-Chart
Template: 46 Questions for Uncovering Community Goals
Template: A Framework For Hosting A Successful Workshop To Gain Alignment On Community Goals
Article: Setting Up Community-Driven Impact Score
Talk: CMX Indispensable Community Talk
Book: The Indispensable Community
It’s possible to prove the value of community by showing how behaviour is changed.
For example, you can show the impact of a community upon retention, call reduction, or some other behavioural metric.
The problem is it’s really damned hard and you’re never going to have a satisfactory result.
The simplest way to show the value of a community isn’t to prove people behave differently – but people think differently.
It’s a lot easier to measure meaningful changes in member attitudes than meaningful changes in member behaviour.
Where Does The Real Value Of Community Show Up?
Another way of thinking about this is to consider where the real value of community shows up.
Take the incredible volume of customers you support. Most people measure this using ‘call deflection’. But a growing body of data shows that people are asking questions in several channels at once and they often ask questions in a community they wouldn’t bother calling support for.
It’s not right to measure this in deflection dollars. But where it will show up is in how people feel about using the products. If they get better results, feel more satisfied, or get unstuck quicker – that shows up in their attitudes.
And this is the most direct benefit of a community – it changes member attitudes.
It’s often quite tragic changes in attitudes aren’t measured. Attitudes drive behaviour.
If you can change the attitude, you can change the behaviour.
What Attitudes Can We Measure?
Let’s imagine you went to your boss tomorrow and presented real, live, data that showed, that since joining your community, members have significantly improved satisfaction with products and the likelihood to recommend you to others.
Imagine you could show your community really moved the needle in how people feel about you and your brand and future purchase intent? That’s a powerful metric. Imagine the associations they have with your brand have changed in a more positive way?
Or, for the golden ticket, imagine you could show that brand preference has markedly increased vs. any competitors. Now members consider your brand superior on a range of different attributes.
Or, if you’re working for a non-profit, imagine you could definitively prove that a member’s quality of life scores have increased. Perhaps you could also show they now feel they are better prepared to handle whatever circumstance they find themselves in since joining your community?
In more specific terminology, we often use terms like:
- Member Satisfaction. (CSAT)
- Net Promoter Score (NPS)
- Quality of life (non-profits)
- Brand attitude
- Brand perception
- Brand preference
But it all rolls up into the same key measure – attitudes have changed.
This is the kind of data that is a lot easier to get and far harder to ignore.
How Do We Measure A Change In Attitudes?
You probably know the common problem with any metric you want to change by now.
You can easily show any community members score higher in almost any metric than non-members. It’s a lot harder to show it’s the community that caused the metric to be higher.
For example, how do you not know it was simply your best and most loyal customers who joined the community in the first place? Establishing any kind of causal relationship which removes the possible presence of confounding variables is difficult.
But these are some relatively simple solutions. They’re not expensive, but they do take time and you need to be careful with setting up the process correctly.
Option 1 – The Controlled Trial
The best solution is to do a controlled test. This is hard to do but it’s not impossible. In this approach, you would segment non-participants at random into two groups. You can send an email to one group with an invitation to join the community and track the results against the other group.
Or, better, only enable one group to see/join the community and not the other group.
In practice, however, this is problematic. For starters, it’s damned hard to stop people from seeing the community. You would have to remove it from search results and configure the technology for the community not to appear for a large group of members. Few organizations are keen on that. And even then you might only be measuring the differences in the characteristics of membership of each group.
If you can do a controlled test, great. But if you can’t, I’d suggest a simpler method…
Option 2: Gather Data When Members Join
The reason why few can measure the results of the community is they only collect data based on their timeline instead of the members’ timeline.
It’s a little like – ‘hey it’s October! Time to run our survey!’
But there’s nothing particularly special about October (or any month of the year) which is especially good to run a survey. In fact, you’re very likely to bias the outcome by limiting your survey to once a month.
This is why it’s far better to measure member behaviour by their tenure. Specifically, a member hits a certain milestone, it’s a good idea to hit them up with a quick survey to measure their attitudes.
The most important of these milestones is when members join! If you get a baseline attitude survey when members join, you can compare it with later surveys and estimate the results. If you ask the same questions again a year later, you get to see the impact of the community.
Why It’s Critical To Capture Member Data When They Join
Let me share an example of why capturing member attitudes when they join is so important.
Last year, I worked with a client whose community reported the lowest NPS of any customer support channel. We’re talking deep in the toilet low!
The community team was getting a lot of criticism for the poor performance of the community.
But was it the community’s fault?
We began doing some exploring. It soon became clear the community was simply attracting the most frustrated members. These were members who had had negative experiences in other channels and were turning to the community as a last resort.
They had the lowest attitude scores before they even joined the community!
The community was still helping these customers. But they often had problems for which there wasn’t a solution. The community helped them realize that. They weren’t happy about it – but at least they could keep looking.
The problem was no one had taken the time to capture members’ attitudes upon joining. They had no idea if the community had improved the results or not.
(aside – and remember here people who have a negative experience tell several times as many as people. Preventing negative word of mouth is a huge benefit).
Now imagine if they had captured the NPS, CSAT, or other scores when members had joined and could compare progress 6 to 12 months later. You can start to get a sense of the impact of the community.
Communities Naturally Attract Members Who Like You More
For example, in the graph below, you can see how much the NPS scores vary by each category of members. For example, avg. non-members today, avg. non-members last year, average new members a year ago, avg. new members today. As you can see here.
This is interesting data, but it doesn’t really show the impact of the community. It more likely shows that people with higher NPS scores might be more likely to join a community and those who like the brand most stay in the community for longer.
What we need to know is the difference between non-members and first-year members over the same time frame. This is where the data starts to get interesting, as you can see below.
The NPS varies (remarkably wildly) from one month to the next. But over the course of the year it appears that community members appear to have a higher difference over that same time frame. You can see this here:
Sure there is plenty of variability, but there’s a clear trendline here. Community drives a higher NPS score.
It’s worth noting this isn’t 100% conclusive.
You might simply be measuring the people within each cohort who bothered to complete a survey a year later – people who might naturally be more predisposed to completing a survey. However, by using random survey sampling and offering a small incentive, you should be able to overcome much of that.
Quick caveat here, if you’re working with a tiny community, this might not work for you. You probably can’t get enough responses without offering some substantial incentives. But for any community which has more than a few thousand members, you should be able to undertake a rotating study and show the results.
How Do We Measure Results?
Before we measure the results, let’s understand what we’re measuring here. Metrics like Net Promoter Score (NPS) and Customer Satisfaction (CSAT) have been extensively covered before.
Let’s focus on the three lesser-known attitudinal metrics we want to cover. Ones which are real game-changers for organizations.
I think of Apple as stylish, easy to use, and expensive.
I think of Logitech as basic, functional, and simple.
I think of Google as useful, friendly, and accessible.
You probably have (very) different perceptions of all three.
But we aggregate the results from a large enough group of people and you will soon start to build a great understanding of how people perceive of each brand.
These perceptions matter a lot. They are a leading indicator of future purchase intent, retention, and likelihood to promote the community.
So imagine your community could profoundly change how people perceive your brand.
Imagine if when members joined the community they used terms like confusing, overwhelming, and fiddly. Then a year later they used terms like automated, supportive, and comprehensive.
In less than a year, you can show you have completely changed members’ attitudes about your brand.
This is incredibly valuable data which too few people ever measure.
Perhaps the only thing more valuable than changing perceptions is changing preferences.
Brand preference is where you compare your brand against those of a selected group of direct and indirect competitors and get feedback from your members on how they rate each of you. It’s one of the most common and powerful impacts of community – but few people ever try to measure it!
Imagine you can go to your boss tomorrow and say because of your community, members have now increased preference of your brand against competitors by xx% points. Or perhaps they simply associate your brand with more positive terms than any competitor.
If you can show that 10,000 people have improved their preference for your brand since being in the community, it becomes a no-brainer to try and get as many people engaged in the community as possible.
Aside, the other major benefit of this kind of research is you get a unique insight into how people think and feel about your brand. I’m often amazed that organizations don’t constantly do these kinds of surveys given just how remarkable the results tend to be. You can provide your marketing and PR teams with precisely the terms they need to use and messaging to deploy to achieve the results they want.
Second aside, you can also take this a step further and measure the relationship between brand preference and purchases in the future to see exactly how strong this connection is.
The Danger of Measuring Brand Attitudes
Let’s quickly highlight an obvious danger of measuring member attitudes.
The obvious danger is the results might show you member attitudes declined since joining the community. On the surface, this suggests the community is having a negative impact.
This is a risk, but even if it is true, you 100% want to know this!
Once you have this data, you can begin interviewing members and drilling a bit deeper into your data to determine the precise cause.
For example, you might find it’s a specific segment of members having a negative experience.
In one client, when we drilled deeper into the data, we found it was people having issues with one particular product line. They were never likely to the solution they wanted because there wasn’t a feasible solution to the major problems they were raising. They felt they were wasting their time on the community and we were seeing hundreds of questions go unanswered.
We came up with a simple solution. If the community couldn’t help with these kinds of issues, it shouldn’t support these kinds of issues. We directed these questions directly to other support channels and closed down this section. Any time you notice the community is doing more harm than good, you can figure out why and take action.
What About Non-Profit Communities?
This might all sound good if you work for a typical business, but what if you’re working for a non-profit? Does any of this apply to you?
I’d argue it applies more to non-profits than any other organization. The only difference is you’re measuring different attitudes. Instead of looking at brand attitudes, you’re looking at more representative metrics such as quality of life, increased capacity, or overall confidence in managing their circumstances.
A Quality of Life Survey, for example, can show the impact of community across a range of well-being factors such as health, relationships, finances, and overall life satisfaction. Sometimes the impact of a community might appear in places you don’t expect.
Setting Up The Survey (Or Poll)
You can set up a survey to capture data both before and after someone has joined a community.
Ideally, you want to have the same people participating in surveys each year. A 1-year frame can work.
One way of doing this is offering a small incentive if they participate in the survey when they join and then offering them a larger one if they complete the survey a year later. Another option is simply to compare the average of newcomers vs. veterans and assume nothing else changed during this time period.
Questions For All Surveys
You can adapt these to your situation and environment. You may want to ask additional identifier questions about the type of customer or individual so you can drill deeper into the data later. You might ask questions about the type of products purchased, gender, age, location etc…
Typically the key questions should include:
Q. For approximately how long have you been visiting the community?
(This helps you to separate members who have just joined from those who have been around for a while. If you’re triggering the survey by member tenure in the community you can skip this question)
Q. How would you rate your overall satisfaction with [the product/community]?
- Very unsatisfied // Unsatisfied // Neutral // Satisfied // Very satisfied
(This is a question which can help establish overall satisfaction with the community environment. You can skip it if you’ve got too many questions, but it can help establish if the community is driving an improvement in results)
Q. If you would like to receive [discount/benefit], please enter your email address below
(This is simply a tracking question to be included at the end. It helps you identify who is participating in the survey and match it up to your customer database. Sometimes you don’t need this, but it’s usually useful to be able to follow up with people who took the survey the previous year)
Example Questions For Brand-Attitude Surveys
If you’re running a survey on brand attitudes, you might ask a combination of the following questions.
(NPS qs.) Q. On a scale of 0 to 10, how likely are you to recommend [brand] to a friend/colleague?
This is the standard NPS question.
(brand perception qs) Q. On a scale of 1 to 10, Please rate how well each of the following traits describes our brand.
List of traits here. This might include a mix of attitude and behavioural traits. For example:
- Value for money.
- Broadest feature set.
- Better quality than other brands.
- Caters to my unique needs.
- Easy to set up and use.
- Great customer support.
- Great documentation.
- The staff seems friendly and supportive.
You can add any set of attributes you like here which people can use to evaluate your brand. Try to be as specific as possible.
Aside – An alternative approach begins with the attributes you want to learn more about and asks members to select which brand most relates to which attribute. This is useful if you know specific attributes are most important.
Resource: Brand Perception Questions
(brand preference qs) Q. Before purchasing from [brand], which other organizations did you consider?
This is good for knowing who you should be comparing against and you can then program each of these options to appear in the evaluative set to ensure you’re not asking people to compare brands they never considered.
(brand preference qs) Q. How important or unimportant were the following factors in your decision to purchase from [brand] rather than any other brand?
- Extremely important
- Very important
- Moderately important
- Slightly important
- Not at all important
- No opinion
This question will help you identify the key factors which drove the purchase decision. List the most likely factors here and include a rating scale along the lines of
(brand preference) Q. What (if anything) might make you switch to a competitor?
You can also suggest possible answers and add an open text box. But it’s generally better to let members complete the answers themselves.
(brand preference) Q. Which brands would you most associate with the following attributes
- Value for money.
- Strong customer support.
- Comprehensive feature set.
- Easy to use
This is the critical question. It’s a direct comparison question where you use the relevant brands provided in the previous answer to populate answers for people to complete. Zero in here on the attributes which you feel best drive purchasing behaviour – you can use the answers from above.
Resource: Brand Preference Questions
Example Questions For Non-Profits
Developing questions for non-profits is a little trickier. Every non-profit supports a different circumstance and ideally, questions should best address that circumstance. This may include:
[non-profits] Q. How would you rate your ‘confidence to handle [circumstance]?
- Not confident at all
- Slightly confident
- Moderately confident
- Somewhat confident
- Very confident
This is the simplest question to use. It provides a simple snapshot answer where people can rate their level of confidence on a single scale and you can track results over time. You should be able to see precisely the impact of the community.
[non-profits] Q. How would you rate [quality of life, mental health, physical health, social life etc..] over the past 4 weeks?
- Very good
You can repeat this question with several variations above to capture the full impact of the community across a range of factors. This lets you identify where the community has shown results in areas where you might otherwise not expect. You can go further to deploy a full quality of life survey using the resource below.
Resource: Quality of Life Questions
Resource: Writing Survey Questions
Behaviour is primarily driven by attitudes. Your community shapes and influences those attitudes in a major way. You’ve probably had countless interactions which you know have had a major impact on your audience – and you probably haven’t tracked the outcome of those interactions.
Notice we use the word ‘track’ rather than ‘measure’ here. Measuring gives you a snapshot of today. That’s interesting, but not very helpful. Tracking attitudes over time will help you understand and prove the impact of your community.
Imagine how powerful it is to have results like this to share with colleagues:
However, if you want this to work you have to set up the data properly. You have to prepare the dataset today for the results you want to show tomorrow (or 6 to 12 months from now).
When execs are against the community, they often dismiss the ROI data which can be complicated and prone to all sorts of attribution issues. However, it’s harder to dismiss attitude data – especially when it’s been properly collected and analysed
Here are some simple next steps
- Decide which attitudes you’re going to track.
- Setup the survey questions and test the survey on a small audience.
- Setup the survey to trigger members based upon tenure (time in or no. visits to the community).
- Automatically pull the data into a visualisation tool (Tableau, Looker, PowerBI etc..)
- Drill into the results to determine which segments/factors most impact the results.
Let FeverBee Calculate Your Attitude Change Score
All of the above takes a lot of effort to set up and ensure the data is collected properly. This is where FeverBee can take care of the process for you. We’ve worked with plenty of clients to measure and analyse the impact of their communities.
We can take on the entire process and simply provide you with the data you need to prove the value of your community.
If you want help, drop us a line.
- It’s easier to measure meaningful changes in attitudes than behavior.
- Track attitudes over time – especially when people join.
- Brand perception, brand preference, and quality of life are valuable data points to have.
Here’s a relatively common story.
A community team is given a goal to achieve. This goal is usually something fairly simple like: “increase engagement by 50% by the end of this year!”.
However, a few months into the year, the engagement metrics haven’t budged.
In fact, the numbers are even beginning to drop slightly. No matter how hard the community team works to improve response rates, time to first response, and improve the platform, the overall engagement metrics simply don’t move.
At the end of the year, the engagement metrics are slightly below what they were the year before. The community team receives a negative performance review. Budgets are cut, team members leave, and the community suffers.
This raises the question, who failed here? It’s a more complicated answer than you might think.
(p.s. if you want the video version of how to set realistic community goals, click here.)
Setting The Right Goals For A Community
Perhaps the best way to begin is by looking at the goals themselves.
They were simply the wrong goals to begin with.
I wrote a whole book about this; engagement is a bad goal. It’s never the best metric to track, it’s simply the easiest. Worse yet, the number of posts, likes, shares, simply feels like a good metric for success.
To realize how wrong this is. I once came across a community manager who skyrocketed engagement overnight by removing the spam filter.
There’s plenty of information about finding the ROI of your community. I won’t rehash the entire topic here. The key thing is your goals should come from discussions with stakeholders and feasibility of behaviors.
1) Stakeholder interviews and analysis. You need to speak to as many stakeholders as possible, understand their unique needs and motivations (you can use this script), and undertake stakeholder mapping (see template) to determine whose needs to prioritize.
2) Needs, desires, and behaviors of members. You need to interview, survey, and study members to determine what they need, what they want, and what they desire. You can learn more about this kind of data here.
During this process we often host a workshop with the data to try to identify the right kind of goals for the community. We tend to set the members’ needs and let stakeholders establish their priorities.
You can see an example of this here:
Using our research and this simple framework, we should be able to identify and prioritize possible goals.
p.s. It’s worth noting this is an idealized approach. The reality is often a lot messier (it’s not unknown for a senior stakeholder to ignore all of this and simply set the goal).
Communities Need Really Specific Targets
In client calls, I often ask what the goal of the community is.
The person I’m speaking with can often give a clear and specific answer.
“The goal of the community is to improve product adoption”.
When I ask what metrics would show success, the answer usually becomes a lot more vague.
Often the answer is “plenty” or “lots”. Or, in the worst-case scenario “we’ll know it when we see it!”.
This leaves the community team in an unfair position. They might achieve a great result only for someone more senior to state “they expected more!”.
We need specific targets we can aim for here. For example:
We want to see 25% of active community members utilizing 2+ services.
Reach 3-month avg. 25% call deflection within two years while maintaining 4.2+ satisfaction score.
Generate leads with a value of $323,440 per quarter for 3 successive quarters.
Increase 3-month member satisfaction by 16% by the end of the year.
The challenge is where do these numbers come from?
Don’t Pluck Targets From Thin Air
Far too often numbers are plucked from thin air i.e. a 50% increase!
Why 50%? No-one knows! It’s just a nice round number that sounds good.
This often leads to a community team having goals which are impossible to achieve.
Is a 50% increase in call deflection a good target?
It might be if there was a 40% increase in the past year.
If activity rose by 40% in the past year, it might be. If it fell by 40% in the past year, it probably isn’t.
To begin finding the right target, we need to know our trends.
Use Trends To Set Good Community Targets
Targets should be based upon current trends with a range which indicates what great, good, ok, and bad look like.
Sometimes you can do a great job in reversing a downward trend but fail to hit your goals because whomever set the goal didn’t realize the community was in a downward trend.
Let’s use client data for a community which has a goal of answering as many questions as possible and in July 2021 answered 3971 questions.
The company wanted to increase this monthly average by 50% within a year.
But is this realistic?
Well, let’s look at the trendline below rather than pluck numbers from the sky.
The trendline suggests at current rates that 5150 answered questions is the current expected result (a 30% increase). 50% would be an extremely high result.
But if you look closer, you might notice something important.
Since April 2020, the number of answered questions has plateaued!
Expecting a big increase when the community has plateaued is a big mistake. Using data that stretches back to Jan 2018 doesn’t make sense to set community targets for July 2022.
Instead we can do something clever; we can forecast the number of answered posts just using the past year of data.
If you’re using google sheets, you can use ”=ARRAYFORMULA(FORECAST(39:A56,C27:C38,A27:A38))” to make predictions about the future.
Now the result (as you can see here) is different:
Notice now the prediction is of 4500 answered questions per month for July (or about a 13% increase over the year).
The Difference Between An Increase And An Improvement
You can improve a metric but still be performing worse than last year.
For example, if you had a 40% increase last year and this year you only get a 10% increase, the numbers will still go up but you’ll be doing worse.
An improvement isn’t about improving the absolute number, it’s about improving beyond the performance achieved the year before.
An easy way to do this is to set a performance improvement (I’d suggest somewhere between 10% and 30% – which should be matched by an increased budget) above and below the trend line.
This is what adding this will look like:
Now we can start setting some rudimentary targets with a 10% performance improvement based upon the previous year of data. This might look like:
- Anything above 5000 is good
- Anything between 4000 and 5000 answered questions per month by July 2022 is ‘ok’
- Anything below 4000 is bad
You can change the upper and lower limits from 10% to any percentage increase (or decrease) you like.
Now you can set targets based upon current trends from the past year of data and can see what a performance increase might look like.
Set Targets Where You Have The Most Influence
What If You Can’t Control The Outcome?
The biggest problem with using the ‘number of. answered questions’ as a goal (and pretty much any engagement target), is that it’s primarily driven by how many people have a question in the first place.
You can’t exercise much control over that.
If your company acquires more customers (or loses customers), that number will rise and fall through no fault or achievement of your own.
Worse yet, many activity-based metrics have a natural curve over time. As you begin answering most questions, people no longer need to ask as many and engagement drops. This is a good result masquerading as a bad outcome on your stats.
So we need to find out what impact you have.
Track These Three Metrics To Identify Your Impact
We want to know how the community compares against other channels.
If, for example, the number of support tickets (or customer support calls) falls by 10% and the number of questions in the community drops by 10%, that’s probably not the community’s fault.
In most cases, we usually want to get the following data:
- No. questions asked in other support channels vs. in the community.
- No. visits to the company website vs. the community website.
- No. new customers each month vs. new community registrations.
Then we look to how closely correlated these are with whichever metrics we’re tracking.
For example, look at the graph below.
We see historically there is a close relationship between an increase in web traffic and answered questions.
If the web traffic suddenly rises or falls, we would expect community participation to rise and fall regardless of how good a job the community team is doing.
The increase in web traffic above should mean a lot more people are now visiting the community. We therefore need to have a model which dynamically updates the forecasts based upon this relationship.
We now use the same FORECAST function to predict this and show what a 10% above or below the predicted line looks like.
You can see now how a big increase in web traffic naturally raises the number of answered questions we should anticipate within the community.
This also raises the expected answer rate. Anything above 6198 by July 2022 is now good and anything below 5071 is a poor result.
This is a very simple explanation of how to set targets. In practice, it can become far more complex. What matters however is now the community team has clear targets based upon actual data which is within their control!
How Do You Use Data To Achieve Your Targets
Goals Should Change Behavior
There isn’t much point in setting a new goal if you’re going to keep doing what you’ve always done.
The point in establishing community goals is to change your activities to align with that goal.
If your goal relates to growth, then you should be doing more activities which drive growth.
If your goal relates to call deflection, then you should be doing more activities which drive call deflection.
Goals ultimately change priorities. That means you do more of some activities and less of others.
But how do we know which activities drive the outcome?
We first need to calculate which activities have historically been strongly correlated with the outcome.
What To Prioritize To Achieve Your Goals
You need two things; a dataset and an informed opinion.
When you have these two things, you can run a multiple regression analysis to determine which variables influence the goal and by how much.
If you don’t know how to do this, find a data person who can help (or reach out to us – we do it for clients).
Let’s use a client example trying to increase member satisfaction.
We ran a multiple regression analysis on a dataset covering 15 variables and discovered the following:
Don’t worry, you don’t need to know what all of that means.
This essentially says there are three statistically significant (and independent) predictors of member satisfaction within the community. Combined they account for 86% of the variability in satisfaction each month.
We use these three predictors as the basis of our strategy.
- Objective 1: Increase the number of event attendees.
- Objective 2: Increase the no. MVPs who make at least 1 post per month.
- Objective 3: Reduce the average time to first response.
Now we repeat the process above to find the right targets for each of these objectives and show what a 10% or 20% performance increase or decrease would look like.
Build Your Community Dashboard
It’s obviously important not to keep targets to yourself but to be able to share them widely and let yourself and your entire team stay on track.
We want to know at a glance if the community is on track to achieve its goals or not. If not, we can make rapid changes in our strategy to ensure it is.
Building a dashboard once you have the data isn’t that difficult. You can use Tableau or PowerBI if you want more powerful functionality.
But we’ve kept it simple and built this one below on Google Sheets.
You might want to click to open up the full image.
We can now track progress over time (we’ve added some dummy data to illustrate). This shows where the community is doing well, where it’s not and, most importantly, it’s tracking the metrics which actually matter!
As you get more data, you can see issues early and address them. You can especially see when a number begins to fall behind its predicted target and try to identify what happened each month.
(p.s. It helps to get familiar with ‘conditional’ cell formatting in Google Sheets (or Excel) to create custom rules for what happens when numbers fall above or below a certain range).
Let’s Build Out Your Strategy
Once you have the right targets in place it becomes a lot easier to build out the overview of the community strategy.
Now you have all the key elements in place:
1) A clear goal
2) Two clear objectives to achieve the goal above.
3) Specific targets to track progress towards those objectives.
4) A set of strategies each aligned to achieving those results
5) A set of tactics (or initiatives) to execute the strategy.
6) Clearly identified ‘must win’ battles which identify the hard part of each strategy.
Believe me, it’s a lot better to be working on a strategy you know is aligned to achieving specific results you can feel comfortable about being held accountable to. It all begins with setting realistic community goals and the right targets.
Yes, targets should be SMART. But they need to be so much more than that.
Good goals should possess the following attributes:
1) They reflect the unique value of the community to the organization and audience.
2) You should have the majority influence over them.
3) They should be based upon current trends.
4) They should show what a % improvement looks like, not just the increase.
5) They should translate into specific actions you can execute on.
It’s okay to have multiple goals (I wouldn’t recommend more than three). What matters is you have some goals which are translated into specific targets to guide your work.
Try not to rush the process of setting good community goals. It’s worth investing a little more time (or getting outside help!) to get it right.