Five Research Techniques We Use To Evaluate Communities

We believe organisations should be making data-driven decisions about their communities.
This means you need to begin with really great data.

I want to share five techniques we use to evaluate a community.

 

1. Community UX Testing

If you want to evaluate the community experience, you can use your own judgement. But the way you look at a community isn’t the way members engage a community. It’s far better to undertake really good UX testing instead.

This means going through the community with members and seeing the community through their eyes. Specifically, this includes:

  • Get members to share their screens.
  • Give them tasks to do (ask/answer questions. search for info).
  • Evaluate where they get stuck or confused.

You should be able to quickly identify a range of issues and take steps to solve them. Prioritise the biggest problems and develop solutions.

IssueSolutionSeverityEffortPriority
Doesn't realise the login option is also the registration option.Easy to tweak to login/register.SevereLowHigh
The 'forgot password' feature isn't working.Needs a developer to review the process and resolve it.SevereLowHigh
Community is 'overwhelming'. Members don't know where to ask questions.Revamp homepage with separate instances for members depending upon profile dataSevereModerateMedium
Members can't find the posts they've recently made. Need to show members the recent posts they've made on the homepage when they visit and in the member profileMediumHighMedium
Members can't find the latest information about products.Need a separate signposted area on the homepage and on discussion pagesMediumModerateMedium
The registration process takes too much time to progress through.Need to reduce redundant pages and information we ask from newcomers.MediumHighLow
Members are unable to connect with people like themselves.Need to create a member directory and solicit metadata from membersLowHighLow

You should be able to build a list of solutions like this quickly.

Resource: How To Undertake Community UX Research

 

2. Analysis of Community Management Performance

It’s a good idea to benchmark how the community is managed. This typically means looking at a couple of things.

1. Review the community against competitors by avg. time to first response, response rate, and accepted solution rate. You can also do this by category for deeper insights.

community activity benchmarks

2. Benchmark the quality of responses against training/standards. Your community team should be trained to respond at this level. Pull a sample of responses and see how they compare against these standards.

community engagement benchmarks

3. Post questions and evaluate responses. We sometimes post questions in the community and see what the responses are like.

4. Test spam/troll processes. You can review how quickly spam is removed and how quickly trolls are addressed.

Resource: How To Improve Your Engagement Skills.

 

3. Acquisition and Onboarding

You should go through and experience the entire member journey yourself from scratch. Remember that in many communities, members don’t arrive at the home page. The first interaction might be a good search. So begin there.

Our evaluation of onboarding would typically include:

  • Join the community ourselves and go through the process with prospective members. You can use the same approach as UX testing above. See where people get stuck or are confused. Consider how long it takes to complete.
  • Look at conversion data and where the drop-off points are. Look at the entire onboarding journey and see where the majority of members are dropping out of the process. Use this as your baseline and suggest improvements.
  • Interview newcomers about their experience. See how they felt about the post-registration experience. Do they even recall any of the emails or messages they received? Were they useful? Could some things be removed?

Tip: Most of the time you can remove the majority of steps and increase the retention rate significantly.

Resource: Case Study: How FeverBee Tripled A Community’s Retention Rates (scripts, templates, and examples)

 

4. Top Member Program

It’s very hard not to be biased about your own top member program (aside, this is often why getting independent consultants like ourselves can help). Some of the things we would look at include:

  • What % of questions are answered by top members? This is a good indication of how effective the program is.
  • Who are the best performing top members? Use this bubble chart approach as before to see which members merit the most attention and which can be removed. Compare top members by # answers, speed of response, and accepted solutions.

super user community comparison

By removing a few top members from one program we increased performance significantly.

  • Interview current members to identify needs/desires /satisfaction. This will let you build a detailed top member persona to cater towards (hint: face-time with staff is usually high on the list).
  • Review recruitment and documentation. Review the documentation of the community. Is it clear how to join, what members get, and how the program operates?

You should typically be able to identify concerns and improvements which can be made here.

Resource: Community Superuser Programs

 

5. Rewards / Benefits

We analyse whether the community is offering the right rewards to members and whether members feel they are getting these rewards. There are plenty of ways to review this. Some of the easiest include:

1. Use surveys where members can identify and prioritise what they want from the community. Make sure members prioritise what they want. That’s the key.

community survey responses

Ask members how important or unimportant factors like the above are.

2. Review positioning vs. other channels. Does the community occupy unique positioning in the eyes of members vs. other channels? Is this aligned with what members need?

community benchmarking

Every community should occupy a unique value proposition for members.

3. Use interviews to identify specific requirements. Use member interviews to compare the community against pre-existing personas and ensure each is getting the value they need.

This should help you identify if the community offers the right rewards to attract and retain members. It should also help you determine if the community is delivering on the promised rewards.

Resource: Common Member Survey Questions
Resource: Common Member Interview Questions

 

Summary

You can use this data to create a simple evaluation showing how the community is doing today and where the focus might want to be.

overall community benchmarks

A simple summary of a detailed community evaluation.

The reason why so many strategies fail to achieve their goal is they aren’t built upon high-quality data. If you have the right data, it becomes a lot easier to identify the right solutions and prioritise what you should be working on.

 

Get Consultancy Support

If you don’t have the time or expertise to do it yourself (or need an objective opinion), get consultancy support.

We have many clients who want updated benchmarks each year to evaluate their performance and track progress. We would be excited to hear from you too.

 

Resources:

Follow This Strategic Approach To Engage Members On 3rd-Party Platforms

Everyone responsible for engaging communities has a decision to make.

Do you persuade members to come to you or do you go to them?

Do you spend your time trying to get people to participate in your platform? Or should you engage members where they are?

The former is great when it works. It enables you to have a lively community on a platform which you can completely customize to your needs.

The problem is it’s becoming harder to do this. It’s painful to spend thousands (even hundreds of thousands) of dollars developing a hosted community experience and struggle to get people to use it.

The latter approach is equally precarious. You can spend countless hours building communities on platforms over which you have little-to-no control. The rules might suddenly change or audiences might quickly change their preferences. Your account might be closed down with little justification.

 

Align With Strategic Trends

As you know, we always align strategies to current trends.

The trends here are quite clear. The popularity of hosted platforms is declining and the popularity of social media platforms continues to rise.

No matter what you do, there is going to be a sizable chunk of your audience who won’t engage on your community platform. As time goes by, this chunk is going to get bigger.

Test this on yourself if you like. How much time do you personally engage in a hosted brand community that isn’t your own? Can you even recall the last time you did it? You’re not alone.

We need an approach which aligns with current trends.

 

A New Mindset for A New Era

The old mindset was to create a central platform and drive everyone to it.

This mindset is outdated today.

The new mindset is to engage members where you can offer and gain unique strategic value.

Community is less about a destination you host and more about an experience you facilitate.

 

Where Should (and Shouldn’t) You Engage Members?

Before we get into the minutia of platforms, let’s remind ourselves of strategic thinking.

Thinking strategically means recognizing opportunity costs. Time spent on one platform is time you can’t spend on another.

A general doesn’t spread troops evenly across every battlefield. They decide which battles are worth fighting and then allocate their resources to win those battles decisively.

Our goal is to find the few places where it’s worth building a community outpost and commit the resources required to deliver a remarkable experience.

The battle we fight is the battle for our audience’s attention. We win it by delivering the value they urgently need that no one else can provide.

 

The Decision-Making Framework

We’ve summarized our thinking about whether to engage on a new platform below.

engagement decision making chart

Much of this is self-explanatory, but it’s worth going through each decision.

 

1) Where Does Your Audience Participate?

It shouldn’t be difficult to discover where audiences participate today.

Create a list of all the platforms you think or know your audience engages in today. List specific accounts/channels rather than broad platforms (i.e. r/brandquestions vs. ‘Reddit’).

Interview a couple of members to see if they suggest any others which aren’t on your list.

Then compile this into a survey you send out to your audience.

Whenever a member selects an answer, the survey should prompt them to explain why they participate in that channel. You can suggest attributes if you like.

This gives you both the list of potential places to consider and the exact value members want when they go to each.

2) Can We Offer Unique Value?

Adding another voice to the conversation doesn’t magically make it a better conversation.

It doesn’t matter how popular any platform is if you can’t offer unique value by hosting or engaging in an outpost there.

Common examples of brand value here might include:

  • Offering a verified/validated source of information amongst rumors and chaos.
  • Providing exclusive information and insights.
  • Answering questions others don’t have the ability to answer.
  • Supporting leaders with more resources.
  • Providing a space where you truly listen to member feedback and act on requests.

This isn’t a comprehensive list, but you get the idea. If you’re going to be on another platform, you should be able to use your unfair advantage to offer unique value members can’t get from anywhere else.

(Note: You must be offering unique value members urgently want. If it’s not abundantly obvious they want it, don’t be there).

Resource: How To Benchmark Your Community

Resource: What Is Your Unfair Advantage?

3) Can We Gain Unique Value?

You should only participate in places which offer you unique value.

If you’re simply engaging with the same audience in another destination, what’s the point? You’re just working twice as hard for the same outcome.

Unique value typically includes some combination of:

  • Access to unique audiences. The platform offers you access to audiences which are difficult to reach through existing channels.
  • Access to unique mindset. Third-party channels are great for getting the real story from members b/c they feel like it’s their space versus a company-hosted space.
  • Unique functionality. This is the ability to deliver information in unique and valuable ways (think YouTube, Instagram, TikTok, event hosting platforms, etc.) which support your community’s goals. Plenty of things are unique, but do they help you achieve your community’s and organization’s goals?
  • Improving brand perception. Will participating improve your brand perception? If all your competitors are on a particular platform. The opposite could also be true. it might make sense for you to be on a platform before competitors are. Sometimes it improves brand perception to be a pioneer (think Adidas on the Metaverse).

The critical lesson is you should increase your reach, relationships, or perception as a result of being on these platforms. If you’re not achieving one of these goals, consider carefully if being on the platform helps or hurts your outcomes.

Remember, it is expensive to host or engage in any platform at a high level. If you’re not going to get unique results from it, don’t do it.

4) Is The Platform Rising Or Falling In Popularity?

For most platforms, you have a choice about whether to host a community there or participate in one. Remember participating in a platform is a lot easier than hosting a community. The latter is a far bigger commitment of resources. But you have to participate the right way right or face the consequences.

Three is a simple framework you can use here. If a platform is rising in popularity, you might want to invest the time to build an outpost there and really commit to it.

On some platforms, you can simply see and track the level of engagement over time. At other times, you can look at public statements of user engagement to get a good picture. Pew Research is also a good source.

However, if a platform is falling in popularity, you probably want to spend far less time on it. You probably don’t want to invest too much time in a platform that’s falling in popularity.

Resource: How Brands Should Participate In A Community

Resource: Popularity of Social Media Engagement

How To Use Each Outpost

Once you have a broad idea of the right platforms to engage, it’s important to engage in each the right way.

You can see common examples of this here:

This varies a lot by the type(s) of community you’re building.

Two different types of community communities can use the same platforms in very different ways.

Test the platform with ideas before making a full commitment to it.

Resource: Five Brand Community Building Models That Succeed (And Why)

Resource: How Brands Should Participate In A Community

 

Don’t Try To Use New Outposts As Inbound Recruiting Tools

In theory, you can create an outpost on popular platforms like Twitter, Reddit, YouTube, etc. and use it as a source of new members for your existing community platform.

In practice, this is not going to happen – at least not at a level that makes the endeavor worthwhile. Attempts to use platforms as a recruiting tool will almost certainly end in disappointment for both you and your audience.

Your audience(s) have chosen a platform for a reason. That reason isn’t the result of a lack of awareness of other platforms. It’s simply a reflection of their preferences.

When you try to persuade people to move from their preferred platform you’re going to cause frustration. It’s far better to accept the preferences of your audiences and create the best possible experience you can on that platform.

 

New Platforms Require New Metrics

You’ve probably already spotted the problem with the above. You can’t use your current measurement system on new platforms.

For starters, this is often an apples-to-oranges comparison. How one platform counts a member, view, or action is different from another. It’s not an even playing field.

The far bigger problem is most of the platforms which will let you build outposts are designed for advertising. You often only get a small sample of aggregated metrics which reflect the needs of advertisers.

If you’re going to build an outpost on a platform you need to accept you’re measuring the success by the metrics available to that platform. Almost every platform offers some method of seeing views, posts, and comments.

You need to measure growth in the metrics available.

 

Next Steps

To get started, here are some unique steps.

  1. Use interviews and surveys to determine where your audience participates today.
  2. Evaluate the unique value people want and get from each platform.
  3. Analyze if you can offer unique value to the platform.
  4. Determine if you can get unique value from the platform.
  5. Research if the platform is rising or falling in popularity.
  6. Use our framework to determine how you might engage each platform by your community type.

 

Resources

Strategy Briefing: Do Forums Have A Future In 2023 and Beyond?

Should you launch your new community on a forum-centred platform?

It depends on whether you’re building a customer support community or not.

If you are, great! Go ahead and use a forum-centred platform.

If you’re not, then the picture is a little murkier.

 

Forum Use Cases Have Slowly Narrowed Recently

The number of community use cases for which forum is the best solution has steadily narrowed over the past decade or so.

Let’s begin with the five primary uses cases of community:

commujnity models chart

As you can see above, these are:

  • Support.
  • Success (or practice).
  • Advocacy/Marketing.
  • Peer/friendship groups.
  • User groups.

Ten years ago, a forum might well have been the best platform for all of the above. However, new platforms have increasingly emerged to better cater to unique use cases than a forum can.

 

What Makes Forums Unique?

We’re so familiar with forums we don’t often reflect on what makes them unique.

The answer is simple. Forums create pages. Lots and lots of pages!

Each discussion (or topic/question) is essentially a new page created at the request of a member. Someone with the answer responds and this answer is documented for others to find. The more pages there are, the more people are likely to find the answer to their question before having to use any other channels.

This is the primary reason why forums deflect support tickets, but chat channels don’t. They’re not indexable. This makes forums much less like chat rooms and more like Wikipedia. Forums won’t wow you aesthetically, but, like Wikipedia, the technology is reliable and it’s easy to find what you want.

A forum is the best tool for documenting the collective experiences of a vast number of customers in a searchable database containing thousands, even millions, of answers.

So is a forum still a good fit for your community or not?

The answer primarily depends upon whether members creating thousands of pages of searchable information would be a great help. If you’re selling simple, retail, products, the answer might be no. But if you’re selling a complex technology product, the answer might well be yes.

 

Why Forums Aren’t Usually The Best Fit For Customer Success Communities

Not long ago, it was common to launch a forum for members to proactively share their advice about a product or topic. That sometimes still happens today, but it’s becoming rarer than before.

These non-support communities have a remarkably high failure rate. The problem is they rely upon a small group of experts willing to share their advice. But if you’re an expert in a topic today, you don’t want to share your wisdom in a forum.

We spoke about this at length here.

Expertise gets buried in forums. Each new post pushes the former off the page. Experts want to share expertise in a place where they can build their reputation, grow their audience, and exercise some measure of control. They can then leverage this platform for tangible and intangible rewards.

Social media platforms (including those like YouTube, Instagram, TikTok, Substack and Medium) are now the preferred choice for experts to share their knowledge. They offer better tools and the grand prize; the opportunity to go viral.

The best approach for brands is encouraging members to share expertise in places which suits them and then aggregating/promoting it within a central location.

 

But What About Reddit?

This approach works well when the potential knowledge gap between newcomers and experts is vast – and thus the number of experts is low. However, when the knowledge gap is low when someone can be considered an expert in months rather than years (i.e. most video games), there is another very powerful option – Reddit.

After some early stumbles, Reddit has become powerful by offering a hybrid forum/chat experience. In Reddit, the best expertise naturally rises to the top through the votes of other members.

This presents an enticing offer. If I share a detailed article on a forum, it will be chronologically buried by each new post that appears – regardless of quality. But if I share it on Reddit, there’s a chance it might attract a lot of votes and spread like wildfire.

While other platforms have the mechanisms of voting Reddit (Discourse), none has yet been able to replicate the culture of voting like Reddit.

 

Peer Groups and Communities of Interest

The rise of group messaging apps (Slack, Teams, Discord) along with the growth of Reddit and Facebook Groups have pretty much swallowed the peer group market (along with the communities of interest). A large number of legacy forums remain, but you will struggle to launch a new one in the modern era.

The problem here is simple. Other platforms have made it easier to chat. By chat, we don’t mean someone asking a question and another person responding. We mean the places where people casually hang out without having to discuss a defined question.

Forums were always clunky substitutes for real-time discussions. They required someone to come up and post a topic of discussion. But then in the real world, conversations flow seamlessly (and effortlessly) from one topic to the next. It’s not surprising that group messaging apps (along with Facebook Groups) have swallowed this market whole. They make it easier to do that.

The standard argument against these tools is once these groups become big, it becomes impossible to follow the conversation. The problem with this logic is most members aren’t trying to follow the conversation. They’re happy to dip in and out whenever suits them – just like they do when they meet their friends.

 

User Groups

Until fairly recently, most user groups relied upon a forum experience. But they always faced the same dilemma; outside of events, they struggled to sustain any meaning level of conversation.

Often they become repositories for event recordings and notices of upcoming events. This was never the right fit for the functionality forums offer.

This situation was ripe for new entrants and platforms like Bevy (and Meetup) have successfully stepped taken the lead here. They provide better event experiences for members. They recognise that people want event-first, discussion-second experiences. They deliver the key functionality that members want better than forum-centred platforms ever could.

 

Advocacy / Marketing Use Case

There are some great examples of forums being leveraged to attract new customers or drive sales (Element14 and Sephora). By bringing people into an environment you control, you have plenty of opportunities to create community-generated revenue.

However, you might struggle to find many examples created this decade. The problem is these communities are essentially communities of interest. Most audiences prefer to have these discussions on more modern experiences like Reddit or social media than older forum platforms. People care far less about finding the answer to a question than learning ‘what’s popular right now’. Social media and popularity-driven platforms are simply better for that.

And if the goal is to drive advocacy, there are often simply better platforms for that than forums. Platforms like BazaarVoice, Olapic, Influitive, and others all make it easier to work with brand advocates, create user-generated content, and attract new customers than a forum does.

 

The Strategic Implications

Our strategy philosophy is always to align our clients to current trends. If you don’t you’re forever fighting an uphill battle. That’s why it’s essential to identify the trend and then determine how best to align with that trend.

The major trend here is the shifting preference of members to use whichever technology best serves their immediate needs (as opposed to a forum-centred platform which supports all of them).

Strategic Implication 1 – Select The Platform Which Aligns With The Trends

If you’re launching a community, ignore the outdated dogma and select the platform that best aligns with future trends.

The above isn’t written as a comprehensive list. By all means, do your research or ask us for help. But make sure you aren’t beholden to ancient dogma or vendor-driven advice about which platform to select.

Pick the platform that best satisfies your use case.

Strategic Implication 2 – New Use Cases Require New Platforms

If you’re shifting or broadening your community’s goal, you probably need to shift or expand the platforms you’re using too.

Many efforts to adapt a forum for a new use case fail because it’s not aligned with member preferences. Members have habits for your current forum and preferences for the behaviours you want them to perform – which involve other platforms.

Just because a use case is enabled for that platform (i.e. user groups, knowledge base, events) doesn’t mean members want to use that platform to do those things. Community vendors have tried (and largely failed) to incorporate more use cases into their platforms. They can replicate a lot of the functionality – but it’s never as good as a platform dedicated solely to that use case.

As you expand your use cases, be prepared to expand the number of platforms you use too.

Good luck.

How We Get Amazing Community Data For Our Clients

For a while now, our community intelligence team has been helping clients gather and analyse data to definitively prove the value and success of their community.

If you’ve ever bemoaned that you “don’t have the data” you want, I would wager a reasonable amount you haven’t developed a specific enough process to get the data you need.

For example, a client recently mentioned they had waited years to have their data team tell them the impact of the community upon retention. But they’ve never once tried to build the process to get that data.

Once you’re up to the professional and elite level, it helps to be much more informed about community data and what you need to do to get the information you need to deliver terrific results.

Every project is different, but this is our standard process for clients.

community data table

There is a collaborative process and there are three stakeholders in this example.

1) The consultants. These are the consultants at FeverBee (but could just as easily be community managers on your team). They bring the client and community expertise to the table.

2) The data analysts. These are the analysts at FeverBee (but they can be the data people in your organisation). They bring analytical expertise to the table.

3) The IT team. These are tech experts on the FeverBee team or (more frequently), the client’s IT team who provide access and may help extract the data.

You may want to add more stakeholders to your process (e.g. your boss and other stakeholders).

Not every step is relevant to every project. Sometimes, for example, you don’t need to do fancy data modelling when you can get the data through your platform or community intelligence tool. So remove the steps you don’t need in the data analysis process.

What’s critical here is the right people are brought into the project at the times when they add value and kept out of the process when they don’t.

 

Defining The Question

We’ve often seen community professionals ask questions like:

“Show me the impact of the community upon retention rate”

This sounds reasonable, but actually, it’s an impossible question for an analyst to answer.

There isn’t a metric labelled ‘impact’ to compare against a metric labelled ‘retention’ in a database. To create those metrics the analyst would have to know what you mean by ‘impact’ and ‘retention’.

The most critical part of this entire process is defining a specific question that an analyst can answer.

If you don’t define a good question, you will never get good data.

This sounds dead easy. But, trust me, having recently gone through this process in a room full of seasoned community professionals, I suspect we all need help with it.

 

The Business Questions

Our first step is to speak to the client and gather as much context as possible about the background and the desired outcomes.

This obviously means finding out the background of the community, why it exists, what goals it serves etc…We’ve spoken a lot about this before, so will skip it here.

Next, we uncover how the data will be used.

Who will see the data? Is it going to just one person or will it be shared widely? What kinds of decisions will be made looking at the data? What format does the data have to be in?

For example, there is a big difference between providing an answer to a one-time question (e.g. how many leads has the community generated?) and creating a methodology which enables the client to track this data point month by month.

The answers to these questions inform the initial business questions our consultants develop.

Some typical questions here might be:

What is the impact of community upon retention?
How many of our leads are sourced from community?
Does community impact our product?

At this stage, we’re not trying to define what these terms mean. We’re simply putting together a list of questions to answer.

 

Define The Answer

Now next to each question or decision, we want to know what a good answer looks like.

We’re not speculating on the desired value here – that would be cheating – we’re simply trying to estimate what the format and structure of an acceptable answer might be.

I.e. We don’t want the client to say “We want to show a $2m+ savings in customer support”. We want the client to say “I want to be able to say what is the average number of tickets deflected per month in a community”.

We talked about this in our recent post on finding the perfect metric, but it’s important to note there isn’t a single way to investigate a question.

You can see the four common approaches below and the type of answers they provide:

4 methods to calculate community value

Notice that each option involves trade-offs between the time/resources required and causation or correlation.

Sometimes we encounter a stakeholder who says “just give me the answer”. In this case, we put together example answers in each of the four options and ask which method they prefer.

By the end of this process, you should know the business question, the desired type of answer, and the broad methodology to take.

Now we convert the business question to a data question.

 

Define The Data Question

Now comes the critical part of the process. We need to turn the business questions into data questions

If we get this wrong, we won’t get the answers the client needs.

As we mentioned before, there isn’t a metric labelled ‘impact’ to compare against a metric labelled ‘retention’ in your database. Every term in the business creations we created needs to be clearly defined.

How you create and define terms can lead you to completely different outcomes.

Here is a simple example.

Is the community below growing, shrinking, or approximately staying the same size?

community growth graph

I asked this in FeverBee’s community data workshop recently.

The room was evenly split between the three options.

The same people are looking at the same data and drawing completely different conclusions!

Isn’t that incredible?!?

This happens because people interpret the question very differently. Some look at the increase in recent months and see growth. Some look at the decrease since its peak and see decline. Some look at a broad variation around the mean and conclude it’s staying the same.

This is why community expertise is required, not just a good analyst.

 

Why Community Expertise Is Required

Let’s use the question above as an example.

An analyst might try to calculate if the community is growing by looking at the number of registered members. But anyone with community expertise knows that’s the wrong metric to use because people don’t tend to delete their accounts. So the number can only go up.

The community could be experiencing a rapid decline in activity and the analyst would still conclude it’s growing as there are more registered members than before.

Likewise, the timeline matters too. I once saw a data analyst gather data since day 1 and conclude it had grown. Well, duh, anything above the zero members the community had on launch day is growth!

Also, imagine a major global event (like a pandemic) happened during the period above. It might be obvious to you not to compare present data to peak or trough pandemic data. But it might not be obvious to an analyst.

This is why you (the person with community expertise) need to refine the question before you give it to a data analyst. Otherwise, you’re leaving too many decisions in the hands of someone who doesn’t have the expertise to answer them.

 

Refine The Question

This is why we refine the questions in data terms which also reflect our expertise in the topic. To evaluate growth, we might refine the question to:

“Is the total number of active participants over the past 3 months greater this year compared with the same period last year?”

Notice how we’re using community expertise to set a question which:

  • Uses a metric which best represents growth or decline.
  • Uses a 3-month average to avoid random fluctuations.
  • Uses year-on-year data to avoid seasonality.

This gives an analyst a clear and specific target to hit. They shouldn’t have to make any critical decisions.

Most importantly, these are all metrics you can easily create from existing data or find in your database.

Example: ‘What is the impact of community upon retention?’

Let’s go back to the retention-based question from before. We can break this down into steps.

Step 1: Define ‘Community’

It sounds simple, but defining who we’re studying here matters and we have some decisions to make.

Do we include anyone who has ever joined the community (even if they haven’t visited in years and never made a contribution)? Can we honestly claim the odds of them staying with a company were impacted by community? If the answer is ‘no’ (and it should be), then where do we draw the line?

Do we only include people who are regularly active members (even if can only generalise the result across active members)?

This isn’t binary and there are other options too. In this case, we might agree that you need to have visited the community at least three times (=>3) in the past year for the community to have influenced your likelihood of remaining a member. Again, this will exclude the majority of people who have visited the community (and means we can only generalise the results across this audience).

Step Two: Define ‘Retention’

Retention can mean many different things.

Do we look at product usage, annual renewal rates, monthly renewal rates etc?

What if the people who purchase the product aren’t those who use it or participate in the community?

What if you sell subscriptions to an organisation that enables access for hundreds of staff – some of whom visit the community when they need help?

We first check if clients have an existing definition we can use. If not, we work with them to craft the definition of retention.

In this case, we might say retention occurs when an organization renews their annual subscription.

Step Three: Check with the analyst

Now check the analyst understands the terms. They should now be able to find (or easily create) the two metrics above and begin studying them. In this case, they will look at the relationship of people who visit the community >3 times each year (compared to those who don’t) and organizations who renew their subscriptions (or not).

There are dozens of ways to define a term

How you define the terms will completely change the outcome you will get. The people with community expertise need to be doing this – not the analyst.

This is just one of many possible definitions. Other examples include:

Has the annual % of customers who renew subscriptions improved since the community was launched in [date]? If so, by how much?

What is the retention rate of members in the lower, middle, and upper quantiles of community visits compared with non-participants?

Is there a statistically significant relationship between community visits and monthly churn rate?

Do members who visit the community at least once per quarter have a higher probability of renewing their annual subscription?

Each option involves trade-offs and assumptions you (and others) need to be comfortable with.

Whatever the business question is, you need to translate it into a data question.

You can see more examples of translating the business question into a data question below:

Business QuestionData Question
Is our community growing?Is the total number of posts over the past 3-months greater this year compared with the same period last year?
Does our community improve retention? Do organizations with at least one member who has visited the community >3 times in the past 12 months have a higher probability of renewing their annual subscription than those who don’t?
Does our community increase customer satisfaction? Is there a positive relationship between the number of posts member make in a community and their satisfaction scores over the past 12 months?
How many support tickets does our community deflect?What is the relationship between the % answer rate in a community and the number of support tickets a product category receives when normalising for product size? What is this number multiple by volume of the % difference?
Has moving to a new platform increased engagement? Is the number of members who have made a contribution to the community higher in the 3 months since the platform migration compared with the 3 months before the platform migration?

 

Assess The Data Available

Now, this is where the analyst and the consultants need to work together to determine if and where this data might already exist. Both have useful expertise to contribute and it’s often something of a fishing expedition to go through with a client to determine

a) If the data exists.
b) Where it is stored
c) Whether it’s possible to gain access to it.

Over time, you get better and better at knowing where the data is likely to be and how to extract it from different systems.

You can see a simplified example below.

community data needed table

You can see here that while we can get most of the data we need, there are some we don’t have access to and can’t gain access to. In this case, we might need to refine our question to reflect the data which is available.

This often means finding a proxy metric which represents the metric we are interested in. For example, we might not know if people are using the product more, but we can see how many times they log in to the CRM for example.

The further we progress through this process, the more we may need to refine our data question to reflect the data available.

 

Evaluating Limitations

Every data question, methodology, and dataset has limitations.

At this stage, it’s good to list the limitations we’re aware of to ensure they are acceptable to the client.

Some common examples of limitations include

  • The methodology can only prove correlation, not causation.
  • The dataset only stretches back [x] months.
  • A major pandemic occurred which may distort the results.
  • Major platform changes may have impacted the results.
  • Some data is unavailable or missing.
  • Data is self-reported.
  • Data isn’t collected from all of the audience.
  • Can only generalise the results to the audience studied.

You can add to these as you go through the process. But for now, it’s good to list the limitations and ensure everyone understands and signs off on them to avoid problems later.

As you go through the process, your analyst may add additional assumptions or limitations in the project.

By this point, you should be able to put all the steps into a template like this.

community data process table

Template: Click here to download this template

 

Handoff To The Analyst

This is the stage where the project can be handed off to our data analysts.

By this point, we (and you) should have

  • Alignment among stakeholders on the purpose of gathering data.
  • An agreed list of business questions to answer.
  • Agreement on the type of answers expected (and visualisation to be shown).
  • Agreement on the data questions.
  • A list of limitations of the approach.
  • Alignment of the data needed and where it will be found.

Now the analyst should be able to work with the IT team and begin undertaking the analyst.

One critical thing is the analyst keeps a record of everything they’ve done with the data.

This ensures the process can be replicated (and turned into a monthly updated dashboard). It also allows others to understand and dig into the answers a little if they want to.

During this process, our analysts will work through the process of collecting, describing, exploring, and verifying the data to build their understanding. This is followed by preparing the dataset for analysis by selecting the right variables, cleaning it, and constructing, integrating, and formatting it.

From there they can begin building the model to answer the question. This includes selecting the methodology, testing it, building the model, and then evaluating the model.

Our consultants check in with the data analysts at the data selection and methodology stage. This ensures there are no changes or assumptions we aren’t aware of.

 

Implementing The Results

The final stage is the implementation of the results.

This means not just presenting an answer but ensuring the client gets the maximum value from the data. There should be both an answer to the question(s) (data points, visualisations etc…) and a methodology which can be used in the future.

For example, if the question is what is the impact of community on retention, we may both create a presentation to show the results, provide context, and share a methodology so this data can be automatically tracked month by month. This provides not only value for today, but value for the future too.

Our consultants now work with clients to deploy the results internally. This might mean implementing the methodology on a dashboard or providing advice for them to do it themselves. At this stage, we also want to set realistic targets for the future.

Finally, we provide training to best understand and interpret the results internally to ensure the client can make data-driven decision-making.

Next Steps

1. Uncover how the data will be used. Who will see it, what will they do with it, and what decisions will they make with it?

2. Identify the ‘type’ of answer they want. You have four broad choices. This will impact the methodology you use.

3. Turn your business questions into data questions. Be really specific with the terminology and time frame so an analyst can find or easily create the data.

4. Identify what data is available and refine your questions if needed. Begin with what you need, what exists, and whether you can gain access.

5. List limitations and assumptions. Create a list of limitations and assumptions baked into the process and ensure people agree on these in advance.

6. Hand over the project to an analyst. Have check-in points when they finalise the data and the methodology they will use.

7. Implement the outcome. Develop training materials, develop the dashboard, and set forecasts/targets for the metrics you’re tracking.

This breakdown spends far more time at the beginning of the process rather than at the end. That’s not an accident. It stresses just how important it is to do the hard collaborative work at the beginning to set really specific questions to answer.

These questions are embedded throughout the entire process and are what will show up on the dashboard.

We’re also clear about who should be involved in when. It’s critically important the people with the most community expertise turn the key questions into data questions. Once that’s done, it becomes a LOT easier for the analyst to run with the project using their skillset.

If you get this right, you can drastically improve the data you’re currently getting from your community today.

 

Get Support

If you don’t have the time, expertise, or confidence to set up your community data setup, let FeverBee take it on for you.

Contact us at [email protected]

 

Resources

COMMUNITY DATA PROCESS TEMPLATE
[ARTICLE] The Perfect Community Metric
[ARTICLE] Designing Your Community Data System
[ARTICLE] How To Prove Your Brand Community Changes Attitudes
[ARTICLE] Community-Driven Impact
[BOOK] Build Your Community (Pearson, 2021)

The Perfect Community Metric – Finding Your North Star

A north star metric is a single metric that represents the success of your community.

Ideally, it’s a single metric all stakeholders can track to see if the community is delivering more value than in the past. If it goes up, you’re doing well. If it goes down, things aren’t going so well.

If you find the right metric, it’s a lot easier to get more support and build more awareness of the community. However, finding the right metric isn’t easy. Every metric has limitations and it’s not easy to find a metric which reflects your organisation’s unique goals and constraints.

In this post, I want to share some of our thinking about finding your north star and finding the perfect metric for your efforts.

 

Why Finding The North Star Is Painfully Difficult

Before we begin, it’s important to be mindful of why this process is painfully difficult. Most previous efforts to find a north star metric suffer from four problems.

1) They don’t reflect value. Many metrics show increases or decreases in activity, but not whether the activity is important or not. This leads to the ‘engagement trap’ problem.

2) Gaming the metrics. Many metrics are easy to game. This undermines the entire value of the metric. Good metrics always need to be accompanied by good judgment (and ethical behaviour).

3) They are driven by external factors. Many metrics change not as a result of community, but as a result of external factors outside of the community team’s control. You shouldn’t be held accountable for metrics you can’t directly influence.

4) They aren’t comparable to other metrics. It’s hard to know if the metric matters if it can’t be compared to any existing metrics. A community-focused metric is great, but how do you compare that to a return on marketing spend?

Be mindful that the challenge isn’t to find a flawless metric. There isn’t one. The challenge is to find out which flaws are acceptable and communicate these well to others.

We will go deeper into each of these flaws and make suggestions about which kinds of metrics might make sense.

Resource: The Tyranny of Metrics
Resource: The Engagement Trap
Resource: Don’t Use Bogus Metrics

 

The Journey Is As Important As The Destination

The process of finding the right north star metric is as important as the metric itself.

You shouldn’t come up with a metric by yourself. That’s a surefire way to set yourself up for endless years of arguing why the metric matters. Instead, you should treat this as a journey of discovery and close collaboration where you listen to the thoughts of others and bring people along on the journey.

Everyone should feel they were involved in finding the final metric. If you don’t do this, you might face a concerted effort of people to unpick the value later.

You can see a typical process to find the north star metric below:

ROI Flowchart Community

At every stage of the process, you need to build greater understanding and keep people informed. This might begin with knowing if a precise dollar value is needed. But that’s far from the end of the journey.

Be mindful that this chart isn’t prescriptive – it’s not designed to cover every possible type of community (non-profits, partners, developers etc…), but it might be a good place to begin investigating the opportunities.

 

PART ONE – FINDING THE DOLLAR VALUE

Is A Precise Dollar Value Required?

The first challenge is to ascertain expectations. Perhaps the most important determination is whether you’re looking for a dollar value return or not.

The method to find a precise dollar value is completely different than if one that has a different goal.

If a dollar value is required, you need to develop and explain a methodology that leads to a precise calculation. The end result is a dollar value return which will likely be compared to other departments to determine if the community merits more investment.

Also, if a dollar value is required, you will need data not just from the community but from whatever systems store the spending and behaviour of customers. You need to combine community data with your CRM data.

If a dollar value is not required, you have more flexibility to design an approach that reflects the unique value of your community. If a dollar value isn’t required, it’s possible to determine value just with community data.

In this approach, you can find a metric that perfectly encapsulates the impact of the community – but might not assign a precise dollar value to the community. This is like the difference between measuring the value of a branding effort vs. a dollar ad spend. The former requires metrics that either show revenue driven by ads or a change in feelings and perceptions (usually acquired by questionnaires).

You begin the process then by answering some critical questions.

These questions include:

  • Who will be looking at the metric?
  • What do they want to know about your community?
  • How will they be using the metric?
  • What will good or bad look like?

Once you’re armed with answers to these questions, you can begin to figure out what kind of metric might make the most sense. Crucially, are people looking for a precise dollar value or a broader sense of impact?

For now, let’s assume a dollar value is required.

Understand The Purpose

The next step is to determine the purpose(s) of the community. We’re not talking about specific goals here, but a broad understanding of why the community exists. This can be singular or there might be multiple reasons. Often the answer is suggested by whichever department the community reports to (which is why shifting departments is such a big deal, the purpose of your community will also change).

Some simple questions to ask here might include:

  • Which department does the community report to (and why?)
  • Who was behind the community’s launch and what was expected?
  • What are the expectations stakeholders have of the community?

If no one seems to know (or seems reluctant to own the decision), it’s often a good idea to host a stakeholder workshop to gain some sense of alignment. This is where you guide members through the process of prioritising what is the best match for a community.

You can see a typical outcome of the process below.

Community Gaining Alignment

You don’t need specific metrics at this stage, but you should have a fairly clear purpose (or set of purposes) to define how the community helps. Is it support? Attract new customers? Retention? Or some other benefit?

Resource: 46 questions for uncovering community goals
Resource: A Framework For Hosting A Successful Workshop To Gain Alignment On Community Goals

The Four Methods Of Finding A Dollar Value

Once we know the purpose, we can start finding the right methodology to calculate community value to a precise dollar value.

You can see the four options below:

4 methods to calculate community value

Each method has some obvious pros and cons. You have to decide between precision or simplicity, correlation or causation.

The former is a question of capabilities, the latter is a question of preferences.

Let’s go a little deeper into each method.

Controlled Experiments

The gold standard is to run an experiment.

However, this rarely happens because experiments usually require:

  • An analyst with experience in running experiments.
  • 3 to 6 months to plan and execute.
  • Removal of access (or not providing access) to a significant percentage of your audience.

If you don’t have the capabilities, resources, and support for this, it won’t be possible to run an experiment. This means you can only determine correlation, not causation. Also, be mindful that an experiment gives you data from a single point in time. If things change, this data might quickly become outdated.

Controlled experiments let you make statements like:

“The community causes members to spend/do [x] more than they otherwise would”

Article: The Real Value Of Your Brand Community

Group Comparison

The second approach is to compare two groups (or more) by a specific condition.

The most common example is to compare spending or retention of members vs. non-members. Statistically, this isn’t too difficult to do. The catch is this provides you with correlation, not causation. This simply indicates there is a relationship between membership and the desired outcome, it doesn’t show which way the relationship goes (i.e. does retention drive engagement or engagement drive retention?).

Nor does it reveal whether there is a confounding variable that influences both variables as you can see below.

confounding varible diagram

This doesn’t make the data redundant (and there are more complex calculations you can do to try and ascertain the unique impact of the community), but overall it’s a useful, interesting, signal.

This lets you make statements like:

“Community members spend/do [x] more than non-members (or comparative group)”

Note: You can (and should) be precise about how you’re comparing groups. For example, you can find groups of people who closely resemble one another before they join a community and then track their spending a year later etc…

Article: What Is A Login, Post, Or Download Worth? An Analysis By FeverBee

 

Surveys and Polls

Surveys are a handy tool which can be used to determine the value of a community. For example, you can ask members about their spending before and after they join the community.

The upsides of using surveys and polls are obvious; they’re easy to set up and analyse the results.

However, surveys suffer from three common problems.

1) Getting enough responses. You usually need around 300 survey responses to have meaningful results. If we assume a 5% survey response rate, you must contact at least 6000 people.

2) Sampling bias. Surveys suffer from a sampling bias. The most active members are also those most likely to respond to survey requests. This means surveys tend to skew towards the happiest and highest-spending members. You need to use a quota system by activity level to gain an accurate picture.

3) Responses are often inaccurate. You can ask members questions like how much they purchase before and after the community (or even track purchases over time), but memories are often fuzzy and the answers might not be accurate (people might also give you the answers they think you want to hear).

This doesn’t mean surveys aren’t useful. For tracking trends over time, they are especially useful. They can also give you a precise value you can easily justify. For example, you can say something like:

“Members tell us they’re spending $343 per year because of the community”

But if you’re going to use them to calculate value just be mindful of the downsides as much as the upsides.

 

Assigning Values To Behaviors

The final approach is to assign values to member behaviours. There are two common ways to achieve this, regression analysis and alternative costs.

Regression Analysis To Assign Values To Behaviors

Let’s tackle regression (or multiple regression) analyses. This is where you look at how a change in one variable relates to another. So you might look at how changes in independent variables like asking questions, reading articles, and receiving answers affect a dependent variable such as additional sales or revenue.

This study will give you coefficients which let you assign a value to this behaviour. This helps you make statements like “We know members who ask a question spend $4 more”.

However, be mindful of the phrasing above. We’re not claiming asking a question causes the increase in spending. We’re simply asserting members who do this one thing also do this other thing. This is correlation, not causation.

This methodology offers a precise value that helps you estimate the potential value of the community and the key behaviours you might want to encourage more of.

You can see an example of this in our work with The Pragmatic Institute.

How Much Is Community Worth

Alternative Costs

Perhaps a more familiar approach is to look at the comparative cost (i.e. what would it cost to achieve the same result if the community didn’t exist?).

The most common form of this is call deflection. This is how much it would cost to support the same number of customers if the community didn’t exist.

In the standard approach, you calculate the value of a view of an answered question and multiply it by the number of answered questions. This doesn’t mean these cost savings were realised (not many people actually reduce the size of their support team). But it gives a theoretical value to a community.

Other approaches include calculating the cost of acquiring customers, and traffic, or creating the same amount of content if the community didn’t exist. First, you find the baseline cost of the work today. Second, you determine the cost of a single incident of the behaviour happening in the community. Then you multiply this figure by the number of behaviours in the community.

This will you let assert an answer like ‘the community is potentially saving $7.04m per year in deflected tickets’

 

Which Approach Should You Use?

If each approach has significant advantages and disadvantages, how do you determine which is the best approach for you?

This depends upon your organisation. You shouldn’t develop the methodology alone because it might later be picked apart due to inherent flaws. If we know every approach has flaws, it’s good to determine which flaws are already acceptable to your organisation (or department) and select that approach.

This means it’s useful to investigate which methods are used to calculate the value of other activities within that function.

This might include questions such as

  • How are dollar returns calculated for activities within that department today?
  • Do they want to adopt a similar method for calculating community activity?
  • What kind of formula or methodology are they expecting?
  • How will the results be used in the decision-making process?

Find out which methodologies are already in use, and you can adopt a similar methodology to the community.

Once you know the methodology, you can zero in on the specific value metric.

 

Zeroing In On The Metric

It’s not possible to cover every possible metric, but it might help to cover some of the most common approaches below based on the methodology.

PurposeApproachCommon Method(s)
Customer SupportValue AssignmentCall Deflection Method
(No. visitors to accepted solutions * % who state they received the answer * % who would otherwise have contacted support (* % in warranty) * avg. cost per ticket.

Monthly Contact Rate
No. members * (avg. contact rate of non-members - avg. contact rate of members) * avg. cost per contact.
Controlled ExperimentExpose or withhold community from a group

Remove community access from one or more product groups and measure the increase or decrease on customer support tickets.
Customer RetentionPaired GroupsMember vs. non-member retention rate
Avg. annual retention rate of members - avg. the annual retention rate of non-members * no. members * avg. value of a customer.

Trial completion rate
% trial completion of new community members - % trial completion of non-members * avg. value of completing trial (CLV) * no. new community members.
Controlled ExperimentInvite one group, block another
Invite one group of prospects to join the community and hide the community from another. Compare the results after a fixed time frame.
Customer AcquisitionValue AssignmentTrack signups from the community
Calculating the avg. lifetime value of members who joined the community prior to becoming a customer.

Track conversions from the corporate site
Conversion rate of visitors to corporate site * visitors from the community.

Track direct sales
Direct sales of products and upgrades promoted to the community.

Track value of leads
Number of leads sourced from community sources * avg. value of a lead.

The right metric for your community might not be above. The key, however, is to engage with others to calculate the right metric for your organisation.

If you need help here, drop us a line.

 

PART TWO – IMPACT

If the value refers to a dollar return, impact refers to everything which isn’t a dollar return.
Much of the work of the community sits several levels beneath calculable value.

For example, if a salesperson uses the community to gather testimonials, how would you quantify the value of that? You can’t attribute the sale to the community, yet few organisations want to do a test comparing presentations with testimonials as opposed to those without. Yet, you know it has an impact.

If you’re not under pressure to show a dollar value return, you have a lot more flexibility to develop a method that truly reflects the impact of your community.

But be mindful of a potential problem here. The further you drift from a dollar value, the more likely it is people will question your metric. Everyone understands dollars, fewer people understand impact.

The challenge is to strike the balance between metrics that perfectly encapsulate the value of the community and those which will be accepted internally.

As a general rule, the more support there is for the community internally, the more you can design community-specific metrics.

This also serves as a good rule of thumb. The less internal support exists for the community, the more you need to skew the metrics in favour of value over impact.

 

Using Existing Benchmarks

The first step is to find out if there are already metrics which can be used to benchmark the community against.

These might not be dollar-value metrics. Instead, they might refer to things like member sentiment (or change in sentiment). In this case, you might compare a community by the quality of answers provided by members or the quantity of content created.

There isn’t any shortage of potential impact metrics you can use here. So begin by asking a few key questions for whatever department or purpose(s) your community serves. These questions might include:

  • How are results measured?
  • What metrics are important? And why?
  • How are those metrics calculated?
  • Who gets the metrics and what do they do with them?
  • Is the community likely to be compared against them?

Based on the department your community serves, there are some common metrics you might need to use here. These include

  • Net Promoter Score
  • Customer Satisfaction Score
  • Customer Contact Rate
  • Task Completion Rate
  • Product Adoption Rate
  • No. Bugs Resolved.
  • No Features Implemented

If you adopt one of these metrics for your community, be mindful of the context.

There are many areas where a community will naturally have advantages or disadvantages against other programs.

One client, for example, had terrible NPS scores compared with other customer support channels. The problem wasn’t the community experience. The problem was the community had been positioned as the final destination for people who hadn’t been able to solve their problems through any other channel. These are the same people who were already frustrated before they arrived in the community. Often they simply had problems which couldn’t be solved and had nowhere else to express their frustration.

Comparing the NPS scores of someone in a community vs. someone who gets personalised help on a support call doesn’t make sense unless cost is considered in the conclusion. This is why you might want to compare NPS scores/cost of each contact as a metric rather than just NPS scores.

Never take a single benchmark from another program and apply it to the community. Always consider the unique context and ensure this is reflected in your recommendations.

 

Is Engagement The Goal?

If you don’t have existing benchmarks to compare against, you have the option to design a unique metric for the community. The most common example is to use some measure of engagement.

We’ve talked endlessly about the challenge of using engagement metrics to serve as the north star (I wrote an entire book about it).

Put simply, the problem with using engagement is many people simply don’t value engagement. Even those who just want to see a lot of engagement in a community recognise not all engagement is good (if they disagree, try removing the spam filter).

We’ve worked with dozens of clients over the year who just want to see a highly engaged community. They might not know precisely what success looks like, but they know having a huge number of people highly engaged in an ecosystem they control is critically important.

This is where you want to probe a little deeper to find out exactly what they want to see.

  • What is your definition of community?
  • What kind of metrics do you want to see?
  • How will you know if the community is achieving its goals?
  • What does success or failure look like?

You might not need precise metrics here, but you should be able to get to a business question which you can then translate into more precise metrics.

 

Setting Engagement Metrics

If engagement is the goal, the next step is to translate the broad goals you’ve been given in discussions into more specific metrics.

This is where you need to ‘operationalize’ a goal. This can sound far more complicated than it really is. Essentially, you look for metrics which reflect what stakeholders have said they’re trying to achieve.

For example “I want to see a thriving community with thousands of members reflecting our diverse audience and lots of healthy discussions”

In this example, you can see the number of members, diversity of the audience, and quality of discussion matters. So you might begin testing metrics such as

  • No. active members (1+ posts in the past 30 days).
  • % of members from each core audience group.
  • No. responses to questions.

There aren’t any rigid rules about this process so go with what works best for you. Notice how important it is to pay close attention to language. If the person had referred to thousands of ‘happy members’, we would have also incorporated satisfaction score in there too.

Develop the potential metrics and check internally if they accurately reflect the business question.

Resource: CMX Indispensable Community Talk.

 

Establishing Community-Driven Impact Score

If engagement isn’t a goal (but you know there aren’t existing benchmarks to use), you have the freedom to come up with a better method. This is where we recommend the Community-Driven Impact Score.

This is a simple process where you ask members what the impact is and multiply this by the no. members represented.

For example

  • On a scale of 0 to 10, to what extent did the community influence your likelihood to renew your subscription?

If you have an average score of 8.8 and 10,000 visitors, that’s a score of 88k. Your job is to make that score go up by increasing the number of visitors or improving the impact of community upon members.

In our minds, this is the purest score which reflects the unique impact of community. However, it requires deep internal support and understanding to be fully utilised.

Resource: Setting Up Community-Driven Impact Score.

 

Next Steps

Begin the process of finding a north star metric with the full process in mind. Once you know the process, you can start guiding others through it and find a metric which works for everyone. This includes

1) Outline the process for your colleagues (and bring everyone along on the journey). There isn’t much point in developing a north-star metric if no one else supports it. Instead, if you bring people with you on the journey of discovery you can ideally land upon a metric which works for everyone. Make sure everyone understands the process of finding your north star.

2) Make sure people understand the limitations of metrics. If you need to calculate the dollar value return, make sure everyone understands the limitations of each approach and try to find how other departments have accepted the trade-off to find the right metric.

3) Use existing methods and benchmarks where possible. If you don’t need to show a dollar value, find out if there are existing benchmarks to compare against and if not you can develop community-specific metrics for your organisation. This might mean tracking some measure of engagement or utilising the Community-Driven Impact score.

4) Repeat this process this when things change. This isn’t a one-time process. As stakeholders and corporate strategies change you will likely need to revive and update the process over time. Renew the effort as often as needed and try to expand the value of the community over time.

If you feel you need expert help to develop this process and gain internal support, contact FeverBee.

 

Resources

Article: The Tyranny of Metrics
Article: The Engagement Trap
Article: Don’t Use Bogus Metrics
Template: The Community ROI Flow-Chart
Template: 46 Questions for Uncovering Community Goals
Template: A Framework For Hosting A Successful Workshop To Gain Alignment On Community Goals
Article: Setting Up Community-Driven Impact Score
Talk: CMX Indispensable Community Talk
Book: The Indispensable Community

Become A Community Consultant at FeverBee (North America only)

We’re looking for a community consultant to join the FeverBee team.

If you want to work at the cutting edge of the community industry, help the world’s largest organisations solve complex community challenges, and master a range of new skills, we want to talk to you.

Put simply, our mission is to help great companies build thriving communities.

We’re the consultancy the world’s top brands turn to when they want to improve how they build their communities. Our recent clients include Google, Microsoft, Intel, Apple, Meta, SAP, HP, Esri, Atlassian, Okta, and many more.

We hire passionate, intelligent people, and give them the autonomy to do exceptional work.

What you will do

You will work with clients in a wide range of sectors and industries. Our work is multi-disciplinary and covers corporate strategy, user experience design, data analysis, internal and external communications, and much more. You will be exposed to a wide variety of skills and resources.

Working with our team, you will help shape community strategies at some of the world’s most prominent organizations. This work includes gathering and analysing quantitative and qualitative information, developing and presenting recommendations, and helping implement solutions to our client’s community challenges.

As part of the FeverBee team, you will receive exceptional training as well as frequent coaching and mentoring from our senior team. Additionally, you’ll receive a stipend to undertake training courses, attend events, and purchase relevant books as needed.

 

Key responsibilities include:

  • Undertake user experience research on behalf of clients.
  • Interview community members to understand needs, pain points, and desires.
  • Interview key stakeholders to uncover challenges, understanding, and enthusiasm for community.
  • Evaluate client communities against best-in-class standards.
  • Work with FeverBee’s data analysts to identify key trends, challenges, and value of a community.
  • Develop playbooks, training materials, operational plans, and roadmaps for clients.
  • Use community research to develop and implement best-in-class solutions

 

Requirements

  • 2 to 5 years of professional community experience with hands-on experience using at least one of the following community platforms: Khoros, Salesforce, Discourse, Vanilla, Insided, HigherLogic, or Verint.
  • Located in North America (this is a remote position).
  • Deep understanding and/or past experience in at least one of these fields; design, UI/UX, data analytics, or business strategy.
  • A Passion for analytics and/or statistics.
  • Excellent ability to work in a team environment, and engage effectively with people at all levels in an organization.
  • Ability to communicate complex ideas effectively, both verbally and in writing, in English.

 

Desired strengths

  • Learner – asks thoughtful questions, catches on quickly, interested in people and ideas.
  • Adaptability – receptive to change, flexible, can collaborate with a diverse range of people.
  • Deliberative – exhibits good judgment, identifies risk, and makes sound decisions.

 

Salary

  • $70k to $90k – Depending upon experience.

 

How To Apply

To apply for this role, send an email to [email protected].

Include your résumé and a link to your LinkedIn profile (if you have one).

Most importantly, tell us:

  • Who you are.
  • What you’ve done.
  • Where you’re going.
  • What drives you.
  • What skills you bring to the table.
  • Why you want to do consultancy work.

In short, sell yourself (but don’t tell fibs!)

Applications close on Dec 2, 2022 (but please don’t leave it until the last minute).

EO/EE statement
FeverBee is an equal opportunity employer and prohibits discrimination and harassment of any kind. All applicants will be considered for employment without attention to age, race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.

Community-Generated Revenue – Maximise Your Community ROI

You might be facing pressure to show your community doesn’t just reduce costs, but directly generates revenue.

In the long term, this pressure may be useful. It’s always a little awkward to ask for more money to support a program designed to reduce costs.

But in the short term, it presents a challenge; how do you generate revenue through community?

This is a question we should welcome and definitively answer. Cost savings are limited by definition. However, there is no cap on the ability to generate more revenue.

Better yet, the more revenue your community generates, the more internal stakeholder support you’re likely to get.

 

How Can Brand Communities Generate Revenue

Brand communities typically generate revenue in three core ways.

These include:

  • Retention (keeping customers you have). Brand communities directly improve the retention rate. This includes improving trial completion rates, increasing the lifetime value of customers, or improving retention of specific niches.
  • Direct sales (selling your members). Communities create opportunities to sell new products to existing members and drive greater product adoption.
  • New customers (attracting new business). Communities attract new customers (and audiences) to the brand’s products and services.

Each method has clear drawbacks. Retention is hard to prove, direct selling can be irritating to members who didn’t ask for it, and it’s not easy to attract customers of rival brands to a community. Yet it’s possible to navigate through each approach and harness the community to show great results.

 

Driving Product Retention Through Community

We all know brand communities can increase customer loyalty (i.e. retention).

But this isn’t achieved simply through members engaging with one another. Often the most engaged members are the most loyal customers, to begin with.

Instead, we have discovered there are very specific mechanisms that increase retention. You can see these below:

How do communities drive ROI

We won’t cover every activity, but retention is achieved when members experience:

  1. Initial success using the products as quickly as possible.
  2. Learning how to get the most value from the products.
  3. Satisfaction with the brand experience.

Each of these, as you can see above, can be influenced by things you directly control.

Here are a few things to consider:

Set Up Groups For Product Newcomers

It’s common to set up a group for newcomers to the community. This isn’t a terrible idea, but it’s not the best idea either. People don’t want to learn about the brand community, they want to learn how to get started with the product. It’s far better to set up groups for people who are new to the product. This is a place where they can be encouraged to ask as many beginner-level questions as possible and get mentored by others.

This works best when you collaborate with sales and success teams who will guide new customers into newcomer groups. If members are on a trial period, it’s critical to guide members into groups where they can experience results as quickly as possible.

You should ask members to frequently share updates and progress, solicit questions on areas which members often don’t understand, and have top community mentors who can provide immediate-hand help to resolve any issues.

Measurement: Track the % of newcomers who complete the trial in a newcomer group compared with those who don’t.

 

Prioritize Questions Of Trialists

You can also improve the retention rate of trialists by prioritizing their questions. This is the audience most likely to give up if they experience frustration. Every hour they have to wait for a response to a problem, their frustration increases.

Set up an automated or manual system to highlight questions from community newcomers (first 30 days) to a superuser group or internal Slack channel. Work with your top members to prioritize responding to these questions and guiding members to provide the best possible responses.

Measurement: Track the satisfaction rate of trialists invited to the community vs. those who aren’t.

 

Build A Library of Great Examples

In our experience, members love seeing great case studies, examples, and breakdowns showing the best (and most innovative) ways to use a product. They use these to guide their own work and get more value from using the product.

Instead of badgering members to share their best advice in wikis (or tribal knowledge base), it’s far, far, better to encourage members to share how they’re using your products and services. Especially if they have a unique approach to it.

A single great case study or breakdown provided by a member of the community is worth dozens of responses. It’s one of the most valuable contributions members can make to your community. You can use a tribal knowledge base tool, interview members, or simply invite members to submit articles for the blog or newsletter.

The key to making this work is ensuring the reward vs. effort equation is in their favour.

Measurement: Regression analysis showing the relationship between article views and retention rate * customer lifetime value.

Resource: How To Get Experts To Contribute To Your Community

 

Upgrade Members Near Renewal Time

If you are managing a community where members renew their subscriptions each year, make sure you provide increased value to members closer to this time.

Create an automated system where a few months before a member’s renewal is due, invite them to a private group of peers to celebrate their one-year anniversary. This should be a group at a more advanced level and ideally where they can find peers to help them with the next stage of their journey.

This provides unique value to members and targets this value at the moment when members might begin considering whether or not to renew their subscriptions.

Measurement: Track the renewal rate of members who receive these notifications vs. those who aren’t.

 

Direct Sales To Community Members

The idea of selling things to community members feels like an anathema to many of us. This is often the result of painful internal discussions where we fight to keep the hounds of salespeople out of our community.

However, if you’re in business, you’re always in sales in some capacity. Don’t shirk from this opportunity, embrace how you can showcase products in a manner which feels authentic to you and your community (e.g through UGC videos, user group meetups, etc). The key is to ensure you’re selling what members truly need and check the sales process adds value to members.

When done well, this is the most direct path to drive immediate value and show the quickest results. You don’t need fancy math to show direct value generated from selling to members.

 

Premium Membership

The easiest option is to charge a fee to be a part of a community (or to access unique areas/features of the community).

The downside of this approach is it immediately limits the size and level of engagement within the community. Yet, it may be possible to create unique user groups, membership tiers, or features that members are happy to pay for. Sometimes this can be wrapped into a broader support package.

You might, for example, pay a small fee for top members to share their best expertise in a private group that only paying members can access. Or offer members unique customisation opportunities to engage in a certain way.

 

Events

Perhaps the most common approach is to charge members a fee to attend community-hosted events and activities. When this is done well, it can be a tremendous revenue driver. You can attract sponsors, new customers, and members paying hundreds, even thousands, of dollars to attend.

The downside is events are cost-heavy. Many community events are loss leaders to attract new customers and build goodwill. While there are great examples of community events being spun into their own business unit, the scale required to make this work (and the variable cost model) makes it a challenge.

 

Courses

A far better match for most organizations is to develop training courses members can take.

If this is structured as an on-demand course, you can offer members courses that may be related to your product but not about your products.

You might not even need to create all the content yourself. Some organizations, like CXL (a client), pay top industry professionals a fixed fee and package access to all courses as part of a unique membership tier.

Courses could also be sold individually. You can reach out to three to four relevant industry experts and pay them a fee to create a course. You then sell the course to community members (or with a special discount to members).

 

Affiliate and Partner Sales

Perhaps the least common route in a brand community is affiliate sales.

When you have access to a large audience, there will be plenty of organizations that will pay a fee to reach them. Sometimes this is directly achieved through advertising. But few brands want advertising in their brand community.

But advertising is just one of many approaches. I know one organization in the pharmaceutical sector which generates millions of dollars a year by recruiting community members to participate in focus groups for a partner.

In this approach, you find partners in your industry who would like to reach your audience. You work with them to offer a strong discount (i.e. the unique value to members), and you gain a percentage figure for every product sold. Be mindful that any promotion is limited in frequency and offers tremendous value to members.

If members begin to express dissatisfaction, this approach should be discontinued.

 

Generate Business From The Community

Your brand community can also generate new business.

If the previous section explained how to sell new products to your existing customers, this section describes how to attract new customers to your existing products.

Of the three approaches, this is by far the most popular with organizations in today’s economic climate.

Brand communities can attract new customers in two primary ways. They can attract new leads and increase the likelihood of existing leads becoming customers.

 

Generating Leads

Generating leads is probably the easiest activity to directly show new business.

Theoretically, every registration is a lead. The problem for most brand communities is the majority of members are already customers of the brand.

The challenge is to attract people who aren’t already customers to join and participate. This raises an obvious question (one which many people launching a new community fail to answer); why would someone who doesn’t use our products join our community?

 

1) Create A ‘Thought Leadership’ Zone

A common answer to the question above is to create a community about the topic, not the brand.

In theory, this can work well (and there are plenty of great examples).

In practice, it’s hard to do. Most efforts struggle. Members don’t have the same urgency to engage and participate. And the people who have the best expertise to share often want to do it in channels where they can build a following.

To do this well, it’s important to make sure you recruit experts early and ensure they’re getting what they want from the experience.

Engaging Experts in your communityIn this approach, you can open up a section of the community (or create a distinct area) for industry-focused topics and, as long as you are transparent, you can attract leads through registrations in the community.

Resource: Why You Should Build A Thought Leadership Community

 

2) Create Gated Resources

A simpler approach than creating an entire community to attract leads is to create resources and activities non-customers would be eager to submit their email addresses to access.

In this approach, you can collaborate with your members to create shared resources (or host events). These can then be promoted to non-members through advertising, social media, and other channels. Every registration can theoretically be counted as a lead.

 

3) Lead-Scoring Through Community Behavior

A powerful, but technically complex, approach is to identify leads from community behavior. In this approach, you might identify leads (or, more likely, upsell opportunities), by seeing how members engage in the community.

This can occur through lead scoring. This means specific behavior in the community is assigned a score. When a member achieved a certain score, they’re considered a lead and receive an invite to the next stage of the process.

A less complex version of this is to manually identify leads. For example, if you notice members discussing a problem one of your products can solve, you might flag this to a salesperson. We used to do this in the FeverBee Experts community.

 

4) Drive Traffic To The Website

A final approach is to optimize the community for search and leverage the community to maximize the traffic it drives to your primary company website.

There are plenty of ways you can optimize a community for search. This includes archiving old and duplicate content, optimizing titles for search, and ensuring most questions receive an accepted solution.

If you get to a point where your community is the secondary driver of traffic to your company homepage, you’re in a great position.

Who would want to reduce support for the community if it has a serious negative impact on web traffic?

 

Lead Conversion

Communities can also play a unique role in improving lead conversion. Sometimes it happens naturally. People want to get backchannel feedback or insights on the product/service they want to purchase. The community lets people ask questions and get help from others who have been in their position. In a branded community, they can also see how they will be supported as an admin and/or user of that product.

You can also stimulate improved lead conversion several mechanisms.

 

1) Showing Community Content On Product Pages.

You can use community-generated content on product pages and throughout the company website. This lets you use the natural social proof created by your community within the product pages where they will have maximum impact

Sephora (a client) is a good example of this.

Sephora Product Community

Looks, reviews, and discussions sourced in the community appear on product pages – this directly increases conversion.

Resource: Community Content Creates Trusted Product Pages

 

2) Surfacing Community Data in Presentations.

Having a thriving community should be a powerful reason for a potential customer to select your product over another.

If you have a community of 30k members who answer hundreds of questions a day and your competitors don’t, that’s a major differentiator.

The problem is the community data often isn’t communicated to prospects – at least not in a decisive way. The size and scale of the community should be prominently mentioned in sales materials as a major strategic differentiator and a reason to purchase.

Make sure the sales team is provided with good, useful, data they can use to highlight the unique value of the community. You can find out exactly what statistics might be most impressive and provide them with data and examples which might be a great fit. If you can build a list of great examples (discussions, case studies), this is a huge benefit.

Resource: Competitive Benchmarking

 

3) Testimonials, Reviews, and Case studies.

Your community is a fantastic tool to generate testimonials, reviews, and case studies which can be used in sales material and featured on comparison sites to directly attract more sales.

If you source 50 reviews for a comparison site from your community, it’s likely you will instantly jump to number one within that category. For an enterprise product, this can generate millions of dollars in sales. It’s one of the quickest possible wins.

To make this work, you need to make this part of the community management process. Regularly invite top members to create case studies, share testimonials, and leave reviews on relevant sites.

Make it as easy as possible by creating a template they can use. The more reviews and testimonials you can source from your members, the more leads you’re likely to convert. You can begin by asking your sales team which kind of testimonials would be most impactful and work with top members to do precisely that.

(p.s. make sure you have a large group of community members willing to serve as reference calls for prospects as well).

Resource: How Quickbase Became No. 1 In Their Category

 

4) Community-Recommended Products

If you’re selling multiple products to members, encourage members to share their product stack / recommended products with one another. This is especially useful for retail brands.

Better yet, show related products next to relevant community discussions and track purchases from these discussions. Having a list of community products recommended by members helps reduce the perceived risk and may increase sales.

 

5) Create Product Hubs / Pages Within A Community

You might also consider creating a product hub or page within a community which pulls together community discussions, one-pagers and data-sheets, feature requests/ideas, videos around key products. This is a highlighted, curated, collection of community and company products all within the same place.

 

Create Community-Generated Revenue

Don’t hide from the revenue discussion. If you have a successful brand community, it’s likely already generating revenue or there are a few small changes you can implement to drive tremendous revenue.

Try a variety of things and uncover which works best for your audiences (colleagues and members). Not every activity will hit, but you will be able to find a combination which works to turn your community into a revenue-generating machine. Make sure you track the results and consistently show how much revenue the community is generating.

It’s one thing to say a company should invest more to reduce costs, it’s a lot easier to show a company they should invest more to drive more revenue.

Feel free to contact us for consultancy help. We’ve helped clients generate millions of dollars from their community efforts. We would love to help you too.

 

Resources

Template: How Do Communities Drive ROI?
Guide: Measuring The ROI of Online Communities
Book: The Indispensable Community
Article: The Million Dollar Community Page
Article: How To Get Experts To Contribute To Your Community
Article: Why You Should Build A Thought Leadership Community
Article: How To Get Experts To Contribute To Your Community
Article: Community Content Creates Trusted Product Pages
Article: How Quickbase Became Number 1 In Their Category
Community: The CXL Playbook Community

Designing Your Community Data System – From Basic To Elite

Many of us haven’t really scratched the surface of what’s possible with our data.

If you want to see what’s possible, I recommend you watch this talk if you haven’t already.

You can do far more with your data than you probably realise.

Note: You should also sign up for our workshop on building community data systems on Nov 2.

Yet just knowing what’s possible doesn’t help us achieve our goals. We also need the knowledge to jump from where are today to get to where we need to go. And that destination is a place where we finally have the data (in the manner we need it) to make the right decisions for our community.

 

Benchmark Your Current Data System

If you’re trying to jump from a novice to an expert overnight, you’re setting yourself up for disappointment.

It’s far better to figure out where you are today and track your progress over time.

community data benchmarks

As you can see above, we benchmark community data skills along the four levels.

To summarise briefly:

  • Basic. Tracking changes in community engagement.
  • Advanced. Tracking what happened and why it happened by using external data.
  • Professional. Evaluating community impact and making data-driven improvements.
  • Elite. Automating the entire process.

These benchmarks themselves are comprised of five attributes:

  • Metrics. Which metrics do you measure grow and expand over time?
  • Extraction. How do you gather the data from different sources?
  • Analysis. Which analyses do you undertake in your community?
  • Summary. How do you summarise and visualise the data?
  • Utility. How do you utilise the data in your strategy?

Before we begin, take a second to benchmark your current level today. You might be farther along in some attributes than others.

If you’re like most people, you’re at the beginner or advanced level. This is good – it means you can make a lot of improvement in a relatively short amount of time!

Resource: Community Data Benchmarks

 

The Basic Level

If you’re relying upon the insights that come up with your community platform, you’re at the basic level.

Almost all of us begin by using our community platforms to track engagement. Yet even within this level, there is a huge variation in the data you have.

The quantity and quality of insights provided by inexpensive tools like Facebook, Circle, and Tribe are very different from those provided by premium enterprise platforms.

Generally speaking, the more you spend the better insights you should have.

At the basic level, you can usually answer the following questions:

  • Is engagement heading in the right direction?
  • Who are the most active members of the community?
  • What are the popular discussions in the community?

Overall, you get a simple snapshot which reflects aggregated data for a given time period in a community. Typically at this level, you’re simply gathering community data and sending screenshots or visuals to colleagues.

However, there are critical insights you won’t have. For example:

  • What’s causing the changes in engagement (and what to optimise for)?
  • Are changing because of your actions or things happening outside of the community?
  • Where are people dropping out of the community journey and why?
  • Is the community having an impact on key business metrics?

As your community expands (or your ambitions expand), not having this data starts to become a bigger and bigger problem.

Sooner or later, you need to move on to a more advanced level of insights.

 

The Advanced Level

The advanced level is when you begin to get serious about community data.

At this level, you’re not just concerned about what happened, you’re concerned about why it happened. The why is what lets you improve your community efforts. This is the foundation stone upon which you build a data-driven community strategy.

e.g. If you know [x] is causing a positive outcome, you probably want to do more of [x]. 

The key difference at this level is you need to combine multiple datasets. You’re still working with aggregated data (i.e. behavior is combined into a metric rather than separated by members), but you need to wrangle a few other sources of data together.

 

Step One: Gather Environment-Level Data

At this stage, gather data on what’s happening outside of the community to see how it impacts things within the community. We could easily write a series of articles about the precise metrics to measure, for now, I’ll recommend our ROI guide and this guide

I would recommend getting monthly time-series data to reveal:

  • No. visitors to the company website. (Google Analytics, Adobe Analytics, Amplitude Analytics etc..).
  • No. unique, new, visitors to the company website (Google Analytics, Adobe Analytics, Amplitude Analytics etc..)
  • No. search visitors to the company website (Unique visitors – source: Search).
  • No. customers/users (sales data/usage data – usually upon request)
  • No. new customers/product activations (sales data).
  • No. support tickets/customer calls per month (help desk software/customer support software – usually upon request).
  • Any proxy metrics for no. developers, admins, users etc…

Download this data from each source individually. Remember to keep rows as months (use transpose if they’re in the wrong format) and ensure it matches up on a monthly time-series basis stretching back up to 36 months (any longer than this and the data is less relevant).

You can usually download these metrics individually from each source. However, it might take a little effort to clean the data so you can create a single spreadsheet. The spreadsheet should have each attribute above as a column and each month should be a row.

Note: It’s difficult to download monthly time-series data from some platforms (e.g. Adobe Analytics). You might need to download data from each month at a time. This is also true of some community platforms. This is painful and takes a little time – but you only need to do it once (thereafter you’re updating the data each month).

 

Step Two: Setup and/or Collect Qualitative Performance Data

Quantitative data can reveal plenty of useful insights, but it doesn’t show whether members are becoming more or less satisfied with the community. This is why it’s also good to have a measure of member satisfaction you can track over time.

This can be done in three ways.

  1. An annual survey. You can set up an annual survey of up to 10 questions to get an ongoing measure of how members feel about the community. This will go out to all members (ideally using the same questions each year).
  2. A quarterly survey. You can set up a survey which goes out to a rotating sample of members at random. This makes it easier to track data over time but is prone to strong fluctuations if you don’t get 300 responses.
  3. A poll. A poll is an ideal solution to track member satisfaction. This appears to members as a pop-up in the community and asks members to rate their satisfaction, whether they got the answer or the overall impact of the community. These results can be aggregated into a summary score over time. I’d strongly recommend setting up the community-driven impact score system at this level.

You can set this up in your platform toll or using Hotjar, Qualtrics, Typeform, SurveyMonkey, or any other tools which suit your goals.

Ideally, you should combine this data with your community data on a monthly time-series basis.

If you can’t get data monthly (it’s hard to do this for surveys unless you sample a rotating group of members), simply share the annual data each month for now.

 

Step Three: Analysing Community Performance

Once you’ve got your dataset, you should be able to analyse community performance.

You can see the kinds of questions you can answer below (but, be mindful, this isn’t a definitive list).

Question:
What are the trends we need to adapt to?
Data
What content / categories / tags are rising or falling in popularity (use % from normalised baseline).Aggregate group activity by category.
What areas, features, sections of community are rising or falling in popularity?Aggregated visitor data from different sections of the community.
Is there any major shift in locations people are coming from?Google analytics - top 10 locations - 100% bar chart with trendines.
Are the type of users changing (more devs, admins etc..).Member profile data (or synched customer data).
Any specific changes between types of visitors?Google Analytics data on devices, regions, languages etc?

To perform this analysis, you might also use your current variables to create new metrics.

For example, you might create metrics like posts per active member, visitor conversion rate etc…You can find an example of what we often look at in this sheet.

You can now create graphs to help you answer questions like:

  • Is the community doing well in attracting attention and acquiring newcomers? (i.e. attracting the expected number of visitors, newcomers, and questions)
  • Does the community have the right number of superusers / is the quality of superusers good enough? (i.e. are the majority of questions being resolved/answered as the community grows?)
  • Are customers more or less likely to turn to the community for support? (i.e. what is the relationship between support tickets vs. community questions?)

You don’t need to use these questions verbatim. Come up with your own questions if you need them. What you should be able to determine though is how well the community is performing and why.

Note: Be mindful that this is always correlational data. This doesn’t make the data bogus, but it should be treated as a relationship which could be explored further.

 

Step Four: Using the data

If you have the expertise, you can run a regression analysis to show the strength of the relationship numerically. But this isn’t always easy to interpret.

Often simply looking at the data on a chart will help you explain what’s happening. 

It’s good to highlight any significant changes within the organisation which might’ve taken place during this time frame to see how they impacted the community (e.g. new product launches, website redesign, Black Friday etc…). 

You can overlay these with the data itself to see if there was any major impact.

You should be able to tell a compelling story here which explains:

  • What’s happened so far?
  • Why did those things happen? 
  • What’s likely to happen in the future (the forecast function is worth learning)?
  • What should you do more of, less of, and change to improve results?

Resource: Setting Up The Community-Driven Impact Score System.

 

The Professional Level

At the professional level, two things change.

  1. You look at member data. Instead of only tracking outcomes by month, you track outcomes by member. For example, you might see if members who have been invited into a newcomer group had a higher product trial completion rate than those who didn’t.
  2. You integrate databases. Instead of getting a single metric from each source per month (i.e. 16,434 customer calls), you get the list of who made the calls and when. You need to combine data from several databases to show the impact of community.

This lets you do a far deeper analysis to determine what is and isn’t working in your community. You can see exactly which activities (or member attributes) had a meaningful impact and which didn’t.

As this example dataset shows, once you have data at the member level the quality of analyses you can undertake improves considerably.

 

Step 1: Identifying sources of member data. 

Member-level analysis requires you to build a database of individuals where each row (observation) is a member rather than a month. This database needs to combine records from several sources.

These sources might include:

  • The community platform.
  • The customer relationship management system (CRM).
  • The mailing list provider (see who did and didn’t open emails).
  • The learning management system (LMS) (see who completed training courses).
  • The call or help-desk system (see how filed tickets).
  • Social media platforms (community intelligence tools).
  • The event /usergroup hosting software (see who attended events).
  • Etc…

In larger organisations, it’s likely there is already some database which integrates a lot of these things. You might simply be able to add community data to it. The challenge is finding out where it is, who owns it, and in what format it’s in.

It’s also common at this stage to use community intelligence tools like CommonRoom and Orbit to create a more detailed picture of member behaviours from a variety of different channels (i.e. you can learn about member engagements on Reddit, Discord, Slack, and other social media channels).

 

Step 2: Extract and combine the data

Sometimes you get lucky and the platforms you’re using have a connector which lets you easily combine data (and keep it updated). It’s far more common, however, that you will need to pull together the different datasets yourself.

This might be by extracting the data from the available APIs, or it might mean manually downloading and combining the data from each source individually. The challenge is often matching up records of different individuals across platforms. If members don’t use a consistent email address, this becomes tricky (but not impossible) to do.

Even if you get the data, the quality of it might vary considerably. API data can be in the form of raw server logs which you then need to process to be useful.

Another challenge is it’s very likely you will be working with datasets which are too large to process on your personal laptop – so you might need to consider virtual solutions for collecting, processing, and securely storing the data (data security is going to be a critical consideration).

As a general principle, cleaning and transforming the data usually takes 80% of the entire scope of the project.

 

Step 3: Analysing the data

Once you have data on individuals, you can still do some really exciting things.

This includes

  • A sequential analysis to determine the order of member behaviors and community participation (i.e. see what came first).
  • Survival analysis to see how retention rate varies by month and member segment.
  • Analyse superuser performance by accepted solution rate.
  • Segment the community by category and see which areas need more attention.
  • See how product usage varies by community participation and types of participation.
  • Identify attributes which cause members to stay for longer.
  • Identify unique clusters of members by community behaviours and build distinct mailing lists of them.
  • Develop detailed journeys of each member.

You should be able to build a model which predicts the outcomes you want to gain from the community and then gain insights into what you need to do to drive those outcomes.

However, this requires expertise in statistics. If you don’t have expertise in this, we can set up this system for you.

Note: you have to be really, really, precise about the definition of any metric you use here. For example, if you want to know the impact of the community on customer retention. You have to define what metrics best represent each of those terms. Two different people can define the terms in two very different ways and come to different conclusions.

 

Step 4: Summarise data in a custom dashboard

Once you have a clean dataset, you can develop a custom dashboard in Tableau, PowerBI, or any tool you’re familiar with. Again, this requires some technical knowledge. But here we would recommend having colour-coded dashboards which help you make the critical decisions you need to make about your community.

Common metrics here would include:

  • Activity-level metrics (no. posts, active members, newcomer retention, and no. active MVPs).
  • Program-level metrics (% questions with a response, average time to first response, accepted solution rate).
  • Satisfaction-level metrics (from polls/surveys).
  • Impact-level metrics (impact upon customer retention, utilisation, or community-driven impact score etc..)

You should know the primary drivers of these metrics and be able to produce performance-based forecasts.

This dashboard should have pre-determined success/failure metrics which identify whether unique community programs succeeded or not. Those programs which succeeded should be provided with additional resources and those which didn’t succeed should be altered or have fewer resources in the future.

From this, everyone can determine both how the community is doing today and what to focus on next.

Resource: Setting Targets and Building A Community Dashboard

Resource: The Dream Community Dataset

 

The Elite Level

The big difference between professional and elite is the automated data pipeline.

A simple overview of a data pipeline is shown via Newsela below:

(p.s. you should watch the full presentation hosted on the Verint/Telligent Community)

At the elite level, you’re not manually pulling and analysing data. It’s not a monthly task you have to (painfully) endure. Instead, it’s a fully automated pipeline of fresh insights.

Typically, data is extracted via APIs (and direct connectors), cleaned, transformed, and stored within a data warehouse. From there a business intelligence or data visualisation tool is used to explore the data. You can provide different colleagues with different levels of access to the data depending on their needs.

This is infinitely more scalable and it gives you regularly updated data (to the day if needed) which others can explore. For larger communities, it’s beneficial to get real-time insights rather than wait until the end of the month to see what’s happening.

The downside is this is a project which requires significant investment. It requires significant resources, expertise, and time. You also need to account for maintenance costs (data pipelines frequently break when platforms are updated or tweaked ).

Before beginning this journey, you need to carefully consider issues such as data quality, integrity, governance, privacy, and security.

The steps you take might vary, but this is what we would usually recommend:

 

Step 1: Create a steering group to define the outcome

Every process is different, but it’s useful to begin by bringing the right people together to start. I’ve lost track of how many data projects stumble because of issues which weren’t foreseen at the time.

The steering group varies but often includes a combination of:

  • Community team
  • Community vendor
  • Company software engineers
  • Customer loyalty/success
  • Marketing
  • Sales
  • Legal
  • IT (esp. Data security-minded folks).
  • Data-specific people

Essentially anyone with any sort of influence or interest in data from the sources you’re likely to collect data from should be included in this process. Your primary responsibility at this point is to learn.

Always begin this process by bringing the right stakeholders together to gather requirements, identify issues and challenges, and understand their needs from the data. It’s likely others want to ensure key outcomes are considered too.

 

Step 2: Decide who will be doing the implementation

The next step is to define the right architecture for your data system.

If you don’t have a data engineering background, please don’t try to do this yourself. Selecting the right architecture and technology requires considerable knowledge of the competing options and a deep understanding of your organization’s own needs and requirements.

In the past, we’ve seen a community data pipeline shut down after two months because the organisation’s data technology stack was moving to a different set of tools.

The earliest decision you need to make is whether you have the resources to do this internally or not. This is whether consultancies can help. You need someone to work with your team to define the requirements in architecture and help select the right technology stack.

 

Step 3: Implement the solution

This is a bigger topic than we will cover here. Perhaps the most important difference in this process is you will first import data from a variety of sources into a data warehouse (Snowflake, BigQuery, Redshift, etc…).

In some cases, this might involve writing a custom webhook to import the data and using separate data orchestration tools. You will also need a data analytics tool which integrates with your data warehouse.

We’re not going to cover the process of building the full data pipeline here. There are far better resources than us to learn about data pipelines. 

Be mindful that this is an intensive process – one that often involves considerable trial and error. This will usually take several months to complete. 

 

Step 4: Training stakeholders to use the data

The final step in the journey is to develop a process for persuading and training stakeholders to use the data to support their work. This applies to both your immediate team and to a broader set of community stakeholders. 

This involves both a combination of emotive-driven storytelling and training workshops where you can guide stakeholders through the process of extracting the data they need. 

I’d recommend having three things in place.

  • A simple roadshow deck explaining the remarkable value of using the tool. This should cover the basics of community, the type of insights people can get, and how to explore the data. 
  • A regular workshop (hosted every 3 to 6 months) for stakeholders can attend where you will help them access and explore the community data. This lets them run analyses themselves, address questions and, most importantly, get time on people’s calendars for the community. 
  • A short guide to support colleagues getting set up on the data visualisation platform – included with some basic do’s and don’t’s. At a minimum, stakeholders should know how to run basic queries and explore the data to find useful insights. 

Resource: How Newsela Build Their Data Pipeline

Resource: Fundamentals of Data Engineering

Resource: What Is A Data Pipeline?

 

Next Steps

If you’ve found this guide useful and want to explore it further, I’d strongly recommend attending our community data workshop on November 2nd.

Ultimately, if you want your community taken seriously you have to take your community data seriously.

Community platforms can show you if engagement is heading in the right direction and who the most active members are, but they’re never going to be able to do much more than that. They can’t answer the critical questions you and your colleagues have. They can’t show a meaningful impact caused by the community.

Many of us need to tread beyond our comfort zone to get the data we need. You don’t need to be a technical person, but you do need to know the process and how to advance to each level. You’re also likely to need outside help.

Begin with the questions you want to answer. You can get a good list of these from speaking with your stakeholders. Refine these questions into specific metrics and start identifying the source of each of these metrics. Be as specific as possible.

Get the environmental data you need and start combining it with community data. Once you can do that, get the member-level data from different databases into a single destination. Now you can consider building an automated system for the whole process.

 

Resources

Community-Driven Impact – A Better Way To Prove The Value of Community

Back in 2018, I stood on stage at the CMX Summit and asked the audience whether they believed it was critical to prove the ROI of their community.

About half the hands in the room shot up.

Excellent!

Next, I asked how many of them could prove the ROI of their community.

The outstretched arms began to wilt. Within seconds, just half a dozen hands remained raised in the air amongst a sea of glum-looking faces.

Now it was time for the kicker.

“If most of you believe it’s critical to measure the ROI of community, but almost no one can do it – how critical can it really be?”

I mean, if it was really critical – life and death critical – wouldn’t anyone who couldn’t prove the ROI be out of work by now?

Either we were all on the slaughterhouse conveyor belt waiting for our turn to get the chop, or we were all very much mistaken about ROI.

Maybe measuring ROI isn’t as critical as we think – at least not in the way we imagine ROI.

Perhaps there’s a better way?

A week ago I hosted the Community-Driven Impact webinar to share a better approach.

It seemed to resonate so I’ll share the highlights here

What Do We Measure If Not ROI?

No one is proclaiming we should be free from the shackles of measurement.

Measurement is ultimately about having the right data to make good decisions.

At the ROI level, these are decisions about whether community programs should have more or less funding.

If we don’t measure anything, we can’t prove anything (i.e. no resources).

We need measurement. We need to be held accountable for results.

Most importantly, we need measurement to show the incredible value of the work we do.

The problem is ROI is a 100+-year-old efficiency ratio from a bygone era.

ROI was developed in a less complex world of tangible products and slower decision-making. It made sense to use ROI when considering major capital investments (‘should we build a new factory?’), it’s less effective in the intangible, service-driven, rapid-decision-making era of today.

It’s certainly possible to measure the ROI of communities, but it’s horribly complex, takes a huge amount of time, and the result is rarely as accurate or as persuasive as you might think.

Most people, when they refer to ROI, don’t mean the percentage return for every dollar spent. What they mean is what is the benefit of this?

Unless you sell membership to the community, your community is several layers removed from profit-level metrics. It doesn’t make sense to use profit-level metrics to evaluate the community.

My argument is twofold.

1) We should be measuring outcomes the community can and does directly influence.

2) We should be measuring these outcomes using a methodology developed precisely for the community.

This is why we’ve recently begun using a new score to measure community.

It’s a score I want to introduce to you today.

We call it Community-Driven Impact.

 

The Problem With Current Measurements

The problem with pretty much every community value metric is it’s not designed for community.

Every method, as you can see below, trades precision for simplicity and must choose between causation and correlation. Neither is close to a great outcome.

methods for measuring community value

Controlled experiments can show great results, but are incredibly difficult and costly to do.

Comparing members to non-members helps set priorities, but doesn’t show causation.

Call deflection is horribly imprecise. Estimates for ‘visitors to deflected tickets’ range from 1 in 20 to 1 in 500. No one really knows if a visitor to an answer received the answer to their question or not. Surveys can help but suffer from a sampling bias.

NPS can show a change in attitudes, but it’s an advocacy score. You might help a member resolve a problem but that doesn’t make them more likely to recommend a brand to friends (honestly, how often do you really recommend any brand to friends?).

What we need is a different measure. A measure which directly shows the unique impact of the community. A measure which colleagues will care about. A measure which is easy to set up and easier to understand.

This is where Community-Driven Impact matters.

 

Introducing Community-Driven Impact

Community-Driven Impact is a metric which combines the two key things people really want to know about your community.

1) How impact does the community have upon visitors? (Average Member Impact)

2) How many people visit the community? (reach)

The method multiplies the reach by Average Member Impact to get a Community-Driven Impact score.

Reach is relatively simple to calculate. It’s simply the number of people who visit the community in any given time frame.

You can set a cut-off for this if you like (i.e. skip visits of less than 30 seconds), but total reach is recommended.

The harder part is calculating the Average Member Impact.

 

Average Member Impact (AMI)

The Average Member Impact is the average impact directly caused by the community on each individual visitor.

 

How To Measure Average Member Impact

The simple method is to adapt the Net Promoter Score question for a community-specific purpose.

We simply ask our members what they felt the impact was.

To put the question in the simplest terms, we ask ‘How has the community impacted [outcome]?’

Behavioral Impact Score Question

You can adapt this question for almost any desired outcome. For example:

How has the [brand] community impacted your ability to achieve [outcome]?

How has the [brand] community influenced your likelihood to utilise [features of product]?

How has the [brand] community influenced your [results of product]?

How did [brand] community influence your decision to [purchase item]

How has the community affected your satisfaction with ?

You can even adapt this question to metrics like call deflection

“Approximately how many times has the community saved you from contacting customer support?”

The magic of this approach is the question is incredibly flexible, it lets members make the causation, and it’s easy to get good data.

You can collect this data by polls or surveys each month/quarter and average the results over time as you see in the chart below:

average member impact graph

How Accurate Is This?

I can hear the howls of criticism already.

“But members don’t give accurate answers to survey questions!”

“Members might have different interpretations of the question!”

“You might suffer from a self-response bias!”

“What if members can’t remember or recall their past behaviour?”

These are valid criticisms. The results won’t be 100% accurate.

But here’s a sneaky secret; neither are most of the other metrics you’re looking at.

Every data point you ever look at is based upon a wide number of assumptions enslaved to a large number of biases. The key is to balance practicality against bias to find the optimal outcome.

The main difference in this approach is we’re not using a black box of complex formulas to show the impact of community. We’re using a completely transparent method to show the impact of community using a measure designed for community.

It’s impossible to criticise this method while still supporting NPS or CSAT as a valid measure of success.

The AMI question gets people to directly connect their community experience to the desired outcome and make an estimate. It takes the burden of causation away from us and lets members highlight what (if any) impact the community has had on their behavior.

The problem is people are generally not great at explaining what drove their behaviour. For example, people claim advertising doesn’t affect them but they keep buying from the biggest advertisers.

This doesn’t invalidate behavior surveys, it instead shows that all behavioral surveys entail a certain level of uncertainty. We should recognise this uncertainty and reference it in our summaries.

The major benefit of calculating Average Member Impact is it doesn’t require us to undertake statistical voodoo to connect the community to the result, it’s a direct question asking for a direct, thoughtful, response. To assume that members can’t understand that is to think extremely poorly of our members.

 

Calculating Community-Driven Impact

Tracking your Average Member Impact is an excellent way to escape the engagement trap (the trap of being measured by the level of engagement).

If you can show more engagement reduces your Average Member Impact score, it doesn’t make sense to continue increasing engagement.

At least in theory it doesn’t.

But there is an obvious problem here.

It doesn’t matter if your Average Member Impact is really high if your reach is really low.

Few brands are going to be impressed if you build a group of 10 really happy members. That’s too many resources devoted to too few people. The level of engagement isn’t the critical number, but it does matter.

Your boss isn’t mistaken for wanting engagement to go up. Where they go wrong is wanting this number to go up without knowing whether increasing engagement impacts impact.

It equally doesn’t make sense to have a community of 100,000 members if the community has no impact on them (i.e. has an Average Member Impact score of zero or below). That’s just a waste of resources.

To balance engagement against impact, we need to calculate Community-Driven Impact.

 

Community-Driven Impact

The Community-Driven Impact score is your Average Member Impact score multiplied by the total number of visitors each month.

In short, it measures how many people the community reaches and the impact the community has upon them.

For example:

  • An AMI score of 2.7 with 10,000 monthly visitors is a Community-Driven Impact score of 27,000.
  • An AMI score of 4.1 with 7,000 monthly visitors is a Community-Driven Impact score of 28,700.

The above example shows the importance of balancing the two metrics against each other. A smaller community with a higher impact is as valuable as a larger community with a smaller impact.

It’s ok if either metric declines as long as the Community-Driven Impact score remains consistent (or increases).

Ultimately, the CDI score is the one which really matters. It combines the value of both reach and impact so people can evaluate community success over time as you can see in the graph below.

community driven impact score graph

Above you can see how both changes in reach and Average Member Impact influence the Community-Driven Impact (CDI) score.

We can see in this example the community is delivering more value over time.

 

This Data Reveals Powerful Insights

These metrics also help you analyse what’s happening in the community.

For example, a client was seeing a long-term decline in monthly visitors but an increase in Average Member Impact.

When we began segmenting the AMI score by member type, we noticed a strong relationship between member tenure and the AMI score.

The AMI score was increasing because people with a lower AMI were leaving the community.

We discovered the community was increasingly catering to the whims of veteran members at the expense of others. They were going to end up with a tiny group of super happy members. Not ideal.

It’s usually a good idea to compare the Average Member Impact score by the following segments

  • Member location/language.
  • Member tenure.
  • Member type.
  • Member participation.
  • Member satisfaction.

You will usually notice segments of members which have lower or higher CDI. This is where you can focus your efforts to achieve the biggest impact. This is how you build a data-driven strategy.

In this example, you can see there are some groups which are doing well and should be grown, others which need to be improved, and some which may not be suitable for the community.

segmented performance graph

If you use a pop-up poll, you can also see if the AMI score changes by where the poll pops up within the community. You might even find the AMI score varies by the category of discussion people are viewing.

This suggests you can focus on specific features or categories to increase the CDI score.

 

What Impacts Your CDI Score?

Once you have a simple score to measure community success, you can start to measure everything you do against that score.

For example, if you’re changing the community platform, adding a new gamification program, hosting more events, introducing a new onboarding program, hiring more staff, or increasing the budget for the community, you can measure the success of all these things by looking at the Community-Driven Impact score.

Statistical buffs amongst you can also look run a multiple regression analysis to see which specific things most influence the community-driven impact score. This might include:

  • No. posts.
  • Average time to first response.
  • No. accepted solutions.
  • No. posts replied to by superuser
  • No. superusers
  • No. events
  • etc…

You can start to see what the statistically significant predictors of the CDI score are and use these as the primary targets to improve in your community strategy. Combine this with qualitative insights from member interviews and you should be able to build a clear picture of what you need to change in the community and why.

You might also discover much of what you’re doing doesn’t have any impact at all and you can halt those activities. You might also notice that other things are closely correlated and develop programs to change these metrics.

For example, if the number of superusers is strongly correlated with the average time to first response and the average time to first response is correlated with the CDI – you might place greater emphasis on recruiting superusers at the expense of other activities which don’t impact CDI.

 

Community-Driven Impact Sits At The Heartbeat Of A Data-Driven Community Strategy

data driven community

This is how you put together a genuinely data-driven community strategy.

There’s no guesswork or erroneous assumptions. You identify the target, examine what impacts that target, and develop the right programs to influence those factors.

There will also be some trial and error. All data is based upon some assumptions which are prone to error and bias. But it’s a process of trial and error which will always eventually land upon the right outcome.

By having a single metric to guide your efforts, you can identify what the results of the community should be and align all your actions to achieve that result.

If you want to learn more or have us set this up for you, drop us a line.

 

Resources

The Beginner’s Guide To Community Management

Today we’re launching The Beginner’s Guide To Community Management.

This is a project which has been some time in the making. Our goal has been to put together a resource which helps newcomers in the industry quickly get up to speed on the basics.

The guide covers:

  • What community management is (and isn’t)
  • What makes the community unique from other company activities
  • Which tools and software to use for a community.
  • What tasks should be undertaken daily, weekly, and monthly.
  • How best to engage members in a community.
  • What metrics matter (and which don’t).

We’ve also thrown in plenty of resources, frameworks, and examples to help us in the journey.

https://www.feverbee.com/guides/the-beginners-guide-to-community-management/

Whether you’re new to the industry or a veteran looking to brush up on the basics, this guide might help.

P.s. please also share it with anyone who might find it useful

This Is How You Identify Your Community’s Biggest Design Issues

The more research we do, the more I’m convinced most community professionals completely misunderstand how members use their sites and what they feel about them.

I say this having sat through hundreds of sessions where members show us how they use the community.

This misunderstanding leads to huge amounts of time being invested in improving aspects of the community experience that members really don’t care much about. Too much time, money, and resources are wasted which could be better invested in higher impact areas.

I want to share our approach to this problem and what we’ve learned from it.

 

Why The Current Approach To User Experience Is Inadequate

The current approach to developing a better community experience is woefully inadequate because of poor research.

Typically, a community person will speak to a few members, ask them what they like and don’t like, and then try to make improvements based on that. Often this leads to really random suggestions floating to the top of the priorities list.

I’ve seen more than a few organisations spending months revamping gamification programs, member profiles, and adding more groups because it came up in member interviews. Later they will wonder why this had no impact.

The problem is members are bad at describing what they want and need.

There is a much, much, better way to improve your community. But it also begins with having the right data to begin with.

 

Watch What Members Do, Not What They Say

Members are terrible at recalling a community experience and how they felt in the moment. Yet this is exactly the kind of information you need.

The secret then is to be there with them in the moment!

The way FeverBee approaches audience research these days is to sit with members and watch how they progress through a community.

I mean this in almost the most literal sense possible. You have to have members sharing their screens as they progress through the community – and ask them questions as they do.

There are several steps to this process.

  • Step 1) Build a list of members to speak to
  • Step 2) Send out the invites and schedule calls.
  • Step 3) Give them tasks to do and see how they do it.
  • Step 4) Identify and prioritise the issues.

Getting each of these steps right is important to understand how to get this process right.

Resource: Rocket Surgery Made Easy (read this – it outlines the process in detail!)

 

Step One – Building A List Of Members To Speak To

The common mistake is to interview the people you know best. These people are usually the most highly engaged members of the community.

The problem is they experience the community in a completely different way from your regular members.

You don’t want to just talk to your superusers, you want to talk to the people who aren’t superusers (yet). You want to build a list of members you can speak to which includes lurkers, newcomers, irregulars, and veterans.

If your community caters to multiple groups, you want that reflected in your research too. This is especially important if you’re dealing with developers, users, administrators etc…or those from diverse geographic locations.

A big word of warning here, don’t recruit people from places like userinterview.com.

While having a group of random people review a site can give you some interesting insights, they won’t behave the way your members do. They won’t begin with the goals and understanding of the topic.

Resource: Build your list of inteviewees

 

Step Two – The Outreach Email (a simple script)

A clumsy outreach email can undermine a good process.

The best outreach messages run along the lines of

“Hi [name],

I’m reaching out because we’re looking to improve our community efforts in the coming year and I’m hoping you might be willing to help.

We’re looking to get your experience on what is and isn’t working for you – and see how you experience the site. 

Would you be willing to spend 30 minutes speaking with [person]? 

No worries if can’t, but it would really help us if you can”

The goal is to keep it personal and low-pressure. We also try not to over-automate the process. We could easily add a calendly link in the first email, but then it would seem less like instruction and not seeking consent.

If we receive a positive response, we can then follow up with….

“That’s great, thank you so much! 

To save time going back and forth, would you mind booking any slot that works for you using this link:

[Calendlylink here]

During the call, we would love to sit and follow how you explore the community in real-time and get your thoughts as you go. You don’t need to prep anything.

Let me know if you have any questions.”

This lets people select a time slot that works for them. Make sure this is connected to a Zoom link too so people can easily participate.

This process has to be friendly, personalised, and solicit their consent before making requests of members to do something.

If you don’t get a response to initial messages, you can add a $20 gift voucher and see if that works. But we’ve usually found we don’t need to do that. Most people are interested in sharing their thoughts and feeling appreciated.

Resource: Calendly (for easy scheduling of interviews)

 

Step Three – Tasks For Members To Participate In

The next critical set is to give the right instructions for members. These calls are typically 30 minutes long (any longer and we find the acceptance rate drops sharply!).

During this process, it’s important to give members specific tasks to do which reflect what you know most members come to the community to do.

Sure, having members casually browse the site and highlight what they like or don’t like is interesting. You might pick up a few pointers. But this isn’t how most people behave. Most people visit the site to do something specific. There’s a certain mindset which is associated with that.

Here are some of the best examples:

  • “You’re looking to solve [common problem], how would you go about doing it? Why did you go there? Why not the community?
  • “You want advice on the best equipment to use to [achieve goal], how would you do it?”
  • “You landed on a discussion page and didn’t find the answer, where would you go next?”
  • “You want to ask a question, how would you do it?”
  • “You want to find information about [common issue], where would you go?”
  • “You’ve noticed an issue you want us to fix, how would you let us know about it?”
  • “There’s something you want changed about our products, how would you tell us about it”

Now watch what members do as they go about solving these problems.

You will quickly get a sense of the issues and how people go about their choices.

 

Task How frequent is this need? What did they do? Why did they do it? Any issues / notes?
Solve a product problem Monthly I search on Google It’s easiest place to get the best information Perceived community as too much effort to find the answer.
Find peers in my position Weekly LinkedIn It’s easier to search by job title. Haven’t seen this as an option
Find latest expertise Monthly Went to the company page and then latest product. Didn’t see any obvious place to get the information in the community. Release notes is hidden and the language is unclear.

 

You might also notice many of these problems might not begin in the community itself, they might begin elsewhere. People might search the documentation or browse the web for answers before visiting the community. These are also good insights to have and to understand their entire journey.

Resource: Interview Script

 

 

Don’t Reveal The Answer In The Question

Notice in the tasks above, we don’t use the common terminology of a community site. We don’t tell people to join groups, connect with others, ask questions, make feature suggestions etc.

If you do this, members simply scan the text and select the words you just gave them. This isn’t how people use the site. The goal isn’t to find out if members can find the community features you’ve named. The goal is to see if members can intuitively achieve their goals using the features the community presents to them.

You want to know if the terminology is right and if the features are positioned where members can see them, and which features members choose to use for which goals.

This is why we don’t tell people to log in to ask a question, we want to see if they figure it out.

This is also why we don’t use ‘feature suggestion’ and instead use “tell us about it”. They have to figure out which sections of the community best guide us to the answer.

 

Don’t Give People ‘Community-Focused’ Tasks

With some exceptions, most people don’t come to the community to do tasks like updating their profile, viewing the gamification score, or joining newcomer groups.

Asking people to do these tasks gives you pretty useless data. The tasks you give people to do should reflect what members want to do before they arrive at the community.

People often want answers to questions, to find people who can help them, and to find the information they need. You want to keep this relatively broad and see how people solve the issue.

 

Prompt People To Share Their Thinking

As you go through the process, you want to prompt members to share what they’re thinking at any given time.

You don’t need to do this every five seconds, but as you watch people try to solve a task they’re given, it’s a good time just to ask them to share what they’re thinking as they do it.

You might get answers like:

“Well I think I need to register, but I can only see a login button and not a register button.” 

This kind of information was useful because the company assumed everyone would realise that to register you simply click the login button. A simple change of text resolved this.

“I can’t be bothered to browse through so many navigation options, so I’m just going to search and see what comes up”

This suggests that the site is overwhelming and people will increasingly rely upon search to resolve their issues.

Another answer might be:

“This is the 4th page of the registration process, it’s really starting to get annoying. I just want to get going”

This was interesting for a client where the registration process was long and they were trying to overly prepare the member for the experience instead of just letting a member dive in and do what they wanted to do.

You don’t need to overdo this. I.e. asking people to do this every 10 seconds just gets annoying (and starts to seem a little insecure). But it’s a good idea to do this when you notice a pause or a moment of uncertainty.

It’s a good idea to use Otter AI or another transcription tool to automatically convert the discussions to text so you can others can analyse it with the recording later.

Resource: Otter AI

 

Step Four – Identify and Prioritise The Issues

As you begin seeing how members really use your site to solve issues, you start to notice the issues.

For example, you might notice no one is reading your onboarding messages and some areas of the community receive almost no clicks.

You might notice people struggle with the search function and often don’t find the answer they want.

In one client, we noticed that rather than browse the community, they would return to Google and search again for a related term with the community brand name and see the results.

You don’t need to do this alone. If you’ve recorded the sessions as you should be able to watch this with colleagues and get a real feel for how people use and experience your community.

List all of these by how critical the issue is.

Here’s a simple scale you can use.

  • 5 – Showstopper – immediately stops the user from progressing any further.
  • 4 – Critical – has a major negative impact on the member experience.
  • 3 – Annoyance – causes a strong negative reaction.
  • 2 – Frustration – causes mild frustration.
  • 1 – Drag – slows the user from achieving their goals.
Issue Cause Severity
The forgot password feature isn’t working. Uncertain why these emails aren’t going through 5
Community is overwhelming – people don’t know where to ask their questions. Too much information on the homepage. Unclear where members should go next. 4
Doesn’t realise the login option is also the registration option. The term ‘register’ isn’t used anywhere 4
Members can’t find the latest information about products. ‘Release notes’ isn’t a term members are familiar with. 3
Registration process takes too much time to progress through. Too many pages have slowed this process down. 3
Members can’t find the posts they’ve recently made. There’s no natural path outside of member profiles to find posts a member has made. 2
Members are unable to connect with people like themselves. There’s no easy way for members to search a member directory by attributes 1

This is just an example, but you get the idea. The majority of issues here are now ranked and can be resolved.

 

Step Five – Prioritise By Effort and Impact

Now we have a prioritised list of issues, we can start thinking about the solutions to each of these issues.

At this stage, you may need to ask around, spend time talking to developers or your platform vendor, and see how other communities have addressed the issue (or get consultancy support).

For each of the issues you identify, you want to know the solution and how much effort the solution will require. You can develop any scale you like, but we prefer a scale such as:

Time Money Misc
1 – Maximum effort 6+ months $50k+ Procurement required or info sec review needed.
2 – High Effort 3 to 6 months $10k to $50k Vendor support required.
3 – Significant Effort 1 to 3 months $1k to $10k Developer needed.
4 – Low Effort 1 to 30 days $100 to $1k Admin support needed
5 – Minimal Effort Less than a day $0 Can personally implement

 

Notice we have inversed the scale here (5 was highest for severity but lowest for effort). This is because we want to prioritise the tasks which give the biggest impact for the least effort.

Now we can put the solutions and prioritised list together.

 

Issue Solution Severity Effort Priority score
Doesn’t realise the login option is also the registration option. Easy tweak to login/register. 4 5 9
The forgot password feature isn’t working. Needs a developer to review the process and resolve it. 5 3 8
Community is overwhelming – people don’t know where to ask their questions. Need a revamp of the community homepage with separate instances for members depending upon profile data. 4 2 6
Members can’t find the posts they’ve recently made. Need to show members the recent posts they’ve made on the homepage when they visit and in the member profile. 2 4 6
Members can’t find the latest information about products. Need a separate signposted area on the homepage and on discussion pages. 3 3 6
The registration process takes too much time to progress through. Need to reduce redundant pages and reduce the information we ask from members. 3 2 5
Members are unable to connect with people like themselves. Need to create a member directory and solicit metadata from members. 1 2 3

 

Now we have a prioritised list of issues identified, potential solutions, and the level of effort required to resolve them. This is something you can use when you build your community roadmap.

You can add and change this over time, but it’s always a good idea to do it.

 

Counter-Intuitive Discoveries You Should Know

In the time we’ve been doing this for clients, we’ve made some discoveries which seem to apply to most communities. So here’s a cheat sheet of things you might want to think about when developing your own community experiences.

  1. The majority of visitors read almost nothing. The overwhelming majority of visitors won’t read more than 2 to 3 words on your homepage. They scan to find the ‘thing’ of interest and then jump to that. You should reduce static homepage text to a minimum.
  2. Members really struggle to follow conversations. Members really want to find the discussions they’ve participated in previously (to check for updates), but they often struggle to do so. They forget what category they posted in and can’t find it from the homepage. This is a huge frustration. It needs to be dead simple for members to find a list of discussions they’ve participated in.
  3. Members switch to a different site quickly. Rather than spending just 5 to 10 seconds trying to get your SSO or two-factor authentication to work, they’re more likely to switch to a different site to find what they need. It’s just too much effort. Sites which have a tendency to log people out often for security reasons cough Salesforce cough struggle with this most.
  4. Most people make a couple of searches for the answer. Asking a question is usually a last resort. Most people with a question will make a few searches and then publish a question.
  5. Members post questions in multiple places. People care more about getting an answer than where they get an answer. Around 50% of people seem to post questions in the community and support at the same time.
  6. Average visitors browse the homepage for 5 to 10 seconds. The average person wants to feel a sense of progress. They browse a homepage for 5 to 10 seconds and then click on something (anything), to feel a sense of momentum. People hate feeling stuck.
  7. Only a tiny percentage of regulars scan the latest discussions. This really surprised us. Despite the common practice of putting the latest questions at the top of the homepage, very few people scan them. Most consider them irrelevant to their problems right now.
  8. Members browse top-level navigation before searching. Most members view what’s in the top-level navigation and click the word that most closely resembles what they’re looking for. What appears in the navigation and how the navigation is structured is critical. Members must be easily able to browse for what they need.
  9. Almost nobody seems to look at the leaderboards. Even the people on the leaderboards don’t seem to look at them often. This is especially the case for static leaderboards based upon cumulative activity since launch as opposed to monthly leaderboards or the past 30 days.
  10. No one reads the welcome message. It really doesn’t matter what appears in the welcome message unless people are confused about the nature of the site. If they don’t understand the community itself, they might glance at the welcome message to comprehend what it’s about. But these are very rare situations.
  11. No one reads the code of conduct. Theoretically, you could put terms and conditions you like in there and people would still agree to it. Although, interestingly, they will read articles titled “The Five Rules Of [community]”. So, maybe give it more of a Fight Club vibe?
  12. Popular discussions. Members generally like seeing the list of popular discussions in a community. This list works best on individual discussion pages as much as on the homepage. However, once a member has seen the list once, they dislike seeing the same discussions again.
  13. Related discussions matter a lot – but only in high-volume communities. At low volume, the related discussions usually aren’t relatable enough. They simply appear because something has to appear.
  14. Recent activity really, really, matters. If the community seems dead, few people ask questions. Recent activity labels are far more important than any static copy/text on the page. No one can recall the liveliness statistics, but they might build up into a more subconscious understanding of how busy the community is.

Even these findings – most of which surprised us – are really only the tip of the iceberg.

The more research you do (with an open mind) the more interesting discoveries you will make.

As long as you’re willing to sacrifice some sacred cows, you can rapidly improve your community experience by following the member journey.

 

Get FeverBee To Undertake Your UX Research

If you don’t have the time or experience doing UX research, we can certainly help. It’s a service we’ve done for many clients and we’ve been able to show rapid improvements.

In this capacity, we can do deep benchmarking of your community experience, undertake interviews with a range of members, and build a detailed list of improvements.

If you like, we can also take this a step further and undertake the design and development work ourselves to bring the community up to the highest possible standard.

If you want help, you can click here to contact us.

 

Resources

Brand Communities Change Customer Attitudes – Here’s How You Prove It

It’s possible to prove the value of community by showing how behaviour is changed.

For example, you can show the impact of a community upon retention, call reduction, or some other behavioural metric.

The problem is it’s really damned hard and you’re never going to have a satisfactory result.

The simplest way to show the value of a community isn’t to prove people behave differently – but people think differently.

It’s a lot easier to measure meaningful changes in member attitudes than meaningful changes in member behaviour.

 

Where Does The Real Value Of Community Show Up?

Another way of thinking about this is to consider where the real value of community shows up.

Take the incredible volume of customers you support. Most people measure this using ‘call deflection’. But a growing body of data shows that people are asking questions in several channels at once and they often ask questions in a community they wouldn’t bother calling support for.

It’s not right to measure this in deflection dollars. But where it will show up is in how people feel about using the products. If they get better results, feel more satisfied, or get unstuck quicker – that shows up in their attitudes.

And this is the most direct benefit of a community – it changes member attitudes.

It’s often quite tragic changes in attitudes aren’t measured. Attitudes drive behaviour.

If you can change the attitude, you can change the behaviour.

 

What Attitudes Can We Measure?

Let’s imagine you went to your boss tomorrow and presented real, live, data that showed, that since joining your community, members have significantly improved satisfaction with products and the likelihood to recommend you to others.

Imagine you could show your community really moved the needle in how people feel about you and your brand and future purchase intent? That’s a powerful metric. Imagine the associations they have with your brand have changed in a more positive way?

Or, for the golden ticket, imagine you could show that brand preference has markedly increased vs. any competitors. Now members consider your brand superior on a range of different attributes.

Or, if you’re working for a non-profit, imagine you could definitively prove that a member’s quality of life scores have increased. Perhaps you could also show they now feel they are better prepared to handle whatever circumstance they find themselves in since joining your community?

In more specific terminology, we often use terms like:

  • Member Satisfaction. (CSAT)
  • Net Promoter Score (NPS)
  • Quality of life (non-profits)
  • Brand attitude
  • Brand perception
  • Brand preference

But it all rolls up into the same key measure – attitudes have changed.

This is the kind of data that is a lot easier to get and far harder to ignore.

 

How Do We Measure A Change In Attitudes?

You probably know the common problem with any metric you want to change by now.

You can easily show any community members score higher in almost any metric than non-members. It’s a lot harder to show it’s the community that caused the metric to be higher.

For example, how do you not know it was simply your best and most loyal customers who joined the community in the first place? Establishing any kind of causal relationship which removes the possible presence of confounding variables is difficult.

But these are some relatively simple solutions. They’re not expensive, but they do take time and you need to be careful with setting up the process correctly.

 

Option 1 – The Controlled Trial

The best solution is to do a controlled test. This is hard to do but it’s not impossible. In this approach, you would segment non-participants at random into two groups. You can send an email to one group with an invitation to join the community and track the results against the other group.

Or, better, only enable one group to see/join the community and not the other group.

In practice, however, this is problematic. For starters, it’s damned hard to stop people from seeing the community. You would have to remove it from search results and configure the technology for the community not to appear for a large group of members. Few organizations are keen on that. And even then you might only be measuring the differences in the characteristics of membership of each group.

If you can do a controlled test, great. But if you can’t, I’d suggest a simpler method…

 

Option 2: Gather Data When Members Join

The reason why few can measure the results of the community is they only collect data based on their timeline instead of the members’ timeline.

It’s a little like – ‘hey it’s October! Time to run our survey!

But there’s nothing particularly special about October (or any month of the year) which is especially good to run a survey. In fact, you’re very likely to bias the outcome by limiting your survey to once a month.

This is why it’s far better to measure member behaviour by their tenure. Specifically, a member hits a certain milestone, it’s a good idea to hit them up with a quick survey to measure their attitudes.

The most important of these milestones is when members join! If you get a baseline attitude survey when members join, you can compare it with later surveys and estimate the results. If you ask the same questions again a year later, you get to see the impact of the community.

 

Why It’s Critical To Capture Member Data When They Join

Let me share an example of why capturing member attitudes when they join is so important.

Last year, I worked with a client whose community reported the lowest NPS of any customer support channel. We’re talking deep in the toilet low!

The community team was getting a lot of criticism for the poor performance of the community.

But was it the community’s fault? 

We began doing some exploring. It soon became clear the community was simply attracting the most frustrated members. These were members who had had negative experiences in other channels and were turning to the community as a last resort.

They had the lowest attitude scores before they even joined the community!

The community was still helping these customers. But they often had problems for which there wasn’t a solution. The community helped them realize that. They weren’t happy about it – but at least they could keep looking.

The problem was no one had taken the time to capture members’ attitudes upon joining. They had no idea if the community had improved the results or not.

(aside – and remember here people who have a negative experience tell several times as many as people. Preventing negative word of mouth is a huge benefit).

Now imagine if they had captured the NPS, CSAT, or other scores when members had joined and could compare progress 6 to 12 months later. You can start to get a sense of the impact of the community.

 

Communities Naturally Attract Members Who Like You More

For example, in the graph below, you can see how much the NPS scores vary by each category of members. For example, avg. non-members today, avg. non-members last year, average new members a year ago, avg. new members today. As you can see here.

graph showing how Communities Naturally Attract Members Who Like You More

This is interesting data, but it doesn’t really show the impact of the community. It more likely shows that people with higher NPS scores might be more likely to join a community and those who like the brand most stay in the community for longer.

What we need to know is the difference between non-members and first-year members over the same time frame. This is where the data starts to get interesting, as you can see below.

graph showing difference between non-members and first year members

The NPS varies (remarkably wildly) from one month to the next. But over the course of the year it appears that community members appear to have a higher difference over that same time frame. You can see this here:

graph showing impacts of Community driven change

Sure there is plenty of variability, but there’s a clear trendline here. Community drives a higher NPS score.

It’s worth noting this isn’t 100% conclusive.

You might simply be measuring the people within each cohort who bothered to complete a survey a year later – people who might naturally be more predisposed to completing a survey. However, by using random survey sampling and offering a small incentive, you should be able to overcome much of that.

This is often known as the ‘difference in differences’ approach. You compare groups and track the impact over time as you see below.

Quick caveat here, if you’re working with a tiny community, this might not work for you. You probably can’t get enough responses without offering some substantial incentives. But for any community which has more than a few thousand members, you should be able to undertake a rotating study and show the results.

 

How Do We Measure Results?

Before we measure the results, let’s understand what we’re measuring here. Metrics like Net Promoter Score (NPS) and Customer Satisfaction (CSAT) have been extensively covered before.

Let’s focus on the three lesser-known attitudinal metrics we want to cover. Ones which are real game-changers for organizations.

Brand Perception

I think of Apple as stylish, easy to use, and expensive.

I think of Logitech as basic, functional, and simple.

I think of Google as useful, friendly, and accessible.

You probably have (very) different perceptions of all three.

But we aggregate the results from a large enough group of people and you will soon start to build a great understanding of how people perceive of each brand.

These perceptions matter a lot. They are a leading indicator of future purchase intent, retention, and likelihood to promote the community.

So imagine your community could profoundly change how people perceive your brand.

Imagine if when members joined the community they used terms like confusing, overwhelming, and fiddly. Then a year later they used terms like automated, supportive, and comprehensive.

In less than a year, you can show you have completely changed members’ attitudes about your brand.

This is incredibly valuable data which too few people ever measure.

 

Brand Preference

Perhaps the only thing more valuable than changing perceptions is changing preferences.

Brand preference is where you compare your brand against those of a selected group of direct and indirect competitors and get feedback from your members on how they rate each of you. It’s one of the most common and powerful impacts of community – but few people ever try to measure it!

Imagine you can go to your boss tomorrow and say because of your community, members have now increased preference of your brand against competitors by xx% points. Or perhaps they simply associate your brand with more positive terms than any competitor.

If you can show that 10,000 people have improved their preference for your brand since being in the community, it becomes a no-brainer to try and get as many people engaged in the community as possible.

Aside, the other major benefit of this kind of research is you get a unique insight into how people think and feel about your brand. I’m often amazed that organizations don’t constantly do these kinds of surveys given just how remarkable the results tend to be. You can provide your marketing and PR teams with precisely the terms they need to use and messaging to deploy to achieve the results they want.

Second aside, you can also take this a step further and measure the relationship between brand preference and purchases in the future to see exactly how strong this connection is.

 

The Danger of Measuring Brand Attitudes

Let’s quickly highlight an obvious danger of measuring member attitudes.

The obvious danger is the results might show you member attitudes declined since joining the community. On the surface, this suggests the community is having a negative impact.

This is a risk, but even if it is true, you 100% want to know this!

Once you have this data, you can begin interviewing members and drilling a bit deeper into your data to determine the precise cause.

For example, you might find it’s a specific segment of members having a negative experience.

In one client, when we drilled deeper into the data, we found it was people having issues with one particular product line. They were never likely to the solution they wanted because there wasn’t a feasible solution to the major problems they were raising. They felt they were wasting their time on the community and we were seeing hundreds of questions go unanswered.

We came up with a simple solution. If the community couldn’t help with these kinds of issues, it shouldn’t support these kinds of issues. We directed these questions directly to other support channels and closed down this section. Any time you notice the community is doing more harm than good, you can figure out why and take action.

 

What About Non-Profit Communities?

This might all sound good if you work for a typical business, but what if you’re working for a non-profit? Does any of this apply to you?

I’d argue it applies more to non-profits than any other organization. The only difference is you’re measuring different attitudes. Instead of looking at brand attitudes, you’re looking at more representative metrics such as quality of life, increased capacity, or overall confidence in managing their circumstances.

A Quality of Life Survey, for example, can show the impact of community across a range of well-being factors such as health, relationships, finances, and overall life satisfaction. Sometimes the impact of a community might appear in places you don’t expect.

 

Setting Up The Survey (Or Poll)

You can set up a survey to capture data both before and after someone has joined a community.

Ideally, you want to have the same people participating in surveys each year. A 1-year frame can work.

One way of doing this is offering a small incentive if they participate in the survey when they join and then offering them a larger one if they complete the survey a year later. Another option is simply to compare the average of newcomers vs. veterans and assume nothing else changed during this time period.

 

Questions For All Surveys

You can adapt these to your situation and environment. You may want to ask additional identifier questions about the type of customer or individual so you can drill deeper into the data later. You might ask questions about the type of products purchased, gender, age, location etc…

Typically the key questions should include:

Q. For approximately how long have you been visiting the community?

(This helps you to separate members who have just joined from those who have been around for a while. If you’re triggering the survey by member tenure in the community you can skip this question)

Q. How would you rate your overall satisfaction with [the product/community]?

  • Very unsatisfied // Unsatisfied // Neutral // Satisfied // Very satisfied

(This is a question which can help establish overall satisfaction with the community environment. You can skip it if you’ve got too many questions, but it can help establish if the community is driving an improvement in results)

Q. If you would like to receive [discount/benefit], please enter your email address below

(This is simply a tracking question to be included at the end. It helps you identify who is participating in the survey and match it up to your customer database. Sometimes you don’t need this, but it’s usually useful to be able to follow up with people who took the survey the previous year)

 

Example Questions For Brand-Attitude Surveys

If you’re running a survey on brand attitudes, you might ask a combination of the following questions.

(NPS qs.) Q. On a scale of 0 to 10, how likely are you to recommend [brand] to a friend/colleague?

This is the standard NPS question. 

(brand perception qs) Q. On a scale of 1 to 10, Please rate how well each of the following traits describes our brand.

List of traits here. This might include a mix of attitude and behavioural traits. For example:

  • Value for money.
  • Broadest feature set.
  • Better quality than other brands.
  • Innovative.
  • Caters to my unique needs.
  • Easy to set up and use.
  • Great customer support.
  • Great documentation.
  • The staff seems friendly and supportive.

You can add any set of attributes you like here which people can use to evaluate your brand. Try to be as specific as possible. 

Aside – An alternative approach begins with the attributes you want to learn more about and asks members to select which brand most relates to which attribute. This is useful if you know specific attributes are most important.

Resource: Brand Perception Questions

(brand preference qs) Q. Before purchasing from [brand], which other organizations did you consider?

This is good for knowing who you should be comparing against and you can then program each of these options to appear in the evaluative set to ensure you’re not asking people to compare brands they never considered. 

(brand preference qs) Q. How important or unimportant were the following factors in your decision to purchase from [brand] rather than any other brand?

  • Extremely important
  • Very important
  • Moderately important
  • Slightly important
  • Not at all important
  • No opinion

This question will help you identify the key factors which drove the purchase decision. List the most likely factors here and include a rating scale along the lines of

(brand preference) Q. What (if anything) might make you switch to a competitor? 

You can also suggest possible answers and add an open text box. But it’s generally better to let members complete the answers themselves. 

(brand preference) Q. Which brands would you most associate with the following attributes

  • Value for money.
  • Strong customer support.
  • Comprehensive feature set.
  • Easy to use
  • Etc…

This is the critical question. It’s a direct comparison question where you use the relevant brands provided in the previous answer to populate answers for people to complete. Zero in here on the attributes which you feel best drive purchasing behaviour – you can use the answers from above. 

Resource: Brand Preference Questions

 

Example Questions For Non-Profits

Developing questions for non-profits is a little trickier. Every non-profit supports a different circumstance and ideally, questions should best address that circumstance. This may include:

[non-profits] Q. How would you rate your ‘confidence to handle [circumstance]?

  • Not confident at all
  • Slightly confident
  • Moderately confident
  • Somewhat confident
  • Very confident

This is the simplest question to use. It provides a simple snapshot answer where people can rate their level of confidence on a single scale and you can track results over time. You should be able to see precisely the impact of the community. 

[non-profits] Q. How would you rate [quality of life, mental health, physical health, social life etc..] over the past 4 weeks?

  • Poor
  • Fair
  • Good
  • Very good
  • Excellent

You can repeat this question with several variations above to capture the full impact of the community across a range of factors. This lets you identify where the community has shown results in areas where you might otherwise not expect. You can go further to deploy a full quality of life survey using the resource below. 

Resource: Quality of Life Questions

Resource: Writing Survey Questions

 

Next Steps

Behaviour is primarily driven by attitudes. Your community shapes and influences those attitudes in a major way. You’ve probably had countless interactions which you know have had a major impact on your audience – and you probably haven’t tracked the outcome of those interactions.

Notice we use the word ‘track’ rather than ‘measure’ here. Measuring gives you a snapshot of today. That’s interesting, but not very helpful. Tracking attitudes over time will help you understand and prove the impact of your community.

Imagine how powerful it is to have results like this to share with colleagues:

However, if you want this to work you have to set up the data properly. You have to prepare the dataset today for the results you want to show tomorrow (or 6 to 12 months from now).

When execs are against the community, they often dismiss the ROI data which can be complicated and prone to all sorts of attribution issues. However, it’s harder to dismiss attitude data – especially when it’s been properly collected and analysed

Here are some simple next steps

  1. Decide which attitudes you’re going to track.
  2. Setup the survey questions and test the survey on a small audience.
  3. Setup the survey to trigger members based upon tenure (time in or no. visits to the community).
  4. Automatically pull the data into a visualisation tool (Tableau, Looker, PowerBI etc..)
  5. Drill into the results to determine which segments/factors most impact the results.

 

Let FeverBee Calculate Your Attitude Change Score

All of the above takes a lot of effort to set up and ensure the data is collected properly. This is where FeverBee can take care of the process for you. We’ve worked with plenty of clients to measure and analyse the impact of their communities.

We can take on the entire process and simply provide you with the data you need to prove the value of your community.

If you want help, drop us a line.

 

Key Takeaways

  1. It’s easier to measure meaningful changes in attitudes than behavior.
  2. Track attitudes over time – especially when people join.
  3. Brand perception, brand preference, and quality of life are valuable data points to have.

 

Resources

 

©2023 FeverBee Limited, Apartment 2410, 251 Southwark Bridge Road, London SE1 6FN England FEVERBEE