How To Develop A Community Strategy

Data-Driven Improvements

Chapters  LinkedIn  Google Plus
 

You should never have to ask what to measure. If you don’t know what to measure, then you don’t know what you are trying to achieve. If you don’t know what you’re trying to achieve, then why measure anything?

A far better question to ask is how to measure. How do you measure if your objectives are achieving your goals? How do you measure if your strategies are achieving your objectives? How do you measure if your tactics are coming through on your strategy? How do you measure if the tactics themselves are well executed? These are the key things worth measuring.

Why bother measuring anything?

Why go to the effort of measuring anything? What do you want to know? What do you plan to do with the data?

There are two common reasons here. The first is you might measure to impress your boss. This helps you keep your job and get more resources for future efforts. This is often part-vanity, too. You want to know how you personally are doing to increase the level of engagement.

The second reason is you measure to improve the work you do. This doesn’t get anywhere near as much attention as it should. Almost all discussions concern ‘what’ you should measure instead of ‘why’ you measure. Data is a valuable tool to make future decisions. You should use data to better allocate your resources to achieve your goal. Data tells you where you have been wasting time and where you can better spend your time to achieve your goals.

This means you need a decision tree in place to handle data. For example, if the level of people participating in a type of discussion falls. Does this mean you need to spend less time on this discussion or spend more time promoting it? If you are collecting data without a decision tree, you’re wasting your time.

The goal of measurement is not to find out what has been happening. The goal of measurement is to improve what you’ve been doing.

Define, Measure, Analyze, Improve

You need to distinguish between three important concepts here: data, analysis, and insights.

  • Data are the raw metrics. Data show you what’s happened (e.g. visitor numbers increased by 5% this month).
  • Analysis tells you why it happened (e.g. this was due to increased investments in search optimization).
  • Insights tell you what you’re going to do with this information (e.g. we should invest more of our time and money in search engine optimization techniques).

These are three parts of the DMAIC framework (define, measure, analyze, improve, control).

In this context, you want to focus on defining what you will measure, analyze, and then make insight-driven improvements.

DEFINE: What Should You Measure?

Measure execution first

Notice these are in reverse chronological order from the strategy. This is important. It’s pointless to measure whether the strategy achieved the objectives if the tactics were not well executed in the first place.

Your data might tell you that the strategy failed. Yet, the tactics to execute the strategy only reached a tiny percentage of the entire audience. It was the execution of the tactic that failed, not the strategy.

The first step is to define what variables you intend to measure. There are four critical questions you want to answer here. These are:

  1. Were the tactics well-executed? (success of the action plan)
  2. Did the tactics amplify the emotion? (success of tactics)
  3. Did the emotion change the behavior? (success of strategy)
  4. Did the behavior generate a return? (success of objectives)

This tells you whether the action plan was successfully executed, whether you chose the right tactics, whether you used the right strategy, and whether you had the right objectives in the first place.

The challenge here is to identify the right proxy metrics for each of these questions. Proxy metrics are metrics that are presumed to reflect the variable we wish to measure.

For example, you might not be able to measure if more of the community members bought a specific new product, but you might be able to track how many members mention a new product or visited a purchase page.

MEASURE: Collect Data To Reveal What Happened

It’s harder to collect good data than what you might think. You might have access to sophisticated analytical tools, but these tools aren’t designed to answer the kinds of questions you have.

The Problem With Google Analytics

Don’t rely upon a single analytical tool (e.g. Google Analytics) for measurement. The current crop of analytical tools are excellent, but with one key problem: they are not designed to measure your community. Google Analytics, for example, is designed to help content creators sell more Google Ads to their audience.

You need to develop a framework to measure your specific community with your specific goals. This is going to take a lot more work than using an off-the-shelf package, but this extra work will yield exactly the data that will let you make improvements and save a huge amount of time later on.

You need to pull good data from a variety of sources and avoid biasing your findings. It’s very easy to cherry pick data to tell any story you would like. Try to avoid this by establishing how you will measure an activity before you initiate it.

To get good data, you usually need to use three distinct techniques:

  1. Direct measurement. This is the easiest type. The data already exists and you just need to find it. Usually, this means you need to collect it from an existing analytics package. Sometimes it might mean collecting the data from other departments within your organization.
  2. Sampling. This is more complicated. This is when you sample a group of members to measure change over a period of time. This is most common when the data doesn’t already exist and you need to track new variables. Perhaps the most common of these are surveys. You often need to set benchmarks before you begin to execute the strategy.
  3. Experiments. This is the most complicated. This is when you run an experiment or a test to see whether the change in a variable had a desired impact. You must set up the parameters of a test before you try to change the variable.

Collecting data is usually the most time-consuming part of this process. Once you have identified how you will measure the variables you have defined, it’s best to outsource this to someone else. For manual labor tasks, there are a variety of data-collection agencies that can usually perform this service for you.

This data will tell you whether or not your strategic plan was successful. If proving success is the only thing you care about, you can stop here. But if you want to use data to improve your efforts, read ahead.

ANALYZE: Explain what happened

The data above will explain whether the variables changed, but it won’t answer the really critical question of why it did or didn’t change. To understand ‘why’, you usually need some context. This usually means more data-points and information than you have collected so far.

Why Did Traffic Increase

For example, you might notice that visitor numbers have risen. This is usually good news. But it’s not actionable news. We might dig into the data a little more and see that this is because search traffic has risen. Digging a little further, we might identify what specific pages and search terms are being targeted.

Analysis usually means making comparisons. This includes:

  1. Examining antecedents. What happened before, after, or duration the change in the variable? Did anything closely correlate with the variables we’re measuring? Correlation doesn’t prove causation, but it might suggest a possible relationship you can test. For example, if the number of new registrations rose, is this because the conversion rate increased, or were you attracting more people to the site in the first place? You could compare the registration rate with the new visitor rate to find the answer. This is an example of an antecedent.
  2. Comparing trends. You might also look at the long-term trends. What is the long-term trend here? Has the trend line changed? If so, by what degree? If you look at the long-term trend of that variable, you will find more information. For example, you might find that the number of new registrations has been rising for several months. This means your activities in the past month might not be the cause. It’s a longer-term trend.
  3. Comparing dates. A similar approach is to compare dates against similar periods. Trend lines can be influenced by holiday or weather, for example. How does the variable change compare against a comparable date? The previous day, week, month, or year for example. How does the number of new registrations compare with the same period last year? Are external factors a likely cause or not?
  4. Comparing against benchmarks. Another approach is to look at benchmarks. How does this compare against other activities of similar size? Are you more or less successful than them? For example, you might notice mentions of your product rose in your community. However, if the level of activity or members also rose by a similar percentage, it’s probably the result of an extraneous variable. Alternatively, you might benchmark against communities of similar size. How does your registration rate compare with theirs? This might reveal what you are doing better or worse than others.
  5. Feedback from team members or community members. Another useful datapoint is feedback from team members or community members. They are often closer to the action and can highlight any small changes which might have had a big impact. For example, small changes in wording on title tags can have a big impact on search rankings. If no-one told you this has changed, you might struggle to explain the data you see. The registration rate might have increased because of something one of your team has changed.

At this stage, you do not want to describe what happened, but you want to explain why it happened. Once you know why something failed or succeeded, you can begin to make useful extrapolations about what to do next.

Improve (what will you do differently?)

This is the critical step that is missing from most improvement efforts. This is where you highlight how you will improve your outcomes by changing how you undertake processes. This can mean a couple of things:

  1. Repeat the process (and fix the problem). Sometimes, the new skills and knowledge you gained the first time around will will help you fix the problems you encountered. This is the best option when there is clearly a mistake that was made and can be fixed the next time round.
  2. Dedicate more resources to the process. This applies in two situations. The first is when a process isn’t working because it hasn’t been properly resourced. If the process isn’t working because not enough time, money and resources have been spent on it, this might be the best option. In practice, it’s rare to double-down on failing processes. The second situation is when a process is working and you’re reinvesting in its future success. It’s usually a good idea to invest more time and effort into the things which are working.
  3. Stop the process. This is an obvious solution for processes that are not working. If something is not cascading into the next level (e.g. tactics affecting strategy), it’s usually best to kill the tactic. Unless you have sunk considerable costs in the process*, have identified a clear problem that can be fixed, or plan to allocate more resources to the process, the best idea is to stop doing it and allocate your limited resources elsewhere.
  4. Try a different process. Another method of improvement is to try something new altogether. This is the riskiest and, unfortunately, the most common option. It’s also the option least likely to succeed.

If you have tried something before, you have learned a lot about what makes it work or not work. You’ve made progress towards finding the right answer. You’re more likely to succeed the next time. In practice, it’s usually easier to spend more time on things that are working than to try something new. Trying something new means starting from scratch on something you haven’t done before.

Working Not Working
Easy benefit from extra resources Allocate more resources Fix the process
No benefit from extra resources Repeat the process Stop the process

You can see the clear choices before you.

You might notice that, so far, we haven’t talked about what analytics package to use or how to set up a dashboard. This is deliberate. It’s far too easy to dive into any analytics package (Google Analytics, Omniture, Community-Analytics, etc.) and find a bunch of data that’s interesting.

But ‘interesting’ and ‘useful’ are polar opposites on the value continuum. You can spend hours looking at interesting data believing you’re making headway. Try not to do this. Don’t ever open an analytics package unless you know exactly what you’re measuring and how you’re going to make your analysis. Then, once you do open it, ignore everything outside of this scope. This is simply noise.

Measuring What Matters

Far too frequently we jump into Google Analytics and begin measuring a whole range of things which really don’t matter much. We measure visitors, users, time on site, bounce rate, goal conversion and more. None of these matter unless they fall within the framework above.

With this in mind, it’s time to define what you need to measure.

* Sunk costs are usually a terrible influence on decision making. They often lead to sending good money after bad money. Here, it is whether the time and effort invested in executing a tactic (e.g. building relationships, establishing skills, etc.) would save you time compared with a similar tactic.

Summary

  1. Don’t open any analytics tool until you know what you’re going to extract from it.
  2. Don’t measure anything until you know what you will do with the information. You need to build the model before you measure.
  3. Use the DMAIC model. Define what you will measure, measure the data, analyze why it happened, and improve it.
  4. Find out what happened. You measure this by finding proxy metrics, directly collecting data, sampling, or running experiments.
  5. Find out why it happened by analyzing by examining contextual information. This includes antecedents, trends, dates, benchmarks, and qualitative data.
  6. Decide what you will do with it. This means how you will improve by repeating the process, fixing the process, stopping the process, or trying a new process.

Chapters

Share

 LinkedIn LinkedIn

 Google Plus Google+