Now you need to answer perhaps the most important question. Are the objectives achieving your goals? If your objective was to get existing members to share their best tips within the community, did that achieve the goal of the community?
It’s easier to set objectives from above. It’s far more difficult to determine if you set the right objectives. It’s hard to be critical of your own work to determine whether it was a success.
Defining What To Measure
The first obvious question to ask is did you achieve your goal? The example goal here was to increase customer satisfaction. Did customer satisfaction of members actually increase during this period? If so, by how much?
This leads into two easy variables to measure:
Measure if community members are more satisfied with products.
Measure if non-members are more satisfied with their products.
The latter is quite important. Many people who access tips might not be members of the community. Yet, their satisfaction is just as likely to increase by accessing the advice. This means we need to measure both members and non-members.
At this level, you can usually rely upon more standard proxies of success. A CSAT score, for example, is measured by a survey asking members to rate their level of satisfaction with the product on a scale of 1 to 10. A net promoter score (NPS) is measured by asking customers how likely they are to recommend the product to a friend.
You want to measure the score of members via a survey. Then you need to use a survey to measure the score of visitors. You might use a pop-up survey to non-registered visitors to track the latter or email a sample of your total audience. Ideally, you want to measure this over an ongoing period of time (i.e. send the survey out to a different group each month) or have it pop up at random each month. Be careful of the non-response bias:
Before and after surveys measuring CSAT score of members.
Before and after surveys measuring CSAT score of visitors.
This will tell you whether customer satisfaction increased. However, it can’t tell you whether this was due to you achieving your objectives or some random factor.
This leads into the second question. Is the change in CSAT score attributable to the community?
Once again, you can look at the correlation between the two scores. You can see the correlation below:
This shows that the goal was achieved to some degree (i.e. the CSAT score increased by a significant amount of both members and non-members). However, the correlation is stronger for non-members than members.
Then, you can measure other possible factors that might account for the change in CSAT score.
Now you can figure out what we need to do differently.
If the correlation is high between objectives and goals, and the goal was achieved, you need to allocate more resources to the community. This might come at the expense of other objectives or simply acquiring more resources from the organization.
If the correlation is high between objectives and goals, but the goal was not achieved, you need to change the strategy to achieve the objectives. You have the right objectives but the strategy is not succeeding.
If the correlation is low between objectives and goals, and the goal was achieved, you need to figure out what is working. It is likely that other objectives are succeeding, or your actions are causing another behavior which is increasing customer satisfaction.
If the correlation is low between objectives and the goal was not achieved, you need to change the objectives. In practice, this often means ending the community efforts as a strategy.
You can go one level further again and use a multiple regression analysis to determine what impact each of the objectives had on the goal and allocate your resources accordingly. This is the best option but requires an understanding of statistics and statistical packages.
Notice the key insights here. These might be hard to extract. If the correlation is much higher for non-members, it suggests that there could be a natural ceiling to the degree to which a community can influence the CSAT score of members.
This might mean the best additional investment might come from getting more people to see the tips in the first place and, thus, a shift in resources to attract newcomers.
You can use conventional measures of ROI as proxy metrics to determine if you set the right objectives.
Run a correlational analysis (or, ideally, a multiple regression analysis) to determine if there is a relationship between the objective you set and the achievement towards the goal.
If the correlation is high, keep the same objectives and either shift the strategy if it is not working, or invest more resources if it is working.
If the correlation is low, you need to set new objectives.