It’s incredibly hard to prove the value of a community.
Unless you have the time, expertise, willpower, and capabilities to do a controlled trial; most data analyses establish a relationship. The direction of that relationship (and whether there are confounding variables) is often left unexplored.
However, this doesn’t mean you can’t gather some really interesting insights. You may not be able to definitively prove the value of the community, but you can estimate the possible value of the community using statistically valid methods.
Pragmatic Institute Needed Help To Measure Their Community
In March 2022, The Pragmatic Institute hired us to help them determine the ROI of their community.
The Pragmatic Institute is a provider of expert training for Data, Design, and Product Teams. The Pragmatic Institute primarily sells to organizations – that often subscribe multiple staff members to courses at any given time.
To support their work, they created an online community hosted by Higher Logic. The community was popular and active. Members were clearly getting value from participating in the community. However, it had proven difficult to show whether engaging in the community led to more sales or better outcomes for The Pragmatic Institute.
FeverBee was tasked with answering three key questions:
1) What is the connection between community participation and the number of courses taken?
2) How might community activity impact training revenue?
3) What kind of community behavior should be encouraged to improve the metrics above?
Step One: Preparing the dataset
The first step of the process was to build a proper database.
We began by preparing the three datasets:
1) The dataset of Pragmatic users from Salesforce.
2) The community activity dataset from Higher Logic
3) The sales data from Salesforce.
Like many organizations, member and customer data was scattered across multiple channels.
The community was hosted on Higher Logic which contained all the engagement-level data. Purchase data lived in one set of Salesforce reports while course completion data lived in another.
To make things even more challenging, the individuals who took the courses weren’t usually the people who paid for them. Finally, the data often contained inconsistencies in how the names of organizations were recorded. Sometimes acronyms were used, other times the entire name was used.
Over several weeks, we cleaned the data, identified and resolved missing variables, and analyzed the outcomes.
Step Two: Running comparisons
Next, we began running comparisons between different segments of members. This included comparing active members with inactive members. We also compared members who have engaged in the community with those that didn’t.
When variables had a sufficient distribution, we performed linear regressions to look for a linear relationship between behavior and revenue generated per organization. We also grouped members by the number of posts they had made to determine if a specific number of engagements was associated with an increased probability of future sales (this is known as ordinal regression)
Step Three: Presenting Results
Once we had undertaken this analysis, we examined and visualized the results to estimate the potential value of the community. This is what we found…
What Is The Pragmatic Community Worth?
Overall, almost every behavior was associated with greater revenue.
- Active members (>1 post) generated 21% more revenue than inactive members.
- Members who make at least one post contribute 44% more revenue.
- Members who make at least one thread contribute 35% more revenue.
- Members who log in to the community at least once contribute 21% more revenue.
- Members who have replied to at least 1 direct message contribute 38% more revenue.
- Members who contribute a document contribute 38% more revenue.
- Members who download a document contribute 20% more revenue.
- The more frequently members log in, the greater the revenue they generate.
By our calculations, community participation was associated with over $1.5m in additional revenue.
Which Behaviors Were Most Valuable?
It’s one thing to know the value of the community, it’s another to dive a little deeper to determine which behaviors are more important in driving that behavior.
In our case, we wanted to know which community behaviors increased the probability of members taking an additional course.
Here the data showed three important metrics.
- Replying to a DM increased the probability of taking an additional course by 16%.
- Creating a document increased the probability of taking an additional course by 23%
- Creating a thread increased the probability of taking an additional course by 201%
The probability of taking a future course was directly related to downloading documents, creating threads, and replying to direct messages.
This helped set some clear and specific strategic priorities within the community.
These priorities included:
- Encouraging members to create threads
- Creating more documents to download.
- Getting more responses from members.
The Value Of Community Intelligence
There are far too many community teams who know their community is incredibly valuable but are unable to prove it.
As we’ve seen with the Pragmatic Institute, even if your data is scattered across several channels, we’re still able to combine this data and evaluate the impact of the community upon revenue, attitudes, and almost any other desired outcome.
If you can clearly define the variables, you can usually measure them. Better yet, we are able to specifically show which behaviors are most important and should be encouraged in the community at the expense of others.
As we’ve seen before, this helps you build a solid community strategy.
Once you can prove the value of the community through statistically valid methods, it becomes a lot easier to gain both internal validation and additional support.
If you know your community has value but you’re not able to definitively prove it yet, I’d suggest getting help from our community intelligence services.