For a long time, public relations was measured by one of two things.
Either the number of press clippings you could achieve for clients or what’s called advertising value equivalent.
Advertising value equivalent is essentially a metric where you multiply the advertising rate in each publication by the column inches you achieve in that publication. Then you combine the total amount to say how much PR is worth.
And while AVE offered a precise, scientific-sounding measure that PR firms loved to share with clients, it increasingly came to be seen as problematic.
In 2010, many of the world’s top communication professionals gathered in Barcelona for the International Association for the Measurement and Evaluation of Communication (AMEC) summit, and they agreed on a set of principles known as the Barcelona Principles.
One principle stated that AVE is not the value of public relations.
But they didn’t select a preferred option. They left the outcome to PR professionals. At the time, many worried that not being able to share a precise dollar-value return would kill the profession. But over the following years, as AVE gradually died as a measurement tool, PR firms’ popularity and success only grew.
In short, bad measurements held PR back for years, and only when they got rid of them could better techniques and processes take their place.
Enterprise community folks are now in a similar place. For a long time, we’ve used call deflection metrics. Call deflection is essentially the number of visitors who viewed answered questions, multiplied by the percentage who say they got the answer they wanted, multiplied by the average cost of a support ticket.
Call deflection value = visitors to answered questions × % who say they got the answer they wanted × average cost of a support ticket
And this ends up giving a very precise value. And typically it gives a very high value. That’s because the measure essentially gives the highest number you can possibly give.
And because the numbers are often so high, they’re often unbelievable.
I remember speaking to one exec who complained that the value of community deflections is three times higher than his entire customer support budget.
There are several reasons for this, and it’s important to understand each one.
1) Communities Attract Non-Critical Questions. When you create a community, you lower the barrier to asking a question. This means you attract a lot of questions people otherwise wouldn’t have made the effort to ask a customer rep. They aren’t ticket deflections. Their value would show up as improved customer satisfaction, product adoption, or increased sentiment and usage.
2) It’s Hard To Prove People Achieved A Resolution. The only real way to prove community resolved an individual question is to ask people if it was resolved. So organizations usually use a poll asking people, “Did it solve your question?” Sometimes, as follow-ups: “Would you have otherwise contacted support?” and “Are you in warranty or not?”
The problem is that polls are heavily influenced by response bias, and we generalize the results across all traffic.
Even if people were only on the community for a couple of seconds and there’s no way they could have received the answer, we still count that. We’ve also found there’s a curvilinear relationship between how long it takes the poll to appear and the percentage response rates.
By delaying the poll by a few extra seconds, you can double the response rate and, theoretically, double your community’s value, which undermines the entire point.
3) People Might File A Ticket Anyway. When we’ve studied how people resolve problems, we’ve found they don’t just ask a question in the community and stop. They might ask the same question in multiple communities and file a support ticket or two because dissipating the frustration matters more than the effort required to resolve the problem.
That means the community often gets credit for people who also filed a ticket.
4) Resolving A Problem Is Messy. Sometimes snippets and fragments of multiple answers are combined together to give a response. Sometimes problems have no resolution. An answer can mean people can stop looking, but it doesn’t necessarily mean there’s a resolution.
While deflection metrics are still widely used, they’re incredibly problematic and have never come remotely close to calculating the true value of a community (or, most likely, actual support costs saved).
The Final Nail In The Call Deflection Coffin
Since the rise of LLMs, the introduction of Google search snippets, and, most importantly, AI overviews, engagement rates across most enterprise communities have dropped significantly.
Many have seen declines in engagement of 20% to 40% year on year, and engagement keeps falling.
This doesn’t mean the community is resolving fewer questions.
It simply means people don’t need to visit the community for answers. It comes up naturally in search, so traditional call deflection methods will show a sharp decline in a community’s value.
At the same time, its real value is likely holding steady or even rising as knowledge spreads through more channels than ever, and this problem will only get worse.
Now we have increasingly complex and precise ways to address this. You can measure ghost deflections, as we spoke about last year, something that Microsoft has done and talked about a lot.
The problem is this is complex and difficult to prove, and sudden changes in the algorithm or Search Console calculations can wreak havoc with your stats.
And this isn’t going to change. It’s only going to get worse.
Usage of LLMs is going up. Fewer people will visit a community in the future than in the past.
People will increasingly retrieve community knowledge through other channels, so the old way of measuring deflection (any calculation that includes website visitors) is dead.
Which raises the question: what should replace it?
The Misleading Quest For A Precise Value
We try to measure the value of a community to sustain it or gain more internal support. CFOs, directors of finance, and other departments use it to decide how much of their budget to allocate to a community against the competing needs of other functions and departments.
But there’s also a misunderstanding in how these decisions are made.
These decisions aren’t based on which function delivers the highest return. They’re based on certainty, the time to achieve that return, and whether people trust the number in the first place.
The problem with call deflection for so long is that the methodology has been sketchy, the numbers are too high, and too few people actually trust that number.
What we need isn’t a higher number.
What we need is a number people can trust, and this number doesn’t need to be a dollar-value return. Most functions that an organization undertakes don’t deliver a dollar-value return. They are a couple of layers removed from that.
What we need to show is the impact on the organization’s priorities, so you don’t need to show a high dollar-value return if you can show a big impact in metrics that matter.
So drop the need to show a dollar-value return and instead focus on impact on customer satisfaction, self-service scores, resolution rates, and whatever metrics matter to the organization.
The Phoenix Measure Rising From The Deflection Ashes
For years we’ve been asking the wrong question.
We’ve been asking: how many calls or tickets does our community prevent (and what’s the value of those calls or tickets)?
The better question is:
What percentage of customer questions can now be answered because of the community?
For example, if the self-serve rate of customer questions is 55% without community knowledge (i.e., official resources/documentation) and 77% with community, that’s a fantastic outcome.
The best way of measuring this is to take a random sample of 100 questions and then answer two questions:
- What % of them can be answered without community knowledge?
- What % of them can be answered with community knowledge?
This gives you your additional coverage rate.
How Do We Measure This?
The ideal scenario is to pull 100 queries from your support tickets / AI logs. Then divide people into two groups at random and ask one group to solve the problem using only official documentation and another to solve it using both community and documentation.
This would give you the most reliable data, but it takes time and resources most teams don’t have.
A simpler approach is to get AI to help.
Unable to use any client’s actual support logs, I asked Claude to take 150 randomly selected Jira Cloud and Jira Service Management Cloud questions from the public Atlassian Community and then answer them in two ways. Once using official documentation and community content, and once using official documentation only.
Part of the problem is that Claude has been trained on community data and official documentation. It can’t forget it even if you ask it to. So I asked Claude to cite every source and the exact sentence it used in its response, then verify that none of the answers were unsourced.
The response was illuminating.
These findings are really interesting for a few reasons.
- The resolution rate with the community is 9 percentage points higher than without. I suspect that understates the true value, but it’s worth noting.
- The community either provides a better answer or the only working answer in 33% of questions. This shows the community’s unique value in an AI world.
- Community adds value to questions where docs have an answer. Community often adds additional context, confirms which answers work, helps people know what to search for, etc.
When we actually drill deeper into where community added this unique value, the answers were truly illuminating:
A quick aside: I ran the same study on the Smartsheet community, and the results were surprisingly consistent.
The community gain was even larger: 84% of questions could be fully answered with official documentation alone, rising to 98.7% with community content added (a 14.7 percentage point gain).
So what does all this mean?
It means the main value of community isn’t just resolving questions that otherwise couldn’t be resolved. It’s improving the knowledge that does exist by adding context and experiences of customers who use these products every day.
You can see some examples of this below:
Notice how frequent and how valuable many of these contributions actually are!
And yet if you’re just measuring deflection, none of this will show up. That’s the problem with deflection today.
What Does This Experiment Really Show About Community Value?
The critical finding of this experiment isn’t that the community can answer questions that documentation can’t (though there are plenty of examples of that, and a clear percentage increase in resolution rate from connecting community to AI bots and including community in RAG search, etc.).
Community adds details and context that official documentation often misses.
It shows what happens when people follow instructions but still get stuck.
It highlights workarounds to problems when things don’t work as expected.
It can identify and document issues and limitations, helping customers avoid repeating mistakes others have already made.
The problem, though, is: where will this value appear? What metrics will you see change as a result of this unique community value? I suspect it’s here:
- Answer quality: More customer questions can be answered accurately and completely because answers can draw on community knowledge, not just documentation. This might show up as more accurate AI answers, better answer ratings, and fewer follow-up questions.
- Customer outcomes: If customers get better answers from the community, they will complete more tasks successfully. They should resolve problems faster and spend less time searching for help or troubleshooting issues.
- Reduced costs: Customers will be able to self-serve better, so demand for support should fall. If agents, both AI and human, can solve problems faster, resolution costs should decline. This also means that customer satisfaction and product adoption will probably improve.
These are the kinds of outcomes we definitely should be measuring. We should stop measuring how many people who visited a community might otherwise have contacted support. That’s a speculative financial metric that misunderstands a community’s real purpose and value.
Instead, a much better test is what happens when we make community knowledge available versus when it isn’t.
What does the additional coverage rate look like then?
What does problem resolution or task completion look like then?
What is the time and effort customers have to expend to resolve their issue?
And how happy are they with those resolutions?
Does AI provide better answers? Do customers complete more tasks? Are problems resolved faster? Are fewer issues escalated and reopened?
I believe the future of many enterprise communities, especially support-centric ones, isn’t about proving how many tickets we prevented. It’s about proving how much better our customers perform because of the knowledge our communities create.
Let Us Do Your Audit
The problem is that many organizations don’t have the time or ability to run experiments like this to prove the unique value their community adds beyond their documentation. If you want help, contact us and let FeverBee undertake the activity for you.
We can run the entire experiment, show the results, and help you communicate the results internally. We can also help you design systems where the organization can best capture the knowledge and deploy the knowledge generated by the community.
If you’re interested, you can click here to contact us. We would love to help you not only prove the value of your community but also ensure your organization is getting the most value from your community.