If you knew there was a little poison in a well, would it matter how big the well was?
I worry that in our excitement to come up with data to measure and prove the value of communities, there’s a lot of poison slipping into the well.
Sometimes it’s clearly bogus stats, other times it’s rather suspect measures of ROI. Often it’s simply a biased interpretation of existing data.
It might feel like you’re doing a good thing to spread positive data about the industry, but if that data is suspect or poorly interpreted it simply poisons the well for good data.
No-one believes video views or ad impression data anymore. Advertising and social platforms have poisoned that well forever. Let’s not go down that path.
So how do we balance the need to build up a body of evidence of our community success without poisoning the well?
A few suggestions…
First, maintain a healthy skepticism about claims of a community’s success from the people running the community. If the community did achieve a remarkable ROI, you should also see an expansion of the community team, further investment in the platform etc…If that’s not happening (and they’re not providing the underlying data or methodology), treat these claims as propaganda.
Second, go looking in Google Scholar. There are thousands of peer-reviewed papers on almost every conceivable topic of communities. Go looking for evidence of what is and isn’t working. You might be surprised how rich this information is. Better yet, hire someone to do a literature review for you on your topic. I’ve done this for both of my books in the past.
Third, commit to genuine inquiry of the value of your community. Hire an outside consultant if you want the right methodology and an unbiased approach. This isn’t cheap, but you’ll get rock-solid data.
This isn’t as fun as participating in the hype cycle, but it’s a lot more value in the long run.
