On a call this week, one prospect mentioned they’ve removed outdated discussions when they’re flagged to the community team, but that’s it.
They’ve never really undertaken a proper curation process or built an ongoing system to ensure the huge volume of information shared in a community over the past decade is accurate.
Every time they migrated platforms, they also migrated all the old discussions.
Here’s why that’s becoming a huge problem.
The Problem With Knowledge Decay
There’s a concept called ‘Knowledge Decay’ in communities (especially those related to products).
In short, answers that were right at the time they were published (and have been marked as the best answer with lots of likes) might no longer be right today.
For example, this weekend I asked ChatGPT a simple question about n8n, the automation tool:
“I’m on n8n Cloud Starter. Whenever I activate a new workflow, one of my other workflows deactivates itself. Why?”
Here’s the answer:
Notice the beginning of its answer (sourced from the community):
“The most likely explanation is that you’re hitting n8n Cloud Starter’s active workflow limit.”
The problem is that the answer is wrong.
It has decayed.
It was right when it was posted, but it isn’t right today.
Last year, the founder announced that these plans are unlimited.
Worse, the post was automatically closed after 90 days, so no one can flag the outdated thread.
Both posts stay live without corrections. When ChatGPT went looking for answers, it led with the wrong one.
What Causes Knowledge Decay?
Knowledge decay happens whenever a product or service changes. Every price change, renamed feature, retired workaround, or policy update makes previously accurate answers inaccurate.
And ChatGPT (and other LLMs) have no reliable way to know which is the best answer. Here, the old thread had 534 views, and the founder’s announcement had over 13,000. ChatGPT still led with the old one.
Is an answer posted yesterday with no likes/best answer better than one posted two years ago with lots of likes and a best answer?
Sometimes the answer is yes; sometimes it’s no. It depends.
But there’s no consistent way for a person, let alone an LLM, to know.
And because nobody goes back to mark our old answers as outdated, this problem grows over time.
Your Community Is Probably Spreading A Lot Of Misinformation
Here’s an illustrative example. A community that never cleans up anything for 10 years will likely have as many incorrect answers as correct ones.
Yikes! It only takes a relatively small percentage of answers to go out of date each year for the community to begin spreading a lot of false information.
You Need To Clean Up Your Community
The solution here is twofold:
- Clean up the decayed content in your community. This is a process of working at scale to identify decayed content, curate it, add freshness signals, and eliminate contradictions.
- Build an ongoing review system. It’s not enough to do this once; you need an ongoing process to prevent it from happening again. You need to implement a program (alongside existing community programs) to properly curate community content.
Both are becoming imperative in the age of AI.
Even if you don’t connect it to your internal AI tools, LLMs are already training on this data and spewing out bad responses. These bad responses make customers more frustrated with your brand.
Worse yet, any of your colleagues might notice LLMs regurgitating bad information they learned from the community and suggest (as one client faced three weeks ago) the community should be closed down!
Learn How To Do This For Yourself
We’ve been helping clients tackle this problem for almost two years.
But now we’re also going to teach you to do it for yourself in our AI-Ready Communities Program.
This program will teach you how to ensure your community adds tremendous value to LLMs without spreading misleading or incorrect information.
We’ll work through together the five critical areas to prepare your community for AI. These topics are:
- Duplication and contradiction. How to find duplicate and contradictory answers in your community and resolve them.
- Decay and freshness. How to spot content that has gone out of date and add the freshness signals AI relies on.
- Authority. How to decide which answers are authoritative, so AI knows which answer to trust.
- Taxonomy. How to standardize your categories and tags so content is easy to find and classify.
- Terminology and keeping it clean. How to make sure the same thing is called the same thing everywhere, and how to stop new mess from building up.
The program runs live every Monday from 23 November to 14 December (5pm UK / 12pm New York / 9am San Francisco). Each session lasts an hour and is recorded, and the prep material opens on Monday 16 November.
The fee for this program is $1,750, but if you sign up before Friday, November 13 using the code EARLYBIRD, you get a $250 discount.
Bring your colleagues!
We’ve found this program works best when you invite colleagues to join you. The fee is per organization, so you can invite as many colleagues as you like, whether they work in the community or not.