P-Values and Random Fluctuations
In statistics, the p-value is the probability of seeing the same (or more extreme) outcome by chance.
For example, I might claim I have a lucky coin that always lands on heads. If I flip the coin twice and it lands on heads both times, does this prove my theory?
Nope. You simply witnessed the 1 in 4 occasions where that outcome was statistically likely to occur.
The lower the p-value, the more likely there is something going on. If my coin landed on heads for the 5th time in a row (p = 0.031), you might want to start inspecting the coin.
Our web traffic rose by 10% yesterday. Is this because yesterday’s blog post was great? Our Google search ranking increased? Or was it just the 1 in 30 days where our traffic fluctuates by 10%.
The danger begins when you attribute random fluctuations to specific events. This will lead you down the fruitless path of trying to replicate false results.
For example, a webinar guest might attract a 20% bigger audience. An activity might increase participation by 15% one month. Is this because of the guest or the interesting discussion topics? Or is it simply the outcome of random fluctuations?
The more you learn about statistics the more skeptical you should become of your observations.