Significance

In statistical analysis the word “significant” means that there is evidence that effect found in the sample exists in the population from which the sample was drawn. The choice of the word “significant” is unfortunate, as it is used to mean something different in common language. Reporters hear a scientist say that there is a significant effect, and tend to think big. Results gets reported as significant, meaning big, and we have effect inflation.

In reality, if we take a large enough sample, even a small effect will show up as significant. Because the sample is large, it is easier to detect and be sure of the existence of small effects in the population. However, this does not mean that the effect is notable or makes any difference.
Unfortunately this confusion is rife in the reporting of medical and educational research. A drug may have a statistically significant effect, which means that there is evidence that it exists in the population, but it may be to reduce incidence from 2 in a thousand to 1 in a thousand, which isn’t really much of a difference. To make matters worse a result like this can also be stated as a 50% decrease, which makes it seem even more miraculous.
This post is more about learning statistics, but those who teach it really need to be alert for this misconception. We have just posted our latest YouTube video explaining significance and usefulness, evidence and strength, association and causation.

Within the video I have tried to give memorable images, which students will hold onto, even if they don’t quite remember the reasoning. The p-value shrinking as the Evidence label grows, aims to help students understand that a small p means more evidence to reject the null. I’m also really pleased with my “p-machine”, turning the mean, standard deviation and sample size into a t statistic, which is then converted to a p-value.
There are just a few really big ideas in statistics, and these are some of them. This forms part of statistical literacy, which is important to all citizens. I hope you may find the video useful in helping students remember.