Understanding the meaning of a confidence interval can take a little effort. The key idea is we want to infer findings from a study to subjects who were not part of the study. Sometimes, reading explanations in different words can help. Let’s pause in our series and see how others have explained what confidence intervals mean:

Harvey Motulsky:

The true population value either lies within the 95% CI you calculated or it doesn’t. There is no way for you to know. If you were to calculate a 95% CI from many samples, you would expect it to include the population proportion in about 95% of the samples and to not include the population proportion in about 5% of the samples. (p 35)

Geoff Cumming & Robert Calin-Jageman:

It’s always correct to think of our CI … as one from the notionally infinite sequence of intervals we’d obtain if the experiment were repeated indefinitely. Each interval is one from the dance of the CIs. (p 113)

… it’s acceptable to say “we are 95% confident our interval includes [the population parameter]”, provided we keep in mind that we’re referring to 95% of the intervals in the dance including [the population parameter]. (p 115)

Stanton Glantz:

The specific 95 percent confidence interval associated with a given set of data will or will not actually include the true size of the treatment effect, but in the long run 95 percent of all possible 95 percent confidence intervals will include the true difference of mean values associated with the treatment. As such, [the confidence interval] describes not only the size of the effect but quantifies the certainty with which one can estimate the size of the treatment effect. (p 227)

Notice the idea of repeated testing to estimate the population value, e.g. “many samples”, “infinite sequence of intervals”, “in the long run”, “all possible confidence intervals” etc. It conveys the idea of testing frequently in order to make inferences. For this reason, estimation statistics are also known as frequency-based or frequentist statistics, in case you have heard these terms.

What is the value of using confidence intervals? Here is one explanation in the context of clinical trials, which is also applicable to exploratory and basic science research.

Stuart Pocock:

The main purpose of a clinical trial [or study making inferences about effects, associations, or other physical phenomena; my addition] should be to estimate the magnitude of improvement of one treatment over another. Although significance tests give the strength of evidence for one treatment being better they do not tell one how much better. Hence, significance tests are not the finale of analysis but should be followed by statistical estimation methods such as confidence limits … (p 206)

So, confidence intervals give an idea of the precision about an effect “in the long run”. This idea is important because ultimately, we want to make inferences about the population when we only have data from samples.

In the next post, we will see how variability in data is quantified.

References

Cumming G & Calin-Jageman R (2017) Introduction to the new statistics. Estimation, open science and beyond. Routledge: New York, USA.

Glantz SA (2005) Primer of biostatistics. (6th Ed) McGraw-Hill: New York, USA.

Motulsky H (2018) Intuitive biostatistics. A nonmathematical guide to statistical thinking. 4th Ed. Oxford University Press: Oxford, UK.

Pocock SJ (2009) Clinical trials. A Practical Approach. John Wiley & Sons: New Jersey, USA.