What is an exact test? An example using Fisher’s Exact test

Suppose you conduct a study to compare an outcome between healthy people and people with stroke, but you want to check whether the proportion of men and women in each group are similar. The Fisher’s Exact test is often used to test differences in proportions between groups, similar to a Chi-Square test. But what is an exact test?

This helpful blog post by statistician Karen Grace-Martin at The Analysis Factor explains what exact tests are. Most statistical tests calculate a p-value based on how a statistic (e.g. a mean or a proportion) is distributed. In contrast, exact tests calculate a p-value empirically. For example, if a pair of dice are rolled and the values of both faces are added, there are 6 x 6 = 36 possible outcomes. Of these, there are 6 possible ways to roll the dice to get total values of 10, 11 and 12. So the empirical probability of rolling a 10 or higher is 6 / 36 = 0.167.

Exact tests calculate the empirical probability of getting an outcome as different or more from the null hypothesis, compared to the outcome observed in the data. For example, the table below lists hypothetical numbers of men and women in the control and stroke groups, with 15 subjects in each group. The Fisher’s Exact test is used to test the hypothesis that the proportion of men and women in each group is the same. The test counts all possible ways that the 30 observations can fall into a 2 x 2 table with the same row and column totals. It then counts the number of tables with proportions that are the same or more different than the observed table, and calculates the exact p-value.

male female total
control 8 7 15
stroke 14 1 15
total 22 8 30

From the table, we might expect the proportions of men and women to be different between groups. The following Python code is used to perform a Fisher’s Exact test on the tabled data:

from scipy.stats import fisher_exact

con_m, con_f = 8, 7
str_m, str_f = 14, 1

oddsratio, pvalue = fisher_exact([[con_m, con_f], [str_m, str_f]])
print('Fisher exact test p-value: {:.4f}'.format(pvalue))

Out[1]:
Fisher exact test p-value: 0.0352

As expected, the exact p-value is statistically significant, showing that the proportions of men and women between groups are different.

Summary

Exact tests calculate empirical probabilities. The Fisher’s Exact test, which is one example, can be used to test for differences in proportions between groups.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s