Suppose you conduct a study to compare an outcome between healthy people and people with stroke, but you want to check whether the proportion of men and women in each group are similar. The Fisher’s Exact test is often used to test differences in proportions between groups, similar to a Chi-Square test. But what is an exact test?

This helpful blog post by statistician Karen Grace-Martin at The Analysis Factor explains what exact tests are. Most statistical tests calculate a p-value based on how a statistic (e.g. a mean or a proportion) is distributed. In contrast, exact tests calculate a p-value empirically. For example, if a pair of dice are rolled and the values of both faces are added, there are 6 x 6 = 36 possible outcomes. Of these, there are 6 possible ways to roll the dice to get total values of 10, 11 and 12. So the empirical probability of rolling a 10 or higher is 6 / 36 = 0.167.

Exact tests calculate the empirical probability of getting an outcome as different or more from the null hypothesis, compared to the outcome observed in the data. For example, the table below lists hypothetical numbers of men and women in the control and stroke groups, with 15 subjects in each group. The Fisher’s Exact test is used to test the hypothesis that the proportion of men and women in each group is the same. The test counts all possible ways that the 30 observations can fall into a 2 x 2 table with the same row and column totals. It then counts the number of tables with proportions that are the same or more different than the observed table, and calculates the exact p-value.

From the table, we might expect the proportions of men and women to be different between groups. The following Python code is used to perform a Fisher’s Exact test on the tabled data:

from scipy.stats import fisher_exact

con_m, con_f = 8, 7
str_m, str_f = 14, 1

oddsratio, pvalue = fisher_exact([[con_m, con_f], [str_m, str_f]])
print('Fisher exact test p-value: {:.4f}'.format(pvalue))

Out[1]:
Fisher exact test p-value: 0.0352

As expected, the exact p-value is statistically significant, showing that the proportions of men and women between groups are different.

Summary

Exact tests calculate empirical probabilities. The Fisher’s Exact test, which is one example, can be used to test for differences in proportions between groups.