Performing a hypothesis test comes with the risk of obtaining either a Type 1 or Type 2 error.

• Type 1 error: Rejecting a true null hypothesis
• Type 2 error: Accepting a false null hypothesis

When analysing different groups, a one-way ANOVA can tell us if there is a statistically significant difference between those groups.

However, it cannot tell us which group is different from another. Moreover, when performing multiple hypothesis tests at once, the probability of obtaining a Type 1 error increases.

In statistics, this is known as the family-wise error rate, which measures the probability that a Type 1 error will be made across any particular hypothesis test.

It is calculated as follows:

1 — (1-α)^n

where:

α = the significance level for a given hypothesis test

n = total number of tests

For instance, if we are using a significance level of 0.05 and we conduct three hypothesis tests, the probability of making a Type 1 error increases to 14.26%, i.e. 1-(1–0.05)³ = 0.1426.

To guard against such a Type 1 error (and also to concurrently conduct pairwise t-tests between each group), a Bonferroni correction is used whereby the significance level is adjusted to reduce the probability of committing a Type 1 error. However, a downside of this test is that the probability of committing a Type 2 error also increases.

Hotel Revenue: Analysing Average Daily Rates Across Distribution Channels

For this example, let us consider a hotel that has collected data on the average daily rate for each of its customers, i.e. the average price that the customer pays per day to stay at the hotel.

The hotel also has information on the distribution channel pertaining to each customer, i.e. Corporate, Direct, and TA/TO. The goal of the analysis is to determine the differences across means in ADR for each of these three groups. The original data was sourced from Antonio, Almeida and Nunes (2019) as referenced below, and 100 samples from each distribution channel were randomly selected.

One-way ANOVA

Let’s start by conducting a one-way ANOVA in R.

> model <- aov(ADR ~ DistributionChannel, data = data)
> summary(model)
                     Df  Sum Sq Mean Sq F value   Pr(>F)    
DistributionChannel   2  236636  118318   33.33 8.76e-14 ***
Residuals           297 1054469    3550                     
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

When analysing the results, we can see that the p-value is highly significant and virtually zero. This means we reject the null hypothesis that no significant differences exist between each group.

However, we can see that the ANOVA test merely indicates that a difference exists between the three distribution channels — it does not tell us anything about the nature of that difference.

Before performing the pairwise p-test, here is a boxplot illustrating the differences across the three groups:

From a visual glance, we can see that the mean ADR across the Direct and TA/TO distribution channels is higher than that of Corporate, and the dispersion across ADR is significantly greater.

Pairwise t-test with Bonferroni correction

However, we would like to analyse this in more detail using a pairwise t-test with a Bonferroni correction.

> pairwise.t.test(data$ADR, data$DistributionChannel, p.adjust.method="bonferroni")Pairwise comparisons using t tests with pooled SDdata:  data$ADR and data$DistributionChannel       Corporate Direct
Direct 4.6e-11   -     
TA/TO  2.4e-11   1P value adjustment method: bonferroni

When looking at the adjusted p-values, we can see that the differences between Corporate and Direct, and Corporate and TA/TO are highly significant as the p-values are near zero. Given that the Bonferroni correction has been used to guard against Type 1 errors, we can be more confident in rejecting the null hypothesis of no significant differences across groups.

That said, we can see that there exists a p-value of 1 between the Direct and TA/TO groups, implying that we cannot reject the null hypothesis of no significant differences between these two groups.

Conclusion

This has been a short introduction to pairwise t-tests and specifically, the use of the Bonferroni correction to guard against Type 1 errors. You have seen:

• The limitations of using a one-way ANOVA
• How to calculate the family-wise error rate
• How to conduct a pairwise t-test using a Bonferroni correction and interpret the results

Many thanks for your time, and any questions or feedback are greatly appreciated.

Michael Grogan Data Science Consultant with expertise in economics, time series analysis, and Bayesian methods