Analysis of Variance (ANOVA)

What is an analysis of variance (ANOVA)?

An analysis of variance (ANOVA for short; Analysis of Variance) is a statistical method. It is used to test whether there are statistically significant differences between the means of two or more groups. In practice, ANOVA is especially useful when comparing three or more groups.

What is ANOVA used for?

• You have two or more groups (e.g., multiple classes, treatments, or products).
• You want to know whether the average values really differ, or whether the difference is just due to chance.

Difference from the t-test

• Independent-samples t-test: compares exactly two groups.
• ANOVA: can compare two or more groups.

Types of analysis of variance

There are different forms. The most common are:

• One-way ANOVA (1 factor)
• Two-way ANOVA (2 factors)

Both variants also exist:

• with repeated measures (the same people/objects are measured multiple times)
• without repeated measures (different people/objects per group)

• One-factor (or one-way) ANOVA
• Two-factor (or two-way) ANOVA
• One-factor ANOVA with repeated measures
• Two-factor ANOVA with repeated measures

Why not calculate multiple t-tests?

ANOVA is used when there are more than two groups. Of course, it would also be a possibility to calculate a t-test for each pair of groups. The problem is that each additional test increases the probability of obtaining at least one false-positive result. If each test uses a significance level of 5%, this does not mean that exactly every 20^th test will be significant by chance.

For example, comparing 20 groups requires 190 pairwise tests. If there are actually no group differences, conducting all these tests without a multiple-comparison correction creates a high probability of at least one significant result by chance. ANOVA first provides one overall test; adjusted post-hoc comparisons can then be used if needed.

Difference between one-way and two-way ANOVA

A one-way analysis of variance examines whether the mean of a metric dependent variable differs across the levels of one factor. For example, it can examine whether mean salary differs by place of residence. If two factors are considered, a two-way analysis of variance can test the main effect of each factor and whether the factors interact.

Two-way analysis of variance tests the main effects of two factors and their interaction. The interaction indicates whether the association between one factor and the dependent variable differs across the levels of the other factor.

Analysis of variance with and without repeated measures

Depending on whether the sample is independent or dependent, either analysis of variance with or without repeated measures is used. If the same person was interviewed at several points in time, the sample is a dependent sample and analysis of variance with repeated measures is used.

One-way ANOVA

The one-way analysis of variance is an extension of the t-test for independent groups. With the t-test only a maximum of two groups can be compared; this is now extended to more than two groups. For two independent groups (k = 2), the classical one-way ANOVA is equivalent to the pooled independent-samples t-test, with F = t². The independent variable is accordingly a categorical variable with at least two levels. The dependent variable is on a metric scale. In the case of the analysis of variance, the independent variable is referred to as the factor.

The aim of ANOVA is to explain as much variance as possible in the dependent variable by dividing it into the groups. Let us consider the following example.

One-way ANOVA example

With the help of the independent variable, e.g. "highest educational qualification" with the three characteristics group 1, group 2 and group 3 should be explained as much variance of the dependent variable "salary" as possible. In the graphic below, under A) a lot of variance can be explained with the three groups and under B) only very little variance.

Accordingly, group membership explains much more of the variation in salary in case A) than in case B). This pattern alone does not establish that group membership causes the salary differences.

In the case of A), the values in the respective groups deviate only slightly from the group mean, the variance within the groups is therefore very small. In the case of B), however, the variance within the groups is large. The variance between the groups is the other way round; it is large in the case of A) and small in the case of B). In the case of B) the group means are close together, in the case of A) they are not.

Analysis of variance hypotheses

The null hypothesis and the alternative hypothesis result from a one-way analysis of variance as follows:

• Null hypothesis H₀: All population group means are equal.
• Alternative hypothesis H₁: At least one population group mean differs.

A significant omnibus ANOVA indicates that not all group means are equal, but it does not identify which groups differ. If no comparisons were planned in advance, adjusted post-hoc tests can be used to investigate the group differences while controlling for multiple comparisons. Common choices include Tukey's test when variances are reasonably equal and the Games-Howell test when they are not.

An example of the one-way analysis of variance would be to investigate whether the daily coffee consumption of students from different fields of study differs significantly.

Assumptions for one-way analysis of variance

• Scale level: The scale level of the dependent variable must be metric, whereas the independent variable must be nominally scaled.
• Homogeneity: The variances in each group should be roughly the same. This can be checked with Levene's test.
• Normality: The residuals within each group should be approximately normally distributed. ANOVA is often reasonably robust to moderate deviations, especially with similar group sizes, but strong skewness or outliers can cause problems. For independent groups, the Kruskal-Wallis test may be an alternative when its assumptions and rank-based interpretation fit the research question.
• Independence: Observations must be independent within and between groups. If the same people or matched observations are measured repeatedly, a repeated-measures method is required.

If there are no independent samples but dependent ones, then a one-way analysis of variance with repeated measures is used.

Welch's ANOVA

If the condition of variance homogeneity is not fulfilled, Welch's ANOVA can be calculated instead of the classical ANOVA. If Levene's test indicates unequal variances, numiqo automatically calculates Welch's ANOVA in addition. Welch's ANOVA is particularly useful when group sizes are also unequal.

Effect size Eta squared (η²)

The best known measures of effect size for analysis of variance are the Eta squared and the partial Eta squared. For a one-way ANOVA, the Eta squared and the partial Eta squared are identical.

Eta squared estimates the proportion of the total sample variance associated with the factor. It is calculated by dividing the between-groups sum of squares by the total sum of squares. Eta squared can be upwardly biased as an estimate of the population effect; omega squared is a less biased alternative.

Two-way analysis of variance

As the name suggests, two-way analysis of variance examines how the mean of a dependent variable differs across the levels of two factors. This extends the one-way analysis of variance by a further categorical independent variable. A two-way ANOVA tests both main effects and the interaction between the factors.

Example with numiqo

One-way analysis of variance:

You want to check whether there is a difference in coffee consumption between students in different subjects. To do this, ask 10 students from each field of study.

After the table above has been copied into the hypothesis test calculator, simply click on Hypothesis test and select the two variables (subject and coffee consumption). The result looks like this:

Here, n is the number of cases in each group, df is the degrees of freedom, F is the ratio of the between-groups mean square to the within-groups mean square, and p is the p-value. In this example, p = 0.505, so the data do not provide evidence that mean coffee consumption differs among the three fields of study.