Degrees of Freedom (df) in Statistics

This tutorial explains how to calculate, use, and interpret degrees of freedom in statistics.

What are Degrees of Freedom in general?

Degrees of freedom are the number of independent pieces of information you have left to vary when estimating something.

To make this intuitive, we first look at an example from mechanics and then apply the same logic to statistics.

Degrees of Freedom Example in Mechanics

In mechanics, the degrees of freedom tell us how many independent ways something can move.

If a point can move left and right, we have one degree of freedom. If it can also move up and down, we have a second. If it can move back and forth, we have a third.

So in total we have three degrees of freedom. Therefore, a point in three dimensional space has three degrees of freedom.

For example a helicopter moving through space has three degrees of freedom for translation: left-right, up-down, and forward-back. Yes, it can also rotate, but we'll ignore rotation here.

But what if we look at an elevator? An elevator just can move up and down, because it's constrained by the shaft. Therefore, because it can just move up and down, it has only one degree of freedom.

If we have two elevators, each one can move up or down. If we treat them as one system, we have two degrees of freedom in total.

If we have three elevators we there are three degrees of freedom, each can move up or down.

So, let's take it a step further with another example, which then will bring us close to what degrees of freedoms are in statistics.

Degrees of Freedom Example

Imagine in a city you've got a small hill with two funiculars moving up and down. The cars are linked by a cable. When one goes up, the other must go down.

Two independent cars would give the system two degrees of freedom, like two elevators. The connecting cable is a constraint, so the system loses one degree of freedom.

Imagine three cars connected by a single cable. Because the cable length is fixed, if the first car moves up by some distance, the combined downward movement of the second and third must equal that same distance.

For example, the second car could move down exactly that amount while the third stays put, or they could split the distance in any proportion.

We can also write this constraint mathematically. If we measure each car's height from the ground, the sum of their heights is constant. For example, h1 + h2 + h3 might always add up to 300 m.

Because one equation ties h1, h2, and h3 together, we lose one degree of freedom. With three variables and one constraint, we have 3 - 1 = 2 degrees of freedom.

In mechanics the constraint is tangible-the cable-and you can also write it as an equation. In statistics, the constraints are less visible, but the idea is the same.

What are Constraints in Statistics?

In statistics, especially in hypothesis testing, we use a sample to make a statement about the population. Here is the key point: we use the sample twice.

First, we use the sample to test our claim. Second, we use the same sample to estimate quantities like the mean.

Let us walk through a t-test and see how degrees of freedom work.

What are the Degrees of Freedom in a t-test?

The t-test-more precisely, the independent-samples t-test-checks whether two groups, for example group A and group B, differ significantly.

So first we need two samples. If we collect 8 people in each group, we have 8 independent data points in each group.

To test whether these two samples differ, we use each group's mean. To calculate it, we use the same data we are testing.

These means are like the cable in the mechanical example-they tie the data points together and create a constraint.

If we have a sample of seven people and calculate the mean from this data, only six of them can vary freely. If we know the mean and the first six values, we automatically know the value of the seventh person.

Just like in the mechanics example: if the cars are connected with a cable and we know the position of the first two cars, we automatically know the position of the last one.

With two groups, we calculate the mean of both samples. We therefore have two constraints and lose two degrees of freedom.

In an independent-samples t-test, the degrees of freedom are n₁ + n₂ - 2, where n₁ and n₂ are the sample sizes of the two groups and the minus 2 comes from the two means.

If we have 9 data points and want to fit a simple linear regression, we estimate two parameters-the slope and the intercept. That costs 2 degrees of freedom. Therefore, for a simple linear regression, the degrees of freedom are the number of data points minus 2.

In statistics, degrees of freedom tell you how many pieces of information in your data are still free to vary after you have used some of them to estimate quantities like a mean.

Why do we need Degrees of Freedom?

In mechanics, degrees of freedom are intuitive: they follow from the requirements of the system. For cable cars, when one car goes down it pulls the other up. This constraint reduces how each car can move.

In statistics, our methods introduce constraints. We use the sample to test a claim and also to estimate quantities (like means or variances). Each estimate uses up a degree of freedom and limits how the data can vary independently.

In hypothesis testing, we compute a p-value using a reference distribution such as the t distribution, chi-square distribution, or F distribution.

These distributions depend on the degrees of freedom. For example, this is the t-distribution for different degrees of freedom:

If you use the wrong degrees of freedom, you get the wrong curve, the wrong p-value, and possibly the wrong conclusion.