Effect size for independent t-test

An effect size indicates the magnitude of an observed effect. Depending on the research question, an effect may be a difference between groups or a relationship between variables.

• How strong is the relationship/difference?
• How strong is the correlation?

The t-test for independent samples checks whether there is a difference between two independent groups. The effect size in the independent t-test tells you how strong the difference between the groups is. In the independent t-test, this is done by comparing differences in means.

This allows different studies to be compared using the effect size.

Difference between effect size and p-value

The p-value says nothing about the size of the effect or difference and depends very much on the sample size.

What does this mean? If there is a difference in the population, the larger the sample, the clearer it will appear in the p-value. If the sample is very large, even very small differences in the population can be detected. These small differences may no longer be relevant at all.

In the left study in the upper graph, we will get, with the same difference, a larger p-value than in the right study, assuming the variability is also the same. Simplified, this is because more people were included in the right study, so the estimate is more precise and the test has more power to detect the difference.

To describe the magnitude of the difference, the effect size is used in addition to the p-value.

Effect Size of the t-test for independent samples

As already said, the p-value does not describe the size of the difference. It indicates how compatible the data are with the null hypothesis under the assumptions of the test. The independent t-test compares two means. One question could be whether there is a mean difference in the salary of men and women. To make such differences more comparable across studies, a standardized effect size can be used.

Calculate effect size in t-test for independent samples

The effect size for a t-test for independent samples is usually calculated using Cohen's d. At the population level, the mean difference is standardized by dividing it by the population standard deviation, as shown below. In a sample, this standard deviation must be estimated. The usual pooled version of Cohen's d uses the pooled within-group sample standard deviation.

In practice, the population standard deviation is unknown and is estimated from the sample. Cohen's d therefore uses sample statistics. Hedges' g applies an additional correction to Cohen's d to reduce its small-sample bias. Hedges' g is usually close to Cohen's d in large samples.

The formula on the right shows how the pooled Cohen's d can be calculated from the test statistic of the conventional pooled-variance independent t-test. It should not be applied directly to Welch's t-test. When group variances differ, researchers should report which standardizer and effect-size definition they use.

In the literature, a variety of different names and symbols are used for the different effect sizes. A detailed discussion on the topic of different effect size measures for the difference of means from two independent groups can be found here: Enzmann, D. (2015). Notes on Effect Size Measures for the Difference of Means From Two Independent Groups: The Case of Cohen's d and Hedges' g (Technical Report).

To combine comparable mean differences across independent studies, use the meta-analysis calculator. Enter each group's sample size, mean, and standard deviation, then choose a mean difference or Hedges' g. The results include the pooled effect and a forest plot.

Interpret Effect Size

The absolute value of Cohen's d or Hedges' g is sometimes interpreted using Cohen's guidelines below.

For example, an effect of 0.5 is a medium effect and means that the difference between the two groups is equal to half a standard deviation. If the effect size is in the middle of two values, the result is a small to medium effect or a medium to large effect. These values are only general guidelines; what counts as important depends on the subject area and context. The sign indicates the direction of the difference and depends on which group mean is subtracted from the other. A confidence interval should also be reported because the estimated effect size is uncertain, especially in small samples.

Calculate Effect Size with numiqo

With numiqo in the independent t-test calculator, the effect size can easily be calculated online. Simply select a metric and a categorical variable and click on effect size.