Hypothesis test

What is a hypothesis test?

Hypothesis tests are statistical test procedures, such as the t-test or an analysis of variance, with which you can test hypotheses based on collected data.

A hypothesis test is used whenever you want to test a hypothesis about the population with the help of a sample. It helps you assess whether the sample provides evidence for a statement about the population, while allowing for sampling uncertainty.

A possible example would be that the company "My-Muesli" would like to know whether their muesli bars really weigh 250 g. For this purpose, a random sample is taken and a hypothesis test is then used to draw conclusions about all the muesli bars produced.

In statistics, hypothesis tests aim to test hypotheses about the population on the basis of sample characteristics.

Hypothesis Testing and the Null Hypothesis

As we know from the previous tutorial on hypotheses, there is always a null and an alternative hypothesis. In "classical" inferential statistics, the null hypothesis is tested using a hypothesis test. The null hypothesis (H0) usually specifies no difference, no relationship, or a particular value for a population parameter.

If you want to be 100% accurate, the null hypothesis H0 can only ever be rejected or not rejected using a hypothesis test. The non-rejection of H0 is not a sufficient reason to conclude that H0 is true. Therefore, the wording "H0 was not rejected" is preferable to "H0 was retained."

Briefly anticipating the p-value: the p-value is compared with a significance level chosen before analyzing the data. If the chosen significance level is 0.05, H0 is rejected when the p-value is less than or equal to 0.05; otherwise, H0 is not rejected.

Probability of Error in hypothesis tests

A decision to reject or not reject H0 can be wrong. But why can such an error occur?

Here is the short answer: Each time you take a sample, you naturally get a different one, which means that the results will vary. In the worst case, the sample may deviate strongly from the population pattern, leading to an incorrect decision. Hypothesis testing controls some of these error risks, but it cannot eliminate them.

Level of Significance

A hypothesis test can never reject the null hypothesis with absolute certainty. There is always a certain probability of error that the null hypothesis is rejected even though it is actually true. This probability of error is called the significance level, denoted by α.

Usually, a significance level of 5% or 1% is set. If a significance level of 5% is set, the testing procedure has a 5% long-run probability of rejecting H0 when H0 is true, provided the test assumptions are satisfied. It does not mean that there is a 5% probability that H0 is true or that the current decision is wrong.

Illustrated by the two-sample t-test, this means that the observed means of two samples have a certain distance to each other. The greater the observed difference between the means relative to its standard error, the less compatible the data are with a null hypothesis of equal population means. The significance level defines how much evidence against H0 is required before H0 is rejected.

The p-value is the probability, assuming H0 and the test assumptions are true, of obtaining a test statistic at least as extreme as the observed one. Accordingly, if the p-value is less than or equal to the significance level, the null hypothesis is rejected; otherwise, it is not rejected.

If, for example, we get a p-value of 0.04, then assuming the null hypothesis is true, the probability of obtaining a test statistic at least as extreme as the one observed is 4%. The p-value is thus less than the significance level of 5% and thus the null hypothesis is rejected.

It is important to note that the significance level is always set before the test and may not be changed afterwards in order to obtain the "desired" statement after all. To ensure a certain degree of comparability, the significance level is usually 5% or 1%. If many hypotheses are tested, the probability of at least one false-positive result increases, so the analysis plan may also need a multiple-testing adjustment.

• p ≤ 0.01: statistically significant at the 1% level
• 0.01 < p ≤ 0.05: statistically significant at the 5% level
• p > 0.05: not statistically significant at the 5% level

Types of Errors

Because decisions are based on sample data, two types of errors can occur. Due to sampling variation, it can happen that the null hypothesis is rejected by chance, although in reality there is no difference, that is, the null hypothesis is true. Conversely, the result of the hypothesis test can also be that the null hypothesis is not rejected, although in reality there is a difference and thus the alternative hypothesis is true.

Accordingly, there are two types of errors in hypothesis testing:

• Type I error: The null hypothesis is rejected although it is true.
• Type II error: The null hypothesis is not rejected although the alternative hypothesis is true.

The probability of a Type I error is controlled by the significance level α. The probability of a Type II error is denoted by β, and the statistical power, 1 − β, is the probability of rejecting H0 when a specified alternative is true. Power depends on factors such as sample size, effect size, variability, and the significance level.

Overall, the following cases arise:

Significance vs. Effect Size

When the p-value is less than or equal to the chosen significance level, we reject H0 and regard the data as evidence for the alternative hypothesis. This does not prove the alternative hypothesis.

However, it is important to keep in mind that just because an effect is statistically significant does not mean that the effect is relevant.

With a very large sample or low variability, even a very small difference between two groups may be statistically significant, although it may not be relevant in practice.

Choosing correct hypothesis test

In order to test hypotheses, various test procedures are available. On the one hand, their suitability depends on the levels of measurement of the variables

• Nominal scale
• Ordinal scale
• Metric scale

and, on the other hand, on the number of samples and whether observations are independent or paired. The study design, research question, and assumptions of the test must also be considered.

numiqo helps you find a suitable test: select the data you want to evaluate, and numiqo will suggest a test based on the selected variables and their scale levels. You should still check whether the study design and the assumptions of the suggested test are appropriate.

Depending on the selected variables, the following tests are calculated:

• t-test one sample
• t-test independent samples
• t-test dependent samples
• Chi-Square test
• Binomial test
• ANOVA with/without repeated measures
• Two-way ANOVA with/without repeated measures
• Wilcoxon test
• Mann-Whitney U test
• Friedman test
• Kruskal-Wallis test
• ...

The following table lists the relevant test procedures. If you know the scale level of the variables in your hypothesis, you can see in the table which test could fit!

If a correlation hypothesis is to be tested, a correlation analysis is calculated. Either the Pearson correlation or the Spearman correlation is then used here.

Examples of hypothesis testing

Is there a difference in the average number of burglaries (dependent variable) in houses with and without alarm systems (independent variable with 2 groups)?

Does average blood pressure differ before and after cigarette consumption in the same participants?

People living in small, medium or large cities (independent variable with three groups) differ in their health awareness (dependent variable).