p-value

What is the p-value?

The p-value indicates the probability that the observed result or an even more extreme result will occur if the null hypothesis is true.

The p-value is used to decide whether the null hypothesis is rejected or not rejected. If the p-value is smaller than the pre-specified significance level (often 5%), the null hypothesis is rejected; otherwise, it is not rejected.

You want to test a claim about the population and have set up a hypothesis for this. Since it is usually not possible to survey the entire population, you survey a sample. This sample will most likely deviate from the population due to chance.

If the null hypothesis applies in your population, for example that the salary of men and women does not differ, then there will typically be some difference in the sample, such as a difference of EUR 300 or more per month. The p-value tells you how likely it is that a difference of EUR 300 or more will occur by chance in the sample if there is no difference in the population.

If the result is a very small probability, you can ask yourself whether the assumption about the population is true at all.

If p = 0.03, then assuming no true difference, a difference of EUR 300 or larger would occur in about 3% of random samples.

Example

The p-value is used to either reject or retain (not reject) the null hypothesis in a hypothesis test. If the calculated p-value is smaller than the significance level, which in most cases is 5%, then the null hypothesis is rejected; otherwise it is not rejected.

Example:

• The null hypothesis is that there is no difference between the salaries of men and women.
• Now a sample is taken with the salary of men and women. These are our observed results.
• We assume that the null hypothesis is true, that is, that there is no difference between the salaries of men and women.
• In the observed sample we find that men earn 150 EUR more per month than women.
• The p-value now indicates how likely it is to draw a sample in which the salary of men and women differ by 150 EUR or more, even though there is no difference in the population.
• If the p-value is, for example, 0.04, it is only 4% likely to draw a sample of 150 EUR or more extreme if there is no difference in salary in the population.

A p-value of 0.04 or 4% means that if there is no difference in salary in the population, it is only 4% likely to draw a sample that is 150 EUR or more extreme.

The probability of 4% is very low, so one can ask whether it is true that men and women in the population earn the same or whether this hypothesis should be rejected.

The question of when the null hypothesis is discarded is answered by the significance level.

Significance Level

The significance level is determined before the test. If the calculated p-value is below this value, the null hypothesis is rejected; otherwise, it is not rejected. As a rule, a significance level of 5% is chosen.

• α ≤ 0.01: highly significant result
• α ≤ 0.05: significant result
• α > 0.05: not significant result

The significance level α is the long-run Type I error rate. If there is a significance level of 5% and the null hypothesis is rejected, the probability that the null hypothesis is actually valid is 5%. So, there is a 5% probability of making a mistake. If the critical value is reduced to 1%, the probability of error is accordingly only 1%, but it is also more difficult to confirm the alternative hypothesis.

One-tailed p-value

Let's say you are examining the reaction time of two groups. Often it is not only whether there is a difference between the two groups that is of interest, but whether one group has a larger or smaller value than the other. In this case, you have a directional hypothesis and calculate a one-sided p-value.

The one-sided p-value is obtained by dividing the two-sided p-value by 2. Here, of course, care must be taken whether the difference or effect under consideration is in the direction of the alternative hypothesis.

Example

Your alternative hypothesis is that group A has greater reaction time values than group B. When analyzing your data you get a two-sided p-value of 0.04.

Now you have to check whether group A really has larger values in your data. If this is the case, the two-sided p-value is divided by two, so you get 0.02.

If this is not the case and the effect or difference goes in the opposite direction from the alternative hypothesis, your p-value is 1 - 0.02, i.e. 0.98.

Don't worry, if you use numiqo, you can specify what kind of hypothesis you have, and numiqo will help you evaluate it.

Calculate p-value

To calculate the p-value a suitable hypothesis test must first be found. After selecting the appropriate hypothesis test, you can calculate the p-value in the statistic calculator numiqo. The best known hypothesis tests are:

• t-test
• Correlation analysis
• Chi-square test
• Regression analysis

For the calculation of the p-value a distribution function is needed which describes the realizations of the sample. If this distribution function is known, it can be determined how probable it is that a drawn sample is less than or equal to a considered value. Classical representatives of these distributions are the t-distribution and the Chi-square distribution.

t-distribution

Chi-square distribution

Statistical tests and the p-value

In order to reject or maintain a hypothesis one needs the p-value. The procedure for using the p-value in statistical tests is:

• 1. Definition of the critical p-value or the significance level e.g. 5%
• 2. Definition of a statistical test procedure e.g. t-tests or correlation analysis
• 3. Calculation of the test statistic from the sample e.g. the t-value in the t-test
• 4. Determination of the p-value for the test statistic e.g. p-value for given t in t-test
• 5. Check whether the p-value is above or below the specified critical p-value e.g. p-value 1% lies below the critical value of 5%