Linear Regression

What is a Linear Regression Analysis?

Linear regression analysis models the relationship between a dependent variable and one or more independent variables.

In short: it explains or predicts one variable by assuming it changes linearly with another.

The model estimates how much the dependent variable changes for a one-unit change in each independent variable and assesses how well the model fits the data.

Types of Regression

• If there is one independent variable, it is called simple linear regression.
• If there are multiple independent variables, it is called multiple linear regression.

Note: In linear regression, the level of measurement of the dependent variable must be metric. If it is categorical, a suitable categorical regression model, such as binary, multinomial, or ordinal logistic regression, is generally used.

Simple Linear Regression

The goal of simple linear regression is to predict the value of a dependent variable based on a single independent variable.

• A stronger linear relationship usually produces more accurate predictions within the range and population represented by the data.
• This strength shows how much of the dependent variable can be explained by the independent variable.

The relationship between the variables can be illustrated in a scatter plot.

• A strong linear relationship is indicated when data points closely align along a straight line.
• A weak relationship is indicated when data points are more widely scattered.
• To determine that line, linear regression uses the method of least squares.

Calculation of a Simple Linear Regression

The regression line can be described by the following equation:

Definition of "Regression coefficients":

• a : point of intersection with the y-axis (y-intercept)
• b : gradient of the straight line (slope)
• y-hat: estimated y value (dependent variable)

In short: By using the equation above, for each x value, the corresponding y value is estimated.

Example Simple Linear Regression

In our example the height of people is used to estimate their weight.

• Independent variable: height
• Dependent variable: weight

• "Perfect" estimation: All points (measured values) lie exactly on a single straight line.
• In practice: This is almost never the case.
• Therefore: A straight line must be found that lies as close as possible to the data points.
• Goal: Keep estimation error (epsilon) as small as possible.

Estimation error

• The goal is to keep the estimation error as small as possible.
• This means that the distance between the fitted value and the observed value should be as small as possible.
• This distance (error) is called the residual.
• A residual is usually written as e. The Greek letter ε represents the unobserved error term in the population model.

When calculating the regression line, the regression coefficients (a and b) are determined so that the sum of the squared residuals is minimal. (Method of Ordinary Least Squares)

Regression coefficient b

The sign of the regression coefficient b indicates the direction of the relationship:

• b > 0: the fitted linear relationship is positive (larger x values are associated with larger fitted y values)
• b < 0: the fitted linear relationship is negative (larger x values are associated with smaller fitted y values)
• b = 0: there is no fitted linear relationship between x and y

Standardized regression coefficients are labeled beta. These values are comparable across predictors because the units are removed.

Multiple Linear Regression

Unlike simple linear regression, multiple linear regression allows more than one independent variable to be considered.

The goal is to estimate a variable based on several other variables. Like in simple linear regression, the variable to be estimated is called the dependent variable (criterion). The variables that are used for the prediction are called independent variables (predictors).

Multiple linear regression is used in many different fields such as market research, health & medicine, economics & finance, or data science & machine learning.

In all these areas, researchers may examine how different factors are associated with a variable. Regression alone does not establish that the factors cause the outcome.

Calculation Multiple Linear Regression

The equation for multiple regression with k independent variables is:

where:

• y is the dependent (response) variable
• x₁, x₂, …, x_k are the independent (explanatory) variables
• b₁, b₂, …, b_k are the regression coefficients
• a is the intersection with the y-axis (y-intercept)

The coefficients can now be interpreted similarly to the linear regression equation.

If all independent variables are 0, the resulting value is a.

If the independent variable x_i increases by one unit, the fitted value of y changes by b_i units, while the other predictors remain constant.

Coefficient of Determination R²

To evaluate how well the regression model can predict or explain the dependent variable, two main measures are used: the coefficient of determination R² and the standard estimation error.

The coefficient of determination R², also known as explained variance, indicates the proportion of variation in the observed dependent variable accounted for by the fitted model. A higher R² indicates a closer fit to the data used, but does not by itself show that the model is appropriate or predicts new data well.

To calculate R², the variance of the estimated values is related to the variance of the observed values:

Adjusted R²

R² cannot decrease when more independent variables are added, even if they provide little useful information. Adjusted R² includes a penalty for adding predictors and is therefore more useful when comparing models with different numbers of predictors.

Standard estimation error

The standard error of the regression is an estimate of the standard deviation of the error term. It gives an impression of the typical size of the residuals. Graphically interpreted, the standard estimation error is the dispersion of the observed values around the regression line.

The coefficient of determination and the standard estimation error are used for simple and multiple linear regression.

Standardized and unstandardized regression coefficient

Regression coefficients can be reported in unstandardized or standardized form. The unstandardized coefficients are the ones used in the regression equation and are abbreviated b.

A standardized coefficient is obtained by multiplying the unstandardized coefficient b_i by the standard deviation of its predictor S_xi and dividing by the standard deviation of the dependent variable S_y. This describes the expected change in y, in standard deviations, for a one-standard-deviation increase in x_i.

Assumptions of Linear Regression

In order to interpret the results of the regression analysis meaningfully, certain conditions must be met.

• Linearity: The mean of the dependent variable must be a linear function of the predictors as they appear in the model.
• Homoscedasticity: The residuals must have a constant variance.
• Normality: For exact small-sample confidence intervals and significance tests, the errors are assumed to be normally distributed.
• No severe multicollinearity: Predictors should not be exact or extremely close linear combinations of one another.
• Independence: Errors should be independent.

Linearity

In simple linear regression, a straight line is fitted to the data points. This line should represent the relationship as well as possible. If the relationship is non-linear, a straight line may represent it poorly.

In the left graph there is a linear relationship between the dependent and independent variable, so the regression line is meaningful.

In the right graph the relationship is clearly non-linear, so fitting a straight line is not meaningful. In that case, the coefficients cannot be interpreted reliably and prediction errors can be larger than expected.

Therefore, it is important to check whether the model represents the relationship adequately. For simple regression, this can be inspected in a scatter plot. For multiple regression, residual plots and partial-residual plots are more informative than separate plots of the outcome against each predictor.

Homoscedasticity

Since in practice the regression model never exactly predicts the dependent variable, there is always an error. This very error must have a constant variance over the predicted range.

To check homoscedasticity, fitted values are usually plotted on the x-axis and residuals on the y-axis. The residuals should have a roughly even vertical spread across the range of fitted values.

If this is not the case, heteroscedasticity is present. In the case of heteroscedasticity, the error variance changes across the range of fitted values.

Normal Distribution of the Error

For exact small-sample confidence intervals and significance tests, the error term (ε) is assumed to be normally distributed. Normality is not required merely to calculate the least-squares line.

There are two ways to check this:

• Analytical methods
• Graphical methods

Analytical methods

• Common tests:
  • Shapiro–Wilk test
  • Kolmogorov–Smirnov test when the reference distribution is fully specified; if its parameters are estimated from the data, an appropriate correction is required
• Interpretation:
  • If the p-value is greater than 0.05, the test has not found sufficient evidence of a deviation from normality.
  • This does not prove that the errors are normally distributed.
• However, analytical tests are used less frequently because:
  • They often indicate normal distribution for small samples.
  • For large samples, they become significant very quickly.
  • This can lead to rejecting the null hypothesis of normal distribution even when deviations are minor.

Graphical methods

Normality tests have little power to detect deviations in small samples and can detect minor, unimportant deviations in large samples. Therefore, they should be considered together with graphical checks.

• Common graphical tools:
  • Histogram
  • QQ-plot (quantile–quantile plot)
• Interpretation of the QQ-plot:
  • The closer the data points lie to the straight line, the closer they are to a normal distribution.

Multicollinearity

Multicollinearity means that two or more independent variables are strongly correlated with one another. The problem with multicollinearity is that the effects of each independent variable cannot be clearly separated from one another.

If, for example, there is a high correlation between x₁ and x₂, then it is difficult to determine b₁ and b₂. If both are e.g. completely equal, the regression model does not know how large b₁ and b₂ should be, becoming unstable.

Multicollinearity is especially problematic when interpreting individual coefficients because it increases their uncertainty. It can also make predictions unstable, particularly when predicting new observations whose predictor patterns differ from those in the training data.

Significance test and Regression

The regression analysis is often carried out in order to make statements about the population based on a sample. Therefore, the regression coefficients are calculated using the data from the sample. Significance tests quantify how compatible the estimates are with specified null hypotheses under the model assumptions. Tests are commonly performed at two levels:

• Significance test for the whole regression model
• Significance test for the regression coefficients

It should be noted, however, that the assumptions in the previous section must be met.

Here it is checked whether the coefficient of determination R² in the population differs from zero. The null hypothesis is therefore that the coefficient of determination R² in the population is zero. To test the null hypothesis, the following F-test is calculated:

The calculated F-value must now be compared with the critical F-value. If the calculated F-value is greater than the critical F-value, the null hypothesis is rejected and the R² deviates from zero in the population. The critical F-value can be read from the F-distribution table. The numerator degrees of freedom are k and the denominator degrees of freedom are n-k-1.

Significance test for the regression coefficients

The next step is to check which variables contribute significantly to the prediction of the dependent variable. This is done by testing whether the slopes (regression coefficients) differ from zero in the population. The following test statistic is calculated:

where b_j is the j^th regression coefficient and s_{b_j} is the standard error of b_j. This test statistic is t-distributed with the degrees of freedom n-k-1. The critical t-value can be read from the t-distribution table.

Confidence intervals should be reported with the coefficients because they show the range of values compatible with the data and model. Predictions should generally be limited to predictor values and populations represented in the data; extrapolation beyond the observed range can be unreliable.

Calculate with numiqo

As an example of linear regression, a model is set up that predicts the body weight of a person. The dependent variable is thus the body weight, while the height, age and gender are chosen as independent variables. The following example data set is available:

After you have copied your data into the statistics calculator, you must select the variables that are relevant for you. Then you receive the results in table form.

Interpretation of the Results

This table shows that the model accounts for 75.4% of the variation in weight in this sample. The standard error of 6.587 kg describes the typical scale of the residuals; it is not the average prediction error for new people.

Regression equation:

Weight = 47.379 · Height + 0.297 · Age + 8.922 · is_male − 24.41

Interpretation of the coefficients

• If age increases by one year, predicted weight increases by 0.297 kg according to the model, while height and sex remain constant.
• For the dichotomous variable sex:
  • The coefficient represents a difference.
  • According to the model, a man's predicted weight is 8.922 kg more than a woman's at the same height and age.
• If all independent variables are zero, the fitted equation gives a weight of −24.41 kg. Because zero height and age are outside the meaningful data range, this intercept has no useful real-world interpretation.

Standardized coefficients (beta)

• Standardized beta coefficients are expressed in standard-deviation units. In multiple regression they are not restricted to the range −1 to +1.
• The absolute beta values can help compare associations among predictors measured on different scales, but they do not by themselves establish importance or causality.
• In this regression analysis, age has the largest absolute standardized coefficient among the included predictors.

Significance of coefficients

• The calculated coefficients refer to the sample used in the regression analysis.
• It must be tested whether the B-values differ from zero only by chance or also in the population.
• The null hypothesis states that the calculated B-value equals zero in the population.
• If the null hypothesis is not rejected, the data do not provide sufficient evidence that the coefficient differs from zero after accounting for the other predictors.

P-values

• The p-value measures how compatible the estimated coefficient is with the null hypothesis, under the model assumptions.
• If a significance level of 0.05 was chosen in advance, p-values below 0.05 are called statistically significant.
• In this example, only age can be considered a significant predictor of a person’s weight.

Lasso, Ridge and PLS Regression

Lasso adds an L₁ penalty to shrink some coefficients to zero. Use it for automatic feature selection and to prevent overfitting. You can use the Lasso Regression Calculator

Ridge applies an L₂ penalty to uniformly shrink coefficients. It keeps all features but reduces their impact to improve stability. You can use the Ridge Regression Calculator

PLS compresses correlated predictors into a few components that capture their covariance with the dependent variable. Use it when predictors are highly correlated or outnumber the cases. You can use the PLS Regression Calculator