Logistic Regression

What is a Logistic Regression Analysis?

Logistic regression is a type of regression analysis used for a categorical dependent variable. Binary logistic regression is used when the outcome has two categories, for example: buys a product and does not buy a product.

Logistic regression analysis is the counterpart of linear regression, in which the dependent variable of the regression model must at least be interval-scaled.

Logistic regression estimates the probability of an outcome category from one or more predictors.

If you need to calculate a logistic regression, you can easily use the Regression Analysis calculator here on numiqo.com.

Goals of a Logistic Regression Analysis

In the basic form of logistic regression, dichotomous variables (0 or 1) are predicted. The model estimates the probability of value 1 (characteristic present).

In medicine, for example, a frequent application is to investigate which variables are associated with a disease. In this case, 0 could stand for not diseased and 1 for diseased. The associations of age, gender, and smoking status with the probability of this disease could then be examined.

Logistic Regression and Probabilities

In linear regression, the independent variables (e.g., age and gender) are used to estimate the specific value of the dependent variable (e.g., body weight).

In logistic regression, on the other hand, the dependent variable is dichotomous (0 or 1) and the probability that outcome 1 occurs is estimated. Returning to the example above, this means: How likely is disease given a person's age, sex, and smoking status?

Calculate Logistic Regression

To build a logistic regression model, the linear regression equation is used as the starting point.

If you simply apply linear regression to a logistic problem, the result looks like this:

As can be seen in the graph, predictions can take values between plus and minus infinity. Logistic regression, however, estimates probabilities rather than raw values. Therefore, the equation must be transformed.

To do this, it is necessary to restrict predicted values to the range between 0 and 1. To ensure that only values between 0 and 1 are possible, the logistic function f is used.

Logistic function

The logistic model is based on the logistic function. The special thing about the logistic function is that for values between minus and plus infinity, it always returns values between 0 and 1.

The logistic function is well suited to describe the probability P(y=1). Applying the logistic function to the linear predictor yields:

This ensures that, no matter where the x values lie, the predictions stay between 0 and 1. The new graph now looks like this:

The probability that the dichotomous dependent variable y equals 1 for given predictor values is given by:

To calculate disease probability in the example above, the model parameters b₁, b₂, b₃ and a must first be determined. Once these have been determined, the equation for the example above is:

Maximum Likelihood Method

To determine the model parameters for the logistic regression equation, the Maximum Likelihood Method is applied. The maximum likelihood method is one of several methods used in statistics to estimate the parameters of a mathematical model. Another well-known estimator is the least squares method, which is used in linear regression.

The Likelihood Function

To understand the maximum likelihood method, we introduce the likelihood function L. L is a function of the unknown parameters in the model; in logistic regression these are b₁,... b_k, a. Therefore we can also write L(b₁,... b_k, a) or L(θ) if the parameters are summarized in θ.

L(θ) measures how compatible different parameter values are with the observed data. The maximum likelihood estimate is the parameter value that gives the observed data the greatest likelihood under the model.

Maximum Likelihood Estimator

The Maximum Likelihood Estimator can be applied to the estimation of complex nonlinear as well as linear models. In logistic regression, the goal is to estimate the parameters b₁,... b_k, a, which maximize the so-called log likelihood function LL(θ). The log likelihood function is simply the logarithm of L(θ).

This optimization is performed numerically, often with algorithms such as Newton–Raphson or iteratively reweighted least squares.

Multinomial logistic regression

Binary logistic regression is used when the dependent variable has exactly two categories. If the dependent variable has three or more mutually exclusive categories without a natural order, multinomial logistic regression can be used. For ordered categories such as low, medium, and high, ordinal logistic regression may be more appropriate.

Multinomial logistic regression estimates all outcome categories together in one joint model. First, one outcome is selected as the reference category. If the dependent variable has K categories, the model estimates K - 1 coefficient equations. Each equation compares one of the remaining categories with the reference category.

Example of multinomial logistic regression

Suppose the dependent variable describes how a person travels to work and has the categories car, public transport, and bicycle. If bicycle is selected as the reference category, the model estimates two equations:

• car compared with bicycle
• public transport compared with bicycle

The equations are estimated jointly, not as separate binary regressions. The model calculates a probability for every category, and the probabilities for one observation always sum to 1. For classification, an observation can be assigned to the category with the largest fitted probability, although a different decision rule may be appropriate when the costs of errors differ.

A coefficient describes how a predictor changes the comparison between a displayed category and the reference category, while the other predictors remain constant. Exponentiating the coefficient gives Exp(B), which is reported as a relative risk ratio. Changing the reference category changes the coefficient comparisons, but it does not change the fitted category probabilities or the overall model fit.

One model per category

Instead of one joint model, a separate binary logistic regression can be fitted for every category, each one separating that category from all the others. This approach is called one-versus-rest. With the categories car, public transport, and bicycle, three models are estimated: car versus not car, public transport versus not public transport, and bicycle versus not bicycle.

Because these models are estimated independently of one another, each returns its own score and the scores of an observation do not sum to 1. An observation is assigned to the category with the largest score. The joint multinomial model is the natural choice when the categories are exhaustive and the interest lies in comparing them with a reference category. One model per category is more robust when a category contains few cases or the joint model does not converge, and every category can be read as a model of its own.

Assumptions and Data Requirements

• Appropriate outcome: Binary logistic regression requires two outcome categories. Multinomial or ordinal logistic regression is used for other categorical outcomes.
• Independent observations: Standard logistic regression assumes that observations are independent. Repeated or clustered observations require a suitable extension.
• Linearity in the logit: Each continuous predictor should have an approximately linear relationship with the log-odds of the outcome. The probability itself does not need to be linear.
• No severe multicollinearity: Predictors should not be exact or extremely close linear combinations of one another.
• No complete separation: A predictor or combination of predictors should not perfectly separate the outcome categories, because ordinary maximum likelihood estimates can then become extremely large or fail to exist.
• Enough data: There should be enough observations and outcome events for the number of coefficients estimated. Sparse categories can produce unstable estimates and wide confidence intervals.

Interpretation of the results

In logistic regression, the relationship between the independent variables and the probability of the outcome is not linear. Therefore, the regression coefficients cannot be interpreted in the same way as in linear regression. Instead, a logistic regression coefficient describes how the log-odds change when an independent variable increases by one unit, while the other variables remain constant. Because log-odds are difficult to interpret directly, the coefficients are usually converted into odds ratios.

For a categorical predictor, categories are represented by indicator variables. Its coefficient compares one category with a chosen reference category, while the other predictors remain constant.

What are log-odds?

If an event has probability p, its odds are calculated as p / (1 - p). The log-odds are the natural logarithm of these odds: ln(p / (1 - p)).

For example, a probability of 0.75 gives odds of 0.75 / 0.25 = 3, or 3 to 1. The corresponding log-odds are ln(3), which is approximately 1.10. A probability of 0.5 has log-odds of 0. Probabilities above 0.5 have positive log-odds, while probabilities below 0.5 have negative log-odds.

Logistic regression models the log-odds because they can take any value from minus infinity to plus infinity, while the resulting predicted probabilities always remain between 0 and 1.

A predictor should not be judged only by a simple correlation or its p-value. Its effect size, confidence interval, subject-matter relevance, and contribution to model fit or predictive performance should also be considered.

Odds Ratios

An odds ratio (OR) is a statistical measure used to determine the strength of association or effect size between two events or groups, often in case-control studies. It compares the odds of an event occurring in one group to the odds of it occurring in another group.

Odds represent the ratio of the probability of an event happening to it not happening. An odds ratio is a ratio of odds, not a ratio of probabilities. In a regression model, an odds ratio above 1 indicates higher odds and an odds ratio below 1 indicates lower odds for a one-unit increase in the predictor, while the other predictors remain constant. Confidence intervals should be reported to show the uncertainty of this estimate.

Pseudo-R squared

In linear regression, the coefficient of determination (R²) indicates the proportion of variance explained by the model. Logistic regression is estimated using maximum likelihood and predicts outcome probabilities. The variance decomposition used for the classical R² therefore does not apply in the same way.

Instead, likelihood-based measures called pseudo R² are used to describe how much the fitted model improves upon a model containing only an intercept. They are useful as approximate model fit indices, but they must not be interpreted as the proportion of variance explained. Different pseudo R² definitions can produce different values and should not be compared as if they were identical measures.

Common measures include Cox and Snell R², Nagelkerke R², and McFadden's R².

Null Model

For the calculation of the Cox and Snell R-square and the Nagelkerke R-square, the likelihood from the so-called null model L₀ and the likelihood L₁ from the calculated model (full model) is needed.

The null model is a model in which no independent variables are included; L₁ is the likelihood of the model with the independent variables.

Cox and Snell R-square

In the Cox and Snell R-square, the ratio of the likelihood function of the null model L₀ and L₁ is compared. The better the full model fits compared to the null model, the lower the ratio between L₀ and L₁. The Cox and Snell R-square is obtained with:

Nagelkerkes R-square

Cox and Snell R-square has a theoretical maximum below 1. Nagelkerke R-square rescales Cox and Snell R-square so that its theoretical maximum is 1. A value closer to 1 indicates a greater improvement over the null model, but it is not a percentage of variance explained.

McFadden's R-square

McFadden's R-square compares the log-likelihood of the fitted model with the log-likelihood of the null model. Higher values indicate a greater improvement over the null model, but the values are not interpreted in the same way as the R² from linear regression.

Chi² Test and Logistic Regression

In logistic regression, the likelihood-ratio chi-square test evaluates whether the fitted model with predictors provides a better fit than a null model containing only an intercept.

Two nested models are compared: the full model containing the selected independent variables and the null model containing no independent variables.

The chi-square statistic is calculated from the difference between the log-likelihoods of these two models. A larger value indicates that the model with predictors improves the fit more strongly relative to the null model.

The null hypothesis states that all predictor coefficients, except the intercept, are zero. In other words, the predictors do not improve model fit. If a significance level of 0.05 is used and the p-value is below 0.05, the null hypothesis is rejected and the model is considered significant overall. This test does not show which individual predictors are significant.

Statistical significance and pseudo R² do not by themselves show how well a model predicts. Predictive use should also assess discrimination, such as with an ROC curve, and calibration, which compares predicted probabilities with observed outcomes, ideally on data not used to fit the model. If probabilities are converted into classifications, results such as sensitivity and specificity depend on the chosen threshold; 0.5 is not automatically the best threshold.

Logistic regression describes associations unless the study design and assumptions justify a causal interpretation. A statistically significant coefficient alone does not show that changing a predictor will cause the outcome probability to change.

Example logistic regression

As an example, purchasing behaviour in an online shop is examined. The data record whether a person buys immediately, at a later time, or not at all after visiting the website. The dependent variable therefore has the following three categories:

• Buy now
• Buy later
• Don't buy

The binary result shown below compares Buy now with not Buy now, combining the other two categories. A multinomial model would instead retain and compare all three categories. Gender, age, and time spent in the online shop are available as predictors.

Logistic regression results

Logistic regressions, similar to linear regression models, can be easily and quickly calculated with numiqo.

To recalculate the example above:

• Copy and paste the table on purchasing behavior in the online store into numiqo’s statistics calculator.
• Select the Regression tab.
• Click on the desired variables.

The results are displayed directly below in table form.