ROC Curve

What is a ROC curve?

A ROC curve is a graphical representation of the performance of a binary classification model across all classification thresholds. Here, ROC stands for Receiver Operating Characteristic.

We want to classify, based on a screening, whether a person has cancer or not.

This classification uses a blood value where higher values indicate cancer. The question is which value we choose as the classification threshold. From which value do we predict disease?

For this, we obtain data from 10 people on their blood value and whether the disease is present.

We could choose a classification threshold of 45, for example. In this case, of the five people with the disease, we would correctly classify four as "diseased" and one incorrectly as "healthy." Thus, we correctly classified 4 out of 5 as "diseased."

This value is called the True Positive Rate (TPR) and is equal to sensitivity.

On the other hand, out of the five healthy individuals, we misclassify two as "diseased" and correctly classify three as "healthy." So, we misclassified 2 out of 5 as "diseased." This value is called the False Positive Rate (FPR).

So for a threshold of 45 we get a True Positive Rate of 4/5, i.e. 0.8, and a False Positive Rate of 2/5, i.e. 0.4.

True Positive Rate and False Positive Rate

The True Positive Rate (TPR) is calculated with this equation:

The True Positive Rate is equal to the true positives divided by the true positives plus the false negatives. The true positives are those correctly classified as "diseased" and the false negatives are those incorrectly classified as "healthy."

The False Positive Rate (FPR) is obtained using this equation:

The False Positive Rate is equal to the false positives divided by the false positives plus the true negatives. The false positives are the healthy individuals misclassified as "diseased" and the true negatives are the individuals correctly classified as "healthy."

How to plot the ROC Curve

We can now calculate the True Positive Rate and False Positive Rate for each threshold. These two values are plotted on the ROC curve. The True Positive Rate is plotted on the y-axis and the False Positive Rate on the x-axis.

Now let's plot the complete ROC curve for our example!

If we choose a very small threshold value, i.e., pushing it all the way to the left, we correctly classify all five diseased individuals. Our True Positive Rate is thus 5 out of 5, i.e. 1.

In the same way, we also misclassify all five healthy people as "diseased." Our False Positive Rate is therefore 5 out of 5, i.e. 1.

This gives us the first point:

Now we can increase the threshold. Here we still classify all five diseased people correctly as "diseased." So we still have a True Positive Rate of 5/5. However, of the five healthy individuals, we now only misclassify 4 out of 5 as "diseased." So we have 4 out of 5, or 0.8.

At the next threshold, we still have a True Positive Rate of 1. All five diseased are correctly classified. The False Positive Rate takes the value of 3/5, so 0.6.

At the next threshold, for the first time, a diseased person is misclassified as "healthy". We therefore obtain a True Positive Rate of 4/5, i.e. 0.8, and a False Positive Rate of 3/5, i.e. 0.6.

We can do this for all other thresholds, finishing the ROC curve. At the marked point below, for example, 80% of the diseased people were correctly classified as "diseased" and 20% of the healthy people were incorrectly classified as "diseased."

AUC value

Using the ROC curve, we can also compare different classification methods. A classifier has better discrimination when its curve is higher and closer to the top-left corner across the relevant thresholds. This area under the ROC curve is summarized by the AUC value. If two ROC curves cross, which classifier is preferable may depend on the range of sensitivity or specificity that matters in practice.

The AUC varies between 0 and 1. An AUC of 0.5 indicates no discrimination, whereas an AUC of 1 indicates perfect discrimination. It can also be interpreted as the probability that a randomly selected positive case receives a higher score than a randomly selected negative case. An AUC below 0.5 may indicate that the score direction or outcome coding has been reversed.

AUC measures discrimination, not calibration: a model can rank cases correctly while producing inaccurate probability estimates. ROC curves also do not directly show the positive predictive value, which can be low when the condition is rare. These measures should therefore be considered alongside calibration and prevalence-dependent measures when predicted probabilities or practical decisions matter.

Confidence interval and p-value for the AUC

The AUC is estimated from a sample and therefore comes with uncertainty. In publications the AUC is almost always reported together with a confidence interval, e.g. "AUC = 0.83 (95% CI 0.75 to 0.91)". numiqo computes the standard error of the AUC using the DeLong method and, from it, the 95% confidence interval. The confidence interval shows the range of AUC values reasonably compatible with the data under the method's assumptions: a narrow interval means a precise estimate, whereas a wide interval indicates substantial uncertainty about the test's discrimination.

In addition, numiqo reports a p-value. It tests the null hypothesis that the AUC equals 0.5, which represents no discrimination. This p-value should not be interpreted on its own, however. With a large sample, even a small difference from 0.5 can be statistically significant. The confidence interval is usually more informative because it shows the estimated level of discrimination and its precision, not just whether the data provide evidence that the AUC differs from 0.5.

Youden index and optimal cutoff

The AUC value tells us how well a test discriminates overall. It does not yet tell us which threshold to use in practice. That is exactly what the Youden index is for, usually abbreviated as J.

The Youden index is computed for every possible cutoff and is defined as:

J = sensitivity + specificity - 1

At an individual cutoff, the Youden index can range from -1 to 1. If J equals 0, the point lies on the diagonal, so sensitivity and specificity together are no better than at a corresponding chance point. Negative values lie below the diagonal, and J equals 1 only for perfect classification. Intuitively, the Youden index is the vertical distance between the ROC curve and the diagonal.

The optimal cutoff is then the threshold at which the Youden index is largest. This is the point where the ROC curve is farthest above the diagonal and where the unweighted sum of sensitivity and specificity is largest. It is optimal according to this criterion, but not necessarily for a particular clinical or practical decision.

Important to note: the Youden index weights missed cases and false alarms equally and does not account for how common the condition actually is. In our cancer-screening example it would probably be far worse to miss a diseased person than to initially misclassify a healthy person as "sick". In such a case it can make sense to deliberately choose a different cutoff with higher sensitivity, even if the Youden index is not at its maximum there.

numiqo computes the Youden index automatically and reports the optimal cutoff together with the corresponding sensitivity and specificity in the "Optimal Cutoff (Youden Index)" table. The calculator also supports screening, confirmatory and prespecified cutoffs. The ROC cutoff-selection guide explains when each approach is appropriate.

Positive and negative likelihood ratio

At the chosen cutoff, sensitivity and specificity are not the only useful measures. The likelihood ratios are often reported in addition, because they answer the question that really matters in practice: how strongly does a test result change the assessment of whether a person has the condition?

The positive likelihood ratio is defined as:

LR+ = sensitivity / (1 - specificity)

To update the odds of disease, the pre-test odds are multiplied by LR+. An LR+ of 5, for example, means that a positive result is five times as likely in a person who actually has the disease as in a healthy one.

The negative likelihood ratio is defined as:

LR− = (1 - sensitivity) / specificity

To update the odds after a negative result, the pre-test odds are multiplied by LR−. Small values provide stronger evidence against the disease, although the resulting probability also depends on the pre-test probability.

Prevalence does not appear directly in the likelihood-ratio formulas. However, the post-test probability still depends on the pre-test probability, and likelihood ratios may vary between populations. As a rough rule of thumb, an LR+ above 10 or an LR− below 0.1 counts as strong evidence. numiqo reports both values at the optimal cutoff in the "Optimal Cutoff (Youden Index)" table.

When low values indicate the condition

In our example, high blood values indicate disease. For many measures it is the other way round: low values indicate the condition, for instance haemoglobin or the heart's pumping function. For the ROC curve it is crucial which way the relationship goes, because that determines from which threshold on a case counts as positive.

This direction should be set from domain knowledge and not read off the data: if you always automatically chose the direction that looks better, the AUC would come out systematically too high. In numiqo you therefore state yourself whether high or low values indicate the positive state. An unexpected AUC below 0.5 can be a reason to check the coding and direction, but the direction should not be reversed solely because the observed sample looks better that way.

The purpose determines the cutoff

A balanced cutoff is not automatically the best threshold for a practical decision. For screening or triage, missing a positive case may be especially costly, so a minimum sensitivity can be specified. For confirmatory or rule-in testing, false positive results may be especially costly, so a minimum specificity can be required. A cutoff that was defined previously should be evaluated without optimizing it again on the validation data.

Moving a cutoff in the direction that classifies more observations as positive generally increases sensitivity and decreases specificity. The numerical direction of that movement depends on the scale and on whether high or low values indicate the positive outcome; it is not universally “toward zero” or “toward one.”

Paired and independent ROC curves

If several tests are measured on the same participants, their ROC curves are paired. A statistical comparison must account for the covariance created by the shared participant rows. Curves from separate participant samples are independent and require an unpaired comparison. Comparing groups from one tidy dataset also creates independent group curves because different rows belong to the groups.

numiqo uses paired or unpaired DeLong comparisons as appropriate and reports Holm-adjusted p-values when several curve pairs are tested. See the guide to comparing ROC curves for data layouts, missing-value handling and interpretation.

Uncertainty in curves and cutoffs

AUCs, cutoff performance and the curve itself are estimates from a sample. numiqo reports DeLong uncertainty for AUCs and stratified bootstrap confidence intervals for selected operating-point measures. Optional shaded bands are pointwise 95% bootstrap intervals for sensitivity at fixed specificity values. They are not a simultaneous confidence region for the entire curve.

A threshold selected and evaluated on the same data can look better than it performs in a new sample. Important decision rules should therefore be validated using independent data. The full calculation details and pROC reference-test scope are documented on the methods and validation page.

ROC curve and logistic regression

How does this relate to logistic regression? We could, for example, create a new classifier using logistic regression. Here we could use, in addition to the blood value, a person's age and gender.

In logistic regression, the estimated value is the probability that a particular person has the disease.

Very often, 50% is used as the threshold to classify whether a person is "diseased" or not. But this does not have to be the case. Any threshold can be used.

Therefore, we can also create a ROC curve by varying the threshold in logistic regression.

Create ROC curve with numiqo

Of course, we can easily output a ROC curve online with numiqo. To do this, we simply copy our data into the table and click on ROC Calculator. Alternatively, you can also create an ROC curve in the Regression Calculator in Logistic Regression.

We simply select the two variables Diseased and Blood Value and specify what we consider a positive event, in our case the answer yes. Now we get the ROC curve. In the table below the ROC curve, you can see the threshold value for each point on the curve.

For several tests or groups, start with one of the paired, independent or grouped example datasets on the calculator page. The Case Processing Summary shows which observations were analyzed and which were excluded because a required value was missing.