ROC Calculator Methods and Validation
Author: Dr. Mathias Jesussek
Updated:
This page documents how numiqo calculates ROC results and how the implementation is checked against the open-source pROC package for R. The comparison is a statistical reference test; numiqo is not affiliated with or endorsed by the pROC project.
Analysis population and direction
A curve is calculated from rows containing both a valid binary outcome and a finite test value. The selected positive outcome defines cases; all other valid outcome categories count as controls. For a high-value-positive analysis, a result is positive when the test value is greater than or equal to the cutoff. For a low-value-positive analysis, it is positive when the value is less than or equal to the cutoff.
Direction is specified by the analyst. Automatically choosing whichever direction gives the larger AUC would use the observed outcome twice and can bias apparent discrimination upward.
AUC and DeLong uncertainty
The empirical AUC is calculated from the ranking of positive and negative scores, with tied positive–negative pairs contributing one half. numiqo calculates the nonparametric DeLong variance from case and control placement values. The AUC table contains the standard error, a normal-approximation 95% confidence interval and a two-sided test of H0: AUC = 0.5. At least two valid positive and two valid negative observations are required for a DeLong variance estimate.
Paired and unpaired AUC comparisons
For paired curves, comparison-specific complete rows are used and the covariance between the two AUC placement-value estimates is included. For independent curves, the variance of the AUC difference is obtained from the separate curve variances. When several curve pairs are compared, Holm-adjusted p-values are reported alongside the unadjusted values.
Cutoff calculations and bootstrap intervals
Candidate optimized cutoffs are derived from distinct observed score values. The Youden analysis maximizes sensitivity + specificity − 1. Screening and confirmatory analyses first enforce the selected sensitivity or specificity constraint and then maximize the complementary measure. A prespecified cutoff is evaluated as entered.
Operating-point intervals use 2,000 stratified bootstrap resamples with a reproducible fixed seed. Positive and negative observations are resampled separately. Optimized cutoffs are selected again within every resample; a prespecified cutoff is held fixed. The displayed intervals are percentile intervals. Expensive bootstrap results are calculated lazily and cached for the current data and settings.
Pointwise ROC confidence bands
When requested, numiqo calculates sensitivity at a grid of fixed specificity values in
each of 2,000 stratified bootstrap samples and displays the 2.5th and 97.5th percentiles.
These are pointwise intervals, comparable in purpose to pROC’s ci.se()
workflow. They are not simultaneous confidence bands and should not be interpreted as
containing the complete population ROC curve with 95% probability.
pROC-based reference tests
Automated reference fixtures compare numiqo with pROC results for missing observations, paired curves, independent curves and grouped data. Reference tests cover AUC estimates, DeLong uncertainty and paired or unpaired comparisons. The comparison scope is stated explicitly because matching selected pROC workflows does not imply that every pROC option or method is implemented.
| Capability | numiqo method | pROC reference workflow |
|---|---|---|
| Empirical AUC | Rank/placement formulation with tie handling | roc() and auc() |
| AUC 95% CI | DeLong placement-value variance | ci.auc(method = "delong") |
| Paired AUC difference | Paired DeLong covariance | roc.test(..., paired = TRUE) |
| Independent AUC difference | Unpaired DeLong comparison | roc.test(..., paired = FALSE) |
| Pointwise sensitivity CI | 2,000 stratified bootstrap resamples | ci.se(..., boot.n = 2000, boot.stratified = TRUE) |
Important scope limits
- The module calculates empirical full ROC curves; it does not fit a smoothed ROC model.
- It does not currently report partial AUC.
- AUC measures discrimination and does not assess probability calibration.
- Pointwise bands do not provide simultaneous coverage of the entire curve.
- Cutoff optimization on the analysis sample requires external validation.
- Groups without valid positive and negative cases cannot produce an interpretable AUC.
Reproducibility and reporting
A reproducible report should identify the actual outcome, positive category, test score, score direction, valid and excluded observations, AUC method, cutoff purpose and any minimum sensitivity or specificity. For multiple curves, also report whether the design was paired or independent and whether p-values were Holm-adjusted.
Related ROC guides
- ROC curve calculator overview
- Choosing and evaluating a cutoff
- Comparing paired and independent curves
- ROC curve and AUC tutorial