Building a recurrence model that is evaluated out of sample
The Cox model in Part 3 was designed to estimate adjusted associations. A prediction model has a different goal. It should produce useful probabilities for records that were not used to fit it.
This analysis focuses on ipsilateral recurrence within 24 months. Records censored before 24 months without recurrence are excluded because their 24-month outcome is unknown. This leaves 528 records and 125 events.
Two prespecified models
The baseline model contains information that could be available early in the clinical pathway.
– Age
– Sex
– Operative indication
– Smoking history
– Pack-years
The perioperative model adds variables observed around surgery and early recovery.
– Previous tube thoracostomy
– Operative bleb or bulla status
– Multiple wedge resection
– Postoperative air-leak duration
– Postoperative complication
The comparison asks how much additional predictive information these perioperative variables provide.
Keep preprocessing inside cross-validation
Five-fold stratified cross-validation is used. Four folds train the model and the remaining fold receives predictions. This process is repeated until every record has a probability from a model that did not train on that record.
Missing pack-years are imputed inside each training fold. Scaling and one-hot encoding are also learned inside the fold. Keeping these steps in one pipeline prevents held-out information from leaking into model development.
The final metrics are calculated from the combined out-of-fold predictions.
Discrimination and overall error
ROC AUC measures how often a randomly selected event receives a higher score than a randomly selected non-event. Average precision summarizes precision across recall levels and is sensitive to event prevalence. The Brier score measures mean squared probability error. Lower Brier values are better.
| Model | ROC AUC | 95% bootstrap CI | Average precision | Brier score |
|—|—:|—:|—:|—:|
| Baseline | 0.618 | 0.560 to 0.672 | 0.340 | 0.175 |
| Perioperative | 0.684 | 0.627 to 0.737 | 0.429 | 0.165 |
The complete output is in `tables/08_prediction_performance.csv`.
The perioperative model has better discrimination and a slightly lower Brier score. Its AUC remains moderate. That is a more useful description than calling the model accurate.
Comparing AUCs with a paired bootstrap
Both models make predictions for the same records. The AUC comparison should preserve this pairing.
The paired bootstrap repeatedly samples records with replacement and calculates both AUCs in the same resample. The distribution of their differences provides a confidence interval.
| Comparison | AUC difference | 95% bootstrap CI | Two-sided bootstrap p-value |
|—|—:|—:|—:|
| Perioperative minus baseline | 0.066 | 0.026 to 0.111 | 0.001 |
The comparison is stored in `tables/09_auc_comparison.csv`.
The estimated improvement is about 0.07 AUC units. The interval excludes zero in this cohort. This result describes internal model performance. It does not show that the model will work in another hospital or time period.
Calibration answers another question
Discrimination concerns ranking. Calibration concerns the probability values themselves.
A model can rank patients correctly and still predict risks that are too high or too low. The calibration intercept detects systematic overprediction or underprediction. The calibration slope detects predictions that are too extreme or too narrow.
| Model | Calibration intercept | Calibration slope |
|—|—:|—:|
| Baseline | -0.253 | 0.766 |
| Perioperative | -0.144 | 0.856 |
Ideal values are 0 for the intercept and 1 for the slope. The perioperative model is closer to these targets. Its slope below 1 still suggests some overfitting or overly variable predictions.
What cross-validation does not solve
Cross-validation reduces the optimism caused by evaluating a model on its training data. It does not create external validation. Records still come from the same data-generating setting and share the same variable definitions.
A clinical prediction model would also need prespecified handling of missing data, temporal validation, external validation, decision thresholds, and an assessment of clinical utility. Calibration would need to be checked again in each new population.
The useful result here is the workflow. Features and preprocessing are declared before evaluation. Predictions are out of fold. Discrimination, probability error, calibration, and model comparison are reported together.
Part 5 examines the stability of the main recurrence result and shows how a future study can be planned around the expected number of events.
Leave a Reply