Chapter 17: Internal validation

Split-sample, cross-validation, and the bootstrap

Several techniques are at the analyst's disposal for internal validation. In a classic JCE paper (Steyerberg, Harrell, 2001, pdf), we compared variants of split-sample validation, cross-validation, and bootstrap validation.

We found that:

1. Split-sample validation analyses gave overly pessimistic estimates of performance, with large variability;

2. Cross-validation on 10% of the sample had low bias and low variability;

3. Bootstrapping provided stable estimates with low bias.

We concluded that split-sample validation is inefficient, and recommended bootstrapping for estimation of internal validity of a predictive logistic regression model. This conclusion was largely confirmed in a later study, with a minor note of caution: “with lower events per variable or lower C statistics, bootstrapping tended to be optimistic but with lower absolute and mean squared errors than .. cross-validation”. In a recent JCE Commentary (Steyerberg&Harrell, 2015, pdf), we stated that 'split-sample validation only works when not needed', i.e. in the situation of very large sample size. This is because a stable prediction model can then be developed on the training part of the data, and a reliable assessment of performance can be obtained in a large test data set. In such a situation of large training and large test part, a better solution is to develop a model on the total data set, with the apparent performance as the best estimate of future performance.

Machine learning and high-dimensional model validation

Machine learning techniques are increasingly used for prediction in high-dimensional data, i.e. all kinds of 'omics' data. We should realize that machine learning techniques are relatively data hungry, and that high-dimensional data pose far more risk of overfitting than classical situations of say 5 to 20 predictors. Hence, validation is very important.

Researchers in this field often resort to old-fashioned split sample techniques. An extreme example was published in Nature Medicine in 2018. Here, researchers split a data set with 54 patients with leukemia in a training part with 44 patients, and a validation part with 10 patients. The endpoint of interest was relapse, which occurred in only 3 patients among the 10 patients in the training set. One does not have to be a statistician to understand that such a small sample size for validation leads to a highly unreliable assessment of performance (JCE 2018).