Ch 5: Overfitting and optimism

Practical Exercise

We discuss causes of overfitting in predictive regression models, and its consequences, specifically optimism in performance estimates (Table 5.1).

These issues are presented in a practical exercise, which starts with re-creating Fig 5.2 in R, and a critical look at Fig 5.3 and 5.4.
Next, we discuss stepwise selection in the presence of pure noise predictors, in line with a paper by Mundry and Nunn.
Finally, we perform bootstrap resampling to study optimism.