Chapter 16: clinical usefulness

Cleveland 2008: discussion on performance measures

Substantial discussion has emerged on measures to quantify the clinical usefulness of a prediction model. This issue was discussed at a symposium in Cleveland, Sept 29, 2008. A related paper in Epidemiology (Jan 2010) aimed to present a perspective on some traditional and modern performance measures (R code here). It was accompanied by a vivid discussion in an Editorial and response from the authors.

A key element for performance evaluation is a validation graph.
At github: R code, from BavoDC/CalibrationCurves: Calibration Performance, maintained by Bavo de Cock. Documentation also at cran: CalibrationCurves: Calibration Performance.

Data sets

Epidemiology 2010, testicular cancer example

Rev Esp Cardiol. 2011, Framingham data

Net Benefit and Decision Curve Analysis (DCA)

Andrew Vickers (@VickersBiostats) maintains a rich source with information on Net Benefit estimation: software, examples, and interpretation. Marshall Brown <mdbrown@fredhutch.org> also developed an R implementation.

Many references are at the STRATOS TG6 site; some below for a selection.

Literature

  1. Original proposal for DCA
    Vickers AJ, Elkin EB.
    Decision curve analysis: a novel method for evaluating prediction models.
    Med Decis Making. 2006 Nov-Dec;26(6):565-74. doi: 10.1177/0272989X06295361

  2. Net benefit as a concept in other papers
    Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD.
    Validity of prognostic models: when is a model clinically useful?
    Semin Urol Oncol. 2002 May;20(2):96-107. doi: 10.1053/suro.2002.32521

Peirce CS
The numerical measure of the success of predictions
Science, 1884

  1. Explanation and guidance
    Vickers AJ, Van Calster B, Steyerberg EW.
    Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.
    BMJ. 2016 Jan 25;352:i6. doi: 10.1136/bmj.i6

    Vickers AJ, van Calster B, Steyerberg EW.
    A simple, step-by-step guide to interpreting decision curve analysis.
    Diagn Progn Res. 2019 Oct 4;3:18. doi: 10.1186/s41512-019-0064-7

    Van Calster B, Wynants L, Verbeek JFM, Verbakel JY, Christodoulou E, Vickers AJ, Roobol MJ, Steyerberg EW.
    Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators.
    Eur Urol. 2018 Dec;74(6):796-804. doi: 10.1016/j.eururo.2018.08.038

  2. Multiple estimates of Net Benefit can be summarized in a meta-analysis
    Wynants L, Riley RD, Timmerman D, Van Calster B.
    Random-effects meta-analysis of the clinical utility of tests and prediction models.
    Stat Med. 2018 May 30;37(12):2034-2052. doi: 10.1002/sim.7653.

  3. Illustration
    Steyerberg EW, Van Calster B, Pencina MJ.
    [Performance measures for prediction models and markers: evaluation of predictions and classifications].
    Rev Esp Cardiol. 2011 Sep;64(9):788-94. doi: 10.1016/j.recesp.2011.04.017