Evidence of Unreliable Data and Poor Data Provenance in Clinical Prediction Model Research and Clinical Practice

Clinical prediction models are often created using large routinely collected datasets. It is essential that prediction models are developed with appropriate data and methods and transparently reported to ensure that decisions are based on reliable predictions. Kaggle is a popular competition website where users learn and apply analysis skills on a range of datasets. We identified two large, publicly available Kaggle datasets, on stroke and diabetes, that lack clear data provenance, but are widely used in clinical prediction models in peer-reviewed publications. The authenticity of both datasets could not be verified and have evidence they are likely to be simulated or fabricated. Data provenance assessment using nine TRIPOD+AI items revealed major deficiencies, with minimal details for either dataset including no information on when, where, why or how the data were collected. From these two datasets, we found 124 clinical prediction model studies. Three prediction models had evidence of use in clinical practice, one model was cited in a medical device patent, and the models were cited in 86 review articles. We recommend that journals and data repositories mandate data provenance reporting to safeguard published research. Prediction models based solely on simulated or fabricated data sets should never be used to directly inform decisions on patient care.