AI learns racial information from the values of vital signs
Velichkovska, B.; Gjoreski, H.; Denkovski, D.; Kalendar, M.; Dankwa-Mullan, I.; Gichoya, J.; Martinez-Martin, N.; Celi, L.; Osmani, V.
Show abstract
BackgroundBias in medical practice is multifaceted, including treatment variations across race-ethnicity, unconscious bias in healthcare providers attitudes, and bias in clinical scores. However, far less is known about the potential racial bias in routinely collected, essential information in clinical decision-making, namely vital signs. Research questionDo vital signs embed racial information that can be learned by AI algorithms? Study Design and MethodsRetrospective cohort study of critically ill patients between 2014 and 2015 from the multi-centre eICU-CRD critical care database involving 335 Intensive Care Units (ICU) based in 208 US hospitals, containing 200,859 patient admissions. We extracted 10,763 critical care admissions of patients aged 18 and over, alive during the first 24 hours after admission to ICU with recorded self-reported race as well as at least two measurement of heart rate, oxygen saturation, respiratory rate, and blood pressure. Pairs of racial subgroups were matched based on age, gender, admission diagnosis and APACHE IV scores. Traditional machine learning algorithms, including XGBoost and Logistic regression were used to predict self-reported race using values of vital signs as an input. ResultsAI models derived from only six vital signs can predict patients self-reported race with an AUC of 0.74 ({+/-} 0.022) between White and Black patients. Technologies used to measure oxygen saturation are a significant source of self-reported racial information (AUC of 0.72 {+/-} 0.028), in addition to blood pressure measurements (AUC of 0.63 {+/-} 0.035). Care delivery practices do not present a significant source of racial information (AUC of 0.57 {+/-} 0.019). However, even when controlling for these known factors, self-reported race can still be learned from vital signs, whose origin we cannot currently explain. InterpretationVital signs embed racial information that can be learned by AI algorithms, posing a significant risk to equitable clinical decision-making. Mitigating measures might be challenging, considering fundamental role of vital signs.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- A proposed de-identification framework for a cohort of children presenting at a health facility in Uganda 94%
- Identification of predictive patient characteristics for assessing the probability of COVID-19 in-hospital mortality 94%
- Predictability and Stability Testing to Assess Clinical Decision Instrument Performance for Children After Blunt Torso Trauma 93%
Similar papers in this journal
- Imputation of PaO2 from SpO2 values from the MIMIC-III Critical Care Database Using Machine-Learning Based Algorithms 95%
- Using explainable machine learning to identify patients at risk of reattendance at discharge from emergency departments 94%
- AKI Risk Score (AKI-RiSc): Developing an Interpretable Clinical Score for Early Identification of Acute Kidney Injury for Patients Presenting to the Emergency Department 93%
Similar papers in this journal
- Predicting mortality in SARS-COV-2 (COVID-19) positive patients in the inpatient setting using a Novel Deep Neural Network 95%
- Image and structured data analysis for prognostication of health outcomes in patients presenting to the Emergency Department during the COVID-19 pandemic 94%
- Predicting Prognosis in COVID-19 Patients using Machine Learning and Readily Available Clinical Data 92%
Similar papers in this journal
- A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis 94%
- A Machine Learning-Based Prediction of Hospital Mortality in Mechanically Ventilated ICU Patients 94%
- REPLICCAR II Study: Data Quality Audit in the Paulista Cardiovascular Surgery Registry 94%
Similar papers in this journal
- Implicit bias in Critical Care Data: Factors affecting sampling frequencies and missingness patterns of clinical and biological variables in ICU Patients 94%
- Development and Validation of ‘Patient Optimizer’ (POP) Algorithms for Predicting Surgical Risk with Machine Learning 94%
- Prediction of Sepsis Mortality in ICU Patients Using Machine Learning Methods 94%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.