Prognostic predictions in psychosis: exploring the complementary role of machine learning models

BACKGROUNDPredicting outcomes in schizophrenia spectrum disorders is challenging due to the variability of individual trajectories. While machine learning (ML) shows promise in outcome prediction, is has not yet been integrated into clinical practice. Understanding how ML models (MLMs) can complement psychiatrists predictions and bridge the gap between MLM capabilities and practical use is key. OBJECTIVEThis study aims to compare the performance of psychiatrists and MLMs in predicting short-term symptomatic and functional remission in patients with first-episode psychosis and explore whether MLMs can improve psychiatrists prognostic accuracy. METHODTwenty-four psychiatrists predicted symptomatic and functional remission probabilities based on written baseline information from 66 patients in the OPTiMiSE trial. ML-generated predictions were then shared with psychiatrists, allowing them to adjust their estimates. A questionnaire assessed trust in MLMs, perceived information gaps, and psychiatrists self-assessed predictive accuracy, which was compared to actual accuracy. FINDINGSThe predictive accuracy of the MLM was comparable to that of psychiatrists for symptomatic remission (MLM: 0.50, psychiatrists: 0.52) and functional remission (MLM: 0.72, psychiatrists: 0.79). Interrater agreement was low but comparable for psychiatrists and the MLM. Although the MLM did not improve overall predictive accuracy, it showed potential in aiding psychiatrists with difficult-to-predict cases. However, psychiatrists struggled to recognize when to rely on the models output and we were unable to determine a clear pattern in these cases based on their characteristics. Psychiatrists could not reliably estimate their predictive accuracy. Psychiatrists expressed moderate to high trust in MLMs for prognostic prediction, but highlighted concerns about the lack of transparency and interpretability of model outputs. CONCLUSIONSMLMs are a promising tool for supporting psychiatric decision-making, particularly in challenging cases. However, their potential remains underutilized due to limitations in predictive accuracy and a lack of clarity in how predictions are generated. Addressing these issues is essential to build trust and foster integration into clinical practice. CLINICAL IMPLICATIONSMLMs are best suited as supplementary tools, providing a second opinion while psychiatrists retain decision-making autonomy. Integrating predictions from both sources may help reduce individual biases and improve accuracy. This approach leverages the strengths of MLMs without compromising clinical responsibility. SUMMARY BOXO_ST_ABSWhat is already known on this topicC_ST_ABSWhile machine learning models (MLMs) show promise in predicting outcomes in psychotic disorders, they have yet to be integrated into clinical practice. Evidence on the predictive accuracy of psychiatrists for these disorders is limited, with only two small studies published before 1990 suggesting moderate accuracy. Comparisons of MLMs and psychiatrists in this context have not been previously conducted. What this study addsThis is the first study to compare the predictive accuracy of psychiatrists with that of an MLM for psychotic disorders and to assess whether an MLM can enhance psychiatrists performance. It highlights that while MLMs do not improve overall accuracy, they may support psychiatrists in difficult cases. Insights into psychiatrists trust in MLMs and the challenges of implementing these models are also provided. How this study might affect research, practice, or policyThe findings emphasize the need for advancements in MLM accuracy, interpretability, and strategies to identify cases where MLMs are most beneficial. These improvements could foster effective integration of MLMs as supplementary tools in clinical practice, aiding psychiatrists in decision-making while maintaining their autonomy.