Effects of temperature settings on information quality of ChatGPT-3.5 responses: A prospective, single-blind, observational cohort study

ObjectiveThe effect of temperature settings on the quality of ChatGPT version 3.5 (OpenAI) responses related to drug information remains unclear. We investigated ChatGPT-3.5s response quality on apixaban information with and without the temperature being set to 0. MethodsOn 6 September 2023, 37 questions regarding apixaban, derived from the frequently asked questions on the Bristol-Myers Squibbs website, were entered into ChatGPT in Japanese. The primary endpoint was the effect of temperature settings on ChatGPT-3.5s responses to apixaban-related questions. The response accuracy, clarity, detail, and adequacy were rated on a 5-point Likert scale by 10 pharmacists, with higher scores indicating higher response quality. Cumulative score means were analyzed using the Mann-Whitney U test. In the subgroup analysis, evaluators were limited to pharmacists at university hospitals. Welchs t-test was employed in sensitivity analysis to validate primary endpoint findings. ResultsThe mean scores for ChatGPT-3.5s apixaban-related responses with (13.08) and without (14.40) the temperature being set to 0 were not significantly different (p = 0.064). Accuracy differed significantly (3.15 vs. 3.54, p = 0.045), whereas clarity, detail, and appropriateness were similar. Subgroup analysis (13.30 vs. 14.21, p = 0.394) and sensitivity analysis confirmed similar results (13.45 vs. 14.52, p = 0.105). ConclusionsChatGPT-3.5 temperature setting does not significantly affect overall responses to apixaban-related inquiries. However, the variance in accuracy suggests that ChatGPT-3.5 is unable to consistently provide precise responses. Hence, it is more suitable as a supplementary tool rather than a primary medical resource.