Evaluation of Large Language Models for Post-Cystectomy Sexual Health Counseling in Women: A Pilot Study

Abstract Objective To evaluate the adherence to guidelines and readability of large language model-generated sexual health information related to female sexual dysfunction following cystectomy, and to determine whether adherence differs across models and prompt formats. A secondary objective was to introduce an analytic strategy using principal component analysis to examine the dimensions of readability metrics. Methods Three large language models (LLMs), ChatGPT, Gemini, and Perplexity were prompted with six clinical questions related to sexual function after cystectomy. Questions were phrased in long-form and short-form language. Responses were independently graded by two reviewers, derived from guideline recommendations. Linear mixed-effects models predicted adherence as functions of LLM, prompt, and reviewer, with clinical questions as a random intercept. Readability was assessed using five metrics, and principal component analysis (PCA) was used to determine latent structure. Results ChatGPT demonstrated the highest (estimated marginal mean [emm] = 0.769), outperforming Gemini (0.499) and Perplexity (0.457). Shorter, less complex prompts elicited higher adherence than more complex, clinical prompts. All models produced content that exceeded recommended reading levels. PCA demonstrated that a single dominant component accounted for 76.7% of variance across readability indices, indicating a shared underlying construct. Conclusion ChatGPT produced the most guideline-concordant information overall. High linguistic complexity was seen across models, highlighting a barrier to patient comprehension. These findings characterize large language models as variable medical information systems whose outputs rely heavily on prompt structure and model type.