Response consistency of ChatGPT-4o for Type 2 Diabetes Nutrition and Physical-activity Recommendations: A Pilot NLP-based Assessment of GPT outputs

Generative AI tools such as ChatGPT are increasingly used by the public to seek guidance on diet and physical activity for type 2 diabetes (T2D) prevention and management. However, the consistency of model outputs across different users and disease-stage scenarios remains insufficiently characterized. This pilot study aims to evaluate the word-level and semantic-level consistency of GPT-4os diet and physical activity responses for type 2 diabetes prevention and management. We designed 12 prompts covering four categories: prediabetes, diagnosed type 2 diabetes (T2D), diagnosed T2D with complications, and general questions that did not specify dysglycemia stage. Word-level similarity was quantified with Term Frequency-Inverse Document Frequency (TF-IDF) cosine scores; sentence-level semantic similarity was measured using large language models (LLMs) - DeBERTa-v3 MNLI to calculate the entailment probabilities. The results showed that mean cosine similarity across users was moderate (0.44-0.66), whereas mean entailment similarity was higher (0.68-0.81). Across stages, word-level similarity was low to moderate (0.28-0.63) and entailment similarity remained moderate to high (0.63-0.80). Low similarity commonly referenced distinct food choices, operational details, safety warnings, and stage-specific suggestions. GPT-4o generated semantically consistent but variably detailed responses and the moderate semantic variation suggested limited differentiation of response content across diabetes-related stages in this pilot consistency assessment.