Back

Authority Signals in AI Cited Health Sources: A Framework for Evaluating Source Credibility in ChatGPT Responses

Jacques, E.; Datuowei, E.; Jones, V.; Basch, C.; Vanderpool, C.; Udeozo, N.; Chapa, G.

2026-01-23 health informatics
10.64898/2026.01.22.26344576 medRxiv
Show abstract

Health information seeking has fundamentally changed since the onset of Large Language Models (LLM), with nearly one third of ChatGPTs 800 million users asking health questions weekly. Understanding the sources of those AI generated responses is vital, as health organizations and providers are also investing in digital strategies to organically improve their ranking, reach and visibility in LLM systems like ChatGPT. As AI search optimization strategies are gaining maturity, this study introduces an Authority Signals Framework, organized in four domains that reflect key components to health information seeking, starting with "Who wrote it ?" (Author Credentials), followed by "Who published it?" (Institutional Affiliation), "How was it vetted?" (Quality Assurance), and "How does AI find it?" (Digital Authority). This descriptive cross-sectional study randomly selected 100 questions from HealthSearchQA which contains 3,173 consumer health questions curated by Google Research from publicly available search engine suggestions. Those questions were entered into ChatGPT 5.2 Pro to record and code the cited sources through the lens of the Authority Signals Frameworks four domains. Descriptive statistics were calculated for all cited sources (n=615), and cross tabulations were conducted to examine distinction among organization types. Over 75% of the sources cited in ChatGPTs health generated responses were from established institutional sources, such as Mayo Clinic, Cleveland Clinic, Wikipedia, National Health Service, PubMed with the remaining citations sourced from alternative health information sources that lacked established institutional backing.

Matching journals

The top 3 journals account for 50% of the predicted probability mass.

50% of probability mass above

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.