Evaluating agentic simulation for local public health estimation

Large language model (LLM)-based generative agents can reproduce aspects of individual human behavior, but whether they can be scaled to geographically grounded populations that reproduce real-world health behaviors remains unclear. Here, we introduce LLMPopSim, a generative population simulation framework that integrates U.S. Census and Centers for Disease Control and Prevention data to construct synthetic individuals and uses an LLM to simulate individual health behaviors whose aggregate outcomes can be evaluated at the community level. We developed the framework using historical Hawaii data and evaluated temporal and geographic generalizability using held-out 2022 cohorts from Hawaii and New York State, with colorectal cancer screening and mammography as proof-of-concept behaviors. Across the four state-outcome evaluations, mean absolute error ranged from 3.5 to 15.0 percentage points and correlations between simulated and observed ZCTA-level prevalence ranged from 0.26 to 0.69. Performance differed across dimensions of population fidelity: colorectal cancer screening predictions preserved geographic ranking more strongly but systematically overestimated prevalence and compressed geographic variation, whereas mammography achieved lower absolute error but weaker geographic correlation and inconsistent preservation of between-community variability. Prediction error was greatest in communities with lower observed screening prevalence and varied across community characteristics without a uniform socioeconomic gradient. These findings demonstrate that individually represented LLM-based synthetic agents can aggregate into population-level patterns that retain measurable features of real-world health behavior across temporal and geographic transfer, while identifying calibration, distributional fidelity and subgroup performance as key challenges for generative population simulation. LLMPopSim provides an empirical foundation for developing synthetic populations that may ultimately enable simulation of heterogeneous population responses to public health interventions.