A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions

Objective: Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (e.g., total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions. Materials and Methods: Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under two encodings: (1) symbolic field-based tokens and (2) semantic natural language audit log action descriptions. A first-order Markov model, which used only the immediate prior action, served as baseline minimal context comparator. Model representation was assessed using next-action prediction accuracy in an early-period test set and two temporally distinct out-of-sample (OOS) periods. Results: In the early period test set, the semantic LLM achieved the highest accuracy (0.7418, 95%CI [0.7415 - 0.7420]) compared to the symbolic LLM (0.3838, 95% CI: 0.3836 - 0.3840) and Markov baseline (0.4553, 95%CI [0.4551 - 0.4555]). The semantic approach was also robust to temporal drift in EHR interaction patterns seen in the two OOS periods (semantic LLM vs Markov accuracy, OOS-1: 0.6509 vs 0.3169; OOS-2: 0.6232 vs 0.2648). Discussion and Conclusions: Semantic LLM relying on audit log action descriptions yielded the highest next-action prediction accuracy, and demonstrated robustness to temporal drift, suggesting that longer sequential context and semantic action descriptions may improve audit log-based sequence modeling. These findings support further development of semantic sequence models for audit log research, including task identification, automated workflow characterization, and safety-focused analyses of clinician-EHR interaction patterns.