Scaling the Prompt: How Batch Size Shapes Performance of Mid-2025 State-of-the-Art LLMs in Automated Title-and-Abstract Screening

BackgroundManual abstract screening is a primary bottleneck in evidence synthesis. Emerging evidence suggests that large language models (LLMs) can automate this task, but their performance when processing multiple records simultaneously in "batches" is uncertain. ObjectivesTo evaluate the classification performance of four state-of-the-art LLMs (Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-5, and GPT-5 mini) in predicting study eligibility across a wide range of batch sizes for a systematic review of randomised controlled trials. MethodsWe used a gold-standard dataset of 790 records (93 inclusions) from a published Cochrane Review. Using the public APIs for each model, batches of 1 to 790 citations were submitted to classify records as Include or Exclude. Performance was assessed using sensitivity and specificity, with internal validation conducted through 10 repeated runs for each model-batch combination. ResultsGemini 2.5 Pro was the most robust model, successfully processing the full 790-record batch. In contrast, GPT-5 failed at batches [≥]400, while GPT-5 mini and Gemini 2.5 Flash failed at the 790-record batch. Overall, all models demonstrated strong performance within their operational ranges, with two notable exceptions: Gemini 2.5 Flash showed low initial sensitivity at batch 1, and GPT-5 minis sensitivity degraded at higher batch sizes (from 0.88 at batch 200 to 0.48 at batch 400). At a practical batch size of 100, Gemini 2.5 Pro achieved the highest sensitivity (1.00, 95% CI 1.00-1.00), whereas GPT-5 delivered the highest specificity (0.98, 95% CI 0.98-0.98). ConclusionState-of-the-art LLMs can effectively screen multiple abstracts per prompt, moving beyond inefficient single-record processing. However, performance is model-dependent, revealing trade-offs between sensitivity and specificity. Therefore, batch size optimisation and strategic model selection are important parameters for successful implementation.