Diagnostics
Evaluation of Large Language Models for Emergency Triage in Urology: A Multi-Model Study
By Qi Cai, Kaipeng Bi, Xinyu Zhang and 5 others.
Abstract
Objective
To evaluate the triage accuracy, response quality, safety, and repeatability of widely used large language models (LLMs) in urologic emergencies and to examine their potential clinical utility and risks.
Methods
We developed 50 standardized clinical vignettes covering 10 categories of urologic emergencies, including testicular torsion, acute urinary retention, and Fournier gangrene. The set comprised high-risk vignettes requiring immediate emergency care, borderline vignettes, and atypical vignettes containing misleading cues. Six commercially available LLMs—Kimi 3.0, DeepSeek V4, Qwen 3.8-MAX, GPT-5.6 Sol (ChatGPT), Claude Opus 4.8, and Ant Afu (AQ)—were evaluated in Chinese-language interactions. Each vignette was submitted three times in independent conversations. Two urologists with more than 10 years of clinical experience assessed triage accuracy using a three-level reference standard (immediate emergency department [ED] evaluation, routine outpatient evaluation, or home observation). They also rated response quality across eight dimensions using a 5-point Likert scale and evaluated medical-error severity (0–3), response stability, and Chinese readability. Overall differences were assessed using the Friedman test, followed by pairwise Wilcoxon signed-rank tests with Holm adjustment.