Diagnostic agreement of large language models with clinical reference diagnoses of mental disorders and associated factors

Objectives

Insufficient diagnostic evidence and blurred diagnostic boundaries may compromise the reliability of clinical judgment and research labels, whereas standardized diagnostic interviews are resource-intensive and difficult to implement widely. Routinely archived inpatient medical records may therefore provide a practical basis for assisted diagnostic review. This study aims to evaluate the agreement of several commonly used general-purpose large language models (LLMs) with clinical reference diagnoses and the stability of their outputs across repeated runs when reviewing psychiatric admission records, analyze the associations of diagnostic information completeness and diagnostic complexity with Top-1 diagnostic agreement, and explore the characteristics of cases with frequent Top-1 disagreement to inform subsequent manual review.