Systematic Scoring Differences in Large Language Model-Based Clinical Benchmarking Reveal Evaluator-Dependent Divergence: Controlled Evaluation Study

Background

Large language model (LLM) benchmarking in clinical medicine is increasingly reliant on LLM-based assessment frameworks; in these systems, one AI scores the outputs of another. This practice relies on the assumptions of evaluator neutrality and rubric consistency. These assumptions have not been empirically tested in clinical fields.

Objective