Comparative Evaluation of Reasoning Consistency Detection Methods on LLM Outputs in Cardiovascular Clinical Scenarios

The advancement of Large Language Models (LLMs) presents opportunities for developing cognitive assistants in the medical field, particularly in cardiovascular care.However, hallucinations and reasoning inconsistencies remain significant challenges, while the evaluation of logical consistency in this domain is limited due to the absence of public datasets with clear ground truth. This study aims to compare the effectiveness of Self-Consistency (LLM), Sentiment Heuristic, and Keyword Overlap methods in detecting reasoning inconsistencies using a synthetic dataset of 90 cardiovascular clinical scenarios (30 scenarios with 3 variants each). Performance was evaluated based on accuracy, precision, recall, and F1-score, and benchmarked against a fine-tuning approach. The results show that the zero-shot detector based on self-consistency achieved the best performance, with an accuracy of 97.78% and an F1-score of 98.31%, while the fine-tuning approach on the limited synthetic dataset failed to surpass these results. These findings indicate that the semantic reasoning capabilities of LLMs have the potential to serve as consistency detectors without additional training, and can serve as a foundation for developing cognitive assistant systems in the cardiovascular domain.