Journal of clinical medicine
Comparison of Internal Validation Strategies for Estimating Machine Learning Model Performance in One-Year Mortality Prediction in Hepatocellular Carcinoma
By Kucukakcali Z, Akbulut S and Sarici KB.
Abstract
Background
Machine learning (ML) performance estimates in hepatocellular carcinoma (HCC) may depend on internal validation strategy, especially in small datasets.
Aim
To develop models for one-year mortality prediction and compare performance, calibration, decision-analytic performance, and uncertainty estimates using a conventional train-test split and nested cross-validation (nCV).
Methods
We analyzed a public dataset of 165 patients with HCC, including 63 one-year deaths. Random Forest (RF), XGBoost, and CatBoost were developed using fold-specific preprocessing, cost-sensitive learning, Optuna-based hyperparameter optimization, and training-based threshold selection. L2-penalized logistic regression provided a conventional benchmark. Models were evaluated using a stratified 80/20 split and five-fold nCV with five inner folds. Six stratified splits assessed partition variability, and unweighted models assessed the effect of class weighting. Discrimination, threshold-dependent classification, calibration, and decision-analytic performance were assessed using ROC-AUC, PR-AUC, balanced accuracy, Matthew's correlation coefficient, Brier score, calibration curves, and decision-curve analysis. Confidence intervals used 2000 bootstrap resamples; SHAP was used for interpretation.