Robust Evaluation Metrics for Assessing Machine Learning Performance Beyond Accuracy

Putra, Ade and Noche, Elmar B. and Gabhane, Diksha and Yeole, Abhay (2026) Robust Evaluation Metrics for Assessing Machine Learning Performance Beyond Accuracy. Journal of Data Science, 2026 (16). pp. 272-291. ISSN 2805-5160

	Text jods2026_16.pdf - Published Version Available under License Creative Commons Attribution. Download (436kB)
	Text 934 - Published Version Available under License Creative Commons Attribution. Download (40kB)

Abstract

The widespread deployment of machine learning (ML) systems in critical domains has exposed the limitations of accuracy-centric evaluation, particularly under conditions involving class imbalance, noise, and distributional shifts. Existing studies frequently employ alternative metrics in isolation and lack a unified framework capable of systematically assessing model robustness, reliability, and decision sensitivity across varying data conditions. To address this gap, this study proposes a structured multi-metric evaluation framework that integrates classification, ranking-based, calibration, and robustness-oriented metrics for comprehensive ML performance assessment. A quantitative experimental design is employed using multiple benchmark datasets with varying statistical characteristics, including balanced and imbalanced distributions. Controlled perturbation scenarios—including noise injection, class imbalance manipulation, and distribution shift simulation—are introduced to emulate realistic deployment environments. Several machine learning models, namely Logistic Regression, Support Vector Machines, Random Forest, and Multi-Layer Perceptron (MLP), are evaluated using metrics such as Accuracy, F1-score, ROC-AUC, PR-AUC, Brier Score, and Expected Calibration Error (ECE). The experimental results demonstrate that accuracy consistently overestimates model effectiveness under adverse conditions, while alternative metrics reveal substantial hidden weaknesses in minority class detection and probability reliability. Among the evaluated models, MLP achieved the strongest overall performance, obtaining a ROC-AUC of 0.94 and PR-AUC of 0.89 under baseline conditions. Furthermore, calibration-oriented metrics exhibited significantly higher sensitivity to perturbation severity compared to accuracy. This study contributes to the advancement of trustworthy artificial intelligence by promoting a comprehensive, context-aware, and robustness-oriented evaluation framework capable of supporting more reliable real-world ML deployment.

Item Type:	Article
Uncontrolled Keywords:	Machine Learning Evaluation, Performance Metrics, Robustness Assessment, Calibration Metrics, Trustworthy Artificial Intelligence
Subjects:	Q Science > QA Mathematics Q Science > QA Mathematics > QA75 Electronic computers. Computer science Q Science > QA Mathematics > QA76 Computer software
Depositing User:	Unnamed user with email masilah.mansor@newinti.edu.my
Date Deposited:	14 Sep 2026 10:06
Last Modified:	14 Sep 2026 10:06
URI:	http://eprints.intimal.edu.my/id/eprint/2371

Actions (login required)

View Item