Scalable AutoML Frameworks for End-to-End Optimization of Data Science Workflows

Udariansyah, Devi and Alshar'e, Marwan and Kumar Yadav, Puneet and Tomar, Arjit (2026) Scalable AutoML Frameworks for End-to-End Optimization of Data Science Workflows. Journal of Data Science, 2026 (08). pp. 132-151. ISSN 2805-5160

	Text 926 - Published Version Available under License Creative Commons Attribution. Download (42kB)
	Text jods2026_08.pdf - Published Version Available under License Creative Commons Attribution. Download (391kB)

Abstract

The increasing complexity of machine learning workflows has created significant challenges in efficiently developing scalable and high-performing predictive models. Although Automated Machine Learning (AutoML) has emerged as a promising solution for reducing manual intervention in data science pipelines, existing approaches still suffer from limited scalability, high computational overhead, fragmented pipeline optimization, and insufficient resource-aware mechanisms. This study addresses these limitations by proposing a scalable end-to-end AutoML framework that integrates automated preprocessing, feature engineering, model selection, hyperparameter optimization, and adaptive resource allocation within a unified optimization architecture. The proposed methodology employs hierarchical search strategies, Bayesian and evolutionary optimization, and dynamic resource scheduling to improve optimization efficiency while maintaining predictive robustness. Experiments were conducted on medium- and large-scale datasets using repeated cross-validation and benchmark comparisons against conventional machine learning models and existing AutoML systems. The results demonstrate that the proposed framework achieved superior predictive performance, obtaining an accuracy of 0.93 ± 0.01 and an AUC of 0.96 ± 0.01, outperforming baseline approaches by approximately 8-12% across multiple evaluation metrics. In addition, the framework reduced optimization runtime by approximately 68% compared to existing AutoML systems while maintaining strong scalability across increasing dataset sizes. This research aims to advance scalable and efficient AutoML systems by providing a reproducible, resource-aware, and integrated framework for automating end-to-end data science workflows.

Item Type:	Article
Uncontrolled Keywords:	AutoML, Scalable Machine Learning, Hyperparameter Optimization, Adaptive Resource Allocation, Data Science Workflows
Subjects:	Q Science > QA Mathematics Q Science > QA Mathematics > QA75 Electronic computers. Computer science Q Science > QA Mathematics > QA76 Computer software
Depositing User:	Unnamed user with email masilah.mansor@newinti.edu.my
Date Deposited:	10 Sep 2026 06:05
Last Modified:	10 Sep 2026 09:34
URI:	http://eprints.intimal.edu.my/id/eprint/2363

Actions (login required)

View Item