A Comparative Analysis of Deep Learning, Traditional Machine Learning, and Statistical Methods for Predictive Modeling

Abstract

Selecting a predictive modeling family for tabular data remains difficult because deep neural networks, tree-based machine learning, and conventional statistical models embody different inductive biases, computational demands, and assumptions. This study presents a leakage-controlled, reproducible comparison of five representative methods logistic regression, shrinkage linear discriminant analysis (LDA), random forest, XGBoost, and a three-hidden-layer multilayer perceptron (MLP) across ten binary classification tasks spanning finance, security, health, physics, transport, social science, and controlled synthetic stress tests. Models were evaluated using five-fold cross-validation repeated twice; the transport choice data were split by traveler to prevent group leakage. Area under the receiver-operating-characteristic curve (AUROC) was the primary endpoint, complemented by balanced accuracy, F1, Matthews correlation coefficient, Brier score, log loss, and computational time. Dataset-level AUROC ranks were compared using a Friedman omnibus test, a Nemenyi critical-difference analysis, and paired Wilcoxon signed-rank tests with Holm multiplicity correction. The omnibus ranking differed significantly across models (χ²(4)=10.88, p=.028). XGBoost achieved the best average rank (1.70) and the highest mean AUROC across tasks (0.906), followed by random forest (rank 2.70; mean AUROC 0.900). The MLP ranked third (3.30; mean AUROC 0.859) and was significantly worse than XGBoost after Holm correction (adjusted p=.039; rank-biserial effect=-0.964). However, XGBoost did not significantly outperform random forest, logistic regression, or LDA after multiplicity correction, underscoring substantial dataset dependence. Statistical models remained highly competitive on low-dimensional and approximately linear tasks, while nonlinear ensemble methods dominated the transport and noisy synthetic problems. The findings support model-family selection through rigorous cross-validated benchmarking rather than an a priori preference for model complexity, and they clarify how contemporary tabular deep-learning advances should be interpreted relative to strong classical baselines.