Optimization of High-Dimensional Functions Using Gradient Descent and Genetic Algorithms with Computational Complexity Analysis
DOI:
https://doi.org/10.66021/Keywords:
High-dimensional optimization, Gradient Descent, Genetic Algorithms, Curse of Dimensionality, Saddle Points, Cooperative Coevolution, Computational Complexity, Adam Optimizer, Large-Scale Global Optimization (LSGO), Hybrid OptimizationAbstract
The optimization of high-dimensional functions poses significant challenges due to the curse of dimensionality, data sparsity, metric concentration, and the proliferation of saddle points in non-convex landscapes. This paper provides a comprehensive analysis of two dominant paradigms in high-dimensional optimization: gradient-based methods and genetic algorithms (GAs). Gradient descent variants, including Stochastic Gradient Descent (SGD), Momentum, Adagrad, and Adam, leverage local curvature information for efficient local refinement, while genetic algorithms employ population-based evolutionary search with selection, crossover, and mutation to explore the global search space. Special attention is given to Large-Scale Global Optimization (LSGO) techniques, particularly Cooperative Coevolution (CC) and variable interaction detection methods such as Min-Hash Decomposition (MHD), which address the exponential growth of the search space through decomposition strategies. A detailed computational complexity analysis compares the scaling behavior of batch gradient descent, SGD, Adam, and genetic algorithms, highlighting trade-offs between precision, convergence speed, and resource consumption. The study further examines hybrid approaches that integrate the global exploratory power of genetic algorithms with the exploitative precision of gradient-based optimizers, such as the Adam RNA-GA framework. Benchmark results from CEC 2013 and aerodynamic shape optimization demonstrate the strengths and limitations of each paradigm. Emerging trends, including Strategic Monte Carlo Optimization (SMCO), Quality-Diversity (QD) optimization, and green optimization frameworks, point toward future directions in the field. This work underscores the necessity of hybrid and strategically guided methods for effectively navigating the complex hyper-surfaces of high-dimensional optimization problems in machine learning, engineering, and scientific computing.
