A MATHEMATICAL ANALYSIS OF GRADIENT DESCENT AND ITS VARIANTS IN ARTIFICIAL INTELLIGENCE

Abstract

Gradient descent is the core AI optimization algorithm : it iteratively steps along the negative gradient to find the parameters that minimize a loss function . In this paper we present a mathematical analysis of the algorithm and its main variants. The aims are to develop the theoretical background of gradient descent, to derive and compare its variants and to analyze their convergence behavior and computational efficiency. Analysis of first and second order derivatives, Hessian and convexity classification of objective functions. Formulation Batch, stochastic and mini-batch descent Momentum and Nesterov acceleration Adaptive methods: AdaGrad, RMSProp and Adam. We consider convergence in terms of smoothness, strong convexity and non-convexity and we differentiate between linear, sublinear and stochastic rates. A numerical experiment on the convex quadratic f(x,y)=x^2+4y^2 verifies the theory quantitatively: full-gradient descent achieves a gradient norm below 〖10〗^(-3) in 19 iterations at a final loss of 4.83x〖10〗^(-8), RMSProp in 38 iterations, while stochastic descent stalls at a variance floor near 3.31x〖10〗^(-3). The results show that no variant dominates: each trades stability, speed and adaptivity against the others, and the guaranties available under convexity degrade to mere stationarity when the landscape is non-convex. The choice of optimizer is mathematical, not a conventional, choice.