Math
Optimisation on a landscape
Gradient descent, the heavy ball, the Nelder–Mead simplex, simulated annealing and a genetic population go down the same surface - the bowl shows the condition number, the banana shows the crawl, Himmelblau shows the basins and Rastrigin shows who gets stuck.
What happens
A lit surface whose height is the function to be minimised, and five ways down it. Gradient descent steps against the slope and, in a narrow valley, zigzags from wall to wall - too big a step and it flies out. Momentum smooths the zigzag and overshoots the bottom. Newton's method jumps to the minimum of the local parabola, brilliant near a bottom and lost far from one. Rosenbrock's banana valley punishes descent, Himmelblau has four bottoms and the start decides which, Rastrigin has a hollow at every integer and the deepest is hard to find. Pick a surface and a start, drop the balls, and count their steps.
Equations used in the module
Equations descent: x' = x − η ∇f; heavy ball: v' = βv − η ∇f, x' = x + v bowl f = ½(x² + κy²): descent converges only for η < 2/κ Rosenbrock (1 − x)² + 100(y − x²)²; Himmelblau, four bottoms; Rastrigin, hollows everywhere Newton: x' = x − H^(−1) ∇f, quadratic near a bottom
