Watch parameters descend a loss surface

Training a model means choosing parameters that reduce an objective. Gradient descent does not inspect every possible setting. It evaluates the derivative at the current point, then moves opposite the local gradient. Repeating that simple operation produces a path through parameter space.

The simulator uses two visible parameters, x and z, so the objective can be drawn as terrain. The vertical coordinate is a compressed display of the analytic loss; the optimizer still receives the exact derivative of the original function. That distinction matters because visualization transforms should never silently change the algorithm being taught.

On the quadratic bowl, every gradient points toward one global minimum. This is the clean case: a modest learning rate converges, a tiny rate crawls, and an excessive rate overshoots. Momentum accumulates a velocity, accelerating along consistent slopes while damping some zig-zag motion.

The Rosenbrock function introduces a curved ravine. The shortest local direction points toward a valley wall, not straight along the valley toward the minimum. A step size that works on the bowl can oscillate or diverge here. The saddle has positive curvature in one direction and negative curvature in another, exposing why nearly flat gradients can delay escape. Himmelblau's function has multiple minima, so starting position influences the solution reached.

Noise is added deterministically from a seeded pseudo-random sequence, making runs reproducible. It approximates stochastic gradient variation rather than a particular mini-batch. Gradient clipping limits the update direction's norm before the optimizer uses it; clipping can prevent an explosion, but it also changes the effective step and does not repair a fundamentally poor objective or schedule.