Gradient Descent

FoundationsIntermediate~8 min

Gradient Descent — Gradient descent is an iterative optimization algorithm that minimizes a loss function by repeatedly stepping in the direction of its negative gradient. The learning rate controls the step size and determines whether it converges or diverges.

Imagine you’re blindfolded on a hilly landscape and you want to find the lowest valley. Gradient descent is the strategy: feel which way is downhill, take a step that direction, repeat.

Loading 3D view…

Start
Descent path
Next step (−α∇J)
Minimum

Descent controls

Data

Loss surface

A round quadratic. The gradient always points straight at the minimum — the easy case.

Start θ₁-2.4

Start θ₂2.6

Or drag on the contour map to drop the start anywhere.

Model

Learning rate α0.10

Step budget70

Loss6.260

Step length0.354

Playback

Step 0 / 67

Speed

Steep — long steps
Shallow — short steps
Flat — converged
↻

Step 0 of 67 — at (-2.40, 2.60) the slope is ∇J = (-2.40, 2.60); α·|∇J| = 0.354 is how far the next step travels — the last one moved 0.000 and loss went 6.260 → 6.260

Loss vs iteration

Break it

The ravine's steep axis has curvature 4, so anything past α = 2/4 = 0.5 makes each step overshoot further than the last.

The idea in plain words

Gradient descent finds the bottom of a valley by feeling which way is downhill and taking a step that direction, over and over. The learning rate is the step size. Nudge it up and the path descends faster; push it to the top and each step overshoots, bouncing to ever-larger loss until it flies off to infinity.

The valley is defined by a loss function, and this is exactly how models like linear regression are fit when there’s no shortcut.

Now, the math

Each parameter θ updates by stepping against the gradient:

\theta \leftarrow \theta - \eta\,\nabla L(\theta)

$\theta$: a model parameter being tuned.
$\eta$: the learning rate — the step size.
$\nabla L(\theta)$: the gradient: the uphill direction of the loss.

▸ Show the derivation

On an elongated bowl, the steepest direction has the largest curvature. Convergence there requires the learning rate to stay below roughly twice the inverse of that curvature; above it, each step more than undoes the last and the loss diverges — which is exactly what the slider lets you trigger.

Trace it by hand

The loss is the interactive's bowl surface — half of x squared plus y squared — whose gradient at any point (x, y) is simply (x, y). Start at theta = (2, 1) with learning rate eta = 0.5 and take two updates. All numbers are exact.

The start point and its loss
$\theta_0 = (2,\ 1), \qquad L(\theta_0) = \tfrac{1}{2}\left(2^2 + 1^2\right) = 2.5$
This is the same bowl surface the interactive uses, with its minimum at the origin.
Feel the slope: the gradient
$\nabla L(\theta_0) = (x,\ y) = (2,\ 1)$
For this bowl the gradient equals the position itself — the farther out you are, the steeper it is.
First update: step against the gradient
$\theta_1 = (2,\ 1) - 0.5\,(2,\ 1) = (1,\ 0.5), \qquad L(\theta_1) = 0.625$
With eta equal to 0.5 on this surface, every coordinate exactly halves.
Second update: repeat from the new point
$\theta_2 = (1,\ 0.5) - 0.5\,(1,\ 0.5) = (0.5,\ 0.25), \qquad L(\theta_2) = 0.15625$
The gradient was recomputed at the new point before stepping — that re-measuring is the whole algorithm.
Watch the loss fall
$L: \ 2.5 \ \to\ 0.625 \ \to\ 0.15625$
Position halves each step and the loss is quadratic, so the loss drops 75 percent per update.

What just happened: Two applications of theta minus eta times the gradient cut the loss from 2.5 to 0.156 — a 75 percent drop per step. Nothing global was needed: the local slope at each point was enough to head to the minimum.

Now Break It

Try this: Crank the learning rate high — the path oscillates wildly and diverges to infinity.

Control: Learning rate slider (set to maximum)

What happens: Diverged! The learning rate is too high — each step overshoots the minimum and loss is increasing.

Where gradient descent is used

Gradient descent is the workhorse that fits nearly every modern model by iteratively stepping downhill on the loss surface. Training a neural network for image recognition, a language model that autocompletes your email, or a recommendation system all reduce to the same loop: compute how the loss changes with respect to each parameter, then nudge every parameter a small step in the direction that reduces error. Logistic regression models in credit scoring, deep networks in speech recognition, and the fine-tuning of large language models are all driven by gradient descent or one of its variants. Its popularity comes from scalability. Because it only needs the gradient at the current point, it can optimize models with billions of parameters without ever examining the entire loss surface.

A common misconception is that gradient descent finds the global minimum. For most non-convex problems, like deep networks, it finds a local minimum or a flat region, and that is usually good enough because many such solutions generalize well. A second pitfall is the learning rate. Too large a step and the loss oscillates or diverges, overshooting the valley; too small and training crawls or stalls in a shallow dip. People also confuse gradient descent with backpropagation: backpropagation is the efficient method for computing the gradient, while gradient descent is the rule that uses that gradient to update parameters. Finally, gradient descent needs a differentiable loss, which is why non-smooth objectives require special handling.

Frequently asked questions

What is gradient descent?

Gradient descent is an optimization method that minimizes a loss function by repeatedly moving parameters in the direction that most reduces the loss. The gradient points in the direction of steepest increase, so the algorithm steps in the opposite direction. Each iteration takes a small step, and over many iterations the parameters settle near a minimum.

What is the difference between gradient descent and backpropagation?

Backpropagation is an efficient algorithm for computing the gradient of the loss with respect to every parameter in a neural network. Gradient descent is the separate step that uses that gradient to update the parameters. In short, backpropagation computes the direction and gradient descent takes the step.

What is the difference between batch, stochastic, and mini-batch gradient descent?

Batch gradient descent computes the gradient over the entire dataset before each update, which is accurate but slow. Stochastic gradient descent updates after every single example, which is fast and noisy. Mini-batch gradient descent, the most common choice, uses small groups of examples to balance speed, stability, and hardware efficiency.

Why does gradient descent get stuck or diverge?

If the learning rate is too high, the steps overshoot the minimum and the loss can oscillate or blow up. If it is too low, progress is extremely slow and the algorithm can stall in a flat region or a shallow local minimum. Poorly scaled features and vanishing gradients in deep networks can also stall progress. Tuning the learning rate and normalizing inputs usually helps.

Does gradient descent always find the best solution?

Not necessarily. For convex problems it converges to the single global minimum, but most deep learning loss surfaces are non-convex with many local minima and saddle points. In practice gradient descent finds a good local minimum that generalizes well, which is usually sufficient. The global optimum is rarely required for strong performance.

What is a learning rate and how do I choose it?

The learning rate controls how big a step gradient descent takes on each update. A good value is large enough to make steady progress but small enough to avoid overshooting. Common practice is to try values on a logarithmic scale, watch the loss curve, and use schedules that reduce the rate over time or warm it up at the start.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.