Backpropagation

Neural NetworksAdvanced~10 min

Backpropagation — Propagate error gradients backward to update every weight.

Backpropagation is how networks learn. It sends the output error backward through the layers using the chain rule, computing how much each weight contributed to the mistake, then nudges every weight to do better.

Forward pass — activations light up

Training set — click a point to trace it

Loss vs iteration

Class 0
Class 1
Traced example
Gradient edge

Gradient magnitude by layer (output → input)

0.003

0.022

0.116

0.476

Backprop controls

Data

Dataset

Points per class10

Spread1.0×

Traced example: (2.2, 7.1), target 0. Click any point on the plot to switch to it.

Model

Hidden layers (depth)3

Neurons per hidden layer4

Activation

Learning rate0.60

Init seed7

Playback

Training

Step 0 / 15

Speed

Random init
Fitting
Settled
↻

Step 0 of 15 — after 0 epochs (1 full-batch updates) — loss 0.686, started at 0.686. The pass below is the update this net would apply next.

One update, step by step

Step 0 / 8

Speed

Forward pass
Compare to target
Backward pass
↻

Step 0 of 8 — the chosen example enters as x = (-0.92, 0.70) — nothing is known about the error yet

Break it

Too large a step and each update overshoots, so the loss climbs instead of falling. Too many sigmoids and the gradient is multiplied down to nothing before it reaches the first layer — the bars on the left go flat while the ones on the right stay tall.

The idea in plain words

Watch the error flow back edge by edge. Make the network deep with sigmoids and the early-layer gradients dim to almost nothing — the vanishing-gradient problem, visible in the shrinking bars. Switch to ReLU to revive them.

Now, the math

The gradient for each weight is a local product, assembled by the chain rule:

\frac{\partial L}{\partial w^{(l)}_{ij}} = \delta^{(l)}_j \, a^{(l-1)}_i

$\delta^{(l)}_j$: the error signal at neuron j in layer l, propagated from the output.
$a^{(l-1)}_i$: the activation that fed into that weight on the forward pass.

▸ Show the derivation

Each δ is the next layer’s δ times the local weight times the activation derivative. Because those derivatives (≤ 0.25 for sigmoid) multiply at every layer, the error signal shrinks exponentially as it travels back, so early layers of deep sigmoid networks barely update. ReLU’s derivative of 1 keeps the signal alive.

Trace it by hand

The smallest network that can show the chain rule: input x → weight w₁ → sigmoid → weight w₂ → output ŷ, with squared loss L = ½(ŷ − y)². Concrete numbers: x = 1, w₁ = 0.5, w₂ = 0.8, target y = 1, learning rate 0.5. Values rounded to 4 decimal places.

Step 1 — forward pass: run the numbers left to right

z_1 = w_1 x = 0.5, \qquad h = \sigma(z_1) = 0.6225, \qquad \hat{y} = w_2\,h = 0.4980

L = \tfrac{1}{2}(\hat{y} - y)^2 = \tfrac{1}{2}(0.4980 - 1)^2 = 0.1260

This is exactly one run of forward propagation; every intermediate value (z₁, h, ŷ) gets cached for the backward pass.

Step 2 — backward to w₂: two local derivatives

\frac{\partial L}{\partial \hat{y}} = \hat{y} - y = -0.5020, \qquad \frac{\partial L}{\partial w_2} = (\hat{y} - y)\,h = -0.5020 \cdot 0.6225 = -0.3125

Step 3 — backward to w₁: the chain grows one link per layer

\frac{\partial L}{\partial w_1} = (\hat{y} - y)\cdot w_2 \cdot \sigma'(z_1) \cdot x = -0.5020 \cdot 0.8 \cdot 0.2350 \cdot 1 = -0.0944

σ′(z₁) = h(1 − h) = 0.6225 · 0.3775 = 0.2350. That factor can never exceed 0.25 — stack ten sigmoid layers and the gradient shrinks by up to 0.25¹⁰, the vanishing-gradient problem.

Step 4 — update both weights with gradient descent

w_2 \leftarrow 0.8 - 0.5\,(-0.3125) = 0.9562, \qquad w_1 \leftarrow 0.5 - 0.5\,(-0.0944) = 0.5472

Rerun the forward pass with the new weights and the loss drops from 0.1260 to 0.0777 — one step of gradient descent, routed through the network by the chain rule.

What just happened: backpropagation never differentiated the whole network at once — it multiplied cheap local derivatives (ŷ − y, then w₂, then σ′, then x) backward along the wire, reusing the values cached on the forward pass. Both gradients came out negative, so both weights rose, and one update cut the loss by almost 40%.

Now Break It

Try this: In a deep sigmoid net the backward gradients shrink toward zero — early layers barely update.

Control: Depth slider with sigmoid activations

What happens: Vanishing gradients! In a deep sigmoid net the backward signal shrinks to nothing — early layers stop learning.

Where backpropagation is used

Backpropagation is the algorithm that makes training deep networks feasible: it efficiently computes how every weight in the network contributed to the final error, so gradient descent knows which direction to nudge each parameter. Popularized for neural networks by Rumelhart, Hinton, and Williams in 1986, it turned multilayer networks from theoretical objects into trainable systems and underlies essentially all modern deep learning, from image classifiers to large language models. Its key trick is reusing computation: rather than recalculating gradients independently for millions of parameters, it propagates error signals backward one layer at a time, so the cost of computing all gradients is comparable to a single forward pass. This efficiency is what makes training networks with billions of parameters practical.

The biggest misconception is that backpropagation is some mysterious learning force; it is simply the chain rule from calculus applied systematically across a network's layers, computing derivatives of the loss with respect to each weight. Another misconception is that backprop trains the network by itself, but it only computes gradients; a separate optimizer such as stochastic gradient descent or Adam uses those gradients to actually update the weights. A practical pitfall is that backpropagation inherits the vanishing and exploding gradient problems, since repeatedly multiplying many small or large derivatives through deep networks can shrink or blow up the signal, which motivates careful initialization, normalization, and architectures like residual connections.

Frequently asked questions

What is backpropagation?

Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in a neural network by propagating error signals backward from the output to the input. These gradients tell an optimizer how to adjust each weight to reduce the error. It is the core method for training deep networks.

Is backpropagation just the chain rule?

Essentially, yes. Backpropagation is a systematic and efficient application of the chain rule of calculus, computing how the loss changes with respect to each parameter layer by layer. Its cleverness lies in reusing intermediate results so that all gradients are found in roughly the cost of one extra forward pass rather than recomputing from scratch.

Does backpropagation update the weights?

No, backpropagation only computes the gradients. A separate optimization algorithm, such as stochastic gradient descent or Adam, uses those gradients to actually update the weights. The two steps work together but are conceptually distinct.

Why does backpropagation need the forward pass first?

The backward pass relies on values computed during the forward pass, including each layer's outputs and the final loss. Without those cached activations, the chain rule cannot be evaluated correctly. That is why every training step runs a forward pass, then a backward pass.

What are exploding and vanishing gradients in backpropagation?

As gradients are propagated back through many layers, repeated multiplication can make them shrink toward zero (vanishing) or grow uncontrollably (exploding). Vanishing gradients stall learning in early layers, while exploding gradients cause unstable updates. Techniques like good initialization, normalization, gradient clipping, and residual connections help manage them.

Who invented backpropagation?

The underlying ideas appeared in several fields over the years, but its popularization for training neural networks is credited to a 1986 paper by Rumelhart, Hinton, and Williams. Their work showed that multilayer networks could learn useful internal representations. This helped revive interest in neural networks.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.