Forward Propagation

Neural NetworksAdvanced~7 min

Forward Propagation — Push inputs through the layers to compute a prediction.

Forward propagation is how a network makes a prediction: feed the inputs into the first layer, pass the outputs to the next, and repeat until the final layer produces an answer. Just matrix multiplies and activations, layer by layer.

Activation of every unit, layer by layer — hover a dot, click a column

Positive activation
Negative activation
Wavefront

Input space (drag the point)

Drag the input point (or use the x₁ / x₂ sliders) and watch the activations ripple forward. Every column of dots is one layer’s worth of numbers — the whole forward pass is just that matrix multiply, repeated.

Forward-pass controls

Data

Input vector

One positive, one negative input — the two weight columns pull against each other.

Input x₁0.8

Input x₂-0.5

Model

Hidden layers3

Neurons per hidden layer5

Hidden activation

Weight-init scale1.0×

Init seed7

The output layer is always a single sigmoid unit — the hidden activation is what you are choosing here.

Playback

Step 0 / 4

Speed

Input
Hidden layers
Output
↻

Step 0 of 4 — the input vector (0.80, -0.50) is loaded into layer 0 — nothing has been multiplied yet

Largest |a| at this layer0.800

Break it

Play it to the end either way: the dots pin to the saturation limits or collapse onto zero, and by the output layer the input has stopped mattering at all.

The idea in plain words

Forward propagation is how a network makes a prediction: feed the inputs into the first layer, pass its outputs to the next, and repeat until the final layer produces an answer. Just matrix multiplies and activations, layer by layer.

Step the wavefront and watch each neuron light up with its value. Poor weight initialization makes those values explode toward the saturation limits or vanish toward zero as they propagate — which is why initialization schemes matter.

Now, the math

Each layer transforms the previous layer’s activations:

a^{(l)} = f\!\left(W^{(l)} a^{(l-1)} + b^{(l)}\right)

$W^{(l)}$: the weight matrix of layer l.
$a^{(l-1)}$: activations arriving from the previous layer.

▸ Show the derivation

If weights are too large, repeated multiplication amplifies the activations layer after layer until they saturate; too small, and they decay toward zero. Initialization schemes (Xavier, He) scale the weights by the layer width so activation variance stays roughly constant with depth.

Trace it by hand

A 2-2-1 network with ReLU hidden units and a sigmoid output. Input a(0) = (1, 2). Hidden weights: row one (0.5, -0.4), row two (0.3, 0.8), biases (0.1, -0.2). Output weights (0.6, -0.5), bias 0.2. Computed with the repo's MLP core; final value rounded to 4 decimal places.

Hidden layer pre-activations
$z^{(1)} = W^{(1)} a^{(0)} + b^{(1)} = \begin{pmatrix} 0.5(1) - 0.4(2) + 0.1 \\ 0.3(1) + 0.8(2) - 0.2 \end{pmatrix} = \begin{pmatrix} -0.2 \\ 1.7 \end{pmatrix}$
Each hidden neuron takes its own weighted view of the same input — one lands negative, one strongly positive.
Apply the hidden activation
$a^{(1)} = \text{ReLU}\!\left(z^{(1)}\right) = \begin{pmatrix} \max(0,\, -0.2) \\ \max(0,\, 1.7) \end{pmatrix} = \begin{pmatrix} 0 \\ 1.7 \end{pmatrix}$
ReLU silences hidden neuron one entirely — only neuron two's signal travels on to the next layer.
Output layer pre-activation
$z^{(2)} = W^{(2)} a^{(1)} + b^{(2)} = 0.6(0) + (-0.5)(1.7) + 0.2 = -0.65$
The silenced neuron contributes exactly nothing: its weight 0.6 is multiplied by 0.
Sigmoid output gives the prediction
$a^{(2)} = \sigma(-0.65) = \frac{1}{1 + e^{0.65}} \approx 0.3430$
The network predicts class 1 with probability about 0.34 — so it leans toward class 0 for this input.

What just happened: The prediction 0.3430 came from nothing but two rounds of multiply-add-activate — and the trace shows ReLU zeroing one hidden neuron, so the whole output for this input flowed through a single active path.

Now Break It

Try this: Poor weight initialization makes activations explode or vanish as they propagate forward.

Control: Weight init scale slider (set very high or low)

What happens: Activations exploding! Bad initialization makes values blow up as they flow through the layers.

Where forward propagation is used

Forward propagation is the process of pushing an input through a network layer by layer to produce a prediction, and it is the step that runs every time a trained model is actually used, whether classifying a photo, transcribing speech, or generating the next word in a sentence. Each layer computes a weighted sum, adds a bias, and applies an activation, feeding its output to the next layer until the final layer emits scores or probabilities. This forward pass is what people mean by inference or serving a model in production, and its efficiency directly determines latency and cost. Understanding the exact sequence of matrix multiplications and activations is essential for reasoning about a network's speed, memory use, and behavior.

A common misconception is that forward propagation and training are the same thing; in fact the forward pass only computes outputs, while learning also requires a backward pass to compute gradients and update weights. Another pitfall is ignoring the shapes of the tensors flowing through: a mismatch between a layer's weight matrix dimensions and its input is one of the most common bugs, producing errors or silently wrong results. Beginners also forget that during training the forward pass must cache intermediate values, the pre-activation sums and layer outputs, because backpropagation needs them; skipping this caching is why pure inference can run with less memory than training.

Frequently asked questions

What is forward propagation?

Forward propagation is the process of feeding an input through a neural network, layer by layer, to compute the output prediction. Each layer applies its weights, bias, and activation function, then passes the result forward. It is the computation performed whenever the model makes a prediction.

What is the difference between the forward pass and the backward pass?

The forward pass sends inputs through the network to produce a prediction and measure the loss. The backward pass then propagates gradients of that loss back through the layers to determine how each weight should change. Forward computes outputs; backward computes how to improve them.

Is forward propagation the same as inference?

Essentially yes. Inference, or using a trained model to make predictions, is exactly a forward pass through the network. The difference is that during inference you usually do not need to store intermediate values or run a backward pass, so it can be more memory efficient.

What happens in each layer during forward propagation?

Each layer takes the outputs of the previous layer, multiplies them by its weight matrix, adds a bias vector, and applies an activation function. The result becomes the input to the next layer. This repeats until the output layer produces the final scores or probabilities.

Why does training cache values during the forward pass?

Backpropagation needs the intermediate quantities computed during the forward pass, such as each layer's pre-activation sums and outputs, to calculate gradients accurately. So during training the forward pass stores these values for later use. Pure inference can skip this storage, which is one reason it uses less memory.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.