Skip to content
ML Visualization

Multilayer Perceptron

Neural NetworksAdvanced~9 min

Multilayer Perceptron — Stack neurons into layers to learn nonlinear boundaries.

Stack neurons into layers and a network can carve any boundary at all. The multilayer perceptron is the workhorse feedforward network — the thing “deep learning” scaled up.

  • Class 0
  • Class 1
  • First-layer neuron lines
Hidden layer 1:
Loss vs iteration
0.8700Max loss on axis: 0.8699

MLP controls

Data
Dataset
20
1.0×
Add points as
Model
6
2
Activation
0.60
320
7

Same data, same architecture, different seed: sometimes the spiral is solved and sometimes it is not. The seed is a real hyperparameter, not a detail.

Playback
Step 0 / 30
Speed
  1. Random init
  2. Fitting
  3. Settled

Step 0 of 30 — epoch 0 of 320 — cross-entropy 0.870; 21 of 40 points still on the wrong side

Break it

One neuron can only draw one line, so the boundary stays straight no matter how long it trains. Too large a learning rate overshoots instead — watch the loss curve stop falling smoothly.

The idea in plain words

Stack neurons into layers and a network can carve any boundary at all. Each hidden neuron learns its own line; together they combine into a complex curved boundary that a single perceptron never could — solving circles, XOR, even spirals.

Watch the boundary reshape as the network trains, and click a first-layer neuron to see the line it learned. Too few neurons can’t bend enough to separate a hard dataset; the loss curve stalls high.

Now, the math

An MLP composes layers of nonlinear transformations:

y^=f ⁣(W(2)f ⁣(W(1)x+b(1))+b(2))\hat{y} = f\!\left(W^{(2)} f\!\left(W^{(1)} x + b^{(1)}\right) + b^{(2)}\right)
ff
the nonlinear activation — what makes stacking meaningful.
W(1),W(2)W^{(1)}, W^{(2)}
the weight matrices of the hidden and output layers.
▸ Show the derivation

The universal approximation theorem says a single hidden layer with enough neurons can approximate any continuous function — but “enough” can be huge. Extra depth lets the network build features hierarchically, solving hard shapes like the spiral with far fewer neurons per layer. It’s trained by backpropagation.

Trace it by hand

One fixed 2-2-1 network, ReLU hidden layer, sigmoid output, evaluated at two inputs. Hidden weights: both rows (1, 1) with biases -0.5 and -1.5, so the two hidden neurons share a direction but fire at different thresholds. Output weights (6, -16), bias -2. Outputs rounded to 4 decimal places, computed with the repo's MLP core.

  1. Hidden layer for the input (1, 0)

    f ⁣(W(1)x+b(1))=(max⁡(0, 1+0−0.5)max⁡(0, 1+0−1.5))=(0.50)f\!\left(W^{(1)} x + b^{(1)}\right) = \begin{pmatrix} \max(0,\, 1 + 0 - 0.5) \\ \max(0,\, 1 + 0 - 1.5) \end{pmatrix} = \begin{pmatrix} 0.5 \\ 0 \end{pmatrix}

    Both hidden neurons measure the same quantity x1 plus x2, but only the low-threshold neuron fires here.

  2. Output for (1, 0)

    y^=σ ⁣(6(0.5)−16(0)−2)=σ(1)≈0.7311\hat{y} = \sigma\!\left(6(0.5) - 16(0) - 2\right) = \sigma(1) \approx 0.7311

    Above 0.5, so the point (1, 0) is classified as class 1.

  3. Hidden layer for the input (1, 1)

    f ⁣(W(1)x+b(1))=(max⁡(0, 1+1−0.5)max⁡(0, 1+1−1.5))=(1.50.5)f\!\left(W^{(1)} x + b^{(1)}\right) = \begin{pmatrix} \max(0,\, 1 + 1 - 0.5) \\ \max(0,\, 1 + 1 - 1.5) \end{pmatrix} = \begin{pmatrix} 1.5 \\ 0.5 \end{pmatrix}

    Moving further along the same diagonal wakes the second, high-threshold neuron — and its output weight is strongly negative.

  4. Output for (1, 1) lands on the other side

    y^=σ ⁣(6(1.5)−16(0.5)−2)=σ(9−8−2)=σ(−1)≈0.2689\hat{y} = \sigma\!\left(6(1.5) - 16(0.5) - 2\right) = \sigma(9 - 8 - 2) = \sigma(-1) \approx 0.2689

    Below 0.5: class 0. The second neuron's -16 weight overpowers the first, folding the boundary back.

  5. The full picture is XOR-shaped

    y^(0,0)=σ(−2)≈0.1192,y^(0,1)=σ(1)≈0.7311\hat{y}(0, 0) = \sigma(-2) \approx 0.1192, \qquad \hat{y}(0, 1) = \sigma(1) \approx 0.7311

    So the four corners score 0.12, 0.73, 0.73, 0.27 — low, high, high, low. No single straight line can produce that pattern.

What just happened: Two hidden neurons watching the same direction with different thresholds let the output layer subtract one stripe from another: (1, 0) scores 0.7311 while (1, 1), further along the very same diagonal, scores 0.2689 — a bent, XOR-solving boundary no single perceptron can draw.

Now Break It

Try this: Too few hidden units can’t bend enough to separate a spiral; too many overfit the noise.

Control: Hidden units slider (set to 1)

What happens: Not enough capacity! With one hidden unit the network can only draw a line — it can’t separate the spiral.

Where multilayer perceptron is used

The multilayer perceptron (MLP) stacks neurons into successive layers so that hidden layers learn intermediate features and the whole network can carve nonlinear decision boundaries that a single neuron never could. This architecture is the foundation of practical deep learning: the universal approximation theorem shows that an MLP with even one sufficiently wide hidden layer can approximate any continuous function to arbitrary precision. MLPs classify handwritten digits, estimate house prices, detect fraud, and appear as the feed-forward blocks inside transformers that power modern language models. Once researchers paired the multilayer structure with backpropagation in the 1980s, the layered network stopped being a curiosity and became the engine behind a huge share of applied machine learning.

A common misconception is that the universal approximation theorem means a shallow MLP is always enough; in practice a single hidden layer might need an impractically enormous number of neurons, whereas adding depth lets the network reuse features and learn the same function far more efficiently. Another pitfall is forgetting that the hidden layers must use nonlinear activations, because without them the multilayer structure collapses back into a plain linear model regardless of how many layers you stack. Beginners also overbuild, assuming more layers always help, but excessively deep or wide MLPs overfit small datasets and can be harder to train without regularization, normalization, and enough data.

Frequently asked questions

What is a multilayer perceptron?
A multilayer perceptron is a neural network made of an input layer, one or more hidden layers, and an output layer, with each neuron connected to those in the next layer. The hidden layers apply nonlinear activations, allowing the network to learn nonlinear relationships. It is the classic fully connected, feed-forward neural network.
How is an MLP different from a single perceptron?
A single perceptron can only draw one straight decision boundary, so it is limited to linearly separable problems. An MLP stacks layers of neurons with nonlinear activations, letting it combine many boundaries into complex, curved regions. This is what allows it to solve problems like XOR that defeat a single perceptron.
What is a hidden layer?
A hidden layer is any layer of neurons between the input and output layers. Its neurons learn intermediate representations, or features, that are not directly observed in the data or the final prediction. Stacking hidden layers lets the network build increasingly abstract features.
What is the universal approximation theorem?
It states that a feed-forward network with at least one hidden layer and a suitable nonlinear activation can approximate any continuous function to any desired accuracy, given enough neurons. It explains why MLPs are so expressive in principle. It does not, however, tell you how to train the network or how many neurons you will realistically need.
Why use deep networks instead of one wide layer?
Although one wide hidden layer can in theory approximate any function, it may require an impractically huge number of neurons. Deeper networks reuse and compose features across layers, often representing the same function with far fewer parameters. Depth also tends to generalize better on complex, structured data.
How does an MLP make a prediction?
It passes the input through each layer in turn, with every neuron computing a weighted sum, adding a bias, and applying an activation function, until the output layer produces the result. This left-to-right process is called forward propagation. Training then adjusts the weights using backpropagation.
← Back to all visualizationsContinue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated .