Activation Functions

Neural NetworksIntermediate~6 min

Activation Functions — The nonlinearity that lets networks bend.

Without a nonlinear activation, stacking layers is pointless — it collapses back to one linear map. Activations like ReLU and sigmoid are what let networks learn curves and complex boundaries.

Function f(x)

Derivative f′(x)

Gradient after N layers (log scale) — click or drag to set the depth

Sigmoid
Tanh
ReLU
Leaky ReLU
GELU
Current x / depth

Activation controls

Data

Input regime

x near the middle of the curve, where every activation still has real slope.

Model

Function

Input x1.5

Stacked layers (depth)1

Playback

Step 0 / 11

Speed

Gradient alive
Fading
Vanished
↻

Step 0 of 11 — 1 stacked Sigmoid layer — each one multiplies the gradient by f′(1.5) = 0.149, leaving 0.1491

Break it

Sigmoid’s derivative peaks at 0.25, so ten layers can multiply the gradient down by a factor of a million. ReLU’s stays exactly 1 for positive inputs — and exactly 0 for negative ones, which kills the unit outright.

The idea in plain words

Without a nonlinear activation, stacking layers is pointless — the whole network collapses back to a single linear map. Activations like ReLU and sigmoid are what let a network learn curves and complex boundaries.

Each function’s derivative is what training actually uses. Sigmoid’s derivative flatlines at the extremes while ReLU’s stays alive — the visual root cause of vanishing gradients in deep networks.

Now, the math

Common activations and their behavior at the extremes:

\sigma(x) = \frac{1}{1+e^{-x}},\qquad \text{ReLU}(x) = \max(0, x)

$\sigma'(x)$: the sigmoid gradient — peaks at 0.25, vanishes for large |x|.
$\text{ReLU}'(x)$: exactly 1 for positive inputs — no shrinkage.

▸ Show the derivation

During backprop, the gradient is multiplied by the activation’s derivative at every layer. Sigmoid derivatives are at most 0.25, so through many layers the product shrinks exponentially toward zero. ReLU’s derivative of 1 preserves the signal — the main reason it replaced sigmoid in deep nets.

Trace it by hand

Push the SAME pre-activation z = 1.5 through sigmoid, tanh, and ReLU, then repeat with z = -2 to expose ReLU's dead zone. All values computed with the repo's activation core and rounded to 4 decimal places.

Sigmoid at z equal to 1.5
$\sigma(1.5) = \frac{1}{1 + e^{-1.5}} \approx 0.8176$
Squashed into the 0-to-1 range — useful when the output should read as a probability.
Tanh at the same z
$\tanh(1.5) \approx 0.9051$
Tanh maps to the range from -1 to 1 and is already near its ceiling at z equal to 1.5.
ReLU at the same z
$\text{ReLU}(1.5) = \max(0,\, 1.5) = 1.5$
For positive inputs ReLU is the identity — no squashing at all, the value passes straight through.
Now the same comparison at z equal to -2
$\sigma(-2) \approx 0.1192, \qquad \tanh(-2) \approx -0.9640, \qquad \text{ReLU}(-2) = \max(0,\, -2) = 0$
Sigmoid and tanh return small but informative values; ReLU clamps the input to exactly zero.
The derivatives tell the training story
$\text{ReLU}'(-2) = 0, \qquad \sigma'(-2) \approx 0.1050, \qquad \text{ReLU}'(1.5) = 1$
At z equal to -2 a ReLU neuron passes zero gradient — it is dead and cannot learn from that input. For positive z its derivative is exactly 1, which is why gradients survive deep ReLU stacks.

What just happened: The same z = 1.5 became 0.8176, 0.9051, or 1.5 depending only on the activation choice — and at z = -2, ReLU output and gradient are both exactly zero while sigmoid still passes a 0.105 gradient: the dead-zone trade-off in two numbers.

Now Break It

Try this: Sigmoid saturates at the extremes, its gradient vanishing — deep sigmoid nets barely learn.

Control: Activation selector (sigmoid) with large inputs

What happens: Vanishing gradient! Sigmoid flattens at the extremes — its gradient dies and deep layers stop learning.

Where activation functions is used

Activation functions are the nonlinear switches that let neural networks bend, curve, and carve up input space in ways a straight line never could, and the choice of activation has repeatedly reshaped what deep learning can achieve. Sigmoid and tanh dominated early networks, but the rise of ReLU (rectified linear unit) around 2010 to 2012 was a major factor in making very deep networks trainable, helping spark the modern deep learning era in computer vision. Today activations like GELU and SiLU power large language models, while softmax turns raw scores into class probabilities at the output of classifiers. Every time a network recognizes a face, translates a sentence, or ranks a search result, activation functions are quietly injecting the nonlinearity that makes it possible.

The central misconception is that activations are a minor detail; in truth, without a nonlinear activation, stacking layers is pointless because the composition of linear maps is still just one linear map, collapsing a deep network into a single-layer model. A second pitfall is the vanishing gradient problem: sigmoid and tanh saturate at their extremes, where their derivatives shrink toward zero, so gradients flowing back through many layers fade away and early layers barely learn. ReLU eases this by keeping a constant gradient for positive inputs, though it introduces the dead neuron problem, where units stuck at negative inputs output zero forever, which variants like leaky ReLU are designed to counter.

Frequently asked questions

What is an activation function?

An activation function is a nonlinear transformation applied to a neuron's weighted sum before it is passed on. It decides how strongly the neuron fires and, crucially, introduces nonlinearity so the network can model curved and complex relationships. Common choices include ReLU, sigmoid, tanh, and softmax.

Why do neural networks need nonlinear activations?

Because stacking linear layers without nonlinearity is mathematically equivalent to a single linear layer, no matter how many you add. A nonlinear activation between layers breaks that collapse and lets the network approximate complicated functions. Without it, a deep network could only ever draw straight decision boundaries.

What is ReLU and why is it popular?

ReLU, the rectified linear unit, outputs the input directly if it is positive and zero otherwise. It is cheap to compute and keeps a healthy, constant gradient for positive inputs, which helps deep networks train faster and avoid vanishing gradients. These properties made it the default activation for many modern architectures.

What are vanishing gradients?

Vanishing gradients occur when the gradients used to update early layers become extremely small, so those layers learn very slowly or not at all. Saturating activations like sigmoid and tanh contribute to this because their derivatives approach zero at the extremes. ReLU and careful initialization help reduce the problem.

When should I use softmax?

Softmax is used at the output of a multi-class classifier to turn raw scores into a probability distribution that sums to one. It is not typically used in hidden layers. For binary classification a single sigmoid output is the usual choice instead.

What is a dead neuron in ReLU?

A dead neuron is a ReLU unit that always receives negative input, so it outputs zero and its gradient is zero, meaning it never updates again. Large learning rates or poor initialization can push neurons into this state. Variants like leaky ReLU allow a small negative slope to keep such neurons alive.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.