Softmax & Multiclass

ClassificationIntermediate~6 min

Softmax & Multiclass — Softmax converts a vector of raw class scores (logits) into probabilities that sum to 1 by exponentiating and normalizing. A temperature parameter sharpens it toward a hard argmax or flattens it toward uniform.

Three or more classes carve the space into colored regions. A bar panel shows raw scores becoming probabilities — and a temperature dial slides softmax from a confident winner-take-all to a flat shrug.

Class 0
Class 1
Class 2
Prototype μ (drag me)

Logits at the query (raw scores, can be negative)

-6.50

-8.11

-8.08

Softmax controls

Data

Classes k3

Points per class24

Spread1.0×

Model

Temperature T1.0

Drag any μ marker on the plot to move a class prototype by hand — the regions re-carve live and every logit changes with it.

Playback

Step 0 / 2

Speed

Logits
Exponentiate
Normalize
↻

Step 0 of 2 — at (5.0, 5.0) the scores are z0 = -6.50, z1 = -8.11, z2 = -8.08 — negative squared distances, not probabilities

Break it

Neither extreme changes which class wins — the argmax is untouched by T. What changes is how much the model claims to know, which is the number people actually quote.

The idea in plain words

Softmax turns a handful of raw class scores into probabilities that sum to one, by exponentiating and normalizing. It’s the multiclass generalization of the sigmoid, and it powers the output layer of nearly every neural network classifier.

A temperature dial controls how peaked it is. Near zero it becomes a hard argmax — one class takes everything — so tiny changes flip the winner. Turn it up and the probabilities flatten toward a uniform shrug.

Now, the math

Softmax with temperature T:

\text{softmax}(z_i) = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}

$z_i$: the raw score (logit) for class i.
$T$: temperature — low sharpens toward argmax, high flattens toward uniform.

▸ Show the derivation

Dividing logits by T before exponentiating rescales the gaps between them. As T → 0 the largest logit dominates completely (probability 1); as T → ∞ all scaled logits approach 0 and the probabilities become equal. This same knob is used to calibrate confidence and to soften targets in model distillation.

Trace it by hand

Three classes with logits z = (2.0, 1.0, 0.1), starting at temperature T = 1. Exponentials and probabilities rounded to 3 decimal places.

Exponentiate each logit
$e^{2.0} = 7.389, \qquad e^{1.0} = 2.718, \qquad e^{0.1} = 1.105, \qquad \textstyle\sum_j e^{z_j} = 11.213$
Normalize into probabilities
$\text{softmax}(z) = \left(\tfrac{7.389}{11.213},\ \tfrac{2.718}{11.213},\ \tfrac{1.105}{11.213}\right) = (0.659,\ 0.242,\ 0.099)$
A logit gap of 1.0 became a probability ratio of e, about 2.72 — softmax works on differences, not absolute values.
Sharpen with T = 0.5
$T = 0.5: \; z/T = (4.0,\ 2.0,\ 0.2) \;\Rightarrow\; p = (0.864,\ 0.117,\ 0.019)$
Halving T doubles every logit gap before exponentiating, so the leader pulls away.
Flatten with T = 5
$T = 5: \; z/T = (0.40,\ 0.20,\ 0.02) \;\Rightarrow\; p = (0.400,\ 0.327,\ 0.273)$
The ranking never changes with temperature — only the confidence does.

What just happened: The same logits (2.0, 1.0, 0.1) produced a 66 percent, an 86 percent, or a 40 percent favorite purely by rescaling with temperature — softmax preserves the winner but T sets how much the winner takes.

Now Break It

Try this: Temperature near zero makes the classifier brittle — the winner flips on tiny changes.

Control: Temperature slider (set very low)

What happens: Winner-take-all! At near-zero temperature softmax becomes a hard argmax that flips on the slightest change.

Where softmax & multiclass is used

The softmax function converts a vector of raw model scores, called logits, into a set of probabilities that are all positive and sum to one, which is exactly what you need to pick among several mutually exclusive classes. It is the standard output layer for multiclass classifiers, from the final layer of neural networks recognizing thousands of object categories to multinomial logistic regression. Every image classifier that reports a most-likely label with a confidence, every language model choosing the next token from a vocabulary, and every speech recognizer selecting among phonemes ends in a softmax. Because the outputs form a proper probability distribution, they plug directly into the cross-entropy loss used to train these models and into downstream decisions like ranking or thresholding.

A frequent misconception is that softmax probabilities measure true confidence or real-world likelihood; they are relative scores that always sum to one, so a model can be completely wrong yet assign 99 percent to the incorrect class, especially on inputs unlike anything it trained on. Another pitfall is confusing softmax with the sigmoid: softmax is for choosing one class among several that compete, whereas independent sigmoids suit multi-label problems where several labels can be true at once. People also forget that softmax is shift-invariant, meaning adding a constant to every logit leaves the output unchanged, which is why stable implementations subtract the maximum logit first to avoid numerical overflow. Finally, a temperature parameter can sharpen or soften the distribution without changing which class ranks highest.

Frequently asked questions

How is softmax different from the sigmoid function?

Sigmoid maps a single score to one independent probability and is used for binary or multi-label problems where labels do not compete. Softmax takes a whole vector of scores and turns them into competing probabilities that sum to one, which suits single-label multiclass problems. In fact, softmax with two classes reduces to the sigmoid.

Do softmax outputs represent true confidence?

Not reliably. They are normalized scores that always sum to one, so the model reports a highest class even for inputs it has never seen anything like. Modern networks are often overconfident, so a high softmax value does not guarantee correctness, and calibration techniques exist to make the numbers more trustworthy.

What are logits?

Logits are the raw, unnormalized scores a model produces before the softmax is applied. They can be any real number, positive or negative, and have no probabilistic meaning on their own. Softmax exponentiates and normalizes them into a valid probability distribution.

Why do stable implementations subtract the maximum logit?

Softmax involves exponentials, and large logits can overflow to infinity in floating-point arithmetic. Because softmax is unchanged when you subtract the same constant from every logit, subtracting the maximum keeps the largest exponent at zero and avoids overflow. This produces identical results with better numerical stability.

What does the temperature parameter do?

Temperature divides the logits before softmax. A high temperature makes the distribution more uniform and less certain, while a low temperature sharpens it toward the top class. It is widely used to control randomness when sampling from language models and to soften targets in knowledge distillation.

How does softmax connect to cross-entropy loss?

Softmax turns scores into a probability for each class, and cross-entropy then measures how far that predicted distribution is from the true label. The two are paired because their combined gradient is simple and numerically well behaved. Training a multiclass model almost always minimizes this softmax cross-entropy.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.