Loss Functions

FoundationsBeginner~5 min

Loss Functions — A loss function quantifies the disagreement between a model’s predictions and the true values as a single number. Training minimizes this number; the choice of loss (such as MSE or MAE) determines how errors are penalized.

How do you measure how wrong your model is? That’s what a loss function does — it takes the gap between prediction and reality and turns it into a single number. Smaller is better.

Data points
Prediction (Squared (MSE))
Absolute (MAE) would sit here
Residual (width = share of loss)

Loss for one point, against its residual r (click a dot to pin that point)

Loss controls

Data

Data shape

A well-behaved cloud around one value. Every loss agrees on roughly where to sit.

Model

Loss

Compare against

Huber δ (where quadratic becomes linear)1.0

Squared (MSE)40.830

Absolute (MAE)6.312

Playback

Prediction0.00

Step 0 / 120

Speed

Under-predicting
At the minimum
Over-predicting
↻

Step 0 of 120 — prediction 0.00 — largest |residual| 7.25, Squared (MSE) 40.830 (its own best is 6.30), Absolute (MAE) 6.312 (best 6.80)

Break it

Squared (MSE) would sit at 6.30; Absolute (MAE) would sit at 6.80. They are 0.50 apart.

The idea in plain words

A loss function scores how wrong the model is, as a single number to minimize. Drag the prediction line and watch both scores move. Then drag one point far away: mean squared error explodes while mean absolute error barely flinches — because squaring turns a big miss into a huge one.

Once you can measure “wrong,” gradient descent minimizes it, and linear regression is the classic model built on squared-error loss.

Now, the math

Two common regression losses over n points:

\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - \hat{y}_i\bigr)^2

\mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n}\bigl|y_i - \hat{y}_i\bigr|

$y_i$: the actual value of point i.
$\hat{y}_i$: the model’s prediction for point i.
$n$: the number of points.
$\sum$: add up over all points.

▸ Show the derivation

Because MSE squares each error, an error of 6 contributes 36 while an error of 1 contributes 1 — so a single far-off outlier dominates the total. MAE grows only linearly, making it far more robust to outliers. The right choice depends on whether large errors should be punished disproportionately.

Trace it by hand

Three points with actual values y = (5, 4, 8) and a fixed prediction line giving y-hat = (4, 6, 5). We compute MSE and MAE exactly, then drag the third point from 8 out to 14 and recompute. All values are exact fractions rounded to 2 decimals.

Compute the three errors
$y_i - \hat{y}_i = (5 - 4,\ 4 - 6,\ 8 - 5) = (1,\ -2,\ 3)$
Both losses start from these same residuals — they differ only in how they punish them.
Mean squared error
$\mathrm{MSE} = \frac{1^2 + (-2)^2 + 3^2}{3} = \frac{14}{3} \approx 4.67$
Squaring already makes the error of 3 contribute 9, nine times what the error of 1 does.
Mean absolute error
$\mathrm{MAE} = \frac{|1| + |-2| + |3|}{3} = \frac{6}{3} = 2$
Under MAE the error of 3 contributes only three times what the error of 1 does.
Drag one point into an outlier
$y_3: 8 \to 14 \quad\Rightarrow\quad y_i - \hat{y}_i = (1,\ -2,\ 9)$
Only one residual changed, from 3 to 9.
Recompute both losses
$\mathrm{MSE} = \frac{1 + 4 + 81}{3} \approx 28.67, \qquad \mathrm{MAE} = \frac{1 + 2 + 9}{3} = 4$
The single squared term 81 is now 94 percent of the entire MSE sum.

What just happened: One moved point multiplied MSE by 6.1 (4.67 to 28.67) but MAE only by 2 (2 to 4). Squaring turns a big miss into a huge one, so a single outlier can dominate MSE while MAE grows only linearly.

Now Break It

Try this: One extreme outlier explodes MSE but barely moves MAE — showing why loss function choice matters.

Control: Drag an outlier point far from the cluster

What happens: MSE is dominated by the outlier! One extreme point has hijacked the entire loss.

Where loss functions is used

A loss function measures how wrong a prediction is, and the choice of loss quietly shapes almost every deployed system. Regression tasks such as forecasting energy demand or estimating delivery times often use mean squared error, which punishes large misses heavily, or mean absolute error when you want robustness to outliers. Classifiers behind spam filters, medical diagnosis, and content moderation typically use cross-entropy, which rewards confident correct answers and sharply penalizes confident mistakes. Object detectors in self-driving cars combine a localization loss for bounding boxes with a classification loss for object type. Ranking systems for search and recommendations use pairwise or listwise losses that care about order rather than exact scores. In every case the loss function is the numerical definition of what good performance means.

A common misconception is that lower loss always means a better product. Loss is a proxy: minimizing it drives learning, but the metric you actually care about, such as revenue, click-through, or diagnostic recall, may diverge from it. A model with lower cross-entropy can still have worse accuracy at your chosen threshold. A second pitfall is picking a loss function that ignores the real cost structure of mistakes. If a false negative in cancer screening is far worse than a false positive, an unweighted loss that treats both errors equally is the wrong objective; class weighting or a custom loss aligns training with real-world stakes. Always ask whether your loss function encodes the errors you truly care about.

Frequently asked questions

What is a loss function in machine learning?

A loss function is a formula that assigns a single number to how far a model's prediction is from the correct answer, with larger values meaning worse predictions. Training works by adjusting the model's parameters to make this number as small as possible across the data. The loss function is therefore the mathematical definition of what the model is trying to achieve.

What is the difference between a loss function and a cost function?

Loss usually refers to the error on a single example, while cost or objective often refers to the average loss over a whole batch or dataset. In everyday use the terms are frequently interchanged. What matters is that both measure prediction error and both are minimized during training.

When should I use cross-entropy versus mean squared error?

Use cross-entropy for classification, where the model outputs probabilities over classes, because it strongly penalizes confident wrong answers and pairs well with softmax or sigmoid outputs. Use mean squared error for regression, where the target is a continuous number. Applying mean squared error to a classifier often produces weak gradients and slower learning.

Why does my loss go down but accuracy not improve?

Loss and accuracy measure different things. Loss reflects the confidence and calibration of every prediction, while accuracy only counts whether the final label is right at a chosen threshold. A model can become better calibrated, lowering loss, without crossing the threshold on enough examples to change accuracy. This gap is normal and often temporary.

How do I handle imbalanced classes in the loss function?

When one class is rare, an unweighted loss lets the model achieve low error by ignoring the minority class. You can assign higher weight to the rare class in the loss, resample the data, or use a specialized loss such as focal loss that focuses on hard examples. The right choice depends on the real cost of each type of mistake.

Can I design my own loss function?

Yes. Custom loss functions let you encode the specific costs of different mistakes, such as penalizing false negatives more heavily than false positives. As long as the loss is differentiable with respect to the model's outputs, gradient-based training can minimize it. Custom losses are common in ranking, detection, and cost-sensitive applications.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.