Naive Bayes

ClassificationIntermediate~7 min

Naive Bayes — Classify using Bayes’ rule and a strong independence assumption.

Naive Bayes flips the question around with Bayes’ rule: instead of asking “what class is this?” it asks “which class most likely produced these features?” The “naive” part assumes every feature is independent.

Class 0
Class 1
NB Gaussian (2σ)
Query point

p(x | class)0.028 / 0.184

p(y | class)0.230 / 0.125

Naive Bayes controls

Data

Dataset

Points per class30

Spread1.0×

Feature correlation ρ0.10

Add points as

Model

Prior P(class 1)0.50

The fit measured 0.50 / 0.50 from the data. Overriding it moves the boundary without touching a single likelihood.

Misclassified0%

Playback

Step 0 / 3

Speed

Prior
Likelihood of x
Likelihood of y
Normalize
↻

Step 0 of 3 — before looking at the point: P(A) = 0.50, P(B) = 0.50

Drag the ringed query point to re-run the trace anywhere on the plane.

Break it

The ellipses stay axis-aligned no matter how the data tilts — that is the “naive” independence assumption. A lopsided prior does the other kind of damage: evidence has to overcome it before it counts.

The idea in plain words

Naive Bayes flips classification around with Bayes’ rule: instead of “what class is this point?” it asks “which class most likely produced these features?” The “naive” part assumes the features are independent, so each class becomes an axis-aligned Gaussian blob.

That assumption is a shortcut. Correlate the features and the true clouds tilt, but Naive Bayes stubbornly keeps its ellipses square to the axes — and starts misclassifying exactly where the tilt matters most.

Now, the math

Bayes’ rule with the independence assumption factorizes the likelihood:

P(y \mid x) \propto P(y) \prod_i P(x_i \mid y)

$P(y)$: the class prior — how common the class is.
$P(x_i\mid y)$: each feature’s likelihood, modeled as a 1-D Gaussian.
$\prod_i$: the naive step: multiply as if features were independent.

▸ Show the derivation

Multiplying per-feature Gaussians is equivalent to a single Gaussian with a diagonal covariance — an ellipse whose axes are parallel to the coordinate axes. Real correlated data has off-diagonal covariance (a tilted ellipse), which the model cannot represent, so its posterior is skewed.

Trace it by hand

Two classes of 4 points each: class 0 has mean (2, 2) and class 1 has mean (5, 5), each with variance 0.5 on both axes and prior 0.5 (fit by the Gaussian Naive Bayes core). Classify the test point x = (4, 4). Likelihoods rounded to 4 decimal places.

Fit the per-class Gaussians
$\mu_0 = (2, 2),\quad \mu_1 = (5, 5),\quad \sigma^2 = 0.5 \text{ per axis},\quad P(y{=}0) = P(y{=}1) = 0.5$
Score class 0: prior times likelihoods
$P(y{=}0)\prod_i P(x_i \mid y{=}0) = 0.5 \times 0.0103 \times 0.0103 = 5.34 \times 10^{-5}$
x = 4 sits 2 units from the class-0 mean on each axis, so each 1-D Gaussian density is tiny: 0.0103.
Score class 1 the same way
$P(y{=}1)\prod_i P(x_i \mid y{=}1) = 0.5 \times 0.2076 \times 0.2076 = 0.0215$
x = 4 is only 1 unit from the class-1 mean on each axis, so each density is 20 times larger.
Normalize into a posterior
$P(y{=}1 \mid x) = \frac{0.0215}{0.0215 + 5.34 \times 10^{-5}} = 0.9975$
The proportionality in Bayes rule resolves by dividing by the sum of the two scores.

What just happened: Being 1 unit versus 2 units from a mean turned into a 400-to-1 posterior ratio — multiplying per-feature Gaussians amplifies distance differences exponentially, which is why Naive Bayes commits so confidently.

Now Break It

Try this: Strongly correlated features break the independence assumption and skew the probabilities.

Control: Correlation slider between features

What happens: Independence violated! These features are correlated, so the naive assumption makes the probabilities overconfident.

Where naive bayes is used

Naive Bayes applies Bayes' rule together with an assumption that features are independent given the class, and despite that strong simplification it remains a workhorse for text classification. Spam filters were among its earliest and most famous successes, scoring emails by how characteristic each word is of spam versus legitimate mail. It powers sentiment analysis, topic labeling, language identification, and document categorization, where the vocabulary is huge but each document is sparse. Because it only needs to count word frequencies per class, it trains in a single pass over the data, scales to millions of documents, and works surprisingly well even with limited examples. This speed and simplicity make it a strong baseline that more complex models must beat before they justify their extra cost.

The central misconception is that the independence assumption must hold for the model to work. Words in real text are clearly not independent, yet the classifier often performs well anyway because it only needs the correct class to receive the highest score, not perfectly accurate probabilities. That said, its probability estimates are frequently overconfident, pushing toward zero or one, so treat them as rankings rather than calibrated likelihoods. A practical pitfall is the zero-frequency problem: if a word never appeared with a class in training, its probability becomes zero and wipes out the whole product, which is why Laplace smoothing is applied. Choosing the wrong variant, such as Gaussian Naive Bayes for word counts instead of multinomial, is another common mistake.

Frequently asked questions

What does the naive in Naive Bayes mean?

It refers to the simplifying assumption that all features are conditionally independent given the class label. This is naive because real features, especially words in text, are usually correlated. The assumption is almost never literally true, but it makes the math tractable and the model fast.

Why does it work if the independence assumption is false?

Classification only requires that the correct class gets the highest score, not that the probabilities are exact. Even when the assumption inflates or distorts individual probabilities, the ranking of classes often stays correct. This is why Naive Bayes can be accurate while its probability outputs are poorly calibrated.

What is Laplace smoothing and why is it needed?

Laplace, or additive, smoothing adds a small count to every feature-class combination so no probability is ever exactly zero. Without it, a single unseen word would multiply the entire class probability by zero and eliminate that class. Smoothing keeps the model robust to words absent from the training data.

Which variant of Naive Bayes should I use?

Multinomial Naive Bayes suits word counts and term frequencies in text. Bernoulli Naive Bayes fits binary present-or-absent features. Gaussian Naive Bayes is for continuous features assumed to follow a normal distribution. Matching the variant to your feature type matters for good results.

Are the probabilities from Naive Bayes trustworthy?

Treat them with caution. The independence assumption tends to make the outputs overconfident, clustering near 0 or 1. They are reliable for ranking and picking the top class, but if you need well-calibrated probabilities you should apply a calibration step afterward.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.