Skip to content
ML Visualization

Decision Boundaries

ClassificationIntermediate~7 min

Decision Boundaries — A decision boundary is the surface a classifier draws to separate classes. Different models draw very different boundaries on the same data: logistic regression a straight line, kNN jagged islands, a tree axis-aligned boxes, an SVM a max-margin street.

Paint a spiral or two moons, then watch four models try to separate it at once — the linear model fails while kNN and the tree contort to fit. The tradeoffs, obvious in a single glance.

Training error at this capacity (click to spotlight a panel)

Compare controls

Data
Dataset
26
1.0×
Paint points as
Model
25
1
0.02
40
0.60
Playback
Step 0 / 12
Speed
  1. Too rigid
  2. Balanced
  3. Contorted

Step 0 of 12 — capacity rung 0 — kNN k = 25, tree depth 1, SVM C = 0.02, logistic 40 epochs · training error Logistic 15%, kNN 19%, Decision Tree 19%, SVM 19%

Four classifiers, one dataset. Drag a point or paint new ones on any panel — all four boundaries refit at once. Scrub the capacity ladder and watch which models can change shape and which are stuck being a line.

Break it

On a spiral or circles the linear models fail outright while kNN and the tree contort to fit — and at the top rung they carve islands around single points.

The idea in plain words

Every classifier draws a different kind of line. Put four on the same data and the tradeoffs are obvious at a glance: logistic regression and SVM draw straight boundaries, kNN draws jagged islands, and a decision tree draws axis-aligned boxes.

On blobs they all look fine. Switch to a spiral or concentric circles and the linear models fail completely while the flexible ones contort to fit — the whole point of having more than one model.

Now, the math

Each model defines its decision region by a different rule:

linear: sign⁡(w⊤x+b)\text{linear:}\ \operatorname{sign}(w^\top x + b)
kNN: majority label of k nearest\text{kNN:}\ \text{majority label of } k \text{ nearest}
w⊤x+bw^\top x + b
a linear score — a straight boundary.
kk
neighborhood size for kNN — small k means jagged regions.
▸ Show the derivation

Linear models can only separate classes a hyperplane can split; a spiral needs a curved boundary they cannot express. kNN and trees are non-parametric — they carve arbitrarily complex regions, at the cost of overfitting on sparse or noisy data.

Trace it by hand

One query point (2, 3), two models trained on the same five labeled points: a linear classifier with w = (1, -1), b = 0.5, and a 3-nearest-neighbor vote over (1,3) and (2,4) and (3,3) of class 1, plus (4,1) and (5,2) of class 0. Distances rounded to 3 decimal places.

  1. Model 1: the linear rule

    sign⁡(w⊤x+b)=sign⁡(1(2)−1(3)+0.5)=sign⁡(−0.5)  ⇒  class 0\operatorname{sign}(w^\top x + b) = \operatorname{sign}(1(2) - 1(3) + 0.5) = \operatorname{sign}(-0.5) \;\Rightarrow\; \text{class } 0

  2. Model 2: measure the kNN distances

    d((2,3),(1,3))=1.000  [class 1]d((2,3),(2,4))=1.000  [class 1]d((2,3),(3,3))=1.000  [class 1]d((2,3),(4,1))=2.828  [class 0]d((2,3),(5,2))=3.162  [class 0]\begin{aligned} d((2,3),(1,3)) &= 1.000 \;[\text{class } 1] \\ d((2,3),(2,4)) &= 1.000 \;[\text{class } 1] \\ d((2,3),(3,3)) &= 1.000 \;[\text{class } 1] \\ d((2,3),(4,1)) &= 2.828 \;[\text{class } 0] \\ d((2,3),(5,2)) &= 3.162 \;[\text{class } 0] \end{aligned}

  3. Model 2: take the majority

    k=3:  nearest are all class 1  ⇒  kNN says class 1k = 3: \; \text{nearest are all class } 1 \;\Rightarrow\; \text{kNN says class } 1

  4. The two models disagree

    linear: y^=0,kNN: y^=1\text{linear: } \hat{y} = 0, \qquad \text{kNN: } \hat{y} = 1

    The query sits on the class-0 side of the straight line but inside a local pocket of class-1 points that the line cannot bend around.

What just happened: Score minus 0.5 says class 0; three unanimous neighbors at distance 1.0 say class 1. Decision boundaries differ because each model answers a different question — global line position versus local neighborhood — and the disagreement zone is exactly where model choice matters.

Now Break It

Try this: The spiral breaks the linear model; a sparse dataset makes kNN jagged.

Control: Preset shape picker (spiral / XOR)

What happens: The linear model can’t separate a spiral — only the non-linear models bend to fit.

Where decision boundaries is used

A decision boundary is the surface where a classifier switches from predicting one class to another, and visualizing it is one of the fastest ways to understand how an algorithm behaves. Comparing several classifiers on the same dataset reveals their inductive biases at a glance: logistic regression carves the plane with a straight line, a decision tree produces axis-aligned rectangular regions, k-nearest neighbors traces a jagged boundary that hugs the data, and a kernelized support vector machine bends smoothly around clusters. Practitioners use these side-by-side plots when teaching, when debugging a model that misbehaves near class edges, and when choosing an algorithm whose shape matches the geometry of the problem. Seeing the boundary also exposes overfitting, which shows up as islands wrapped tightly around individual points.

The most common misconception is that a more complicated boundary is a better boundary. A wiggly frontier that perfectly separates the training points often reflects noise rather than signal, and it usually generalizes worse than a smoother one. Another pitfall is forgetting that a boundary lives in the full feature space, not just the two dimensions you happen to plot; a model can look confused in a 2D projection while separating the classes cleanly in higher dimensions. People also assume every classifier draws one connected boundary, but methods like k-nearest neighbors and decision trees can create several disconnected regions for the same class. Finally, the visual boundary is only as trustworthy as the data density around it, since regions with no training examples are essentially guesses.

Frequently asked questions

What exactly is a decision boundary?
It is the set of points in feature space where a classifier is equally undecided between classes, so an infinitesimal move across it flips the prediction. On one side the model predicts one class and on the other side it predicts another. For a two-feature problem you can draw it as a line or curve on a plane.
Why do different classifiers produce such different shapes?
Each algorithm has its own inductive bias, meaning built-in assumptions about what a good separation looks like. Linear models assume classes can be split by a flat surface, trees split along one feature at a time producing rectangular regions, and neighbor-based methods follow the local density of the data. Those assumptions directly shape the boundary.
Does a smoother boundary always generalize better?
Not always, but a very jagged boundary is a warning sign of overfitting. Smoothness reflects a simpler hypothesis that is less sensitive to noise, which often generalizes better. The right amount of complexity depends on how much true structure exists in the data, so validation on held-out data is the real test.
Can a boundary be non-linear even for a linear model?
Yes, if you transform the inputs first. Adding polynomial or interaction features lets a linear classifier draw a curved boundary in the original space while remaining linear in the expanded feature space. This is the same idea that the kernel trick exploits implicitly.
Why should I visualize boundaries in only two dimensions?
Two dimensions are all we can plot directly, so they are a teaching and debugging aid rather than the full truth. A real model often uses many features, and its true boundary is a high-dimensional surface. Projections can be misleading, so treat 2D plots as intuition rather than proof.
← Back to all visualizationsContinue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated .