Support Vector Machine

ClassificationIntermediate~8 min

Support Vector Machine — Find the boundary with the widest margin between classes.

A support vector machine doesn’t just find any boundary — it finds the one with the widest possible margin, the biggest gap between the two classes. Only the closest points (the support vectors) matter.

Class 0
Class 1
Boundary
Margins

Points inside the margin vs iteration

Margin width∞

Violations32 / 32

SVM controls

Data

Dataset

Points per class16

Spread1.0×

Model

C (soft margin)1.00

Sub-gradient steps200

Drag a non-support point — nothing moves. Drag a ringed support vector and the whole street re-solves. Small C widens the margin until it swallows violations.

Playback

Step 0 / 200

Speed

Plane swings
Margin widens
Settled
↻

Step 0 of 200 — sub-gradient step 0 at C = 1.00 — margin 2/‖w‖ = ∞, 32 of 32 points still inside the margin, ‖w‖ = 0.000

Break it

The idea in plain words

An SVM doesn’t just find a separating line — it finds the one with the widest possible margin, the biggest empty street between the two classes. Only the closest points, the support vectors, touch that street and determine it.

Drag a point far from the boundary and nothing changes. Drag a support vector and the whole street re-solves. When the classes overlap, the soft-margin parameter C decides how many violations to tolerate for a wider margin — the bridge to the kernel trick.

Now, the math

The SVM maximizes the margin, equivalently minimizing ‖w‖ subject to the labels:

\text{margin} = \frac{2}{\lVert w\rVert}

$\lVert w\rVert$: the weight norm — smaller means a wider margin.
$C$: soft-margin strength — large C punishes violations, small C tolerates them.

▸ Show the derivation

The support vectors are the points with margin ≤ 1; the solution depends only on them, which is why moving other points does nothing. Trained here with Pegasos — sub-gradient descent on the hinge loss plus an L2 term whose weight is set by C.

Trace it by hand

Six points in centered coordinates: class 0 at (-2,-2), (-2,0), (-1,-1) and class 1 at (2,2), (2,0), (1,1). Trained with the Pegasos core (C = 1, 400 iterations); weights rounded to 3 decimal places.

The learned boundary
$w = (0.505,\ 0.495), \qquad b \approx 0.000$
By symmetry of the data the true optimum is a 45-degree line through the origin; Pegasos lands within 0.01 of it.
Margin width from the weight norm
$\lVert w\rVert = \sqrt{0.505^2 + 0.495^2} = 0.707, \qquad \text{margin} = \frac{2}{\lVert w\rVert} = 2.83$
Find the support vectors
$y_i(w^\top x_i + b): \; (-1,-1) \mapsto 1.000, \quad (1,1) \mapsto 1.000, \quad (-2,0) \mapsto 1.010, \quad (2,0) \mapsto 1.010$
The far corners (-2,-2) and (2,2) have margin 2.000 — safely outside the street, so they are not support vectors and moving them changes nothing.
Score a new point
$w^\top x + b = 0.505(1) + 0.495(-1) + 0.000 = 0.010 > 0 \;\Rightarrow\; \text{class } 1$
A score of 0.010 against margin scores of 1.0 says this point sits almost exactly on the boundary.

What just happened: The four points with signed margin about 1.0 are the entire solution — they pin the street of width 2.83, while points at margin 2.0 could be deleted without moving the boundary at all.

Now Break It

Try this: Tiny C ignores misclassifications and picks a huge sloppy margin; huge C overfits to every point.

Control: C (regularization) slider

What happens: Mis-tuned C! Too small ignores errors for a fat margin; too large overfits to noise.

Where support vector machine is used

A support vector machine separates classes by finding the boundary that leaves the widest possible margin between them, which tends to generalize well because it commits to the most confident separation rather than any that merely works. SVMs were long the state of the art for text categorization and remain strong on high-dimensional, sparse data where the number of features exceeds the number of samples. They are used in bioinformatics for classifying gene expression profiles, in image recognition before deep learning dominated, and in handwriting and face detection pipelines. Their strength on small-to-medium datasets with many features, combined with solid theoretical guarantees, keeps them relevant as a reliable baseline and as a component in specialized systems where labeled data is scarce.

A common misunderstanding is that the margin is defined by all the training points, when in fact only the support vectors, the examples closest to the boundary, determine it; the rest could be removed without changing the result. Another pitfall is expecting SVMs to draw only straight boundaries, but with a kernel they can carve highly non-linear frontiers by operating in an implicit higher-dimensional space. Beginners also overlook the regularization parameter C, which trades off a wider margin against fewer training errors; setting it too high invites overfitting while too low underfits. Finally, SVMs do not output probabilities natively, so their raw scores are signed distances to the boundary and need an extra calibration step to become probabilities.

Frequently asked questions

What are support vectors?

Support vectors are the training points that lie closest to the decision boundary, right on the edge of the margin. They are the only points that determine where the boundary sits, so removing any other point would not change the model. Their name gives the algorithm its title.

What does maximizing the margin achieve?

The margin is the gap between the boundary and the nearest points of each class. Choosing the boundary that makes this gap as wide as possible gives the model a buffer against noise and tends to improve generalization to unseen data. This maximum-margin principle is the core idea of SVMs.

What is the C parameter?

C controls the trade-off between a wide margin and correctly classifying every training point. A large C penalizes misclassifications heavily, producing a narrower margin that may overfit, while a small C allows more margin violations for a smoother, more general boundary. It is typically tuned with cross-validation.

How do SVMs handle non-linear data?

They use the kernel trick, which implicitly maps the data into a higher-dimensional space where a linear separator exists, without ever computing that mapping explicitly. Common kernels include the radial basis function and polynomial kernels. The result is a curved boundary in the original feature space.

Can an SVM give probability estimates?

Not directly. An SVM produces a signed distance from the boundary rather than a probability. To obtain probabilities you fit a calibration model, such as Platt scaling, on top of the SVM scores, which adds computation and requires held-out data.

Do SVMs scale to very large datasets?

Kernel SVMs scale poorly because their training cost grows steeply with the number of samples, making them slow on millions of rows. Linear SVMs are much faster and handle large, high-dimensional data well. For huge non-linear problems, other methods are usually more practical.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.