K-Nearest Neighbors

ClassificationBeginner~6 min

K-Nearest Neighbors — K-nearest neighbors is a supervised learning algorithm that classifies a point by a majority vote of its k closest labeled examples under a distance metric. It does no training — it simply stores the data and measures distance at prediction time.

Want to classify something? Just look at the closest examples you’ve already seen and go with the majority. That’s KNN — no training needed, just memory and a sense of distance.

Class A
Class B
Query point

The 5 nearest, sorted (hover to locate, click to scrub)

KNN controls

Data

Dataset

Points per class22

Spread1.0×

Model

k (neighbors)5

Distance

Vote weighting

Query predictionClass A (5–0)

Drag the ringed query point anywhere on the map.

Playback

Step 0 / 5

Speed

Measure every distance
Keep the k nearest
Tally the votes
↻

Step 0 of 5 — measured all 44 distances under Euclidean (L2) — only the 5 smallest get a vote

Break it

The idea in plain words

KNN doesn’t train — it memorizes the data. To classify a new point, it looks at the k closest labeled examples and takes a majority vote. Drag the query point around and watch its predicted class flip as its neighborhood changes.

With k = 1 the boundary bends around every noisy point (overfitting); with k as large as the dataset it always returns the global majority (underfitting). It’s a useful contrast to a fitted model like linear regression.

Now, the math

Neighbors are ranked by Euclidean distance:

d(p, q) = \sqrt{\sum_j (p_j - q_j)^2}

$p,\ q$: two points being compared.
$p_j$: the j-th feature (coordinate) of point p.
$k$: how many nearest neighbors vote.

▸ Show the derivation

k controls the bias–variance balance: small k gives a flexible, high-variance boundary that chases noise; large k averages over a wide neighborhood, raising bias until the model ignores local structure entirely.

Trace it by hand

Classify the query q = (3, 3) against five labeled points: (4,4) B, (5,3) A, (1,3) A, (2,5) B, (6,6) B. Distances are Euclidean, rounded to 3 decimal places.

Distance to the first point
$d(q, (4,4)) = \sqrt{(4-3)^2 + (4-3)^2} = \sqrt{2} = 1.414$
Distances to the other four
$\begin{aligned} d(q,(5,3)) &= \sqrt{4+0} = 2.000 \\ d(q,(1,3)) &= \sqrt{4+0} = 2.000 \\ d(q,(2,5)) &= \sqrt{1+4} = 2.236 \\ d(q,(6,6)) &= \sqrt{9+9} = 4.243 \end{aligned}$
Vote with k = 3
$k{=}3: \{B\,(1.414),\; A\,(2.000),\; A\,(2.000)\} \Rightarrow 2 \text{ votes A},\ 1 \text{ vote B} \Rightarrow A$
The single closest point is a B, but the vote overrules it.
Vote with k = 5
$k{=}5: 3 \text{ votes B},\ 2 \text{ votes A} \Rightarrow B$
Same query, same data — only k changed, and the answer flipped.

What just happened: Five square roots and two votes: k = 3 predicts A while k = 5 predicts B for the very same query. The choice of k is not a detail — it decides the answer near class borders.

Now Break It

Try this: k=1 memorizes every noisy point; k=N always predicts the majority class regardless of position.

Control: k slider (set to 1, then to maximum)

What happens: k=1: Memorizing noise — the boundary is jagged and overfitting. k=max: Ignoring all structure — predicting the majority class everywhere.

Where k-nearest neighbors is used

K-nearest neighbors classifies a new point by looking at the labels of its closest training examples and taking a majority vote, which makes it a natural fit for problems where similar inputs should share an outcome. Recommendation systems use neighbor search to suggest items enjoyed by people with similar tastes. Content-based image and document retrieval find the most similar examples to a query, and modern semantic search over vector embeddings is essentially nearest-neighbor lookup at massive scale. It also appears in anomaly detection, where a point far from all its neighbors is flagged as unusual, and in medical decision support, where clinicians surface historically similar cases. Because it stores the data rather than fitting parameters, it adapts instantly whenever new labeled examples arrive.

The biggest misconception is that k-nearest neighbors does no training, so it must be cheap. Training is trivial, but every prediction requires comparing the query against many stored points, which makes inference slow and memory-hungry on large datasets unless you use spatial indexes or approximate search. A second pitfall is ignoring feature scaling: because the method relies on distances, a feature measured in thousands will dominate one measured in fractions, so standardization is essential. People also underestimate the curse of dimensionality, where in high dimensions all points become roughly equidistant and the notion of nearest loses meaning. Finally, choosing k too small makes predictions noisy while choosing it too large blurs genuine class boundaries.

Frequently asked questions

How do I choose the value of k?

There is no universal best value, so it is tuned with cross-validation. Small k makes the model sensitive to noise and outliers, while large k smooths predictions but can wash out real boundaries. A common practical starting point is an odd number near the square root of the number of training examples, then refine from there.

Why is feature scaling so important for KNN?

The algorithm ranks neighbors by distance, and distance is dominated by whichever feature has the largest numeric range. Without scaling, a feature like annual income in dollars would overwhelm a feature like age in years. Standardizing or normalizing features puts them on comparable footing so distances reflect true similarity.

Is KNN really training-free?

The fitting step just stores the labeled data, so it is often called a lazy learner. The real cost is deferred to prediction time, when the model must find the nearest neighbors of each query. For large datasets this is handled with structures like KD-trees, ball trees, or approximate nearest-neighbor libraries.

What is the curse of dimensionality in this context?

As the number of features grows, the space becomes so vast that all points tend toward being equally far apart, which erodes the meaning of nearest. Distances become less discriminative and the method degrades. Reducing dimensionality or selecting informative features helps restore useful neighborhoods.

Can KNN be used for regression too?

Yes. Instead of voting on class labels, KNN regression averages the target values of the nearest neighbors to produce a numeric prediction. Weighted variants give closer neighbors more influence, and the same concerns about scaling and dimensionality apply.

What distance metric should I use?

Euclidean distance is the default and works well for continuous, scaled features. Manhattan distance can be more robust in higher dimensions, and cosine similarity suits text or embedding vectors where orientation matters more than magnitude. The right choice depends on what similar means for your data.

← Back to all visualizations Continue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated August 6, 2026.