Skip to content
ML Visualization

Word Embeddings

Deep LearningIntermediate~9 min

Word Embeddings — A word embedding maps each word to a dense vector of numbers so that words used in similar contexts sit close together, and directions in the space capture relationships — famously, king − man + woman lands near queen.

Explore a galaxy of real GloVe word vectors in 3D: similar words cluster, and meaning becomes geometry. Do arithmetic on words — king minus man plus woman — and watch the answer, queen, light up. Then ask for a relationship the text never showed and watch the analogy fall apart.

Loading the word galaxy…

The idea in plain words

A word embedding turns every word into a list of numbers — a vector — chosen so that words appearing in similar contexts end up with similar vectors. Plotted in space, synonyms and related words cluster together, and whole regions correspond to themes like countries, family roles, or colors. The model never sees a dictionary; it learns these positions purely from which words tend to appear near each other across billions of sentences.

The surprising part is that directions carry meaning too. The step from man to king is almost the same direction as woman to queen — a “royalty” direction — so you can do arithmetic on words. These vectors are what feed modern language models: before attention can relate words, each token is first turned into an embedding like the ones you are exploring here.

Now, the math

Similarity between two word vectors is measured by the cosine of the angle between them:

sim(u,v)=u⋅v∥u∥ ∥v∥\mathrm{sim}(u, v) = \frac{u \cdot v}{\lVert u\rVert\,\lVert v\rVert}

An analogy “a is to b as c is to ?” is solved by vector arithmetic, then a nearest-neighbor search:

answer=arg⁡max⁡w sim(vw,  vb−va+vc)\mathrm{answer} = \arg\max_{w}\ \mathrm{sim}\big(v_w,\; v_b - v_a + v_c\big)
vwv_w
the embedding vector for word w.
u⋅vu \cdot v
the dot product — large when vectors point the same way.
cos⁡\cos
cosine similarity ignores length and compares only direction.
▸ Show the derivation

The demo uses real 50-dimensional GloVe vectors. You cannot see 50 dimensions, so the galaxy projects them down to three with principal component analysis, keeping the directions of greatest variance. Analogies and neighbor rankings are computed on the full 50-dimensional vectors, so the math is exact even though the picture is a shadow of the true space.

Now Break It

Try this: Ask for an analogy with no basis in text — cat is to table as music is to what — and the top match is arbitrary with a low similarity score.

Control: Analogy builder (the "Shaky analogy" preset)

What happens: Weak analogy! Word vectors only capture patterns that actually co-occur in text — a relationship the corpus never saw returns a low-confidence, essentially random word.

Where word embeddings is used

Word embeddings quietly power much of modern language technology. Search engines use them to match a query to documents that share meaning rather than exact words; recommendation systems embed products and users into the same space to suggest related items; and every large language model begins by converting tokens into embeddings before any deeper processing. The same idea generalizes far beyond words — embeddings represent images, songs, molecules, and graph nodes, letting systems measure similarity and cluster items by learned meaning instead of hand-written rules.

A common misconception is that embeddings “understand” words. They only capture statistical co-occurrence from the training text, which is why they also absorb its biases — analogies can reproduce stereotypes present in the data, a serious fairness concern. Another pitfall is trusting every analogy: the famous king−man+woman result is unusually clean, and many triples return vague or wrong matches because the relationship simply is not encoded linearly. Finally, classic GloVe and word2vec give each word a single fixed vector, so they cannot distinguish the two senses of “bank” — that limitation is exactly what contextual models were built to fix.

Frequently asked questions

What is a word embedding?
A word embedding is a dense vector of numbers assigned to a word so that words used in similar contexts have similar vectors. The vectors are learned from large amounts of text, and distances and directions between them capture semantic relationships. This lets software treat meaning as geometry.
How does king minus man plus woman equal queen?
In a good embedding space, the direction that separates male and female words is roughly consistent, and so is the direction for royalty. Subtracting man and adding woman shifts the king vector along the gender direction while keeping the royalty component, landing near queen. The result is found by taking the vector b minus a plus c and searching for the closest word.
What is the difference between word2vec and GloVe?
Both produce static word vectors, but they train differently. Word2vec learns by predicting nearby words with a sliding context window, while GloVe factorizes a global word co-occurrence count matrix. In practice they yield similar, high-quality embeddings; this visualization uses pretrained GloVe vectors.
What is cosine similarity and why use it instead of distance?
Cosine similarity measures the angle between two vectors, ignoring their lengths. For word embeddings, direction carries the meaning while magnitude often reflects word frequency, so comparing angles is more reliable than straight-line distance. A cosine near 1 means very similar, near 0 means unrelated, and negative means opposite.
Do word embeddings capture bias?
Yes. Because embeddings are learned from human-written text, they absorb the statistical associations in that text, including gender, racial, and cultural stereotypes. Analogy arithmetic can surface these biases, which is why practitioners audit and sometimes debias embeddings before using them in sensitive applications.
How are these related to large language models?
Embeddings are the first step inside a language model: each token is converted to a vector before attention and other layers process it. Modern models learn contextual embeddings that change with the surrounding sentence, unlike the fixed vectors here, but the core idea of representing meaning as vectors is the same foundation.
← Back to all visualizationsContinue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated .