What does word2vec actually learn?

Researchers have unveiled a complete closed-form theory of how word2vec — the embedding algorithm that paved the way for modern language models — actually learns. The new paper proves that, in realistic training regimes, word2vec's learning problem reduces to unweighted least-squares matrix factorization, with gradient flow dynamics solvable in closed form. The punchline: the final learned representations are simply given by principal component analysis (PCA) of a matrix built from corpus statistics.

A Decade-Old Question About a Minimal Language Model

word2vec learns dense vector representations of words via a contrastive algorithm; after training, semantic similarity between words is captured by the angle between their embeddings. The learned vectors famously exhibit striking linear structure — linear subspaces encode interpretable concepts such as gender, verb tense, or dialect, and enable analogy completion like "man : woman :: king : queen" through vector addition. This "linear representation hypothesis" has drawn renewed attention because large language models show the same behavior, underpinning semantic inspection of internal representations and novel model steering techniques.

Yet despite word2vec's status as a precursor to modern language models, researchers lacked a quantitative, predictive theory of its learning process. Understanding word2vec, the authors argue, is a prerequisite to understanding feature learning in more sophisticated language modeling tasks.

One Concept at a Time, in Discrete Steps

The theory's central prediction concerns learning dynamics. When all embedding vectors are initialized randomly but very close to the origin — effectively zero-dimensional — the model learns one concept (an orthogonal linear subspace) at a time, in a sequence of discrete learning steps. Each step increments the rank of the embedding matrix and decreases the loss, giving word embeddings more room to express meaning. Crucially, once a subspace is learned, it does not rotate, making these subspaces the model's learned features.

The features themselves have a simple form: they are the top eigenvectors of a target matrix M* built entirely from measurable corpus statistics and algorithmic hyperparameters, where each entry combines word co-occurrence probabilities P(i,j) and unigram probabilities P(i). When constructed from Wikipedia statistics and diagonalized, the top eigenvector selects words tied to celebrity biographies, the second to government and municipal administration, the third to geographical and cartographical descriptors, and so on. Training, in effect, finds a sequence of optimal low-rank approximations of M*.

Distribution-Free Theory, Verified on Benchmarks

Notably, the "mild approximations" required — a quartic approximation of the objective around the origin, a constraint on hyperparameters, small initial weights, and vanishingly small gradient descent steps — involve no assumptions about the data distribution. Such distribution-agnostic descriptions of learning dynamics are rare; the authors state this is the first for a practical natural language task. Empirically, the approximations stay faithful to real word2vec: on the standard analogy completion benchmark, word2vec achieves 68% accuracy, the approximate model 66%, and the classical PPMI alternative just 51%.

The paper also applies the theory to abstract linear representations of binary concepts such as masculine/feminine or past/future. word2vec builds these over a sequence of noisy learning steps, and their geometry matches a spiked random matrix model. Early in training, semantic signal dominates; later, noise can take over and degrade the model's ability to resolve those representations.

Where This Leaves the Field

This is one of the first complete closed-form theories of feature learning in a minimal yet genuinely relevant language task, and a step toward analytical descriptions of practical machine learning algorithms. What to watch next: whether similar distribution-agnostic analyses can extend to deeper, non-linear language models, and whether the noise-driven late-training degradation of linear representations has parallels in today's larger systems.