GloVe (Global Vectors for Word Representation), from Stanford, takes a different route to learning embeddings than Word2Vec — instead of predicting words from local context windows, it directly factorizes global word co-occurrence statistics computed across the entire corpus.
The Core Idea — Global Co-occurrence, Not Local Prediction
Word2Vec learns from many small, local context windows, one prediction task at a time. GloVe instead first builds one giant matrix \(\mathbf{X}\), where \(X_{ij}\) counts how often word \(i\) and word \(j\) co-occur across the entire training corpus, then learns embeddings such that their dot products approximate the logarithm of these co-occurrence counts.
Formula (Simplified)
The training objective adjusts the embedding vectors (and bias terms) so that this approximation holds as closely as possible across every observed word pair, weighted to downweight extremely rare or extremely common co-occurrences.
GloVe vs Word2Vec — What Actually Differs
| Word2Vec | GloVe | |
|---|---|---|
| Training signal | Local context windows, one prediction task at a time | Global co-occurrence statistics, computed once across the whole corpus |
| Underlying approach | A neural prediction task (implicit factorization) | Explicit matrix factorization of co-occurrence counts |
| Practical quality | Comparable, often near-identical downstream performance for many tasks | Comparable — no consistently dominant winner between the two |
In practice, both methods tend to produce embeddings with broadly similar useful properties (including the analogy behavior from Word Embeddings) — the choice between them has historically mattered less than the choice of training corpus size and quality.
Code — Using Pretrained GloVe Vectors
import gensim.downloader as api
glove_model = api.load("glove-wiki-gigaword-300") # pretrained on Wikipedia + Gigaword
print(glove_model.most_similar("computer", topn=3))
print(glove_model.similarity("king", "queen"))Common Mistakes
- Assuming GloVe and Word2Vec produce embeddings that can be directly mixed or compared numerically — even though both produce dense vectors of similar dimensionality, they're trained independently and their vector spaces aren't aligned; a "cat" vector from GloVe isn't meaningfully comparable to a "cat" vector from Word2Vec via a raw dot product.
- Assuming one of GloVe or Word2Vec is a strict theoretical improvement over the other — both are well-established, and the more practically important choice today is usually whether to use static embeddings (either one) at all, versus modern contextual embeddings (next note).
Interview Relevance
Q: "What's the key difference between how GloVe and Word2Vec learn their embeddings?" Word2Vec learns from local context windows via a prediction task (CBOW or skip-gram), processing the corpus one small window at a time. GloVe instead first computes global co-occurrence statistics across the entire corpus into one matrix, then directly factorizes that matrix so embeddings' dot products approximate log co-occurrence counts — a more explicitly global, statistics-first approach rather than a local, prediction-task-first one.
Practice Question
Why might GloVe's use of global co-occurrence statistics help it capture some corpus-wide patterns that a purely local context-window approach might be slower to pick up on?
