Word embeddings replace one-hot encoding's huge, meaningless vectors with much smaller, dense vectors — learned such that words with similar meanings end up close together in the embedding space, restoring exactly the notion of similarity one-hot encoding lacked.
The Core Idea
Instead of a vector as long as the entire vocabulary, each word gets a dense vector of a fixed, much smaller size (commonly 100–300 dimensions) — every value in this vector is a learned parameter, adjusted during training so that words used in similar contexts end up with similar vectors, following the "distributional hypothesis": words that appear in similar contexts tend to have similar meanings.
The Famous Analogy Property
Well-trained embeddings, remarkably, capture semantic relationships as consistent directions in the embedding space — the vector difference between "king" and "man" ends up pointing in a similar direction as the difference between "queen" and "woman" (both roughly encoding "male → female"), letting simple vector arithmetic solve word analogies. This emergent property was one of the most striking early demonstrations of what learned embeddings capture.
How Embeddings Are Implemented
\(\mathbf{E}\) is an embedding matrix of shape \((\text{vocab\_size}, d_{\text{embed}})\) — essentially a lookup table. Looking up a word's embedding is simply indexing into this matrix at that word's token id row. \(\mathbf{E}\) itself is a learnable parameter, updated via backpropagation exactly like any other weight matrix.
Code
import torch
import torch.nn as nn
embedding_layer = nn.Embedding(num_embeddings=10000, embedding_dim=300)
token_ids = torch.tensor([42, 7, 891]) # 3 words, represented as their vocabulary indices
embeddings = embedding_layer(token_ids)
print(embeddings.shape) # (3, 300) -- one dense 300-dim vector per word
# Compare to one-hot: same 3 words would need (3, 10000) -- one-hot is over 30x larger hereTwo Ways to Get Embeddings
| Approach | Detail |
|---|---|
| Train from scratch | The embedding matrix starts randomly initialized and is learned entirely from your specific task's training data |
| Use pretrained embeddings | Load embeddings already trained on a huge external corpus (Word2Vec, GloVe — the next few notes), often as a strong starting point, sometimes frozen, sometimes further fine-tuned |
Common Mistakes
- Training an embedding layer from scratch on a very small dataset — with limited data, the embedding matrix may never see enough examples of many words to learn meaningful representations; pretrained embeddings are usually a better starting point in this case.
- Forgetting that
nn.Embeddingis just a lookup table with learnable parameters — it's mathematically equivalent to multiplying a one-hot vector by the embedding matrix, but implemented much more efficiently as a direct index lookup instead of an actual matrix multiplication.
Interview Relevance
Q: "What does it mean for word embeddings to capture semantic similarity, and how is that actually achieved?" Words used in similar contexts across a large training corpus end up with similar embedding vectors — this happens automatically during training because the embedding matrix's parameters are updated via backpropagation to make the model's predictions (whatever the specific training objective is — predicting a word from context, or vice versa) more accurate, and words that behave similarly in context end up needing similar vector representations to support similar predictions.
Practice Question
Mathematically, why is nn.Embedding(vocab_size, embed_dim) equivalent to multiplying a one-hot vector by a \((\text{vocab\_size}, \text{embed\_dim})\) matrix, even though it's implemented as a direct lookup rather than a matrix multiplication?
