CBOW (Continuous Bag of Words) is the first of Word2Vec's two training architectures — it predicts a target word from the (averaged) representation of its surrounding context words.
The Task, Concretely
Given a window of surrounding words, predict the missing word in the middle. For the sentence "the quick brown fox jumps," with a window size of 2, one training example uses context {"the", "quick", "fox", "jumps"} to predict the target word "brown."
Formula
\(m\) is the context window radius (e.g. 2 means 2 words on each side). The embeddings of all context words are simply averaged together into one vector, which is then used to predict the target word via a softmax over the entire vocabulary.
Diagram
All context words are averaged into one vector, then used to predict the single missing target word.
Code
from gensim.models import Word2Vec
sentences = [["the", "quick", "brown", "fox", "jumps"],
["the", "lazy", "dog", "sleeps"]]
model = Word2Vec(sentences, vector_size=100, window=2, sg=0) # sg=0 selects CBOW specifically
print(model.wv["fox"].shape) # (100,) -- the learned embedding for "fox"Why CBOW Trains Faster
Because CBOW averages multiple context words into a single training signal per example, it makes fewer, "smoother" updates per pass through the data compared to skip-gram (next note), which generates a separate training example per context word. This makes CBOW noticeably faster to train, especially on large corpora — a genuine practical tradeoff against skip-gram's typically stronger performance on rare words, covered in the next note's comparison.
Common Mistakes
- Assuming CBOW predicts multiple words at once — it predicts exactly one target word per training example; the averaging happens on the input (context) side, not the output side.
- Using too small a context window for a task that needs longer-range context, or too large a window for a task focused on tight local relationships — window size is a real, tunable hyperparameter affecting what kind of similarity the resulting embeddings capture.
Interview Relevance
Q: "What does CBOW predict, and why is it generally faster to train than skip-gram?" CBOW predicts a single target word from the averaged embeddings of its surrounding context words. It's faster because each context window produces exactly one training example (one prediction task), whereas skip-gram (predicting each context word separately from the target) produces multiple training examples per window — more total updates for the same amount of raw text.
Practice Question
For the sentence "she plays guitar well" with window size 1 and target word "guitar," what context words would CBOW average together as input?
