Naive Bayes for Text Classification

A complete, realistic text classification workflow using Naive Bayes — from raw documents to a trained spam filter — tying together TF-IDF/Bag-of-Words feature extraction with the algorithm's specific strengths for this exact problem type.

Why Naive Bayes Is Such a Strong Fit for Text

• Text naturally produces very high-dimensional, sparse features (thousands of possible words) — Naive Bayes handles this efficiently, unlike distance-based methods that struggle in high dimensions
• It's extremely fast to train, even on large document collections
• The independence assumption, while technically wrong for language, doesn't hurt classification accuracy nearly as much as intuition might suggest

Full Workflow — Bag-of-Words + Multinomial NB

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score

emails = [
    "free money free offer act now",
    "win a free prize click here",
    "urgent free cash reward waiting",
    "meeting schedule tomorrow morning",
    "project update meeting notes attached",
    "quarterly report meeting agenda",
]
labels = [1, 1, 1, 0, 0, 0]   # 1 = spam, 0 = not spam

X_train_text, X_test_text, y_train, y_test = train_test_split(
    emails, labels, test_size=0.34, random_state=42, stratify=labels
)

vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(X_train_text)   # fit vocabulary on TRAINING data only
X_test = vectorizer.transform(X_test_text)           # transform test data using that SAME vocabulary

model = MultinomialNB()
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(accuracy_score(y_test, predictions))
print(classification_report(y_test, predictions))

Using TF-IDF Instead of Raw Counts

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer()
X_train_tfidf = tfidf_vectorizer.fit_transform(X_train_text)
X_test_tfidf = tfidf_vectorizer.transform(X_test_text)

model_tfidf = MultinomialNB()
model_tfidf.fit(X_train_tfidf, y_train)
print(model_tfidf.score(X_test_tfidf, y_test))
# In practice, results with raw counts vs TF-IDF for Multinomial NB are often similar --
# TF-IDF's benefit is more pronounced for other algorithms (like linear models)

Inspecting What the Model Learned

import numpy as np

feature_names = vectorizer.get_feature_names_out()
log_probs = model.feature_log_prob_   # log P(word | class), one row per class

spam_class_index = list(model.classes_).index(1)
top_spam_words = np.argsort(log_probs[spam_class_index])[-5:]
print([feature_names[i] for i in top_spam_words])   # the words most strongly associated with spam

This is a genuinely useful debugging and explainability step — reading off which words the model considers most spam-indicative is a direct, human-readable check on whether the model learned something sensible.

Handling New/Unseen Words at Prediction Time

new_email = ["free vacation offer waiting"]
new_email_vec = vectorizer.transform(new_email)   # words not in the training vocabulary are simply ignored
print(model.predict(new_email_vec))
print(model.predict_proba(new_email_vec))

CountVectorizer.transform() silently ignores any word not seen during fit_transform() on the training set — this is generally the right behavior (an unknown word contributes no signal either way), but it's worth knowing explicitly rather than being surprised by it.

Common Mistakes

• Calling fit_transform() on the test set instead of just transform() — this would build a different vocabulary from test data, causing a shape mismatch and, more subtly, leaking test-set vocabulary information.
• Not removing extremely common, uninformative words when they add noise for a given classification task — though for spam/ham classification specifically, even common words can carry useful signal.
• Evaluating only accuracy on a dataset where spam and ham aren't evenly represented — see Imbalanced Data.

Interview Relevance

Q: "Why is Naive Bayes historically such a common choice for spam filtering specifically?" It's fast enough for real-time filtering at scale, handles the high-dimensional sparse features text naturally produces well, requires relatively little training data to perform reasonably, and its probabilistic output naturally supports a tunable spam-confidence threshold.

Practice Question

After training a spam classifier, you notice the word "conference" is among the top spam-indicative words — clearly a labeling or data quality issue. What would you check first?

Related ML Notes

• Text Features
• Multinomial Naive Bayes
• Spam Detection Project