By the end of this lesson, you will understand how K-Nearest Neighbors (KNN) predicts outcomes by looking at similar data points and be able to implement a basic classifier using Python.
What it is
K-Nearest Neighbors (KNN) is a non-parametric, instance-based learning algorithm. Unlike models that learn a general rule (like a line or tree), KNN stores the entire training dataset. When making a prediction for a new data point, it identifies the k closest examples in the training set based on distance metrics (usually Euclidean distance). For classification, it assigns the majority label among those neighbors; for regression, it averages their values.
Mental Model: Imagine you are trying to guess if a fruit is an apple or an orange. You look at the five fruits most similar in size and color to yours. If four are apples and one is an orange, you classify your fruit as an apple.
Related terms: Lazy learner, Distance metric, Curse of dimensionality.
Why it matters
- Simplicity: It requires no complex mathematical assumptions about data distribution.
- Flexibility: Works well for both classification and regression tasks without changing the core logic.
- No Training Phase: The model "trains" instantly because it just stores data; computation happens during prediction.
- Baseline Performance: Often provides a strong baseline for comparison against more complex algorithms.
Syntax or steps
- Choose K: Select the number of neighbors (e.g.,
k=3). Odd numbers help avoid ties in binary classification. - Calculate Distances: Compute the distance between the query point and every point in the training set.
- Select Neighbors: Identify the
kpoints with the smallest distances. - Predict: Return the mode (classification) or mean (regression) of these neighbors.
Example
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
# 1. Load data
data = load_iris()
X, y = data.data, data.target
# 2. Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# 3. Initialize the model with k=5
knn = KNeighborsClassifier(n_neighbors=5)
# 4. Train the model (stores data)
knn.fit(X_train, y_train)
# 5. Make predictions
y_pred = knn.predict(X_test)
# 6. Evaluate
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2f}")Explanation: We use the Iris dataset. train_test_split ensures we evaluate on unseen data. KNeighborsClassifier is initialized with n_neighbors=5. The fit method simply stores the training data. predict calculates distances for each test sample and returns the majority class label.
Common mistakes
- Ignoring Feature Scaling: KNN relies on distance. Features with larger ranges (e.g., age 0-100 vs. salary 0-100,000) dominate the calculation. Always normalize or standardize data before applying KNN.
- Choosing Poor K Values: A very small
kleads to overfitting (noise sensitivity); a very largekleads to underfitting (oversmoothing). Use cross-validation to tunek. - High Dimensional Data: In high dimensions, all points become roughly equidistant ("curse of dimensionality"), degrading performance. Consider dimensionality reduction techniques like PCA first.
- Computational Cost: Prediction time grows linearly with the size of the training set. KNN is slow for massive datasets compared to parametric models.
When to use it
| Scenario | Use KNN? | Alternative |
|---|---|---|
| Small to medium dataset, low dimensions | Yes | - |
| Real-time prediction required | No | Decision Trees / Logistic Regression |
| Data has many irrelevant features | No | Random Forests / SVM |
| Non-linear boundaries needed | Yes | Kernel SVM / Neural Networks |
Practice
Guided Exercise: Modify the example above to use k=1 and k=15. Observe how the accuracy changes. Which value yields higher accuracy? Why might k=1 be risky?
Challenge: Implement feature scaling using StandardScaler from scikit-learn before fitting the KNN model. Compare the accuracy with and without scaling.
Quick check
Question: Why must you scale features when using KNN?
Answer: Because KNN uses distance calculations (like Euclidean distance), unscaled features with larger magnitudes will disproportionately influence the distance, biasing the neighbor selection toward those features regardless of their actual predictive power.
Summary
KNN is a powerful, intuitive algorithm that predicts based on similarity rather than learned parameters. While simple to implement, its effectiveness heavily depends on proper feature scaling and careful selection of the k parameter to balance bias and variance.