By the end of this lesson, you will understand how decision trees partition data to make predictions and how random forests improve accuracy by combining many trees.
What it is
A Decision Tree is a flowchart-like structure where each internal node represents a feature test, each branch represents the outcome of the test, and each leaf node holds a prediction. It splits data recursively to maximize purity (e.g., using Gini impurity or entropy).
A Random Forest is an ensemble method that builds multiple decision trees during training. Each tree sees a random subset of data (bootstrapping) and a random subset of features at each split. The final prediction is the average (regression) or majority vote (classification) of all trees.
Related terms: Overfitting, Bagging, Feature Importance, Impurity Reduction.
Why it matters
- Interpretability: Single trees provide clear rules humans can understand.
- Robustness: Random forests reduce variance and overfitting compared to single trees.
- No Scaling Needed: Trees are invariant to monotonic transformations of input features.
- Feature Selection: They naturally rank which variables matter most for prediction.
Syntax or steps
- Split: Choose the best feature and threshold to divide the dataset into two groups.
- Recurse: Repeat the splitting process on each subgroup until a stopping criterion is met (e.g., max depth reached or pure nodes).
- Predict: For a new data point, traverse the tree based on its feature values until reaching a leaf.
- Ensemble (Forest): Aggregate predictions from many independently trained trees.
Example
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
# Load data
data = load_iris()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Train Decision Tree
dt_clf = DecisionTreeClassifier(max_depth=5, random_state=42)
dt_clf.fit(X_train, y_train)
dt_pred = dt_clf.predict(X_test)
# Train Random Forest
rf_clf = RandomForestClassifier(n_estimators=100, random_state=42)
rf_clf.fit(X_train, y_train)
rf_pred = rf_clf.predict(X_test)
print(f"Decision Tree Accuracy: {accuracy_score(y_test, dt_pred):.2f}")
print(f"Random Forest Accuracy: {accuracy_score(y_test, rf_pred):.2f}")
This code loads the Iris dataset, splits it, trains both models, and compares their accuracy. The max_depth parameter prevents the single tree from becoming too complex, while n_estimators defines the number of trees in the forest.
Common mistakes
- Overfitting with deep trees: Unconstrained trees memorize noise. Fix by limiting
max_depth,min_samples_leaf, or using pruning. - Ignoring feature importance bias: High-cardinality categorical features may appear more important simply because they offer more split points. Use permutation importance instead.
- Assuming linear relationships: Trees struggle with smooth, continuous boundaries requiring many splits. Consider gradient boosting or linear models if interpretability isn't critical.
- Using too few estimators: A small forest behaves like a single noisy tree. Increase
n_estimatorsuntil performance stabilizes.
When to use it
| Scenario | Best Choice | Reason |
|---|---|---|
| Need simple, explainable rules | Decision Tree | Single path logic is easy to visualize and audit. |
| High accuracy required, tabular data | Random Forest | Ensembling reduces variance and handles non-linearities well. |
| Limited computational resources | Decision Tree | Faster to train and predict than large forests. |
Practice
Guided Exercise: Modify the example above to print the top 3 most important features from the Random Forest model using rf_clf.feature_importances_.
Challenge: Train a Decision Tree without setting max_depth. Compare its training accuracy vs. testing accuracy. What does the gap indicate?
Hint: A large gap indicates overfitting; the tree learned the training noise but fails to generalize.
Quick check
Q: Why do Random Forests generally outperform single Decision Trees?
A: By averaging many decorrelated trees, Random Forests reduce variance and mitigate the risk of any single tree making a poor prediction due to specific data quirks.
Summary
Decision trees offer intuitive, rule-based modeling but are prone to overfitting. Random forests solve this by aggregating many trees, providing robust, high-accuracy predictions for tabular data while retaining some interpretability through feature importance scores.