Removing Duplicates

Learn how to remove duplicate rows from a Pandas DataFrame using drop_duplicates(), ensuring your dataset contains only unique records based on specific columns or the entire row.

What it is

Duplicate removal is the process of identifying and deleting redundant entries in a dataset. In Python's Pandas library, this is handled by the DataFrame.drop_duplicates() method. A "duplicate" can be defined as an exact match across all columns or a match within a subset of key columns (e.g., removing users with the same email address but keeping different purchase dates).

Mental Model: Think of it as filtering a list where you keep the first occurrence of a value and discard subsequent identical values. The related term unique() returns distinct values from a Series, while duplicated() returns a boolean mask indicating which rows are duplicates.

Why it matters

• Data Integrity: Prevents skewed analysis caused by double-counting events or entities.
• Performance: Reduces memory usage and speeds up operations like grouping and joining.
• Accuracy: Ensures metrics like averages or counts reflect true unique occurrences.
• Cleanliness: Removes artifacts from data merging or scraping processes that often introduce redundancy.

Syntax or steps

The basic syntax is df.drop_duplicates(subset=None, keep='first', inplace=False).

1. subset: Specify column names to check for duplicates. If None, all columns are considered.
2. keep: Determines which duplicate to retain: 'first' (default), 'last', or False (remove all duplicates).
3. inplace: If True, modifies the original DataFrame; otherwise, returns a new one.

Example

import pandas as pd

# Create sample data with duplicates
data = {
    'Name': ['Alice', 'Bob', 'Alice', 'Charlie', 'Bob'],
    'Age': [25, 30, 25, 35, 31],
    'City': ['NYC', 'LA', 'NYC', 'Chicago', 'LA']
}
df = pd.DataFrame(data)

print("Original DataFrame:")
print(df)

# Remove exact duplicates across all columns
df_clean = df.drop_duplicates()

print("\nAfter removing exact duplicates:")
print(df_clean)

# Remove duplicates based only on 'Name' (keeping the first occurrence)
df_unique_names = df.drop_duplicates(subset=['Name'])

print("\nAfter removing duplicates based on 'Name':")
print(df_unique_names)

Explanation: The first call removes rows where all values match previous rows. The second call uses subset=['Name'], so if two rows have the same name, only the first one encountered is kept, regardless of age or city differences.

Common mistakes

• Ignoring Order: By default, keep='first' retains the earliest row. If your data is unsorted, you might keep outdated information. Sort the DataFrame before dropping duplicates if necessary.
• Forgetting Assignment: df.drop_duplicates() does not modify df unless inplace=True. Always assign the result: df = df.drop_duplicates().
• Case Sensitivity: "Alice" and "alice" are treated as different. Use str.lower() on relevant columns before deduplication if case-insensitive matching is required.
• NaN Handling: Rows with NaN in the subset columns may behave unexpectedly depending on version. Ensure critical keys are non-null or handle them explicitly.

When to use it

Compare drop_duplicates() with groupby().first():

Use drop_duplicates() when you want to clean raw data. Use groupby() when you need to consolidate information from duplicates into a single record.

Practice

Guided Exercise: Create a DataFrame with columns ID and Status. Add three rows with ID 1 (Status: Active, Pending, Active). Use drop_duplicates(subset=['ID'], keep='last') to see which status remains.

Challenge: How would you remove duplicates based on Email but keep the row with the most recent Login_Date? Hint: Sort by date descending before dropping duplicates.

Quick check

Q: What happens if you set keep=False in drop_duplicates()?
A: All duplicate rows are removed, including the first occurrence. Only completely unique rows remain.

Summary

Pandas drop_duplicates() is essential for cleaning datasets by removing redundant entries based on full-row matches or specific key columns. Mastering the subset and keep parameters allows precise control over which records survive, ensuring accurate downstream analysis.