Building directly on what Data Exploration surfaces, data cleaning is the practical process of actually fixing or removing the problems found — corrupt files, duplicates, and label errors.
Common Cleaning Tasks
| Issue | Typical Fix |
|---|---|
| Corrupt or unreadable files | Detect and remove (or attempt to repair) programmatically before they crash training mid-run |
| Duplicate or near-duplicate examples | Deduplicate, especially critical across train/validation/test splits, to avoid data leakage |
| Mislabeled examples | Re-label manually if feasible, or remove if the error rate for a specific subset is severe and re-labeling isn't practical |
| Missing values (for tabular/structured features) | Impute, drop, or explicitly encode "missingness" as its own signal, depending on the specific pattern of missingness |
Code — Detecting Corrupt Image Files
from PIL import Image
import os
def find_corrupt_images(image_dir):
corrupt_files = []
for filename in os.listdir(image_dir):
filepath = os.path.join(image_dir, filename)
try:
img = Image.open(filepath)
img.verify() # checks the file is a valid, readable image without fully decoding it
except Exception:
corrupt_files.append(filepath)
return corrupt_files
corrupt = find_corrupt_images("images/")
print(f"Found {len(corrupt)} corrupt files")Code — Detecting Near-Duplicates Across Splits
import hashlib
def file_hash(filepath):
with open(filepath, 'rb') as f:
return hashlib.md5(f.read()).hexdigest()
train_hashes = {file_hash(f) for f in train_files}
test_hashes = {file_hash(f) for f in test_files}
overlap = train_hashes & test_hashes
if overlap:
print(f"WARNING: {len(overlap)} exact duplicate files found across train and test sets!")This exact check directly prevents the data leakage problem flagged in Dataset Train/Val/Test Split — a test example that's an exact (or near) duplicate of a training example produces an artificially inflated, misleading test performance estimate.
Common Mistakes
- Cleaning data after splitting into train/val/test without re-checking for duplicates across the splits — cleaning steps should be applied consistently, with the split-leakage check specifically done across the final splits, not just within the raw combined dataset.
- Removing every example with any detected imperfection indiscriminately, rather than assessing severity — sometimes a minor, correctable issue is better fixed than the example simply discarded, especially if data is limited.
Interview Relevance
Q: "Why is checking for duplicate examples across your train/validation/test splits specifically important, not just within the raw dataset?" If the same (or a near-identical) example appears in both the training set and the test set, the model may effectively be evaluated on data it has already memorized during training — producing an artificially inflated, misleading test performance estimate that won't hold up on genuinely new, unseen data. This directly undermines the entire purpose of holding out a test set, exactly the concern flagged in Dataset Train/Val/Test Split.
Practice Question
Why might it be better to fix a minor, correctable labeling error rather than simply discarding the affected example, especially in a small dataset?
