Filtering Arrays

By the end of this lesson, you will be able to select specific elements from a NumPy array based on logical conditions using boolean indexing.

What it is

Filtering arrays in Python, specifically within the numpy library, refers to the process of extracting a subset of data that meets certain criteria. The core mechanism is boolean indexing. When you apply a condition (like a > 2) to an array, NumPy generates a temporary "mask" array of True and False values. This mask is then used to select only the elements where the condition was true.

Related terms include logical operators (&, |, ~) and vectorization, which allows these operations to happen efficiently across entire arrays without explicit loops.

Why it matters

• Performance: Boolean indexing is implemented in C under the hood, making it significantly faster than iterating through lists with Python loops.
• Data Cleaning: It allows you to easily remove outliers or invalid entries (e.g., negative values in a dataset meant for counts).
• Analysis: You can quickly isolate specific segments of data for statistical analysis, such as finding all sales above a certain threshold.
• Readability: The syntax closely resembles mathematical notation, making code easier to understand at a glance.

Syntax or steps

The basic pattern for filtering a NumPy array arr with a condition cond is:

filtered_arr = arr[cond]

To combine multiple conditions, use bitwise operators instead of logical ones:

• & for AND (both conditions must be true)
• | for OR (at least one condition must be true)
• ~ for NOT (invert the condition)

Note: Each individual condition must be wrapped in parentheses when combining them.

Example

import numpy as np

# Create a sample array
data = np.array([10, 25, 30, 45, 60, 75])

# Filter: Select elements greater than 30
high_values = data[data > 30]
print("Values > 30:", high_values)

# Filter: Select elements between 20 and 50 (inclusive)
# Note the parentheses around each condition
mid_range = data[(data >= 20) & (data <= 50)]
print("Values 20-50:", mid_range)

# Filter: Select elements NOT equal to 45
not_45 = data[data != 45]
print("Not 45:", not_45)

Explanation:

1. data > 30 creates a boolean mask [False, False, False, True, True, True].
2. data[mask] returns only the elements corresponding to True.
3. In the second example, (data >= 20) & (data <= 50) combines two masks. The & operator ensures both conditions are met simultaneously.

Common mistakes

• Using Python keywords: Using and or or instead of & and |. This raises a ValueError because Python cannot evaluate the truth value of an array.
• Missing Parentheses: Writing data[data > 20 & data < 50] fails due to operator precedence. Always wrap conditions: data[(data > 20) & (data < 50)].
• Modifying the Original Array Accidentally: Filtering creates a copy by default. If you want to modify the original array in place, use assignment: data[data > 30] = 0.
• Confusing Index Position with Value: Remember that data[data > 30] filters by value. To filter by index position, use np.where() or slice notation.

When to use it

Boolean indexing is best for simple, direct value-based filtering. For more complex logic or when you need the indices of matching elements, consider alternatives.

Practice

Guided Exercise: Given temps = np.array([-5, 0, 15, 22, 30, -2]), write code to extract all temperatures that are freezing or below (<= 0).

Hint: Use temps[temps <= 0].

Challenge: Extract all temperatures that are strictly positive but less than 25 degrees.

Solution Hint: Combine conditions with &: temps[(temps > 0) & (temps < 25)].

Quick check

Question: Why does arr[arr > 5 and arr < 10] fail?

Answer: The Python keyword and expects single boolean values, not arrays. NumPy requires the bitwise operator & to perform element-wise logical AND operations on arrays.

Summary

Boolean indexing provides a powerful, vectorized way to filter NumPy arrays by applying logical conditions directly to the data. By mastering the use of bitwise operators and proper parenthesization, you can efficiently clean and analyze large datasets without resorting to slow Python loops.