Loops & List Comprehensions

By the end of this lesson, you will be able to choose between traditional loops and list comprehensions to transform data efficiently in Python.

What it is

In data science, we often need to iterate over datasets to clean, filter, or transform values. Python offers two primary mechanisms for this: loops (for, while) and list comprehensions. Loops are imperative statements that execute code block-by-block. List comprehensions are declarative expressions that generate a new list by applying an expression to each item in an iterable, optionally filtering items based on a condition. The mental model for comprehensions is "mathematical set-builder notation": [expression for item in iterable if condition].

Why it matters

• Readability: Comprehensions often express intent more clearly than multi-line loops for simple transformations.
• Performance: In CPython, list comprehensions are typically faster than equivalent for loops with .append() because they avoid repeated method lookups.
• Data Cleaning: Quickly filter out nulls or outliers from large lists without verbose boilerplate.
• Feature Engineering: Generate derived features (e.g., squared values, normalized scores) in a single line.

Syntax or steps

A basic list comprehension follows this structure:

[output_expression for variable in input_iterable if filter_condition]

Compare this to the equivalent for loop:

1. Initialize an empty list.
2. Iterate through the source data.
3. Check the condition (if applicable).
4. Apply the transformation.
5. Append the result to the list.

Example

Suppose we have a list of raw sensor readings containing negative errors and valid positive measurements. We want to square only the valid positive readings.

raw_readings = [10, -5, 20, -3, 40]

# Traditional Loop Approach
cleaned_loop = []
for x in raw_readings:
    if x > 0:
        cleaned_loop.append(x ** 2)

# List Comprehension Approach
cleaned_comp = [x ** 2 for x in raw_readings if x > 0]

print(cleaned_comp)

Explanation: In the comprehension, x ** 2 is the output expression. for x in raw_readings iterates over the source. if x > 0 filters out negative values before squaring. Both methods produce [100, 400, 1600], but the comprehension is concise and optimized.

Common mistakes

• Over-complicating logic: Do not use nested comprehensions for complex business logic. If your comprehension requires multiple lines of reasoning, revert to a standard function with a loop for readability.
• Modifying the original list: Never try to remove items from a list while iterating over it directly. Always create a new list via comprehension or loop.
• Ignoring memory usage: List comprehensions build the entire list in memory at once. For massive datasets (millions of rows), consider generators (using parentheses instead of brackets) or vectorized operations in libraries like NumPy/Pandas.
• Side effects: Avoid calling functions with side effects (like printing or writing files) inside a comprehension. It makes debugging difficult and violates functional programming principles.

When to use it

Practice

Guided Exercise: Given temps = [72, 85, 60, 90, 75], write a list comprehension to convert Fahrenheit to Celsius using the formula (F - 32) * 5/9, rounded to one decimal place.

Challenge: Create a dictionary comprehension that maps each temperature in temps to its Celsius value.

Hint: Use {key: value for key in iterable}.

Quick check

Q: Why might a list comprehension be preferred over a for loop with .append()?

A: It is generally faster due to internal optimizations and more readable for simple transformations.

Summary

List comprehensions provide a Pythonic, efficient way to transform and filter data sequences. While powerful, they should be reserved for simple operations; complex logic or large-scale numerical data processing is better handled by explicit loops or specialized libraries like Pandas.