By the end of this lesson, you will be able to load a CSV file into Python using pandas and inspect its structure, data types, and summary statistics.
What it is
Working with data in Python typically involves the pandas library, which provides powerful data structures like the DataFrame. A DataFrame is essentially a table where rows represent records and columns represent variables. Loading data means reading external files (like CSV or Excel) into memory, while inspecting involves checking dimensions, column names, data types, and missing values before analysis begins.
Why it matters
- Quality Control: Inspecting data reveals errors, such as incorrect data types or unexpected missing values, early in the process.
- Efficiency: Understanding the shape and size of your dataset helps you choose appropriate algorithms and memory management strategies.
- Context: Summary statistics provide an immediate overview of distributions, helping you identify outliers or skewness.
- Reproducibility: Standardized loading and inspection steps ensure that analyses can be repeated reliably by others.
Syntax or steps
- Import the
pandaslibrary. - Use
pd.read_csv()to load a file into a DataFrame variable. - Call
.head()to view the first few rows. - Call
.info()to check column names, non-null counts, and data types. - Call
.describe()for numerical summary statistics.
Example
import pandas as pd
# 1. Load the dataset
df = pd.read_csv('sales_data.csv')
# 2. Inspect the first 5 rows
print("First 5 Rows:")
print(df.head())
# 3. Check structure and data types
print("\nData Info:")
df.info()
# 4. Get summary statistics for numeric columns
print("\nSummary Statistics:")
print(df.describe())Explanation: The code imports pandas as pd. It reads 'sales_data.csv' into df. df.head() displays the top rows to verify content. df.info() lists each column's name, count of non-null entries, and dtype (e.g., int64, float64, object). df.describe() calculates mean, standard deviation, min/max, and quartiles for numeric columns only.
Common mistakes
- Ignoring Data Types: Treating IDs as numbers instead of strings can lead to incorrect calculations. Always check
df.dtypes. - Assuming Clean Data: Failing to check for nulls via
df.isnull().sum()can cause errors in later modeling steps. - Overloading Memory: Loading entire large datasets without specifying necessary columns (
usecols) can crash low-memory systems. - Misinterpreting Indexes: Pandas adds a default integer index unless specified; do not confuse this with actual data columns.
When to use it
| Scenario | Recommended Tool | Reason |
|---|---|---|
| Small to Medium Tabular Data | Pandas | Rich API for cleaning, grouping, and joining. |
| Huge Datasets (Big Data) | Spark / Dask | Distributed processing handles data larger than RAM. |
| Simple File Reading | Python csv module | No dependency on heavy libraries like pandas. |
Practice
Guided Exercise: Create a small CSV file named test.csv with three columns: Name, Age, Score. Load it using pandas and print the average age.
Challenge: Modify the code to drop any rows where 'Score' is missing, then re-run .info() to confirm the row count decreased.
Quick check
Question: Which method would you use to see the number of rows and columns in a DataFrame?
Answer: Use the .shape attribute (e.g., df.shape), which returns a tuple like (rows, columns).
Summary
Loading and inspecting data are foundational steps in any Python analytics workflow. Using pandas, you can quickly assess data quality, structure, and statistical properties, ensuring that subsequent analysis is built on a reliable foundation.
