Reading JSON

By the end of this lesson, you will be able to load JSON data into Python using both the standard library and pandas, handling common file structures correctly.

What it is

JSON (JavaScript Object Notation) is a lightweight format for storing and transporting data. In Python, reading JSON involves parsing text files containing key-value pairs and arrays into native Python objects like dictionaries and lists. The json module handles raw parsing, while pandas extends this capability by converting structured JSON directly into DataFrames for analysis.

Why it matters

• Interoperability: JSON is the universal language for web APIs and configuration files.
• Data Analysis: Converting JSON to DataFrames allows for immediate statistical operations and visualization.
• Simplicity: Human-readable structure makes debugging data pipelines easier than binary formats.
• Flexibility: Supports nested structures that can be flattened or normalized as needed.

Syntax or steps

To read JSON, you generally follow two paths depending on your goal:

1. Standard Library: Open the file in text mode, then use json.load() to parse it into a dictionary or list.
2. Pandas: Use pd.read_json() if the JSON represents tabular data (a list of records or a column-oriented structure).

Example

import json
import pandas as pd

# 1. Standard Library Approach
with open('data.json', 'r') as file:
    data_dict = json.load(file)
    print(type(data_dict)) # Output: <class 'dict'>

# 2. Pandas Approach (Assuming data.json contains a list of records)
# Example content of data.json: [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
df = pd.read_json('data.json')
print(df.head())

In the first block, json.load() reads the entire file object and returns a Python dictionary. This is best when you need to access specific keys manually. In the second block, pd.read_json() automatically infers the structure. If the JSON is a list of objects, pandas creates rows from each object and columns from their keys.

Common mistakes

• Using json.loads() instead of json.load(): loads() expects a string, while load() expects a file object. Passing a file path string to loads() causes an error.
• Ignoring encoding issues: Always specify encoding='utf-8' in open() to prevent errors with special characters.
• Mismatched JSON structure in Pandas: read_json() fails if the JSON is not strictly tabular. Nested objects may require normalize=True or manual flattening.
• Memory overload: Loading massive JSON files entirely into memory can crash scripts. For large datasets, consider streaming parsers or chunking.

When to use it

Use json.load() when you need full control over the data structure or when the JSON is highly irregular. Use pd.read_json() when the data is already flat or semi-flat and you intend to perform analytical operations immediately.

Practice

Guided Exercise: Create a file named sample.json with the content [{"user": "admin", "active": true}, {"user": "guest", "active": false}]. Write a script to load this into a DataFrame and filter for active users.

Challenge: Modify the script to handle a JSON file where the root element is a dictionary containing a key "records" which holds the list of users. Hint: You may need to extract the list before passing it to pandas or use the orient parameter.

Quick check

Question: What is the primary difference between json.load() and json.loads()?

Answer: json.load() parses a JSON document from a file-like object, whereas json.loads() parses a JSON document from a string.

Summary

Reading JSON in Python is straightforward using the built-in json module for general data structures or pandas for tabular data. Choosing the right tool depends on whether you need raw dictionary access or immediate analytical capabilities. Always verify your JSON structure matches the expected input format of your chosen parser.