Batch inference runs a model on a large accumulated set of inputs at once, on a schedule, rather than responding to individual requests as they arrive — the right choice whenever real-time responses aren't actually required.
When Batch Inference Is the Right Choice
| Example Use Case | Why Batch Fits |
|---|---|
| Nightly product recommendation refresh for all users | Predictions don't need to reflect the last few minutes of activity — a daily refresh is entirely adequate |
| Periodic churn-risk scoring across a customer base | Business decisions based on this score are made on a similarly relaxed timescale |
| Processing a large archive of documents for classification | No individual request is waiting on any single document's result in real time |
Code — A Simple Batch Inference Job
import torch
from torch.utils.data import DataLoader
model.eval()
dataset = load_full_dataset_to_score() # e.g. all users, all documents
loader = DataLoader(dataset, batch_size=256, shuffle=False)
all_predictions = []
with torch.no_grad():
for batch in loader:
predictions = model(batch)
all_predictions.append(predictions)
all_predictions = torch.cat(all_predictions)
save_predictions_to_storage(all_predictions) # e.g. write to a database or data warehouseWhy Batch Inference Is Often Simpler and More Efficient
Without a real-time latency constraint, batch inference can use much larger batch sizes than real-time serving typically allows, maximizing hardware (especially GPU) utilization and overall throughput. It also avoids the operational complexity of maintaining an always-available, low-latency serving endpoint — the job simply runs, completes, and the results are stored for later use.
Batch Inference Infrastructure Patterns
Batch jobs are commonly scheduled via a workflow orchestrator (e.g. Airflow, or a simple cron job) and run on compute that only needs to exist for the duration of the job — a good fit for the cost-efficient, transient compute patterns available on most cloud platforms, since there's no need to keep an endpoint running and staffed with capacity around the clock.
Common Mistakes
- Building real-time serving infrastructure for a use case that would be perfectly well served, more simply and cheaply, by a scheduled batch job — real-time serving carries meaningfully more operational complexity and cost that should be justified by an actual need for low latency.
- Using a small batch size for a batch inference job out of habit (carried over from real-time serving code) when there's no latency constraint requiring it — this leaves throughput on the table unnecessarily.
Interview Relevance
Q: "Why would you choose batch inference over real-time inference for a use case like nightly product recommendations, even though real-time serving is technically possible?" Real-time serving carries meaningfully more operational complexity and cost — maintaining an always-available, low-latency endpoint — that isn't justified when the actual use case (recommendations refreshed once a day) doesn't require immediate responses. Batch inference can also use much larger batch sizes than real-time serving typically allows, since there's no per-request latency constraint, maximizing hardware utilization and overall throughput for the same total workload.
Practice Question
Why can batch inference typically use larger batch sizes than real-time inference serving, and why does that matter for throughput?
